-----============= acceptance-small: sanity-flr ============----- Fri Sep 11 10:08:18 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg638-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:08:44 (1789135724) === oleg638-client.virtnet: executing check_config_client /mnt/lustre oleg638-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg638-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8f6244d4a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f6244d4a000.idle_timeout=debug disable quota as required oleg638-server: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Setting lod.lustre-MDT0000-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Updated after 3s: want '1' got '1' === sanity-flr: finish setup 10:09:32 (1789135772) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7627] == sanity-flr test 0a: lfs mirror create with -N option == 10:09:36 (1789135776) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:09:51 (1789135791) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:09:54 (1789135794) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:09:57 (1789135797) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:10:22 (1789135822) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:10:25 (1789135825) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 10:10:29 (1789135829) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8f6244d4a000.stats=clear osc.lustre-OST0001-osc-ffff8f6244d4a000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:10:37 (1789135837) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:10:47 (1789135847) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:10:54 (1789135854) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:10:57 (1789135857) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:11:04 (1789135864) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 10:11:13 (1789135873) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:11:20 (1789135880) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:11:23 (1789135883) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:11:30 (1789135890) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:11:40 (1789135900) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:11:47 (1789135907) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.669312 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159743 s, 6.6 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:11:57 (1789135917) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.732845 s, 28.6 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:12:07 (1789135927) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:12:17 (1789135937) Stopping /mnt/lustre-ost1 (opts:) on oleg638-server oleg638-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f6244d4a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f6244d4a000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg638-server: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg638-client: oleg638-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg638-server: oleg638-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg638-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg638-server oleg638-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f6244d4a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f6244d4a000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg638-server: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg638-client: oleg638-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg638-server: oleg638-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg638-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (72s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:13:30 (1789136010) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.36204 s, 17.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg638-server oleg638-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f6244d4a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f6244d4a000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg638-server: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg638-client: oleg638-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg638-server: oleg638-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg638-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:14:09 (1789136049) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg638-server oleg638-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f6244d4a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f6244d4a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg638-server: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg638-client: oleg638-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg638-server: oleg638-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg638-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg638-server oleg638-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f6244d4a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f6244d4a000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg638-server: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg638-client: oleg638-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg638-server: oleg638-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg638-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:15:32 (1789136132) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.52418 s, 9.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 11 10:15 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.037s user 0m0.000s sys 0m0.184s Stopping /mnt/lustre-ost1 (opts:) on oleg638-server oleg638-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f6244d4a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f6244d4a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.214s user 0m0.001s sys 0m0.182s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg638-server: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg638-client: oleg638-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg638-server: oleg638-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg638-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg638-server oleg638-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f6244d4a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f6244d4a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.265s user 0m0.000s sys 0m0.147s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg638-server: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg638-client: oleg638-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg638-server: oleg638-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg638-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg638-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f6244d4a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f6244d4a000.ost_server_uuid in FULL state after 0 sec PASS 33b (91s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:17:03 (1789136223) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:17:07 (1789136227) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:17:10 (1789136230) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 10:17:14 (1789136234) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 10:17:22 (1789136242) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.476464 s, 26.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:17:34 (1789136254) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244009 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033568 s, 31.2 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:17:42 (1789136262) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:17:49 (1789136269) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00132023 s, 776 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000897489 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000819171 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00121765 s, 841 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 10:17:58 (1789136278) create test files with size 15754876 .. 60fe5b1bd08dda38759b23f53d1c0de4 - 0cdf2b9803e571adfb3e34a7963fbccc - d91b8230b6336df7e1a827d539a16555 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg638-client.virtnet /mnt/lustre (opts:) Starting client: oleg638-client.virtnet: -o user_xattr,flock 192.168.206.138@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 10:18:18 (1789136298) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:18:33 (1789136313) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:18:40 (1789136320) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958758 s, 10.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529162 s, 19.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 10:18:50 (1789136330) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165628 s, 12.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.441777 s, 9.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.2437 s, 8.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 10:19:07 (1789136347) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 10:19:10 (1789136350) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:19:13 (1789136353) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418119 s, 25.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 10:19:21 (1789136361) llite.lustre-ffff8f6246424000.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 10:19:28 (1789136368) llite.lustre-ffff8f6246424000.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349752 s, 30.0 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521511 s, 20.1 MB/s **verify components PASS 43d (7s) == sanity-flr test 44a: lfs mirror split check =========== 10:19:35 (1789136375) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:19:38 (1789136378) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xca:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xca:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:19:48 (1789136388) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.253354 s, 41.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg638-client.virtnet /mnt/lustre (opts:) Starting client: oleg638-client.virtnet: -o user_xattr,flock 192.168.206.138@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:19:59 (1789136399) llite.lustre-ffff8f62453c5800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.277155 s, 37.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg638-client.virtnet /mnt/lustre (opts:) Starting client: oleg638-client.virtnet: -o user_xattr,flock 192.168.206.138@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8f6243853800.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:20:11 (1789136411) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:20:57 (1789136457) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:21:32 (1789136492) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 10:21:37 (1789136497) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.286938 s, 11.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.1684 s, 18.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa7d86a80 (nosync,stale) Mirror 2: 0xa7d86a80 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 34dfe8783aeaa8b198d7a8b946b2c076 - mirror 1 checksum: 65af777da5895e4c23a446f7b01221c7 - mirror 2 checksum: 34dfe8783aeaa8b198d7a8b946b2c076 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T10:21:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T10:21:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 34dfe8783aeaa8b198d7a8b946b2c076 - mirror 1 checksum: 65af777da5895e4c23a446f7b01221c7 - mirror 2 checksum: 65af777da5895e4c23a446f7b01221c7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T10:21:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T10:21:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:21:52 (1789136512) Failing mds1 on oleg638-server Stopping /mnt/lustre-mds1 (opts:) on oleg638-server 10:21:59 (1789136519) shut down facet: mds1 facet_host: oleg638-server facet_failover_host: oleg638-server Failover mds1 to oleg638-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg638-server: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg638-client: oleg638-server: ssh exited with exit code 1 Started lustre-MDT0000 10:22:30 (1789136550) targets are mounted 10:22:30 (1789136550) facet_failover done oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472862 s, 22.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe1:0x0] } PASS 50A (59s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:22:51 (1789136571) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:22:55 (1789136575) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:22:57 (1789136577) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00498806 s, 3.3 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:23:04 (1789136584) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:23:06 (1789136586) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0101133 s, 405 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0549946 s, 74.5 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:23:13 (1789136593) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:23:32 (1789136612) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:23:58 (1789136638) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:24:20 (1789136660) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:24:43 (1789136683) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 10:24:49 (1789136689) mirror create pid 55771 mirror split pid 55772 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55771 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55772 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 10:25:55 (1789136755) Seed set to 6757 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1789136757.653597 write 0x00de42 thru 0x012aa0 (0x4c5f bytes) 000005 1789136757.656380 write 0x02ae5c thru 0x0326cf (0x7874 bytes) 000012 1789136757.677849 read 0x027298 thru 0x02b0fa (0x3e63 bytes) 000016 1789136757.688096 write 0x0141e4 thru 0x017052 (0x2e6f bytes) 000019 1789136757.710398 mapread 0x01d2f3 thru 0x021066 (0x3d74 bytes) 000020 1789136757.742480 mapwrite 0x0171e9 thru 0x0237be (0xc5d6 bytes) 000021 1789136757.811584 MIRROR_EXTEND to 2 mirrors 000022 1789136758.661538 mapread 0x02c545 thru 0x0326cf (0x618b bytes) truncating to largest ever: 0xf76e 000023 1789136758.704493 trunc from 0x0326d0 to 0x00f76d (0x22f62 bytes) 000024 1789136758.787041 mapread 0x008fee thru 0x00f76d (0x6780 bytes) 000027 1789136758.812160 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) 000029 1789136759.019180 read 0x00bb45 thru 0x00e850 (0x2d0c bytes) 000030 1789136759.028561 mapwrite 0x03792d thru 0x03888e (0xf62 bytes) 000031 1789136759.137984 mapwrite 0x001e8f thru 0x007229 (0x539b bytes) 000033 1789136759.229317 mapread 0x01d79b thru 0x02b809 (0xe06f bytes) 000034 1789136759.268229 mapwrite 0x017f04 thru 0x02360b (0xb708 bytes) 000035 1789136759.321185 MIRROR_EXTEND to 2 mirrors 000036 1789136760.166001 write 0x025001 thru 0x02fcd3 (0xacd3 bytes) 000037 1789136760.329807 write 0x0357ed thru 0x035ba6 (0x3ba bytes) 000038 1789136760.338591 write 0x015e2d thru 0x0174f8 (0x16cc bytes) 000039 1789136760.349015 mapwrite 0x02e1e0 thru 0x02fed0 (0x1cf1 bytes) truncating to largest ever: 0x22b8c 000040 1789136760.444907 trunc from 0x03888f to 0x022b8b (0x15d03 bytes) 000041 1789136760.524766 MIRROR_RESYNC 2 mirrors 000042 1789136761.306534 MIRROR_RESYNC 2 mirrors 000043 1789136761.760109 mapwrite 0x0319ab thru 0x03d8dd (0xbf33 bytes) 000044 1789136761.921747 read 0x0203d2 thru 0x020b7c (0x7ab bytes) 000047 1789136761.929802 write 0x016bd7 thru 0x0264d4 (0xf8fe bytes) 000049 1789136761.943628 trunc from 0x03d8de to 0x011b7e (0x2bd5f bytes) 000051 1789136761.965754 write 0x00c36e thru 0x016948 (0xa5db bytes) 000052 1789136761.978072 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x16949): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16949): Mirror 2: 0x252cd56e (stale) CRC-32 checksum value for chunk [0, 0x16949): Mirror 1: 0x4b7ed317 Mirror 3: 0x4b7ed317 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16949: skipped 000055 1789136762.875978 mapwrite 0x009530 thru 0x012143 (0x8c14 bytes) 000058 1789136763.079616 trunc from 0x016949 to 0x009c02 (0xcd46 bytes) 000059 1789136763.123403 write 0x00cb09 thru 0x00e707 (0x1bff bytes) 000061 1789136763.125040 read 0x002b44 thru 0x005ced (0x31aa bytes) 000062 1789136763.136615 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xe708): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe708): Mirror 2: 0x7b16d51d (stale) Mirror 3: 0xfedb0342 (stale) CRC-32 checksum value for chunk [0, 0xe708): Mirror 1: 0x75a7b335 Mirror 4: 0x75a7b335 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe708: skipped 000066 1789136764.017245 trunc from 0x00e708 to 0x014970 (0x6269 bytes) 000067 1789136764.131821 mapwrite 0x03ad49 thru 0x03fe22 (0x50da bytes) 000068 1789136764.244693 mapread 0x03dc35 thru 0x03fe22 (0x21ee bytes) 000069 1789136764.247515 mapread 0x03a92f thru 0x03fe22 (0x54f4 bytes) 000072 1789136764.253176 mapread 0x035642 thru 0x03fe22 (0xa7e1 bytes) truncating to largest ever: 0x23aea 000073 1789136764.254832 trunc from 0x03fe23 to 0x023ae9 (0x1c339 bytes) 000074 1789136764.293207 trunc from 0x023aea to 0x02230d (0x17dc bytes) 000076 1789136764.313019 trunc from 0x02230e to 0x0192da (0x9033 bytes) 000078 1789136764.333360 write 0x0251b5 thru 0x031184 (0xbfd0 bytes) 000079 1789136764.340539 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31185): Mirror 2: 0xdcff3a35 (stale) Mirror 3: 0x23739988 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31185): Mirror 2: 0xdcff3a35 (stale) Mirror 3: 0x23739988 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31185: skipped 000080 1789136764.757436 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31185): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31185): Mirror 2: 0xdcff3a35 (stale) Mirror 3: 0x23739988 (stale) CRC-32 checksum value for chunk [0, 0x31185): Mirror 1: 0xafe09503 Mirror 4: 0xafe09503 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31185: skipped 000081 1789136765.846710 write 0x002c9d thru 0x007a01 (0x4d65 bytes) 000082 1789136765.942098 read 0x013539 thru 0x019b3b (0x6603 bytes) truncating to largest ever: 0x266e7 000083 1789136765.956700 trunc from 0x031185 to 0x0266e6 (0xaa9e bytes) truncating to largest ever: 0x38d6b 000087 1789136765.984828 trunc from 0x0266e7 to 0x038d6a (0x12684 bytes) 000089 1789136766.021617 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38d6b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d6b): Mirror 2: 0xbea19473 (stale) Mirror 3: 0x578e76ee (stale) Mirror 4: 0x44c0ee30 (stale) CRC-32 checksum value for chunk [0, 0x38d6b): Mirror 1: 0x1045c15e Mirror 5: 0x1045c15e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d6b: skipped 000090 1789136767.196705 mapwrite 0x01d7fc thru 0x0252b4 (0x7ab9 bytes) 000091 1789136767.508567 trunc from 0x038d6b to 0x037a38 (0x1332 bytes) 000092 1789136767.554364 write 0x00346d thru 0x00d682 (0xa216 bytes) 000094 1789136767.566640 write 0x027e35 thru 0x02996e (0x1b3a bytes) 000096 1789136767.582017 read 0x006da5 thru 0x00d216 (0x6472 bytes) 000097 1789136767.584594 mapwrite 0x0209dd thru 0x0220cd (0x16f1 bytes) 000103 1789136767.660170 mapread 0x018aab thru 0x0230cf (0xa625 bytes) 000104 1789136767.664718 write 0x00c40f thru 0x01b255 (0xee47 bytes) 000107 1789136767.681119 MIRROR_RESYNC 5 mirrors 000110 1789136769.251352 mapwrite 0x01b03e thru 0x021aec (0x6aaf bytes) 000111 1789136769.429589 mapread 0x003c75 thru 0x0116b9 (0xda45 bytes) 000112 1789136769.449786 write 0x00c50c thru 0x00f735 (0x322a bytes) 000113 1789136769.465526 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37a39): Mirror 2: 0x99131e96 (stale) Mirror 3: 0x99131e96 (stale) Mirror 5: 0x99131e96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a39): Mirror 2: 0x99131e96 (stale) Mirror 3: 0x99131e96 (stale) Mirror 5: 0x99131e96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a39: skipped 000114 1789136769.893555 write 0x00da94 thru 0x00ffe9 (0x2556 bytes) 000116 1789136769.905570 mapread 0x01c273 thru 0x023f2b (0x7cb9 bytes) 000118 1789136769.946548 mapwrite 0x01940e thru 0x022679 (0x926c bytes) 000119 1789136770.009043 trunc from 0x037a39 to 0x02cdb2 (0xac86 bytes) 000120 1789136770.028211 read 0x00e1a2 thru 0x0167c2 (0x8621 bytes) 000121 1789136770.046364 MIRROR_RESYNC 4 mirrors 000122 1789136770.965142 read 0x019f86 thru 0x01fd7d (0x5df8 bytes) 000123 1789136770.974502 mapwrite 0x0089c3 thru 0x017cf8 (0xf336 bytes) 000125 1789136771.132993 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cdb3): Mirror 3: 0xe9b014d (stale) Mirror 5: 0xe9b014d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cdb3): Mirror 3: 0xe9b014d (stale) Mirror 5: 0xe9b014d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cdb3: skipped 000126 1789136771.458799 mapwrite 0x00c82f thru 0x01248f (0x5c61 bytes) 000127 1789136771.584615 MIRROR_RESYNC 3 mirrors 000129 1789136772.599858 mapwrite 0x03e4c5 thru 0x03ffff (0x1b3b bytes) 000131 1789136772.767536 mapwrite 0x01b757 thru 0x01f247 (0x3af1 bytes) 000132 1789136772.838206 trunc from 0x040000 to 0x0344ac (0xbb53 bytes) 000134 1789136772.866866 write 0x03013b thru 0x03d71e (0xd5e4 bytes) 000135 1789136772.877107 mapwrite 0x02e0a8 thru 0x03d936 (0xf88f bytes) truncating to largest ever: 0x3b0ba 000138 1789136772.977490 trunc from 0x03d937 to 0x03b0b9 (0x287d bytes) 000142 1789136773.003584 trunc from 0x03b0ba to 0x018f01 (0x221b8 bytes) 000144 1789136773.020445 read 0x018d76 thru 0x018f01 (0x18c bytes) 000147 1789136773.023504 mapread 0x015615 thru 0x018733 (0x311f bytes) 000149 1789136773.027973 write 0x03c519 thru 0x03ffff (0x3ae7 bytes) 000151 1789136773.032027 mapread 0x00579f thru 0x00c00a (0x686c bytes) 000153 1789136773.050384 trunc from 0x040000 to 0x0183ad (0x27c52 bytes) 000155 1789136773.073595 MIRROR_RESYNC 3 mirrors 000159 1789136773.991048 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) 000160 1789136774.432445 mapread 0x00c495 thru 0x017119 (0xac85 bytes) 000161 1789136774.457524 write 0x0230cd thru 0x0304ae (0xd3e2 bytes) 000162 1789136774.523511 write 0x03b86e thru 0x03f6dc (0x3e6f bytes) 000163 1789136774.532721 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) 000165 1789136774.741285 read 0x01b776 thru 0x020e17 (0x56a2 bytes) 000169 1789136774.763822 read 0x0236a2 thru 0x02c681 (0x8fe0 bytes) 000171 1789136774.792570 read 0x0085ac thru 0x0126ab (0xa100 bytes) 000172 1789136774.794704 mapwrite 0x02afc1 thru 0x032bc4 (0x7c04 bytes) 000176 1789136774.882945 mapread 0x02399b thru 0x02403c (0x6a2 bytes) 000184 1789136774.890338 write 0x03fcc3 thru 0x03ffff (0x33d bytes) 000185 1789136774.891310 mapwrite 0x030cd5 thru 0x03c50f (0xb83b bytes) 000186 1789136774.936616 mapread 0x028dee thru 0x02aa7b (0x1c8e bytes) 000188 1789136774.943715 MIRROR_EXTEND to 2 mirrors 000190 1789136775.827011 mapwrite 0x034366 thru 0x03ffff (0xbc9a bytes) 000192 1789136776.054283 write 0x011a4e thru 0x01beda (0xa48d bytes) 000195 1789136776.066187 trunc from 0x040000 to 0x037428 (0x8bd7 bytes) 000197 1789136776.111324 trunc from 0x037429 to 0x027d81 (0xf6a7 bytes) 000198 1789136776.174777 mapwrite 0x01d7a5 thru 0x02509f (0x78fb bytes) 000199 1789136776.301606 MIRROR_RESYNC 2 mirrors 000203 1789136777.002068 write 0x03f8a1 thru 0x03ffff (0x75f bytes) 000204 1789136777.069059 read 0x024142 thru 0x030b6f (0xca2e bytes) 000207 1789136777.080160 trunc from 0x040000 to 0x0166e0 (0x2991f bytes) 000210 1789136777.133336 mapread 0x005e3e thru 0x00d624 (0x77e7 bytes) 000211 1789136777.162216 read 0x010477 thru 0x0166e0 (0x626a bytes) 000212 1789136777.174073 write 0x02018d thru 0x0216c3 (0x1537 bytes) 000214 1789136777.176680 write 0x02017f thru 0x02778c (0x760e bytes) 000216 1789136777.178736 mapread 0x008fec thru 0x011e69 (0x8e7e bytes) 000217 1789136777.180498 mapwrite 0x0127d2 thru 0x0133bb (0xbea bytes) 000219 1789136777.225420 trunc from 0x02778d to 0x034be0 (0xd454 bytes) 000221 1789136777.243567 read 0x00c0b6 thru 0x00d33c (0x1287 bytes) 000222 1789136777.245468 mapwrite 0x00a4b0 thru 0x01614f (0xbca0 bytes) 000223 1789136777.320158 read 0x018fb6 thru 0x025332 (0xc37d bytes) 000224 1789136777.340229 write 0x003f99 thru 0x005326 (0x138e bytes) 000225 1789136777.342014 mapread 0x034a45 thru 0x034be0 (0x19c bytes) 000226 1789136777.347424 MIRROR_RESYNC 2 mirrors 000228 1789136778.126017 write 0x02700e thru 0x03458d (0xd580 bytes) 000230 1789136778.199509 mapwrite 0x01de9b thru 0x023801 (0x5967 bytes) 000231 1789136778.273346 read 0x024a0c thru 0x02c501 (0x7af6 bytes) 000232 1789136778.273940 write 0x01e6f9 thru 0x028043 (0x994b bytes) 000233 1789136778.275030 mapread 0x007a3c thru 0x00a8e7 (0x2eac bytes) 000234 1789136778.284222 mapwrite 0x024821 thru 0x027e64 (0x3644 bytes) 000236 1789136778.320418 MIRROR_RESYNC 2 mirrors 000237 1789136778.967579 MIRROR_EXTEND to 3 mirrors 000239 1789136779.596310 read 0x03045f thru 0x03096f (0x511 bytes) 000240 1789136779.601104 trunc from 0x034be1 to 0x00f52f (0x256b1 bytes) 000242 1789136779.688538 read 0x0047c0 thru 0x0098dd (0x511e bytes) 000243 1789136779.700216 read 0x000507 thru 0x00154b (0x1045 bytes) 000244 1789136779.701322 read 0x00bf88 thru 0x00e036 (0x20af bytes) 000251 1789136779.705521 mapread 0x001478 thru 0x007c0c (0x6795 bytes) 000252 1789136779.706770 trunc from 0x00f530 to 0x026b51 (0x17622 bytes) 000253 1789136779.735774 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26b52): Mirror 3: 0xd6e55e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26b52): Mirror 3: 0xd6e55e8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26b52: skipped 000254 1789136780.101773 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26b52): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26b52): Mirror 3: 0xd6e55e8 (stale) CRC-32 checksum value for chunk [0, 0x26b52): Mirror 1: 0x60288bb9 Mirror 4: 0x60288bb9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26b52: skipped 000255 1789136781.164239 write 0x024824 thru 0x027946 (0x3123 bytes) 000256 1789136781.282721 read 0x00dbf3 thru 0x01c974 (0xed82 bytes) 000257 1789136781.320852 mapread 0x00f22a thru 0x0145fe (0x53d5 bytes) 000259 1789136781.322193 read 0x019184 thru 0x01ddde (0x4c5b bytes) 000261 1789136781.347763 mapwrite 0x025603 thru 0x02d27e (0x7c7c bytes) 000262 1789136781.435875 write 0x035a52 thru 0x03ffff (0xa5ae bytes) 000263 1789136781.444318 mapwrite 0x015171 thru 0x02404b (0xeedb bytes) 000264 1789136781.510966 write 0x00d5c1 thru 0x01168e (0x40ce bytes) 000266 1789136781.512830 write 0x013a50 thru 0x01e23b (0xa7ec bytes) 000268 1789136781.515363 trunc from 0x040000 to 0x02d03f (0x12fc0 bytes) 000269 1789136781.535122 mapwrite 0x01f471 thru 0x0254d7 (0x6067 bytes) 000272 1789136781.594637 mapread 0x01afe6 thru 0x01d14f (0x216a bytes) 000273 1789136781.600408 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d040): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d040): Mirror 3: 0x330734ac (stale) Mirror 4: 0xcc5beba (stale) CRC-32 checksum value for chunk [0, 0x2d040): Mirror 1: 0x2218ad44 Mirror 5: 0x2218ad44 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d040: skipped 000274 1789136782.376023 mapread 0x010383 thru 0x01d4bd (0xd13b bytes) 000278 1789136782.406511 mapwrite 0x00fed1 thru 0x013189 (0x32b9 bytes) 000280 1789136782.640065 write 0x021e3a thru 0x0251b8 (0x337f bytes) 000282 1789136782.658576 write 0x031391 thru 0x033d01 (0x2971 bytes) 000283 1789136782.662283 mapwrite 0x034c8a thru 0x03d7ac (0x8b23 bytes) 000285 1789136782.780483 mapwrite 0x019a5e thru 0x028ed7 (0xf47a bytes) 000289 1789136782.877783 trunc from 0x03d7ad to 0x0235ef (0x1a1bd bytes) 000291 1789136782.911476 mapwrite 0x0307d3 thru 0x03d1c0 (0xc9ee bytes) 000292 1789136783.031172 write 0x011875 thru 0x01eddb (0xd567 bytes) 000295 1789136783.034150 write 0x011415 thru 0x01d4aa (0xc096 bytes) 000296 1789136783.035546 trunc from 0x03d1c1 to 0x0391f3 (0x3fcd bytes) 000297 1789136783.056395 read 0x02d851 thru 0x0391f3 (0xb9a3 bytes) 000298 1789136783.057780 trunc from 0x0391f4 to 0x01f28e (0x19f65 bytes) 000299 1789136783.081745 trunc from 0x01f28f to 0x011d16 (0xd578 bytes) 000300 1789136783.098308 mapwrite 0x00a2f2 thru 0x018f5a (0xec69 bytes) 000301 1789136783.193548 mapwrite 0x037e3d thru 0x03f690 (0x7854 bytes) 000303 1789136783.322907 trunc from 0x03f691 to 0x0344ee (0xb1a2 bytes) 000305 1789136783.349598 mapread 0x01ee83 thru 0x02e58d (0xf70b bytes) 000306 1789136783.378229 mapwrite 0x00c870 thru 0x0103f6 (0x3b87 bytes) 000308 1789136783.444134 mapwrite 0x028bdf thru 0x02f459 (0x687b bytes) 000309 1789136783.491714 write 0x012fed thru 0x0220ba (0xf0ce bytes) 000310 1789136783.493794 trunc from 0x0344ef to 0x01ab31 (0x199bd bytes) 000311 1789136783.515778 write 0x013fc1 thru 0x022b95 (0xebd5 bytes) 000313 1789136783.518115 write 0x03691a thru 0x03f482 (0x8b69 bytes) 000315 1789136783.520234 mapread 0x01532e thru 0x018815 (0x34e8 bytes) 000319 1789136783.523405 read 0x00242e thru 0x00fe5d (0xda30 bytes) 000320 1789136783.531039 write 0x014415 thru 0x01f792 (0xb37e bytes) 000321 1789136783.532327 trunc from 0x03f483 to 0x019b37 (0x2594b bytes) 000322 1789136783.548978 mapread 0x01558b thru 0x019b37 (0x45ad bytes) 000324 1789136783.550407 mapread 0x006e9f thru 0x007cb5 (0xe17 bytes) 000325 1789136783.551134 write 0x037e37 thru 0x03e116 (0x62e0 bytes) 000326 1789136783.552755 trunc from 0x03e117 to 0x0110df (0x2d037 bytes) 000327 1789136783.573549 write 0x017e55 thru 0x0277c2 (0xf96e bytes) 000328 1789136783.582019 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x277c3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x277c3): Mirror 3: 0x5051d01b (stale) Mirror 4: 0x5573f978 (stale) Mirror 5: 0x3d6b04c (stale) CRC-32 checksum value for chunk [0, 0x277c3): Mirror 1: 0xc8ff4f8 Mirror 6: 0xc8ff4f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x277c3: skipped 000329 1789136784.385974 trunc from 0x0277c3 to 0x033c0c (0xc44a bytes) 000330 1789136784.516054 mapread 0x026e6e thru 0x02b155 (0x42e8 bytes) 000333 1789136784.531103 trunc from 0x033c0d to 0x03a97e (0x6d72 bytes) 000335 1789136784.555794 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a97f): Mirror 3: 0xed21aeba (stale) Mirror 5: 0x55e59b0c (stale) Mirror 6: 0x5ac1a044 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a97f): Mirror 3: 0xed21aeba (stale) Mirror 5: 0x55e59b0c (stale) Mirror 6: 0x5ac1a044 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a97f: skipped 000338 1789136784.987177 mapwrite 0x024f5d thru 0x026678 (0x171c bytes) 000339 1789136785.081100 write 0x03833c thru 0x03ffff (0x7cc4 bytes) 000342 1789136785.093662 trunc from 0x040000 to 0x00b922 (0x346dd bytes) 000343 1789136785.131871 read 0x005c86 thru 0x00b922 (0x5c9d bytes) 000345 1789136785.147633 read 0x0049be thru 0x00b922 (0x6f65 bytes) 000346 1789136785.163359 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb923): Mirror 3: 0xc1aa9f1a (stale) Mirror 5: 0xc1aa9f1a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb923): Mirror 3: 0xc1aa9f1a (stale) Mirror 5: 0xc1aa9f1a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb923: skipped 000347 1789136785.490030 mapwrite 0x029e6c thru 0x0309c3 (0x6b58 bytes) 000350 1789136785.568540 mapwrite 0x00460a thru 0x0097aa (0x51a1 bytes) 000351 1789136785.627548 mapwrite 0x02039d thru 0x02e0ca (0xdd2e bytes) 000360 1789136785.728611 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x309c4): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x309c4): Mirror 3: 0x4a2d00a5 (stale) Mirror 5: 0xead89538 (stale) CRC-32 checksum value for chunk [0, 0x309c4): Mirror 1: 0xe9a1953b Mirror 6: 0xe9a1953b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x309c4: skipped 000362 1789136786.683012 mapwrite 0x00f331 thru 0x01366f (0x433f bytes) 000364 1789136786.922094 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x309c4): Mirror 3: 0x4a2d00a5 (stale) Mirror 6: 0xe9a1953b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x309c4): Mirror 3: 0x4a2d00a5 (stale) Mirror 6: 0xe9a1953b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x309c4: skipped 000366 1789136787.327973 mapread 0x01d828 thru 0x02cefa (0xf6d3 bytes) 000367 1789136787.363246 mapread 0x009586 thru 0x018fa5 (0xfa20 bytes) 000369 1789136787.382131 mapread 0x017533 thru 0x023e6a (0xc938 bytes) 000372 1789136787.384434 trunc from 0x0309c4 to 0x02e17a (0x2849 bytes) 000373 1789136787.404345 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e17b): Mirror 6: 0x4edb4461 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e17b): Mirror 6: 0x4edb4461 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e17b: skipped 000375 1789136787.776609 mapwrite 0x00acc7 thru 0x010b87 (0x5ec1 bytes) 000376 1789136787.920057 trunc from 0x02e17b to 0x012c59 (0x1b521 bytes) 000378 1789136787.947460 mapwrite 0x03939c thru 0x039f00 (0xb65 bytes) 000379 1789136788.090310 mapwrite 0x0059b6 thru 0x0094a6 (0x3af1 bytes) 000380 1789136788.142629 mapread 0x0292f8 thru 0x029883 (0x58c bytes) 000381 1789136788.161662 mapread 0x017158 thru 0x021db5 (0xac5e bytes) 000382 1789136788.180090 trunc from 0x039f01 to 0x00142c (0x38ad4 bytes) 000383 1789136788.204862 write 0x00a0e9 thru 0x0169dc (0xc8f4 bytes) 000385 1789136788.206850 mapwrite 0x004bdf thru 0x009064 (0x4486 bytes) 000386 1789136788.307411 write 0x0388ad thru 0x03ffff (0x7753 bytes) 000388 1789136788.309187 mapwrite 0x01fdcd thru 0x02040c (0x640 bytes) 000390 1789136788.402932 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa858d4da (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8f9e6fe9 Mirror 7: 0x8f9e6fe9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000392 1789136789.370050 MIRROR_RESYNC 3 mirrors 000395 1789136790.282413 write 0x02809f thru 0x0287c4 (0x726 bytes) 000396 1789136790.365542 mapwrite 0x03a315 thru 0x03ffff (0x5ceb bytes) 000397 1789136790.433948 trunc from 0x040000 to 0x00ec64 (0x3139b bytes) 000398 1789136790.454640 mapread 0x001d09 thru 0x00c6ff (0xa9f7 bytes) 000399 1789136790.476081 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xec65): Mirror 7: 0x48b2c153 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xec65): Mirror 7: 0x48b2c153 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xec65: skipped 000400 1789136790.846063 trunc from 0x00ec65 to 0x0378f2 (0x28c8e bytes) 000401 1789136790.868106 read 0x00d238 thru 0x011abe (0x4887 bytes) 000402 1789136790.878857 mapwrite 0x02c5ac thru 0x03585c (0x92b1 bytes) 000403 1789136791.001772 mapread 0x002473 thru 0x007bb2 (0x5740 bytes) 000406 1789136791.024786 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x378f3): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x378f3): Mirror 7: 0x82f4c3dc (stale) CRC-32 checksum value for chunk [0, 0x378f3): Mirror 1: 0x3883c763 Mirror 8: 0x3883c763 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x378f3: skipped 000409 1789136792.128699 MIRROR_RESYNC 3 mirrors 000413 1789136793.211172 mapwrite 0x004803 thru 0x00c72b (0x7f29 bytes) 000415 1789136793.396801 mapread 0x020dc0 thru 0x02a8e7 (0x9b28 bytes) 000416 1789136793.432374 read 0x02460b thru 0x02501d (0xa13 bytes) 000418 1789136793.435185 write 0x02f811 thru 0x035fbc (0x67ac bytes) 000419 1789136793.443412 mapwrite 0x000995 thru 0x004972 (0x3fde bytes) 000423 1789136793.507206 write 0x018d87 thru 0x026577 (0xd7f1 bytes) 000425 1789136793.516170 trunc from 0x0378f3 to 0x01e871 (0x19081 bytes) 000426 1789136793.543745 write 0x01a5ba thru 0x025022 (0xaa69 bytes) 000427 1789136793.545205 mapwrite 0x003ac6 thru 0x011fd6 (0xe511 bytes) 000429 1789136793.611149 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x25023): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25023): Mirror 7: 0xaf99b390 (stale) Mirror 8: 0xaf99b390 (stale) CRC-32 checksum value for chunk [0, 0x25023): Mirror 1: 0x5000f1af Mirror 9: 0x5000f1af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25023: skipped 000430 1789136794.343418 trunc from 0x025023 to 0x038ffe (0x13fdc bytes) 000432 1789136794.515767 MIRROR_RESYNC 4 mirrors 000434 1789136796.090396 read 0x018341 thru 0x01a5f5 (0x22b5 bytes) 000436 1789136796.096237 mapread 0x025733 thru 0x031a75 (0xc343 bytes) 000442 1789136796.138067 mapread 0x01d48d thru 0x01f5fc (0x2170 bytes) 000446 1789136796.154550 trunc from 0x038fff to 0x004369 (0x34c95 bytes) 000447 1789136796.286976 trunc from 0x00436a to 0x00a9b2 (0x6649 bytes) 000449 1789136796.318938 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xa9b3): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa9b3): Mirror 7: 0x4640f70b (stale) Mirror 8: 0x4640f70b (stale) Mirror 9: 0x4640f70b (stale) CRC-32 checksum value for chunk [0, 0xa9b3): Mirror 1: 0xe12aee82 Mirror 10: 0xe12aee82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa9b3: skipped 000450 1789136797.577874 read 0x001329 thru 0x00a9b2 (0x968a bytes) 000455 1789136797.594072 mapread 0x006637 thru 0x00a9b2 (0x437c bytes) 000456 1789136797.595093 trunc from 0x00a9b3 to 0x03186e (0x26ebc bytes) 000458 1789136797.702198 read 0x009762 thru 0x016376 (0xcc15 bytes) 000462 1789136797.721876 read 0x01c3a1 thru 0x022035 (0x5c95 bytes) 000463 1789136797.724123 read 0x02914c thru 0x03186e (0x8723 bytes) 000466 1789136797.735834 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3186f): Mirror 7: 0xa60940d4 (stale) Mirror 9: 0xa60940d4 (stale) Mirror 10: 0x1710d904 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3186f): Mirror 7: 0xa60940d4 (stale) Mirror 9: 0xa60940d4 (stale) Mirror 10: 0x1710d904 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3186f: skipped truncating to largest ever: 0x3c2a2 000468 1789136798.175019 trunc from 0x03186f to 0x03c2a1 (0xaa33 bytes) 000469 1789136798.201469 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c2a2): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c2a2): Mirror 7: 0x36a4b99e (stale) Mirror 9: 0x36a4b99e (stale) Mirror 10: 0x1f50ef3a (stale) CRC-32 checksum value for chunk [0, 0x3c2a2): Mirror 1: 0x1f50ef3a Mirror 11: 0x1f50ef3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c2a2: skipped 000471 1789136799.675417 MIRROR_RESYNC 5 mirrors 000472 1789136801.045046 trunc from 0x03c2a2 to 0x022838 (0x19a69 bytes) 000474 1789136801.140606 mapread 0x00d9c1 thru 0x01a00d (0xc64d bytes) 000476 1789136801.185148 mapwrite 0x022a65 thru 0x02a00f (0x75ab bytes) 000482 1789136801.278182 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a010): Mirror 7: 0x779301b8 (stale) Mirror 9: 0x779301b8 (stale) Mirror 10: 0x779301b8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a010): Mirror 7: 0x779301b8 (stale) Mirror 9: 0x779301b8 (stale) Mirror 10: 0x779301b8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a010: skipped 000484 1789136801.763395 mapwrite 0x013c46 thru 0x01e4ca (0xa885 bytes) 000487 1789136801.859994 read 0x016d9a thru 0x0198b4 (0x2b1b bytes) 000490 1789136801.861810 write 0x005a74 thru 0x0115f6 (0xbb83 bytes) 000491 1789136801.868494 write 0x0283c9 thru 0x02a2e3 (0x1f1b bytes) 000492 1789136801.886464 mapwrite 0x009e78 thru 0x010770 (0x68f9 bytes) 000493 1789136801.937819 mapread 0x00e958 thru 0x012341 (0x39ea bytes) 000494 1789136801.939037 mapread 0x0163eb thru 0x01d041 (0x6c57 bytes) 000495 1789136801.940372 mapwrite 0x025403 thru 0x031a43 (0xc641 bytes) 000497 1789136802.055129 mapread 0x00212c thru 0x00e283 (0xc158 bytes) 000498 1789136802.065859 mapwrite 0x01817a thru 0x01a252 (0x20d9 bytes) 000499 1789136802.106445 mapread 0x00c101 thru 0x019cf8 (0xdbf8 bytes) 000501 1789136802.109003 read 0x014ea0 thru 0x0210a7 (0xc208 bytes) 000503 1789136802.112780 mapwrite 0x0363fd thru 0x03865a (0x225e bytes) 000504 1789136802.242204 read 0x02e0d7 thru 0x02eb21 (0xa4b bytes) 000505 1789136802.246097 read 0x02168f thru 0x023144 (0x1ab6 bytes) 000506 1789136802.248286 mapwrite 0x01294c thru 0x0215c0 (0xec75 bytes) 000507 1789136802.340426 trunc from 0x03865b to 0x0333ba (0x52a0 bytes) 000508 1789136802.377468 trunc from 0x0333bb to 0x0167ca (0x1cbf0 bytes) truncating to largest ever: 0x3e3ca 000511 1789136802.402003 trunc from 0x0167cb to 0x03e3c9 (0x27bff bytes) 000512 1789136802.425218 mapwrite 0x039a39 thru 0x03ffff (0x65c7 bytes) 000513 1789136802.560005 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x326385db (stale) Mirror 10: 0x326385db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x326385db (stale) Mirror 10: 0x326385db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000514 1789136803.063974 mapwrite 0x02b484 thru 0x0363aa (0xaf27 bytes) 000516 1789136803.212775 mapwrite 0x008391 thru 0x010715 (0x8385 bytes) 000518 1789136803.285166 trunc from 0x040000 to 0x031e07 (0xe1f8 bytes) 000519 1789136803.312992 mapread 0x01742b thru 0x020f11 (0x9ae7 bytes) 000522 1789136803.336479 mapread 0x024969 thru 0x029f76 (0x560e bytes) 000524 1789136803.338543 trunc from 0x031e08 to 0x00b5ae (0x26859 bytes) 000525 1789136803.357549 read 0x0003d1 thru 0x00b5ae (0xb1de bytes) 000526 1789136803.369843 MIRROR_RESYNC 3 mirrors 000528 1789136804.198748 write 0x031ca2 thru 0x03fc26 (0xdf85 bytes) 000531 1789136804.265052 write 0x03aef2 thru 0x03ca2e (0x1b3d bytes) 000532 1789136804.267372 mapwrite 0x023c20 thru 0x024529 (0x90a bytes) 000533 1789136804.339625 mapwrite 0x030554 thru 0x03e732 (0xe1df bytes) 000534 1789136804.394815 write 0x00d483 thru 0x017227 (0x9da5 bytes) 000535 1789136804.413829 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fc27): Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc27): Mirror 7: 0x86a3f9ea (stale) Mirror 10: 0x86a3f9ea (stale) CRC-32 checksum value for chunk [0, 0x3fc27): Mirror 1: 0xec12352c Mirror 11: 0xec12352c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc27: skipped 000536 1789136805.849820 read 0x0362b9 thru 0x03fc26 (0x996e bytes) 000538 1789136805.866548 write 0x001841 thru 0x00fb4f (0xe30f bytes) 000539 1789136806.005306 write 0x005e48 thru 0x0083e1 (0x259a bytes) 000541 1789136806.006959 trunc from 0x03fc27 to 0x035c86 (0x9fa0 bytes) 000543 1789136806.046505 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35c87): Mirror 7: 0x2e0b0d8b (stale) Mirror 10: 0x2e0b0d8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35c87): Mirror 7: 0x2e0b0d8b (stale) Mirror 10: 0x2e0b0d8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35c87: skipped 000544 1789136806.389181 mapread 0x009677 thru 0x012302 (0x8c8c bytes) 000545 1789136806.428152 read 0x027214 thru 0x02ef88 (0x7d75 bytes) 000547 1789136806.439987 mapwrite 0x02c4ee thru 0x03bf33 (0xfa46 bytes) 000550 1789136806.554879 read 0x031ed8 thru 0x03bf33 (0xa05c bytes) 000551 1789136806.555485 read 0x00b830 thru 0x013a78 (0x8249 bytes) 000552 1789136806.556166 mapwrite 0x012fe2 thru 0x020733 (0xd752 bytes) 000553 1789136806.681093 write 0x019c0d thru 0x02054d (0x6941 bytes) 000554 1789136806.684375 write 0x026d99 thru 0x027605 (0x86d bytes) 000555 1789136806.685486 trunc from 0x03bf34 to 0x024c51 (0x172e2 bytes) 000556 1789136806.713043 mapread 0x01387e thru 0x01714b (0x38ce bytes) 000558 1789136806.719406 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24c52): Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24c52): Mirror 7: 0x46be7fe1 (stale) Mirror 10: 0x46be7fe1 (stale) CRC-32 checksum value for chunk [0, 0x24c52): Mirror 1: 0x5e940e29 Mirror 11: 0x5e940e29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24c52: skipped 000559 1789136807.530098 trunc from 0x024c52 to 0x00781f (0x1d432 bytes) 000560 1789136807.651951 trunc from 0x007820 to 0x032f13 (0x2b6f4 bytes) 000562 1789136807.699288 write 0x02c9d8 thru 0x0392d6 (0xc8ff bytes) 000563 1789136807.710161 mapwrite 0x03320e thru 0x0348da (0x16cd bytes) 000564 1789136807.770904 write 0x01e2b3 thru 0x0243a7 (0x60f5 bytes) 000566 1789136807.790180 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x392d7): Mirror 7: 0xae704233 (stale) Mirror 10: 0xae704233 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x392d7): Mirror 7: 0xae704233 (stale) Mirror 10: 0xae704233 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x392d7: skipped 000568 1789136808.180586 write 0x03315e thru 0x0357bb (0x265e bytes) 000569 1789136808.195868 mapread 0x016f40 thru 0x023dd5 (0xce96 bytes) 000570 1789136808.241864 mapread 0x02edee thru 0x038a30 (0x9c43 bytes) 000572 1789136808.289491 mapread 0x036bca thru 0x0392d6 (0x270d bytes) 000575 1789136808.294120 mapread 0x003d80 thru 0x005bb1 (0x1e32 bytes) 000576 1789136808.310611 read 0x02584b thru 0x02ff9f (0xa755 bytes) 000577 1789136808.311556 read 0x02b1ee thru 0x02eb10 (0x3923 bytes) 000578 1789136808.312473 mapread 0x02af97 thru 0x02f0fb (0x4165 bytes) 000579 1789136808.313768 trunc from 0x0392d7 to 0x0340d7 (0x51ff bytes) 000580 1789136808.347810 write 0x01c5d6 thru 0x028757 (0xc182 bytes) 000581 1789136808.350382 trunc from 0x0340d8 to 0x02c6a9 (0x7a2e bytes) 000582 1789136808.372758 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c6aa): Mirror 10: 0x86621151 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c6aa): Mirror 10: 0x86621151 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c6aa: skipped 000584 1789136808.686239 write 0x00aa4c thru 0x0141a9 (0x975e bytes) 000585 1789136808.700998 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) 000586 1789136808.847877 mapread 0x02181b thru 0x022e9a (0x1680 bytes) 000589 1789136808.862011 trunc from 0x02c6aa to 0x039fea (0xd941 bytes) 000590 1789136808.886246 mapread 0x01a7ec thru 0x01d157 (0x296c bytes) 000591 1789136808.896215 mapwrite 0x025b05 thru 0x02c189 (0x6685 bytes) 000592 1789136808.948087 mapwrite 0x001926 thru 0x005950 (0x402b bytes) truncating to largest ever: 0x3f695 000593 1789136809.009382 trunc from 0x039feb to 0x03f694 (0x56aa bytes) 000594 1789136809.028387 mapread 0x0264ac thru 0x028868 (0x23bd bytes) 000596 1789136809.030056 trunc from 0x03f695 to 0x004e2f (0x3a865 bytes) 000597 1789136809.046240 write 0x03bbc7 thru 0x03ffff (0x4439 bytes) 000599 1789136809.050546 mapread 0x02e500 thru 0x03c4b9 (0xdfba bytes) 000606 1789136809.077442 read 0x03db37 thru 0x03ffff (0x24c9 bytes) 000613 1789136809.088333 read 0x02a99c thru 0x033003 (0x8668 bytes) 000614 1789136809.088941 trunc from 0x040000 to 0x01bf62 (0x2409d bytes) 000615 1789136809.122103 write 0x03998d thru 0x03deb2 (0x4526 bytes) 000626 1789136809.145575 read 0x0131ce thru 0x01788b (0x46be bytes) truncating to largest ever: 0x3fd68 000628 1789136809.172985 trunc from 0x03deb3 to 0x03fd67 (0x1eb5 bytes) 000631 1789136809.211851 write 0x0182ba thru 0x018640 (0x387 bytes) 000635 1789136809.218859 mapwrite 0x03e3d5 thru 0x03ffff (0x1c2b bytes) 000637 1789136809.379107 write 0x02d5bd thru 0x035970 (0x83b4 bytes) 000640 1789136809.384576 trunc from 0x040000 to 0x029376 (0x16c89 bytes) 000641 1789136809.411407 mapwrite 0x03e1c7 thru 0x03ffff (0x1e39 bytes) 000642 1789136809.502725 trunc from 0x040000 to 0x001c04 (0x3e3fb bytes) 000643 1789136809.524497 trunc from 0x001c05 to 0x0306ec (0x2eae8 bytes) 000644 1789136809.546637 trunc from 0x0306ed to 0x009b15 (0x26bd7 bytes) 000650 1789136809.568384 read 0x00540a thru 0x00854b (0x3142 bytes) 000653 1789136809.578416 trunc from 0x009b16 to 0x027ebf (0x1e3aa bytes) 000655 1789136809.597371 mapwrite 0x037a95 thru 0x03ffff (0x856b bytes) 000658 1789136809.699140 MIRROR_EXTEND to 2 mirrors 000660 1789136810.846685 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) 000663 1789136810.989456 mapread 0x0262fb thru 0x032895 (0xc59b bytes) 000664 1789136811.019010 mapwrite 0x01f4cc thru 0x023eee (0x4a23 bytes) 000666 1789136811.103432 MIRROR_EXTEND to 2 mirrors 000669 1789136812.283437 write 0x012c69 thru 0x01704a (0x43e2 bytes) 000671 1789136812.376479 write 0x00dac9 thru 0x01396d (0x5ea5 bytes) 000673 1789136812.395397 mapwrite 0x027d4f thru 0x02d658 (0x590a bytes) 000675 1789136812.495971 write 0x0200a3 thru 0x0296f3 (0x9651 bytes) 000676 1789136812.502958 write 0x003b30 thru 0x00fac6 (0xbf97 bytes) 000678 1789136812.510852 mapwrite 0x027e39 thru 0x037682 (0xf84a bytes) 000679 1789136812.572029 trunc from 0x040000 to 0x01e8ed (0x21712 bytes) 000682 1789136812.602123 MIRROR_RESYNC 2 mirrors 000683 1789136813.555816 read 0x01156d thru 0x0174fe (0x5f92 bytes) 000684 1789136813.582344 read 0x007ad8 thru 0x011a8f (0x9fb8 bytes) 000685 1789136813.600135 mapread 0x015515 thru 0x01cd90 (0x787c bytes) 000687 1789136813.603946 read 0x00ccbe thru 0x011730 (0x4a73 bytes) 000688 1789136813.606517 mapwrite 0x03c580 thru 0x03ffff (0x3a80 bytes) 000689 1789136813.762379 mapwrite 0x021b6c thru 0x023e0a (0x229f bytes) 000691 1789136813.817798 write 0x0002bf thru 0x00c8a2 (0xc5e4 bytes) 000696 1789136813.821476 mapwrite 0x02e9b0 thru 0x030d6d (0x23be bytes) 000697 1789136813.865343 trunc from 0x040000 to 0x02fc93 (0x1036c bytes) 000699 1789136813.907981 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) 000700 1789136814.148378 read 0x01434a thru 0x0184b8 (0x416f bytes) 000704 1789136814.164392 mapread 0x0189ff thru 0x01e5d3 (0x5bd5 bytes) 000705 1789136814.192487 write 0x036260 thru 0x03ffff (0x9da0 bytes) 000708 1789136814.196717 mapread 0x018584 thru 0x021d0b (0x9788 bytes) 000709 1789136814.198342 trunc from 0x040000 to 0x031e47 (0xe1b8 bytes) 000711 1789136814.231077 read 0x00f432 thru 0x016d4d (0x791c bytes) 000713 1789136814.252485 write 0x0248b7 thru 0x02c3f9 (0x7b43 bytes) 000714 1789136814.254248 mapread 0x00175f thru 0x002549 (0xdeb bytes) 000719 1789136814.259916 mapwrite 0x002682 thru 0x00b11f (0x8a9e bytes) 000720 1789136814.415584 mapread 0x00064d thru 0x00688e (0x6242 bytes) 000722 1789136814.417402 write 0x02e4dd thru 0x03b236 (0xcd5a bytes) 000723 1789136814.421504 mapwrite 0x01aa43 thru 0x024ac9 (0xa087 bytes) 000725 1789136814.543127 MIRROR_EXTEND to 2 mirrors 000726 1789136815.528053 mapread 0x026287 thru 0x02ffa8 (0x9d22 bytes) 000727 1789136815.561498 read 0x037743 thru 0x03a98f (0x324d bytes) 000728 1789136815.573906 read 0x002c89 thru 0x00e4db (0xb853 bytes) 000730 1789136815.595376 trunc from 0x03b237 to 0x032381 (0x8eb5 bytes) 000731 1789136815.763897 trunc from 0x032382 to 0x02bd59 (0x6628 bytes) 000733 1789136815.800135 mapread 0x013308 thru 0x01ffff (0xccf8 bytes) 000734 1789136815.909116 read 0x01e307 thru 0x02574b (0x7445 bytes) 000736 1789136815.941855 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2bd5a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bd5a): Mirror 2: 0x9e7f7d0d (stale) CRC-32 checksum value for chunk [0, 0x2bd5a): Mirror 1: 0x9e7f7d0d Mirror 3: 0x9e7f7d0d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bd5a: skipped 000737 1789136816.980153 mapwrite 0x006a83 thru 0x00aef6 (0x4474 bytes) 000739 1789136817.169663 MIRROR_RESYNC 3 mirrors 000740 1789136818.113961 mapwrite 0x038dde thru 0x03ffff (0x7222 bytes) 000744 1789136818.330338 mapwrite 0x030669 thru 0x0373b2 (0x6d4a bytes) 000745 1789136818.400016 trunc from 0x040000 to 0x026251 (0x19dae bytes) 000747 1789136818.422186 trunc from 0x026252 to 0x02b644 (0x53f3 bytes) 000748 1789136818.442343 MIRROR_RESYNC 3 mirrors 000749 1789136819.283916 write 0x02c11c thru 0x037a60 (0xb945 bytes) 000751 1789136819.365350 MIRROR_RESYNC 3 mirrors 000752 1789136820.344323 write 0x024871 thru 0x02f098 (0xa828 bytes) 000754 1789136820.441063 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37a61): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a61): Mirror 2: 0x985277f6 (stale) Mirror 3: 0x985277f6 (stale) CRC-32 checksum value for chunk [0, 0x37a61): Mirror 1: 0xf209201a Mirror 4: 0xf209201a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a61: skipped 000755 1789136821.364324 mapwrite 0x02ef3c thru 0x037d90 (0x8e55 bytes) 000756 1789136821.543419 trunc from 0x037d91 to 0x002dec (0x34fa4 bytes) 000757 1789136821.581375 trunc from 0x002ded to 0x0317bd (0x2e9d1 bytes) 000758 1789136821.616995 MIRROR_RESYNC 4 mirrors 000760 1789136822.651466 mapwrite 0x003ab5 thru 0x00f8d9 (0xbe25 bytes) 000761 1789136822.899414 trunc from 0x0317be to 0x03976e (0x7fb1 bytes) 000762 1789136822.961084 MIRROR_RESYNC 4 mirrors 000764 1789136824.049904 mapwrite 0x00d2e9 thru 0x01b6bd (0xe3d5 bytes) 000766 1789136824.250412 trunc from 0x03976f to 0x02f094 (0xa6da bytes) 000767 1789136824.273353 trunc from 0x02f095 to 0x013fed (0x1b0a7 bytes) 000768 1789136824.292344 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13fee): Mirror 3: 0x20dcf26b (stale) Mirror 4: 0x20dcf26b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13fee): Mirror 3: 0x20dcf26b (stale) Mirror 4: 0x20dcf26b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13fee: skipped 000771 1789136824.642468 mapwrite 0x017d77 thru 0x02061b (0x88a5 bytes) 000774 1789136824.752037 read 0x00046c thru 0x001dec (0x1981 bytes) 000775 1789136824.765982 read 0x01759e thru 0x02061b (0x907e bytes) 000777 1789136824.767172 mapread 0x00a344 thru 0x019460 (0xf11d bytes) 000778 1789136824.780108 read 0x00990a thru 0x00e2e5 (0x49dc bytes) 000779 1789136824.781100 trunc from 0x02061c to 0x01d1b0 (0x346b bytes) 000780 1789136824.803932 mapread 0x018ac0 thru 0x01b7a3 (0x2ce4 bytes) 000781 1789136824.805941 write 0x0264bf thru 0x02b0d6 (0x4c18 bytes) 000785 1789136824.810864 mapwrite 0x01396b thru 0x022303 (0xe999 bytes) 000786 1789136824.910021 mapwrite 0x02b0b0 thru 0x02baf8 (0xa49 bytes) 000787 1789136825.025521 read 0x029ed4 thru 0x02baf8 (0x1c25 bytes) 000788 1789136825.027417 trunc from 0x02baf9 to 0x00e306 (0x1d7f2 bytes) 000789 1789136825.048675 write 0x037c42 thru 0x03ce9f (0x525e bytes) 000790 1789136825.058976 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cea0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cea0): Mirror 3: 0x3a655669 (stale) Mirror 4: 0x3a655669 (stale) CRC-32 checksum value for chunk [0, 0x3cea0): Mirror 1: 0xd0a18cca Mirror 5: 0xd0a18cca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cea0: skipped 000794 1789136826.220618 mapread 0x00dee9 thru 0x01dea6 (0xffbe bytes) 000795 1789136826.252933 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cea0): Mirror 3: 0x3a655669 (stale) Mirror 4: 0x3a655669 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cea0): Mirror 3: 0x3a655669 (stale) Mirror 4: 0x3a655669 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cea0: skipped 000796 1789136826.795642 read 0x005a1d thru 0x012b93 (0xd177 bytes) 000797 1789136826.806634 read 0x023a62 thru 0x0325ca (0xeb69 bytes) 000798 1789136826.819915 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cea0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cea0): Mirror 3: 0x3a655669 (stale) Mirror 4: 0x3a655669 (stale) CRC-32 checksum value for chunk [0, 0x3cea0): Mirror 1: 0xd0a18cca Mirror 5: 0xd0a18cca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cea0: skipped 000799 1789136827.980780 read 0x027448 thru 0x02b226 (0x3ddf bytes) 000800 1789136827.997461 read 0x039933 thru 0x03a228 (0x8f6 bytes) 000801 1789136828.032167 mapread 0x03b160 thru 0x03ce9f (0x1d40 bytes) 000804 1789136828.033744 mapread 0x03b997 thru 0x03ce9f (0x1509 bytes) 000805 1789136828.034687 read 0x03ac5d thru 0x03ce9f (0x2243 bytes) 000807 1789136828.035781 write 0x017005 thru 0x01739a (0x396 bytes) 000809 1789136828.176803 mapwrite 0x011ea6 thru 0x0176ac (0x5807 bytes) 000811 1789136828.274143 write 0x03c2b0 thru 0x03ffff (0x3d50 bytes) 000812 1789136828.280540 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8435fe5a (stale) Mirror 4: 0x8435fe5a (stale) Mirror 5: 0x50853f05 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x88d721d7 Mirror 6: 0x88d721d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000814 1789136829.431615 trunc from 0x040000 to 0x01c48f (0x23b70 bytes) 000815 1789136829.564943 read 0x00c0ad thru 0x0155e9 (0x953d bytes) 000816 1789136829.576363 read 0x00bec9 thru 0x01a382 (0xe4ba bytes) 000819 1789136829.589926 trunc from 0x01c490 to 0x0396a1 (0x1d212 bytes) 000820 1789136829.620897 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x396a2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x396a2): Mirror 3: 0x8398ec5d (stale) Mirror 4: 0x8398ec5d (stale) Mirror 5: 0xfe862d5f (stale) Mirror 6: 0xd006004 (stale) CRC-32 checksum value for chunk [0, 0x396a2): Mirror 1: 0x6f049962 Mirror 7: 0x6f049962 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x396a2: skipped 000823 1789136831.005167 read 0x012f68 thru 0x014ef1 (0x1f8a bytes) 000824 1789136831.018351 MIRROR_RESYNC 6 mirrors 000825 1789136832.392645 MIRROR_EXTEND to 7 mirrors 000827 1789136833.502050 read 0x013972 thru 0x016763 (0x2df2 bytes) 000828 1789136833.513386 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) 000829 1789136833.930234 write 0x011c5f thru 0x01afb1 (0x9353 bytes) 000830 1789136834.022205 mapwrite 0x021622 thru 0x02eb65 (0xd544 bytes) 000833 1789136834.170421 write 0x03cb06 thru 0x03ffff (0x34fa bytes) 000834 1789136834.171467 mapwrite 0x03250c thru 0x03b16d (0x8c62 bytes) 000835 1789136834.295947 read 0x0211fe thru 0x021a32 (0x835 bytes) 000837 1789136834.311040 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde2aa384 (stale) Mirror 5: 0xde2aa384 (stale) Mirror 6: 0xde2aa384 (stale) Mirror 7: 0xde2aa384 (stale) Mirror 8: 0xde2aa384 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5cae6039 Mirror 9: 0x5cae6039 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000838 1789136835.247622 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde2aa384 (stale) Mirror 5: 0xde2aa384 (stale) Mirror 6: 0xde2aa384 (stale) Mirror 7: 0xde2aa384 (stale) Mirror 8: 0xde2aa384 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5cae6039 Mirror 9: 0x5cae6039 Mirror 10: 0x5cae6039 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000839 1789136836.324868 write 0x00364a thru 0x0037fa (0x1b1 bytes) 000841 1789136836.407058 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde2aa384 (stale) Mirror 5: 0xde2aa384 (stale) Mirror 6: 0xde2aa384 (stale) Mirror 8: 0xde2aa384 (stale) Mirror 9: 0x5cae6039 (stale) Mirror 10: 0x5cae6039 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde2aa384 (stale) Mirror 5: 0xde2aa384 (stale) Mirror 6: 0xde2aa384 (stale) Mirror 8: 0xde2aa384 (stale) Mirror 9: 0x5cae6039 (stale) Mirror 10: 0x5cae6039 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000842 1789136836.917910 trunc from 0x040000 to 0x030ab7 (0xf548 bytes) 000844 1789136836.950718 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30ab8): Mirror 4: 0x885580b9 (stale) Mirror 5: 0x885580b9 (stale) Mirror 6: 0x885580b9 (stale) Mirror 8: 0x885580b9 (stale) Mirror 10: 0x10b292d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30ab8): Mirror 4: 0x885580b9 (stale) Mirror 5: 0x885580b9 (stale) Mirror 6: 0x885580b9 (stale) Mirror 8: 0x885580b9 (stale) Mirror 10: 0x10b292d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30ab8: skipped 000845 1789136837.563855 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x30ab8): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30ab8): Mirror 4: 0x885580b9 (stale) Mirror 5: 0x885580b9 (stale) Mirror 6: 0x885580b9 (stale) Mirror 8: 0x885580b9 (stale) Mirror 10: 0x10b292d7 (stale) CRC-32 checksum value for chunk [0, 0x30ab8): Mirror 1: 0x89153b03 Mirror 11: 0x89153b03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30ab8: skipped 000847 1789136838.708321 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x30ab8): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30ab8): Mirror 4: 0x885580b9 (stale) Mirror 5: 0x885580b9 (stale) Mirror 6: 0x885580b9 (stale) Mirror 8: 0x885580b9 (stale) Mirror 10: 0x10b292d7 (stale) CRC-32 checksum value for chunk [0, 0x30ab8): Mirror 1: 0x89153b03 Mirror 11: 0x89153b03 Mirror 12: 0x89153b03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30ab8: skipped 000848 1789136839.974932 mapread 0x015622 thru 0x0232ef (0xdcce bytes) 000849 1789136840.023747 mapwrite 0x02f03a thru 0x037555 (0x851c bytes) 000851 1789136840.250393 MIRROR_RESYNC 8 mirrors 000856 1789136841.625685 mapread 0x026891 thru 0x02d4b1 (0x6c21 bytes) 000859 1789136841.666933 trunc from 0x037556 to 0x0009b9 (0x36b9c bytes) 000860 1789136841.769102 read 0x0006cf thru 0x0006f7 (0x29 bytes) 000865 1789136841.778515 mapwrite 0x00c3e3 thru 0x019530 (0xd14e bytes) 000867 1789136841.898718 read 0x004b0d thru 0x00f93b (0xae2f bytes) 000868 1789136841.939814 MIRROR_RESYNC 8 mirrors 000869 1789136843.476843 mapread 0x00959d thru 0x00e315 (0x4d79 bytes) 000871 1789136843.503682 trunc from 0x019531 to 0x028172 (0xec42 bytes) 000873 1789136843.565610 read 0x009192 thru 0x00c291 (0x3100 bytes) 000874 1789136843.577650 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28173): Mirror 5: 0xe6017964 (stale) Mirror 6: 0xe6017964 (stale) Mirror 8: 0xe6017964 (stale) Mirror 10: 0xe6017964 (stale) Mirror 11: 0xe6017964 (stale) Mirror 12: 0xe6017964 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28173): Mirror 5: 0xe6017964 (stale) Mirror 6: 0xe6017964 (stale) Mirror 8: 0xe6017964 (stale) Mirror 10: 0xe6017964 (stale) Mirror 11: 0xe6017964 (stale) Mirror 12: 0xe6017964 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28173: skipped 000877 1789136844.036082 read 0x0125ef thru 0x01b38e (0x8da0 bytes) 000879 1789136844.070906 mapread 0x022ce8 thru 0x028172 (0x548b bytes) 000880 1789136844.128963 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28173): Mirror 5: 0xe6017964 (stale) Mirror 6: 0xe6017964 (stale) Mirror 8: 0xe6017964 (stale) Mirror 11: 0xe6017964 (stale) Mirror 12: 0xe6017964 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28173): Mirror 5: 0xe6017964 (stale) Mirror 6: 0xe6017964 (stale) Mirror 8: 0xe6017964 (stale) Mirror 11: 0xe6017964 (stale) Mirror 12: 0xe6017964 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28173: skipped 000881 1789136844.520757 write 0x013064 thru 0x01f761 (0xc6fe bytes) 000882 1789136844.534232 read 0x026e35 thru 0x028172 (0x133e bytes) 000883 1789136844.543596 trunc from 0x028173 to 0x0320eb (0x9f79 bytes) 000884 1789136844.567976 mapwrite 0x023576 thru 0x02e848 (0xb2d3 bytes) 000885 1789136844.684532 trunc from 0x0320ec to 0x02527b (0xce70 bytes) 000886 1789136844.707881 read 0x00c553 thru 0x014e46 (0x88f4 bytes) 000887 1789136844.724843 mapwrite 0x03b177 thru 0x03ffff (0x4e89 bytes) 000890 1789136844.800979 MIRROR_RESYNC 6 mirrors 000891 1789136846.612160 mapread 0x02f32b thru 0x0310f0 (0x1dc6 bytes) 000892 1789136846.637494 mapwrite 0x018781 thru 0x019979 (0x11f9 bytes) 000895 1789136846.847004 trunc from 0x040000 to 0x02f61b (0x109e4 bytes) 000896 1789136846.875055 mapread 0x009d1b thru 0x0165ac (0xc892 bytes) 000899 1789136846.887181 write 0x0188b6 thru 0x01e5d2 (0x5d1d bytes) 000900 1789136846.888750 write 0x02ad86 thru 0x039880 (0xeafb bytes) 000902 1789136846.895817 mapread 0x028762 thru 0x031713 (0x8fb2 bytes) 000903 1789136846.912079 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39881): Mirror 6: 0xbda97440 (stale) Mirror 8: 0xbda97440 (stale) Mirror 11: 0xbda97440 (stale) Mirror 12: 0xbda97440 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39881): Mirror 6: 0xbda97440 (stale) Mirror 8: 0xbda97440 (stale) Mirror 11: 0xbda97440 (stale) Mirror 12: 0xbda97440 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39881: skipped 000904 1789136847.338223 mapwrite 0x0186d1 thru 0x021f17 (0x9847 bytes) 000905 1789136847.431957 mapread 0x0018e6 thru 0x00ee37 (0xd552 bytes) 000907 1789136847.451947 mapread 0x020f6e thru 0x02fc5d (0xecf0 bytes) 000908 1789136847.496189 trunc from 0x039881 to 0x01ffef (0x19891 bytes) 000911 1789136847.529043 trunc from 0x01fff0 to 0x037ae7 (0x17af8 bytes) 000914 1789136847.552204 mapread 0x02af38 thru 0x02f2db (0x43a4 bytes) 000915 1789136847.567769 mapread 0x003b86 thru 0x004518 (0x993 bytes) 000916 1789136847.569770 trunc from 0x037ae8 to 0x01a474 (0x1d673 bytes) 000917 1789136847.586913 trunc from 0x01a475 to 0x03f39a (0x24f26 bytes) 000918 1789136847.602561 read 0x00ca9f thru 0x015882 (0x8de4 bytes) 000920 1789136847.611210 mapread 0x02f674 thru 0x037355 (0x7ce2 bytes) 000921 1789136847.622554 read 0x004f00 thru 0x006fa9 (0x20aa bytes) 000924 1789136847.628700 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f39b): Mirror 6: 0x7fa873c9 (stale) Mirror 8: 0x7fa873c9 (stale) Mirror 12: 0x7fa873c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f39b): Mirror 6: 0x7fa873c9 (stale) Mirror 8: 0x7fa873c9 (stale) Mirror 12: 0x7fa873c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f39b: skipped 000925 1789136847.984328 write 0x018c6c thru 0x022713 (0x9aa8 bytes) 000928 1789136847.996679 trunc from 0x03f39b to 0x0023c1 (0x3cfd9 bytes) 000929 1789136848.023019 mapwrite 0x03ac19 thru 0x03ffff (0x53e7 bytes) 000930 1789136848.161082 trunc from 0x040000 to 0x01d742 (0x228bd bytes) 000931 1789136848.190182 trunc from 0x01d743 to 0x02af67 (0xd825 bytes) 000932 1789136848.209809 mapread 0x010f5b thru 0x01227f (0x1325 bytes) 000933 1789136848.223706 trunc from 0x02af68 to 0x006ff4 (0x23f73 bytes) 000936 1789136848.256066 mapread 0x001c6d thru 0x006ff4 (0x5388 bytes) 000938 1789136848.275026 mapwrite 0x0238ac thru 0x0304b8 (0xcc0d bytes) 000939 1789136848.417678 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x304b9): Mirror 8: 0xacc0b1dc (stale) Mirror 12: 0xacc0b1dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x304b9): Mirror 8: 0xacc0b1dc (stale) Mirror 12: 0xacc0b1dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x304b9: skipped 000942 1789136848.724008 write 0x02160c thru 0x031386 (0xfd7b bytes) 000946 1789136848.738669 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31387): Mirror 8: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31387): Mirror 8: 0xcaf8c060 (stale) Mirror 12: 0xcaf8c060 (stale) CRC-32 checksum value for chunk [0, 0x31387): Mirror 1: 0x94c3f665 Mirror 13: 0x94c3f665 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31387: skipped 000948 1789136849.963541 trunc from 0x031387 to 0x0144b0 (0x1ced6 bytes) 000949 1789136850.083377 MIRROR_RESYNC 4 mirrors 000950 1789136851.223333 mapread 0x003fc3 thru 0x006110 (0x214e bytes) 000952 1789136851.255243 trunc from 0x0144b1 to 0x013362 (0x114e bytes) 000954 1789136851.327006 MIRROR_RESYNC 4 mirrors 000955 1789136852.346237 MIRROR_EXTEND to 5 mirrors 000956 1789136853.346835 read 0x00454e thru 0x0062be (0x1d71 bytes) 000957 1789136853.352765 mapread 0x0053f9 thru 0x008429 (0x3031 bytes) 000958 1789136853.371866 write 0x0054aa thru 0x013b33 (0xe68a bytes) 000959 1789136853.468117 read 0x00ee30 thru 0x013b33 (0x4d04 bytes) 000960 1789136853.470855 write 0x01cfb5 thru 0x026ba7 (0x9bf3 bytes) 000961 1789136853.473434 read 0x01ec27 thru 0x020d29 (0x2103 bytes) 000964 1789136853.475359 read 0x0232a3 thru 0x026ba7 (0x3905 bytes) 000967 1789136853.477818 read 0x016ba6 thru 0x0176ce (0xb29 bytes) 000969 1789136853.494132 MIRROR_RESYNC 5 mirrors 000970 1789136854.599487 mapread 0x020f6b thru 0x021f5b (0xff1 bytes) 000971 1789136854.616963 mapread 0x0062e8 thru 0x010353 (0xa06c bytes) 000972 1789136854.633504 read 0x002350 thru 0x005929 (0x35da bytes) 000973 1789136854.635326 write 0x03be82 thru 0x03ffff (0x417e bytes) 000974 1789136854.724388 write 0x02dfb2 thru 0x03028d (0x22dc bytes) 000977 1789136854.738154 read 0x017060 thru 0x02405f (0xd000 bytes) 000978 1789136854.752686 mapread 0x0294e3 thru 0x02f256 (0x5d74 bytes) 000979 1789136854.754715 mapwrite 0x01e6cc thru 0x029886 (0xb1bb bytes) 000981 1789136854.822097 read 0x035c07 thru 0x0362cc (0x6c6 bytes) 000982 1789136854.822758 mapread 0x018624 thru 0x01d8ff (0x52dc bytes) 000983 1789136854.834865 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x96a32cc0 (stale) Mirror 12: 0x96a32cc0 (stale) Mirror 13: 0x96a32cc0 (stale) Mirror 14: 0x96a32cc0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe14d5f66 Mirror 15: 0xe14d5f66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000984 1789136855.899113 write 0x022289 thru 0x02cb1c (0xa894 bytes) 000985 1789136856.016108 write 0x00db5f thru 0x016f0f (0x93b1 bytes) 000990 1789136856.035822 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bf:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x96a32cc0 (stale) Mirror 12: 0x96a32cc0 (stale) Mirror 13: 0x96a32cc0 (stale) Mirror 14: 0x96a32cc0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x96a32cc0 (stale) Mirror 12: 0x96a32cc0 (stale) Mirror 13: 0x96a32cc0 (stale) Mirror 14: 0x96a32cc0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000993 1789136856.493243 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x96a32cc0 (stale) Mirror 12: 0x96a32cc0 (stale) Mirror 13: 0x96a32cc0 (stale) Mirror 14: 0x96a32cc0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe17af092 Mirror 15: 0xe17af092 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000994 1789136857.403476 trunc from 0x040000 to 0x0207b6 (0x1f849 bytes) 000995 1789136857.523187 mapread 0x016ff5 thru 0x0207b6 (0x97c2 bytes) 000998 1789136857.542539 write 0x02756a thru 0x03586c (0xe303 bytes) 001000 1789136857.549148 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3586d): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3586d): Mirror 8: 0xc393624c (stale) Mirror 12: 0xc393624c (stale) Mirror 13: 0xc393624c (stale) Mirror 14: 0xc393624c (stale) Mirror 15: 0x3bdf7d81 (stale) CRC-32 checksum value for chunk [0, 0x3586d): Mirror 1: 0x9f1d170 Mirror 16: 0x9f1d170 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3586d: skipped All operations completed A-OK! PASS 70a (107s) == sanity-flr test 71: check mirror extend parent fid ==== 10:27:43 (1789136863) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 10:27:46 (1789136866) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.294597 s, 7.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131851 s, 8.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 10:27:53 (1789136873) Starting client: oleg638-client.virtnet: -o user_xattr,flock 192.168.206.138@tcp:/lustre /mnt/lustre2 Starting client: oleg638-client.virtnet: -o user_xattr,flock 192.168.206.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1143936 .. Extending file size to 1461568 .. failed Extending file size to 6724832 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 5085440 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6908672 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7102208 .. Extending file size to 7206176 .. Extending file size to 7890336 .. Extending file size to 8066368 .. Extending file size to 7045152 .. Extending file size to 7601024 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8023488 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9144800 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8137856 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8213600 .. Extending file size to 9160448 .. Extending file size to 8232704 .. done Waiting 59022 59023 59024 59025 59028 192.168.206.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg638-client.virtnet /mnt/lustre2 (opts:) 192.168.206.138@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg638-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (76s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:29:09 (1789136949) Starting client: oleg638-client.virtnet: -o user_xattr,flock 192.168.206.138@tcp:/lustre /mnt/lustre2 Starting client: oleg638-client.virtnet: -o user_xattr,flock 192.168.206.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5787136 .. Extending file size to 8104928 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9114848 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8310432 .. Extending file size to 8477248 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8815264 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 60739 60740 60742 60743 60746 done 192.168.206.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg638-client.virtnet /mnt/lustre2 (opts:) 192.168.206.138@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg638-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:30:25 (1789137025) Starting client: oleg638-client.virtnet: -o user_xattr,flock 192.168.206.138@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119876 s, 17.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.364709 s, 5.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x833f85e (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.206.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg638-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:30:44 (1789137044) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150299 s, 14.0 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 10:30:52 (1789137052) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x216:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33f:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33f:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x340:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x323:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:30:59 (1789137059) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.368567 s, 5.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x341:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x324:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x325:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:31:07 (1789137067) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.232217 s, 9.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x343:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x326:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x343:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x326:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x327:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:31:15 (1789137075) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.183264 s, 11.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x344:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x328:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x344:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x328:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x329:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:31:48 (1789137108) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg638-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127346 s, 8.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x345:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x345:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x346:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:32:46 (1789137166) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg638-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x347:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.247545 s, 8.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x347:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x347:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (61s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:33:46 (1789137226) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg638-server mds-ost sync done. Creating new pool test_204f oleg638-server: Pool lustre.test_204f created Adding targets to pool oleg638-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg638-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199756 s, 10.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg638-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg638-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg638-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:35:01 (1789137301) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:35:08 (1789137308) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:35:15 (1789137315) oleg638-server: Pool lustre.test_206 created Adding targets to pool oleg638-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg638-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg638-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg638-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg638-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:35:38 (1789137338) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422377 s, 24.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279353 s, 37.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x90be7ebf Mirror 3: 0x90be7ebf PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:35:49 (1789137349) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:35:52 (1789137352) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:35:56 (1789137356) Starting client: oleg638-client.virtnet: -o user_xattr,flock 192.168.206.138@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.121511 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1320000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1be0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1be0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1be0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2480000 Read this (4 bytes): BBBB 192.168.206.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg638-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:36:31 (1789137391) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0221096 s, 371 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:36:37 (1789137397) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066784 s, 15.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x23b lmm_fid: [0x200000404:0x23b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x23b lmm_fid: [0x200000404:0x23b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x337:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834491 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 10:36:38.000000000 -0400 Modify: 2026-09-11 10:36:39.000000000 -0400 Change: 2026-09-11 10:36:40.000000000 -0400 Birth: 2026-09-11 10:36:39.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:36:44 (1789137404) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:36:47 (1789137407) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.449047 s, 91.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0514545 s, 79.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 11 10:36 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 11 10:36 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:36:54 (1789137414) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:37:00 (1789137420) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1728 sec =========== 10:37:09 (1789137429) osc.lustre-OST0000-osc-ffff8f6243853800.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffff8f6243853800.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 obdfilter.lustre-OST0000.stats_compr= snapshot_time 1789137432.370430367 secs.nsecs start_time 1789136166.559703270 secs.nsecs elapsed_time 1265.810727097 secs.nsecs uncompressed_objects 2816 samples [reqs] obdfilter.lustre-OST0001.stats_compr= snapshot_time 1789137432.370488548 secs.nsecs start_time 1789136200.154421127 secs.nsecs elapsed_time 1232.216067421 secs.nsecs uncompressed_objects 2838 samples [reqs] === sanity-flr: start cleanup 10:37:11 (1789137431) === === sanity-flr: finish cleanup 10:37:17 (1789137437) ===