-----============= acceptance-small: sanity-flr ============----- Thu Jun 11 18:21:03 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg617-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 18:21:13 (1781216473) === oleg617-client.virtnet: executing check_config_client /mnt/lustre oleg617-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg617-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff952685f5f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff952685f5f000.idle_timeout=debug disable quota as required oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 18:21:31 (1781216491) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8207] == sanity-flr test 0a: lfs mirror create with -N option == 18:21:32 (1781216492) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:21:39 (1781216499) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:21:41 (1781216501) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:21:42 (1781216502) PASS 0d (12s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:21:54 (1781216514) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:21:56 (1781216516) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 18:21:57 (1781216517) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff952685f5f000.stats=clear osc.lustre-OST0001-osc-ffff952685f5f000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:22:01 (1781216521) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--compress|-Z] [--compress-chunk=SIZE] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:22:06 (1781216526) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:22:10 (1781216530) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 18:22:12 (1781216532) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:22:15 (1781216535) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 18:22:20 (1781216540) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:22:23 (1781216543) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:22:27 (1781216547) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:22:30 (1781216550) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:22:35 (1781216555) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:22:38 (1781216558) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.693744 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480357 s, 21.8 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:22:43 (1781216563) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.264052 s, 79.4 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 18:22:47 (1781216567) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:22:51 (1781216571) Stopping /mnt/lustre-ost1 (opts:) on oleg617-server oleg617-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff952685f5f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff952685f5f000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg617-server: oleg617-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg617-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg617-server oleg617-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff952685f5f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff952685f5f000.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg617-server: oleg617-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg617-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (37s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:23:28 (1781216608) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.147225 s, 42.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg617-server oleg617-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff952685f5f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff952685f5f000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg617-server: oleg617-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg617-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (17s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:23:45 (1781216625) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg617-server oleg617-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff952685f5f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff952685f5f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg617-server: oleg617-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg617-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg617-server oleg617-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff952685f5f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff952685f5f000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg617-server: oleg617-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg617-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (37s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:24:22 (1781216662) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.22883 s, 42.7 MB/s -rw-r--r-- 1 root root 52428800 Jun 11 18:24 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.453s user 0m0.000s sys 0m0.170s Stopping /mnt/lustre-ost1 (opts:) on oleg617-server oleg617-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff952685f5f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff952685f5f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.545s user 0m0.001s sys 0m0.178s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg617-server: oleg617-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg617-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg617-server oleg617-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff952685f5f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff952685f5f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.552s user 0m0.000s sys 0m0.179s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg617-server: oleg617-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg617-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg617-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff952685f5f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff952685f5f000.ost_server_uuid in FULL state after 0 sec PASS 33b (40s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:25:02 (1781216702) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:25:04 (1781216704) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:25:06 (1781216706) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 18:25:07 (1781216707) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 18:25:11 (1781216711) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.307913 s, 40.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:25:17 (1781216717) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153981 s, 68.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015991 s, 65.6 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:25:21 (1781216721) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:25:26 (1781216726) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00246857 s, 415 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00161831 s, 633 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000739953 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000949693 s, 1.1 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 18:25:30 (1781216730) create test files with size 15748869 .. 728d6e62c5e1866a2bae6c0c7110b484 - b8436bff3f03a2cf6c99dd60d4802b0b - b690fc7697480af92551c114d36cdc23 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg617-client.virtnet /mnt/lustre (opts:) Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre PASS 37 (10s) == sanity-flr test 38: resync ============================ 18:25:40 (1781216740) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:25:46 (1781216746) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:25:49 (1781216749) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033989 s, 30.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154162 s, 68.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 18:25:53 (1781216753) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0480999 s, 43.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.136278 s, 30.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0791335 s, 26.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (7s) == sanity-flr test 42: lfs mirror verify ================= 18:26:00 (1781216760) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 18:26:02 (1781216762) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:26:03 (1781216763) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225401 s, 46.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 43c: read prefer uncompressed mirror == 18:26:07 (1781216767) llite.lustre-ffff952687ea8000.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (4s) == sanity-flr test 44a: lfs mirror split check =========== 18:26:11 (1781216771) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.156492 s, 20.1 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:26:15 (1781216775) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:26:20 (1781216780) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.137017 s, 76.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg617-client.virtnet /mnt/lustre (opts:) Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:26:28 (1781216788) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9526885f3800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.131162 s, 79.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg617-client.virtnet /mnt/lustre (opts:) Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff95268461b000.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:26:35 (1781216795) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (18s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:26:53 (1781216813) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (15s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:27:09 (1781216829) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 18:27:10 (1781216830) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.117353 s, 26.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.06722 s, 46.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x50bff0b8 (nosync,stale) Mirror 2: 0x50bff0b8 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 60bf18667918890101dc94cfab5cd21c - mirror 1 checksum: 761ec812c12852cb16a18618fb1b928d - mirror 2 checksum: 60bf18667918890101dc94cfab5cd21c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 18:27:10 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 18:27:10 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 60bf18667918890101dc94cfab5cd21c - mirror 1 checksum: 761ec812c12852cb16a18618fb1b928d - mirror 2 checksum: 761ec812c12852cb16a18618fb1b928d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 18:27:10 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 18:27:10 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 18:27:14 (1781216834) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144516 s, 72.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:27:18 (1781216838) Failing mds1 on oleg617-server Stopping /mnt/lustre-mds1 (opts:) on oleg617-server 18:27:26 (1781216846) shut down facet: mds1 facet_host: oleg617-server facet_failover_host: oleg617-server Failover mds1 to oleg617-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-MDT0000 18:27:42 (1781216862) targets are mounted 18:27:42 (1781216862) facet_failover done oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161235 s, 65.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } PASS 50A (33s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:27:51 (1781216871) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0333486 s, 614 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0102761 s, 2.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:27:55 (1781216875) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.034529 s, 593 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00538815 s, 3.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:28:00 (1781216880) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00462674 s, 3.5 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:28:03 (1781216883) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.025757 s, 795 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:28:07 (1781216887) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00650549 s, 630 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0199894 s, 205 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:28:10 (1781216890) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x13:0x0]: Invalid argument (22) PASS 60b (10s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:28:20 (1781216900) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:28:41 (1781216921) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:28:59 (1781216939) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:29:18 (1781216958) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 18:29:21 (1781216961) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57685 mirror split pid 57686 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57685 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57686 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 18:30:25 (1781217025) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 7025 000001 1781217025.990586 mapwrite 0x017fb0 thru 0x024690 (0xc6e1 bytes) fallocating to largest ever: 0x13d4f 000002 1781217026.042047 fallocate 0x005063 thru 0x013d4e (0xecec bytes) fallocating to largest ever: 0x31a47 000003 1781217026.045901 fallocate 0x02d8ba thru 0x031a46 (0x418d bytes) 000005 1781217026.049750 trunc from 0x017b80 to 0x01fd40 (0x81c1 bytes) 000007 1781217026.066225 read 0x0083e8 thru 0x00fa2a (0x7643 bytes) 000008 1781217026.073782 mapwrite 0x01816a thru 0x022569 (0xa400 bytes) 000009 1781217026.108510 write 0x00c494 thru 0x018185 (0xbcf2 bytes) 000010 1781217026.110530 punch from 0x010461 to 0x012fc8 (0x2b68 bytes) 000011 1781217026.132715 trunc from 0x00505c to 0x0070cc (0x2071 bytes) 000012 1781217026.136733 punch from 0x019504 to 0x021bb4 (0x86b1 bytes) 000013 1781217026.142678 mapread 0x00b74e thru 0x015f08 (0xa7bb bytes) 000014 1781217026.148183 trunc from 0x009cf2 to 0x014ebc (0xb1cb bytes) 000015 1781217026.157409 trunc from 0x011155 to 0x01bb67 (0xaa13 bytes) 000016 1781217026.162358 mapread 0x005117 thru 0x00b676 (0x6560 bytes) 000017 1781217026.171095 write 0x030521 thru 0x037541 (0x7021 bytes) 000018 1781217026.172566 punch from 0x029d39 to 0x02c47a (0x2742 bytes) 000020 1781217026.180200 trunc from 0x005477 to 0x007b05 (0x268f bytes) 000021 1781217026.184262 mapwrite 0x03d8c4 thru 0x03ffff (0x273c bytes) 000022 1781217026.221116 trunc from 0x019ba9 to 0x0282aa (0xe702 bytes) 000023 1781217026.225901 mapwrite 0x0200f7 thru 0x02ac85 (0xab8f bytes) fallocating to largest ever: 0x3259a 000024 1781217026.260402 fallocate 0x023bfd thru 0x032599 (0xe99d bytes) 000026 1781217026.265137 write 0x005baf thru 0x0127e4 (0xcc36 bytes) 000027 1781217026.272632 mapwrite 0x033ede thru 0x03ffff (0xc122 bytes) 000028 1781217026.291806 trunc from 0x040000 to 0x023970 (0x1c68f bytes) 000029 1781217026.302348 punch from 0x00d31e to 0x01c21f (0xef02 bytes) 000030 1781217026.308545 trunc from 0x00ef2f to 0x01388b (0x495d bytes) 000031 1781217026.315702 mapwrite 0x01a427 thru 0x01ecd4 (0x48ae bytes) 000032 1781217026.338945 read 0x01ad57 thru 0x01badf (0xd89 bytes) 000034 1781217026.340186 trunc from 0x01339a to 0x021b36 (0xe79d bytes) 000035 1781217026.348024 mapwrite 0x02020e thru 0x028fcd (0x8dc0 bytes) 000036 1781217026.384276 read 0x0122e3 thru 0x0186c0 (0x63de bytes) 000038 1781217026.386330 mapread 0x010d58 thru 0x016520 (0x57c9 bytes) 000039 1781217026.387617 read 0x002f9b thru 0x010063 (0xd0c9 bytes) 000040 1781217026.395352 MIRROR_EXTEND to 2 mirrors 000041 1781217026.756677 mapwrite 0x008f05 thru 0x014e5a (0xbf56 bytes) 000042 1781217026.824645 read 0x011acb thru 0x01668c (0x4bc2 bytes) 000043 1781217026.825357 trunc from 0x028fce to 0x02aee1 (0x1f14 bytes) 000044 1781217026.848767 punch from 0x00b68f to 0x00e6f3 (0x3065 bytes) 000045 1781217026.856005 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2aee2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aee2): Mirror 2: 0xbaab60e9 (stale) CRC-32 checksum value for chunk [0, 0x2aee2): Mirror 1: 0x7f7422b0 Mirror 3: 0x7f7422b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aee2: skipped 000046 1781217027.457365 read 0x00e09d thru 0x01a9ba (0xc91e bytes) 000048 1781217027.465850 mapread 0x01f55d thru 0x020302 (0xda6 bytes) 000049 1781217027.474724 punch from 0x023fc2 to 0x02667c (0x26bb bytes) fallocating to largest ever: 0x3fb63 000050 1781217027.501532 fallocate 0x03d376 thru 0x03fb62 (0x27ed bytes) 000051 1781217027.508125 trunc from 0x02aee2 to 0x0274a8 (0x3a39 bytes) 000053 1781217027.535555 fallocate 0x0165e7 thru 0x01711c (0xb36 bytes) 000054 1781217027.545176 fallocate 0x02bb5c thru 0x0369a7 (0xae4c bytes) 000055 1781217027.549702 trunc from 0x0274a9 to 0x0038fc (0x23bac bytes) 000056 1781217027.567686 write 0x03a031 thru 0x03ffff (0x5fcf bytes) 000057 1781217027.569553 trunc from 0x040000 to 0x016077 (0x29f88 bytes) 000058 1781217027.585696 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x16078): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16078): Mirror 2: 0xa5a1ae92 (stale) Mirror 3: 0x8b929066 (stale) CRC-32 checksum value for chunk [0, 0x16078): Mirror 1: 0xa76bf50 Mirror 4: 0xa76bf50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16078: skipped 000059 1781217028.057978 trunc from 0x003eb6 to 0x00566c (0x17b7 bytes) 000060 1781217028.086737 mapread 0x0146ac thru 0x016077 (0x19cc bytes) 000062 1781217028.099593 trunc from 0x016078 to 0x00be34 (0xa243 bytes) 000063 1781217028.121181 write 0x03b2a8 thru 0x03ea55 (0x37ae bytes) 000064 1781217028.125473 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ea56): Mirror 2: 0x3a57f805 (stale) Mirror 3: 0xe8a37c2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea56): Mirror 2: 0x3a57f805 (stale) Mirror 3: 0xe8a37c2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea56: skipped 000066 1781217028.369656 trunc from 0x03ea56 to 0x01d3de (0x21677 bytes) 000067 1781217028.382796 mapwrite 0x01a22f thru 0x0272d1 (0xd0a3 bytes) 000068 1781217028.441672 fallocate 0x021c73 thru 0x023237 (0x15c5 bytes) 000069 1781217028.447929 punch from 0x01c354 to 0x02209d (0x5d4a bytes) 000070 1781217028.455195 trunc from 0x017646 to 0x01d089 (0x5a44 bytes) 000071 1781217028.462030 punch from 0x01b95d to 0x01d987 (0x202b bytes) 000072 1781217028.465564 trunc from 0x020793 to 0x0272d1 (0x6b3f bytes) 000074 1781217028.472207 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x272d2): Mirror 3: 0xb58a7826 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x272d2): Mirror 3: 0xb58a7826 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x272d2: skipped 000076 1781217028.648008 write 0x006df9 thru 0x00f823 (0x8a2b bytes) 000077 1781217028.658804 write 0x03e97a thru 0x03ffff (0x1686 bytes) 000079 1781217028.660463 trunc from 0x01a405 to 0x01ea93 (0x468f bytes) 000080 1781217028.663720 mapread 0x01d514 thru 0x01eef2 (0x19df bytes) 000081 1781217028.670518 write 0x03f315 thru 0x03ffff (0xceb bytes) 000082 1781217028.671244 trunc from 0x040000 to 0x032745 (0xd8ba bytes) 000083 1781217028.683687 fallocate 0x01c235 thru 0x02178d (0x5559 bytes) 000084 1781217028.688247 trunc from 0x003235 to 0x007ef8 (0x4cc4 bytes) 000085 1781217028.698096 read 0x00dd04 thru 0x015ebc (0x81b9 bytes) 000086 1781217028.705960 mapread 0x0188c9 thru 0x025738 (0xce70 bytes) 000087 1781217028.712216 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000088 1781217028.812665 punch from 0x01548d to 0x022eb1 (0xda25 bytes) 000089 1781217028.817018 write 0x03e3e5 thru 0x03ffff (0x1c1b bytes) 000090 1781217028.821609 punch from 0x01e545 to 0x02e46d (0xff29 bytes) 000091 1781217028.830268 MIRROR_EXTEND to 2 mirrors 000092 1781217029.232662 MIRROR_EXTEND to 3 mirrors 000093 1781217029.674020 read 0x02b342 thru 0x03b30a (0xffc9 bytes) 000094 1781217029.680655 trunc from 0x040000 to 0x02c362 (0x13c9d bytes) 000095 1781217029.737815 trunc from 0x019562 to 0x022de5 (0x9884 bytes) 000096 1781217029.748597 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c363): Mirror 3: 0x7039f994 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c363): Mirror 3: 0x7039f994 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c363: skipped 000097 1781217029.976927 punch from 0x02070b to 0x02557a (0x4e70 bytes) 000098 1781217029.980699 trunc from 0x00be4b to 0x01a6a8 (0xe85e bytes) 000099 1781217029.989867 trunc from 0x02c363 to 0x00bdeb (0x20577 bytes) 000101 1781217030.005489 fallocate 0x029ec6 thru 0x0349a2 (0xaadd bytes) 000102 1781217030.009680 fallocate 0x02df7c thru 0x033dff (0x5e84 bytes) 000103 1781217030.014088 read 0x013be1 thru 0x01e62c (0xaa4c bytes) 000104 1781217030.028881 write 0x019c14 thru 0x029b8c (0xff79 bytes) 000105 1781217030.038530 trunc from 0x0349a3 to 0x03da72 (0x90d0 bytes) 000106 1781217030.057348 punch from 0x003580 to 0x0046e2 (0x1163 bytes) 000107 1781217030.063590 write 0x01926f thru 0x028eb8 (0xfc4a bytes) 000108 1781217030.065635 trunc from 0x03da73 to 0x02df30 (0xfb42 bytes) 000109 1781217030.084893 write 0x015653 thru 0x01affe (0x59ac bytes) 000110 1781217030.087241 fallocate 0x00d8ee thru 0x01a544 (0xcc57 bytes) 000111 1781217030.100287 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2df31): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df31): Mirror 3: 0x44fbf56a (stale) CRC-32 checksum value for chunk [0, 0x2df31): Mirror 1: 0xbe807bf9 Mirror 4: 0xbe807bf9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df31: skipped 000112 1781217030.592755 punch from 0x01f337 to 0x021992 (0x265c bytes) 000113 1781217030.618087 trunc from 0x02df31 to 0x0331f0 (0x52c0 bytes) 000114 1781217030.664718 trunc from 0x0331f1 to 0x01e8d1 (0x1491f bytes) 000115 1781217030.682603 fallocate 0x02ef35 thru 0x039d0e (0xadda bytes) 000116 1781217030.691132 trunc from 0x0352f7 to 0x0377f9 (0x2503 bytes) 000117 1781217030.695982 mapwrite 0x02ef15 thru 0x037e71 (0x8f5d bytes) 000118 1781217030.735198 trunc from 0x00c972 to 0x012acf (0x615e bytes) 000119 1781217030.738440 read 0x037215 thru 0x039d0e (0x2afa bytes) 000120 1781217030.752945 write 0x0250f1 thru 0x02fe0c (0xad1c bytes) 000121 1781217030.762795 mapwrite 0x000e94 thru 0x00e507 (0xd674 bytes) 000122 1781217030.792995 mapwrite 0x0121af thru 0x020876 (0xe6c8 bytes) 000123 1781217030.838050 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39d0f): Mirror 4: 0xa23bdcce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d0f): Mirror 4: 0xa23bdcce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d0f: skipped 000125 1781217031.029504 mapwrite 0x03bf6f thru 0x03d5f2 (0x1684 bytes) 000126 1781217031.079728 write 0x03b353 thru 0x03d31d (0x1fcb bytes) 000127 1781217031.081019 mapread 0x02ac29 thru 0x032683 (0x7a5b bytes) 000129 1781217031.092504 mapwrite 0x014350 thru 0x023047 (0xecf8 bytes) 000130 1781217031.127799 read 0x03744d thru 0x03d5f2 (0x61a6 bytes) 000131 1781217031.128286 trunc from 0x0011e0 to 0x00e42e (0xd24f bytes) 000132 1781217031.138992 MIRROR_RESYNC 2 mirrors 000133 1781217031.553932 MIRROR_RESYNC 2 mirrors 000134 1781217031.779355 trunc from 0x03d5f3 to 0x01f398 (0x1e25a bytes) 000135 1781217031.850246 punch from 0x01cae6 to 0x01f398 (0x28b3 bytes) 000136 1781217031.858959 MIRROR_RESYNC 2 mirrors 000140 1781217032.218552 fallocate 0x027155 thru 0x033db0 (0xcc5c bytes) 000141 1781217032.243836 write 0x0282b0 thru 0x02ef1f (0x6c70 bytes) 000142 1781217032.254239 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33db1): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33db1): Mirror 4: 0x4cb5e1a2 (stale) CRC-32 checksum value for chunk [0, 0x33db1): Mirror 1: 0xdad3ab83 Mirror 5: 0xdad3ab83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33db1: skipped 000143 1781217032.831913 mapread 0x013da5 thru 0x01c5c4 (0x8820 bytes) 000144 1781217032.847103 read 0x00c06b thru 0x01589b (0x9831 bytes) 000145 1781217032.853894 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000146 1781217033.044001 read 0x0189cd thru 0x025a91 (0xd0c5 bytes) 000147 1781217033.053824 mapwrite 0x029000 thru 0x034da0 (0xbda1 bytes) 000149 1781217033.114051 write 0x02c9f4 thru 0x039893 (0xcea0 bytes) 000150 1781217033.115852 read 0x035b7d thru 0x039893 (0x3d17 bytes) 000151 1781217033.117113 trunc from 0x016e90 to 0x0183fa (0x156b bytes) 000152 1781217033.121024 punch from 0x0198bc to 0x01c0a4 (0x27e9 bytes) 000153 1781217033.124966 read 0x016fbb thru 0x025151 (0xe197 bytes) 000154 1781217033.132939 trunc from 0x029c40 to 0x03450b (0xa8cc bytes) 000155 1781217033.142663 punch from 0x01bda2 to 0x02298e (0x6bed bytes) 000156 1781217033.147516 fallocate 0x01b663 thru 0x01c5a3 (0xf41 bytes) 000157 1781217033.154192 trunc from 0x039894 to 0x027fae (0x118e5 bytes) 000159 1781217033.167917 trunc from 0x0057a2 to 0x009f84 (0x47e3 bytes) 000160 1781217033.173635 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000161 1781217033.252613 MIRROR_EXTEND to 2 mirrors 000162 1781217033.689541 trunc from 0x0185a7 to 0x02767e (0xf0d8 bytes) 000163 1781217033.721734 mapread 0x026699 thru 0x027fae (0x1916 bytes) 000164 1781217033.736949 mapwrite 0x0067f0 thru 0x00bd18 (0x5529 bytes) 000166 1781217033.763985 trunc from 0x014e92 to 0x01e812 (0x9981 bytes) 000167 1781217033.768311 mapread 0x0154d6 thru 0x01b619 (0x6144 bytes) 000168 1781217033.779115 punch from 0x00ad7f to 0x0167fa (0xba7c bytes) 000169 1781217033.787110 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000170 1781217033.896224 mapwrite 0x03e839 thru 0x03ffff (0x17c7 bytes) 000172 1781217033.947929 read 0x01e0cc thru 0x0243c1 (0x62f6 bytes) 000173 1781217033.960042 punch from 0x0135e3 to 0x01e24a (0xac68 bytes) 000174 1781217033.967216 MIRROR_EXTEND to 2 mirrors 000175 1781217034.509330 read 0x016fb5 thru 0x01d790 (0x67dc bytes) 000176 1781217034.518337 read 0x0321d5 thru 0x032487 (0x2b3 bytes) 000178 1781217034.529269 trunc from 0x015445 to 0x01ec69 (0x9825 bytes) 000180 1781217034.562966 trunc from 0x040000 to 0x003495 (0x3cb6a bytes) 000182 1781217034.588515 mapwrite 0x03b914 thru 0x03ffff (0x46ec bytes) 000183 1781217034.648533 trunc from 0x040000 to 0x0275d1 (0x18a2e bytes) 000184 1781217034.662497 trunc from 0x0275d2 to 0x02ba38 (0x4467 bytes) 000185 1781217034.676833 punch from 0x0257d1 to 0x029a9f (0x42cf bytes) 000186 1781217034.684592 read 0x014476 thru 0x016692 (0x221d bytes) 000187 1781217034.693890 mapread 0x000e39 thru 0x00bf22 (0xb0ea bytes) 000188 1781217034.695308 mapwrite 0x00e617 thru 0x01bab3 (0xd49d bytes) 000189 1781217034.716260 mapread 0x00b718 thru 0x010075 (0x495e bytes) 000190 1781217034.717368 read 0x0162ae thru 0x02410c (0xde5f bytes) 000191 1781217034.718036 mapwrite 0x0013f1 thru 0x00d3b1 (0xbfc1 bytes) 000192 1781217034.743093 fallocate 0x0375a7 thru 0x03a624 (0x307e bytes) 000193 1781217034.747985 write 0x000913 thru 0x00fe27 (0xf515 bytes) 000194 1781217034.749414 write 0x0186f5 thru 0x025e85 (0xd791 bytes) 000195 1781217034.750724 trunc from 0x03a625 to 0x03c9cf (0x23ab bytes) 000196 1781217034.768476 mapwrite 0x00e965 thru 0x0180b3 (0x974f bytes) 000199 1781217034.790341 write 0x03e653 thru 0x03ffff (0x19ad bytes) fallocating to largest ever: 0x40000 000200 1781217034.791466 fallocate 0x03d641 thru 0x03ffff (0x29bf bytes) 000201 1781217034.800417 punch from 0x011d17 to 0x01e0f9 (0xc3e3 bytes) 000202 1781217034.807827 punch from 0x00c282 to 0x010273 (0x3ff2 bytes) 000203 1781217034.812658 mapwrite 0x01fb9b thru 0x025407 (0x586d bytes) 000204 1781217034.833549 write 0x0264dd thru 0x0349cb (0xe4ef bytes) 000205 1781217034.850093 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5c713b93 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x52414d3b Mirror 3: 0x52414d3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000206 1781217035.526014 mapread 0x017eb2 thru 0x01c367 (0x44b6 bytes) 000207 1781217035.539704 punch from 0x00f2b6 to 0x019ca2 (0xa9ed bytes) 000208 1781217035.563123 trunc from 0x00e8ab to 0x013854 (0x4faa bytes) 000209 1781217035.569809 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x52414d3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x52414d3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000210 1781217035.752240 mapread 0x02ba6e thru 0x03941f (0xd9b2 bytes) 000211 1781217035.775435 punch from 0x023ffc to 0x0298a6 (0x58ab bytes) 000212 1781217035.780455 trunc from 0x0274a8 to 0x02fc09 (0x8762 bytes) 000213 1781217035.788666 read 0x00a797 thru 0x01475e (0x9fc8 bytes) 000214 1781217035.795994 write 0x0338c3 thru 0x03bdc7 (0x8505 bytes) 000215 1781217035.800502 mapread 0x01fa95 thru 0x02bc46 (0xc1b2 bytes) 000216 1781217035.821964 mapread 0x01b7d7 thru 0x022dcc (0x75f6 bytes) 000217 1781217035.823125 mapwrite 0x03e8d6 thru 0x03ffff (0x172a bytes) 000218 1781217035.846198 MIRROR_RESYNC 2 mirrors 000219 1781217036.315171 MIRROR_RESYNC 2 mirrors 000220 1781217036.515260 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000221 1781217036.604891 punch from 0x034e55 to 0x03f01f (0xa1cb bytes) 000222 1781217036.618480 trunc from 0x0296f5 to 0x0323d5 (0x8ce1 bytes) 000224 1781217036.624198 trunc from 0x0356f0 to 0x03cfa2 (0x78b3 bytes) 000225 1781217036.632942 mapread 0x03d42c thru 0x03ffff (0x2bd4 bytes) 000226 1781217036.645095 punch from 0x01507c to 0x01e7f1 (0x9776 bytes) 000227 1781217036.648893 punch from 0x02c342 to 0x032fcc (0x6c8b bytes) 000228 1781217036.651760 write 0x0313e3 thru 0x03e7d6 (0xd3f4 bytes) 000229 1781217036.664002 write 0x00b2cc thru 0x00e2b3 (0x2fe8 bytes) 000230 1781217036.670633 trunc from 0x040000 to 0x033f42 (0xc0bd bytes) 000232 1781217036.683696 trunc from 0x033f43 to 0x033b06 (0x43c bytes) 000233 1781217036.696563 trunc from 0x033b07 to 0x01bbbb (0x17f4b bytes) 000234 1781217036.713497 punch from 0x000e73 to 0x0092c4 (0x8452 bytes) 000235 1781217036.726468 MIRROR_EXTEND to 2 mirrors 000236 1781217037.183597 read 0x010e92 thru 0x012df5 (0x1f64 bytes) 000237 1781217037.189516 write 0x034023 thru 0x035594 (0x1572 bytes) 000238 1781217037.218398 fallocate 0x01e4a4 thru 0x0208b5 (0x2412 bytes) 000239 1781217037.227924 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35595): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35595): Mirror 2: 0xb7356eff (stale) CRC-32 checksum value for chunk [0, 0x35595): Mirror 1: 0x2ffeb3b2 Mirror 3: 0x2ffeb3b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35595: skipped 000240 1781217037.891531 mapread 0x00dbfd thru 0x015dfd (0x8201 bytes) 000241 1781217037.914785 punch from 0x01e1f0 to 0x02541c (0x722d bytes) 000242 1781217037.941223 fallocate 0x013169 thru 0x01a951 (0x77e9 bytes) 000243 1781217037.947579 fallocate 0x0246af thru 0x02f0e0 (0xaa32 bytes) 000244 1781217037.955991 mapwrite 0x01b4d9 thru 0x021f61 (0x6a89 bytes) 000245 1781217037.979236 punch from 0x00f479 to 0x019620 (0xa1a8 bytes) 000246 1781217037.986968 fallocate 0x0076d3 thru 0x0154a1 (0xddcf bytes) 000247 1781217037.991567 punch from 0x009eea to 0x00c7dc (0x28f3 bytes) 000249 1781217037.996181 mapread 0x010322 thru 0x01f603 (0xf2e2 bytes) 000251 1781217038.010685 mapwrite 0x017bff thru 0x0200fc (0x84fe bytes) 000253 1781217038.029093 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35595): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35595): Mirror 2: 0xb7356eff (stale) Mirror 3: 0x2ffeb3b2 (stale) CRC-32 checksum value for chunk [0, 0x35595): Mirror 1: 0xc58f670b Mirror 4: 0xc58f670b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35595: skipped 000254 1781217038.688896 fallocate 0x0066b5 thru 0x00b8a2 (0x51ee bytes) 000255 1781217038.733794 read 0x01dca8 thru 0x02a02d (0xc386 bytes) 000256 1781217038.747601 fallocate 0x039493 thru 0x03ddb6 (0x4924 bytes) 000257 1781217038.752571 trunc from 0x009e42 to 0x015d5d (0xbf1c bytes) 000258 1781217038.757181 write 0x003318 thru 0x00f3b6 (0xc09f bytes) 000260 1781217038.777737 MIRROR_RESYNC 4 mirrors 000261 1781217039.869956 MIRROR_EXTEND to 5 mirrors 000262 1781217040.430108 read 0x02e042 thru 0x03cfd9 (0xef98 bytes) 000263 1781217040.439482 trunc from 0x03ddb7 to 0x00997e (0x34438 bytes) 000264 1781217040.485953 trunc from 0x00997f to 0x01c472 (0x12af4 bytes) 000265 1781217040.515907 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1c473): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c473): Mirror 2: 0x31497dd4 (stale) Mirror 3: 0x31497dd4 (stale) Mirror 4: 0x31497dd4 (stale) Mirror 5: 0x31497dd4 (stale) CRC-32 checksum value for chunk [0, 0x1c473): Mirror 1: 0xd98fc42b Mirror 6: 0xd98fc42b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c473: skipped 000266 1781217041.195939 mapread 0x01a169 thru 0x01c472 (0x230a bytes) 000268 1781217041.214145 trunc from 0x0178f1 to 0x01c472 (0x4b82 bytes) 000269 1781217041.252091 trunc from 0x01c473 to 0x01a0ef (0x2383 bytes) 000270 1781217041.272624 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1a0f0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a0f0): Mirror 2: 0xf53c21b6 (stale) Mirror 3: 0xf53c21b6 (stale) Mirror 4: 0xf53c21b6 (stale) Mirror 5: 0xf53c21b6 (stale) Mirror 6: 0x49b2c0f6 (stale) CRC-32 checksum value for chunk [0, 0x1a0f0): Mirror 1: 0x49b2c0f6 Mirror 7: 0x49b2c0f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a0f0: skipped 000271 1781217041.915966 mapread 0x0157ce thru 0x016642 (0xe75 bytes) 000272 1781217041.928707 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1a0f0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a0f0): Mirror 2: 0xf53c21b6 (stale) Mirror 3: 0xf53c21b6 (stale) Mirror 4: 0xf53c21b6 (stale) Mirror 5: 0xf53c21b6 (stale) Mirror 6: 0x49b2c0f6 (stale) CRC-32 checksum value for chunk [0, 0x1a0f0): Mirror 1: 0x49b2c0f6 Mirror 7: 0x49b2c0f6 Mirror 8: 0x49b2c0f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a0f0: skipped 000273 1781217042.544690 fallocate 0x002aec thru 0x00e5fb (0xbb10 bytes) 000275 1781217042.583647 mapwrite 0x013956 thru 0x01ee35 (0xb4e0 bytes) 000278 1781217042.656650 read 0x01c465 thru 0x01e28f (0x1e2b bytes) 000279 1781217042.658069 trunc from 0x00172f to 0x0106eb (0xefbd bytes) 000280 1781217042.662319 read 0x0164cd thru 0x019dab (0x38df bytes) 000281 1781217042.669261 MIRROR_RESYNC 8 mirrors 000282 1781217043.497861 trunc from 0x01ee36 to 0x02463e (0x5809 bytes) 000285 1781217043.530963 punch from 0x00d96d to 0x0132a5 (0x5939 bytes) 000286 1781217043.536597 punch from 0x019ad2 to 0x01d924 (0x3e53 bytes) 000287 1781217043.540024 mapwrite 0x009a4a thru 0x016e37 (0xd3ee bytes) 000288 1781217043.592678 write 0x020f18 thru 0x026c74 (0x5d5d bytes) 000289 1781217043.602883 write 0x012073 thru 0x01418b (0x2119 bytes) 000290 1781217043.604123 punch from 0x012b29 to 0x01c663 (0x9b3b bytes) 000291 1781217043.614744 mapread 0x017aa6 thru 0x0209a9 (0x8f04 bytes) 000292 1781217043.625831 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26c75): Mirror 3: 0x4e83948b (stale) Mirror 4: 0x4e83948b (stale) Mirror 5: 0x4e83948b (stale) Mirror 6: 0x4e83948b (stale) Mirror 7: 0x4e83948b (stale) Mirror 8: 0x4e83948b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26c75): Mirror 3: 0x4e83948b (stale) Mirror 4: 0x4e83948b (stale) Mirror 5: 0x4e83948b (stale) Mirror 6: 0x4e83948b (stale) Mirror 7: 0x4e83948b (stale) Mirror 8: 0x4e83948b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26c75: skipped 000293 1781217043.948725 write 0x02688b thru 0x02b4b9 (0x4c2f bytes) 000294 1781217043.956774 fallocate 0x02c339 thru 0x038274 (0xbf3c bytes) 000295 1781217043.961983 mapread 0x022c81 thru 0x025083 (0x2403 bytes) 000296 1781217043.971068 trunc from 0x0097e8 to 0x014a90 (0xb2a9 bytes) 000297 1781217043.976246 trunc from 0x01fd5d to 0x027acb (0x7d6f bytes) 000300 1781217043.988650 trunc from 0x02aea7 to 0x02b4b9 (0x613 bytes) 000301 1781217044.001034 MIRROR_RESYNC 7 mirrors 000302 1781217045.160701 trunc from 0x00a71d to 0x015ed4 (0xb7b8 bytes) 000303 1781217045.184591 fallocate 0x009b23 thru 0x00c9c8 (0x2ea6 bytes) 000304 1781217045.188121 mapread 0x01ea5e thru 0x023eec (0x548f bytes) 000305 1781217045.202918 mapread 0x012adc thru 0x01d08c (0xa5b1 bytes) 000306 1781217045.209401 mapwrite 0x018aee thru 0x01a3ce (0x18e1 bytes) 000308 1781217045.222294 write 0x008cf6 thru 0x00f041 (0x634c bytes) 000309 1781217045.228504 trunc from 0x02b4ba to 0x014036 (0x17483 bytes) 000310 1781217045.242107 read 0x00e1eb thru 0x014036 (0x5e4c bytes) 000311 1781217045.242942 punch from 0x00c5cd to 0x014036 (0x7a6a bytes) 000312 1781217045.256538 read 0x00af69 thru 0x00cb1b (0x1bb3 bytes) 000313 1781217045.267744 punch from 0x001ada to 0x01064d (0xeb74 bytes) 000314 1781217045.277681 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x14037): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14037): Mirror 3: 0x5f09703c (stale) Mirror 4: 0x5f09703c (stale) Mirror 5: 0x5f09703c (stale) Mirror 6: 0x5f09703c (stale) Mirror 7: 0x5f09703c (stale) Mirror 8: 0x5f09703c (stale) CRC-32 checksum value for chunk [0, 0x14037): Mirror 1: 0x5f09703c Mirror 9: 0x5f09703c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14037: skipped 000315 1781217045.756715 trunc from 0x014037 to 0x03b686 (0x27650 bytes) 000316 1781217045.808403 mapread 0x00d6af thru 0x01b63f (0xdf91 bytes) 000317 1781217045.825639 read 0x00c9cb thru 0x017e32 (0xb468 bytes) 000318 1781217045.826805 mapread 0x010e80 thru 0x0147d4 (0x3955 bytes) 000319 1781217045.830327 read 0x02462c thru 0x025b7d (0x1552 bytes) 000320 1781217045.838223 punch from 0x006ba0 to 0x00b098 (0x44f9 bytes) 000321 1781217045.844491 mapwrite 0x02efaa thru 0x03970f (0xa766 bytes) 000322 1781217045.867926 MIRROR_RESYNC 8 mirrors 000323 1781217046.780219 mapread 0x026265 thru 0x02f64a (0x93e6 bytes) 000324 1781217046.796905 mapwrite 0x023803 thru 0x023b75 (0x373 bytes) 000325 1781217046.857149 mapread 0x0082bd thru 0x00832a (0x6e bytes) 000326 1781217046.864246 mapread 0x000c13 thru 0x003546 (0x2934 bytes) 000327 1781217046.869872 punch from 0x011bd0 to 0x012771 (0xba2 bytes) 000328 1781217046.876520 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b687): Mirror 3: 0x2e0ee85f (stale) Mirror 4: 0x2e0ee85f (stale) Mirror 5: 0x2e0ee85f (stale) Mirror 6: 0x2e0ee85f (stale) Mirror 8: 0x2e0ee85f (stale) Mirror 9: 0x2e0ee85f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b687): Mirror 3: 0x2e0ee85f (stale) Mirror 4: 0x2e0ee85f (stale) Mirror 5: 0x2e0ee85f (stale) Mirror 6: 0x2e0ee85f (stale) Mirror 8: 0x2e0ee85f (stale) Mirror 9: 0x2e0ee85f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b687: skipped 000329 1781217047.189335 trunc from 0x03b687 to 0x035584 (0x6102 bytes) 000330 1781217047.208553 fallocate 0x00f250 thru 0x01e1d2 (0xef83 bytes) 000331 1781217047.215748 read 0x008aa9 thru 0x015d90 (0xd2e8 bytes) 000332 1781217047.226874 punch from 0x03537c to 0x035584 (0x209 bytes) 000333 1781217047.232200 write 0x02f9ef thru 0x039e92 (0xa4a4 bytes) 000334 1781217047.243116 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39e93): Mirror 3: 0x7ba7f704 (stale) Mirror 4: 0x7ba7f704 (stale) Mirror 5: 0x7ba7f704 (stale) Mirror 8: 0x7ba7f704 (stale) Mirror 9: 0x7ba7f704 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e93): Mirror 3: 0x7ba7f704 (stale) Mirror 4: 0x7ba7f704 (stale) Mirror 5: 0x7ba7f704 (stale) Mirror 8: 0x7ba7f704 (stale) Mirror 9: 0x7ba7f704 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e93: skipped 000335 1781217047.484834 fallocate 0x00635f thru 0x0107e0 (0xa482 bytes) 000336 1781217047.488572 mapwrite 0x01b76d thru 0x01d008 (0x189c bytes) 000337 1781217047.521221 mapwrite 0x03a1f5 thru 0x03ffff (0x5e0b bytes) 000338 1781217047.561541 write 0x00d8c3 thru 0x00f8c4 (0x2002 bytes) 000339 1781217047.571796 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaa85b13f (stale) Mirror 4: 0xaa85b13f (stale) Mirror 5: 0xaa85b13f (stale) Mirror 9: 0xaa85b13f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaa85b13f (stale) Mirror 4: 0xaa85b13f (stale) Mirror 5: 0xaa85b13f (stale) Mirror 9: 0xaa85b13f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000340 1781217047.773965 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaa85b13f (stale) Mirror 4: 0xaa85b13f (stale) Mirror 9: 0xaa85b13f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaa85b13f (stale) Mirror 4: 0xaa85b13f (stale) Mirror 9: 0xaa85b13f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000341 1781217047.996564 mapwrite 0x00f07f thru 0x01071d (0x169f bytes) 000342 1781217048.024574 MIRROR_RESYNC 4 mirrors 000343 1781217048.824081 mapread 0x006b5c thru 0x010ff4 (0xa499 bytes) 000344 1781217048.834836 mapwrite 0x011472 thru 0x0131cc (0x1d5b bytes) 000345 1781217048.903845 read 0x02cc40 thru 0x033848 (0x6c09 bytes) 000346 1781217048.909646 write 0x039702 thru 0x03fc26 (0x6525 bytes) 000347 1781217048.917248 fallocate 0x0231f9 thru 0x029561 (0x6369 bytes) 000348 1781217048.922863 trunc from 0x02c6f9 to 0x0334f0 (0x6df8 bytes) 000349 1781217048.929944 trunc from 0x040000 to 0x019a89 (0x26576 bytes) 000350 1781217048.944681 write 0x008f70 thru 0x010743 (0x77d4 bytes) 000352 1781217048.956839 read 0x0030dc thru 0x009dbb (0x6ce0 bytes) 000353 1781217048.960312 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x19a8a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19a8a): Mirror 3: 0x43468325 (stale) Mirror 4: 0x43468325 (stale) Mirror 9: 0x43468325 (stale) CRC-32 checksum value for chunk [0, 0x19a8a): Mirror 1: 0x78d4b578 Mirror 10: 0x78d4b578 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19a8a: skipped 000354 1781217049.549960 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x19a8a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19a8a): Mirror 3: 0x43468325 (stale) Mirror 4: 0x43468325 (stale) Mirror 9: 0x43468325 (stale) CRC-32 checksum value for chunk [0, 0x19a8a): Mirror 1: 0x78d4b578 Mirror 10: 0x78d4b578 Mirror 11: 0x78d4b578 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19a8a: skipped 000355 1781217050.089533 trunc from 0x019a8a to 0x0170fb (0x298e bytes) 000356 1781217050.127122 mapread 0x0052d2 thru 0x013750 (0xe47f bytes) 000357 1781217050.148625 fallocate 0x038ccc thru 0x03f3c1 (0x66f6 bytes) 000358 1781217050.153240 punch from 0x02e1d8 to 0x037964 (0x978d bytes) 000359 1781217050.157164 fallocate 0x019551 thru 0x020c7e (0x772e bytes) 000360 1781217050.161510 fallocate 0x037f3b thru 0x03ffff (0x80c5 bytes) 000361 1781217050.165724 write 0x03f513 thru 0x03ffff (0xaed bytes) 000362 1781217050.172648 write 0x000e18 thru 0x006606 (0x57ef bytes) 000363 1781217050.177735 trunc from 0x040000 to 0x0274e2 (0x18b1d bytes) 000365 1781217050.196761 read 0x014497 thru 0x015da5 (0x190f bytes) 000366 1781217050.197750 mapread 0x0022e1 thru 0x00451a (0x223a bytes) 000367 1781217050.199241 punch from 0x01862d to 0x01c9f4 (0x43c8 bytes) 000369 1781217050.204330 punch from 0x023f6d to 0x0274e2 (0x3576 bytes) 000371 1781217050.209242 mapread 0x00f009 thru 0x00f0ea (0xe2 bytes) 000372 1781217050.210257 mapwrite 0x0287f6 thru 0x02e773 (0x5f7e bytes) 000373 1781217050.248687 write 0x007169 thru 0x0104bc (0x9354 bytes) 000374 1781217050.249973 mapwrite 0x000151 thru 0x00f3d6 (0xf286 bytes) 000375 1781217050.270015 fallocate 0x031adf thru 0x03f92f (0xde51 bytes) 000376 1781217050.274712 punch from 0x000cfb to 0x00502a (0x4330 bytes) 000377 1781217050.278428 mapwrite 0x023425 thru 0x028f8e (0x5b6a bytes) 000379 1781217050.296778 trunc from 0x02e774 to 0x03cae8 (0xe375 bytes) 000380 1781217050.311397 read 0x00c0e6 thru 0x01ae62 (0xed7d bytes) 000381 1781217050.322764 mapwrite 0x03ac43 thru 0x03ecb5 (0x4073 bytes) 000382 1781217050.366264 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ecb6): Mirror 3: 0x5316109d (stale) Mirror 9: 0x5316109d (stale) Mirror 10: 0xe7b4e215 (stale) Mirror 11: 0xe7b4e215 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ecb6): Mirror 3: 0x5316109d (stale) Mirror 9: 0x5316109d (stale) Mirror 10: 0xe7b4e215 (stale) Mirror 11: 0xe7b4e215 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ecb6: skipped 000383 1781217050.646052 punch from 0x019a86 to 0x023de9 (0xa364 bytes) 000384 1781217050.651870 punch from 0x00dfa0 to 0x010413 (0x2474 bytes) 000385 1781217050.658105 write 0x03a388 thru 0x03ffff (0x5c78 bytes) 000386 1781217050.666318 mapwrite 0x00e445 thru 0x01996f (0xb52b bytes) 000387 1781217050.707881 MIRROR_RESYNC 5 mirrors 000388 1781217051.669300 read 0x014b4e thru 0x0155be (0xa71 bytes) 000389 1781217051.673514 punch from 0x01a640 to 0x029066 (0xea27 bytes) 000390 1781217051.703790 write 0x03c0a7 thru 0x03ffff (0x3f59 bytes) 000391 1781217051.709798 trunc from 0x040000 to 0x01c595 (0x23a6a bytes) 000392 1781217051.726555 write 0x03e242 thru 0x03ffff (0x1dbe bytes) 000394 1781217051.728765 mapwrite 0x029c1c thru 0x030aba (0x6e9f bytes) 000395 1781217051.754392 trunc from 0x00853b to 0x00b56c (0x3032 bytes) 000396 1781217051.758244 trunc from 0x040000 to 0x01ce6e (0x23191 bytes) 000398 1781217051.772089 mapwrite 0x037135 thru 0x03ffff (0x8ecb bytes) 000399 1781217051.800351 write 0x028225 thru 0x02ef3f (0x6d1b bytes) 000400 1781217051.808705 trunc from 0x021310 to 0x024e90 (0x3b81 bytes) 000401 1781217051.813934 write 0x01f3b2 thru 0x027cff (0x894e bytes) 000402 1781217051.821651 write 0x03eb9b thru 0x03ffff (0x1465 bytes) 000403 1781217051.822714 write 0x036be2 thru 0x03cf4f (0x636e bytes) 000404 1781217051.824535 read 0x025885 thru 0x034df9 (0xf575 bytes) 000405 1781217051.828634 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x46f396d2 (stale) Mirror 9: 0x46f396d2 (stale) Mirror 10: 0x46f396d2 (stale) Mirror 11: 0x46f396d2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xec76a5d8 Mirror 12: 0xec76a5d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000406 1781217052.503890 fallocate 0x00b13c thru 0x016b26 (0xb9eb bytes) 000407 1781217052.531774 write 0x0005d7 thru 0x004aa3 (0x44cd bytes) 000408 1781217052.541132 trunc from 0x02f922 to 0x03f683 (0xfd62 bytes) 000409 1781217052.549185 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x46f396d2 (stale) Mirror 10: 0x46f396d2 (stale) Mirror 11: 0x46f396d2 (stale) Mirror 12: 0xec76a5d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x46f396d2 (stale) Mirror 10: 0x46f396d2 (stale) Mirror 11: 0x46f396d2 (stale) Mirror 12: 0xec76a5d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000410 1781217052.789464 write 0x004458 thru 0x011bae (0xd757 bytes) 000411 1781217052.808560 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x46f396d2 (stale) Mirror 10: 0x46f396d2 (stale) Mirror 11: 0x46f396d2 (stale) Mirror 12: 0xec76a5d8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x470a6cdc Mirror 13: 0x470a6cdc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000412 1781217053.428825 read 0x0353e8 thru 0x03ffff (0xac18 bytes) 000414 1781217053.435599 mapwrite 0x00ed91 thru 0x01e43f (0xf6af bytes) 000415 1781217053.505862 mapread 0x02cf65 thru 0x03b107 (0xe1a3 bytes) 000416 1781217053.520794 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x46f396d2 (stale) Mirror 10: 0x46f396d2 (stale) Mirror 11: 0x46f396d2 (stale) Mirror 13: 0x470a6cdc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x46f396d2 (stale) Mirror 10: 0x46f396d2 (stale) Mirror 11: 0x46f396d2 (stale) Mirror 13: 0x470a6cdc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000417 1781217053.796964 mapread 0x00c7d9 thru 0x00ff77 (0x379f bytes) 000418 1781217053.812917 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x46f396d2 (stale) Mirror 10: 0x46f396d2 (stale) Mirror 13: 0x470a6cdc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x46f396d2 (stale) Mirror 10: 0x46f396d2 (stale) Mirror 13: 0x470a6cdc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000419 1781217054.019773 trunc from 0x03934f to 0x03ffff (0x6cb1 bytes) 000420 1781217054.027146 MIRROR_RESYNC 4 mirrors 000421 1781217054.529366 MIRROR_EXTEND to 5 mirrors 000422 1781217054.970809 punch from 0x03d7d9 to 0x03ffff (0x2827 bytes) 000423 1781217055.003553 punch from 0x000201 to 0x00eda4 (0xeba4 bytes) 000424 1781217055.008686 fallocate 0x01eec8 thru 0x02afa6 (0xc0df bytes) 000425 1781217055.017109 trunc from 0x040000 to 0x02f923 (0x106dc bytes) 000426 1781217055.056711 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f924): Mirror 10: 0x199fea19 (stale) Mirror 13: 0x199fea19 (stale) Mirror 14: 0x199fea19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f924): Mirror 10: 0x199fea19 (stale) Mirror 13: 0x199fea19 (stale) Mirror 14: 0x199fea19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f924: skipped 000427 1781217055.314607 trunc from 0x02f924 to 0x013fa0 (0x1b983 bytes) 000428 1781217055.338130 mapwrite 0x033859 thru 0x038110 (0x48b8 bytes) 000429 1781217055.399115 read 0x025b6e thru 0x02e511 (0x89a4 bytes) 000430 1781217055.408155 trunc from 0x021498 to 0x022331 (0xe9a bytes) 000431 1781217055.413933 mapread 0x0051dc thru 0x008c9b (0x3ac0 bytes) 000433 1781217055.426141 mapwrite 0x0133e5 thru 0x01df91 (0xabad bytes) 000434 1781217055.474052 write 0x03bc5b thru 0x03ca3e (0xde4 bytes) 000435 1781217055.475307 trunc from 0x03ca3f to 0x00647f (0x365bf bytes) 000436 1781217055.488692 trunc from 0x006480 to 0x00008e (0x63f1 bytes) 000437 1781217055.506001 read 0x00003d thru 0x00008e (0x52 bytes) 000438 1781217055.510539 read 0x000074 thru 0x00008e (0x1b bytes) 000439 1781217055.511299 mapread 0x00000e thru 0x00008e (0x81 bytes) 000440 1781217055.512169 mapread 0x00004c thru 0x00008e (0x43 bytes) 000441 1781217055.513074 trunc from 0x00008f to 0x015947 (0x158b9 bytes) 000442 1781217055.527908 read 0x004618 thru 0x006d1b (0x2704 bytes) 000443 1781217055.549147 fallocate 0x027960 thru 0x033aaf (0xc150 bytes) 000444 1781217055.554869 read 0x00f0c7 thru 0x014d5a (0x5c94 bytes) 000445 1781217055.555836 punch from 0x00e224 to 0x00f306 (0x10e3 bytes) 000446 1781217055.560880 mapwrite 0x02889b thru 0x031d8c (0x94f2 bytes) 000447 1781217055.596091 mapwrite 0x030bde thru 0x0333db (0x27fe bytes) 000449 1781217055.631265 trunc from 0x01e76e to 0x0296a6 (0xaf39 bytes) 000450 1781217055.636567 trunc from 0x0333dc to 0x02d223 (0x61b8 bytes) 000451 1781217055.651015 read 0x01f0c6 thru 0x02aec6 (0xbe01 bytes) 000452 1781217055.661957 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d224): Mirror 13: 0x4bd4534 (stale) Mirror 14: 0x4bd4534 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d224): Mirror 13: 0x4bd4534 (stale) Mirror 14: 0x4bd4534 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d224: skipped 000453 1781217055.825218 mapwrite 0x02c5eb thru 0x0328b3 (0x62c9 bytes) 000454 1781217055.863011 fallocate 0x02cf9b thru 0x03873a (0xb7a0 bytes) 000455 1781217055.866874 read 0x033e4e thru 0x0359ae (0x1b61 bytes) 000456 1781217055.872481 mapread 0x02e6a7 thru 0x03873a (0xa094 bytes) 000457 1781217055.877509 mapwrite 0x00a73c thru 0x0135dc (0x8ea1 bytes) 000458 1781217055.903588 write 0x03db0d thru 0x03ffff (0x24f3 bytes) 000459 1781217055.904930 fallocate 0x000040 thru 0x006ae6 (0x6aa7 bytes) 000460 1781217055.910575 trunc from 0x040000 to 0x006e4a (0x391b5 bytes) 000461 1781217055.922028 trunc from 0x00694e to 0x006e4a (0x4fd bytes) 000462 1781217055.932353 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6e4b): Mirror 14: 0x3e5e02f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6e4b): Mirror 14: 0x3e5e02f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6e4b: skipped 000463 1781217056.111033 fallocate 0x0360ec thru 0x03b42b (0x5340 bytes) 000464 1781217056.114905 mapwrite 0x025412 thru 0x029240 (0x3e2f bytes) 000465 1781217056.161087 punch from 0x020f61 to 0x029240 (0x82e0 bytes) 000466 1781217056.165483 mapwrite 0x0269cc thru 0x02e132 (0x7767 bytes) 000468 1781217056.201561 trunc from 0x00eaed to 0x01778a (0x8c9e bytes) 000469 1781217056.206629 MIRROR_RESYNC 2 mirrors 000470 1781217056.536703 fallocate 0x01efc3 thru 0x02d51b (0xe559 bytes) 000473 1781217056.568079 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e133): Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e133): Mirror 14: 0x3f26eff1 (stale) CRC-32 checksum value for chunk [0, 0x2e133): Mirror 1: 0x3f26eff1 Mirror 15: 0x3f26eff1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e133: skipped 000474 1781217057.051179 read 0x00c302 thru 0x010a34 (0x4733 bytes) 000475 1781217057.055670 mapread 0x00908a thru 0x00dfdb (0x4f52 bytes) 000476 1781217057.065893 trunc from 0x02e133 to 0x03a32f (0xc1fd bytes) 000477 1781217057.109977 trunc from 0x03a330 to 0x01cb70 (0x1d7bf bytes) 000478 1781217057.134677 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1cb71): Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cb71): Mirror 14: 0x12086a3d (stale) Mirror 15: 0x12086a3d (stale) CRC-32 checksum value for chunk [0, 0x1cb71): Mirror 1: 0x12086a3d Mirror 16: 0x12086a3d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cb71: skipped 000479 1781217057.601392 punch from 0x006e2d to 0x008ffc (0x21d0 bytes) 000480 1781217057.622461 fallocate 0x0340a0 thru 0x0376f4 (0x3655 bytes) 000481 1781217057.625393 write 0x00c778 thru 0x01bb93 (0xf41c bytes) 000482 1781217057.636107 fallocate 0x03bcb8 thru 0x03ffff (0x4348 bytes) 000483 1781217057.640807 write 0x016c2c thru 0x01a6b8 (0x3a8d bytes) 000484 1781217057.646458 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x376f5): Mirror 14: 0xb67a1a45 (stale) Mirror 15: 0xb67a1a45 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376f5): Mirror 14: 0xb67a1a45 (stale) Mirror 15: 0xb67a1a45 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376f5: skipped 000485 1781217057.822308 mapwrite 0x026348 thru 0x029e35 (0x3aee bytes) 000486 1781217057.843230 read 0x023cb3 thru 0x033444 (0xf792 bytes) 000487 1781217057.847454 trunc from 0x0226e1 to 0x0321dd (0xfafd bytes) 000488 1781217057.850793 mapread 0x017cda thru 0x021432 (0x9759 bytes) 000489 1781217057.860933 read 0x008fc8 thru 0x010c3b (0x7c74 bytes) 000490 1781217057.867740 mapwrite 0x013c41 thru 0x01a5e2 (0x69a2 bytes) 000491 1781217057.889025 MIRROR_RESYNC 3 mirrors 000492 1781217058.361051 mapread 0x01fdce thru 0x02e91b (0xeb4e bytes) 000493 1781217058.382742 trunc from 0x01bf53 to 0x01e1ec (0x229a bytes) 000494 1781217058.405801 trunc from 0x0019eb to 0x0109cc (0xefe2 bytes) 000495 1781217058.410047 mapwrite 0x008786 thru 0x01701f (0xe89a bytes) 000498 1781217058.444464 punch from 0x01820b to 0x020592 (0x8388 bytes) 000499 1781217058.449106 trunc from 0x007dc3 to 0x011ff5 (0xa233 bytes) 000500 1781217058.452801 fallocate 0x017bf9 thru 0x018e2c (0x1234 bytes) 000501 1781217058.456855 trunc from 0x02e573 to 0x0376f4 (0x9182 bytes) 000502 1781217058.460650 punch from 0x006379 to 0x00ee19 (0x8aa1 bytes) 000503 1781217058.466227 mapread 0x016a21 thru 0x01ac51 (0x4231 bytes) 000504 1781217058.472515 fallocate 0x000b68 thru 0x00f3f7 (0xe890 bytes) 000505 1781217058.477750 mapwrite 0x000706 thru 0x001a1b (0x1316 bytes) 000506 1781217058.493462 trunc from 0x011fe9 to 0x01210b (0x123 bytes) 000507 1781217058.497035 trunc from 0x0204e6 to 0x02e08a (0xdba5 bytes) 000508 1781217058.503712 write 0x03d34b thru 0x03ffff (0x2cb5 bytes) 000509 1781217058.504979 read 0x01a204 thru 0x02088b (0x6688 bytes) 000510 1781217058.510042 fallocate 0x02117b thru 0x024bb2 (0x3a38 bytes) 000511 1781217058.515577 read 0x01786d thru 0x022bce (0xb362 bytes) 000512 1781217058.532941 trunc from 0x011f4e to 0x01df2b (0xbfde bytes) 000513 1781217058.540287 write 0x02099f thru 0x0280a1 (0x7703 bytes) 000515 1781217058.542233 trunc from 0x040000 to 0x024197 (0x1be68 bytes) 000517 1781217058.557870 read 0x014d44 thru 0x01d3cb (0x8688 bytes) 000518 1781217058.581979 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24198): Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24198): Mirror 14: 0x155655ed (stale) Mirror 15: 0x155655ed (stale) CRC-32 checksum value for chunk [0, 0x24198): Mirror 1: 0xcffc7c43 Mirror 16: 0xcffc7c43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24198: skipped 000519 1781217059.050655 write 0x03d582 thru 0x03ffff (0x2a7e bytes) 000520 1781217059.090683 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xe5afd31e (stale) Mirror 16: 0x788315f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xe5afd31e (stale) Mirror 16: 0x788315f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000521 1781217059.282540 write 0x02ddb9 thru 0x03c8dc (0xeb24 bytes) 000522 1781217059.295366 write 0x02165c thru 0x02f129 (0xdace bytes) 000523 1781217059.301553 read 0x008629 thru 0x00f497 (0x6e6f bytes) 000524 1781217059.307363 mapread 0x0141e9 thru 0x01b900 (0x7718 bytes) 000525 1781217059.312829 punch from 0x01fa4e to 0x020a1b (0xfce bytes) 000528 1781217059.316740 read 0x00ae9a thru 0x00d5e6 (0x274d bytes) 000529 1781217059.317515 punch from 0x0072b6 to 0x00fd1b (0x8a66 bytes) 000530 1781217059.320782 trunc from 0x040000 to 0x016504 (0x29afb bytes) 000531 1781217059.349238 mapwrite 0x00959f thru 0x00bad7 (0x2539 bytes) 000532 1781217059.367104 mapwrite 0x00735d thru 0x009546 (0x21ea bytes) 000533 1781217059.388303 trunc from 0x0104e4 to 0x016504 (0x6021 bytes) 000534 1781217059.399241 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x16505): Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16505): Mirror 15: 0x76095db1 (stale) Mirror 16: 0xf3246f2 (stale) CRC-32 checksum value for chunk [0, 0x16505): Mirror 1: 0x5e99d0f7 Mirror 17: 0x5e99d0f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16505: skipped 000535 1781217059.868560 punch from 0x006735 to 0x006d7b (0x647 bytes) 000536 1781217059.895140 fallocate 0x034582 thru 0x03ffff (0xba7e bytes) 000537 1781217059.899208 punch from 0x006ba5 to 0x013a25 (0xce81 bytes) 000538 1781217059.914161 MIRROR_RESYNC 4 mirrors 000539 1781217060.395641 fallocate 0x03b4c8 thru 0x03ffff (0x4b38 bytes) 000540 1781217060.427117 punch from 0x019d35 to 0x0200cb (0x6397 bytes) 000541 1781217060.430496 punch from 0x010136 to 0x010310 (0x1db bytes) 000542 1781217060.437474 trunc from 0x024223 to 0x02b5ce (0x73ac bytes) 000543 1781217060.446210 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xc6c136a6 (stale) Mirror 16: 0xc6c136a6 (stale) Mirror 17: 0xc6c136a6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc6c136a6 Mirror 18: 0xc6c136a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000544 1781217060.953015 trunc from 0x040000 to 0x0156fd (0x2a902 bytes) 000545 1781217060.989157 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x156fe): Mirror 16: 0x293f3cbe (stale) Mirror 17: 0x293f3cbe (stale) Mirror 18: 0x293f3cbe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x156fe): Mirror 16: 0x293f3cbe (stale) Mirror 17: 0x293f3cbe (stale) Mirror 18: 0x293f3cbe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x156fe: skipped 000546 1781217061.205933 write 0x02ffa4 thru 0x03f233 (0xf290 bytes) 000548 1781217061.212729 trunc from 0x012c0c to 0x01b815 (0x8c0a bytes) 000549 1781217061.221936 mapread 0x014d53 thru 0x01eb87 (0x9e35 bytes) 000550 1781217061.237446 mapread 0x00cb8d thru 0x019d1e (0xd192 bytes) 000551 1781217061.245559 trunc from 0x03f234 to 0x03bfcd (0x3266 bytes) 000552 1781217061.257520 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bfce): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfce): Mirror 16: 0xd9b53202 (stale) Mirror 17: 0xd9b53202 (stale) Mirror 18: 0xd9b53202 (stale) CRC-32 checksum value for chunk [0, 0x3bfce): Mirror 1: 0xef365c12 Mirror 19: 0xef365c12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfce: skipped 000553 1781217061.766066 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bfce): Mirror 16: 0xd9b53202 (stale) Mirror 17: 0xd9b53202 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfce): Mirror 16: 0xd9b53202 (stale) Mirror 17: 0xd9b53202 (stale) CRC-32 checksum value for chunk [0, 0x3bfce): Mirror 1: 0xef365c12 Mirror 19: 0xef365c12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfce: skipped 000554 1781217062.071345 trunc from 0x03bfce to 0x02a135 (0x11e98 bytes) 000555 1781217062.100397 read 0x00821b thru 0x0163e4 (0xe1ca bytes) 000556 1781217062.106207 trunc from 0x028e69 to 0x02a135 (0x12cd bytes) 000557 1781217062.109583 trunc from 0x01cea2 to 0x02a135 (0xd294 bytes) 000558 1781217062.114719 punch from 0x008ae1 to 0x0174d5 (0xe9f5 bytes) 000559 1781217062.118431 mapwrite 0x02ffb4 thru 0x0357c1 (0x580e bytes) 000560 1781217062.161428 mapread 0x005ecf thru 0x012ba1 (0xccd3 bytes) 000561 1781217062.180734 read 0x003336 thru 0x00b481 (0x814c bytes) 000562 1781217062.181497 punch from 0x015129 to 0x02482b (0xf703 bytes) 000563 1781217062.185352 read 0x010b21 thru 0x01176d (0xc4d bytes) 000564 1781217062.189749 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x357c2): Mirror 16: 0x95e956f8 (stale) Mirror 17: 0x95e956f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x357c2): Mirror 16: 0x95e956f8 (stale) Mirror 17: 0x95e956f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x357c2: skipped 000565 1781217062.400725 write 0x00baea thru 0x00e213 (0x272a bytes) 000566 1781217062.412353 punch from 0x02113b to 0x027fe3 (0x6ea9 bytes) 000567 1781217062.417051 punch from 0x015a40 to 0x01694e (0xf0f bytes) 000568 1781217062.425475 MIRROR_RESYNC 3 mirrors 000570 1781217062.877804 mapread 0x01f579 thru 0x023f0b (0x4993 bytes) 000571 1781217062.910485 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000572 1781217063.097962 mapread 0x010900 thru 0x018b78 (0x8279 bytes) 000573 1781217063.107541 read 0x009163 thru 0x0117f5 (0x8693 bytes) 000574 1781217063.111314 mapwrite 0x0373e5 thru 0x03c3f1 (0x500d bytes) 000575 1781217063.162046 write 0x0113d7 thru 0x0114c6 (0xf0 bytes) 000576 1781217063.171534 mapwrite 0x030469 thru 0x03fe40 (0xf9d8 bytes) 000577 1781217063.205734 mapread 0x010c58 thru 0x012d1a (0x20c3 bytes) 000578 1781217063.207624 fallocate 0x02b667 thru 0x03132c (0x5cc6 bytes) 000579 1781217063.217834 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fe41): Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe41): Mirror 17: 0x476c48c4 (stale) CRC-32 checksum value for chunk [0, 0x3fe41): Mirror 1: 0xe1b73d61 Mirror 18: 0xe1b73d61 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe41: skipped 000580 1781217063.760853 write 0x0263b2 thru 0x028334 (0x1f83 bytes) 000581 1781217063.812890 read 0x01ddd5 thru 0x028d84 (0xafb0 bytes) 000582 1781217063.821273 trunc from 0x03fe41 to 0x027319 (0x18b27 bytes) 000583 1781217063.858969 MIRROR_RESYNC 3 mirrors 000584 1781217064.448134 mapread 0x010664 thru 0x0156ee (0x508b bytes) 000585 1781217064.473365 trunc from 0x01eca2 to 0x022e5e (0x41bd bytes) 000586 1781217064.514655 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2731a): Mirror 18: 0xce5281c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2731a): Mirror 18: 0xce5281c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2731a: skipped 000587 1781217064.754364 trunc from 0x02731a to 0x017739 (0xfbe0 bytes) 000588 1781217064.776194 write 0x018123 thru 0x01bc27 (0x3b05 bytes) 000589 1781217064.778686 write 0x033978 thru 0x03db61 (0xa1ea bytes) 000590 1781217064.780253 trunc from 0x016fe3 to 0x01c35c (0x537a bytes) 000591 1781217064.792850 read 0x0367ed thru 0x03ce21 (0x6635 bytes) 000592 1781217064.797605 MIRROR_RESYNC 2 mirrors 000593 1781217065.345537 trunc from 0x009469 to 0x013187 (0x9d1f bytes) 000594 1781217065.373536 mapwrite 0x038487 thru 0x03ffff (0x7b79 bytes) 000595 1781217065.429447 read 0x0307d5 thru 0x034787 (0x3fb3 bytes) 000596 1781217065.439208 mapwrite 0x01a991 thru 0x028c86 (0xe2f6 bytes) 000597 1781217065.493662 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x76822e59 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x53ec0dcc Mirror 19: 0x53ec0dcc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000598 1781217066.178634 MIRROR_RESYNC 3 mirrors 000599 1781217066.751520 punch from 0x035b4e to 0x03ffff (0xa4b2 bytes) 000600 1781217066.789714 MIRROR_RESYNC 3 mirrors 000601 1781217067.536196 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000602 1781217067.789660 trunc from 0x02afc3 to 0x02e347 (0x3385 bytes) 000603 1781217067.824961 fallocate 0x01f76f thru 0x02690e (0x71a0 bytes) 000604 1781217067.830497 trunc from 0x004706 to 0x00d8cb (0x91c6 bytes) 000605 1781217067.835905 trunc from 0x02f242 to 0x030fae (0x1d6d bytes) 000606 1781217067.839424 mapwrite 0x01e281 thru 0x024d21 (0x6aa1 bytes) 000607 1781217067.876725 mapread 0x015689 thru 0x01a232 (0x4baa bytes) 000608 1781217067.890684 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000610 1781217067.989972 fallocate 0x03f81f thru 0x03ffff (0x7e1 bytes) 000611 1781217067.996546 write 0x01beba thru 0x025c67 (0x9dae bytes) 000612 1781217068.008961 fallocate 0x013269 thru 0x0210a7 (0xde3f bytes) 000613 1781217068.013435 mapread 0x032dca thru 0x03ffff (0xd236 bytes) 000614 1781217068.026355 write 0x0327d4 thru 0x0338b6 (0x10e3 bytes) 000615 1781217068.027465 punch from 0x019fba to 0x01c878 (0x28bf bytes) 000617 1781217068.043868 mapread 0x03799a thru 0x03ffff (0x8666 bytes) 000618 1781217068.045167 write 0x003257 thru 0x011c4c (0xe9f6 bytes) 000619 1781217068.056562 write 0x022437 thru 0x025c50 (0x381a bytes) 000620 1781217068.057841 trunc from 0x040000 to 0x038a12 (0x75ed bytes) 000621 1781217068.071408 mapwrite 0x02b690 thru 0x02f73e (0x40af bytes) 000622 1781217068.099438 write 0x000e8a thru 0x0073ef (0x6566 bytes) 000623 1781217068.106681 mapwrite 0x000510 thru 0x005f2c (0x5a1d bytes) 000624 1781217068.124468 punch from 0x037ae0 to 0x038a12 (0xf33 bytes) 000625 1781217068.128202 read 0x012302 thru 0x0138bf (0x15be bytes) 000626 1781217068.134930 write 0x019d47 thru 0x024544 (0xa7fe bytes) 000628 1781217068.144410 trunc from 0x038a13 to 0x009272 (0x2f7a0 bytes) 000629 1781217068.158405 trunc from 0x009273 to 0x03cea8 (0x33c36 bytes) 000630 1781217068.172157 mapwrite 0x01ca19 thru 0x026a8a (0xa072 bytes) 000631 1781217068.208456 fallocate 0x036bcb thru 0x038607 (0x1a3d bytes) 000633 1781217068.217937 punch from 0x00113c to 0x0090f7 (0x7fbc bytes) 000635 1781217068.224622 trunc from 0x035405 to 0x03cea8 (0x7aa4 bytes) 000636 1781217068.233502 MIRROR_EXTEND to 2 mirrors 000637 1781217068.679309 trunc from 0x03cea9 to 0x025786 (0x17722 bytes) 000638 1781217068.730414 MIRROR_RESYNC 2 mirrors 000639 1781217069.183634 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000640 1781217069.283925 fallocate 0x01a13b thru 0x024308 (0xa1ce bytes) 000641 1781217069.299644 write 0x03d7fe thru 0x03ffff (0x2802 bytes) 000642 1781217069.306604 mapwrite 0x03735d thru 0x03ca94 (0x5738 bytes) 000643 1781217069.336929 punch from 0x02e52f to 0x039491 (0xaf63 bytes) 000645 1781217069.342346 mapwrite 0x03cf97 thru 0x03ffff (0x3069 bytes) 000646 1781217069.361712 mapwrite 0x0262a1 thru 0x02da84 (0x77e4 bytes) 000647 1781217069.387172 write 0x013240 thru 0x01c579 (0x933a bytes) 000648 1781217069.399965 write 0x03c9b2 thru 0x03fdb1 (0x3400 bytes) 000649 1781217069.401582 punch from 0x028ee3 to 0x02d763 (0x4881 bytes) 000650 1781217069.406584 trunc from 0x032f81 to 0x037e86 (0x4f06 bytes) 000652 1781217069.412339 read 0x030ce7 thru 0x03ffff (0xf319 bytes) 000653 1781217069.423110 fallocate 0x0032d2 thru 0x008f97 (0x5cc6 bytes) 000654 1781217069.429318 write 0x017666 thru 0x01f726 (0x80c1 bytes) 000655 1781217069.431368 trunc from 0x0282c1 to 0x02ce31 (0x4b71 bytes) 000656 1781217069.440713 MIRROR_EXTEND to 2 mirrors 000657 1781217070.091762 mapwrite 0x0008ce thru 0x00cf4f (0xc682 bytes) 000658 1781217070.206005 write 0x03b0de thru 0x03ff7a (0x4e9d bytes) 000660 1781217070.232837 mapread 0x00007d thru 0x00100e (0xf92 bytes) 000661 1781217070.234274 mapwrite 0x03fbde thru 0x03ffff (0x422 bytes) 000662 1781217070.254831 read 0x003876 thru 0x00afba (0x7745 bytes) 000664 1781217070.256546 trunc from 0x01831c to 0x019da4 (0x1a89 bytes) 000665 1781217070.271583 trunc from 0x040000 to 0x008581 (0x37a7e bytes) 000666 1781217070.323896 write 0x031b5a thru 0x03c8f2 (0xad99 bytes) 000669 1781217070.328665 write 0x015221 thru 0x01a136 (0x4f16 bytes) 000670 1781217070.336823 mapread 0x01315d thru 0x01af93 (0x7e37 bytes) 000671 1781217070.346038 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000672 1781217070.512767 trunc from 0x03c8f3 to 0x03e059 (0x1767 bytes) 000673 1781217070.545648 trunc from 0x03e05a to 0x00187b (0x3c7de bytes) 000674 1781217070.566571 MIRROR_EXTEND to 2 mirrors 000676 1781217071.157051 punch from 0x0009b6 to 0x00187b (0xec6 bytes) 000677 1781217071.188936 write 0x003efb thru 0x01223f (0xe345 bytes) 000678 1781217071.195806 read 0x003305 thru 0x0112ca (0xdfc6 bytes) 000679 1781217071.196958 read 0x010a64 thru 0x01223f (0x17dc bytes) 000680 1781217071.198072 mapwrite 0x0212a1 thru 0x024cad (0x3a0d bytes) 000681 1781217071.243405 trunc from 0x022e3c to 0x024cad (0x1e72 bytes) 000682 1781217071.248347 trunc from 0x024cae to 0x027009 (0x235c bytes) 000683 1781217071.266250 read 0x014e09 thru 0x021802 (0xc9fa bytes) 000684 1781217071.267213 mapwrite 0x006911 thru 0x0099d4 (0x30c4 bytes) 000685 1781217071.288949 mapread 0x01554a thru 0x016b5e (0x1615 bytes) 000686 1781217071.290229 trunc from 0x009989 to 0x0195bd (0xfc35 bytes) 000687 1781217071.303266 MIRROR_RESYNC 2 mirrors 000688 1781217071.991947 punch from 0x01c799 to 0x01f05b (0x28c3 bytes) 000689 1781217072.032083 punch from 0x01a408 to 0x027009 (0xcc02 bytes) 000690 1781217072.045305 read 0x0104d3 thru 0x01d0be (0xcbec bytes) 000691 1781217072.054794 trunc from 0x0153b2 to 0x020142 (0xad91 bytes) 000692 1781217072.060913 punch from 0x01e872 to 0x020612 (0x1da1 bytes) 000693 1781217072.064325 mapread 0x008dde thru 0x013b34 (0xad57 bytes) 000694 1781217072.075863 read 0x0190e2 thru 0x0233f2 (0xa311 bytes) 000695 1781217072.083158 write 0x00b60d thru 0x00d3b2 (0x1da6 bytes) 000696 1781217072.084247 punch from 0x00ca4d to 0x019787 (0xcd3b bytes) 000697 1781217072.096364 trunc from 0x00b1ea to 0x0195fa (0xe411 bytes) 000698 1781217072.103011 read 0x01b612 thru 0x02376d (0x815c bytes) 000699 1781217072.105432 fallocate 0x01a22e thru 0x01b9c7 (0x179a bytes) 000700 1781217072.110761 mapwrite 0x009011 thru 0x011fc2 (0x8fb2 bytes) 000701 1781217072.145601 trunc from 0x023aa8 to 0x0260b6 (0x260f bytes) 000702 1781217072.149617 trunc from 0x02700a to 0x0292cf (0x22c6 bytes) 000703 1781217072.186586 fallocate 0x014e0c thru 0x01957d (0x4772 bytes) 000704 1781217072.190856 mapwrite 0x00a5e0 thru 0x0164f0 (0xbf11 bytes) 000705 1781217072.215761 trunc from 0x0292d0 to 0x01df14 (0xb3bb bytes) 000706 1781217072.226982 mapread 0x015012 thru 0x01cda5 (0x7d94 bytes) 000707 1781217072.228184 mapread 0x0019a4 thru 0x007fda (0x6637 bytes) 000708 1781217072.229200 punch from 0x012ddf to 0x01df14 (0xb136 bytes) 000709 1781217072.234066 fallocate 0x01991f thru 0x01a92f (0x1011 bytes) 000710 1781217072.239199 trunc from 0x01df15 to 0x034ec0 (0x16fac bytes) 000712 1781217072.254232 mapwrite 0x03c85d thru 0x03ffff (0x37a3 bytes) 000713 1781217072.286939 read 0x008a85 thru 0x00a0dd (0x1659 bytes) 000714 1781217072.288350 trunc from 0x040000 to 0x02c5e9 (0x13a16 bytes) 000715 1781217072.299681 trunc from 0x02c5ea to 0x01589c (0x16d4d bytes) 000716 1781217072.309162 mapwrite 0x0060c9 thru 0x010429 (0xa361 bytes) 000717 1781217072.326983 mapwrite 0x024101 thru 0x02c57d (0x847d bytes) 000718 1781217072.365437 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000719 1781217072.474602 trunc from 0x01ba25 to 0x01ff29 (0x4505 bytes) 000720 1781217072.478298 trunc from 0x029989 to 0x02c57d (0x2bf5 bytes) 000722 1781217072.482958 mapwrite 0x003410 thru 0x012b70 (0xf761 bytes) 000723 1781217072.531103 mapread 0x0258b6 thru 0x027025 (0x1770 bytes) 000724 1781217072.541236 punch from 0x01faa6 to 0x02c57d (0xcad8 bytes) 000725 1781217072.551533 trunc from 0x00a968 to 0x00f2ba (0x4953 bytes) 000727 1781217072.561846 mapread 0x01e9a9 thru 0x02975b (0xadb3 bytes) 000728 1781217072.579879 read 0x01414d thru 0x019489 (0x533d bytes) 000729 1781217072.580735 read 0x01a68a thru 0x02316e (0x8ae5 bytes) 000730 1781217072.581666 read 0x026be7 thru 0x027e34 (0x124e bytes) 000732 1781217072.583764 trunc from 0x02c57e to 0x00440a (0x28173 bytes) 000734 1781217072.611206 trunc from 0x00440b to 0x033bcb (0x2f7c1 bytes) 000736 1781217072.632594 fallocate 0x01c957 thru 0x02495f (0x8009 bytes) 000737 1781217072.636941 fallocate 0x02d576 thru 0x02d6d1 (0x15c bytes) 000738 1781217072.641552 trunc from 0x033bcc to 0x0141ab (0x1fa20 bytes) 000739 1781217072.652729 mapwrite 0x00269a thru 0x005439 (0x2da0 bytes) 000740 1781217072.669863 fallocate 0x023c46 thru 0x028635 (0x49f0 bytes) 000741 1781217072.676322 mapwrite 0x001cfd thru 0x00515a (0x345e bytes) 000742 1781217072.691142 trunc from 0x00c423 to 0x0141ab (0x7d89 bytes) 000743 1781217072.695656 fallocate 0x01c83f thru 0x01cc35 (0x3f7 bytes) 000745 1781217072.701403 write 0x025837 thru 0x031dcf (0xc599 bytes) 000746 1781217072.708584 MIRROR_EXTEND to 2 mirrors 000747 1781217073.267874 fallocate 0x02c6a6 thru 0x0360d7 (0x9a32 bytes) 000748 1781217073.296593 trunc from 0x031dd0 to 0x00492a (0x2d4a5 bytes) 000749 1781217073.320630 mapread 0x000ac3 thru 0x00492a (0x3e68 bytes) 000750 1781217073.345202 punch from 0x003e16 to 0x00492a (0xb15 bytes) 000751 1781217073.349610 write 0x02d8dc thru 0x0309a7 (0x30cc bytes) 000752 1781217073.354963 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000753 1781217073.485259 fallocate 0x0133b6 thru 0x01cca3 (0x98ee bytes) 000754 1781217073.490222 mapwrite 0x0326e1 thru 0x035d2a (0x364a bytes) 000755 1781217073.526444 trunc from 0x035d2b to 0x02e54c (0x77de bytes) 000756 1781217073.539196 fallocate 0x017d54 thru 0x020cf9 (0x8fa6 bytes) 000757 1781217073.542338 fallocate 0x00fd47 thru 0x012319 (0x25d3 bytes) 000758 1781217073.545774 mapread 0x0047ff thru 0x005754 (0xf56 bytes) 000759 1781217073.564516 MIRROR_EXTEND to 2 mirrors 000760 1781217073.976348 write 0x0366d6 thru 0x037d03 (0x162e bytes) 000762 1781217074.015728 mapwrite 0x00ab4f thru 0x01a2d1 (0xf783 bytes) 000763 1781217074.068017 punch from 0x00eb2a to 0x01a12b (0xb602 bytes) 000764 1781217074.074311 punch from 0x007f29 to 0x0101f4 (0x82cc bytes) 000765 1781217074.082264 trunc from 0x02243d to 0x028d77 (0x693b bytes) 000766 1781217074.087236 read 0x011e62 thru 0x01ed02 (0xcea1 bytes) 000768 1781217074.102187 MIRROR_RESYNC 2 mirrors 000769 1781217074.586882 mapread 0x00581a thru 0x010bea (0xb3d1 bytes) 000770 1781217074.609367 mapread 0x03131e thru 0x037d03 (0x69e6 bytes) 000771 1781217074.617949 trunc from 0x01175d to 0x0184d9 (0x6d7d bytes) 000772 1781217074.651948 mapwrite 0x018b73 thru 0x01f49f (0x692d bytes) 000773 1781217074.683793 fallocate 0x0255be thru 0x027a28 (0x246b bytes) 000774 1781217074.688262 read 0x01a1b0 thru 0x0231b5 (0x9006 bytes) 000775 1781217074.692742 punch from 0x00ec4d to 0x013828 (0x4bdc bytes) 000776 1781217074.697300 trunc from 0x01ee05 to 0x0289a4 (0x9ba0 bytes) 000777 1781217074.701142 punch from 0x02a785 to 0x0352a0 (0xab1c bytes) 000778 1781217074.709266 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000780 1781217074.852928 read 0x033991 thru 0x037d03 (0x4373 bytes) 000782 1781217074.869449 mapwrite 0x0394bb thru 0x03d519 (0x405f bytes) 000783 1781217074.936151 mapwrite 0x030b76 thru 0x03cb40 (0xbfcb bytes) 000784 1781217074.960255 mapread 0x00b767 thru 0x012ba7 (0x7441 bytes) 000785 1781217074.968669 punch from 0x01b61e to 0x024238 (0x8c1b bytes) 000787 1781217074.974330 mapread 0x02a3dd thru 0x02c27e (0x1ea2 bytes) 000788 1781217074.975328 read 0x0263de thru 0x02f547 (0x916a bytes) 000789 1781217074.985017 punch from 0x008321 to 0x00e50b (0x61eb bytes) 000790 1781217074.993358 MIRROR_EXTEND to 2 mirrors 000791 1781217075.460773 trunc from 0x03d51a to 0x0080b5 (0x35464 bytes) 000792 1781217075.503017 trunc from 0x0080b6 to 0x00f69f (0x75ea bytes) 000793 1781217075.522241 trunc from 0x00f6a0 to 0x016d7c (0x76dd bytes) 000794 1781217075.541279 trunc from 0x016d7d to 0x037400 (0x20684 bytes) 000795 1781217075.556437 fallocate 0x00f7b7 thru 0x0135e8 (0x3e32 bytes) 000796 1781217075.561830 write 0x03ffe0 thru 0x03ffff (0x20 bytes) 000797 1781217075.566619 read 0x01b4bc thru 0x023f68 (0x8aad bytes) 000799 1781217075.575960 read 0x00c0a3 thru 0x015f19 (0x9e77 bytes) 000800 1781217075.582945 read 0x020119 thru 0x02a92e (0xa816 bytes) 000801 1781217075.583871 write 0x0292fb thru 0x034089 (0xad8f bytes) 000802 1781217075.585121 trunc from 0x040000 to 0x010ed1 (0x2f12e bytes) 000803 1781217075.602453 write 0x03c9be thru 0x03ffff (0x3642 bytes) 000804 1781217075.606090 write 0x00e130 thru 0x00f7a2 (0x1673 bytes) 000805 1781217075.607269 punch from 0x002aa1 to 0x00f633 (0xcb93 bytes) 000806 1781217075.623184 mapwrite 0x037eb0 thru 0x03d6f3 (0x5844 bytes) 000807 1781217075.650197 punch from 0x01a1b0 to 0x0272bf (0xd110 bytes) 000809 1781217075.655610 punch from 0x01084b to 0x01a256 (0x9a0c bytes) 000810 1781217075.660234 trunc from 0x03dea8 to 0x03ffff (0x2158 bytes) 000811 1781217075.665069 fallocate 0x00a888 thru 0x01053d (0x5cb6 bytes) 000812 1781217075.670124 mapread 0x022ed1 thru 0x02caae (0x9bde bytes) 000813 1781217075.702855 trunc from 0x037491 to 0x039486 (0x1ff6 bytes) 000814 1781217075.708637 punch from 0x00db05 to 0x01973b (0xbc37 bytes) 000815 1781217075.712796 trunc from 0x040000 to 0x036836 (0x97c9 bytes) 000816 1781217075.727859 read 0x01e9af thru 0x02b9d0 (0xd022 bytes) 000817 1781217075.737048 read 0x033574 thru 0x036836 (0x32c3 bytes) 000818 1781217075.738329 mapwrite 0x01cb93 thru 0x01f437 (0x28a5 bytes) 000819 1781217075.756318 mapwrite 0x02a843 thru 0x02d9d6 (0x3194 bytes) 000820 1781217075.784262 mapwrite 0x01b824 thru 0x029d7f (0xe55c bytes) 000821 1781217075.810026 trunc from 0x02526f to 0x02ad5d (0x5aef bytes) 000822 1781217075.814780 trunc from 0x028875 to 0x031717 (0x8ea3 bytes) 000823 1781217075.819448 read 0x01cc07 thru 0x01e9e1 (0x1ddb bytes) 000824 1781217075.820168 mapread 0x00cad1 thru 0x01aac8 (0xdff8 bytes) 000825 1781217075.822126 fallocate 0x006437 thru 0x010972 (0xa53c bytes) 000826 1781217075.826496 trunc from 0x030c68 to 0x0345ba (0x3953 bytes) 000827 1781217075.830635 trunc from 0x019e2d to 0x024eba (0xb08e bytes) 000828 1781217075.837274 mapread 0x011dce thru 0x013e54 (0x2087 bytes) 000829 1781217075.838723 write 0x02e846 thru 0x03415c (0x5917 bytes) 000830 1781217075.851670 MIRROR_RESYNC 2 mirrors 000831 1781217076.348331 MIRROR_EXTEND to 3 mirrors 000832 1781217076.829305 write 0x003367 thru 0x00e7c6 (0xb460 bytes) 000834 1781217076.870903 fallocate 0x0070dd thru 0x009f8b (0x2eaf bytes) 000835 1781217076.874284 trunc from 0x024f02 to 0x02aafd (0x5bfc bytes) 000836 1781217076.878891 mapread 0x0346c0 thru 0x034e23 (0x764 bytes) 000837 1781217076.884923 read 0x007622 thru 0x00ae49 (0x3828 bytes) 000838 1781217076.885649 write 0x018815 thru 0x023e94 (0xb680 bytes) 000839 1781217076.894210 read 0x00640b thru 0x009f69 (0x3b5f bytes) 000840 1781217076.894931 read 0x031c0d thru 0x036836 (0x4c2a bytes) 000841 1781217076.896086 punch from 0x01706e to 0x01c47e (0x5411 bytes) 000842 1781217076.906028 mapwrite 0x009c67 thru 0x009de9 (0x183 bytes) 000844 1781217076.915840 trunc from 0x036837 to 0x030549 (0x62ed bytes) 000845 1781217076.928883 fallocate 0x0204c3 thru 0x0253e9 (0x4f27 bytes) 000846 1781217076.935015 punch from 0x001c29 to 0x009f8c (0x8364 bytes) 000847 1781217076.938978 write 0x0332c4 thru 0x039383 (0x60c0 bytes) 000848 1781217076.944045 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39384): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39384): Mirror 2: 0xde8f338d (stale) Mirror 3: 0xde8f338d (stale) CRC-32 checksum value for chunk [0, 0x39384): Mirror 1: 0xf5bdcc4a Mirror 4: 0xf5bdcc4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39384: skipped 000849 1781217077.562158 mapwrite 0x01dbb1 thru 0x0202b8 (0x2708 bytes) 000850 1781217077.638891 fallocate 0x01db57 thru 0x021db5 (0x425f bytes) 000851 1781217077.643741 punch from 0x0307f7 to 0x039383 (0x8b8d bytes) 000853 1781217077.649194 trunc from 0x01e5b8 to 0x022e73 (0x48bc bytes) 000854 1781217077.652920 fallocate 0x007c7c thru 0x014917 (0xcc9c bytes) 000855 1781217077.656711 fallocate 0x0384b3 thru 0x03ffff (0x7b4d bytes) 000856 1781217077.666989 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39384): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39384): Mirror 2: 0xde8f338d (stale) Mirror 3: 0xde8f338d (stale) Mirror 4: 0xf5bdcc4a (stale) CRC-32 checksum value for chunk [0, 0x39384): Mirror 1: 0x8377403 Mirror 5: 0x8377403 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39384: skipped 000857 1781217078.406406 read 0x023643 thru 0x0243d0 (0xd8e bytes) 000859 1781217078.410516 punch from 0x01626e to 0x025498 (0xf22b bytes) 000860 1781217078.435037 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39384): Mirror 3: 0xde8f338d (stale) Mirror 4: 0xf5bdcc4a (stale) Mirror 5: 0x8377403 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39384): Mirror 3: 0xde8f338d (stale) Mirror 4: 0xf5bdcc4a (stale) Mirror 5: 0x8377403 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39384: skipped 000863 1781217078.680562 punch from 0x02ccc3 to 0x034b04 (0x7e42 bytes) 000864 1781217078.685597 write 0x031cd1 thru 0x03e82a (0xcb5a bytes) 000865 1781217078.694034 trunc from 0x03e82b to 0x021991 (0x1ce99 bytes) 000866 1781217078.709977 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21992): Mirror 3: 0x3194fa3 (stale) Mirror 5: 0x6b3eb39c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21992): Mirror 3: 0x3194fa3 (stale) Mirror 5: 0x6b3eb39c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21992: skipped 000867 1781217078.884519 fallocate 0x013225 thru 0x022b3b (0xf917 bytes) 000868 1781217078.888050 mapwrite 0x026727 thru 0x02ee78 (0x8752 bytes) 000869 1781217078.938227 write 0x0243c7 thru 0x02d2d0 (0x8f0a bytes) 000870 1781217078.939634 write 0x01714d thru 0x01743b (0x2ef bytes) 000871 1781217078.949100 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ee79): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee79): Mirror 3: 0x22fc6056 (stale) Mirror 5: 0xa8cabd59 (stale) CRC-32 checksum value for chunk [0, 0x2ee79): Mirror 1: 0xb3d91f38 Mirror 6: 0xb3d91f38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee79: skipped 000872 1781217079.490922 mapread 0x0084fb thru 0x016e04 (0xe90a bytes) 000873 1781217079.510338 punch from 0x02b3e8 to 0x02ee78 (0x3a91 bytes) 000876 1781217079.540997 mapwrite 0x00bb4c thru 0x014660 (0x8b15 bytes) 000877 1781217079.568554 mapread 0x023b08 thru 0x02533c (0x1835 bytes) 000878 1781217079.575931 mapwrite 0x01e3f6 thru 0x0273a9 (0x8fb4 bytes) 000879 1781217079.594197 punch from 0x02063f to 0x026f32 (0x68f4 bytes) 000880 1781217079.599170 trunc from 0x02ee79 to 0x024f40 (0x9f38 bytes) 000881 1781217079.620956 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x24f41): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24f41): Mirror 3: 0xc22414e3 (stale) Mirror 5: 0x738c486b (stale) Mirror 6: 0xd0948666 (stale) CRC-32 checksum value for chunk [0, 0x24f41): Mirror 1: 0xe9bdeaf2 Mirror 7: 0xe9bdeaf2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24f41: skipped 000882 1781217080.300877 write 0x009f3e thru 0x00e1c3 (0x4286 bytes) 000883 1781217080.337541 fallocate 0x03cd85 thru 0x03ffff (0x327b bytes) 000885 1781217080.344097 write 0x01e649 thru 0x02e146 (0xfafe bytes) 000886 1781217080.354712 trunc from 0x02e147 to 0x007f52 (0x261f4 bytes) 000887 1781217080.372185 mapread 0x0067fc thru 0x007f52 (0x1757 bytes) 000891 1781217080.380604 fallocate 0x02e181 thru 0x03cf8c (0xee0c bytes) 000892 1781217080.387673 MIRROR_RESYNC 5 mirrors 000893 1781217080.912537 fallocate 0x0360a6 thru 0x037a03 (0x195e bytes) 000894 1781217080.948161 MIRROR_RESYNC 5 mirrors 000896 1781217081.460483 punch from 0x028b25 to 0x02eacc (0x5fa8 bytes) 000897 1781217081.505290 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37a04): Mirror 5: 0x3b3e206e (stale) Mirror 6: 0x3b3e206e (stale) Mirror 7: 0x3b3e206e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a04): Mirror 5: 0x3b3e206e (stale) Mirror 6: 0x3b3e206e (stale) Mirror 7: 0x3b3e206e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a04: skipped 000898 1781217081.730590 punch from 0x010869 to 0x017ef4 (0x768c bytes) 000899 1781217081.737598 MIRROR_RESYNC 4 mirrors 000900 1781217082.238951 MIRROR_RESYNC 4 mirrors 000901 1781217082.491648 trunc from 0x01f544 to 0x0279ac (0x8469 bytes) 000902 1781217082.511221 mapread 0x001e98 thru 0x0090ac (0x7215 bytes) 000903 1781217082.526398 mapwrite 0x01f29f thru 0x02ce4a (0xdbac bytes) 000904 1781217082.557932 punch from 0x02bd7b to 0x034301 (0x8587 bytes) 000905 1781217082.563811 trunc from 0x037a04 to 0x02681c (0x111e7 bytes) 000906 1781217082.577255 fallocate 0x0177e7 thru 0x022f4c (0xb766 bytes) 000907 1781217082.580700 read 0x0139a1 thru 0x014f0d (0x156d bytes) 000908 1781217082.585312 write 0x02c420 thru 0x034fb4 (0x8b95 bytes) 000909 1781217082.587652 punch from 0x00603c to 0x0101b9 (0xa17e bytes) 000910 1781217082.594443 mapread 0x00a63c thru 0x013e97 (0x985c bytes) 000911 1781217082.601454 punch from 0x002357 to 0x01029c (0xdf46 bytes) 000912 1781217082.605572 punch from 0x02fc2a to 0x034fb4 (0x538b bytes) 000913 1781217082.615657 mapread 0x01bebe thru 0x01fb7b (0x3cbe bytes) 000914 1781217082.619708 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34fb5): Mirror 6: 0xe9d5769e (stale) Mirror 7: 0xe9d5769e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34fb5): Mirror 6: 0xe9d5769e (stale) Mirror 7: 0xe9d5769e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34fb5: skipped 000915 1781217082.823670 read 0x018f36 thru 0x0190e6 (0x1b1 bytes) 000916 1781217082.830155 fallocate 0x03b679 thru 0x03ffff (0x4987 bytes) 000917 1781217082.834523 write 0x0153cc thru 0x01686b (0x14a0 bytes) 000920 1781217082.844988 mapwrite 0x02c344 thru 0x0331a1 (0x6e5e bytes) 000921 1781217082.865627 fallocate 0x0363f2 thru 0x03ffff (0x9c0e bytes) 000922 1781217082.872900 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa1a8394 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa1a8394 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000923 1781217083.092369 fallocate 0x032b4e thru 0x0399de (0x6e91 bytes) 000924 1781217083.097502 mapread 0x027d29 thru 0x0314be (0x9796 bytes) 000926 1781217083.120139 punch from 0x00c03d to 0x00ffd4 (0x3f98 bytes) 000927 1781217083.123850 punch from 0x02e1f8 to 0x032910 (0x4719 bytes) 000928 1781217083.129104 trunc from 0x024632 to 0x02b1b0 (0x6b7f bytes) 000929 1781217083.135312 mapread 0x02456a thru 0x02fe02 (0xb899 bytes) 000930 1781217083.153597 trunc from 0x0219d8 to 0x0243c6 (0x29ef bytes) 000931 1781217083.158319 fallocate 0x02f10e thru 0x030352 (0x1245 bytes) 000933 1781217083.162672 punch from 0x00642a to 0x012f20 (0xcaf7 bytes) 000934 1781217083.167202 mapread 0x039c54 thru 0x03ffff (0x63ac bytes) 000935 1781217083.173393 trunc from 0x040000 to 0x03eddd (0x1222 bytes) 000937 1781217083.193092 mapread 0x01cbf5 thru 0x02634f (0x975b bytes) 000939 1781217083.206512 MIRROR_RESYNC 2 mirrors 000940 1781217083.653317 MIRROR_EXTEND to 3 mirrors 000941 1781217084.037226 punch from 0x02be3f to 0x038b2f (0xccf1 bytes) 000942 1781217084.062021 trunc from 0x035359 to 0x03eddd (0x9a85 bytes) 000943 1781217084.067658 mapread 0x03a3b7 thru 0x03da76 (0x36c0 bytes) 000944 1781217084.083246 read 0x000dae thru 0x00e640 (0xd893 bytes) 000945 1781217084.088897 write 0x031461 thru 0x0340dc (0x2c7c bytes) 000946 1781217084.096535 trunc from 0x03edde to 0x0268dd (0x18500 bytes) 000948 1781217084.135202 mapwrite 0x012ba6 thru 0x013c14 (0x106f bytes) 000949 1781217084.152628 read 0x003ebe thru 0x010823 (0xc966 bytes) 000950 1781217084.153298 fallocate 0x039023 thru 0x03971f (0x6fd bytes) 000951 1781217084.158838 trunc from 0x0268de to 0x034adb (0xe1fe bytes) 000952 1781217084.168120 punch from 0x00a2fc to 0x00d8c2 (0x35c7 bytes) 000953 1781217084.171844 trunc from 0x034adc to 0x017359 (0x1d782 bytes) 000954 1781217084.186652 mapwrite 0x03fb49 thru 0x03ffff (0x4b7 bytes) 000955 1781217084.216074 mapread 0x01e30b thru 0x027438 (0x912e bytes) 000956 1781217084.217335 trunc from 0x040000 to 0x009b52 (0x364ad bytes) 000957 1781217084.228336 trunc from 0x0049f2 to 0x009789 (0x4d98 bytes) 000958 1781217084.233714 mapread 0x00215a thru 0x0076b9 (0x5560 bytes) 000959 1781217084.251157 trunc from 0x0027da to 0x009b52 (0x7379 bytes) 000960 1781217084.260528 MIRROR_RESYNC 3 mirrors 000961 1781217084.574095 MIRROR_RESYNC 3 mirrors 000962 1781217084.754571 punch from 0x000ae0 to 0x009b52 (0x9073 bytes) 000963 1781217084.779056 punch from 0x0069fb to 0x009b52 (0x3158 bytes) 000964 1781217084.783137 trunc from 0x009b53 to 0x02590d (0x1bdbb bytes) 000965 1781217084.813981 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2590e): Mirror 8: 0x17964937 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2590e): Mirror 8: 0x17964937 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2590e: skipped 000966 1781217084.994339 fallocate 0x0163b5 thru 0x0259a8 (0xf5f4 bytes) 000967 1781217085.002485 MIRROR_RESYNC 2 mirrors 000968 1781217085.317270 write 0x0152be thru 0x024849 (0xf58c bytes) 000969 1781217085.345621 mapread 0x00a896 thru 0x018d24 (0xe48f bytes) 000970 1781217085.359176 punch from 0x0215cb to 0x02526b (0x3ca1 bytes) 000971 1781217085.373863 trunc from 0x0259a9 to 0x036076 (0x106ce bytes) 000973 1781217085.391683 mapwrite 0x0266e7 thru 0x027230 (0xb4a bytes) 000974 1781217085.412359 mapwrite 0x01b2cc thru 0x02308e (0x7dc3 bytes) 000975 1781217085.434811 trunc from 0x024963 to 0x02fd9b (0xb439 bytes) 000976 1781217085.439325 mapwrite 0x031995 thru 0x03e9b3 (0xd01f bytes) 000977 1781217085.487142 read 0x03b7e5 thru 0x03e9b3 (0x31cf bytes) 000978 1781217085.487950 trunc from 0x03e9b4 to 0x02d9ef (0x10fc4 bytes) 000979 1781217085.501349 trunc from 0x02b3b3 to 0x02d9ef (0x263d bytes) 000980 1781217085.504581 punch from 0x00f73c to 0x01c1e6 (0xcaab bytes) 000981 1781217085.509937 trunc from 0x02d9f0 to 0x031aa5 (0x40b6 bytes) 000983 1781217085.526273 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31aa6): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31aa6): Mirror 8: 0xffde66d4 (stale) CRC-32 checksum value for chunk [0, 0x31aa6): Mirror 1: 0xd6020c65 Mirror 9: 0xd6020c65 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31aa6: skipped 000984 1781217085.980289 trunc from 0x031aa6 to 0x0347cc (0x2d27 bytes) 000985 1781217086.005251 trunc from 0x00c113 to 0x00d89e (0x178c bytes) 000986 1781217086.008304 fallocate 0x001f6d thru 0x0067f6 (0x488a bytes) 000987 1781217086.011944 fallocate 0x03a9f5 thru 0x03ffff (0x560b bytes) 000988 1781217086.015513 trunc from 0x0347cd to 0x012976 (0x21e56 bytes) 000989 1781217086.028601 punch from 0x00b903 to 0x012976 (0x7074 bytes) 000990 1781217086.031866 mapread 0x006d18 thru 0x00e289 (0x7572 bytes) 000991 1781217086.044674 punch from 0x004479 to 0x012976 (0xe4fe bytes) 000992 1781217086.049561 mapwrite 0x03b6d8 thru 0x03ffff (0x4928 bytes) 000993 1781217086.076732 fallocate 0x01ed22 thru 0x02e285 (0xf564 bytes) 000995 1781217086.081505 fallocate 0x00ff76 thru 0x014ffc (0x5087 bytes) 000996 1781217086.085013 trunc from 0x028443 to 0x033f7c (0xbb3a bytes) 000997 1781217086.088177 trunc from 0x005593 to 0x00e274 (0x8ce2 bytes) 000998 1781217086.091851 trunc from 0x0119f5 to 0x0168a2 (0x4eae bytes) 000999 1781217086.095664 fallocate 0x03e521 thru 0x03ffff (0x1adf bytes) 001000 1781217086.099134 mapwrite 0x002155 thru 0x0103c6 (0xe272 bytes) All operations completed A-OK! PASS 70a (63s) == sanity-flr test 71: check mirror extend parent fid ==== 18:31:28 (1781217088) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x36b:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (5s) == sanity-flr test 72: check append on FLR file ========== 18:31:33 (1781217093) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122431 s, 17.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x139:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780985 s, 13.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x139:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x139:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13a:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 18:31:38 (1781217098) Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre2 Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4743040 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3377856 .. failed Extending file size to 8702464 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8044352 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8410496 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9300800 .. Extending file size to 8486016 .. Extending file size to 9125856 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9384128 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 62645 62646 62647 62650 62652 192.168.206.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg617-client.virtnet /mnt/lustre2 (opts:) 192.168.206.117@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg617-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (69s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:32:46 (1781217166) Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre2 Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6543104 .. Extending file size to 1941984 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6677760 .. Extending file size to 8209408 .. Extending file size to 6984448 .. Extending file size to 7704256 .. Extending file size to 8605088 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8864768 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9168544 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8696480 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 65776 65777 65779 65780 65783 192.168.206.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg617-client.virtnet /mnt/lustre2 (opts:) 192.168.206.117@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg617-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:33:53 (1781217233) Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2720 1268584 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2324 1268980 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17768 3589252 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25712 3581308 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43480 7170560 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2720 1268584 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2324 1268980 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17768 3589252 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25712 3581308 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43480 7170560 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0561061 s, 37.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0886343 s, 23.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xf492ec5f (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.206.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg617-client.virtnet /mnt/lustre2 (opts:) PASS 200c (9s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:34:03 (1781217243) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0565519 s, 37.1 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 18:34:06 (1781217246) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x155:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:34:10 (1781217250) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0509791 s, 41.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x479:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x479:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x47a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:34:14 (1781217254) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0421691 s, 49.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x156:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x156:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x157:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:34:19 (1781217259) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0492911 s, 42.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x158:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x158:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x159:0x0] } PASS 204c (27s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:34:46 (1781217286) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg617-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446459 s, 23.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x159:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x159:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15a:0x0] } PASS 204d (34s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:35:20 (1781217320) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg617-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x47a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x47b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.041277 s, 50.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x47a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x47b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x47b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x47a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x47b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x47b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:35:56 (1781217356) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg617-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg617-server: Pool lustre.test_204f created Adding targets to pool oleg617-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg617-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.034848 s, 60.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg617-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg617-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg617-server: Pool lustre.test_204f destroyed PASS 204f (48s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:36:44 (1781217404) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:36:47 (1781217407) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:36:50 (1781217410) oleg617-server: Pool lustre.test_206 created Adding targets to pool oleg617-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg617-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg617-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg617-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg617-server: Pool lustre.test_206 destroyed PASS 206 (17s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:37:07 (1781217427) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320261 s, 32.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165642 s, 63.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x62de3f54 Mirror 3: 0x62de3f54 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:37:11 (1781217431) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:37:12 (1781217432) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:37:13 (1781217433) Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0178992 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1b00000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2250000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2250000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2250000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x19e0000 Read this (4 bytes): BBBB 192.168.206.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg617-client.virtnet /mnt/lustre2 (opts:) PASS 209a (19s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:37:32 (1781217452) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00987742 s, 829 kB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:37:35 (1781217455) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217933 s, 48.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb0 lmm_fid: [0x200000405:0xb0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb0 lmm_fid: [0x200000405:0xb0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x165:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611312 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-11 18:37:36.000000000 -0400 Modify: 2026-06-11 18:37:36.000000000 -0400 Change: 2026-06-11 18:37:37.000000000 -0400 Birth: 2026-06-11 18:37:35.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:37:39 (1781217459) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275419 s, 38.1 MB/s oleg617-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg617-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg617-server: O/280000401/d6/166: File not found by ext2_lookup PASS 210b (16s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:37:55 (1781217475) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.126687 s, 323 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0157987 s, 259 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 11 18:37 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xb5:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 11 18:37 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 18:37:59 (1781217479) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 18:38:03 (1781217483) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (5s) == sanity-flr test complete, duration 1023 sec =========== 18:38:07 (1781217487) === sanity-flr: start cleanup 18:38:08 (1781217488) === === sanity-flr: finish cleanup 18:38:12 (1781217492) ===