-----============= acceptance-small: sanity-flr ============----- Fri Sep 4 22:05:03 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg110-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 22:05:27 (1788573927) === oleg110-client.virtnet: executing check_config_client /mnt/lustre oleg110-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg110-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff92a9452cb800.idle_timeout=debug osc.lustre-OST0001-osc-ffff92a9452cb800.idle_timeout=debug disable quota as required oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 22:06:08 (1788573968) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8235] == sanity-flr test 0a: lfs mirror create with -N option == 22:06:10 (1788573970) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:06:28 (1788573988) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:06:32 (1788573992) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:06:35 (1788573995) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:06:58 (1788574018) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:07:02 (1788574022) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 22:07:06 (1788574026) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff92a9452cb800.stats=clear osc.lustre-OST0001-osc-ffff92a9452cb800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:07:13 (1788574033) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--not-pool POOL_NAME] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pin] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:07:23 (1788574043) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:07:31 (1788574051) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:07:34 (1788574054) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:07:42 (1788574062) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 22:07:51 (1788574071) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:08:00 (1788574080) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:08:07 (1788574087) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:08:15 (1788574095) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:08:26 (1788574106) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:08:35 (1788574115) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.55042 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105247 s, 10.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:08:45 (1788574125) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.401448 s, 34.0 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:08:56 (1788574136) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:09:09 (1788574149) Stopping /mnt/lustre-ost1 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92a9452cb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92a9452cb800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg110-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92a9452cb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92a9452cb800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg110-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (82s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:10:31 (1788574231) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.423367 s, 12.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92a9452cb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92a9452cb800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg110-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (50s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:11:21 (1788574281) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92a9452cb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92a9452cb800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg110-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92a9452cb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92a9452cb800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg110-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (91s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:12:52 (1788574372) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.77012 s, 13.9 MB/s -rw-r--r-- 1 root root 52428800 Sep 4 22:12 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.184s user 0m0.001s sys 0m0.180s Stopping /mnt/lustre-ost1 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92a9452cb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92a9452cb800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.315s user 0m0.000s sys 0m0.198s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg110-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92a9452cb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92a9452cb800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.514s user 0m0.001s sys 0m0.149s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg110-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg110-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92a9452cb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92a9452cb800.ost_server_uuid in FULL state after 0 sec PASS 33b (110s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:14:43 (1788574483) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:14:46 (1788574486) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:14:50 (1788574490) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 22:14:54 (1788574494) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 22:15:03 (1788574503) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.536207 s, 23.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:15:19 (1788574519) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081465 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844477 s, 12.4 MB/s PASS 36b (13s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:15:32 (1788574532) PASS 36c (11s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:15:43 (1788574543) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00474238 s, 216 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00107533 s, 952 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000785462 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00424393 s, 241 kB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 22:15:57 (1788574557) create test files with size 15734325 .. 73cf544c80042bdf07d7dc6d87c5b3e3 - 31e98a4c726273b09446f01979106dc8 - 54f984c7da24e342a554748e94f311a9 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre PASS 37 (30s) == sanity-flr test 38: resync ============================ 22:16:28 (1788574588) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:16:44 (1788574604) PASS 39 (10s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:16:54 (1788574614) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908688 s, 11.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270346 s, 38.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 22:17:04 (1788574624) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0896411 s, 23.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.365164 s, 11.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182361 s, 11.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 22:17:21 (1788574641) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 22:17:26 (1788574646) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:17:30 (1788574650) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498796 s, 21.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 22:17:39 (1788574659) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.430632 s, 7.3 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:17:51 (1788574671) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:18:04 (1788574684) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.565475 s, 18.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (19s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:18:23 (1788574703) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff92a943b0b000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.415714 s, 25.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff92a944d62000.xattr_cache=1 PASS 44e (21s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:18:45 (1788574725) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (57s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:19:42 (1788574782) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (52s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:20:34 (1788574834) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 22:20:39 (1788574839) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.261092 s, 12.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0705836 s, 44.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x6bcc265f (nosync,stale) Mirror 2: 0x6bcc265f (nosync,stale) ** verify mirror 2 doesn't change original checksum: d193ace65e1b3e1471aeb2ecce95dd35 - mirror 1 checksum: ec708dcc88e3eb2426d8750c2bf34350 - mirror 2 checksum: d193ace65e1b3e1471aeb2ecce95dd35 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T22:20:43-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T22:20:43-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d193ace65e1b3e1471aeb2ecce95dd35 - mirror 1 checksum: ec708dcc88e3eb2426d8750c2bf34350 - mirror 2 checksum: ec708dcc88e3eb2426d8750c2bf34350 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T22:20:43-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T22:20:43-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 22:20:53 (1788574853) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039829 s, 26.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (10s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:21:03 (1788574863) Failing mds1 on oleg110-server Stopping /mnt/lustre-mds1 (opts:) on oleg110-server 22:21:17 (1788574877) shut down facet: mds1 facet_host: oleg110-server facet_failover_host: oleg110-server Failover mds1 to oleg110-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-MDT0000 22:21:55 (1788574915) targets are mounted 22:21:55 (1788574915) facet_failover done oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290872 s, 36.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 50A (74s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:22:17 (1788574937) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0828868 s, 247 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0206182 s, 993 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (13s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:22:31 (1788574951) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0782074 s, 262 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0126281 s, 1.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:22:42 (1788574962) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00900742 s, 1.8 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:22:51 (1788574971) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.101893 s, 201 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (11s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:23:02 (1788574982) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00990721 s, 413 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0251404 s, 163 kB/s PASS 60a (11s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:23:13 (1788574993) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:23:38 (1788575018) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:24:09 (1788575049) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:24:34 (1788575074) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:25:01 (1788575101) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 22:25:08 (1788575108) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57624 mirror split pid 57625 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57624 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57625 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 22:26:18 (1788575178) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 5180 skipping zero length punch hole skipping zero length punch hole 000003 1788575180.906327 MIRROR_EXTEND to 2 mirrors 000004 1788575181.464314 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) skipping zero length punch hole skipping zero size read 000008 1788575181.795405 write 0x02e40b thru 0x02ed74 (0x96a bytes) truncating to largest ever: 0x19dc6 000009 1788575181.804090 trunc from 0x02ed75 to 0x019dc5 (0x14faf bytes) 000011 1788575181.837532 read 0x00f82a thru 0x019dc5 (0xa59c bytes) 000012 1788575181.858023 read 0x000c66 thru 0x00a18a (0x9525 bytes) 000013 1788575181.880512 mapread 0x007d12 thru 0x0155a9 (0xd898 bytes) 000015 1788575181.887201 write 0x0187b7 thru 0x019347 (0xb91 bytes) 000016 1788575181.889067 write 0x01ba3a thru 0x026bbe (0xb185 bytes) 000017 1788575181.892243 mapwrite 0x017247 thru 0x02224c (0xb006 bytes) 000018 1788575181.942179 trunc from 0x0013de to 0x00d38f (0xbfb2 bytes) 000019 1788575181.970229 trunc from 0x00750c to 0x01385b (0xc350 bytes) truncating to largest ever: 0x297c8 000020 1788575181.980508 trunc from 0x026bbf to 0x0297c7 (0x2c09 bytes) 000022 1788575182.037323 write 0x02e49f thru 0x035800 (0x7362 bytes) 000023 1788575182.048032 mapwrite 0x029d45 thru 0x031029 (0x72e5 bytes) 000024 1788575182.110030 read 0x003a0f thru 0x004384 (0x976 bytes) 000025 1788575182.143301 MIRROR_EXTEND to 2 mirrors 000026 1788575183.023226 MIRROR_RESYNC 2 mirrors 000028 1788575183.536410 trunc from 0x005811 to 0x006593 (0xd83 bytes) truncating to largest ever: 0x301d2 000029 1788575183.620225 trunc from 0x035801 to 0x0301d1 (0x562f bytes) 000030 1788575183.745725 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) 000031 1788575184.050530 read 0x001dfa thru 0x005847 (0x3a4e bytes) 000033 1788575184.069826 mapwrite 0x000ed2 thru 0x00dffd (0xd12c bytes) 000034 1788575184.118456 mapread 0x0277b6 thru 0x02e1ea (0x6a35 bytes) 000035 1788575184.137245 punch from 0x024d89 to 0x025c2d (0xea5 bytes) 000036 1788575184.144776 trunc from 0x026050 to 0x027fa2 (0x1f53 bytes) 000037 1788575184.152540 mapwrite 0x03d9c8 thru 0x03ffff (0x2638 bytes) truncating to largest ever: 0x33180 000038 1788575184.213115 trunc from 0x040000 to 0x03317f (0xce80 bytes) 000039 1788575184.229975 read 0x00f4b8 thru 0x01a8af (0xb3f8 bytes) 000040 1788575184.253425 trunc from 0x033180 to 0x008368 (0x2ae17 bytes) 000044 1788575184.284527 mapread 0x001978 thru 0x007d33 (0x63bc bytes) 000046 1788575184.295886 trunc from 0x001017 to 0x004b31 (0x3b1b bytes) 000047 1788575184.302242 fallocate 0x020cbf thru 0x02b0a4 (0xa3e6 bytes) 000048 1788575184.313674 punch from 0x0039e4 to 0x008368 (0x4985 bytes) 000049 1788575184.319712 read 0x005229 thru 0x005e05 (0xbdd bytes) 000050 1788575184.332288 read 0x006d23 thru 0x008368 (0x1646 bytes) fallocating to largest ever: 0x35928 000051 1788575184.334198 fallocate 0x02ca62 thru 0x035927 (0x8ec6 bytes) 000052 1788575184.341297 punch from 0x0174c5 to 0x01b521 (0x405d bytes) 000053 1788575184.347776 punch from 0x02b37e to 0x035927 (0xa5aa bytes) 000054 1788575184.354185 write 0x016471 thru 0x0204ef (0xa07f bytes) 000055 1788575184.370141 read 0x02d5ed thru 0x035927 (0x833b bytes) 000056 1788575184.370873 trunc from 0x035928 to 0x031b7d (0x3daa bytes) 000057 1788575184.400811 trunc from 0x031b7e to 0x01a4bd (0x176c0 bytes) 000058 1788575184.448491 punch from 0x0070fa to 0x008639 (0x1540 bytes) 000059 1788575184.456164 mapwrite 0x03bc48 thru 0x03ffff (0x43b8 bytes) 000060 1788575184.519198 read 0x01733e thru 0x018204 (0xec7 bytes) 000062 1788575184.522937 trunc from 0x0041c5 to 0x00ef98 (0xadd4 bytes) 000064 1788575184.531512 write 0x00483d thru 0x0086e3 (0x3ea7 bytes) 000065 1788575184.549213 trunc from 0x01d4c4 to 0x023751 (0x628e bytes) 000066 1788575184.561532 write 0x00a5bc thru 0x00b189 (0xbce bytes) 000067 1788575184.562855 mapwrite 0x00679b thru 0x010e6b (0xa6d1 bytes) 000068 1788575184.595314 fallocate 0x00c931 thru 0x00ec30 (0x2300 bytes) 000069 1788575184.602650 mapwrite 0x02e44a thru 0x034198 (0x5d4f bytes) 000070 1788575184.630905 write 0x0196eb thru 0x020068 (0x697e bytes) 000071 1788575184.643276 punch from 0x0282d0 to 0x03764c (0xf37d bytes) 000072 1788575184.651008 fallocate 0x006e37 thru 0x00b613 (0x47dd bytes) 000073 1788575184.656826 mapread 0x0046da thru 0x013f28 (0xf84f bytes) 000074 1788575184.659133 fallocate 0x025307 thru 0x02d61f (0x8319 bytes) 000075 1788575184.665294 mapwrite 0x0145d5 thru 0x0166d7 (0x2103 bytes) 000077 1788575184.688938 read 0x00b8f9 thru 0x0126f6 (0x6dfe bytes) 000078 1788575184.689760 mapread 0x03b0bb thru 0x03ffff (0x4f45 bytes) 000079 1788575184.691388 write 0x006176 thru 0x0121cc (0xc057 bytes) 000080 1788575184.696017 read 0x03cc10 thru 0x03f86b (0x2c5c bytes) 000084 1788575184.707192 MIRROR_EXTEND to 2 mirrors 000085 1788575185.732922 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) 000086 1788575185.983943 mapread 0x000f4b thru 0x006741 (0x57f7 bytes) 000087 1788575186.065156 write 0x02e161 thru 0x036b52 (0x89f2 bytes) 000088 1788575186.104213 trunc from 0x00126e to 0x00aee2 (0x9c75 bytes) 000089 1788575186.125581 punch from 0x03a954 to 0x03ffff (0x56ac bytes) 000090 1788575186.143751 write 0x021fe6 thru 0x02d6ab (0xb6c6 bytes) 000091 1788575186.172925 read 0x001718 thru 0x010514 (0xedfd bytes) 000092 1788575186.186689 write 0x00bc4d thru 0x017137 (0xb4eb bytes) 000094 1788575186.195526 trunc from 0x023376 to 0x02be7c (0x8b07 bytes) 000095 1788575186.215478 mapread 0x00389c thru 0x0072ec (0x3a51 bytes) 000096 1788575186.224289 trunc from 0x040000 to 0x01ebc5 (0x2143a bytes) 000097 1788575186.243547 mapwrite 0x02c07a thru 0x02ef08 (0x2e8f bytes) 000099 1788575186.292943 mapwrite 0x01dfd8 thru 0x01e18a (0x1b3 bytes) 000101 1788575186.327265 mapwrite 0x02c12a thru 0x02d7bf (0x1696 bytes) 000102 1788575186.355029 MIRROR_EXTEND to 2 mirrors 000103 1788575187.329173 mapread 0x01b99d thru 0x02a18f (0xe7f3 bytes) 000104 1788575187.374019 trunc from 0x027de7 to 0x02918a (0x13a4 bytes) 000105 1788575187.432363 mapwrite 0x017e25 thru 0x0205b8 (0x8794 bytes) 000106 1788575187.517239 mapread 0x007e67 thru 0x0092b8 (0x1452 bytes) 000107 1788575187.531265 mapread 0x02df43 thru 0x02ef08 (0xfc6 bytes) 000108 1788575187.550564 mapread 0x00062c thru 0x008eee (0x88c3 bytes) 000109 1788575187.567113 punch from 0x00ee98 to 0x019d03 (0xae6c bytes) 000110 1788575187.574857 mapwrite 0x02bfb9 thru 0x035e2b (0x9e73 bytes) 000111 1788575187.709110 trunc from 0x016e5c to 0x01d5d8 (0x677d bytes) 000112 1788575187.722282 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35e2c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e2c): Mirror 2: 0x83e888a5 (stale) CRC-32 checksum value for chunk [0, 0x35e2c): Mirror 1: 0xd2e08dd5 Mirror 3: 0xd2e08dd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e2c: skipped 000113 1788575189.295791 trunc from 0x035e2c to 0x02cc52 (0x91d9 bytes) 000114 1788575189.406085 fallocate 0x027c2b thru 0x02e340 (0x6716 bytes) 000115 1788575189.424600 mapread 0x01f155 thru 0x01f4d6 (0x382 bytes) 000116 1788575189.458757 fallocate 0x0290a3 thru 0x02cd94 (0x3cf2 bytes) 000119 1788575189.496926 punch from 0x007f9a to 0x015526 (0xd58d bytes) 000120 1788575189.531513 mapwrite 0x020b87 thru 0x023ff4 (0x346e bytes) 000121 1788575189.588071 mapread 0x01129a thru 0x01b96a (0xa6d1 bytes) 000122 1788575189.606492 mapwrite 0x01a2c0 thru 0x021199 (0x6eda bytes) 000123 1788575189.660677 mapwrite 0x036f61 thru 0x03e76a (0x780a bytes) 000124 1788575189.774637 punch from 0x030552 to 0x03bec8 (0xb977 bytes) 000125 1788575189.783444 trunc from 0x03e76b to 0x01adc4 (0x239a6 bytes) 000126 1788575189.820468 write 0x00a4df thru 0x017ebf (0xd9e1 bytes) 000127 1788575189.821908 punch from 0x0079bb to 0x00b471 (0x3ab7 bytes) 000128 1788575189.861597 MIRROR_RESYNC 3 mirrors 000129 1788575191.401032 trunc from 0x01adc5 to 0x006bff (0x141c5 bytes) 000131 1788575191.467129 punch from 0x000fdb to 0x005c17 (0x4c3d bytes) 000132 1788575191.478293 trunc from 0x001e9c to 0x006bff (0x4d64 bytes) 000133 1788575191.485768 fallocate 0x0125d6 thru 0x01e6a4 (0xc0cf bytes) 000134 1788575191.490726 write 0x02459f thru 0x02afd1 (0x6a33 bytes) 000135 1788575191.493077 read 0x00ae8d thru 0x01613b (0xb2af bytes) 000136 1788575191.519376 mapread 0x0228f6 thru 0x025208 (0x2913 bytes) 000137 1788575191.530482 trunc from 0x02afd2 to 0x02886e (0x2763 bytes) 000138 1788575191.562353 trunc from 0x0180e6 to 0x022871 (0xa78c bytes) 000139 1788575191.573890 punch from 0x0275d6 to 0x028093 (0xabe bytes) 000140 1788575191.590559 mapread 0x01e348 thru 0x026dea (0x8aa3 bytes) 000141 1788575191.599311 punch from 0x011ee6 to 0x017eb2 (0x5fcd bytes) 000142 1788575191.604769 punch from 0x0133a5 to 0x01dbbd (0xa819 bytes) 000143 1788575191.609979 trunc from 0x00b938 to 0x01427c (0x8945 bytes) 000144 1788575191.614187 read 0x01d6d8 thru 0x0219bf (0x42e8 bytes) 000145 1788575191.624267 trunc from 0x018c87 to 0x01b19d (0x2517 bytes) 000146 1788575191.636974 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2886f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2886f): Mirror 2: 0xa0a5bd02 (stale) Mirror 3: 0xa0a5bd02 (stale) CRC-32 checksum value for chunk [0, 0x2886f): Mirror 1: 0xade1cfa6 Mirror 4: 0xade1cfa6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2886f: skipped 000147 1788575192.459143 mapread 0x0215c9 thru 0x02606a (0x4aa2 bytes) 000148 1788575192.494294 punch from 0x0176b1 to 0x0234cb (0xbe1b bytes) 000149 1788575192.570871 fallocate 0x010e4a thru 0x01d339 (0xc4f0 bytes) 000150 1788575192.579302 read 0x025803 thru 0x027566 (0x1d64 bytes) 000151 1788575192.590560 mapwrite 0x02fabb thru 0x032037 (0x257d bytes) 000153 1788575192.686706 trunc from 0x032038 to 0x027b71 (0xa4c6 bytes) 000154 1788575192.716580 mapwrite 0x00ea17 thru 0x016c04 (0x81ee bytes) 000155 1788575192.745682 read 0x01139e thru 0x011f4d (0xbb0 bytes) 000156 1788575192.746474 punch from 0x018a84 to 0x01ef92 (0x650f bytes) 000157 1788575192.751986 trunc from 0x0077da to 0x014e9c (0xd6c3 bytes) 000158 1788575192.757063 mapwrite 0x028d0b thru 0x029374 (0x66a bytes) 000159 1788575192.813075 trunc from 0x00f069 to 0x019b95 (0xab2d bytes) 000160 1788575192.820138 write 0x011993 thru 0x01faab (0xe119 bytes) 000161 1788575192.832138 trunc from 0x01e93e to 0x029374 (0xaa37 bytes) 000162 1788575192.851191 mapread 0x01add1 thru 0x023091 (0x82c1 bytes) 000163 1788575192.884095 read 0x004af0 thru 0x008652 (0x3b63 bytes) 000164 1788575192.885817 trunc from 0x029375 to 0x0214b3 (0x7ec1 bytes) 000165 1788575192.906991 read 0x0056b4 thru 0x006fe0 (0x192d bytes) 000166 1788575192.908121 write 0x01a057 thru 0x01f2dd (0x5287 bytes) 000167 1788575192.911018 fallocate 0x0253e7 thru 0x035337 (0xff51 bytes) 000168 1788575192.918025 fallocate 0x032181 thru 0x033cfe (0x1b7e bytes) 000170 1788575192.927587 punch from 0x025904 to 0x02c160 (0x685d bytes) 000171 1788575192.939409 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33cff): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33cff): Mirror 2: 0xa61fd2a1 (stale) Mirror 3: 0xa61fd2a1 (stale) Mirror 4: 0x914d4a0f (stale) CRC-32 checksum value for chunk [0, 0x33cff): Mirror 1: 0xe6aecd40 Mirror 5: 0xe6aecd40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33cff: skipped 000172 1788575194.197958 mapwrite 0x03703b thru 0x03c02a (0x4ff0 bytes) 000173 1788575194.365136 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c02b): Mirror 3: 0x9f9df003 (stale) Mirror 4: 0x44093d09 (stale) Mirror 5: 0xf9c47082 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c02b): Mirror 3: 0x9f9df003 (stale) Mirror 4: 0x44093d09 (stale) Mirror 5: 0xf9c47082 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c02b: skipped 000174 1788575194.977394 mapwrite 0x03a739 thru 0x03f154 (0x4a1c bytes) 000175 1788575195.043181 mapwrite 0x01b9ed thru 0x02673b (0xad4f bytes) 000176 1788575195.123479 mapwrite 0x0257b1 thru 0x032bf4 (0xd444 bytes) 000178 1788575195.164431 trunc from 0x028d08 to 0x02ad4d (0x2046 bytes) zero_range to largest ever: 0x37827 000181 1788575195.171945 trunc from 0x02ba87 to 0x037826 (0xbda0 bytes) 000182 1788575195.177939 mapwrite 0x01e682 thru 0x028ed6 (0xa855 bytes) 000183 1788575195.206460 trunc from 0x03f155 to 0x0269e0 (0x18774 bytes) 000185 1788575195.236540 MIRROR_RESYNC 4 mirrors 000186 1788575196.453461 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) 000187 1788575197.077674 MIRROR_EXTEND to 4 mirrors 000188 1788575198.059950 read 0x01bb4a thru 0x0207f3 (0x4caa bytes) 000189 1788575198.082094 MIRROR_EXTEND to 5 mirrors fallocating to largest ever: 0x40000 000190 1788575198.935856 fallocate 0x03f685 thru 0x03ffff (0x97b bytes) 000191 1788575199.004000 trunc from 0x01ffd0 to 0x0269e0 (0x6a11 bytes) 000192 1788575199.017778 MIRROR_RESYNC 5 mirrors 000193 1788575200.592328 write 0x024e04 thru 0x028534 (0x3731 bytes) 000194 1788575200.628293 punch from 0x014563 to 0x020645 (0xc0e3 bytes) 000195 1788575200.633232 write 0x03711e thru 0x03b6ec (0x45cf bytes) 000196 1788575200.635577 trunc from 0x03b6ed to 0x03aef7 (0x7f5 bytes) 000197 1788575200.673241 mapwrite 0x03ce82 thru 0x03ffff (0x317e bytes) 000199 1788575200.727016 fallocate 0x03e307 thru 0x03ffff (0x1cf9 bytes) 000200 1788575200.733606 read 0x03f432 thru 0x03ffff (0xbce bytes) 000201 1788575200.734737 read 0x023a29 thru 0x0323e8 (0xe9c0 bytes) 000202 1788575200.750021 fallocate 0x015824 thru 0x019ed5 (0x46b2 bytes) 000204 1788575200.756302 mapwrite 0x03ff4b thru 0x03ffff (0xb5 bytes) 000205 1788575200.771436 trunc from 0x040000 to 0x013df4 (0x2c20b bytes) 000207 1788575200.797383 trunc from 0x013df5 to 0x02aad2 (0x16cde bytes) 000208 1788575200.812943 trunc from 0x02aad3 to 0x037d14 (0xd242 bytes) 000209 1788575200.844314 read 0x00eb6a thru 0x010123 (0x15ba bytes) 000210 1788575200.845416 mapread 0x000e3f thru 0x005c53 (0x4e15 bytes) 000211 1788575200.848448 mapwrite 0x021f89 thru 0x0295d9 (0x7651 bytes) 000212 1788575200.892765 write 0x03abd9 thru 0x03ffff (0x5427 bytes) 000213 1788575200.908143 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x91cc7cb8 (stale) Mirror 5: 0x91cc7cb8 (stale) Mirror 6: 0x91cc7cb8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x91cc7cb8 (stale) Mirror 5: 0x91cc7cb8 (stale) Mirror 6: 0x91cc7cb8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000215 1788575201.447745 read 0x03ae7c thru 0x03ffff (0x5184 bytes) 000216 1788575201.463005 MIRROR_RESYNC 4 mirrors 000217 1788575202.661025 fallocate 0x010c86 thru 0x01d9b6 (0xcd31 bytes) 000218 1788575202.748291 read 0x006f56 thru 0x0083aa (0x1455 bytes) 000219 1788575202.757351 read 0x026ba8 thru 0x02b33c (0x4795 bytes) 000220 1788575202.772939 trunc from 0x0354bf to 0x03dc15 (0x8757 bytes) 000221 1788575202.782191 punch from 0x0266bc to 0x02dc3b (0x7580 bytes) 000222 1788575202.791187 write 0x038ac6 thru 0x03f89b (0x6dd6 bytes) 000223 1788575202.801231 read 0x00d66e thru 0x01284c (0x51df bytes) 000224 1788575202.802132 read 0x00c57c thru 0x011246 (0x4ccb bytes) 000225 1788575202.805190 punch from 0x032137 to 0x0366a1 (0x456b bytes) 000226 1788575202.813658 trunc from 0x000cd5 to 0x007257 (0x6583 bytes) 000227 1788575202.821176 punch from 0x01a623 to 0x02695f (0xc33d bytes) 000228 1788575202.827699 trunc from 0x026269 to 0x03197d (0xb715 bytes) 000229 1788575202.835571 trunc from 0x040000 to 0x00c100 (0x33eff bytes) 000230 1788575202.871157 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xc101): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc101): Mirror 4: 0x9461c3a0 (stale) Mirror 5: 0x9461c3a0 (stale) Mirror 6: 0x9461c3a0 (stale) CRC-32 checksum value for chunk [0, 0xc101): Mirror 1: 0x14ca8fe7 Mirror 7: 0x14ca8fe7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc101: skipped 000231 1788575203.779719 mapread 0x006f26 thru 0x00c100 (0x51db bytes) 000232 1788575203.832439 write 0x01897a thru 0x01dc34 (0x52bb bytes) 000233 1788575203.894551 fallocate 0x01b7af thru 0x0232d8 (0x7b2a bytes) 000234 1788575203.906299 read 0x00a8d9 thru 0x01121a (0x6942 bytes) 000235 1788575203.919064 write 0x007f9d thru 0x0139bc (0xba20 bytes) 000236 1788575203.932672 trunc from 0x00f5a5 to 0x01a768 (0xb1c4 bytes) 000237 1788575203.954955 mapread 0x01a2b7 thru 0x01dc34 (0x397e bytes) 000238 1788575203.970382 mapwrite 0x012fc6 thru 0x01cea7 (0x9ee2 bytes) 000239 1788575204.056192 write 0x010e94 thru 0x011c1b (0xd88 bytes) 000240 1788575204.057299 mapread 0x000c9c thru 0x00f69f (0xea04 bytes) 000241 1788575204.128594 punch from 0x014837 to 0x01dc34 (0x93fe bytes) 000242 1788575204.143976 trunc from 0x01dc35 to 0x01b57b (0x26b9 bytes) 000243 1788575204.208374 trunc from 0x01b57c to 0x013269 (0x8312 bytes) 000245 1788575204.239547 mapread 0x000d11 thru 0x007a63 (0x6d53 bytes) 000246 1788575204.257095 MIRROR_RESYNC 5 mirrors 000247 1788575205.572212 trunc from 0x01326a to 0x012fee (0x27b bytes) 000248 1788575205.603049 trunc from 0x012fef to 0x030d50 (0x1dd62 bytes) 000249 1788575205.613899 write 0x028f50 thru 0x0291bd (0x26e bytes) 000250 1788575205.623590 trunc from 0x0056c5 to 0x010dd5 (0xb711 bytes) 000251 1788575205.628130 fallocate 0x00327e thru 0x00fe15 (0xcb98 bytes) 000252 1788575205.633932 mapwrite 0x036503 thru 0x03770f (0x120d bytes) 000253 1788575205.700724 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37710): Mirror 5: 0xa674a3f8 (stale) Mirror 6: 0xa674a3f8 (stale) Mirror 7: 0xa674a3f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37710): Mirror 5: 0xa674a3f8 (stale) Mirror 6: 0xa674a3f8 (stale) Mirror 7: 0xa674a3f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37710: skipped 000254 1788575206.123109 write 0x036bfe thru 0x03ffff (0x9402 bytes) 000255 1788575206.145901 mapwrite 0x00be5c thru 0x0132b2 (0x7457 bytes) 000256 1788575206.230306 read 0x009665 thru 0x00bd22 (0x26be bytes) 000257 1788575206.232228 read 0x03139c thru 0x033f88 (0x2bed bytes) 000258 1788575206.240821 read 0x01e4e1 thru 0x023eff (0x5a1f bytes) 000259 1788575206.241706 fallocate 0x031b99 thru 0x03b160 (0x95c8 bytes) 000260 1788575206.250148 write 0x0118db thru 0x01d8e7 (0xc00d bytes) 000261 1788575206.252008 fallocate 0x011f83 thru 0x0206b1 (0xe72f bytes) 000262 1788575206.256465 fallocate 0x02099f thru 0x021967 (0xfc9 bytes) 000263 1788575206.264258 trunc from 0x00ee1e to 0x01432f (0x5512 bytes) 000264 1788575206.311926 mapwrite 0x030956 thru 0x03da39 (0xd0e4 bytes) 000265 1788575206.379551 mapwrite 0x02c9d9 thru 0x03b1a2 (0xe7ca bytes) 000266 1788575206.426161 mapwrite 0x005d9a thru 0x00b465 (0x56cc bytes) 000267 1788575206.462980 write 0x013168 thru 0x01c8bd (0x9756 bytes) 000268 1788575206.479350 mapwrite 0x0387ff thru 0x03f434 (0x6c36 bytes) 000269 1788575206.519819 mapread 0x01b904 thru 0x024e7e (0x957b bytes) 000270 1788575206.521988 write 0x02cd62 thru 0x030a89 (0x3d28 bytes) 000271 1788575206.529316 fallocate 0x02dad5 thru 0x032f37 (0x5463 bytes) 000272 1788575206.548542 write 0x00bd29 thru 0x013ea7 (0x817f bytes) 000273 1788575206.549757 fallocate 0x01598a thru 0x01cda1 (0x7418 bytes) 000274 1788575206.562761 write 0x00db60 thru 0x018a49 (0xaeea bytes) 000275 1788575206.564099 trunc from 0x040000 to 0x035d6b (0xa294 bytes) 000277 1788575206.608216 trunc from 0x035d6c to 0x01d63e (0x1872d bytes) 000278 1788575206.667880 trunc from 0x019a22 to 0x01d63e (0x3c1d bytes) 000279 1788575206.682703 punch from 0x006313 to 0x01609f (0xfd8d bytes) 000280 1788575206.727591 trunc from 0x0063cc to 0x00f0bb (0x8cf0 bytes) 000281 1788575206.742602 write 0x02deeb thru 0x03b926 (0xda3c bytes) 000282 1788575206.747869 fallocate 0x034f6a thru 0x036c17 (0x1cae bytes) 000283 1788575206.754564 mapwrite 0x020c85 thru 0x020fb8 (0x334 bytes) 000285 1788575206.787696 read 0x0168de thru 0x016eeb (0x60e bytes) 000286 1788575206.828035 fallocate 0x02056c thru 0x02ba4e (0xb4e3 bytes) 000287 1788575206.833882 fallocate 0x028783 thru 0x035876 (0xd0f4 bytes) 000289 1788575206.840037 fallocate 0x016140 thru 0x01fddd (0x9c9e bytes) 000290 1788575206.845952 fallocate 0x0217aa thru 0x0286b5 (0x6f0c bytes) 000291 1788575206.851043 mapread 0x00a2a9 thru 0x01079f (0x64f7 bytes) 000293 1788575206.853233 mapwrite 0x0149e2 thru 0x01b98d (0x6fac bytes) 000294 1788575206.869728 mapread 0x01104e thru 0x01ecdc (0xdc8f bytes) 000295 1788575206.870369 fallocate 0x02bc8c thru 0x032955 (0x6cca bytes) 000296 1788575206.875310 punch from 0x018e7a to 0x019a94 (0xc1b bytes) 000297 1788575206.879991 mapread 0x01f7ed thru 0x0252ce (0x5ae2 bytes) 000299 1788575206.881378 trunc from 0x03b927 to 0x01952d (0x223f9 bytes) 000300 1788575206.897867 trunc from 0x014da0 to 0x0157fd (0xa5e bytes) 000301 1788575206.904238 trunc from 0x00abb0 to 0x016dff (0xc250 bytes) 000302 1788575206.910556 fallocate 0x01028b thru 0x01bf91 (0xbd07 bytes) 000303 1788575206.916345 mapwrite 0x021e2d thru 0x02ef7a (0xd14e bytes) 000304 1788575207.023261 fallocate 0x037b98 thru 0x03a0ae (0x2517 bytes) 000306 1788575207.034101 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 5: 0xcb71a32 (stale) Mirror 6: 0xcb71a32 (stale) Mirror 7: 0xcb71a32 (stale) CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 1: 0x86c19b49 Mirror 8: 0x86c19b49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef7b: skipped 000307 1788575208.369829 trunc from 0x0056d9 to 0x00a2d1 (0x4bf9 bytes) 000308 1788575208.452065 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 5: 0xcb71a32 (stale) Mirror 6: 0xcb71a32 (stale) Mirror 7: 0xcb71a32 (stale) Mirror 8: 0x86c19b49 (stale) CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 1: 0x5b21cb4d Mirror 9: 0x5b21cb4d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef7b: skipped 000309 1788575209.865516 mapwrite 0x01c8c2 thru 0x01e8b4 (0x1ff3 bytes) 000310 1788575210.037499 read 0x024fb0 thru 0x02ef7a (0x9fcb bytes) 000311 1788575210.047706 write 0x018450 thru 0x0232c6 (0xae77 bytes) 000312 1788575210.058561 fallocate 0x029704 thru 0x0323ef (0x8cec bytes) 000313 1788575210.080571 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 5: 0xcb71a32 (stale) Mirror 6: 0xcb71a32 (stale) Mirror 7: 0xcb71a32 (stale) Mirror 8: 0x86c19b49 (stale) Mirror 9: 0x5b21cb4d (stale) CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 1: 0x65cbc012 Mirror 10: 0x65cbc012 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef7b: skipped 000314 1788575211.095642 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 5: 0xcb71a32 (stale) Mirror 6: 0xcb71a32 (stale) Mirror 7: 0xcb71a32 (stale) Mirror 8: 0x86c19b49 (stale) Mirror 9: 0x5b21cb4d (stale) CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 1: 0x65cbc012 Mirror 10: 0x65cbc012 Mirror 11: 0x65cbc012 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef7b: skipped 000315 1788575212.430138 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 5: 0xcb71a32 (stale) Mirror 6: 0xcb71a32 (stale) Mirror 7: 0xcb71a32 (stale) Mirror 8: 0x86c19b49 (stale) Mirror 9: 0x5b21cb4d (stale) CRC-32 checksum value for chunk [0, 0x2ef7b): Mirror 1: 0x65cbc012 Mirror 10: 0x65cbc012 Mirror 11: 0x65cbc012 Mirror 12: 0x65cbc012 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef7b: skipped 000316 1788575213.397073 fallocate 0x0118be thru 0x01c54e (0xac91 bytes) 000317 1788575213.449981 trunc from 0x02ef7b to 0x008534 (0x26a46 bytes) 000318 1788575213.577332 trunc from 0x008535 to 0x02626a (0x1dd36 bytes) 000319 1788575213.614095 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2626b): Mirror 5: 0xe3c7da64 (stale) Mirror 6: 0xe3c7da64 (stale) Mirror 8: 0x6bd8feb (stale) Mirror 9: 0xd1102562 (stale) Mirror 10: 0x94496dad (stale) Mirror 11: 0x94496dad (stale) Mirror 12: 0x94496dad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2626b): Mirror 5: 0xe3c7da64 (stale) Mirror 6: 0xe3c7da64 (stale) Mirror 8: 0x6bd8feb (stale) Mirror 9: 0xd1102562 (stale) Mirror 10: 0x94496dad (stale) Mirror 11: 0x94496dad (stale) Mirror 12: 0x94496dad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2626b: skipped 000320 1788575214.169813 write 0x02d81f thru 0x03c806 (0xefe8 bytes) 000321 1788575214.175136 trunc from 0x02174d to 0x02500a (0x38be bytes) 000322 1788575214.178095 trunc from 0x01784e to 0x024c0c (0xd3bf bytes) 000323 1788575214.180998 trunc from 0x03c807 to 0x00226c (0x3a59a bytes) 000324 1788575214.191203 write 0x0332d9 thru 0x034163 (0xe8b bytes) 000325 1788575214.192341 fallocate 0x037a8f thru 0x037b0e (0x80 bytes) 000326 1788575214.197376 punch from 0x00b0b8 to 0x0149e3 (0x992c bytes) 000327 1788575214.201782 trunc from 0x030e2f to 0x037b0e (0x6ce0 bytes) 000328 1788575214.207805 fallocate 0x00ce40 thru 0x0123a4 (0x5565 bytes) 000329 1788575214.211268 trunc from 0x037b0f to 0x003182 (0x3498c bytes) 000330 1788575214.221349 mapwrite 0x01e175 thru 0x026792 (0x861e bytes) 000331 1788575214.255202 read 0x022323 thru 0x026792 (0x4470 bytes) 000332 1788575214.258853 trunc from 0x005699 to 0x0091d1 (0x3b39 bytes) 000333 1788575214.263271 punch from 0x024cef to 0x026792 (0x1aa4 bytes) 000334 1788575214.269304 trunc from 0x026793 to 0x037434 (0x10ca2 bytes) 000336 1788575214.284865 MIRROR_RESYNC 8 mirrors 000337 1788575216.001336 fallocate 0x01046f thru 0x01a115 (0x9ca7 bytes) 000338 1788575216.061009 fallocate 0x039b64 thru 0x03ffff (0x649c bytes) 000339 1788575216.071054 mapwrite 0x0384f6 thru 0x03b4a2 (0x2fad bytes) 000340 1788575216.135778 mapwrite 0x01081c thru 0x0178fd (0x70e2 bytes) 000341 1788575216.180702 punch from 0x010c23 to 0x01ad2a (0xa108 bytes) 000342 1788575216.203045 trunc from 0x01df25 to 0x025843 (0x791f bytes) 000343 1788575216.208981 mapread 0x034b27 thru 0x036ca6 (0x2180 bytes) 000345 1788575216.213024 fallocate 0x006f6d thru 0x00af2d (0x3fc1 bytes) 000346 1788575216.217643 trunc from 0x03b4a3 to 0x030cd3 (0xa7cf bytes) 000347 1788575216.237580 mapread 0x002271 thru 0x003af5 (0x1885 bytes) 000348 1788575216.256802 mapwrite 0x02810e thru 0x029910 (0x1803 bytes) 000350 1788575216.284886 mapread 0x01648e thru 0x01a212 (0x3d85 bytes) 000351 1788575216.298059 write 0x01fcee thru 0x02dcaf (0xdfc2 bytes) 000352 1788575216.305713 fallocate 0x00da35 thru 0x0139fd (0x5fc9 bytes) 000353 1788575216.315935 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30cd4): Mirror 5: 0x39a73279 (stale) Mirror 6: 0x39a73279 (stale) Mirror 8: 0x39a73279 (stale) Mirror 9: 0x39a73279 (stale) Mirror 10: 0x39a73279 (stale) Mirror 11: 0x39a73279 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30cd4): Mirror 5: 0x39a73279 (stale) Mirror 6: 0x39a73279 (stale) Mirror 8: 0x39a73279 (stale) Mirror 9: 0x39a73279 (stale) Mirror 10: 0x39a73279 (stale) Mirror 11: 0x39a73279 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30cd4: skipped 000354 1788575216.832003 read 0x02eb82 thru 0x030cd3 (0x2152 bytes) 000355 1788575216.843867 read 0x01243b thru 0x0178ef (0x54b5 bytes) 000356 1788575216.867774 trunc from 0x030cd4 to 0x020209 (0x10aca bytes) 000357 1788575216.895068 punch from 0x019689 to 0x020209 (0x6b81 bytes) 000358 1788575216.898819 trunc from 0x0139b9 to 0x020209 (0xc851 bytes) 000359 1788575216.902672 read 0x0085a3 thru 0x014684 (0xc0e2 bytes) 000360 1788575216.909002 write 0x03472d thru 0x03fd52 (0xb626 bytes) 000362 1788575216.913426 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd53): Mirror 6: 0xdc7b1f9f (stale) Mirror 8: 0xdc7b1f9f (stale) Mirror 9: 0xdc7b1f9f (stale) Mirror 10: 0xdc7b1f9f (stale) Mirror 11: 0xdc7b1f9f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd53): Mirror 6: 0xdc7b1f9f (stale) Mirror 8: 0xdc7b1f9f (stale) Mirror 9: 0xdc7b1f9f (stale) Mirror 10: 0xdc7b1f9f (stale) Mirror 11: 0xdc7b1f9f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd53: skipped 000363 1788575217.347686 mapwrite 0x02a182 thru 0x03583a (0xb6b9 bytes) 000364 1788575217.441320 trunc from 0x015108 to 0x01d186 (0x807f bytes) 000366 1788575217.464273 mapwrite 0x0016ca thru 0x0055e7 (0x3f1e bytes) 000367 1788575217.509044 read 0x024b24 thru 0x0310db (0xc5b8 bytes) 000368 1788575217.516338 punch from 0x015e06 to 0x01655e (0x759 bytes) 000369 1788575217.522336 trunc from 0x03fd53 to 0x03d0bf (0x2c93 bytes) 000370 1788575217.549649 fallocate 0x036b03 thru 0x03ffff (0x94fd bytes) 000371 1788575217.557041 write 0x034ad6 thru 0x03d427 (0x8952 bytes) 000372 1788575217.558526 trunc from 0x009fad to 0x019be4 (0xfc38 bytes) 000373 1788575217.567723 write 0x00a8cf thru 0x014540 (0x9c72 bytes) 000374 1788575217.586317 read 0x0383fc thru 0x038f4d (0xb52 bytes) 000375 1788575217.587707 trunc from 0x040000 to 0x021509 (0x1eaf6 bytes) 000376 1788575217.636133 write 0x01f2db thru 0x0224f1 (0x3217 bytes) 000377 1788575217.659116 write 0x02a4e1 thru 0x02d9a4 (0x34c4 bytes) 000378 1788575217.660420 fallocate 0x01d25a thru 0x02466b (0x7412 bytes) 000379 1788575217.665466 trunc from 0x02bff3 to 0x02d9a4 (0x19b2 bytes) 000380 1788575217.689533 mapread 0x00c789 thru 0x0171ae (0xaa26 bytes) 000382 1788575217.690988 trunc from 0x0261ec to 0x027997 (0x17ac bytes) 000383 1788575217.704590 punch from 0x02168e to 0x02400a (0x297d bytes) 000384 1788575217.734082 punch from 0x01ebbd to 0x025973 (0x6db7 bytes) 000385 1788575217.739842 punch from 0x0255c6 to 0x02d9a4 (0x83df bytes) 000386 1788575217.749306 fallocate 0x0079c5 thru 0x017127 (0xf763 bytes) 000387 1788575217.764375 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2d9a5): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d9a5): Mirror 6: 0x467d08fe (stale) Mirror 8: 0x467d08fe (stale) Mirror 9: 0x467d08fe (stale) Mirror 10: 0x467d08fe (stale) Mirror 11: 0x467d08fe (stale) CRC-32 checksum value for chunk [0, 0x2d9a5): Mirror 1: 0x818cdb13 Mirror 12: 0x818cdb13 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d9a5: skipped 000388 1788575218.711425 punch from 0x025a42 to 0x02d9a4 (0x7f63 bytes) 000389 1788575218.757938 write 0x008069 thru 0x00a90f (0x28a7 bytes) 000390 1788575218.776049 fallocate 0x035317 thru 0x03635e (0x1048 bytes) 000391 1788575218.792253 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2d9a5): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d9a5): Mirror 6: 0x467d08fe (stale) Mirror 8: 0x467d08fe (stale) Mirror 9: 0x467d08fe (stale) Mirror 10: 0x467d08fe (stale) Mirror 11: 0x467d08fe (stale) Mirror 12: 0x818cdb13 (stale) CRC-32 checksum value for chunk [0, 0x2d9a5): Mirror 1: 0x8201c5a4 Mirror 13: 0x8201c5a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d9a5: skipped 000392 1788575219.474502 read 0x0266b2 thru 0x028d0e (0x265d bytes) 000393 1788575219.491283 mapwrite 0x01c417 thru 0x020472 (0x405c bytes) 000394 1788575219.622825 MIRROR_RESYNC 8 mirrors 000395 1788575221.485515 fallocate 0x00c462 thru 0x00fdc1 (0x3960 bytes) 000396 1788575221.519176 mapread 0x00af3a thru 0x00f4de (0x45a5 bytes) 000397 1788575221.542957 trunc from 0x02d9a5 to 0x0081d3 (0x257d1 bytes) 000398 1788575221.558121 trunc from 0x0011e1 to 0x0081d3 (0x6ff3 bytes) 000399 1788575221.566501 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x81d4): Mirror 6: 0xc5878673 (stale) Mirror 8: 0xc5878673 (stale) Mirror 9: 0xc5878673 (stale) Mirror 11: 0xc5878673 (stale) Mirror 12: 0xc5878673 (stale) Mirror 13: 0xc5878673 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x81d4): Mirror 6: 0xc5878673 (stale) Mirror 8: 0xc5878673 (stale) Mirror 9: 0xc5878673 (stale) Mirror 11: 0xc5878673 (stale) Mirror 12: 0xc5878673 (stale) Mirror 13: 0xc5878673 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x81d4: skipped 000400 1788575221.957302 read 0x005232 thru 0x0081d3 (0x2fa2 bytes) 000401 1788575221.966027 trunc from 0x0081d4 to 0x031756 (0x29583 bytes) 000402 1788575221.986100 mapread 0x007719 thru 0x00b7d7 (0x40bf bytes) 000403 1788575222.003174 mapread 0x010dbe thru 0x01f83a (0xea7d bytes) 000404 1788575222.018202 read 0x02fc96 thru 0x031756 (0x1ac1 bytes) 000405 1788575222.031031 write 0x032c9c thru 0x03a8b5 (0x7c1a bytes) 000406 1788575222.032406 read 0x02bfcd thru 0x038f66 (0xcf9a bytes) 000407 1788575222.033848 write 0x01cd23 thru 0x02bb1f (0xedfd bytes) 000408 1788575222.036099 write 0x015cae thru 0x01d2c7 (0x761a bytes) 000410 1788575222.038779 mapread 0x024f1e thru 0x02fd89 (0xae6c bytes) 000411 1788575222.040046 trunc from 0x0276b9 to 0x03410e (0xca56 bytes) 000412 1788575222.057339 mapread 0x0333aa thru 0x03a8b5 (0x750c bytes) 000413 1788575222.071114 MIRROR_RESYNC 7 mirrors 000414 1788575224.320883 punch from 0x031f40 to 0x033603 (0x16c4 bytes) 000415 1788575224.367437 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a8b6): Mirror 6: 0x4e862691 (stale) Mirror 8: 0x4e862691 (stale) Mirror 9: 0x4e862691 (stale) Mirror 12: 0x4e862691 (stale) Mirror 13: 0x4e862691 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a8b6): Mirror 6: 0x4e862691 (stale) Mirror 8: 0x4e862691 (stale) Mirror 9: 0x4e862691 (stale) Mirror 12: 0x4e862691 (stale) Mirror 13: 0x4e862691 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a8b6: skipped 000416 1788575224.885578 trunc from 0x000ab4 to 0x00df0c (0xd459 bytes) 000417 1788575224.901415 fallocate 0x03b3a0 thru 0x03df9b (0x2bfc bytes) 000419 1788575224.914019 mapwrite 0x0088b1 thru 0x016abb (0xe20b bytes) 000420 1788575224.971550 fallocate 0x003e1b thru 0x0062b1 (0x2497 bytes) 000421 1788575224.982282 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3df9c): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df9c): Mirror 6: 0xd1fe5495 (stale) Mirror 8: 0xd1fe5495 (stale) Mirror 9: 0xd1fe5495 (stale) Mirror 12: 0xd1fe5495 (stale) Mirror 13: 0xd1fe5495 (stale) CRC-32 checksum value for chunk [0, 0x3df9c): Mirror 1: 0x19be6889 Mirror 14: 0x19be6889 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df9c: skipped 000422 1788575226.047978 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3df9c): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df9c): Mirror 6: 0xd1fe5495 (stale) Mirror 8: 0xd1fe5495 (stale) Mirror 9: 0xd1fe5495 (stale) Mirror 12: 0xd1fe5495 (stale) Mirror 13: 0xd1fe5495 (stale) CRC-32 checksum value for chunk [0, 0x3df9c): Mirror 1: 0x19be6889 Mirror 14: 0x19be6889 Mirror 15: 0x19be6889 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df9c: skipped 000423 1788575227.058527 fallocate 0x0008cf thru 0x006f73 (0x66a5 bytes) 000424 1788575227.170248 trunc from 0x03df9c to 0x002a86 (0x3b515 bytes) 000425 1788575227.216419 trunc from 0x000050 to 0x002a86 (0x2a37 bytes) 000426 1788575227.227443 read 0x001129 thru 0x002a86 (0x195e bytes) 000427 1788575227.232745 trunc from 0x000082 to 0x002a86 (0x2a05 bytes) 000428 1788575227.238210 read 0x00112a thru 0x002a86 (0x195d bytes) 000429 1788575227.247015 trunc from 0x0021ff to 0x002a86 (0x888 bytes) 000430 1788575227.252018 mapread 0x00228a thru 0x002a86 (0x7fd bytes) 000431 1788575227.261893 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a87): Mirror 6: 0x8e0cd72c (stale) Mirror 8: 0x8e0cd72c (stale) Mirror 9: 0x8e0cd72c (stale) Mirror 12: 0x8e0cd72c (stale) Mirror 14: 0x8e0cd72c (stale) Mirror 15: 0x8e0cd72c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a87): Mirror 6: 0x8e0cd72c (stale) Mirror 8: 0x8e0cd72c (stale) Mirror 9: 0x8e0cd72c (stale) Mirror 12: 0x8e0cd72c (stale) Mirror 14: 0x8e0cd72c (stale) Mirror 15: 0x8e0cd72c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a87: skipped 000432 1788575227.593056 punch from 0x000133 to 0x002a86 (0x2954 bytes) 000433 1788575227.598868 trunc from 0x002a87 to 0x02998e (0x26f08 bytes) 000434 1788575227.625537 read 0x02690f thru 0x02998e (0x3080 bytes) 000435 1788575227.638117 MIRROR_RESYNC 7 mirrors 000436 1788575229.005898 fallocate 0x0257ea thru 0x026f9b (0x17b2 bytes) 000437 1788575229.056689 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2998f): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2998f): Mirror 6: 0x2a37aff2 (stale) Mirror 8: 0x2a37aff2 (stale) Mirror 9: 0x2a37aff2 (stale) Mirror 12: 0x2a37aff2 (stale) Mirror 14: 0x2a37aff2 (stale) Mirror 15: 0x2a37aff2 (stale) CRC-32 checksum value for chunk [0, 0x2998f): Mirror 1: 0x2a37aff2 Mirror 16: 0x2a37aff2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2998f: skipped 000438 1788575230.355169 punch from 0x001c8a to 0x004317 (0x268e bytes) 000440 1788575230.413556 read 0x01b6a5 thru 0x01f7de (0x413a bytes) 000441 1788575230.438199 trunc from 0x01922c to 0x021da1 (0x8b76 bytes) 000442 1788575230.444909 write 0x014e48 thru 0x01adad (0x5f66 bytes) 000443 1788575230.462072 read 0x001a09 thru 0x0075d2 (0x5bca bytes) 000444 1788575230.473500 fallocate 0x03c509 thru 0x03ffff (0x3af7 bytes) 000445 1788575230.484397 trunc from 0x02998f to 0x007fe7 (0x219a7 bytes) 000446 1788575230.536459 punch from 0x006a9d to 0x007fe7 (0x154b bytes) 000447 1788575230.545089 write 0x003ecc thru 0x01142f (0xd564 bytes) 000448 1788575230.579005 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x11430): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11430): Mirror 6: 0x5df182da (stale) Mirror 8: 0x5df182da (stale) Mirror 9: 0x5df182da (stale) Mirror 12: 0x5df182da (stale) Mirror 14: 0x5df182da (stale) Mirror 15: 0x5df182da (stale) Mirror 16: 0x5df182da (stale) CRC-32 checksum value for chunk [0, 0x11430): Mirror 1: 0x7d6ea19b Mirror 17: 0x7d6ea19b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11430: skipped 000449 1788575231.773595 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x11430): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11430): Mirror 6: 0x5df182da (stale) Mirror 8: 0x5df182da (stale) Mirror 9: 0x5df182da (stale) Mirror 12: 0x5df182da (stale) Mirror 14: 0x5df182da (stale) Mirror 15: 0x5df182da (stale) Mirror 16: 0x5df182da (stale) CRC-32 checksum value for chunk [0, 0x11430): Mirror 1: 0x7d6ea19b Mirror 17: 0x7d6ea19b Mirror 18: 0x7d6ea19b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11430: skipped 000450 1788575232.869968 write 0x027264 thru 0x034bbb (0xd958 bytes) 000451 1788575232.913799 mapread 0x02b1a1 thru 0x031e19 (0x6c79 bytes) 000452 1788575232.914965 trunc from 0x0124e5 to 0x016713 (0x422f bytes) 000453 1788575232.919579 write 0x022ca2 thru 0x02e8ce (0xbc2d bytes) 000455 1788575232.925433 mapwrite 0x01e5ca thru 0x02c5ce (0xe005 bytes) 000456 1788575232.963808 trunc from 0x00cf66 to 0x01c7ad (0xf848 bytes) 000458 1788575232.971416 read 0x0218c7 thru 0x030aeb (0xf225 bytes) 000459 1788575232.972818 fallocate 0x01cbcb thru 0x027db7 (0xb1ed bytes) 000460 1788575232.982339 punch from 0x0028de to 0x00e82f (0xbf52 bytes) 000461 1788575232.990692 read 0x02d29d thru 0x03155c (0x42c0 bytes) 000462 1788575232.991476 mapread 0x0278b3 thru 0x02ba24 (0x4172 bytes) 000463 1788575232.993156 read 0x006b48 thru 0x00e181 (0x763a bytes) 000464 1788575233.032743 write 0x023c6b thru 0x028520 (0x48b6 bytes) 000465 1788575233.034464 fallocate 0x02d803 thru 0x03ca5a (0xf258 bytes) 000466 1788575233.039373 trunc from 0x005e7c to 0x006d44 (0xec9 bytes) 000467 1788575233.045110 trunc from 0x0016ec to 0x00ad66 (0x967b bytes) 000468 1788575233.050794 read 0x02b7af thru 0x033b71 (0x83c3 bytes) 000469 1788575233.051686 mapwrite 0x024e56 thru 0x02b8ac (0x6a57 bytes) 000470 1788575233.070513 trunc from 0x02bc40 to 0x02c796 (0xb57 bytes) 000471 1788575233.076266 mapread 0x0117cc thru 0x01a299 (0x8ace bytes) 000472 1788575233.082268 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3ca5b): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca5b): Mirror 6: 0xe86508b7 (stale) Mirror 8: 0xe86508b7 (stale) Mirror 9: 0xe86508b7 (stale) Mirror 12: 0xe86508b7 (stale) Mirror 14: 0xe86508b7 (stale) Mirror 15: 0xe86508b7 (stale) Mirror 16: 0xe86508b7 (stale) Mirror 17: 0x6752632d (stale) Mirror 18: 0x6752632d (stale) CRC-32 checksum value for chunk [0, 0x3ca5b): Mirror 1: 0x91194d0 Mirror 19: 0x91194d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca5b: skipped 000473 1788575234.235463 trunc from 0x03ca5b to 0x02b370 (0x116ea bytes) 000474 1788575234.337352 fallocate 0x02d782 thru 0x02e2eb (0xb6a bytes) 000475 1788575234.341991 trunc from 0x00b4aa to 0x010ced (0x5844 bytes) 000477 1788575234.348132 read 0x005805 thru 0x006e8a (0x1686 bytes) 000478 1788575234.353032 punch from 0x026c72 to 0x0277ac (0xb3b bytes) 000479 1788575234.358520 punch from 0x0193bc to 0x024927 (0xb56c bytes) 000480 1788575234.371592 MIRROR_RESYNC 11 mirrors 000482 1788575237.257252 punch from 0x01bed3 to 0x01eaa3 (0x2bd1 bytes) 000483 1788575237.313442 mapwrite 0x021610 thru 0x02358d (0x1f7e bytes) 000484 1788575237.378144 write 0x02dde7 thru 0x03b938 (0xdb52 bytes) 000485 1788575237.393300 trunc from 0x03b939 to 0x0076d0 (0x34268 bytes) 000486 1788575237.422244 read 0x003c77 thru 0x0076d0 (0x3a5a bytes) 000487 1788575237.444655 fallocate 0x01274f thru 0x0212cb (0xeb7d bytes) 000488 1788575237.451157 trunc from 0x0076d1 to 0x0034a6 (0x422a bytes) 000490 1788575237.511789 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x34a7): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a7): Mirror 6: 0xbc3b10b2 (stale) Mirror 8: 0xbc3b10b2 (stale) Mirror 9: 0xbc3b10b2 (stale) Mirror 12: 0xbc3b10b2 (stale) Mirror 14: 0xbc3b10b2 (stale) Mirror 15: 0xbc3b10b2 (stale) Mirror 16: 0xbc3b10b2 (stale) Mirror 17: 0xbc3b10b2 (stale) Mirror 18: 0xbc3b10b2 (stale) Mirror 19: 0xbc3b10b2 (stale) CRC-32 checksum value for chunk [0, 0x34a7): Mirror 1: 0xbc3b10b2 Mirror 20: 0xbc3b10b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a7: skipped 000492 1788575239.163574 fallocate 0x01f48f thru 0x024b0f (0x5681 bytes) 000493 1788575239.318656 mapwrite 0x01ac51 thru 0x01ecb2 (0x4062 bytes) 000495 1788575239.401752 trunc from 0x01de17 to 0x022fd7 (0x51c1 bytes) 000496 1788575239.408535 trunc from 0x01f9e2 to 0x024b0f (0x512e bytes) 000497 1788575239.415584 mapwrite 0x00791f thru 0x00a726 (0x2e08 bytes) 000498 1788575239.461062 trunc from 0x024b10 to 0x027263 (0x2754 bytes) 000499 1788575239.503167 punch from 0x025636 to 0x027263 (0x1c2e bytes) 000500 1788575239.510570 punch from 0x0063cc to 0x00d678 (0x72ad bytes) 000501 1788575239.518362 fallocate 0x017cc4 thru 0x01bc88 (0x3fc5 bytes) 000502 1788575239.526388 fallocate 0x0098ca thru 0x0120f4 (0x882b bytes) 000503 1788575239.535178 read 0x009be3 thru 0x0169f1 (0xce0f bytes) 000504 1788575239.569588 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x27264): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27264): Mirror 6: 0xbe33bdb7 (stale) Mirror 8: 0xbe33bdb7 (stale) Mirror 9: 0xbe33bdb7 (stale) Mirror 12: 0xbe33bdb7 (stale) Mirror 14: 0xbe33bdb7 (stale) Mirror 15: 0xbe33bdb7 (stale) Mirror 16: 0xbe33bdb7 (stale) Mirror 17: 0xbe33bdb7 (stale) Mirror 18: 0xbe33bdb7 (stale) Mirror 19: 0xbe33bdb7 (stale) Mirror 20: 0xd27b443 (stale) CRC-32 checksum value for chunk [0, 0x27264): Mirror 1: 0xefeecf8c Mirror 21: 0xefeecf8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27264: skipped 000505 1788575241.795789 mapread 0x01f79b thru 0x023833 (0x4099 bytes) 000506 1788575241.847413 write 0x00cc35 thru 0x016ac8 (0x9e94 bytes) 000507 1788575241.932345 read 0x00e121 thru 0x01a9d3 (0xc8b3 bytes) 000508 1788575241.941290 fallocate 0x0058c2 thru 0x00f495 (0x9bd4 bytes) 000509 1788575241.949466 punch from 0x0156ca to 0x01b337 (0x5c6e bytes) 000510 1788575241.970235 mapread 0x001452 thru 0x004612 (0x31c1 bytes) 000511 1788575241.983858 trunc from 0x00176f to 0x001839 (0xcb bytes) 000512 1788575241.989540 trunc from 0x027264 to 0x013229 (0x1403a bytes) 000513 1788575242.020799 write 0x0330ba thru 0x03eb89 (0xbad0 bytes) 000516 1788575242.031247 trunc from 0x03eb8a to 0x013fcc (0x2abbd bytes) 000517 1788575242.058291 mapread 0x009fe8 thru 0x00a64d (0x666 bytes) 000518 1788575242.067128 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x13fcd): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13fcd): Mirror 6: 0x751463ee (stale) Mirror 8: 0x751463ee (stale) Mirror 9: 0x751463ee (stale) Mirror 12: 0x751463ee (stale) Mirror 14: 0x751463ee (stale) Mirror 15: 0x751463ee (stale) Mirror 16: 0x751463ee (stale) Mirror 17: 0x751463ee (stale) Mirror 18: 0x751463ee (stale) Mirror 19: 0x751463ee (stale) Mirror 20: 0x751463ee (stale) Mirror 21: 0x751463ee (stale) CRC-32 checksum value for chunk [0, 0x13fcd): Mirror 1: 0xc1c3978e Mirror 22: 0xc1c3978e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13fcd: skipped 000519 1788575243.304925 trunc from 0x00d4b7 to 0x013fcc (0x6b16 bytes) 000520 1788575243.382945 trunc from 0x013fcd to 0x025137 (0x1116b bytes) 000521 1788575243.491706 trunc from 0x021e3f to 0x025137 (0x32f9 bytes) 000522 1788575243.515358 MIRROR_RESYNC 14 mirrors 000523 1788575247.264989 write 0x0393b8 thru 0x03ffff (0x6c48 bytes) 000524 1788575247.314075 punch from 0x00d4c3 to 0x011a70 (0x45ae bytes) 000525 1788575247.324881 mapwrite 0x008a12 thru 0x0130ed (0xa6dc bytes) 000526 1788575247.372687 trunc from 0x00ffe9 to 0x017854 (0x786c bytes) 000527 1788575247.385737 read 0x009b65 thru 0x00f08a (0x5526 bytes) 000528 1788575247.390125 read 0x035662 thru 0x03ffff (0xa99e bytes) 000530 1788575247.404856 fallocate 0x0275f1 thru 0x0291ab (0x1bbb bytes) 000531 1788575247.409825 mapwrite 0x0041e2 thru 0x008493 (0x42b2 bytes) 000532 1788575247.424203 write 0x0019ce thru 0x00f670 (0xdca3 bytes) 000533 1788575247.426533 read 0x027414 thru 0x02b842 (0x442f bytes) 000534 1788575247.427362 punch from 0x001c28 to 0x009b8e (0x7f67 bytes) 000535 1788575247.471042 trunc from 0x040000 to 0x00aa15 (0x355ea bytes) 000536 1788575247.508159 fallocate 0x01fb62 thru 0x02eadf (0xef7e bytes) 000537 1788575247.521983 trunc from 0x00a231 to 0x00aa15 (0x7e5 bytes) 000538 1788575247.532246 trunc from 0x004907 to 0x00aa15 (0x610f bytes) 000539 1788575247.540980 trunc from 0x00aa16 to 0x030143 (0x2572e bytes) 000540 1788575247.573869 punch from 0x027ed9 to 0x030143 (0x826b bytes) 000541 1788575247.600716 punch from 0x018300 to 0x024790 (0xc491 bytes) 000542 1788575247.611047 punch from 0x02b325 to 0x030143 (0x4e1f bytes) 000543 1788575247.620693 trunc from 0x030144 to 0x0254a0 (0xaca3 bytes) 000544 1788575247.673080 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x254a1): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x254a1): Mirror 6: 0xe7017c99 (stale) Mirror 8: 0xe7017c99 (stale) Mirror 9: 0xe7017c99 (stale) Mirror 12: 0xe7017c99 (stale) Mirror 14: 0xe7017c99 (stale) Mirror 15: 0xe7017c99 (stale) Mirror 16: 0xe7017c99 (stale) Mirror 17: 0xe7017c99 (stale) Mirror 18: 0xe7017c99 (stale) Mirror 19: 0xe7017c99 (stale) Mirror 20: 0xe7017c99 (stale) Mirror 21: 0xe7017c99 (stale) Mirror 22: 0xe7017c99 (stale) CRC-32 checksum value for chunk [0, 0x254a1): Mirror 1: 0x7fef6ba9 Mirror 23: 0x7fef6ba9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x254a1: skipped 000545 1788575249.198873 trunc from 0x0254a1 to 0x004e1a (0x20686 bytes) 000546 1788575249.280700 write 0x006aa5 thru 0x0162fd (0xf859 bytes) 000547 1788575249.282549 mapread 0x009306 thru 0x0162fd (0xcff8 bytes) 000548 1788575249.292203 MIRROR_RESYNC 15 mirrors 000549 1788575252.122372 trunc from 0x007dcd to 0x0162fd (0xe531 bytes) 000550 1788575252.182518 mapread 0x00c9fe thru 0x0162fd (0x9900 bytes) 000551 1788575252.201058 punch from 0x002984 to 0x007568 (0x4be5 bytes) 000552 1788575252.206883 trunc from 0x003dcd to 0x006b57 (0x2d8b bytes) 000553 1788575252.211061 punch from 0x01337b to 0x0162fd (0x2f83 bytes) 000554 1788575252.215599 mapread 0x00c4a2 thru 0x0162fd (0x9e5c bytes) 000555 1788575252.236418 mapwrite 0x014920 thru 0x014c83 (0x364 bytes) 000556 1788575252.265564 MIRROR_SPLIT mirror 6 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x162fe): Mirror 6: 0xd8fc73f1 (stale) Mirror 8: 0xd8fc73f1 (stale) Mirror 9: 0xd8fc73f1 (stale) Mirror 12: 0xd8fc73f1 (stale) Mirror 14: 0xd8fc73f1 (stale) Mirror 16: 0xd8fc73f1 (stale) Mirror 17: 0xd8fc73f1 (stale) Mirror 18: 0xd8fc73f1 (stale) Mirror 19: 0xd8fc73f1 (stale) Mirror 20: 0xd8fc73f1 (stale) Mirror 21: 0xd8fc73f1 (stale) Mirror 22: 0xd8fc73f1 (stale) Mirror 23: 0xd8fc73f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x162fe): Mirror 6: 0xd8fc73f1 (stale) Mirror 8: 0xd8fc73f1 (stale) Mirror 9: 0xd8fc73f1 (stale) Mirror 12: 0xd8fc73f1 (stale) Mirror 14: 0xd8fc73f1 (stale) Mirror 16: 0xd8fc73f1 (stale) Mirror 17: 0xd8fc73f1 (stale) Mirror 18: 0xd8fc73f1 (stale) Mirror 19: 0xd8fc73f1 (stale) Mirror 20: 0xd8fc73f1 (stale) Mirror 21: 0xd8fc73f1 (stale) Mirror 22: 0xd8fc73f1 (stale) Mirror 23: 0xd8fc73f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x162fe: skipped 000557 1788575253.139430 punch from 0x007355 to 0x00ebc7 (0x7873 bytes) 000559 1788575253.150435 trunc from 0x0162fe to 0x03c0b5 (0x25db8 bytes) 000560 1788575253.189083 mapread 0x013f57 thru 0x021948 (0xd9f2 bytes) 000561 1788575253.268832 punch from 0x024dde to 0x02f24e (0xa471 bytes) 000562 1788575253.276038 write 0x036479 thru 0x03ffff (0x9b87 bytes) 000563 1788575253.285488 mapwrite 0x025ba9 thru 0x034e9b (0xf2f3 bytes) 000564 1788575253.356466 fallocate 0x0121a1 thru 0x01766f (0x54cf bytes) 000565 1788575253.377834 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb5db1087 (stale) Mirror 8: 0xb5db1087 (stale) Mirror 9: 0xb5db1087 (stale) Mirror 12: 0xb5db1087 (stale) Mirror 14: 0xb5db1087 (stale) Mirror 16: 0xb5db1087 (stale) Mirror 17: 0xb5db1087 (stale) Mirror 18: 0xb5db1087 (stale) Mirror 19: 0xb5db1087 (stale) Mirror 20: 0xb5db1087 (stale) Mirror 21: 0xb5db1087 (stale) Mirror 22: 0xb5db1087 (stale) Mirror 23: 0xb5db1087 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3b9a1ce5 Mirror 24: 0x3b9a1ce5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000566 1788575255.205903 trunc from 0x040000 to 0x039d9b (0x6264 bytes) 000567 1788575255.325035 punch from 0x017fe7 to 0x01aa05 (0x2a1f bytes) 000568 1788575255.334974 fallocate 0x001c55 thru 0x007a0b (0x5db7 bytes) 000569 1788575255.347495 write 0x01f93d thru 0x02c7f2 (0xceb6 bytes) 000570 1788575255.364103 mapread 0x011f65 thru 0x0201f9 (0xe295 bytes) 000571 1788575255.426243 write 0x0306ab thru 0x033ff3 (0x3949 bytes) 000572 1788575255.438257 fallocate 0x012a45 thru 0x015f32 (0x34ee bytes) 000573 1788575255.453218 write 0x01aada thru 0x025a45 (0xaf6c bytes) 000574 1788575255.454548 write 0x0357ed thru 0x0378bb (0x20cf bytes) 000575 1788575255.479115 fallocate 0x01f67c thru 0x029af8 (0xa47d bytes) 000576 1788575255.488282 write 0x02e02c thru 0x02ee10 (0xde5 bytes) 000577 1788575255.500595 trunc from 0x0029d8 to 0x01215f (0xf788 bytes) 000578 1788575255.516383 mapwrite 0x018725 thru 0x020020 (0x78fc bytes) 000579 1788575255.550916 punch from 0x026e1e to 0x029c06 (0x2de9 bytes) 000580 1788575255.559045 read 0x00b0e4 thru 0x01aa46 (0xf963 bytes) 000581 1788575255.571927 mapwrite 0x008d52 thru 0x018197 (0xf446 bytes) 000582 1788575255.619074 fallocate 0x02d069 thru 0x036d22 (0x9cba bytes) 000583 1788575255.624578 trunc from 0x02d583 to 0x03781b (0xa299 bytes) 000584 1788575255.631652 write 0x01c2c0 thru 0x02b224 (0xef65 bytes) 000585 1788575255.637999 punch from 0x020685 to 0x0222dd (0x1c59 bytes) 000586 1788575255.682899 trunc from 0x009442 to 0x017cba (0xe879 bytes) 000587 1788575255.687099 mapwrite 0x01bad6 thru 0x023483 (0x79ae bytes) 000588 1788575255.731539 mapread 0x00865d thru 0x016aea (0xe48e bytes) 000589 1788575255.786415 mapread 0x01dde3 thru 0x02be83 (0xe0a1 bytes) 000590 1788575255.793590 punch from 0x016bd6 to 0x01c375 (0x57a0 bytes) 000591 1788575255.806587 mapwrite 0x029214 thru 0x03406f (0xae5c bytes) 000592 1788575255.869275 punch from 0x020a29 to 0x021f3d (0x1515 bytes) 000593 1788575255.882592 fallocate 0x013d45 thru 0x01c466 (0x8722 bytes) 000594 1788575255.892112 mapwrite 0x01b109 thru 0x01ed08 (0x3c00 bytes) 000595 1788575255.931922 trunc from 0x02917f to 0x02fc95 (0x6b17 bytes) 000597 1788575255.941886 read 0x017a53 thru 0x025ca1 (0xe24f bytes) 000598 1788575255.948950 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x39d9c): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d9c): Mirror 6: 0xdca20f39 (stale) Mirror 8: 0xdca20f39 (stale) Mirror 9: 0xdca20f39 (stale) Mirror 12: 0xdca20f39 (stale) Mirror 14: 0xdca20f39 (stale) Mirror 16: 0xdca20f39 (stale) Mirror 17: 0xdca20f39 (stale) Mirror 18: 0xdca20f39 (stale) Mirror 19: 0xdca20f39 (stale) Mirror 20: 0xdca20f39 (stale) Mirror 21: 0xdca20f39 (stale) Mirror 22: 0xdca20f39 (stale) Mirror 23: 0xdca20f39 (stale) Mirror 24: 0xd577e069 (stale) CRC-32 checksum value for chunk [0, 0x39d9c): Mirror 1: 0x73b6eb4f Mirror 25: 0x73b6eb4f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d9c: skipped 000599 1788575257.970191 fallocate 0x005b8e thru 0x00994d (0x3dc0 bytes) 000600 1788575258.104626 mapwrite 0x00c7cf thru 0x011031 (0x4863 bytes) 000602 1788575258.183073 MIRROR_RESYNC 16 mirrors 000603 1788575266.335447 mapread 0x026125 thru 0x02764d (0x1529 bytes) 000604 1788575266.342669 mapwrite 0x02e67a thru 0x037513 (0x8e9a bytes) 000605 1788575266.457549 trunc from 0x039d9c to 0x01e27d (0x1bb1e bytes) 000606 1788575266.508575 read 0x0109d9 thru 0x01e27d (0xd8a5 bytes) 000608 1788575266.521441 trunc from 0x011812 to 0x0141ba (0x29a9 bytes) 000609 1788575266.537321 fallocate 0x015196 thru 0x0228ac (0xd717 bytes) 000610 1788575266.547368 fallocate 0x0311a5 thru 0x032283 (0x10df bytes) 000611 1788575266.560416 write 0x001add thru 0x00969e (0x7bc2 bytes) 000612 1788575266.570883 trunc from 0x001a97 to 0x00b135 (0x969f bytes) 000613 1788575266.595511 fallocate 0x023854 thru 0x02e73d (0xaeea bytes) 000614 1788575266.600161 fallocate 0x024c58 thru 0x031057 (0xc400 bytes) 000615 1788575266.610925 MIRROR_SPLIT mirror 9 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32284): Mirror 6: 0xf1e64986 (stale) Mirror 8: 0xf1e64986 (stale) Mirror 9: 0xf1e64986 (stale) Mirror 12: 0xf1e64986 (stale) Mirror 14: 0xf1e64986 (stale) Mirror 16: 0xf1e64986 (stale) Mirror 17: 0xf1e64986 (stale) Mirror 18: 0xf1e64986 (stale) Mirror 20: 0xf1e64986 (stale) Mirror 21: 0xf1e64986 (stale) Mirror 22: 0xf1e64986 (stale) Mirror 23: 0xf1e64986 (stale) Mirror 24: 0xf1e64986 (stale) Mirror 25: 0xf1e64986 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32284): Mirror 6: 0xf1e64986 (stale) Mirror 8: 0xf1e64986 (stale) Mirror 9: 0xf1e64986 (stale) Mirror 12: 0xf1e64986 (stale) Mirror 14: 0xf1e64986 (stale) Mirror 16: 0xf1e64986 (stale) Mirror 17: 0xf1e64986 (stale) Mirror 18: 0xf1e64986 (stale) Mirror 20: 0xf1e64986 (stale) Mirror 21: 0xf1e64986 (stale) Mirror 22: 0xf1e64986 (stale) Mirror 23: 0xf1e64986 (stale) Mirror 24: 0xf1e64986 (stale) Mirror 25: 0xf1e64986 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32284: skipped 000616 1788575267.743812 write 0x038ff5 thru 0x03ffff (0x700b bytes) 000618 1788575267.764253 mapread 0x02d745 thru 0x03bfd9 (0xe895 bytes) 000619 1788575267.830244 punch from 0x015af2 to 0x01df6a (0x8479 bytes) 000620 1788575267.858117 MIRROR_SPLIT mirror 11 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x12f50cdb (stale) Mirror 8: 0x12f50cdb (stale) Mirror 9: 0x12f50cdb (stale) Mirror 12: 0x12f50cdb (stale) Mirror 14: 0x12f50cdb (stale) Mirror 16: 0x12f50cdb (stale) Mirror 17: 0x12f50cdb (stale) Mirror 18: 0x12f50cdb (stale) Mirror 20: 0x12f50cdb (stale) Mirror 21: 0x12f50cdb (stale) Mirror 23: 0x12f50cdb (stale) Mirror 24: 0x12f50cdb (stale) Mirror 25: 0x12f50cdb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x12f50cdb (stale) Mirror 8: 0x12f50cdb (stale) Mirror 9: 0x12f50cdb (stale) Mirror 12: 0x12f50cdb (stale) Mirror 14: 0x12f50cdb (stale) Mirror 16: 0x12f50cdb (stale) Mirror 17: 0x12f50cdb (stale) Mirror 18: 0x12f50cdb (stale) Mirror 20: 0x12f50cdb (stale) Mirror 21: 0x12f50cdb (stale) Mirror 23: 0x12f50cdb (stale) Mirror 24: 0x12f50cdb (stale) Mirror 25: 0x12f50cdb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000621 1788575269.221788 MIRROR_SPLIT mirror 7 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x12f50cdb (stale) Mirror 8: 0x12f50cdb (stale) Mirror 9: 0x12f50cdb (stale) Mirror 12: 0x12f50cdb (stale) Mirror 14: 0x12f50cdb (stale) Mirror 16: 0x12f50cdb (stale) Mirror 18: 0x12f50cdb (stale) Mirror 20: 0x12f50cdb (stale) Mirror 21: 0x12f50cdb (stale) Mirror 23: 0x12f50cdb (stale) Mirror 24: 0x12f50cdb (stale) Mirror 25: 0x12f50cdb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x12f50cdb (stale) Mirror 8: 0x12f50cdb (stale) Mirror 9: 0x12f50cdb (stale) Mirror 12: 0x12f50cdb (stale) Mirror 14: 0x12f50cdb (stale) Mirror 16: 0x12f50cdb (stale) Mirror 18: 0x12f50cdb (stale) Mirror 20: 0x12f50cdb (stale) Mirror 21: 0x12f50cdb (stale) Mirror 23: 0x12f50cdb (stale) Mirror 24: 0x12f50cdb (stale) Mirror 25: 0x12f50cdb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000622 1788575270.232888 write 0x02e876 thru 0x034e82 (0x660d bytes) 000623 1788575270.272206 write 0x01841d thru 0x022b4f (0xa733 bytes) 000624 1788575270.284431 write 0x036836 thru 0x03ffff (0x97ca bytes) 000625 1788575270.294550 trunc from 0x040000 to 0x00127a (0x3ed85 bytes) 000626 1788575270.352853 punch from 0x000725 to 0x00127a (0xb56 bytes) 000627 1788575270.370360 MIRROR_RESYNC 13 mirrors 000628 1788575271.963024 read 0x0009af thru 0x00127a (0x8cc bytes) 000629 1788575271.969691 mapread 0x000b85 thru 0x00127a (0x6f6 bytes) 000630 1788575271.970236 fallocate 0x03550f thru 0x03ffff (0xaaf1 bytes) 000631 1788575272.067428 punch from 0x016f16 to 0x0242d3 (0xd3be bytes) 000632 1788575272.074901 fallocate 0x033a1a thru 0x03a6e6 (0x6ccd bytes) 000633 1788575272.087679 write 0x01c18d thru 0x02af3d (0xedb1 bytes) 000634 1788575272.104338 fallocate 0x01322e thru 0x01c314 (0x90e7 bytes) 000635 1788575272.116152 mapread 0x00aed1 thru 0x00fb59 (0x4c89 bytes) 000636 1788575272.142738 mapread 0x00f7da thru 0x010976 (0x119d bytes) 000637 1788575272.144400 write 0x009c11 thru 0x00fe71 (0x6261 bytes) 000638 1788575272.151158 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 16: 0xe20eea22 (stale) Mirror 18: 0xe20eea22 (stale) Mirror 20: 0xe20eea22 (stale) Mirror 21: 0xe20eea22 (stale) Mirror 23: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 16: 0xe20eea22 (stale) Mirror 18: 0xe20eea22 (stale) Mirror 20: 0xe20eea22 (stale) Mirror 21: 0xe20eea22 (stale) Mirror 23: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000639 1788575272.642521 trunc from 0x040000 to 0x02a5bf (0x15a40 bytes) 000640 1788575272.658205 trunc from 0x02a5c0 to 0x02250b (0x80b4 bytes) 000641 1788575272.667235 mapread 0x01eb03 thru 0x02250b (0x3a09 bytes) 000642 1788575272.683800 trunc from 0x02250c to 0x019cbb (0x8850 bytes) 000643 1788575272.712140 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19cbc): Mirror 6: 0x96a957e5 (stale) Mirror 8: 0x96a957e5 (stale) Mirror 9: 0x96a957e5 (stale) Mirror 14: 0x96a957e5 (stale) Mirror 16: 0x96a957e5 (stale) Mirror 18: 0x96a957e5 (stale) Mirror 20: 0x96a957e5 (stale) Mirror 21: 0x96a957e5 (stale) Mirror 23: 0x96a957e5 (stale) Mirror 25: 0x96a957e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19cbc): Mirror 6: 0x96a957e5 (stale) Mirror 8: 0x96a957e5 (stale) Mirror 9: 0x96a957e5 (stale) Mirror 14: 0x96a957e5 (stale) Mirror 16: 0x96a957e5 (stale) Mirror 18: 0x96a957e5 (stale) Mirror 20: 0x96a957e5 (stale) Mirror 21: 0x96a957e5 (stale) Mirror 23: 0x96a957e5 (stale) Mirror 25: 0x96a957e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19cbc: skipped 000644 1788575273.201521 punch from 0x00b24f to 0x00f385 (0x4137 bytes) 000645 1788575273.209200 MIRROR_RESYNC 11 mirrors 000646 1788575276.011087 mapread 0x018dcb thru 0x019cbb (0xef1 bytes) 000648 1788575276.031342 write 0x006e98 thru 0x00d4fe (0x6667 bytes) 000649 1788575276.099644 read 0x017105 thru 0x01892b (0x1827 bytes) 000650 1788575276.106477 read 0x006d4e thru 0x00ae29 (0x40dc bytes) 000651 1788575276.107091 punch from 0x00b701 to 0x019cbb (0xe5bb bytes) 000652 1788575276.148432 punch from 0x00d704 to 0x00ea04 (0x1301 bytes) 000653 1788575276.159152 mapwrite 0x0360d3 thru 0x03fd13 (0x9c41 bytes) 000654 1788575276.327351 trunc from 0x03fd14 to 0x000ef0 (0x3ee23 bytes) 000655 1788575276.353998 mapread 0x000211 thru 0x000ef0 (0xce0 bytes) 000656 1788575276.357846 read 0x00084c thru 0x000ef0 (0x6a5 bytes) 000657 1788575276.358952 write 0x0364e6 thru 0x03ffff (0x9b1a bytes) 000658 1788575276.362100 punch from 0x0251ca to 0x02ef83 (0x9dba bytes) 000660 1788575276.374823 mapwrite 0x03f19d thru 0x03ffff (0xe63 bytes) 000661 1788575276.395722 punch from 0x02c960 to 0x02d647 (0xce8 bytes) 000662 1788575276.402499 trunc from 0x02fecb to 0x0343a5 (0x44db bytes) 000663 1788575276.412759 trunc from 0x03ea35 to 0x03ffff (0x15cb bytes) 000664 1788575276.421410 read 0x022808 thru 0x0298ec (0x70e5 bytes) 000665 1788575276.453362 fallocate 0x014fd4 thru 0x019dad (0x4dda bytes) 000667 1788575276.474051 write 0x02e6da thru 0x02f286 (0xbad bytes) 000668 1788575276.480677 write 0x0059eb thru 0x00bcfc (0x6312 bytes) 000669 1788575276.482678 mapread 0x023c23 thru 0x02aba9 (0x6f87 bytes) 000670 1788575276.483927 mapwrite 0x03cedc thru 0x03ffff (0x3124 bytes) 000671 1788575276.522074 trunc from 0x040000 to 0x03688a (0x9775 bytes) 000672 1788575276.553071 read 0x010db4 thru 0x01c300 (0xb54d bytes) 000673 1788575276.566435 MIRROR_RESYNC 11 mirrors 000674 1788575280.449752 write 0x018df5 thru 0x019f76 (0x1182 bytes) 000675 1788575280.485454 trunc from 0x03688b to 0x02f698 (0x71f2 bytes) 000676 1788575280.511832 write 0x0074f1 thru 0x00d97a (0x648a bytes) 000677 1788575280.520702 fallocate 0x02ccf3 thru 0x038f91 (0xc29f bytes) 000679 1788575280.525775 trunc from 0x038f92 to 0x025610 (0x13981 bytes) 000680 1788575280.541015 mapread 0x01bb2e thru 0x02229e (0x6771 bytes) 000681 1788575280.562928 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25611): Mirror 6: 0xb8ff330f (stale) Mirror 8: 0xb8ff330f (stale) Mirror 9: 0xb8ff330f (stale) Mirror 14: 0xb8ff330f (stale) Mirror 18: 0xb8ff330f (stale) Mirror 20: 0xb8ff330f (stale) Mirror 21: 0xb8ff330f (stale) Mirror 23: 0xb8ff330f (stale) Mirror 25: 0xb8ff330f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25611): Mirror 6: 0xb8ff330f (stale) Mirror 8: 0xb8ff330f (stale) Mirror 9: 0xb8ff330f (stale) Mirror 14: 0xb8ff330f (stale) Mirror 18: 0xb8ff330f (stale) Mirror 20: 0xb8ff330f (stale) Mirror 21: 0xb8ff330f (stale) Mirror 23: 0xb8ff330f (stale) Mirror 25: 0xb8ff330f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25611: skipped 000682 1788575281.175484 trunc from 0x0254c9 to 0x025610 (0x148 bytes) 000683 1788575281.180326 mapwrite 0x008a80 thru 0x0187e5 (0xfd66 bytes) 000684 1788575281.231923 trunc from 0x025611 to 0x0270ad (0x1a9d bytes) 000685 1788575281.267428 write 0x0018b7 thru 0x00317e (0x18c8 bytes) 000686 1788575281.274653 read 0x00287d thru 0x005637 (0x2dbb bytes) 000687 1788575281.275585 write 0x03b7fc thru 0x03ffff (0x4804 bytes) 000688 1788575281.277581 fallocate 0x00a466 thru 0x00ad11 (0x8ac bytes) 000689 1788575281.282975 fallocate 0x02ede2 thru 0x03d3a2 (0xe5c1 bytes) 000690 1788575281.288084 read 0x00211e thru 0x00f9ad (0xd890 bytes) 000691 1788575281.289602 fallocate 0x02f36d thru 0x03b121 (0xbdb5 bytes) 000692 1788575281.294595 fallocate 0x026852 thru 0x03680a (0xffb9 bytes) 000693 1788575281.302897 punch from 0x01761e to 0x01ca2c (0x540f bytes) 000694 1788575281.310263 read 0x02c76b thru 0x03b144 (0xe9da bytes) 000696 1788575281.328253 punch from 0x028589 to 0x028db0 (0x828 bytes) 000697 1788575281.333580 mapread 0x03adc8 thru 0x03ffff (0x5238 bytes) 000698 1788575281.335971 punch from 0x030a36 to 0x0388d5 (0x7ea0 bytes) 000699 1788575281.349190 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x26db08cc (stale) Mirror 9: 0x26db08cc (stale) Mirror 14: 0x26db08cc (stale) Mirror 18: 0x26db08cc (stale) Mirror 20: 0x26db08cc (stale) Mirror 21: 0x26db08cc (stale) Mirror 23: 0x26db08cc (stale) Mirror 25: 0x26db08cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x26db08cc (stale) Mirror 9: 0x26db08cc (stale) Mirror 14: 0x26db08cc (stale) Mirror 18: 0x26db08cc (stale) Mirror 20: 0x26db08cc (stale) Mirror 21: 0x26db08cc (stale) Mirror 23: 0x26db08cc (stale) Mirror 25: 0x26db08cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000700 1788575282.120866 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x26db08cc (stale) Mirror 14: 0x26db08cc (stale) Mirror 18: 0x26db08cc (stale) Mirror 20: 0x26db08cc (stale) Mirror 21: 0x26db08cc (stale) Mirror 23: 0x26db08cc (stale) Mirror 25: 0x26db08cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x26db08cc (stale) Mirror 14: 0x26db08cc (stale) Mirror 18: 0x26db08cc (stale) Mirror 20: 0x26db08cc (stale) Mirror 21: 0x26db08cc (stale) Mirror 23: 0x26db08cc (stale) Mirror 25: 0x26db08cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000701 1788575282.680346 punch from 0x03ccc2 to 0x03ffff (0x333e bytes) 000702 1788575282.689048 mapwrite 0x01b41a thru 0x028bad (0xd794 bytes) 000703 1788575282.744271 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x26db08cc (stale) Mirror 18: 0x26db08cc (stale) Mirror 20: 0x26db08cc (stale) Mirror 21: 0x26db08cc (stale) Mirror 23: 0x26db08cc (stale) Mirror 25: 0x26db08cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x26db08cc (stale) Mirror 18: 0x26db08cc (stale) Mirror 20: 0x26db08cc (stale) Mirror 21: 0x26db08cc (stale) Mirror 23: 0x26db08cc (stale) Mirror 25: 0x26db08cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000704 1788575283.245110 fallocate 0x028330 thru 0x03753d (0xf20e bytes) 000705 1788575283.251509 mapread 0x012573 thru 0x019e42 (0x78d0 bytes) 000706 1788575283.283017 trunc from 0x040000 to 0x0032e6 (0x3cd19 bytes) 000707 1788575283.322648 mapwrite 0x026ec6 thru 0x0301e7 (0x9322 bytes) 000708 1788575283.456779 read 0x027c16 thru 0x0281ae (0x599 bytes) 000709 1788575283.457601 read 0x013013 thru 0x020320 (0xd30e bytes) 000710 1788575283.484554 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x301e8): Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301e8): Mirror 14: 0x947268c (stale) Mirror 18: 0x947268c (stale) Mirror 20: 0x947268c (stale) Mirror 21: 0x947268c (stale) Mirror 23: 0x947268c (stale) Mirror 25: 0x947268c (stale) CRC-32 checksum value for chunk [0, 0x301e8): Mirror 1: 0x28c09984 Mirror 26: 0x28c09984 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301e8: skipped 000711 1788575284.568023 trunc from 0x001ff1 to 0x00ac7b (0x8c8b bytes) 000712 1788575284.609560 trunc from 0x024fa3 to 0x02f6f3 (0xa751 bytes) 000713 1788575284.616775 trunc from 0x0301e8 to 0x02c1c3 (0x4024 bytes) 000714 1788575284.643433 punch from 0x012320 to 0x016e4d (0x4b2e bytes) 000715 1788575284.648834 write 0x003d39 thru 0x012751 (0xea19 bytes) 000716 1788575284.661289 trunc from 0x02c1c4 to 0x032a75 (0x68b2 bytes) 000718 1788575284.681816 mapread 0x0233c8 thru 0x030ad4 (0xd70d bytes) 000719 1788575284.713400 write 0x025867 thru 0x032413 (0xcbad bytes) 000720 1788575284.720152 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x32a76): Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32a76): Mirror 14: 0xd2f1885f (stale) Mirror 18: 0xd2f1885f (stale) Mirror 20: 0xd2f1885f (stale) Mirror 21: 0xd2f1885f (stale) Mirror 23: 0xd2f1885f (stale) Mirror 25: 0xd2f1885f (stale) Mirror 26: 0x6a0f4395 (stale) CRC-32 checksum value for chunk [0, 0x32a76): Mirror 1: 0xdc1bcc68 Mirror 27: 0xdc1bcc68 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32a76: skipped 000721 1788575285.884974 read 0x02c299 thru 0x02e5f2 (0x235a bytes) 000722 1788575285.922730 trunc from 0x032a76 to 0x01703f (0x1ba36 bytes) 000723 1788575286.014506 mapread 0x009f30 thru 0x00c519 (0x25ea bytes) 000724 1788575286.044261 MIRROR_RESYNC 9 mirrors 000725 1788575287.908330 punch from 0x0039da to 0x00598d (0x1fb4 bytes) 000727 1788575287.945557 mapwrite 0x00d1f4 thru 0x01adfa (0xdc07 bytes) 000728 1788575288.007535 write 0x02163e thru 0x024579 (0x2f3c bytes) 000729 1788575288.008737 mapread 0x0238ac thru 0x02406f (0x7c4 bytes) 000731 1788575288.010982 fallocate 0x0193a5 thru 0x01ba68 (0x26c4 bytes) 000732 1788575288.016879 trunc from 0x02457a to 0x004193 (0x203e6 bytes) 000733 1788575288.029506 mapread 0x003ab5 thru 0x004193 (0x6df bytes) 000734 1788575288.037976 punch from 0x00018b to 0x004193 (0x4009 bytes) 000735 1788575288.043191 punch from 0x001707 to 0x004193 (0x2a8d bytes) 000736 1788575288.047531 write 0x0336db thru 0x03ffff (0xc925 bytes) 000737 1788575288.050551 punch from 0x00c7e5 to 0x01b006 (0xe822 bytes) 000739 1788575288.055993 trunc from 0x040000 to 0x03944a (0x6bb5 bytes) 000740 1788575288.077087 trunc from 0x033af4 to 0x0390d1 (0x55de bytes) 000742 1788575288.090524 write 0x02b0b5 thru 0x039b23 (0xea6f bytes) 000743 1788575288.100179 punch from 0x023587 to 0x0304cb (0xcf45 bytes) 000744 1788575288.113492 write 0x02257a thru 0x027f73 (0x59fa bytes) 000745 1788575288.123289 fallocate 0x001f79 thru 0x002a42 (0xaca bytes) 000746 1788575288.128211 punch from 0x00d589 to 0x012405 (0x4e7d bytes) 000747 1788575288.132535 mapwrite 0x03455e thru 0x03ffff (0xbaa2 bytes) 000748 1788575288.172067 write 0x0095a3 thru 0x010f53 (0x79b1 bytes) 000749 1788575288.185353 trunc from 0x036614 to 0x03f492 (0x8e7f bytes) 000750 1788575288.191158 read 0x01140b thru 0x0167c9 (0x53bf bytes) 000751 1788575288.200549 punch from 0x01e408 to 0x025423 (0x701c bytes) 000752 1788575288.206410 fallocate 0x02a9a9 thru 0x031f9b (0x75f3 bytes) 000753 1788575288.211038 fallocate 0x0048d3 thru 0x00fc2b (0xb359 bytes) 000754 1788575288.215750 mapwrite 0x00860c thru 0x00a0d3 (0x1ac8 bytes) 000755 1788575288.239300 punch from 0x00861b to 0x00de17 (0x57fd bytes) 000756 1788575288.244344 mapwrite 0x01a184 thru 0x01eb3f (0x49bc bytes) 000757 1788575288.268163 punch from 0x02dd03 to 0x02e3cf (0x6cd bytes) 000758 1788575288.273757 read 0x01d192 thru 0x023f9e (0x6e0d bytes) 000759 1788575288.286196 mapwrite 0x013356 thru 0x015b67 (0x2812 bytes) 000760 1788575288.303054 write 0x0068ac thru 0x013a98 (0xd1ed bytes) 000762 1788575288.306525 write 0x0345cd thru 0x03ffff (0xba33 bytes) 000763 1788575288.308693 punch from 0x020aa4 to 0x027d6c (0x72c9 bytes) 000764 1788575288.314975 mapread 0x00adfb thru 0x01a7fb (0xfa01 bytes) 000765 1788575288.316718 write 0x03ce0a thru 0x03ffff (0x31f6 bytes) 000766 1788575288.318318 fallocate 0x038517 thru 0x03ffff (0x7ae9 bytes) 000767 1788575288.322948 mapwrite 0x02d68b thru 0x02e8ca (0x1240 bytes) 000769 1788575288.342363 trunc from 0x040000 to 0x0248bd (0x1b742 bytes) 000770 1788575288.362073 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x248be): Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x248be): Mirror 14: 0xf80d40b9 (stale) Mirror 18: 0xf80d40b9 (stale) Mirror 20: 0xf80d40b9 (stale) Mirror 21: 0xf80d40b9 (stale) Mirror 23: 0xf80d40b9 (stale) Mirror 25: 0xf80d40b9 (stale) Mirror 26: 0xf80d40b9 (stale) Mirror 27: 0xf80d40b9 (stale) CRC-32 checksum value for chunk [0, 0x248be): Mirror 1: 0xb01afdbb Mirror 28: 0xb01afdbb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x248be: skipped 000771 1788575289.192182 read 0x02194f thru 0x0248bd (0x2f6f bytes) 000772 1788575289.206364 punch from 0x00ea6b to 0x01413d (0x56d3 bytes) 000773 1788575289.258543 read 0x01162b thru 0x01711e (0x5af4 bytes) 000774 1788575289.277788 write 0x032101 thru 0x03365e (0x155e bytes) 000776 1788575289.301259 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3365f): Mirror 18: 0x7d68ce59 (stale) Mirror 20: 0x7d68ce59 (stale) Mirror 21: 0x7d68ce59 (stale) Mirror 23: 0x7d68ce59 (stale) Mirror 25: 0x7d68ce59 (stale) Mirror 26: 0x7d68ce59 (stale) Mirror 27: 0x7d68ce59 (stale) Mirror 28: 0xc782c287 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3365f): Mirror 18: 0x7d68ce59 (stale) Mirror 20: 0x7d68ce59 (stale) Mirror 21: 0x7d68ce59 (stale) Mirror 23: 0x7d68ce59 (stale) Mirror 25: 0x7d68ce59 (stale) Mirror 26: 0x7d68ce59 (stale) Mirror 27: 0x7d68ce59 (stale) Mirror 28: 0xc782c287 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3365f: skipped 000777 1788575289.795985 punch from 0x02e59d to 0x03365e (0x50c2 bytes) 000778 1788575289.801257 punch from 0x009d70 to 0x00ecc7 (0x4f58 bytes) 000779 1788575289.810682 write 0x01b18b thru 0x029e0c (0xec82 bytes) 000780 1788575289.828415 MIRROR_RESYNC 9 mirrors 000781 1788575292.912252 trunc from 0x03365f to 0x01fbc4 (0x13a9a bytes) 000782 1788575292.951268 trunc from 0x011011 to 0x01eab7 (0xdaa7 bytes) 000783 1788575292.959194 fallocate 0x017402 thru 0x023919 (0xc518 bytes) 000784 1788575292.970990 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2391a): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2391a): Mirror 18: 0x813f04f9 (stale) Mirror 20: 0x813f04f9 (stale) Mirror 21: 0x813f04f9 (stale) Mirror 23: 0x813f04f9 (stale) Mirror 25: 0x813f04f9 (stale) Mirror 26: 0x813f04f9 (stale) Mirror 27: 0x813f04f9 (stale) Mirror 28: 0x813f04f9 (stale) CRC-32 checksum value for chunk [0, 0x2391a): Mirror 1: 0x260166a0 Mirror 29: 0x260166a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2391a: skipped 000785 1788575294.320301 read 0x000c8b thru 0x008ffd (0x8373 bytes) 000786 1788575294.344365 trunc from 0x01612d to 0x023919 (0xd7ed bytes) 000787 1788575294.400315 trunc from 0x02391a to 0x03a53e (0x16c25 bytes) 000789 1788575294.432359 fallocate 0x0197d0 thru 0x028936 (0xf167 bytes) 000790 1788575294.441792 write 0x033efc thru 0x03f8fa (0xb9ff bytes) 000791 1788575294.450730 mapread 0x016830 thru 0x01e463 (0x7c34 bytes) 000792 1788575294.475865 fallocate 0x004de0 thru 0x01246c (0xd68d bytes) 000793 1788575294.482907 fallocate 0x013f88 thru 0x01af10 (0x6f89 bytes) 000794 1788575294.489838 read 0x01e266 thru 0x02936b (0xb106 bytes) 000796 1788575294.504596 fallocate 0x0333b9 thru 0x0350ef (0x1d37 bytes) 000797 1788575294.515908 read 0x02b153 thru 0x02c457 (0x1305 bytes) 000798 1788575294.524240 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f8fb): Mirror 18: 0xab2317da (stale) Mirror 20: 0xab2317da (stale) Mirror 21: 0xab2317da (stale) Mirror 23: 0xab2317da (stale) Mirror 25: 0xab2317da (stale) Mirror 26: 0xab2317da (stale) Mirror 28: 0xab2317da (stale) Mirror 29: 0x6faaa01d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8fb): Mirror 18: 0xab2317da (stale) Mirror 20: 0xab2317da (stale) Mirror 21: 0xab2317da (stale) Mirror 23: 0xab2317da (stale) Mirror 25: 0xab2317da (stale) Mirror 26: 0xab2317da (stale) Mirror 28: 0xab2317da (stale) Mirror 29: 0x6faaa01d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8fb: skipped 000799 1788575295.193029 mapread 0x0211d7 thru 0x021500 (0x32a bytes) 000800 1788575295.200310 mapread 0x003805 thru 0x00c89e (0x909a bytes) 000801 1788575295.216986 read 0x01c14d thru 0x0278ee (0xb7a2 bytes) 000802 1788575295.233579 read 0x00bf66 thru 0x018ff2 (0xd08d bytes) 000803 1788575295.242986 read 0x016f8c thru 0x023a42 (0xcab7 bytes) 000804 1788575295.244571 mapread 0x00932d thru 0x014db2 (0xba86 bytes) 000805 1788575295.245545 mapread 0x004a1d thru 0x00619f (0x1783 bytes) 000806 1788575295.247751 mapwrite 0x009a4e thru 0x00c732 (0x2ce5 bytes) 000807 1788575295.263882 trunc from 0x03f8fb to 0x03d8f0 (0x200a bytes) 000808 1788575295.287786 fallocate 0x02adc0 thru 0x02cda8 (0x1fe9 bytes) 000809 1788575295.293580 write 0x013b2b thru 0x018c81 (0x5157 bytes) 000811 1788575295.297363 write 0x019cd6 thru 0x0231be (0x94e9 bytes) 000812 1788575295.304556 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3d8f1): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d8f1): Mirror 18: 0x476f45a0 (stale) Mirror 20: 0x476f45a0 (stale) Mirror 21: 0x476f45a0 (stale) Mirror 23: 0x476f45a0 (stale) Mirror 25: 0x476f45a0 (stale) Mirror 26: 0x476f45a0 (stale) Mirror 28: 0x476f45a0 (stale) Mirror 29: 0xa508c082 (stale) CRC-32 checksum value for chunk [0, 0x3d8f1): Mirror 1: 0xbad1f7d Mirror 30: 0xbad1f7d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d8f1: skipped 000813 1788575296.730168 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3d8f1): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d8f1): Mirror 18: 0x476f45a0 (stale) Mirror 20: 0x476f45a0 (stale) Mirror 21: 0x476f45a0 (stale) Mirror 23: 0x476f45a0 (stale) Mirror 25: 0x476f45a0 (stale) Mirror 26: 0x476f45a0 (stale) Mirror 28: 0x476f45a0 (stale) Mirror 29: 0xa508c082 (stale) CRC-32 checksum value for chunk [0, 0x3d8f1): Mirror 1: 0xbad1f7d Mirror 30: 0xbad1f7d Mirror 31: 0xbad1f7d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d8f1: skipped 000814 1788575298.044716 punch from 0x0032b0 to 0x00ceec (0x9c3d bytes) 000815 1788575298.111456 trunc from 0x03607e to 0x03d250 (0x71d3 bytes) 000816 1788575298.117020 trunc from 0x03d8f1 to 0x01aa8f (0x22e61 bytes) 000817 1788575298.172478 write 0x003cca thru 0x006f54 (0x328b bytes) 000818 1788575298.184504 mapread 0x008970 thru 0x015f9b (0xd62c bytes) 000820 1788575298.211366 read 0x00ca40 thru 0x00f7a7 (0x2d68 bytes) 000821 1788575298.215894 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1aa90): Mirror 20: 0xb452c333 (stale) Mirror 21: 0xb452c333 (stale) Mirror 23: 0xb452c333 (stale) Mirror 25: 0xb452c333 (stale) Mirror 26: 0xb452c333 (stale) Mirror 28: 0xb452c333 (stale) Mirror 29: 0x4fe12277 (stale) Mirror 30: 0x12e31ca0 (stale) Mirror 31: 0x12e31ca0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1aa90): Mirror 20: 0xb452c333 (stale) Mirror 21: 0xb452c333 (stale) Mirror 23: 0xb452c333 (stale) Mirror 25: 0xb452c333 (stale) Mirror 26: 0xb452c333 (stale) Mirror 28: 0xb452c333 (stale) Mirror 29: 0x4fe12277 (stale) Mirror 30: 0x12e31ca0 (stale) Mirror 31: 0x12e31ca0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1aa90: skipped 000822 1788575298.701149 read 0x019d90 thru 0x01aa8f (0xd00 bytes) 000823 1788575298.706726 punch from 0x0163f0 to 0x0170db (0xcec bytes) 000824 1788575298.715310 MIRROR_RESYNC 10 mirrors 000825 1788575301.605052 read 0x00de1c thru 0x01a73f (0xc924 bytes) 000826 1788575301.618897 read 0x002484 thru 0x00be0a (0x9987 bytes) 000827 1788575301.630875 read 0x01a5f7 thru 0x01aa8f (0x499 bytes) 000829 1788575301.631422 fallocate 0x0381a9 thru 0x038571 (0x3c9 bytes) 000830 1788575301.685513 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x38572): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38572): Mirror 20: 0xef0f811 (stale) Mirror 21: 0xef0f811 (stale) Mirror 23: 0xef0f811 (stale) Mirror 25: 0xef0f811 (stale) Mirror 26: 0xef0f811 (stale) Mirror 28: 0xef0f811 (stale) Mirror 29: 0xef0f811 (stale) Mirror 30: 0xef0f811 (stale) Mirror 31: 0xef0f811 (stale) CRC-32 checksum value for chunk [0, 0x38572): Mirror 1: 0xef0f811 Mirror 32: 0xef0f811 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38572: skipped 000831 1788575303.504962 mapwrite 0x03949e thru 0x03ffff (0x6b62 bytes) 000833 1788575303.631240 punch from 0x013e7f to 0x01459e (0x720 bytes) 000834 1788575303.640960 trunc from 0x040000 to 0x03907f (0x6f80 bytes) 000835 1788575303.667874 trunc from 0x02b102 to 0x02e768 (0x3667 bytes) 000836 1788575303.673701 punch from 0x030558 to 0x03907f (0x8b28 bytes) 000837 1788575303.679236 fallocate 0x017ce3 thru 0x01fdaf (0x80cd bytes) 000839 1788575303.684667 mapwrite 0x00b570 thru 0x00be89 (0x91a bytes) 000840 1788575303.715212 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39080): Mirror 20: 0xfe6ac7c7 (stale) Mirror 21: 0xfe6ac7c7 (stale) Mirror 23: 0xfe6ac7c7 (stale) Mirror 25: 0xfe6ac7c7 (stale) Mirror 26: 0xfe6ac7c7 (stale) Mirror 29: 0xfe6ac7c7 (stale) Mirror 30: 0xfe6ac7c7 (stale) Mirror 31: 0xfe6ac7c7 (stale) Mirror 32: 0xfe6ac7c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39080): Mirror 20: 0xfe6ac7c7 (stale) Mirror 21: 0xfe6ac7c7 (stale) Mirror 23: 0xfe6ac7c7 (stale) Mirror 25: 0xfe6ac7c7 (stale) Mirror 26: 0xfe6ac7c7 (stale) Mirror 29: 0xfe6ac7c7 (stale) Mirror 30: 0xfe6ac7c7 (stale) Mirror 31: 0xfe6ac7c7 (stale) Mirror 32: 0xfe6ac7c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39080: skipped 000841 1788575304.245933 mapwrite 0x03934f thru 0x03ef18 (0x5bca bytes) 000842 1788575304.308591 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ef19): Mirror 20: 0xdbcb6b1c (stale) Mirror 21: 0xdbcb6b1c (stale) Mirror 23: 0xdbcb6b1c (stale) Mirror 25: 0xdbcb6b1c (stale) Mirror 26: 0xdbcb6b1c (stale) Mirror 30: 0xdbcb6b1c (stale) Mirror 31: 0xdbcb6b1c (stale) Mirror 32: 0xdbcb6b1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef19): Mirror 20: 0xdbcb6b1c (stale) Mirror 21: 0xdbcb6b1c (stale) Mirror 23: 0xdbcb6b1c (stale) Mirror 25: 0xdbcb6b1c (stale) Mirror 26: 0xdbcb6b1c (stale) Mirror 30: 0xdbcb6b1c (stale) Mirror 31: 0xdbcb6b1c (stale) Mirror 32: 0xdbcb6b1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef19: skipped 000843 1788575304.764776 punch from 0x000054 to 0x00681d (0x67ca bytes) 000844 1788575304.772081 punch from 0x006c4f to 0x01300d (0xc3bf bytes) 000845 1788575304.777201 fallocate 0x01527d thru 0x01b97e (0x6702 bytes) 000846 1788575304.789775 MIRROR_RESYNC 9 mirrors 000847 1788575306.827172 mapwrite 0x01ee80 thru 0x020e14 (0x1f95 bytes) 000848 1788575306.942906 trunc from 0x00c537 to 0x00d876 (0x1340 bytes) 000849 1788575306.953774 read 0x005908 thru 0x007faa (0x26a3 bytes) 000850 1788575306.961508 punch from 0x00fefb to 0x014bd4 (0x4cda bytes) 000851 1788575306.969125 punch from 0x00c2ec to 0x01c25e (0xff73 bytes) 000852 1788575306.985005 write 0x02c1bb thru 0x02d9f7 (0x183d bytes) 000853 1788575307.005941 trunc from 0x026cf5 to 0x03549b (0xe7a7 bytes) 000855 1788575307.023071 mapwrite 0x001dea thru 0x00d7f0 (0xba07 bytes) 000856 1788575307.061486 fallocate 0x00984f thru 0x01949d (0xfc4f bytes) 000857 1788575307.068506 mapwrite 0x027474 thru 0x035117 (0xdca4 bytes) 000858 1788575307.119730 write 0x02b4c9 thru 0x02f2e9 (0x3e21 bytes) 000859 1788575307.121806 fallocate 0x003871 thru 0x00c638 (0x8dc8 bytes) 000860 1788575307.126635 trunc from 0x03ef19 to 0x014780 (0x2a798 bytes) 000861 1788575307.146064 write 0x020bce thru 0x02f2ca (0xe6fd bytes) 000862 1788575307.148146 fallocate 0x015ff5 thru 0x01d412 (0x741e bytes) 000863 1788575307.157643 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f2cb): Mirror 20: 0xb49db3bf (stale) Mirror 21: 0xb49db3bf (stale) Mirror 23: 0xb49db3bf (stale) Mirror 25: 0xb49db3bf (stale) Mirror 30: 0xb49db3bf (stale) Mirror 31: 0xb49db3bf (stale) Mirror 32: 0xb49db3bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f2cb): Mirror 20: 0xb49db3bf (stale) Mirror 21: 0xb49db3bf (stale) Mirror 23: 0xb49db3bf (stale) Mirror 25: 0xb49db3bf (stale) Mirror 30: 0xb49db3bf (stale) Mirror 31: 0xb49db3bf (stale) Mirror 32: 0xb49db3bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f2cb: skipped 000864 1788575307.989327 mapwrite 0x00dfae thru 0x012cc4 (0x4d17 bytes) 000865 1788575308.031531 fallocate 0x011c46 thru 0x01e6b1 (0xca6c bytes) 000866 1788575308.036445 fallocate 0x0368a7 thru 0x03fe0a (0x9564 bytes) 000867 1788575308.041770 read 0x01505e thru 0x01bce2 (0x6c85 bytes) 000868 1788575308.048944 fallocate 0x006b58 thru 0x0089a5 (0x1e4e bytes) 000869 1788575308.058370 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2f2cb): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f2cb): Mirror 20: 0xb49db3bf (stale) Mirror 21: 0xb49db3bf (stale) Mirror 23: 0xb49db3bf (stale) Mirror 25: 0xb49db3bf (stale) Mirror 30: 0xb49db3bf (stale) Mirror 31: 0xb49db3bf (stale) Mirror 32: 0xb49db3bf (stale) CRC-32 checksum value for chunk [0, 0x2f2cb): Mirror 1: 0x5d295d Mirror 33: 0x5d295d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f2cb: skipped 000870 1788575309.006355 trunc from 0x00e10a to 0x013ccf (0x5bc6 bytes) 000871 1788575309.066754 mapread 0x0138dd thru 0x020267 (0xc98b bytes) 000872 1788575309.106842 trunc from 0x0120d5 to 0x0185c3 (0x64ef bytes) 000873 1788575309.116806 write 0x02e95f thru 0x03c471 (0xdb13 bytes) 000874 1788575309.132801 read 0x001bbc thru 0x00c23a (0xa67f bytes) 000875 1788575309.150498 punch from 0x0262e7 to 0x02aba1 (0x48bb bytes) 000877 1788575309.159743 trunc from 0x03c472 to 0x008476 (0x33ffb bytes) 000878 1788575309.240195 trunc from 0x008477 to 0x02ba92 (0x2361c bytes) 000879 1788575309.282804 mapwrite 0x01af38 thru 0x0243ee (0x94b7 bytes) 000880 1788575309.335307 mapwrite 0x03b049 thru 0x03ffff (0x4fb7 bytes) 000881 1788575309.417385 trunc from 0x040000 to 0x00724c (0x38db3 bytes) 000882 1788575309.447586 MIRROR_RESYNC 9 mirrors 000883 1788575310.761992 read 0x0063b2 thru 0x00724c (0xe9b bytes) 000884 1788575310.768683 fallocate 0x018c7c thru 0x023e3b (0xb1c0 bytes) 000885 1788575310.817663 fallocate 0x02748a thru 0x035afa (0xe671 bytes) 000886 1788575310.823552 trunc from 0x0024b3 to 0x00724c (0x4d9a bytes) 000887 1788575310.828786 punch from 0x0066bf to 0x00724c (0xb8e bytes) 000889 1788575310.834503 trunc from 0x0006d4 to 0x004f17 (0x4844 bytes) 000890 1788575310.839977 mapread 0x002c56 thru 0x00724c (0x45f7 bytes) 000891 1788575310.863143 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x724d): Mirror 20: 0x9c34170a (stale) Mirror 21: 0x9c34170a (stale) Mirror 23: 0x9c34170a (stale) Mirror 25: 0x9c34170a (stale) Mirror 30: 0x9c34170a (stale) Mirror 31: 0x9c34170a (stale) Mirror 33: 0x9c34170a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x724d): Mirror 20: 0x9c34170a (stale) Mirror 21: 0x9c34170a (stale) Mirror 23: 0x9c34170a (stale) Mirror 25: 0x9c34170a (stale) Mirror 30: 0x9c34170a (stale) Mirror 31: 0x9c34170a (stale) Mirror 33: 0x9c34170a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x724d: skipped 000892 1788575311.251119 MIRROR_RESYNC 8 mirrors 000894 1788575312.336095 trunc from 0x00724d to 0x02fa10 (0x287c4 bytes) 000895 1788575312.375294 write 0x0385d1 thru 0x03ffff (0x7a2f bytes) 000897 1788575312.385650 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0xe20eea22 (stale) Mirror 21: 0xe20eea22 (stale) Mirror 23: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 30: 0xe20eea22 (stale) Mirror 31: 0xe20eea22 (stale) Mirror 33: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc1d961a4 Mirror 34: 0xc1d961a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000898 1788575313.691226 trunc from 0x040000 to 0x0026df (0x3d920 bytes) 000899 1788575313.768868 mapwrite 0x01774c thru 0x025e62 (0xe717 bytes) 000900 1788575313.919435 punch from 0x014b61 to 0x01f040 (0xa4e0 bytes) 000901 1788575313.941723 read 0x018420 thru 0x01cd8d (0x496e bytes) 000902 1788575313.955604 mapread 0x0115a3 thru 0x014c6b (0x36c9 bytes) 000903 1788575313.978169 write 0x022467 thru 0x026f94 (0x4b2e bytes) 000904 1788575313.979390 mapread 0x01b656 thru 0x01fe64 (0x480f bytes) 000905 1788575314.000598 MIRROR_RESYNC 9 mirrors 000906 1788575315.306126 trunc from 0x014b32 to 0x024466 (0xf935 bytes) 000907 1788575315.344567 trunc from 0x026f95 to 0x013185 (0x13e0f bytes) 000909 1788575315.370293 read 0x0063eb thru 0x00fa4d (0x9663 bytes) 000911 1788575315.380550 trunc from 0x00fd34 to 0x013185 (0x3452 bytes) 000912 1788575315.386043 punch from 0x0073db to 0x00a5ea (0x3210 bytes) 000913 1788575315.391940 trunc from 0x013186 to 0x028339 (0x151b4 bytes) 000914 1788575315.409849 fallocate 0x02dda6 thru 0x03d2f4 (0xf54f bytes) 000915 1788575315.417081 read 0x010501 thru 0x010a4f (0x54f bytes) 000916 1788575315.432418 write 0x031655 thru 0x033c79 (0x2625 bytes) 000917 1788575315.434487 fallocate 0x023650 thru 0x032856 (0xf207 bytes) 000918 1788575315.439547 mapread 0x0206e9 thru 0x02b6a1 (0xafb9 bytes) 000920 1788575315.452944 read 0x015886 thru 0x021fb2 (0xc72d bytes) 000921 1788575315.457735 write 0x02706b thru 0x0270b8 (0x4e bytes) 000924 1788575315.474795 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x33c7a): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c7a): Mirror 20: 0xf1c5aad8 (stale) Mirror 21: 0xf1c5aad8 (stale) Mirror 23: 0xf1c5aad8 (stale) Mirror 25: 0xf1c5aad8 (stale) Mirror 30: 0xf1c5aad8 (stale) Mirror 31: 0xf1c5aad8 (stale) Mirror 33: 0xf1c5aad8 (stale) Mirror 34: 0xf1c5aad8 (stale) CRC-32 checksum value for chunk [0, 0x33c7a): Mirror 1: 0x2974f1f8 Mirror 35: 0x2974f1f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c7a: skipped 000925 1788575316.640420 read 0x005b4d thru 0x0064e2 (0x996 bytes) 000926 1788575316.656713 trunc from 0x0264d4 to 0x02b959 (0x5486 bytes) 000928 1788575316.712183 fallocate 0x007e2e thru 0x01649f (0xe672 bytes) 000929 1788575316.718485 trunc from 0x00947b to 0x013cf4 (0xa87a bytes) 000930 1788575316.723598 read 0x02e84c thru 0x033c79 (0x542e bytes) 000931 1788575316.737598 fallocate 0x01bff1 thru 0x02756d (0xb57d bytes) 000933 1788575316.751345 mapread 0x001f8f thru 0x00252b (0x59d bytes) 000934 1788575316.752497 mapread 0x0189f5 thru 0x02056c (0x7b78 bytes) 000935 1788575316.755501 read 0x026cf7 thru 0x0309a1 (0x9cab bytes) 000937 1788575316.757266 read 0x00d05e thru 0x0114be (0x4461 bytes) 000938 1788575316.757981 fallocate 0x026dce thru 0x02a472 (0x36a5 bytes) 000939 1788575316.762870 fallocate 0x0324e3 thru 0x03ffff (0xdb1d bytes) 000940 1788575316.766229 fallocate 0x00bd28 thru 0x01769f (0xb978 bytes) 000941 1788575316.770081 trunc from 0x040000 to 0x02ee15 (0x111ea bytes) 000942 1788575316.804063 trunc from 0x011cbf to 0x0172e9 (0x562b bytes) 000943 1788575316.818848 MIRROR_RESYNC 10 mirrors 000945 1788575318.345065 write 0x02d4f4 thru 0x02d59c (0xa9 bytes) 000946 1788575318.387046 trunc from 0x02ee16 to 0x003da9 (0x2b06c bytes) 000947 1788575318.416156 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3daa): Mirror 21: 0x61401a12 (stale) Mirror 23: 0x61401a12 (stale) Mirror 25: 0x61401a12 (stale) Mirror 30: 0x61401a12 (stale) Mirror 31: 0x61401a12 (stale) Mirror 33: 0x61401a12 (stale) Mirror 34: 0x61401a12 (stale) Mirror 35: 0x61401a12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3daa): Mirror 21: 0x61401a12 (stale) Mirror 23: 0x61401a12 (stale) Mirror 25: 0x61401a12 (stale) Mirror 30: 0x61401a12 (stale) Mirror 31: 0x61401a12 (stale) Mirror 33: 0x61401a12 (stale) Mirror 34: 0x61401a12 (stale) Mirror 35: 0x61401a12 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3daa: skipped 000948 1788575318.970575 read 0x000dac thru 0x003da9 (0x2ffe bytes) 000949 1788575318.978281 fallocate 0x009009 thru 0x00b581 (0x2579 bytes) 000950 1788575318.987479 mapread 0x0031f4 thru 0x003da9 (0xbb6 bytes) 000951 1788575318.989580 mapwrite 0x02b24c thru 0x02f87a (0x462f bytes) 000952 1788575319.094080 mapread 0x00a95c thru 0x017a5d (0xd102 bytes) 000953 1788575319.111565 read 0x002b05 thru 0x008353 (0x584f bytes) 000954 1788575319.112304 fallocate 0x01f2ed thru 0x021e57 (0x2b6b bytes) 000955 1788575319.121657 MIRROR_RESYNC 9 mirrors 000956 1788575321.272455 MIRROR_RESYNC 9 mirrors 000957 1788575322.087063 fallocate 0x025b28 thru 0x02632c (0x805 bytes) 000958 1788575322.126970 trunc from 0x01a7d2 to 0x0218a8 (0x70d7 bytes) 000959 1788575322.132545 mapread 0x00a173 thru 0x00ecb2 (0x4b40 bytes) 000960 1788575322.156303 mapwrite 0x0183c7 thru 0x0265ed (0xe227 bytes) 000961 1788575322.196872 punch from 0x0286c5 to 0x02e557 (0x5e93 bytes) 000962 1788575322.202064 punch from 0x02602c to 0x02f87a (0x984f bytes) 000963 1788575322.208275 punch from 0x02819f to 0x028b21 (0x983 bytes) 000964 1788575322.212869 fallocate 0x00a2c8 thru 0x011826 (0x755f bytes) 000966 1788575322.218354 mapread 0x012b3c thru 0x020d0d (0xe1d2 bytes) 000967 1788575322.220305 mapwrite 0x02af6d thru 0x03a765 (0xf7f9 bytes) 000969 1788575322.292329 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a766): Mirror 21: 0x43e032fd (stale) Mirror 23: 0x43e032fd (stale) Mirror 25: 0x43e032fd (stale) Mirror 30: 0x43e032fd (stale) Mirror 31: 0x43e032fd (stale) Mirror 33: 0x43e032fd (stale) Mirror 35: 0x43e032fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a766): Mirror 21: 0x43e032fd (stale) Mirror 23: 0x43e032fd (stale) Mirror 25: 0x43e032fd (stale) Mirror 30: 0x43e032fd (stale) Mirror 31: 0x43e032fd (stale) Mirror 33: 0x43e032fd (stale) Mirror 35: 0x43e032fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a766: skipped 000970 1788575322.814729 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a766): Mirror 21: 0x43e032fd (stale) Mirror 25: 0x43e032fd (stale) Mirror 30: 0x43e032fd (stale) Mirror 31: 0x43e032fd (stale) Mirror 33: 0x43e032fd (stale) Mirror 35: 0x43e032fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a766): Mirror 21: 0x43e032fd (stale) Mirror 25: 0x43e032fd (stale) Mirror 30: 0x43e032fd (stale) Mirror 31: 0x43e032fd (stale) Mirror 33: 0x43e032fd (stale) Mirror 35: 0x43e032fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a766: skipped 000971 1788575323.434262 read 0x007b84 thru 0x0172ee (0xf76b bytes) 000972 1788575323.451121 mapwrite 0x03590c thru 0x03ffff (0xa6f4 bytes) 000973 1788575323.531028 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xb0bb5182 (stale) Mirror 30: 0xb0bb5182 (stale) Mirror 31: 0xb0bb5182 (stale) Mirror 33: 0xb0bb5182 (stale) Mirror 35: 0xb0bb5182 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xb0bb5182 (stale) Mirror 30: 0xb0bb5182 (stale) Mirror 31: 0xb0bb5182 (stale) Mirror 33: 0xb0bb5182 (stale) Mirror 35: 0xb0bb5182 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000974 1788575324.007493 trunc from 0x01fb68 to 0x028112 (0x85ab bytes) 000976 1788575324.019499 mapread 0x012d1e thru 0x01cf8d (0xa270 bytes) 000977 1788575324.054287 read 0x03ccd8 thru 0x03ffff (0x3328 bytes) 000978 1788575324.064855 punch from 0x0284fd to 0x036af5 (0xe5f9 bytes) 000979 1788575324.121839 fallocate 0x03cc2b thru 0x03ffff (0x33d5 bytes) 000980 1788575324.159106 MIRROR_RESYNC 6 mirrors 000981 1788575325.638023 mapwrite 0x02169c thru 0x024522 (0x2e87 bytes) 000983 1788575325.786403 punch from 0x01a0fb to 0x02475c (0xa662 bytes) 000984 1788575325.800040 read 0x00f40f thru 0x019b12 (0xa704 bytes) 000985 1788575325.811328 fallocate 0x03810f thru 0x03ffff (0x7ef1 bytes) 000986 1788575325.822505 punch from 0x030e53 to 0x03770d (0x68bb bytes) 000987 1788575325.833130 mapwrite 0x0037aa thru 0x005abb (0x2312 bytes) 000990 1788575325.884336 mapwrite 0x020e74 thru 0x02f4b8 (0xe645 bytes) 000991 1788575325.944779 trunc from 0x040000 to 0x037c47 (0x83b8 bytes) 000992 1788575325.975158 fallocate 0x003807 thru 0x0106c6 (0xcec0 bytes) 000993 1788575325.984491 mapwrite 0x03e2db thru 0x03ffff (0x1d25 bytes) 000994 1788575326.027832 trunc from 0x02624b to 0x0359c0 (0xf776 bytes) 000995 1788575326.034550 trunc from 0x00b27b to 0x00edac (0x3b32 bytes) 000996 1788575326.041549 punch from 0x004b20 to 0x01216f (0xd650 bytes) 000997 1788575326.054213 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x2fd2558d (stale) Mirror 30: 0x2fd2558d (stale) Mirror 31: 0x2fd2558d (stale) Mirror 33: 0x2fd2558d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x2fd2558d (stale) Mirror 30: 0x2fd2558d (stale) Mirror 31: 0x2fd2558d (stale) Mirror 33: 0x2fd2558d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000998 1788575326.425866 trunc from 0x040000 to 0x03f5c6 (0xa39 bytes) 000999 1788575326.443567 punch from 0x03e477 to 0x03f5c6 (0x1150 bytes) 001000 1788575326.447872 write 0x011a2f thru 0x0174b2 (0x5a84 bytes) All operations completed A-OK! PASS 70a (155s) == sanity-flr test 71: check mirror extend parent fid ==== 22:28:53 (1788575333) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x19a:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 22:29:06 (1788575346) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.171195 s, 12.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143718 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 22:29:16 (1788575356) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6591936 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6457696 .. done Extending file size to 8267936 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7272800 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8996736 .. Extending file size to 8278144 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8907744 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61466 61467 61468 61471 61472 done 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) 192.168.201.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (90s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:30:46 (1788575446) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3370432 .. Extending file size to 1346688 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8686624 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6616000 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8053408 .. Extending file size to 8055360 .. Extending file size to 9045568 .. Extending file size to 8799136 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8911936 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62897 62898 62900 62901 62903 done 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) 192.168.201.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:32:09 (1788575529) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2536 1268768 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1269244 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25484 3581536 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17656 3589364 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43140 7170900 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2536 1268768 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2060 1269244 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25484 3581536 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17656 3589364 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43140 7170900 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0884512 s, 23.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.250766 s, 8.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xddcdc7af (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:32:31 (1788575551) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139162 s, 15.1 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 22:32:40 (1788575560) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:32:49 (1788575569) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0838104 s, 25.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:32:58 (1788575578) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0979373 s, 21.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x208:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x209:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x209:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x208:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x209:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x209:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20a:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:33:07 (1788575587) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1272 s, 16.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:33:41 (1788575621) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg110-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554654 s, 18.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } PASS 204d (51s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:34:32 (1788575672) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg110-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135098 s, 15.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:35:24 (1788575724) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg110-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg110-server: Pool lustre.test_204f created Adding targets to pool oleg110-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg110-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119803 s, 17.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg110-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg110-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg110-server: Pool lustre.test_204f destroyed PASS 204f (72s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:36:36 (1788575796) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:36:45 (1788575805) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:36:53 (1788575813) oleg110-server: Pool lustre.test_206 created Adding targets to pool oleg110-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg110-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg110-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg110-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg110-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:37:22 (1788575842) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686364 s, 15.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265674 s, 39.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x11b27cc7 Mirror 3: 0x11b27cc7 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:37:32 (1788575852) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:37:36 (1788575856) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:37:40 (1788575860) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0893073 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ca0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x22c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x22c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x22c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xee0000 Read this (4 bytes): BBBB 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:38:16 (1788575896) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0172419 s, 475 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:38:22 (1788575902) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447875 s, 23.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa9 lmm_fid: [0x200000405:0xa9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa9 lmm_fid: [0x200000405:0xa9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611305 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 22:38:24.000000000 -0400 Modify: 2026-09-04 22:38:24.000000000 -0400 Change: 2026-09-04 22:38:25.000000000 -0400 Birth: 2026-09-04 22:38:24.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:38:29 (1788575909) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291748 s, 35.9 MB/s oleg110-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg110-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg110-server: O/280000401/d28/15c: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:38:53 (1788575933) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.223126 s, 184 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0199413 s, 205 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 4 22:38 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xae:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 4 22:38 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:39:00 (1788575940) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:39:07 (1788575947) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 2051 sec =========== 22:39:16 (1788575956) === sanity-flr: start cleanup 22:39:18 (1788575958) === === sanity-flr: finish cleanup 22:39:26 (1788575966) ===