-----============= acceptance-small: sanity-flr ============----- Sat Aug 15 17:03:38 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg146-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 17:04:01 (1786827841) === oleg146-client.virtnet: executing check_config_client /mnt/lustre oleg146-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg146-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b38460b0800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b38460b0800.idle_timeout=debug disable quota as required oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 17:04:46 (1786827886) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8231] == sanity-flr test 0a: lfs mirror create with -N option == 17:04:48 (1786827888) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 17:05:07 (1786827907) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 17:05:12 (1786827912) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 17:05:15 (1786827915) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 17:05:40 (1786827940) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 17:05:44 (1786827944) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 17:05:48 (1786827948) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b38460b0800.stats=clear osc.lustre-OST0001-osc-ffff9b38460b0800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:05:56 (1786827956) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:06:07 (1786827967) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:06:16 (1786827976) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 17:06:19 (1786827979) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:06:29 (1786827989) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 17:06:39 (1786827999) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:06:47 (1786828007) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:06:56 (1786828016) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:07:05 (1786828025) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:07:18 (1786828038) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:07:27 (1786828047) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 1.01258 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163377 s, 6.4 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:07:38 (1786828058) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.369707 s, 25.5 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 17:07:48 (1786828068) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:08:01 (1786828081) Stopping /mnt/lustre-ost1 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b38460b0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b38460b0800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg146-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b38460b0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b38460b0800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg146-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (79s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:09:19 (1786828159) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.117547 s, 26.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b38460b0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b38460b0800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg146-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:10:02 (1786828202) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b38460b0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b38460b0800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg146-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b38460b0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b38460b0800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg146-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (76s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:11:18 (1786828278) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.76369 s, 19.0 MB/s -rw-r--r-- 1 root root 52428800 Aug 15 17:11 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.670s user 0m0.001s sys 0m0.175s Stopping /mnt/lustre-ost1 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b38460b0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b38460b0800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.001s user 0m0.001s sys 0m0.189s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg146-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b38460b0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b38460b0800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.206s user 0m0.001s sys 0m0.187s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg146-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg146-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b38460b0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b38460b0800.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:12:55 (1786828375) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:12:59 (1786828379) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:13:03 (1786828383) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 17:13:07 (1786828387) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 17:13:15 (1786828395) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.448703 s, 28.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:13:27 (1786828407) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505534 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06217 s, 16.9 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:13:36 (1786828416) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:13:44 (1786828424) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00146311 s, 700 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00426579 s, 240 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00107602 s, 952 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00108546 s, 943 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 17:13:55 (1786828435) create test files with size 15759922 .. e6220aa05a684d9b37fb760e20d014f5 - 52a691ceed0c73f59cf22b9e52814a39 - 7d9bc18a6b4dff477e9e217a24d7d7fb - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg146-client.virtnet /mnt/lustre (opts:) Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 17:14:22 (1786828462) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:14:37 (1786828477) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:14:44 (1786828484) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515746 s, 20.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280287 s, 37.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 17:14:53 (1786828493) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0777855 s, 27.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.278669 s, 15.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.176525 s, 11.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 17:15:09 (1786828509) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 17:15:13 (1786828513) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:15:17 (1786828517) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231905 s, 45.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 17:15:27 (1786828527) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.156738 s, 20.1 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:15:39 (1786828539) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:15:50 (1786828550) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.381652 s, 27.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg146-client.virtnet /mnt/lustre (opts:) Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:16:02 (1786828562) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9b3842b38800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.300824 s, 34.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg146-client.virtnet /mnt/lustre (opts:) Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9b384737a000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:16:19 (1786828579) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (45s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:17:04 (1786828624) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (43s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:17:48 (1786828668) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 17:17:51 (1786828671) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.299163 s, 10.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.215776 s, 14.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x318fb32b (nosync,stale) Mirror 2: 0x318fb32b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 69bd1069d3d5c71bc88729c6b7b0ad19 - mirror 1 checksum: 50da97a4d502d5a36472c7a09d3f0f66 - mirror 2 checksum: 69bd1069d3d5c71bc88729c6b7b0ad19 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-15T17:17:53-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-15T17:17:53-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 69bd1069d3d5c71bc88729c6b7b0ad19 - mirror 1 checksum: 50da97a4d502d5a36472c7a09d3f0f66 - mirror 2 checksum: 50da97a4d502d5a36472c7a09d3f0f66 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-15T17:17:53-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-15T17:17:53-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 17:18:02 (1786828682) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735623 s, 14.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:18:10 (1786828690) Failing mds1 on oleg146-server Stopping /mnt/lustre-mds1 (opts:) on oleg146-server 17:18:24 (1786828704) shut down facet: mds1 facet_host: oleg146-server facet_failover_host: oleg146-server Failover mds1 to oleg146-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0000 17:18:57 (1786828737) targets are mounted 17:18:57 (1786828737) facet_failover done oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558133 s, 18.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 50A (68s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:19:18 (1786828758) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.114221 s, 179 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0211883 s, 967 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (14s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:19:32 (1786828772) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.116985 s, 175 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0128271 s, 1.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:19:43 (1786828783) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0167051 s, 981 kB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:19:52 (1786828792) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0632735 s, 324 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:20:01 (1786828801) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0127158 s, 322 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0221646 s, 185 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:20:09 (1786828809) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:20:31 (1786828831) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (33s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 17:21:04 (1786828864) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 17:21:29 (1786828889) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 17:21:53 (1786828913) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 17:22:01 (1786828921) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57583 mirror split pid 57584 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7817: 57583 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57584 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 17:23:09 (1786828989) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 8991 truncating to largest ever: 0x118eb 000001 1786828992.244993 trunc from 00000000 to 0x0118ea (0x118eb bytes) 000002 1786828992.263566 write 0x01ea5f thru 0x025a06 (0x6fa8 bytes) 000003 1786828992.267426 trunc from 0x025a07 to 0x010a03 (0x15003 bytes) 000005 1786828992.301056 mapread 0x00aa04 thru 0x010a03 (0x6000 bytes) 000006 1786828992.339002 mapwrite 0x0398df thru 0x03ffff (0x6721 bytes) 000007 1786828992.466692 mapread 0x00048f thru 0x000874 (0x3e6 bytes) 000008 1786828992.518601 write 0x011ee6 thru 0x018c93 (0x6dae bytes) 000009 1786828992.536653 mapread 0x01fe20 thru 0x021176 (0x1357 bytes) 000011 1786828992.573809 punch from 0x0260d6 to 0x0347bd (0xe6e8 bytes) fallocating to largest ever: 0x33ad7 000012 1786828992.593202 fallocate 0x033223 thru 0x033ad6 (0x8b4 bytes) 000014 1786828992.612542 trunc from 0x01c22a to 0x02aad6 (0xe8ad bytes) truncating to largest ever: 0x3c900 000015 1786828992.653165 trunc from 0x040000 to 0x03c8ff (0x3700 bytes) truncating to largest ever: 0x3e893 000016 1786828992.673906 trunc from 0x03c900 to 0x03e892 (0x1f93 bytes) 000017 1786828992.717317 write 0x03dd2f thru 0x03ffff (0x22d1 bytes) 000018 1786828992.733536 trunc from 0x007cf2 to 0x00a373 (0x2682 bytes) 000019 1786828992.748196 write 0x004856 thru 0x00bf16 (0x76c1 bytes) 000020 1786828992.751402 fallocate 0x00e8e5 thru 0x01acc2 (0xc3de bytes) 000022 1786828992.765816 write 0x02ccb4 thru 0x02e137 (0x1484 bytes) 000023 1786828992.786298 trunc from 0x040000 to 0x009f43 (0x360bc bytes) 000026 1786828992.843293 read 0x008e68 thru 0x009f43 (0x10dc bytes) 000027 1786828992.844853 mapwrite 0x03d901 thru 0x03ffff (0x26ff bytes) 000029 1786828992.916794 read 0x035902 thru 0x03a2e4 (0x49e3 bytes) 000030 1786828992.938756 trunc from 0x040000 to 0x03dcb3 (0x234c bytes) 000031 1786828992.964302 trunc from 0x02aabc to 0x02b9d5 (0xf1a bytes) 000033 1786828992.976333 read 0x03142e thru 0x036552 (0x5125 bytes) 000034 1786828992.979012 trunc from 0x0279fc to 0x02cfef (0x55f4 bytes) 000035 1786828992.989119 write 0x004336 thru 0x00b197 (0x6e62 bytes) 000036 1786828992.993374 trunc from 0x03dcb4 to 0x0126dc (0x2b5d7 bytes) 000038 1786828993.019334 trunc from 0x003499 to 0x0099e2 (0x654a bytes) 000039 1786828993.044064 fallocate 0x03dee8 thru 0x03e55a (0x673 bytes) 000041 1786828993.053679 trunc from 0x03e55b to 0x012b6c (0x2b9ee bytes) 000042 1786828993.072505 mapwrite 0x008073 thru 0x010fe5 (0x8f73 bytes) 000045 1786828993.118105 punch from 0x007a55 to 0x00b1e0 (0x378c bytes) 000046 1786828993.125314 write 0x02b81e thru 0x038862 (0xd045 bytes) 000047 1786828993.130086 trunc from 0x038863 to 0x01fe05 (0x18a5d bytes) 000048 1786828993.153477 write 0x036602 thru 0x03ffff (0x99fe bytes) 000049 1786828993.162140 write 0x035da7 thru 0x03e494 (0x86ee bytes) 000051 1786828993.179160 mapwrite 0x03d8c7 thru 0x03ffff (0x2739 bytes) 000052 1786828993.210432 mapwrite 0x00a7f1 thru 0x019d6e (0xf57e bytes) 000053 1786828993.283134 MIRROR_EXTEND to 2 mirrors 000054 1786828994.376029 MIRROR_EXTEND to 3 mirrors 000055 1786828995.243282 mapread 0x008767 thru 0x013e5d (0xb6f7 bytes) 000056 1786828995.291203 fallocate 0x00eac6 thru 0x013bd1 (0x510c bytes) 000057 1786828995.337734 trunc from 0x040000 to 0x033606 (0xc9f9 bytes) 000058 1786828995.391380 mapwrite 0x03545d thru 0x03a994 (0x5538 bytes) 000059 1786828995.460581 mapwrite 0x012f48 thru 0x016709 (0x37c2 bytes) 000060 1786828995.506690 mapread 0x000572 thru 0x00dda9 (0xd838 bytes) 000061 1786828995.541256 mapwrite 0x0174c3 thru 0x019b0e (0x264c bytes) 000062 1786828995.564264 mapwrite 0x02acf2 thru 0x032b35 (0x7e44 bytes) 000063 1786828995.623247 write 0x008bcc thru 0x00e5f6 (0x5a2b bytes) 000066 1786828995.626339 trunc from 0x03a995 to 0x03671e (0x4276 bytes) 000067 1786828995.650342 write 0x038d07 thru 0x03ffff (0x72f9 bytes) 000068 1786828995.655521 mapwrite 0x0111d6 thru 0x020b77 (0xf9a2 bytes) 000069 1786828995.711639 fallocate 0x02bf93 thru 0x03747f (0xb4ed bytes) 000071 1786828995.724021 MIRROR_RESYNC 3 mirrors 000072 1786828996.689302 fallocate 0x0134c2 thru 0x01e692 (0xb1d1 bytes) 000073 1786828996.721014 read 0x02c07a thru 0x02de86 (0x1e0d bytes) 000074 1786828996.735752 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9a1c46e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9a1c46e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000075 1786828997.508478 fallocate 0x01297e thru 0x01513d (0x27c0 bytes) 000076 1786828997.530247 mapread 0x03ae01 thru 0x03ffff (0x51ff bytes) 000078 1786828997.591540 mapwrite 0x0050a1 thru 0x00eecc (0x9e2c bytes) 000079 1786828997.662270 MIRROR_RESYNC 2 mirrors 000080 1786828998.630926 mapwrite 0x00efde thru 0x017a6f (0x8a92 bytes) 000081 1786828998.716473 mapread 0x000e08 thru 0x003417 (0x2610 bytes) 000082 1786828998.726655 mapwrite 0x00b22f thru 0x0100cc (0x4e9e bytes) 000083 1786828998.743435 fallocate 0x03659a thru 0x0389ba (0x2421 bytes) 000084 1786828998.749049 trunc from 0x040000 to 0x028338 (0x17cc7 bytes) 000085 1786828998.770092 fallocate 0x037566 thru 0x03a1ce (0x2c69 bytes) 000086 1786828998.775601 mapread 0x02ff2f thru 0x03a1ce (0xa2a0 bytes) 000088 1786828998.796735 trunc from 0x012b3d to 0x012b6e (0x32 bytes) 000089 1786828998.802064 read 0x036657 thru 0x03a1ce (0x3b78 bytes) 000090 1786828998.802808 mapread 0x022921 thru 0x025d3c (0x341c bytes) 000091 1786828998.812957 mapwrite 0x01645b thru 0x0184a3 (0x2049 bytes) 000093 1786828998.840957 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a1cf): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1cf): Mirror 2: 0x2744f2ca (stale) CRC-32 checksum value for chunk [0, 0x3a1cf): Mirror 1: 0x4a735f15 Mirror 3: 0x4a735f15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1cf: skipped 000094 1786828999.631867 read 0x024bb5 thru 0x0282a0 (0x36ec bytes) 000096 1786828999.661033 read 0x004e2e thru 0x00fc73 (0xae46 bytes) 000097 1786828999.686282 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a1cf): Mirror 2: 0x2744f2ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1cf): Mirror 2: 0x2744f2ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1cf: skipped 000098 1786828999.963352 mapwrite 0x00a0bc thru 0x00dd68 (0x3cad bytes) 000099 1786829000.010058 read 0x0357fa thru 0x03a1ce (0x49d5 bytes) 000100 1786829000.019364 read 0x0211a6 thru 0x022acb (0x1926 bytes) 000101 1786829000.029955 punch from 0x007eef to 0x0107b1 (0x88c3 bytes) 000102 1786829000.036060 read 0x017e77 thru 0x01ba26 (0x3bb0 bytes) 000103 1786829000.036986 read 0x013bd9 thru 0x01583a (0x1c62 bytes) 000106 1786829000.039383 trunc from 0x03a1cf to 0x02a50f (0xfcbf bytes) 000107 1786829000.070622 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000108 1786829000.214524 write 0x01d164 thru 0x023eac (0x6d49 bytes) 000109 1786829000.231128 mapread 0x000e6f thru 0x00dd55 (0xcee7 bytes) 000110 1786829000.269228 punch from 0x012f22 to 0x014278 (0x1357 bytes) 000111 1786829000.274391 punch from 0x011999 to 0x018032 (0x669a bytes) 000113 1786829000.281298 write 0x0111ad thru 0x017ae2 (0x6936 bytes) 000114 1786829000.294526 mapwrite 0x03da33 thru 0x03ffff (0x25cd bytes) 000115 1786829000.349310 fallocate 0x02ffeb thru 0x033163 (0x3179 bytes) 000116 1786829000.358038 mapwrite 0x028dfe thru 0x02e261 (0x5464 bytes) 000117 1786829000.387409 mapread 0x00a7aa thru 0x014aa6 (0xa2fd bytes) 000118 1786829000.388960 trunc from 0x040000 to 0x031605 (0xe9fa bytes) 000119 1786829000.405991 punch from 0x023cfe to 0x02e8ae (0xabb1 bytes) 000120 1786829000.413141 write 0x032f55 thru 0x03e099 (0xb145 bytes) 000121 1786829000.417141 trunc from 0x00ab73 to 0x0199ed (0xee7b bytes) 000122 1786829000.428055 mapwrite 0x027b06 thru 0x02ffdb (0x84d6 bytes) 000123 1786829000.480651 MIRROR_EXTEND to 2 mirrors 000124 1786829001.181607 fallocate 0x00af26 thru 0x012685 (0x7760 bytes) 000125 1786829001.249012 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000126 1786829001.478929 trunc from 0x03e09a to 0x001f12 (0x3c187 bytes) 000127 1786829001.592296 MIRROR_EXTEND to 2 mirrors 000128 1786829002.719177 fallocate 0x017d45 thru 0x01e2cf (0x658b bytes) 000129 1786829002.753371 punch from 0x00264d to 0x0106a6 (0xe05a bytes) 000130 1786829002.759322 trunc from 0x0058ee to 0x006363 (0xa76 bytes) 000131 1786829002.764182 write 0x038922 thru 0x03bb4f (0x322e bytes) 000132 1786829002.772636 mapread 0x027f05 thru 0x02f77b (0x7877 bytes) 000133 1786829002.815178 fallocate 0x002d35 thru 0x00a420 (0x76ec bytes) 000134 1786829002.820940 fallocate 0x025c5b thru 0x0343d6 (0xe77c bytes) 000135 1786829002.826444 fallocate 0x0000b1 thru 0x00c15a (0xc0aa bytes) 000136 1786829002.835622 mapread 0x03262c thru 0x038f53 (0x6928 bytes) 000137 1786829002.846024 write 0x02d842 thru 0x038b42 (0xb301 bytes) 000138 1786829002.849823 mapread 0x019b67 thru 0x027918 (0xddb2 bytes) 000139 1786829002.870613 read 0x008537 thru 0x00a64e (0x2118 bytes) 000140 1786829002.879049 mapread 0x00be85 thru 0x00d750 (0x18cc bytes) 000141 1786829002.895592 fallocate 0x0199b4 thru 0x019c2c (0x279 bytes) 000142 1786829002.908694 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bb50): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb50): Mirror 2: 0xbd1e949a (stale) CRC-32 checksum value for chunk [0, 0x3bb50): Mirror 1: 0x3e0d4fd2 Mirror 3: 0x3e0d4fd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb50: skipped 000143 1786829003.753293 fallocate 0x03898b thru 0x03cf4c (0x45c2 bytes) 000144 1786829003.795952 mapwrite 0x03115a thru 0x03ffff (0xeea6 bytes) 000145 1786829003.868345 write 0x0055c5 thru 0x00b722 (0x615e bytes) 000146 1786829003.879168 write 0x036304 thru 0x03d6f0 (0x73ed bytes) 000147 1786829003.880464 trunc from 0x040000 to 0x033399 (0xcc66 bytes) 000148 1786829003.908799 punch from 0x00a393 to 0x00e56e (0x41dc bytes) 000149 1786829003.925204 read 0x010f93 thru 0x01a957 (0x99c5 bytes) 000150 1786829003.935649 read 0x006095 thru 0x00cb02 (0x6a6e bytes) 000151 1786829003.947140 mapwrite 0x0154b0 thru 0x01dd5a (0x88ab bytes) 000153 1786829003.971362 punch from 0x00e0f5 to 0x019bd9 (0xbae5 bytes) 000154 1786829003.984495 MIRROR_RESYNC 3 mirrors 000155 1786829005.367379 punch from 0x02b4e6 to 0x02f9f7 (0x4512 bytes) 000156 1786829005.402335 read 0x00d333 thru 0x013124 (0x5df2 bytes) 000157 1786829005.414287 mapread 0x0032fa thru 0x00a02b (0x6d32 bytes) 000158 1786829005.437070 mapwrite 0x034999 thru 0x0354fa (0xb62 bytes) 000159 1786829005.496747 MIRROR_RESYNC 3 mirrors 000160 1786829006.789199 trunc from 0x0354fb to 0x03b8de (0x63e4 bytes) 000161 1786829006.866171 trunc from 0x01796d to 0x022957 (0xafeb bytes) 000162 1786829006.871895 trunc from 0x03b8df to 0x003e70 (0x37a6e bytes) 000163 1786829006.909276 trunc from 0x000e87 to 0x003e70 (0x2fea bytes) 000164 1786829006.914950 mapread 0x002acd thru 0x003e70 (0x13a4 bytes) 000165 1786829006.931218 MIRROR_RESYNC 3 mirrors 000166 1786829007.629186 punch from 0x0017eb to 0x0028bc (0x10d2 bytes) 000167 1786829007.680449 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e71): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e71): Mirror 2: 0xca687500 (stale) Mirror 3: 0xca687500 (stale) CRC-32 checksum value for chunk [0, 0x3e71): Mirror 1: 0xca687500 Mirror 4: 0xca687500 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e71: skipped 000168 1786829008.626994 punch from 0x0001ed to 0x003e70 (0x3c84 bytes) 000169 1786829008.684228 mapread 0x002ae1 thru 0x003364 (0x884 bytes) 000170 1786829008.699442 read 0x000d3f thru 0x003a07 (0x2cc9 bytes) 000171 1786829008.706264 fallocate 0x020771 thru 0x028881 (0x8111 bytes) fallocating to largest ever: 0x40000 000172 1786829008.712295 fallocate 0x03c015 thru 0x03ffff (0x3feb bytes) 000173 1786829008.717821 write 0x003e2b thru 0x009a97 (0x5c6d bytes) 000174 1786829008.728768 mapwrite 0x009d1c thru 0x0161ea (0xc4cf bytes) 000175 1786829008.785555 read 0x01467b thru 0x018765 (0x40eb bytes) 000176 1786829008.790074 read 0x01d7c0 thru 0x028881 (0xb0c2 bytes) 000177 1786829008.804848 trunc from 0x028882 to 0x010aef (0x17d92 bytes) 000178 1786829008.823344 mapread 0x001489 thru 0x00cd23 (0xb89b bytes) 000179 1786829008.825863 read 0x00882f thru 0x00d6d0 (0x4ea2 bytes) 000180 1786829008.826761 trunc from 0x010af0 to 0x009337 (0x77b8 bytes) 000181 1786829008.842965 trunc from 0x009338 to 0x029e43 (0x20b0c bytes) 000182 1786829008.856614 trunc from 0x0174a4 to 0x01b732 (0x428f bytes) 000183 1786829008.863109 fallocate 0x03c87d thru 0x03ffff (0x3783 bytes) 000184 1786829008.869103 mapread 0x00b238 thru 0x01a5c6 (0xf38f bytes) 000185 1786829008.884294 trunc from 0x029e44 to 0x028935 (0x150e bytes) 000186 1786829008.918164 trunc from 0x0222ab to 0x028935 (0x668b bytes) 000187 1786829008.935346 trunc from 0x028936 to 0x03acd4 (0x1239f bytes) 000189 1786829008.985638 write 0x00a457 thru 0x010592 (0x613c bytes) 000190 1786829008.987551 trunc from 0x03acd5 to 0x03bdf8 (0x1124 bytes) 000191 1786829009.011625 mapread 0x01a2f8 thru 0x026bf6 (0xc8ff bytes) 000192 1786829009.046951 mapwrite 0x038a67 thru 0x03ffff (0x7599 bytes) 000193 1786829009.111820 read 0x03ad17 thru 0x03ffff (0x52e9 bytes) 000194 1786829009.113792 trunc from 0x040000 to 0x01886a (0x27795 bytes) 000195 1786829009.141954 punch from 0x0019c8 to 0x001d7c (0x3b5 bytes) 000196 1786829009.148177 fallocate 0x0232a0 thru 0x02f2a4 (0xc005 bytes) 000197 1786829009.157684 trunc from 0x02699c to 0x02f2a4 (0x8909 bytes) 000198 1786829009.164763 trunc from 0x02f2a5 to 0x023b25 (0xb77f bytes) 000199 1786829009.186231 mapwrite 0x036437 thru 0x03ffff (0x9bc9 bytes) 000200 1786829009.249070 mapwrite 0x037d7a thru 0x03ffff (0x8286 bytes) 000202 1786829009.275267 trunc from 0x040000 to 0x037fe4 (0x801b bytes) 000203 1786829009.298905 punch from 0x0328d7 to 0x0353a2 (0x2acc bytes) 000204 1786829009.316267 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37fe5): Mirror 3: 0xdce69400 (stale) Mirror 4: 0xdce69400 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fe5): Mirror 3: 0xdce69400 (stale) Mirror 4: 0xdce69400 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fe5: skipped 000206 1786829009.708575 trunc from 0x037fe5 to 0x007346 (0x30c9e bytes) 000207 1786829009.755813 read 0x0006f4 thru 0x007346 (0x6c53 bytes) 000208 1786829009.770397 trunc from 0x007347 to 0x008ad3 (0x178d bytes) 000209 1786829009.798757 write 0x00b009 thru 0x0113d7 (0x63cf bytes) 000210 1786829009.801010 trunc from 0x00e24b to 0x0113d7 (0x318d bytes) 000211 1786829009.819937 write 0x01cbf1 thru 0x029a38 (0xce48 bytes) 000212 1786829009.822371 mapread 0x014fff thru 0x0184ab (0x34ad bytes) 000213 1786829009.840901 fallocate 0x005b88 thru 0x00c078 (0x64f1 bytes) 000214 1786829009.845970 fallocate 0x01c95e thru 0x0239dc (0x707f bytes) 000215 1786829009.851890 read 0x027f5d thru 0x028c78 (0xd1c bytes) 000216 1786829009.852823 read 0x02136f thru 0x029a38 (0x86ca bytes) 000217 1786829009.853933 fallocate 0x020e12 thru 0x024f91 (0x4180 bytes) 000218 1786829009.859630 trunc from 0x029a39 to 0x0340c2 (0xa68a bytes) 000219 1786829009.875415 trunc from 0x0340c3 to 0x00e572 (0x25b50 bytes) 000220 1786829009.893193 read 0x00e3d3 thru 0x00e572 (0x1a0 bytes) 000222 1786829009.904578 fallocate 0x034c73 thru 0x037a35 (0x2dc3 bytes) 000223 1786829009.911671 fallocate 0x00cc72 thru 0x01ba75 (0xee04 bytes) 000224 1786829009.916964 write 0x02ae1b thru 0x02c4fd (0x16e3 bytes) 000225 1786829009.923444 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c4fe): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c4fe): Mirror 3: 0x8ab97ea2 (stale) Mirror 4: 0x8ab97ea2 (stale) CRC-32 checksum value for chunk [0, 0x2c4fe): Mirror 1: 0xdc566dda Mirror 5: 0xdc566dda lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c4fe: skipped 000226 1786829011.362185 read 0x023580 thru 0x026704 (0x3185 bytes) 000227 1786829011.413481 MIRROR_RESYNC 4 mirrors 000228 1786829012.910327 fallocate 0x00fa6a thru 0x01b9c1 (0xbf58 bytes) 000229 1786829012.979677 write 0x038b93 thru 0x03ffff (0x746d bytes) 000230 1786829012.982998 fallocate 0x037fb6 thru 0x03fe5a (0x7ea5 bytes) 000231 1786829012.990844 punch from 0x000c32 to 0x005fe8 (0x53b7 bytes) 000232 1786829012.999090 punch from 0x001656 to 0x00a1a0 (0x8b4b bytes) 000233 1786829013.008612 read 0x037b59 thru 0x03983e (0x1ce6 bytes) 000234 1786829013.016560 fallocate 0x00e56b thru 0x015ff2 (0x7a88 bytes) 000236 1786829013.024961 trunc from 0x02c6be to 0x034ef7 (0x883a bytes) 000237 1786829013.032030 punch from 0x036cef to 0x039759 (0x2a6b bytes) 000238 1786829013.049447 trunc from 0x028162 to 0x036a76 (0xe915 bytes) 000239 1786829013.058214 punch from 0x03f27d to 0x03ffff (0xd83 bytes) 000240 1786829013.066041 read 0x03d5cd thru 0x03ffd0 (0x2a04 bytes) 000241 1786829013.085399 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb5f32847 (stale) Mirror 4: 0xb5f32847 (stale) Mirror 5: 0xb5f32847 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb475e619 Mirror 6: 0xb475e619 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000242 1786829014.309125 write 0x00e134 thru 0x0124af (0x437c bytes) 000243 1786829014.435993 mapread 0x00e96d thru 0x014902 (0x5f96 bytes) 000245 1786829014.480059 MIRROR_RESYNC 5 mirrors 000246 1786829016.209057 mapwrite 0x01e577 thru 0x02a319 (0xbda3 bytes) 000247 1786829016.338803 write 0x010b9f thru 0x01b00e (0xa470 bytes) 000248 1786829016.357510 MIRROR_RESYNC 5 mirrors 000249 1786829017.619282 read 0x0025cb thru 0x00621d (0x3c53 bytes) 000250 1786829017.627628 read 0x02f2dc thru 0x039044 (0x9d69 bytes) 000251 1786829017.642082 mapread 0x022e20 thru 0x030519 (0xd6fa bytes) 000252 1786829017.644491 write 0x035ef0 thru 0x03ffff (0xa110 bytes) 000253 1786829017.688907 write 0x0267e6 thru 0x02ba64 (0x527f bytes) 000254 1786829017.701915 trunc from 0x040000 to 0x030bc4 (0xf43b bytes) 000255 1786829017.725860 trunc from 0x030bc5 to 0x005360 (0x2b864 bytes) 000256 1786829017.767593 write 0x0011d4 thru 0x008cca (0x7af7 bytes) 000257 1786829017.775333 trunc from 0x008ccb to 0x028dc3 (0x200f9 bytes) 000259 1786829017.801895 fallocate 0x02bb0a thru 0x0353be (0x98b5 bytes) 000260 1786829017.809199 mapwrite 0x024a86 thru 0x033533 (0xeaae bytes) 000261 1786829017.948875 write 0x000cc7 thru 0x008e0a (0x8144 bytes) 000262 1786829017.956661 mapwrite 0x033636 thru 0x0348bd (0x1288 bytes) 000263 1786829017.980680 mapread 0x00caac thru 0x012742 (0x5c97 bytes) 000264 1786829018.019240 trunc from 0x0353bf to 0x01494f (0x20a6f bytes) 000265 1786829018.033592 write 0x0230fa thru 0x0246cb (0x15d2 bytes) 000266 1786829018.034841 trunc from 0x0246cc to 0x00b77f (0x18f4c bytes) 000267 1786829018.050910 mapwrite 0x02dd85 thru 0x033b2f (0x5dab bytes) 000268 1786829018.119780 fallocate 0x023efe thru 0x02933c (0x543f bytes) 000269 1786829018.126345 fallocate 0x010295 thru 0x01fb5d (0xf8c9 bytes) 000271 1786829018.131878 punch from 0x011d6d to 0x021ab5 (0xfd49 bytes) 000272 1786829018.136442 mapread 0x015048 thru 0x0244ed (0xf4a6 bytes) 000273 1786829018.215041 trunc from 0x02a64c to 0x033b2f (0x94e4 bytes) 000274 1786829018.221357 read 0x00c914 thru 0x013d66 (0x7453 bytes) 000275 1786829018.231905 punch from 0x0061be to 0x00cf47 (0x6d8a bytes) 000276 1786829018.238461 read 0x01b91c thru 0x01f441 (0x3b26 bytes) 000277 1786829018.239279 mapread 0x02cfd9 thru 0x031622 (0x464a bytes) 000278 1786829018.240823 fallocate 0x0001c3 thru 0x00c1de (0xc01c bytes) 000280 1786829018.247779 write 0x010b92 thru 0x01861e (0x7a8d bytes) 000281 1786829018.250368 trunc from 0x033b30 to 0x008230 (0x2b8ff bytes) 000282 1786829018.269532 trunc from 0x008231 to 0x03fbbb (0x3798b bytes) 000283 1786829018.290308 punch from 0x005d7e to 0x012d59 (0xcfdc bytes) 000284 1786829018.296340 punch from 0x012869 to 0x01b2c4 (0x8a5c bytes) 000285 1786829018.301658 trunc from 0x03fbbc to 0x006682 (0x39539 bytes) 000286 1786829018.318482 mapread 0x004acd thru 0x006682 (0x1bb6 bytes) 000287 1786829018.327994 trunc from 0x006683 to 0x01b12b (0x14aa9 bytes) 000288 1786829018.342684 write 0x01ce20 thru 0x02a854 (0xda35 bytes) 000289 1786829018.345107 mapread 0x00b9f4 thru 0x011ee6 (0x64f3 bytes) 000290 1786829018.359118 mapread 0x01b0e2 thru 0x01b8fa (0x819 bytes) 000291 1786829018.360595 mapwrite 0x00667f thru 0x00bd2a (0x56ac bytes) 000292 1786829018.379135 fallocate 0x03042b thru 0x03ffff (0xfbd5 bytes) 000293 1786829018.386346 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a855): Mirror 4: 0xe882d6ed (stale) Mirror 5: 0xe882d6ed (stale) Mirror 6: 0xe882d6ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a855): Mirror 4: 0xe882d6ed (stale) Mirror 5: 0xe882d6ed (stale) Mirror 6: 0xe882d6ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a855: skipped 000294 1786829018.812640 mapread 0x001830 thru 0x002b0d (0x12de bytes) 000295 1786829018.829267 fallocate 0x012af8 thru 0x0202bc (0xd7c5 bytes) 000296 1786829018.834849 punch from 0x00a875 to 0x0166bd (0xbe49 bytes) 000297 1786829018.848084 fallocate 0x031c31 thru 0x036a2c (0x4dfc bytes) 000298 1786829018.860598 punch from 0x0094bb to 0x017e39 (0xe97f bytes) 000299 1786829018.873287 write 0x0171f1 thru 0x01d66b (0x647b bytes) 000300 1786829018.917992 write 0x013d47 thru 0x0204cb (0xc785 bytes) 000301 1786829018.949624 write 0x00c561 thru 0x019374 (0xce14 bytes) 000302 1786829018.967601 trunc from 0x02a855 to 0x00dc64 (0x1cbf0 bytes) 000303 1786829019.003504 mapwrite 0x001d3f thru 0x00d99d (0xbc5f bytes) 000304 1786829019.064437 fallocate 0x02d396 thru 0x03971a (0xc385 bytes) 000305 1786829019.076353 mapread 0x0078a2 thru 0x008429 (0xb88 bytes) 000306 1786829019.079550 trunc from 0x033c45 to 0x03971a (0x5ad6 bytes) 000307 1786829019.087530 read 0x0291d3 thru 0x030b13 (0x7941 bytes) 000308 1786829019.102547 trunc from 0x03971b to 0x0236a4 (0x16076 bytes) 000309 1786829019.128586 punch from 0x0038d1 to 0x012003 (0xe733 bytes) 000310 1786829019.137208 write 0x018b5f thru 0x021a2a (0x8ecc bytes) 000311 1786829019.156264 punch from 0x020a05 to 0x02293f (0x1f3b bytes) 000312 1786829019.175192 read 0x008e96 thru 0x01477f (0xb8ea bytes) 000313 1786829019.193993 mapread 0x01f9a2 thru 0x0236a4 (0x3d03 bytes) 000314 1786829019.198308 mapwrite 0x01cf4d thru 0x027f3d (0xaff1 bytes) 000315 1786829019.293954 punch from 0x022375 to 0x024665 (0x22f1 bytes) 000316 1786829019.302253 fallocate 0x00e543 thru 0x01da57 (0xf515 bytes) 000317 1786829019.322176 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27f3e): Mirror 4: 0xa6a93174 (stale) Mirror 6: 0xa6a93174 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27f3e): Mirror 4: 0xa6a93174 (stale) Mirror 6: 0xa6a93174 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27f3e: skipped 000318 1786829019.682585 trunc from 0x027f3e to 0x01e77b (0x97c2 bytes) 000319 1786829019.716129 write 0x03c59e thru 0x03f0c1 (0x2b24 bytes) 000320 1786829019.717685 trunc from 0x03f0c2 to 0x037381 (0x7d40 bytes) 000321 1786829019.732075 punch from 0x0230e1 to 0x02da48 (0xa968 bytes) 000322 1786829019.740477 trunc from 0x037382 to 0x0017a8 (0x35bd9 bytes) 000323 1786829019.751777 read 0x00103c thru 0x0017a8 (0x76d bytes) 000324 1786829019.755506 punch from 0x001500 to 0x0017a8 (0x2a9 bytes) 000325 1786829019.759300 trunc from 0x0017a9 to 0x02badd (0x2a335 bytes) 000326 1786829019.778342 trunc from 0x02bade to 0x032873 (0x6d96 bytes) 000327 1786829019.792792 write 0x02b925 thru 0x030b15 (0x51f1 bytes) 000328 1786829019.800904 trunc from 0x01e864 to 0x023fcc (0x5769 bytes) 000330 1786829019.807791 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32874): Mirror 4: 0x415869e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32874): Mirror 4: 0x415869e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32874: skipped 000331 1786829020.092283 punch from 0x02ed0d to 0x032873 (0x3b67 bytes) 000332 1786829020.097046 read 0x0159f6 thru 0x01778b (0x1d96 bytes) 000333 1786829020.105996 punch from 0x003f2c to 0x008982 (0x4a57 bytes) 000334 1786829020.111793 trunc from 0x00572b to 0x00d6dc (0x7fb2 bytes) 000336 1786829020.120520 punch from 0x025558 to 0x032873 (0xd31c bytes) 000337 1786829020.126019 trunc from 0x00f673 to 0x012d0a (0x3698 bytes) 000338 1786829020.131886 fallocate 0x00bf54 thru 0x014f53 (0x9000 bytes) 000339 1786829020.137496 mapwrite 0x00429e thru 0x0135b3 (0xf316 bytes) 000340 1786829020.194671 write 0x02046e thru 0x028b1b (0x86ae bytes) 000341 1786829020.211356 fallocate 0x004af5 thru 0x0083a9 (0x38b5 bytes) 000342 1786829020.217756 fallocate 0x02e34f thru 0x02e5ce (0x280 bytes) 000343 1786829020.229939 MIRROR_RESYNC 2 mirrors 000344 1786829020.847823 mapread 0x023bee thru 0x024451 (0x864 bytes) 000345 1786829020.859959 mapread 0x01d741 thru 0x0285ec (0xaeac bytes) 000346 1786829020.887388 MIRROR_EXTEND to 3 mirrors 000348 1786829021.562050 trunc from 0x032874 to 0x03f64d (0xcdda bytes) 000349 1786829021.651992 trunc from 0x03f64e to 0x031eb9 (0xd794 bytes) 000350 1786829021.680492 trunc from 0x031eba to 0x02a20a (0x7caf bytes) 000351 1786829021.699812 fallocate 0x02c45e thru 0x02ea13 (0x25b6 bytes) 000352 1786829021.706500 mapread 0x01b5f0 thru 0x0224c0 (0x6ed1 bytes) 000353 1786829021.760293 trunc from 0x02a20b to 0x0135dd (0x16c2d bytes) 000354 1786829021.818508 MIRROR_RESYNC 3 mirrors 000355 1786829022.319818 fallocate 0x01f7b8 thru 0x028be5 (0x942e bytes) 000356 1786829022.353547 read 0x011963 thru 0x0135dd (0x1c7b bytes) 000357 1786829022.369515 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x135de): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x135de): Mirror 4: 0x15d7b79a (stale) Mirror 5: 0x15d7b79a (stale) CRC-32 checksum value for chunk [0, 0x135de): Mirror 1: 0x15d7b79a Mirror 6: 0x15d7b79a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x135de: skipped 000358 1786829023.403744 fallocate 0x01e498 thru 0x023b7a (0x56e3 bytes) 000359 1786829023.459102 punch from 0x005c01 to 0x011eea (0xc2ea bytes) 000360 1786829023.469281 mapwrite 0x015797 thru 0x022e09 (0xd673 bytes) 000361 1786829023.532670 fallocate 0x006c2a thru 0x013d83 (0xd15a bytes) 000362 1786829023.541027 fallocate 0x01c75b thru 0x02653b (0x9de1 bytes) 000363 1786829023.546806 trunc from 0x023b7b to 0x02debd (0xa343 bytes) 000365 1786829023.599519 mapwrite 0x026b2f thru 0x0367ae (0xfc80 bytes) 000366 1786829023.675014 trunc from 0x0367af to 0x037bcd (0x141f bytes) 000367 1786829023.700692 trunc from 0x037bce to 0x02238b (0x15842 bytes) 000368 1786829023.736371 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2238c): Mirror 4: 0x64b4f430 (stale) Mirror 5: 0x64b4f430 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2238c): Mirror 4: 0x64b4f430 (stale) Mirror 5: 0x64b4f430 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2238c: skipped 000369 1786829024.063093 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2238c): Mirror 5: 0x64b4f430 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2238c): Mirror 5: 0x64b4f430 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2238c: skipped 000370 1786829024.382902 read 0x007c03 thru 0x015a01 (0xddff bytes) 000371 1786829024.403676 trunc from 0x003c53 to 0x009610 (0x59be bytes) 000372 1786829024.422466 read 0x00c155 thru 0x010159 (0x4005 bytes) 000373 1786829024.423293 trunc from 0x02238c to 0x00cfca (0x153c1 bytes) 000374 1786829024.468351 punch from 0x00169e to 0x00cfca (0xb92d bytes) 000375 1786829024.475843 write 0x02b927 thru 0x038977 (0xd051 bytes) 000376 1786829024.480268 mapwrite 0x00622f thru 0x013640 (0xd412 bytes) 000377 1786829024.547819 write 0x037cb5 thru 0x03ffff (0x834b bytes) 000378 1786829024.568979 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x3173fe6d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xef61529f Mirror 6: 0xef61529f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000379 1786829025.698432 trunc from 0x040000 to 0x0315a3 (0xea5c bytes) 000380 1786829025.765681 fallocate 0x022cc2 thru 0x0232d0 (0x60f bytes) 000383 1786829025.771969 fallocate 0x0042b6 thru 0x004cbd (0xa08 bytes) 000384 1786829025.784041 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x315a4): Mirror 6: 0xf3a25b4e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x315a4): Mirror 6: 0xf3a25b4e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x315a4: skipped 000386 1786829026.047893 mapwrite 0x022eb8 thru 0x03029f (0xd3e8 bytes) 000387 1786829026.112243 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x315a4): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x315a4): Mirror 6: 0xf3a25b4e (stale) CRC-32 checksum value for chunk [0, 0x315a4): Mirror 1: 0xc760ad7 Mirror 7: 0xc760ad7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x315a4: skipped 000388 1786829026.906967 mapread 0x01a0cd thru 0x01af56 (0xe8a bytes) 000389 1786829026.934823 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000390 1786829027.361001 trunc from 0x026a85 to 0x02f449 (0x89c5 bytes) 000391 1786829027.402970 write 0x03540a thru 0x039184 (0x3d7b bytes) 000392 1786829027.414833 trunc from 0x01184a to 0x017d2c (0x64e3 bytes) 000393 1786829027.420053 punch from 0x032df7 to 0x039184 (0x638e bytes) 000394 1786829027.436160 read 0x02cf9a thru 0x02f96a (0x29d1 bytes) 000395 1786829027.444458 mapwrite 0x012052 thru 0x01a5cd (0x857c bytes) 000396 1786829027.490210 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000397 1786829027.622445 read 0x02ccc2 thru 0x039184 (0xc4c3 bytes) 000398 1786829027.634746 punch from 0x007758 to 0x00fc91 (0x853a bytes) 000399 1786829027.641886 mapread 0x025d57 thru 0x033e31 (0xe0db bytes) 000400 1786829027.664897 punch from 0x018b5a to 0x023a89 (0xaf30 bytes) 000402 1786829027.673540 mapwrite 0x01d765 thru 0x01fdd8 (0x2674 bytes) 000404 1786829027.700845 write 0x03a1fc thru 0x03ffff (0x5e04 bytes) 000407 1786829027.708882 fallocate 0x00a590 thru 0x0108a9 (0x631a bytes) 000409 1786829027.720139 trunc from 0x038d5f to 0x039a4e (0xcf0 bytes) 000410 1786829027.725550 mapwrite 0x0253f1 thru 0x030ab8 (0xb6c8 bytes) 000411 1786829027.748439 punch from 0x006885 to 0x01111d (0xa899 bytes) 000413 1786829027.754957 punch from 0x0278a9 to 0x02ae4b (0x35a3 bytes) 000415 1786829027.760243 fallocate 0x0017ee thru 0x00e246 (0xca59 bytes) 000416 1786829027.765484 mapread 0x008412 thru 0x01003d (0x7c2c bytes) 000419 1786829027.779046 write 0x01712a thru 0x019faf (0x2e86 bytes) 000420 1786829027.780559 write 0x00bad5 thru 0x01a0ca (0xe5f6 bytes) 000421 1786829027.784496 read 0x03eb45 thru 0x03ff39 (0x13f5 bytes) 000422 1786829027.785960 mapread 0x00b9be thru 0x01a46a (0xeaad bytes) 000423 1786829027.788213 trunc from 0x040000 to 0x013880 (0x2c77f bytes) 000424 1786829027.830678 read 0x00ab3a thru 0x013880 (0x8d47 bytes) 000425 1786829027.832780 mapwrite 0x030509 thru 0x03cf3c (0xca34 bytes) 000426 1786829027.896798 read 0x00f3d2 thru 0x016403 (0x7032 bytes) 000427 1786829027.897770 mapwrite 0x016635 thru 0x01824a (0x1c16 bytes) 000429 1786829027.912970 mapread 0x01777f thru 0x0204ec (0x8d6e bytes) 000430 1786829027.914091 fallocate 0x01c6a9 thru 0x01d3a9 (0xd01 bytes) 000431 1786829027.919455 trunc from 0x00f942 to 0x01612f (0x67ee bytes) 000432 1786829027.924828 punch from 0x02724d to 0x032a84 (0xb838 bytes) 000433 1786829027.936373 MIRROR_EXTEND to 2 mirrors 000434 1786829028.899486 mapwrite 0x03e34f thru 0x03ffff (0x1cb1 bytes) 000435 1786829028.980389 write 0x03ddbc thru 0x03ea0b (0xc50 bytes) 000436 1786829028.986375 fallocate 0x031073 thru 0x03f118 (0xe0a6 bytes) 000437 1786829029.001947 write 0x019f67 thru 0x01df8b (0x4025 bytes) 000439 1786829029.024816 fallocate 0x03b8b9 thru 0x03ffff (0x4747 bytes) 000440 1786829029.030622 fallocate 0x022445 thru 0x029bae (0x776a bytes) 000441 1786829029.036752 trunc from 0x040000 to 0x023347 (0x1ccb8 bytes) 000442 1786829029.060868 trunc from 0x023348 to 0x0294e9 (0x61a2 bytes) 000443 1786829029.087920 MIRROR_RESYNC 2 mirrors 000444 1786829029.852394 trunc from 0x00c9cc to 0x00f6ef (0x2d24 bytes) 000445 1786829029.881342 mapread 0x025b41 thru 0x0294e9 (0x39a9 bytes) 000447 1786829029.901484 fallocate 0x031535 thru 0x03a830 (0x92fc bytes) 000448 1786829029.906498 read 0x030c47 thru 0x035428 (0x47e2 bytes) 000449 1786829029.916663 fallocate 0x021ab5 thru 0x02c899 (0xade5 bytes) 000451 1786829029.922601 write 0x022a0b thru 0x030161 (0xd757 bytes) 000452 1786829029.926102 mapwrite 0x01e9ea thru 0x020a98 (0x20af bytes) 000454 1786829029.963134 trunc from 0x03a831 to 0x030a4f (0x9de1 bytes) 000455 1786829029.987774 mapwrite 0x01d230 thru 0x022f7a (0x5d4b bytes) 000457 1786829030.008623 mapwrite 0x00a4cc thru 0x0183fa (0xdf2f bytes) 000458 1786829030.048841 read 0x00c0f9 thru 0x0173a7 (0xb2af bytes) 000459 1786829030.049764 read 0x0072bb thru 0x00a9d4 (0x371a bytes) 000460 1786829030.050444 mapread 0x0056e8 thru 0x00d56b (0x7e84 bytes) 000461 1786829030.052640 write 0x033ecd thru 0x03ffff (0xc133 bytes) 000462 1786829030.055472 punch from 0x03d659 to 0x03ffff (0x29a7 bytes) 000463 1786829030.073854 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000464 1786829030.267543 read 0x00ab6e thru 0x00c51d (0x19b0 bytes) 000465 1786829030.284665 punch from 0x026316 to 0x035df3 (0xfade bytes) 000466 1786829030.302250 read 0x00ae97 thru 0x010b34 (0x5c9e bytes) 000467 1786829030.323573 read 0x0178c8 thru 0x017e06 (0x53f bytes) 000468 1786829030.324373 write 0x001e73 thru 0x006a6f (0x4bfd bytes) 000469 1786829030.331330 read 0x03a63c thru 0x03ffff (0x59c4 bytes) 000470 1786829030.332258 mapread 0x029240 thru 0x033377 (0xa138 bytes) 000471 1786829030.343718 trunc from 0x040000 to 0x001569 (0x3ea96 bytes) 000472 1786829030.375454 trunc from 0x00156a to 0x0241d3 (0x22c6a bytes) 000473 1786829030.393227 fallocate 0x02b0a7 thru 0x035032 (0x9f8c bytes) 000474 1786829030.409113 read 0x012750 thru 0x022740 (0xfff1 bytes) 000475 1786829030.424841 punch from 0x01f8be to 0x0233c2 (0x3b05 bytes) 000476 1786829030.430082 mapread 0x0031d0 thru 0x00c927 (0x9758 bytes) 000477 1786829030.431239 mapwrite 0x006f3b thru 0x007625 (0x6eb bytes) 000478 1786829030.465762 write 0x001f29 thru 0x007e1d (0x5ef5 bytes) 000479 1786829030.470493 mapread 0x02b33b thru 0x035032 (0x9cf8 bytes) 000480 1786829030.472177 trunc from 0x035033 to 0x018c65 (0x1c3cd bytes) 000481 1786829030.491873 mapread 0x005e0f thru 0x00fed9 (0xa0cb bytes) 000482 1786829030.494127 read 0x0060b4 thru 0x00bbd3 (0x5b20 bytes) 000483 1786829030.495671 trunc from 0x017d81 to 0x018c65 (0xee5 bytes) 000484 1786829030.500748 trunc from 0x018c66 to 0x038b92 (0x1ff2d bytes) 000485 1786829030.516659 punch from 0x024a6c to 0x031676 (0xcc0b bytes) 000486 1786829030.522497 mapread 0x01b673 thru 0x01eb3f (0x34cd bytes) 000487 1786829030.533820 trunc from 0x0299e3 to 0x031daa (0x83c8 bytes) 000488 1786829030.539423 write 0x02cb3d thru 0x032316 (0x57da bytes) 000489 1786829030.546560 mapread 0x00b988 thru 0x01b3d0 (0xfa49 bytes) 000490 1786829030.549623 mapread 0x006cd4 thru 0x00bf31 (0x525e bytes) 000491 1786829030.551431 mapwrite 0x01fbde thru 0x0232a2 (0x36c5 bytes) 000493 1786829030.571325 mapread 0x013345 thru 0x01556a (0x2226 bytes) 000494 1786829030.573147 write 0x01a84b thru 0x01cf2c (0x26e2 bytes) 000496 1786829030.575469 trunc from 0x0211d6 to 0x02681d (0x5648 bytes) 000497 1786829030.580687 mapwrite 0x00f542 thru 0x014eb3 (0x5972 bytes) 000500 1786829030.601144 mapread 0x01b427 thru 0x021dba (0x6994 bytes) 000501 1786829030.606820 mapread 0x005237 thru 0x0087b0 (0x357a bytes) 000503 1786829030.609585 read 0x0082d9 thru 0x00c34a (0x4072 bytes) 000504 1786829030.610858 write 0x01e3dd thru 0x02b0fa (0xcd1e bytes) 000505 1786829030.618422 fallocate 0x024b1c thru 0x02cf73 (0x8458 bytes) 000506 1786829030.623591 fallocate 0x00c202 thru 0x010912 (0x4711 bytes) 000507 1786829030.629465 trunc from 0x021896 to 0x02b54d (0x9cb8 bytes) 000508 1786829030.646824 write 0x030ce2 thru 0x034833 (0x3b52 bytes) 000509 1786829030.648165 trunc from 0x026776 to 0x02750c (0xd97 bytes) 000510 1786829030.653132 mapwrite 0x01aed4 thru 0x02721c (0xc349 bytes) 000512 1786829030.685538 fallocate 0x025a14 thru 0x035063 (0xf650 bytes) 000513 1786829030.690996 trunc from 0x038b93 to 0x00fb97 (0x28ffb bytes) 000514 1786829030.706583 fallocate 0x01401e thru 0x01b389 (0x736c bytes) 000515 1786829030.712421 trunc from 0x01a9c8 to 0x01b389 (0x9c2 bytes) 000516 1786829030.717778 mapread 0x0083a4 thru 0x01783e (0xf49b bytes) 000517 1786829030.754484 trunc from 0x00d07d to 0x013d4d (0x6cd1 bytes) 000518 1786829030.763860 trunc from 0x01b38a to 0x03db67 (0x227de bytes) 000519 1786829030.776342 fallocate 0x029eb9 thru 0x0355db (0xb723 bytes) 000520 1786829030.782462 punch from 0x0114ab to 0x01e169 (0xccbf bytes) 000521 1786829030.787875 mapwrite 0x013fa8 thru 0x01a9a7 (0x6a00 bytes) 000522 1786829030.846984 mapwrite 0x023afb thru 0x031743 (0xdc49 bytes) 000523 1786829030.946686 punch from 0x036cc2 to 0x03db67 (0x6ea6 bytes) 000524 1786829030.952490 trunc from 0x028197 to 0x02de17 (0x5c81 bytes) 000525 1786829030.957612 trunc from 0x025521 to 0x02a071 (0x4b51 bytes) 000526 1786829030.963500 mapread 0x01c5c1 thru 0x022f00 (0x6940 bytes) 000527 1786829031.000865 mapwrite 0x01d957 thru 0x0269b0 (0x905a bytes) 000528 1786829031.030152 trunc from 0x03db68 to 0x026bac (0x16fbb bytes) 000529 1786829031.050286 trunc from 0x01f918 to 0x026bac (0x7295 bytes) 000530 1786829031.055844 mapread 0x016224 thru 0x024b7b (0xe958 bytes) 000531 1786829031.091375 trunc from 0x026bad to 0x015d4a (0x10e62 bytes) 000532 1786829031.106573 mapwrite 0x0279cc thru 0x03373a (0xbd6f bytes) 000533 1786829031.220065 read 0x012fdc thru 0x021dce (0xedf3 bytes) 000534 1786829031.235426 punch from 0x0212c5 to 0x02ebc6 (0xd902 bytes) 000535 1786829031.242422 mapwrite 0x004d26 thru 0x007bf3 (0x2ece bytes) 000536 1786829031.262257 punch from 0x0204e2 to 0x02f545 (0xf064 bytes) 000537 1786829031.269080 mapread 0x006275 thru 0x01475b (0xe4e7 bytes) 000539 1786829031.273475 mapwrite 0x00dfb2 thru 0x01c88e (0xe8dd bytes) 000540 1786829031.293097 write 0x00a60b thru 0x0154fe (0xaef4 bytes) 000541 1786829031.294778 punch from 0x0129cc to 0x02101f (0xe654 bytes) 000542 1786829031.310804 fallocate 0x02f0e9 thru 0x02f1c7 (0xdf bytes) 000543 1786829031.315975 mapwrite 0x01dac7 thru 0x02155f (0x3a99 bytes) 000544 1786829031.359190 trunc from 0x015dd8 to 0x01b82b (0x5a54 bytes) 000546 1786829031.366806 fallocate 0x01d913 thru 0x01ef3b (0x1629 bytes) 000547 1786829031.373346 fallocate 0x000386 thru 0x001df7 (0x1a72 bytes) 000548 1786829031.381993 punch from 0x006924 to 0x0111fd (0xa8da bytes) 000549 1786829031.388302 trunc from 0x007cfd to 0x0089dd (0xce1 bytes) 000550 1786829031.393729 mapwrite 0x028f34 thru 0x02a27f (0x134c bytes) 000551 1786829031.421141 trunc from 0x001ba8 to 0x0061ba (0x4613 bytes) 000552 1786829031.427879 trunc from 0x03373b to 0x02a7f6 (0x8f44 bytes) 000553 1786829031.450870 read 0x000bb9 thru 0x008451 (0x7899 bytes) 000555 1786829031.463415 fallocate 0x0342f9 thru 0x03c325 (0x802d bytes) 000556 1786829031.469003 read 0x0046bc thru 0x00fd25 (0xb66a bytes) 000557 1786829031.472906 mapread 0x00f99e thru 0x01c4d1 (0xcb34 bytes) 000558 1786829031.475049 write 0x032a6d thru 0x03ffff (0xd593 bytes) 000559 1786829031.479966 punch from 0x03c689 to 0x03ffff (0x3977 bytes) 000560 1786829031.505683 fallocate 0x01459f thru 0x016887 (0x22e9 bytes) 000561 1786829031.520314 trunc from 0x040000 to 0x0180b0 (0x27f4f bytes) 000562 1786829031.553054 fallocate 0x03e78f thru 0x03fe15 (0x1687 bytes) 000563 1786829031.561276 mapwrite 0x014eac thru 0x018ce9 (0x3e3e bytes) 000564 1786829031.610164 fallocate 0x035f82 thru 0x03b739 (0x57b8 bytes) 000565 1786829031.622011 read 0x008f3e thru 0x01472f (0xb7f2 bytes) 000566 1786829031.623453 write 0x0391e3 thru 0x03ffff (0x6e1d bytes) 000567 1786829031.637309 write 0x026747 thru 0x02ddba (0x7674 bytes) 000568 1786829031.639985 trunc from 0x040000 to 0x014689 (0x2b976 bytes) 000569 1786829031.663123 trunc from 0x00b071 to 0x014689 (0x9619 bytes) 000570 1786829031.673656 trunc from 0x001df9 to 0x0024dc (0x6e4 bytes) 000571 1786829031.683795 mapread 0x00e7bc thru 0x0135cd (0x4e12 bytes) 000572 1786829031.784487 trunc from 0x006e93 to 0x00aedf (0x404d bytes) 000573 1786829031.807113 trunc from 0x010286 to 0x014689 (0x4404 bytes) 000575 1786829031.818934 fallocate 0x0050f8 thru 0x00af9e (0x5ea7 bytes) 000576 1786829031.832458 fallocate 0x027353 thru 0x02fdde (0x8a8c bytes) 000577 1786829031.841807 read 0x02168d thru 0x02f56e (0xdee2 bytes) 000578 1786829031.872654 fallocate 0x00a59c thru 0x018239 (0xdc9e bytes) 000579 1786829031.878646 read 0x00b8e4 thru 0x00d9ac (0x20c9 bytes) 000582 1786829031.880608 write 0x0203ff thru 0x026cfc (0x68fe bytes) 000583 1786829031.892115 mapwrite 0x0212a9 thru 0x02f1c8 (0xdf20 bytes) 000584 1786829031.951059 punch from 0x01db7d to 0x023211 (0x5695 bytes) 000585 1786829031.964118 mapwrite 0x00065b thru 0x00fbf3 (0xf599 bytes) 000586 1786829032.038316 mapwrite 0x02c35f thru 0x02f663 (0x3305 bytes) 000588 1786829032.056619 trunc from 0x02fddf to 0x03f4b5 (0xf6d7 bytes) 000589 1786829032.079366 trunc from 0x027d8a to 0x03287a (0xaaf1 bytes) 000590 1786829032.089050 trunc from 0x03f4b6 to 0x0143d7 (0x2b0de bytes) 000591 1786829032.112417 read 0x00bed1 thru 0x00e42c (0x255c bytes) 000592 1786829032.113474 mapread 0x006564 thru 0x00fe8a (0x9927 bytes) 000593 1786829032.117800 trunc from 0x0143d8 to 0x0366e4 (0x2230d bytes) 000594 1786829032.145358 trunc from 0x0366e5 to 0x005a4b (0x30c99 bytes) 000595 1786829032.191531 trunc from 0x0038eb to 0x0051d2 (0x18e8 bytes) 000596 1786829032.202684 fallocate 0x039aea thru 0x03a22b (0x742 bytes) 000597 1786829032.210608 mapwrite 0x028c65 thru 0x030c95 (0x8031 bytes) 000598 1786829032.292689 trunc from 0x03a22c to 0x0060f2 (0x34139 bytes) 000599 1786829032.314410 trunc from 0x0025a3 to 0x005abb (0x3519 bytes) 000600 1786829032.323248 mapwrite 0x03d51a thru 0x03ffff (0x2ae6 bytes) 000601 1786829032.379813 write 0x02109d thru 0x022df8 (0x1d5c bytes) 000602 1786829032.395410 write 0x01bbee thru 0x02b8d2 (0xfce5 bytes) 000603 1786829032.416679 mapwrite 0x00921b thru 0x00ec34 (0x5a1a bytes) 000604 1786829032.481428 trunc from 0x040000 to 0x00dde5 (0x3221a bytes) 000606 1786829032.504777 punch from 0x00854f to 0x00dde5 (0x5897 bytes) 000607 1786829032.512118 mapread 0x00cb19 thru 0x00dde5 (0x12cd bytes) 000608 1786829032.527266 mapwrite 0x01cfc8 thru 0x027d08 (0xad41 bytes) 000609 1786829032.670440 punch from 0x005248 to 0x00f2b5 (0xa06e bytes) 000610 1786829032.677693 trunc from 0x01daf3 to 0x01eba0 (0x10ae bytes) 000612 1786829032.689725 read 0x01e6e8 thru 0x027d08 (0x9621 bytes) 000613 1786829032.717536 write 0x03f0b1 thru 0x03ffff (0xf4f bytes) 000614 1786829032.718792 write 0x01a62c thru 0x01b8f1 (0x12c6 bytes) 000616 1786829032.726081 read 0x010833 thru 0x011d54 (0x1522 bytes) 000617 1786829032.727361 read 0x00f9d4 thru 0x01b380 (0xb9ad bytes) 000618 1786829032.729622 trunc from 0x040000 to 0x010639 (0x2f9c6 bytes) 000619 1786829032.751358 trunc from 0x00b2cf to 0x00efd2 (0x3d04 bytes) 000620 1786829032.759681 write 0x006df8 thru 0x00bd13 (0x4f1c bytes) 000621 1786829032.767194 mapread 0x00c8c1 thru 0x010639 (0x3d79 bytes) 000622 1786829032.777109 mapread 0x00ae67 thru 0x00f82c (0x49c6 bytes) 000623 1786829032.779044 mapread 0x0014e2 thru 0x00febb (0xe9da bytes) 000624 1786829032.782056 trunc from 0x01063a to 0x016ada (0x64a1 bytes) 000625 1786829032.855500 MIRROR_EXTEND to 2 mirrors 000626 1786829033.556067 trunc from 0x01605f to 0x016ada (0xa7c bytes) 000627 1786829033.582089 punch from 0x003666 to 0x005786 (0x2121 bytes) 000628 1786829033.589575 punch from 0x0072b8 to 0x007f8d (0xcd6 bytes) 000629 1786829033.594324 fallocate 0x028680 thru 0x03086b (0x81ec bytes) 000630 1786829033.600110 mapread 0x011277 thru 0x01b854 (0xa5de bytes) 000631 1786829033.618971 trunc from 0x015533 to 0x024ef2 (0xf9c0 bytes) 000632 1786829033.624502 trunc from 0x02d595 to 0x03086b (0x32d7 bytes) 000633 1786829033.628902 mapread 0x01a432 thru 0x020eea (0x6ab9 bytes) 000634 1786829033.639082 read 0x007655 thru 0x009d7f (0x272b bytes) 000635 1786829033.650838 fallocate 0x011327 thru 0x011b0f (0x7e9 bytes) 000636 1786829033.658853 fallocate 0x01c7a8 thru 0x021bb0 (0x5409 bytes) 000638 1786829033.672230 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000639 1786829033.825414 punch from 0x01c626 to 0x02b71e (0xf0f9 bytes) 000640 1786829033.830443 mapwrite 0x0297ef thru 0x02d242 (0x3a54 bytes) 000641 1786829033.876140 trunc from 0x03086c to 0x03e627 (0xddbc bytes) 000642 1786829033.905325 mapread 0x034ba3 thru 0x03e627 (0x9a85 bytes) 000643 1786829033.924431 write 0x01339e thru 0x018095 (0x4cf8 bytes) 000644 1786829033.935736 write 0x0259d5 thru 0x0312c2 (0xb8ee bytes) 000645 1786829033.938450 mapread 0x038d2e thru 0x03e627 (0x58fa bytes) 000646 1786829033.941185 mapwrite 0x002206 thru 0x00f080 (0xce7b bytes) 000647 1786829033.982292 MIRROR_EXTEND to 2 mirrors 000648 1786829035.278225 mapwrite 0x02a2ab thru 0x032cf1 (0x8a47 bytes) 000649 1786829035.409548 trunc from 0x02c634 to 0x038c98 (0xc665 bytes) 000650 1786829035.430120 fallocate 0x032484 thru 0x03e4f0 (0xc06d bytes) 000651 1786829035.446449 trunc from 0x03e628 to 0x03cc95 (0x1992 bytes) 000652 1786829035.490365 punch from 0x02c745 to 0x035cfa (0x95b6 bytes) 000653 1786829035.506261 mapwrite 0x00fe2d thru 0x01a9b6 (0xab8a bytes) 000654 1786829035.567888 mapread 0x00dad5 thru 0x0183a3 (0xa8cf bytes) 000655 1786829035.595540 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3cc96): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc96): Mirror 2: 0x92ef8d09 (stale) CRC-32 checksum value for chunk [0, 0x3cc96): Mirror 1: 0xa194a929 Mirror 3: 0xa194a929 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc96: skipped 000658 1786829036.604237 write 0x014b1f thru 0x015f2c (0x140e bytes) 000659 1786829036.662583 punch from 0x010ee7 to 0x01c06b (0xb185 bytes) 000660 1786829036.678154 read 0x03122e thru 0x031560 (0x333 bytes) 000661 1786829036.683356 fallocate 0x010230 thru 0x01155b (0x132c bytes) 000662 1786829036.691260 trunc from 0x0132b6 to 0x01e9e5 (0xb730 bytes) 000663 1786829036.696777 read 0x02ab88 thru 0x034a4a (0x9ec3 bytes) 000664 1786829036.707931 mapread 0x03815d thru 0x03c905 (0x47a9 bytes) 000665 1786829036.709363 write 0x03a5a4 thru 0x03ffff (0x5a5c bytes) 000666 1786829036.711560 trunc from 0x040000 to 0x036f1b (0x90e4 bytes) 000667 1786829036.732175 mapread 0x019783 thru 0x024e7f (0xb6fd bytes) 000668 1786829036.738209 MIRROR_RESYNC 3 mirrors 000669 1786829037.540472 MIRROR_RESYNC 3 mirrors 000670 1786829037.902189 mapwrite 0x014fe3 thru 0x016632 (0x1650 bytes) 000671 1786829038.013188 trunc from 0x028d9f to 0x033dfb (0xb05d bytes) 000672 1786829038.030850 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36f1c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f1c): Mirror 2: 0x2dbd3eed (stale) Mirror 3: 0x2dbd3eed (stale) CRC-32 checksum value for chunk [0, 0x36f1c): Mirror 1: 0xf1dbfbba Mirror 4: 0xf1dbfbba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f1c: skipped 000673 1786829039.150040 fallocate 0x03e98a thru 0x03ffff (0x1676 bytes) 000674 1786829039.179970 read 0x02f6f3 thru 0x03b1db (0xbae9 bytes) 000675 1786829039.188412 mapread 0x03a6f0 thru 0x03c8fa (0x220b bytes) 000676 1786829039.194199 write 0x026ef2 thru 0x02b773 (0x4882 bytes) 000677 1786829039.202775 mapread 0x02af5a thru 0x0363d6 (0xb47d bytes) 000678 1786829039.213143 punch from 0x02532e to 0x02fd1c (0xa9ef bytes) 000679 1786829039.226670 read 0x005da1 thru 0x00a644 (0x48a4 bytes) 000680 1786829039.250501 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xef8e211c (stale) Mirror 4: 0xd9b220ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xef8e211c (stale) Mirror 4: 0xd9b220ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000681 1786829039.471307 trunc from 0x03ce07 to 0x03fc18 (0x2e12 bytes) 000682 1786829039.474992 fallocate 0x005ef8 thru 0x00e3b0 (0x84b9 bytes) 000683 1786829039.478193 trunc from 0x040000 to 0x02548f (0x1ab70 bytes) 000684 1786829039.490525 write 0x010cf4 thru 0x0128d8 (0x1be5 bytes) 000685 1786829039.496464 mapwrite 0x021543 thru 0x0220e8 (0xba6 bytes) 000686 1786829039.524336 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x25490): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25490): Mirror 3: 0xc0ae5501 (stale) Mirror 4: 0x314b3c4a (stale) CRC-32 checksum value for chunk [0, 0x25490): Mirror 1: 0xb5746dae Mirror 5: 0xb5746dae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25490: skipped 000687 1786829040.286145 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25490): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25490): Mirror 3: 0xc0ae5501 (stale) Mirror 4: 0x314b3c4a (stale) CRC-32 checksum value for chunk [0, 0x25490): Mirror 1: 0xb5746dae Mirror 5: 0xb5746dae Mirror 6: 0xb5746dae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25490: skipped 000689 1786829041.567963 fallocate 0x025c06 thru 0x035938 (0xfd33 bytes) 000690 1786829041.620117 mapread 0x0154ca thru 0x01ab7c (0x56b3 bytes) 000691 1786829041.648331 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35939): Mirror 4: 0x8a59438e (stale) Mirror 5: 0x5862a4e4 (stale) Mirror 6: 0x5862a4e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35939): Mirror 4: 0x8a59438e (stale) Mirror 5: 0x5862a4e4 (stale) Mirror 6: 0x5862a4e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35939: skipped 000692 1786829042.097142 mapread 0x026c2e thru 0x02af7e (0x4351 bytes) 000693 1786829042.125539 MIRROR_RESYNC 4 mirrors 000694 1786829043.771679 fallocate 0x00602f thru 0x013aa6 (0xda78 bytes) 000695 1786829043.801367 fallocate 0x00be9a thru 0x011cb1 (0x5e18 bytes) 000696 1786829043.804474 punch from 0x027b45 to 0x035938 (0xddf4 bytes) 000697 1786829043.809433 read 0x00b125 thru 0x00b55c (0x438 bytes) 000699 1786829043.813313 trunc from 0x010bd0 to 0x014523 (0x3954 bytes) 000700 1786829043.816640 write 0x019a5e thru 0x01e98d (0x4f30 bytes) 000702 1786829043.823267 fallocate 0x01ef1d thru 0x02a6d1 (0xb7b5 bytes) 000703 1786829043.826477 mapwrite 0x0384b8 thru 0x03c87e (0x43c7 bytes) 000704 1786829043.876304 read 0x0370da thru 0x0381dc (0x1103 bytes) 000705 1786829043.877163 punch from 0x0364c9 to 0x03c2f3 (0x5e2b bytes) 000706 1786829043.885264 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c87f): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c87f): Mirror 4: 0x71526e81 (stale) Mirror 5: 0x71526e81 (stale) Mirror 6: 0x71526e81 (stale) CRC-32 checksum value for chunk [0, 0x3c87f): Mirror 1: 0x86d9b3c8 Mirror 7: 0x86d9b3c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c87f: skipped 000707 1786829044.753395 mapread 0x0100ea thru 0x013f91 (0x3ea8 bytes) 000709 1786829044.838345 write 0x034f97 thru 0x03ffff (0xb069 bytes) 000710 1786829044.884509 mapread 0x0139ff thru 0x01a452 (0x6a54 bytes) 000711 1786829044.899583 MIRROR_RESYNC 5 mirrors 000712 1786829046.720871 write 0x01d478 thru 0x02d14d (0xfcd6 bytes) 000713 1786829046.750352 mapwrite 0x00e6e0 thru 0x019bf2 (0xb513 bytes) 000714 1786829046.831693 punch from 0x023df9 to 0x032b1d (0xed25 bytes) 000715 1786829046.841272 punch from 0x00d16e to 0x0101a1 (0x3034 bytes) 000716 1786829046.851009 punch from 0x007df6 to 0x010ad8 (0x8ce3 bytes) 000717 1786829046.858955 fallocate 0x0076e3 thru 0x01479d (0xd0bb bytes) 000719 1786829046.865801 mapwrite 0x020973 thru 0x0275bd (0x6c4b bytes) 000720 1786829046.901605 trunc from 0x027e31 to 0x032b38 (0xad08 bytes) 000721 1786829046.909523 mapread 0x0214b7 thru 0x02e9bb (0xd505 bytes) 000722 1786829046.944413 MIRROR_RESYNC 5 mirrors 000723 1786829048.705582 trunc from 0x040000 to 0x02b7d3 (0x1482c bytes) 000724 1786829048.780947 trunc from 0x02b7d4 to 0x02b45c (0x377 bytes) 000725 1786829048.845631 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2b45d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b45d): Mirror 4: 0xbd6d6be3 (stale) Mirror 5: 0xbd6d6be3 (stale) Mirror 6: 0xbd6d6be3 (stale) Mirror 7: 0xbd6d6be3 (stale) CRC-32 checksum value for chunk [0, 0x2b45d): Mirror 1: 0xbd6d6be3 Mirror 8: 0xbd6d6be3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b45d: skipped 000726 1786829050.431389 punch from 0x0071ec to 0x00f4f6 (0x830b bytes) 000727 1786829050.509332 fallocate 0x03a463 thru 0x03ffff (0x5b9d bytes) 000729 1786829050.518962 mapread 0x00bbb5 thru 0x00e46d (0x28b9 bytes) 000730 1786829050.544414 trunc from 0x02b45d to 0x031be3 (0x6787 bytes) 000731 1786829050.598286 read 0x01ff09 thru 0x02b1ae (0xb2a6 bytes) 000732 1786829050.612819 trunc from 0x031be4 to 0x00dd86 (0x23e5d bytes) 000733 1786829050.656177 mapwrite 0x01754d thru 0x021911 (0xa3c5 bytes) 000735 1786829050.757464 punch from 0x011648 to 0x01af20 (0x98d9 bytes) 000736 1786829050.764680 mapwrite 0x008c05 thru 0x0166aa (0xdaa6 bytes) 000737 1786829050.816363 write 0x03761a thru 0x037e2a (0x811 bytes) 000738 1786829050.819691 trunc from 0x00f3f9 to 0x016289 (0x6e91 bytes) 000739 1786829050.825934 punch from 0x02caa4 to 0x034b6d (0x80ca bytes) 000740 1786829050.830234 trunc from 0x01f8bd to 0x02c26d (0xc9b1 bytes) 000741 1786829050.836248 mapread 0x011c0c thru 0x019a3d (0x7e32 bytes) 000742 1786829050.844867 trunc from 0x037e2b to 0x01642a (0x21a00 bytes) 000743 1786829050.859998 fallocate 0x006af2 thru 0x00b557 (0x4a66 bytes) 000744 1786829050.866246 mapwrite 0x00494e thru 0x005017 (0x6ca bytes) 000745 1786829050.888188 punch from 0x011e08 to 0x0122a8 (0x4a1 bytes) 000746 1786829050.893259 mapwrite 0x00d586 thru 0x01495f (0x73da bytes) 000747 1786829050.915495 mapread 0x015e60 thru 0x01642a (0x5cb bytes) 000748 1786829050.917500 trunc from 0x01642b to 0x029ce7 (0x138bd bytes) 000749 1786829050.949087 trunc from 0x029ce8 to 0x03ecba (0x14fd3 bytes) 000750 1786829050.965658 punch from 0x0176cd to 0x01b7b0 (0x40e4 bytes) 000751 1786829050.974950 trunc from 0x03ecbb to 0x018f9c (0x25d1e bytes) 000752 1786829050.995488 trunc from 0x016c0b to 0x018f9c (0x2392 bytes) 000753 1786829051.001549 read 0x00385b thru 0x00f5a1 (0xbd47 bytes) 000754 1786829051.002427 mapwrite 0x01efbe thru 0x022407 (0x344a bytes) 000755 1786829051.058243 fallocate 0x01f3e6 thru 0x0235d0 (0x41eb bytes) 000756 1786829051.065389 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x235d1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x235d1): Mirror 4: 0xf4b8fb75 (stale) Mirror 5: 0xf4b8fb75 (stale) Mirror 6: 0xf4b8fb75 (stale) Mirror 7: 0xf4b8fb75 (stale) Mirror 8: 0xf4b8fb75 (stale) CRC-32 checksum value for chunk [0, 0x235d1): Mirror 1: 0xe5abf1cd Mirror 9: 0xe5abf1cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x235d1: skipped 000757 1786829052.220954 fallocate 0x014e0e thru 0x01f3a4 (0xa597 bytes) 000759 1786829052.289625 trunc from 0x019adc to 0x01c4e8 (0x2a0d bytes) 000760 1786829052.296126 read 0x011faf thru 0x01e5ba (0xc60c bytes) 000761 1786829052.310209 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x235d1): Mirror 4: 0xf4b8fb75 (stale) Mirror 5: 0xf4b8fb75 (stale) Mirror 6: 0xf4b8fb75 (stale) Mirror 7: 0xf4b8fb75 (stale) Mirror 9: 0xe5abf1cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x235d1): Mirror 4: 0xf4b8fb75 (stale) Mirror 5: 0xf4b8fb75 (stale) Mirror 6: 0xf4b8fb75 (stale) Mirror 7: 0xf4b8fb75 (stale) Mirror 9: 0xe5abf1cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x235d1: skipped 000762 1786829052.798532 read 0x01b3d4 thru 0x0235d0 (0x81fd bytes) 000763 1786829052.809827 mapread 0x022826 thru 0x0235d0 (0xdab bytes) 000765 1786829052.812971 read 0x0005f2 thru 0x001477 (0xe86 bytes) 000766 1786829052.824637 fallocate 0x009d63 thru 0x00d64e (0x38ec bytes) 000767 1786829052.852524 MIRROR_RESYNC 6 mirrors 000768 1786829055.334641 punch from 0x00b223 to 0x01add3 (0xfbb1 bytes) 000769 1786829055.420451 punch from 0x01e62a to 0x0235d0 (0x4fa7 bytes) 000770 1786829055.452369 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x235d1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x235d1): Mirror 4: 0xe5abf1cd (stale) Mirror 5: 0xe5abf1cd (stale) Mirror 6: 0xe5abf1cd (stale) Mirror 7: 0xe5abf1cd (stale) Mirror 9: 0xe5abf1cd (stale) CRC-32 checksum value for chunk [0, 0x235d1): Mirror 1: 0x67669d15 Mirror 10: 0x67669d15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x235d1: skipped 000771 1786829056.835048 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x235d1): Mirror 4: 0xe5abf1cd (stale) Mirror 5: 0xe5abf1cd (stale) Mirror 6: 0xe5abf1cd (stale) Mirror 7: 0xe5abf1cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x235d1): Mirror 4: 0xe5abf1cd (stale) Mirror 5: 0xe5abf1cd (stale) Mirror 6: 0xe5abf1cd (stale) Mirror 7: 0xe5abf1cd (stale) CRC-32 checksum value for chunk [0, 0x235d1): Mirror 1: 0x67669d15 Mirror 10: 0x67669d15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x235d1: skipped 000772 1786829057.585784 punch from 0x00084d to 0x00f42f (0xebe3 bytes) 000773 1786829057.651896 mapread 0x0228f9 thru 0x0235d0 (0xcd8 bytes) 000774 1786829057.687606 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x235d1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x235d1): Mirror 4: 0xe5abf1cd (stale) Mirror 5: 0xe5abf1cd (stale) Mirror 6: 0xe5abf1cd (stale) Mirror 7: 0xe5abf1cd (stale) Mirror 10: 0x67669d15 (stale) CRC-32 checksum value for chunk [0, 0x235d1): Mirror 1: 0xd52e23ee Mirror 11: 0xd52e23ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x235d1: skipped 000775 1786829059.261526 mapread 0x01ea7e thru 0x0235d0 (0x4b53 bytes) 000776 1786829059.299226 trunc from 0x0235d1 to 0x0079ed (0x1bbe3 bytes) 000777 1786829059.369509 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x79ee): Mirror 4: 0x46e99694 (stale) Mirror 5: 0x46e99694 (stale) Mirror 6: 0x46e99694 (stale) Mirror 10: 0x46e99694 (stale) Mirror 11: 0x47c24225 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x79ee): Mirror 4: 0x46e99694 (stale) Mirror 5: 0x46e99694 (stale) Mirror 6: 0x46e99694 (stale) Mirror 10: 0x46e99694 (stale) Mirror 11: 0x47c24225 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x79ee: skipped 000778 1786829059.750592 mapwrite 0x00e304 thru 0x01a040 (0xbd3d bytes) 000780 1786829059.822324 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a041): Mirror 4: 0x97fdd9d9 (stale) Mirror 5: 0x97fdd9d9 (stale) Mirror 10: 0x1e2bc59 (stale) Mirror 11: 0x7df380a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a041): Mirror 4: 0x97fdd9d9 (stale) Mirror 5: 0x97fdd9d9 (stale) Mirror 10: 0x1e2bc59 (stale) Mirror 11: 0x7df380a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a041: skipped 000781 1786829060.342604 fallocate 0x03f314 thru 0x03fbdf (0x8cc bytes) 000782 1786829060.348424 mapread 0x004a15 thru 0x008ddd (0x43c9 bytes) 000783 1786829060.376646 MIRROR_RESYNC 5 mirrors 000785 1786829061.253879 mapwrite 0x03a994 thru 0x03ffff (0x566c bytes) 000786 1786829061.318889 write 0x00f883 thru 0x01f82e (0xffac bytes) 000787 1786829061.331126 write 0x017886 thru 0x02496c (0xd0e7 bytes) 000788 1786829061.340755 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe351bb86 (stale) Mirror 10: 0xe351bb86 (stale) Mirror 11: 0xe351bb86 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe351bb86 (stale) Mirror 10: 0xe351bb86 (stale) Mirror 11: 0xe351bb86 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000789 1786829061.667319 punch from 0x03490a to 0x03c525 (0x7c1c bytes) 000791 1786829061.678197 punch from 0x013338 to 0x01361b (0x2e4 bytes) 000792 1786829061.683553 mapread 0x02d63f thru 0x02fa62 (0x2424 bytes) 000793 1786829061.701165 trunc from 0x00c6a4 to 0x00cdce (0x72b bytes) 000794 1786829061.706230 punch from 0x005815 to 0x00dda3 (0x858f bytes) 000795 1786829061.711217 trunc from 0x040000 to 0x005cc9 (0x3a336 bytes) 000796 1786829061.740571 MIRROR_RESYNC 4 mirrors 000797 1786829062.505304 write 0x021d51 thru 0x028344 (0x65f4 bytes) 000798 1786829062.545920 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28345): Mirror 10: 0x66b79738 (stale) Mirror 11: 0x66b79738 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28345): Mirror 10: 0x66b79738 (stale) Mirror 11: 0x66b79738 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28345: skipped 000799 1786829063.143035 mapwrite 0x02bcb9 thru 0x03b917 (0xfc5f bytes) 000800 1786829063.326533 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b918): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b918): Mirror 10: 0x65ccf048 (stale) Mirror 11: 0x65ccf048 (stale) CRC-32 checksum value for chunk [0, 0x3b918): Mirror 1: 0x426b0ae Mirror 12: 0x426b0ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b918: skipped 000801 1786829064.253973 write 0x016027 thru 0x01ce27 (0x6e01 bytes) 000802 1786829064.313058 write 0x02ff23 thru 0x0392f3 (0x93d1 bytes) 000803 1786829064.331524 trunc from 0x01a6c3 to 0x025014 (0xa952 bytes) 000804 1786829064.343988 mapread 0x00111d thru 0x00d284 (0xc168 bytes) 000805 1786829064.361020 trunc from 0x03b918 to 0x036690 (0x5287 bytes) 000806 1786829064.408665 write 0x03f20a thru 0x03ffff (0xdf6 bytes) 000807 1786829064.410329 fallocate 0x024ead thru 0x02d11a (0x826e bytes) 000808 1786829064.420889 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x6fd54c7 (stale) Mirror 11: 0x6fd54c7 (stale) Mirror 12: 0x26fed997 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb9e93813 Mirror 13: 0xb9e93813 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000809 1786829065.641672 read 0x0358fc thru 0x03a1d9 (0x48de bytes) 000810 1786829065.672728 read 0x027831 thru 0x02d248 (0x5a18 bytes) 000811 1786829065.712126 MIRROR_RESYNC 5 mirrors 000812 1786829067.036186 mapread 0x0083e9 thru 0x014d7d (0xc995 bytes) 000813 1786829067.095382 read 0x035499 thru 0x03f345 (0x9ead bytes) 000814 1786829067.127367 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000815 1786829067.671552 write 0x0160cb thru 0x022192 (0xc0c8 bytes) 000816 1786829067.785663 fallocate 0x01ed73 thru 0x0211f7 (0x2485 bytes) 000817 1786829067.808263 mapread 0x026b4b thru 0x0293f5 (0x28ab bytes) 000818 1786829067.835448 write 0x006653 thru 0x011f00 (0xb8ae bytes) 000819 1786829067.846241 fallocate 0x029cec thru 0x039600 (0xf915 bytes) 000820 1786829067.852526 read 0x00488c thru 0x00f928 (0xb09d bytes) 000821 1786829067.859192 mapread 0x010bcd thru 0x01dfb2 (0xd3e6 bytes) 000822 1786829067.868185 fallocate 0x01d58c thru 0x020d7d (0x37f2 bytes) 000823 1786829067.873079 write 0x018122 thru 0x01c9e4 (0x48c3 bytes) 000824 1786829067.874139 read 0x026ed1 thru 0x02c65d (0x578d bytes) 000825 1786829067.874778 read 0x021f01 thru 0x024ab2 (0x2bb2 bytes) 000826 1786829067.876622 trunc from 0x040000 to 0x01af3d (0x250c2 bytes) 000827 1786829067.902567 fallocate 0x0042ad thru 0x006196 (0x1eea bytes) 000828 1786829067.911257 write 0x0235ad thru 0x02e31f (0xad73 bytes) 000829 1786829067.915180 trunc from 0x02e320 to 0x02855a (0x5dc5 bytes) 000830 1786829067.929466 punch from 0x017842 to 0x026717 (0xeed6 bytes) 000831 1786829067.952001 read 0x023812 thru 0x02855a (0x4d49 bytes) 000833 1786829067.965652 mapwrite 0x03361c thru 0x03b63b (0x8020 bytes) 000836 1786829068.004857 fallocate 0x023826 thru 0x02a142 (0x691d bytes) 000837 1786829068.016942 trunc from 0x01c900 to 0x02c033 (0xf734 bytes) 000838 1786829068.024306 read 0x02bd5a thru 0x039369 (0xd610 bytes) 000839 1786829068.039936 punch from 0x022338 to 0x0275a2 (0x526b bytes) 000841 1786829068.054692 mapwrite 0x01888f thru 0x02045a (0x7bcc bytes) 000842 1786829068.085118 write 0x02d304 thru 0x03249d (0x519a bytes) 000843 1786829068.086192 mapwrite 0x00d89a thru 0x011601 (0x3d68 bytes) 000844 1786829068.106493 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b63c): Mirror 12: 0x9e53afb3 (stale) Mirror 13: 0x9e53afb3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b63c): Mirror 12: 0x9e53afb3 (stale) Mirror 13: 0x9e53afb3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b63c: skipped 000845 1786829068.418200 mapread 0x018779 thru 0x0252c8 (0xcb50 bytes) 000846 1786829068.446979 write 0x03cbee thru 0x03ce91 (0x2a4 bytes) 000847 1786829068.451403 punch from 0x03ad72 to 0x03ce91 (0x2120 bytes) 000848 1786829068.461439 read 0x019270 thru 0x0231d3 (0x9f64 bytes) 000849 1786829068.465896 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ce92): Mirror 13: 0xe928b89d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce92): Mirror 13: 0xe928b89d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce92: skipped 000850 1786829068.656410 mapread 0x004df2 thru 0x008347 (0x3556 bytes) 000851 1786829068.669958 trunc from 0x03ce92 to 0x002cf6 (0x3a19b bytes) 000852 1786829068.689455 MIRROR_RESYNC 2 mirrors 000853 1786829069.056883 trunc from 0x002cf7 to 0x02419f (0x214a9 bytes) 000854 1786829069.159030 read 0x00889a thru 0x00dab5 (0x521c bytes) 000855 1786829069.208433 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x241a0): Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x241a0): Mirror 13: 0xe18428e5 (stale) CRC-32 checksum value for chunk [0, 0x241a0): Mirror 1: 0xe18428e5 Mirror 14: 0xe18428e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x241a0: skipped 000856 1786829069.952639 mapwrite 0x01aaf7 thru 0x02498b (0x9e95 bytes) 000858 1786829070.045288 write 0x01b983 thru 0x02688b (0xaf09 bytes) 000859 1786829070.047950 trunc from 0x02688c to 0x017734 (0xf157 bytes) 000860 1786829070.077760 read 0x00c7d9 thru 0x0134d4 (0x6cfc bytes) 000861 1786829070.087074 mapwrite 0x02845f thru 0x02dea0 (0x5a42 bytes) 000862 1786829070.139305 mapread 0x00d44a thru 0x017060 (0x9c17 bytes) 000863 1786829070.150916 mapread 0x01daa0 thru 0x023333 (0x5894 bytes) 000864 1786829070.161895 write 0x03fd39 thru 0x03ffff (0x2c7 bytes) 000865 1786829070.164122 fallocate 0x0262ac thru 0x0270f8 (0xe4d bytes) 000866 1786829070.170626 mapread 0x032d57 thru 0x03cd61 (0xa00b bytes) 000867 1786829070.194474 MIRROR_RESYNC 3 mirrors 000868 1786829071.176587 mapread 0x03ff3f thru 0x03ffff (0xc1 bytes) 000869 1786829071.184139 read 0x014680 thru 0x01ed58 (0xa6d9 bytes) 000870 1786829071.195567 trunc from 0x040000 to 0x00c50f (0x33af0 bytes) 000871 1786829071.337145 trunc from 0x00c510 to 0x01941d (0xcf0e bytes) 000872 1786829071.362191 mapread 0x00a3b1 thru 0x0146e3 (0xa333 bytes) 000873 1786829071.392197 fallocate 0x02c33b thru 0x02cdb7 (0xa7d bytes) 000874 1786829071.398106 mapwrite 0x039d5d thru 0x03b70b (0x19af bytes) 000875 1786829071.474365 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b70c): Mirror 14: 0x904abbc1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b70c): Mirror 14: 0x904abbc1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b70c: skipped 000876 1786829071.773519 trunc from 0x008912 to 0x011b68 (0x9257 bytes) 000877 1786829071.783471 write 0x0231a2 thru 0x027c82 (0x4ae1 bytes) 000878 1786829071.802159 trunc from 0x03b70c to 0x011d45 (0x299c6 bytes) 000879 1786829071.819889 mapwrite 0x014d6b thru 0x022066 (0xd2fc bytes) 000880 1786829071.973473 read 0x01f0ee thru 0x022066 (0x2f79 bytes) 000881 1786829071.974227 read 0x00bc0c thru 0x0198b7 (0xdcac bytes) 000882 1786829071.989411 mapread 0x0200bb thru 0x022066 (0x1fac bytes) 000883 1786829071.991525 trunc from 0x022067 to 0x031940 (0xf8da bytes) 000884 1786829072.017492 mapwrite 0x01e46e thru 0x0271f1 (0x8d84 bytes) 000885 1786829072.068343 trunc from 0x013af7 to 0x022399 (0xe8a3 bytes) 000886 1786829072.074439 read 0x0226e9 thru 0x023274 (0xb8c bytes) 000887 1786829072.108318 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000888 1786829072.260189 read 0x0140de thru 0x01bd93 (0x7cb6 bytes) 000890 1786829072.274462 mapread 0x0059eb thru 0x00a3d3 (0x49e9 bytes) 000892 1786829072.296346 mapwrite 0x032221 thru 0x03bf00 (0x9ce0 bytes) 000893 1786829072.358620 read 0x00cc5d thru 0x00d947 (0xceb bytes) 000894 1786829072.359327 punch from 0x01ec4b to 0x0240d0 (0x5486 bytes) 000895 1786829072.370171 mapwrite 0x0306c7 thru 0x03b21e (0xab58 bytes) 000896 1786829072.407540 write 0x00ce30 thru 0x01b5a8 (0xe779 bytes) 000897 1786829072.409437 read 0x022dfc thru 0x02bd74 (0x8f79 bytes) 000898 1786829072.420182 mapread 0x02cd6b thru 0x0374da (0xa770 bytes) 000899 1786829072.421471 trunc from 0x03bf01 to 0x021cad (0x1a253 bytes) 000900 1786829072.443011 punch from 0x01b488 to 0x01e86f (0x33e8 bytes) 000902 1786829072.469955 write 0x021e1f thru 0x022c84 (0xe66 bytes) 000903 1786829072.471666 trunc from 0x022c85 to 0x027cbc (0x5038 bytes) 000904 1786829072.484816 fallocate 0x02e24a thru 0x02e7ce (0x585 bytes) 000906 1786829072.497007 mapwrite 0x003fc0 thru 0x006563 (0x25a4 bytes) 000907 1786829072.514894 trunc from 0x02e7cf to 0x00d565 (0x21269 bytes) 000908 1786829072.534405 trunc from 0x00d566 to 0x019d9f (0xc83a bytes) 000909 1786829072.552405 mapread 0x005787 thru 0x00aba3 (0x541d bytes) 000910 1786829072.553664 fallocate 0x00e030 thru 0x00e929 (0x8fa bytes) 000912 1786829072.558282 mapread 0x00e3ae thru 0x01738d (0x8fe0 bytes) 000913 1786829072.605897 read 0x017264 thru 0x019d9f (0x2b3c bytes) 000914 1786829072.612113 punch from 0x0060bc to 0x00fc75 (0x9bba bytes) 000916 1786829072.631792 read 0x0134ac thru 0x019d9f (0x68f4 bytes) 000920 1786829072.636070 mapwrite 0x0094e7 thru 0x0168d8 (0xd3f2 bytes) 000921 1786829072.723370 mapread 0x010f93 thru 0x019d9f (0x8e0d bytes) 000922 1786829072.724548 read 0x016a0c thru 0x019d9f (0x3394 bytes) 000923 1786829072.725199 mapread 0x00d0a2 thru 0x013345 (0x62a4 bytes) 000924 1786829072.735782 punch from 0x00530f to 0x007d10 (0x2a02 bytes) 000926 1786829072.755489 mapwrite 0x01f2ea thru 0x01fea6 (0xbbd bytes) 000927 1786829072.820122 mapwrite 0x02ed29 thru 0x033234 (0x450c bytes) 000929 1786829072.877856 mapwrite 0x015378 thru 0x0210f8 (0xbd81 bytes) 000930 1786829072.890822 read 0x0123de thru 0x013fab (0x1bce bytes) 000931 1786829072.891314 trunc from 0x02a52b to 0x033234 (0x8d0a bytes) 000932 1786829072.894998 trunc from 0x00338f to 0x00580b (0x247d bytes) 000933 1786829072.898424 trunc from 0x033235 to 0x03328b (0x57 bytes) 000934 1786829072.911084 mapwrite 0x009569 thru 0x012e10 (0x98a8 bytes) 000936 1786829072.930167 read 0x0228f6 thru 0x023b30 (0x123b bytes) 000938 1786829072.931905 write 0x029072 thru 0x034804 (0xb793 bytes) 000939 1786829072.934383 trunc from 0x034805 to 0x03af22 (0x671e bytes) 000942 1786829072.959395 write 0x03e0f0 thru 0x03ffff (0x1f10 bytes) 000944 1786829072.963132 mapwrite 0x008531 thru 0x00a1ff (0x1ccf bytes) 000945 1786829072.989040 trunc from 0x040000 to 0x02bc5f (0x143a0 bytes) 000946 1786829073.014840 punch from 0x0149b5 to 0x022f97 (0xe5e3 bytes) 000947 1786829073.021049 read 0x000804 thru 0x0088b8 (0x80b5 bytes) 000948 1786829073.030691 trunc from 0x02bc60 to 0x03a654 (0xe9f5 bytes) 000949 1786829073.064640 trunc from 0x01475f to 0x016f4b (0x27ed bytes) 000950 1786829073.073495 punch from 0x024247 to 0x030a8e (0xc848 bytes) 000951 1786829073.079872 write 0x0329b0 thru 0x03bcba (0x930b bytes) 000952 1786829073.096930 punch from 0x028d1f to 0x02a64d (0x192f bytes) 000953 1786829073.112111 trunc from 0x03bcbb to 0x018fa5 (0x22d15 bytes) 000955 1786829073.156160 trunc from 0x01140b to 0x018fa5 (0x7b9b bytes) 000956 1786829073.170277 read 0x007d7a thru 0x013b08 (0xbd8f bytes) 000957 1786829073.189405 punch from 0x018570 to 0x018fa5 (0xa36 bytes) 000958 1786829073.219143 MIRROR_EXTEND to 2 mirrors 000961 1786829073.960788 trunc from 0x018fa6 to 0x03a2e7 (0x21342 bytes) 000962 1786829074.077005 trunc from 0x03a2e8 to 0x0116fd (0x28bea bytes) 000963 1786829074.112838 mapwrite 0x036538 thru 0x03ffff (0x9ac8 bytes) 000964 1786829074.239298 fallocate 0x02316d thru 0x02eb15 (0xb9a9 bytes) 000966 1786829074.251900 trunc from 0x01722f to 0x02715a (0xff2c bytes) 000967 1786829074.263611 fallocate 0x01cbbb thru 0x029602 (0xca48 bytes) 000968 1786829074.274196 read 0x027a2d thru 0x02beaf (0x4483 bytes) 000969 1786829074.296995 MIRROR_RESYNC 2 mirrors 000970 1786829075.217942 read 0x0265fe thru 0x031ed7 (0xb8da bytes) 000971 1786829075.237436 punch from 0x0003e7 to 0x004177 (0x3d91 bytes) 000972 1786829075.263568 mapwrite 0x02c4ce thru 0x031c11 (0x5744 bytes) 000973 1786829075.356488 MIRROR_RESYNC 2 mirrors 000974 1786829076.210260 fallocate 0x03311b thru 0x03cac2 (0x99a8 bytes) 000975 1786829076.261505 trunc from 0x040000 to 0x011cb9 (0x2e346 bytes) 000976 1786829076.339179 trunc from 0x011cba to 0x005e29 (0xbe90 bytes) 000977 1786829076.422809 mapread 0x0050d8 thru 0x005e29 (0xd52 bytes) 000978 1786829076.446637 read 0x00019a thru 0x005e29 (0x5c90 bytes) 000979 1786829076.468929 fallocate 0x03f480 thru 0x03ffff (0xb80 bytes) 000981 1786829076.490404 trunc from 0x040000 to 0x036217 (0x9de8 bytes) 000982 1786829076.530520 trunc from 0x033cb0 to 0x036176 (0x24c7 bytes) 000983 1786829076.542839 mapread 0x02fae1 thru 0x036217 (0x6737 bytes) 000984 1786829076.598205 trunc from 0x036218 to 0x03ae9b (0x4c84 bytes) 000985 1786829076.623576 mapread 0x022fbe thru 0x025ed6 (0x2f19 bytes) 000986 1786829076.643928 mapread 0x00533d thru 0x008faa (0x3c6e bytes) 000987 1786829076.659973 write 0x01f977 thru 0x020162 (0x7ec bytes) 000988 1786829076.662519 mapwrite 0x0004a5 thru 0x001f06 (0x1a62 bytes) 000989 1786829076.709556 mapread 0x02652d thru 0x02e48e (0x7f62 bytes) 000990 1786829076.711124 trunc from 0x03ae9c to 0x03d261 (0x23c6 bytes) 000991 1786829076.732058 trunc from 0x02b169 to 0x0360df (0xaf77 bytes) 000992 1786829076.742992 mapread 0x023ef1 thru 0x02cbac (0x8cbc bytes) 000993 1786829076.759273 MIRROR_RESYNC 2 mirrors 000994 1786829077.413351 mapread 0x02fe4a thru 0x0348f1 (0x4aa8 bytes) 000995 1786829077.447081 mapwrite 0x03eff2 thru 0x03ffff (0x100e bytes) 000996 1786829077.585540 fallocate 0x00951a thru 0x00f54f (0x6036 bytes) 000997 1786829077.609044 fallocate 0x014d93 thru 0x0163dc (0x164a bytes) 000998 1786829077.618641 trunc from 0x040000 to 0x02f556 (0x10aa9 bytes) 000999 1786829077.679511 write 0x01988c thru 0x0198b8 (0x2d bytes) 001000 1786829077.696025 read 0x01f51d thru 0x0278db (0x83bf bytes) All operations completed A-OK! PASS 70a (93s) == sanity-flr test 71: check mirror extend parent fid ==== 17:24:42 (1786829082) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x58:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 17:24:54 (1786829094) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2df:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13578 s, 15.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2df:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100819 s, 10.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e0:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2df:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e0:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2df:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ff:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 17:25:03 (1786829103) Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre2 Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1829120 .. Extending file size to 2403264 .. failed Extending file size to 5923232 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7504960 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6480576 .. Extending file size to 7939584 .. Extending file size to 8733280 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8415616 .. Extending file size to 9184512 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61359 61361 61362 61363 61364 done 192.168.201.146@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg146-client.virtnet /mnt/lustre2 (opts:) 192.168.201.146@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg146-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (91s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:26:34 (1786829194) Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre2 Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 326080 .. Extending file size to 6275776 .. failed Extending file size to 6885728 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7854272 .. Extending file size to 8516288 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7484448 .. Extending file size to 7615776 .. Extending file size to 8744384 .. Extending file size to 9236192 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62850 62851 62852 62855 62856 done 192.168.201.146@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg146-client.virtnet /mnt/lustre2 (opts:) 192.168.201.146@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg146-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:27:54 (1786829274) Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1992 1269312 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24832 3582188 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18952 3588068 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43784 7170256 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1992 1269312 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24832 3582188 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18952 3588068 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43784 7170256 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108883 s, 19.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.222226 s, 9.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x94364f80 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.146@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg146-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:28:15 (1786829295) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0825495 s, 25.4 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 17:28:24 (1786829304) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e1:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f6:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f6:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f7:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x314:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:28:32 (1786829312) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0712511 s, 29.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x49:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:28:40 (1786829320) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0551157 s, 38.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x315:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x315:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x316:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:28:47 (1786829327) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0843794 s, 24.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x317:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x317:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x318:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:29:21 (1786829361) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg146-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155543 s, 6.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fb:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x319:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fb:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x319:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fc:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:30:17 (1786829417) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg146-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112546 s, 18.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (50s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:31:07 (1786829467) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg146-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg146-server: Pool lustre.test_204f created Adding targets to pool oleg146-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg146-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0909403 s, 23.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg146-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg146-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg146-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:32:20 (1786829540) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:32:28 (1786829548) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:32:37 (1786829557) oleg146-server: Pool lustre.test_206 created Adding targets to pool oleg146-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg146-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg146-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg146-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg146-server: Pool lustre.test_206 destroyed PASS 206 (32s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:33:09 (1786829589) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922727 s, 11.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354578 s, 29.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x449abf77 Mirror 3: 0x449abf77 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:33:22 (1786829602) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:33:26 (1786829606) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:33:30 (1786829610) Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0848845 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x890000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1c40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1c40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1c40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2040000 Read this (4 bytes): BBBB 192.168.201.146@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg146-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:34:07 (1786829647) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.044637 s, 184 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:34:15 (1786829655) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439934 s, 23.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x204 lmm_fid: [0x200000405:0x204:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x204 lmm_fid: [0x200000405:0x204:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x309:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611652 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-15 17:34:17.000000000 -0400 Modify: 2026-08-15 17:34:17.000000000 -0400 Change: 2026-08-15 17:34:18.000000000 -0400 Birth: 2026-08-15 17:34:15.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:34:24 (1786829664) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392376 s, 26.7 MB/s oleg146-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg146-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg146-server: O/280000401/d7/327: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:34:48 (1786829688) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.345869 s, 118 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.036623 s, 112 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 15 17:34 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x209:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 15 17:34 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 17:34:59 (1786829699) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 17:35:10 (1786829710) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1898 sec =========== 17:35:19 (1786829719) === sanity-flr: start cleanup 17:35:21 (1786829721) === === sanity-flr: finish cleanup 17:35:29 (1786829729) ===