-----============= acceptance-small: sanity-flr ============----- Mon Aug 17 05:59:42 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg301-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 06:00:02 (1786960802) === oleg301-client.virtnet: executing check_config_client /mnt/lustre oleg301-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg301-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9140d1ad6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9140d1ad6800.idle_timeout=debug disable quota as required oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 06:00:44 (1786960844) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8227] == sanity-flr test 0a: lfs mirror create with -N option == 06:00:46 (1786960846) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:01:02 (1786960862) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:01:05 (1786960865) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:01:08 (1786960868) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:01:30 (1786960890) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:01:34 (1786960894) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 06:01:38 (1786960898) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9140d1ad6800.stats=clear osc.lustre-OST0001-osc-ffff9140d1ad6800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:01:47 (1786960907) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:01:59 (1786960919) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:02:07 (1786960927) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:02:11 (1786960931) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:02:19 (1786960939) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 06:02:29 (1786960949) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:02:37 (1786960957) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:02:45 (1786960965) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:02:53 (1786960973) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:03:03 (1786960983) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:03:09 (1786960989) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.564122 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114997 s, 9.1 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:03:17 (1786960997) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.206018 s, 25.4 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:03:25 (1786961005) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:03:35 (1786961015) Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9140d1ad6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9140d1ad6800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9140d1ad6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9140d1ad6800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg301-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (69s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:04:44 (1786961084) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.50722 s, 18.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9140d1ad6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9140d1ad6800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (38s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:05:22 (1786961122) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9140d1ad6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9140d1ad6800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9140d1ad6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9140d1ad6800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg301-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (74s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:06:36 (1786961196) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.92547 s, 17.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 17 06:06 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.809s user 0m0.002s sys 0m0.185s Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9140d1ad6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9140d1ad6800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.002s user 0m0.001s sys 0m0.198s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9140d1ad6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9140d1ad6800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.090s user 0m0.001s sys 0m0.180s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg301-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg301-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9140d1ad6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9140d1ad6800.ost_server_uuid in FULL state after 0 sec PASS 33b (93s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:08:09 (1786961289) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:08:14 (1786961294) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:08:18 (1786961298) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 06:08:21 (1786961301) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 06:08:28 (1786961308) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.426929 s, 29.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:08:40 (1786961320) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397562 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282959 s, 37.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:08:47 (1786961327) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:08:55 (1786961335) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00377216 s, 271 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000786828 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000775726 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0064235 s, 159 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 06:09:06 (1786961346) create test files with size 15731909 .. f517b8a6c9c2a6c49382170a25131ad1 - 751b05b262815678a6186e1e59cd8ce1 - f266c0982f52f3be3c761798a00d0883 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg301-client.virtnet /mnt/lustre (opts:) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 06:09:30 (1786961370) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:09:48 (1786961388) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:09:56 (1786961396) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141229 s, 7.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071005 s, 14.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 06:10:07 (1786961407) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108158 s, 19.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.391355 s, 10.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135841 s, 15.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 06:10:24 (1786961424) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 06:10:28 (1786961428) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:10:32 (1786961432) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438692 s, 23.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 06:10:39 (1786961439) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.385954 s, 8.2 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:10:50 (1786961450) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:11:00 (1786961460) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.301397 s, 34.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg301-client.virtnet /mnt/lustre (opts:) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:11:12 (1786961472) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9140d13dc800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.295876 s, 35.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg301-client.virtnet /mnt/lustre (opts:) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9140c9cb8000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:11:28 (1786961488) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:12:08 (1786961528) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:12:46 (1786961566) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 06:12:50 (1786961570) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.340106 s, 9.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.263244 s, 11.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf1a9de2a (nosync,stale) Mirror 2: 0xf1a9de2a (nosync,stale) ** verify mirror 2 doesn't change original checksum: b61623552cebdca6e9d9a6b0d4789be6 - mirror 1 checksum: e40094a60c79289d8a6211104701fe54 - mirror 2 checksum: b61623552cebdca6e9d9a6b0d4789be6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-17T06:12:54-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-17T06:12:54-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b61623552cebdca6e9d9a6b0d4789be6 - mirror 1 checksum: e40094a60c79289d8a6211104701fe54 - mirror 2 checksum: e40094a60c79289d8a6211104701fe54 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-17T06:12:54-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-17T06:12:54-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } PASS 48 (14s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 06:13:04 (1786961584) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323657 s, 32.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:13:11 (1786961591) Failing mds1 on oleg301-server Stopping /mnt/lustre-mds1 (opts:) on oleg301-server 06:13:23 (1786961603) shut down facet: mds1 facet_host: oleg301-server facet_failover_host: oleg301-server Failover mds1 to oleg301-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0000 06:13:55 (1786961635) targets are mounted 06:13:55 (1786961635) facet_failover done oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369616 s, 28.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } PASS 50A (62s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:14:13 (1786961653) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.069347 s, 295 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0338968 s, 604 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (13s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:14:26 (1786961666) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0567194 s, 361 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0288124 s, 711 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:14:36 (1786961676) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0142371 s, 1.2 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:14:45 (1786961685) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0870223 s, 235 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:14:55 (1786961695) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00577097 s, 710 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0203895 s, 201 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:15:02 (1786961702) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:15:28 (1786961728) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:15:56 (1786961756) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:16:20 (1786961780) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:16:45 (1786961805) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 06:16:52 (1786961812) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57580 mirror split pid 57581 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57580 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57581 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 06:18:00 (1786961880) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 1882 truncating to largest ever: 0x37bdd 000001 1786961883.267943 trunc from 00000000 to 0x037bdc (0x37bdd bytes) 000002 1786961883.290158 read 0x0155fc thru 0x017926 (0x232b bytes) 000003 1786961883.302800 fallocate 0x019797 thru 0x01d4c8 (0x3d32 bytes) 000004 1786961883.311173 fallocate 0x007f01 thru 0x010e76 (0x8f76 bytes) 000005 1786961883.318598 punch from 0x034100 to 0x035071 (0xf72 bytes) 000006 1786961883.331134 MIRROR_EXTEND to 2 mirrors 000007 1786961884.125155 MIRROR_EXTEND to 3 mirrors 000008 1786961885.043742 punch from 0x011c57 to 0x0186a2 (0x6a4c bytes) 000009 1786961885.094478 trunc from 0x002406 to 0x003143 (0xd3e bytes) 000010 1786961885.103227 write 0x0245ac thru 0x02e983 (0xa3d8 bytes) 000011 1786961885.122364 read 0x030fb0 thru 0x035544 (0x4595 bytes) 000012 1786961885.143766 write 0x027988 thru 0x029037 (0x16b0 bytes) 000013 1786961885.151215 trunc from 0x037bdd to 0x009d7a (0x2de62 bytes) 000014 1786961885.220621 punch from 0x0052ee to 0x007848 (0x255b bytes) 000016 1786961885.228832 fallocate 0x030a42 thru 0x032ab5 (0x2074 bytes) 000017 1786961885.250419 punch from 0x00742c to 0x009d7a (0x294f bytes) 000018 1786961885.275984 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9d7b): Mirror 3: 0xdb3b1f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9d7b): Mirror 3: 0xdb3b1f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9d7b: skipped 000019 1786961885.652200 read 0x003923 thru 0x008184 (0x4862 bytes) 000020 1786961885.685836 punch from 0x002f35 to 0x009d7a (0x6e46 bytes) 000021 1786961885.728743 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000022 1786961885.930416 trunc from 0x009d7b to 0x0037ad (0x65cd bytes) 000023 1786961885.966161 MIRROR_EXTEND to 2 mirrors 000024 1786961886.727353 mapread 0x00354e thru 0x0037ad (0x260 bytes) 000025 1786961886.758489 read 0x00344f thru 0x0037ad (0x35f bytes) 000026 1786961886.771737 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000027 1786961886.925793 trunc from 0x0037ae to 0x011ef8 (0xe74b bytes) 000028 1786961886.976043 MIRROR_EXTEND to 2 mirrors 000029 1786961887.845919 trunc from 0x00a394 to 0x011ef8 (0x7b65 bytes) 000030 1786961887.902966 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x11ef9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11ef9): Mirror 2: 0x98a541b (stale) CRC-32 checksum value for chunk [0, 0x11ef9): Mirror 1: 0x98a541b Mirror 3: 0x98a541b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11ef9: skipped 000031 1786961888.927232 trunc from 0x00dca4 to 0x010b57 (0x2eb4 bytes) 000032 1786961888.997642 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11ef9): Mirror 3: 0x98a541b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11ef9): Mirror 3: 0x98a541b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11ef9: skipped 000033 1786961889.325317 mapread 0x000379 thru 0x008ecf (0x8b57 bytes) 000034 1786961889.352835 trunc from 0x011ef9 to 0x01daca (0xbbd2 bytes) 000035 1786961889.385849 mapwrite 0x00c151 thru 0x01561e (0x94ce bytes) 000036 1786961889.436713 fallocate 0x00a835 thru 0x00f0a0 (0x486c bytes) fallocating to largest ever: 0x37dcc 000037 1786961889.447761 fallocate 0x03285b thru 0x037dcb (0x5571 bytes) 000038 1786961889.457593 trunc from 0x02aa4f to 0x031a4a (0x6ffc bytes) 000039 1786961889.494342 mapwrite 0x025e6a thru 0x02b826 (0x59bd bytes) 000041 1786961889.555162 trunc from 0x02ef54 to 0x037dcb (0x8e78 bytes) 000042 1786961889.567442 mapread 0x0372d9 thru 0x037dcb (0xaf3 bytes) 000043 1786961889.580248 write 0x02c6f5 thru 0x03bf13 (0xf81f bytes) 000044 1786961889.582330 write 0x00d7d7 thru 0x0127ef (0x5019 bytes) 000045 1786961889.586793 read 0x0201ab thru 0x02e804 (0xe65a bytes) 000046 1786961889.593804 trunc from 0x03bf14 to 0x029e12 (0x12101 bytes) 000047 1786961889.626384 trunc from 0x029e13 to 0x02faa3 (0x5c91 bytes) 000048 1786961889.645399 fallocate 0x015c9c thru 0x01f09f (0x9404 bytes) 000049 1786961889.650686 mapread 0x01b490 thru 0x02284e (0x73bf bytes) 000050 1786961889.657213 fallocate 0x01e0ac thru 0x021b93 (0x3ae8 bytes) 000051 1786961889.662979 mapwrite 0x011950 thru 0x01fd9a (0xe44b bytes) 000052 1786961889.682839 fallocate 0x00ec34 thru 0x019aa5 (0xae72 bytes) 000053 1786961889.688074 trunc from 0x01bcdb to 0x0247d6 (0x8afc bytes) 000054 1786961889.693702 mapread 0x00bc73 thru 0x01a10a (0xe498 bytes) 000055 1786961889.696072 mapwrite 0x039121 thru 0x03ffff (0x6edf bytes) 000056 1786961889.740661 write 0x01c852 thru 0x01e0ff (0x18ae bytes) 000057 1786961889.751043 punch from 0x003c71 to 0x00d927 (0x9cb7 bytes) 000058 1786961889.760515 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbca88813 Mirror 4: 0xbca88813 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000059 1786961890.930510 trunc from 0x00783e to 0x00ff8e (0x8751 bytes) 000060 1786961891.012550 trunc from 0x040000 to 0x025ec7 (0x1a138 bytes) 000061 1786961891.068582 trunc from 0x025ec8 to 0x020a20 (0x54a7 bytes) 000062 1786961891.138506 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20a21): Mirror 4: 0xb7cbd187 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20a21): Mirror 4: 0xb7cbd187 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20a21: skipped 000063 1786961891.693710 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000064 1786961891.888409 trunc from 0x020a21 to 0x00362c (0x1d3f4 bytes) fallocating to largest ever: 0x3b45a 000065 1786961891.936056 fallocate 0x037c7b thru 0x03b459 (0x37df bytes) 000066 1786961891.947124 fallocate 0x027d92 thru 0x02b0d2 (0x3341 bytes) 000067 1786961891.963326 trunc from 0x02b0d3 to 0x00c5ae (0x1eb24 bytes) 000068 1786961892.003547 trunc from 0x00c5af to 0x0155c0 (0x9012 bytes) 000069 1786961892.031547 read 0x0100a5 thru 0x010752 (0x6ae bytes) 000070 1786961892.043743 mapwrite 0x00f668 thru 0x01f5c6 (0xff5f bytes) 000071 1786961892.178165 trunc from 0x00f49c to 0x012cf8 (0x385d bytes) 000072 1786961892.197704 trunc from 0x001c02 to 0x009c55 (0x8054 bytes) 000073 1786961892.219974 trunc from 0x01f5c7 to 0x03329c (0x13cd6 bytes) 000074 1786961892.253103 write 0x03c3f0 thru 0x03ffff (0x3c10 bytes) 000075 1786961892.261985 write 0x0106f9 thru 0x01447e (0x3d86 bytes) 000076 1786961892.273255 fallocate 0x02cadf thru 0x02e85f (0x1d81 bytes) 000078 1786961892.284383 mapwrite 0x010b0c thru 0x01bc19 (0xb10e bytes) 000079 1786961892.329763 write 0x01e35e thru 0x028ee7 (0xab8a bytes) 000081 1786961892.342682 punch from 0x022c1b to 0x023750 (0xb36 bytes) 000082 1786961892.364349 mapread 0x02854c thru 0x034830 (0xc2e5 bytes) 000083 1786961892.409982 punch from 0x02b644 to 0x037b65 (0xc522 bytes) 000084 1786961892.417390 punch from 0x023267 to 0x02e84e (0xb5e8 bytes) fallocating to largest ever: 0x40000 000085 1786961892.424256 fallocate 0x03c423 thru 0x03ffff (0x3bdd bytes) 000086 1786961892.430367 mapwrite 0x02ab82 thru 0x02b99b (0xe1a bytes) 000087 1786961892.462463 punch from 0x03e919 to 0x03ffff (0x16e7 bytes) 000088 1786961892.473024 fallocate 0x01c529 thru 0x025bf0 (0x96c8 bytes) 000089 1786961892.479360 read 0x0104ca thru 0x01ad19 (0xa850 bytes) 000091 1786961892.486342 write 0x027367 thru 0x032583 (0xb21d bytes) 000093 1786961892.493424 trunc from 0x040000 to 0x01ff87 (0x20078 bytes) 000094 1786961892.540106 write 0x028e90 thru 0x02f71e (0x688f bytes) 000095 1786961892.544334 mapread 0x008860 thru 0x012903 (0xa0a4 bytes) 000096 1786961892.561438 read 0x02ce99 thru 0x02f71e (0x2886 bytes) 000097 1786961892.562979 read 0x0148a4 thru 0x01ab2c (0x6289 bytes) 000098 1786961892.564216 trunc from 0x02f71f to 0x00f782 (0x1ff9c bytes) 000099 1786961892.587642 trunc from 0x009092 to 0x00f782 (0x66f1 bytes) 000100 1786961892.600371 mapread 0x00dbc9 thru 0x00f782 (0x1bba bytes) 000102 1786961892.613916 fallocate 0x033c90 thru 0x039890 (0x5c01 bytes) 000103 1786961892.621270 trunc from 0x019eb9 to 0x02676d (0xc8b5 bytes) 000104 1786961892.630940 mapwrite 0x008130 thru 0x009d42 (0x1c13 bytes) 000105 1786961892.682851 write 0x01d1bd thru 0x0253aa (0x81ee bytes) 000106 1786961892.701997 trunc from 0x039891 to 0x00fedc (0x299b4 bytes) 000108 1786961892.732409 trunc from 0x007598 to 0x00dbd3 (0x663c bytes) 000109 1786961892.739971 mapread 0x004bb7 thru 0x00fedc (0xb326 bytes) 000110 1786961892.822256 trunc from 0x000f34 to 0x003d77 (0x2e44 bytes) 000111 1786961892.830476 punch from 0x004521 to 0x00c19d (0x7c7d bytes) 000113 1786961892.843434 trunc from 0x006805 to 0x00fc32 (0x942e bytes) 000114 1786961892.854751 trunc from 0x00a4b8 to 0x00e9db (0x4524 bytes) 000115 1786961892.873533 read 0x00b5b3 thru 0x00fedc (0x492a bytes) 000116 1786961892.899387 fallocate 0x01fd22 thru 0x0280b8 (0x8397 bytes) 000117 1786961892.936737 trunc from 0x010ff7 to 0x0124df (0x14e9 bytes) 000118 1786961892.950543 mapread 0x025921 thru 0x0280b8 (0x2798 bytes) 000119 1786961892.975837 write 0x0137fe thru 0x0143b5 (0xbb8 bytes) 000120 1786961892.999181 MIRROR_EXTEND to 2 mirrors 000121 1786961894.053592 trunc from 0x01baa9 to 0x0280b8 (0xc610 bytes) 000122 1786961894.130750 mapread 0x007a8d thru 0x00bec6 (0x443a bytes) 000123 1786961894.178664 mapwrite 0x0107cf thru 0x0150d2 (0x4904 bytes) 000124 1786961894.232707 trunc from 0x0280b9 to 0x011776 (0x16942 bytes) 000125 1786961894.261236 fallocate 0x023fe2 thru 0x03116f (0xd18e bytes) 000126 1786961894.267772 mapwrite 0x03a98d thru 0x03ffff (0x5673 bytes) 000127 1786961894.309886 mapwrite 0x029942 thru 0x033790 (0x9e4f bytes) 000128 1786961894.361119 write 0x020dbf thru 0x02122b (0x46d bytes) 000129 1786961894.370979 write 0x00eec7 thru 0x011a9f (0x2bd9 bytes) 000130 1786961894.372089 trunc from 0x027b63 to 0x036e32 (0xf2d0 bytes) 000131 1786961894.376690 write 0x01f257 thru 0x027c41 (0x89eb bytes) 000132 1786961894.387758 punch from 0x02b96c to 0x0377fc (0xbe91 bytes) 000133 1786961894.393040 mapwrite 0x001ac4 thru 0x00f3e9 (0xd926 bytes) 000134 1786961894.449693 MIRROR_RESYNC 2 mirrors 000135 1786961895.507829 fallocate 0x021fb3 thru 0x02eb0e (0xcb5c bytes) 000137 1786961895.579403 punch from 0x02f457 to 0x0385c2 (0x916c bytes) 000139 1786961895.591741 punch from 0x01b512 to 0x02ac3f (0xf72e bytes) 000140 1786961895.612042 mapwrite 0x00a1c1 thru 0x011562 (0x73a2 bytes) 000141 1786961895.706481 fallocate 0x013565 thru 0x01b203 (0x7c9f bytes) 000142 1786961895.716366 mapwrite 0x0376f0 thru 0x03ffff (0x8910 bytes) 000143 1786961895.832032 mapread 0x006f5e thru 0x00c742 (0x57e5 bytes) 000144 1786961895.834419 trunc from 0x040000 to 0x0163e0 (0x29c1f bytes) 000145 1786961895.925770 MIRROR_RESYNC 2 mirrors 000146 1786961896.513242 mapwrite 0x01bfa2 thru 0x023c3c (0x7c9b bytes) 000147 1786961896.627788 punch from 0x00437f to 0x012f5d (0xebdf bytes) 000148 1786961896.648180 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x23c3d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23c3d): Mirror 2: 0x663db1d6 (stale) CRC-32 checksum value for chunk [0, 0x23c3d): Mirror 1: 0x2f92e167 Mirror 3: 0x2f92e167 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23c3d: skipped 000149 1786961897.703749 fallocate 0x01246f thru 0x020d49 (0xe8db bytes) 000150 1786961897.739175 trunc from 0x023c3d to 0x0291cb (0x558f bytes) 000151 1786961897.772608 MIRROR_RESYNC 3 mirrors 000152 1786961898.938944 MIRROR_EXTEND to 4 mirrors 000153 1786961899.654571 read 0x007ea5 thru 0x0124f8 (0xa654 bytes) 000154 1786961899.679073 write 0x02ff4d thru 0x0379da (0x7a8e bytes) 000155 1786961899.738976 write 0x011b6c thru 0x0208ff (0xed94 bytes) 000157 1786961899.762125 mapwrite 0x037c52 thru 0x038f48 (0x12f7 bytes) 000158 1786961899.859778 fallocate 0x030aee thru 0x0345b9 (0x3acc bytes) 000159 1786961899.866679 mapwrite 0x021009 thru 0x0285f6 (0x75ee bytes) 000160 1786961899.912048 read 0x023527 thru 0x029180 (0x5c5a bytes) 000161 1786961899.918346 MIRROR_RESYNC 4 mirrors 000162 1786961901.235403 MIRROR_RESYNC 4 mirrors 000163 1786961901.833249 mapread 0x00c509 thru 0x01799b (0xb493 bytes) 000164 1786961901.864319 fallocate 0x038c31 thru 0x03ffff (0x73cf bytes) 000165 1786961901.894637 mapwrite 0x02ac04 thru 0x036103 (0xb500 bytes) 000166 1786961901.937822 MIRROR_RESYNC 4 mirrors 000167 1786961903.070164 punch from 0x01dfb9 to 0x0234d3 (0x551b bytes) 000168 1786961903.103338 mapread 0x02f132 thru 0x031fd2 (0x2ea1 bytes) 000169 1786961903.125911 fallocate 0x00ffdb thru 0x016fbb (0x6fe1 bytes) 000170 1786961903.131592 trunc from 0x038f49 to 0x01cb31 (0x1c417 bytes) 000171 1786961903.157077 fallocate 0x009478 thru 0x00f85a (0x63e3 bytes) 000172 1786961903.163927 trunc from 0x01cb32 to 0x01bfb0 (0xb81 bytes) 000173 1786961903.181289 trunc from 0x0027aa to 0x00f5db (0xce32 bytes) 000174 1786961903.188138 write 0x022379 thru 0x022968 (0x5f0 bytes) 000175 1786961903.194747 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22969): Mirror 2: 0x9fef63e6 (stale) Mirror 4: 0x9fef63e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22969): Mirror 2: 0x9fef63e6 (stale) Mirror 4: 0x9fef63e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22969: skipped 000176 1786961903.713822 mapread 0x0043e1 thru 0x00c92d (0x854d bytes) 000177 1786961903.771031 write 0x01560d thru 0x01ee11 (0x9805 bytes) 000178 1786961903.798615 write 0x0188d4 thru 0x021688 (0x8db5 bytes) 000179 1786961903.814370 mapread 0x013db6 thru 0x01748b (0x36d6 bytes) 000180 1786961903.836894 read 0x00c9a9 thru 0x00fe0b (0x3463 bytes) 000182 1786961903.845114 read 0x004d04 thru 0x00f40f (0xa70c bytes) 000183 1786961903.848421 mapwrite 0x00c331 thru 0x00e576 (0x2246 bytes) 000184 1786961903.890060 read 0x01889f thru 0x022968 (0xa0ca bytes) 000185 1786961903.902094 trunc from 0x022969 to 0x00a200 (0x18768 bytes) 000186 1786961903.924946 punch from 0x001e5a to 0x00a200 (0x83a7 bytes) 000187 1786961903.936093 trunc from 0x002cca to 0x00a200 (0x7537 bytes) 000188 1786961903.944892 mapwrite 0x03c39d thru 0x03e19a (0x1dfe bytes) 000189 1786961904.003553 mapread 0x02b78c thru 0x030f42 (0x57b7 bytes) 000190 1786961904.005243 punch from 0x02db36 to 0x030c6a (0x3135 bytes) 000191 1786961904.018789 trunc from 0x03c64e to 0x03e19a (0x1b4d bytes) 000194 1786961904.027812 mapwrite 0x02b150 thru 0x034003 (0x8eb4 bytes) 000195 1786961904.102640 read 0x022699 thru 0x0322f1 (0xfc59 bytes) 000197 1786961904.114360 write 0x022f2c thru 0x02e933 (0xba08 bytes) 000198 1786961904.119953 punch from 0x0164c5 to 0x01b0ed (0x4c29 bytes) 000199 1786961904.151569 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e19b): Mirror 2: 0xe2d134d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e19b): Mirror 2: 0xe2d134d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e19b: skipped 000200 1786961904.701919 mapread 0x0337a3 thru 0x033b07 (0x365 bytes) 000201 1786961904.721287 mapwrite 0x004aa7 thru 0x00eb7f (0xa0d9 bytes) 000202 1786961904.783425 mapread 0x02570d thru 0x02ef88 (0x987c bytes) 000203 1786961904.827360 punch from 0x020570 to 0x02a12d (0x9bbe bytes) 000204 1786961904.837536 mapwrite 0x03a59b thru 0x03c7bd (0x2223 bytes) 000206 1786961904.886055 MIRROR_RESYNC 2 mirrors 000207 1786961905.929714 trunc from 0x03e19b to 0x006ac3 (0x376d7 bytes) 000208 1786961906.007883 punch from 0x003602 to 0x006ac3 (0x34c2 bytes) 000209 1786961906.019006 trunc from 0x006ac4 to 0x03610a (0x2f647 bytes) 000210 1786961906.060293 read 0x01b655 thru 0x0213c5 (0x5d71 bytes) 000211 1786961906.074806 write 0x02e845 thru 0x03e07e (0xf83a bytes) 000212 1786961906.091751 fallocate 0x034105 thru 0x03978f (0x568b bytes) 000213 1786961906.106646 read 0x0358eb thru 0x03e07e (0x8794 bytes) 000214 1786961906.108577 write 0x03406f thru 0x03972e (0x56c0 bytes) 000215 1786961906.111012 mapwrite 0x002f5a thru 0x004ddb (0x1e82 bytes) 000216 1786961906.192118 mapread 0x0287a7 thru 0x02f234 (0x6a8e bytes) 000217 1786961906.195937 mapread 0x02e22e thru 0x0305bc (0x238f bytes) 000218 1786961906.208363 MIRROR_RESYNC 2 mirrors 000220 1786961907.012565 mapwrite 0x01c6b1 thru 0x02769b (0xafeb bytes) 000221 1786961907.160570 write 0x01617b thru 0x01b977 (0x57fd bytes) 000222 1786961907.162329 read 0x03427c thru 0x03e07e (0x9e03 bytes) 000223 1786961907.188233 fallocate 0x03e1f8 thru 0x03ffff (0x1e08 bytes) 000224 1786961907.194421 fallocate 0x02259f thru 0x02aadd (0x853f bytes) 000225 1786961907.200600 trunc from 0x00a6b2 to 0x00e3c8 (0x3d17 bytes) 000226 1786961907.206140 fallocate 0x029f8c thru 0x035ce5 (0xbd5a bytes) 000227 1786961907.216581 MIRROR_RESYNC 2 mirrors 000228 1786961908.047983 punch from 0x006ee4 to 0x012bf0 (0xbd0d bytes) 000229 1786961908.131349 punch from 0x031f56 to 0x039793 (0x783e bytes) 000230 1786961908.149548 mapread 0x006995 thru 0x0118ae (0xaf1a bytes) 000231 1786961908.234438 mapwrite 0x02a9f9 thru 0x02ed1a (0x4322 bytes) 000232 1786961908.282243 MIRROR_RESYNC 2 mirrors 000233 1786961909.078504 MIRROR_EXTEND to 3 mirrors 000234 1786961910.151805 write 0x016d02 thru 0x022da1 (0xc0a0 bytes) 000235 1786961910.206585 mapwrite 0x007547 thru 0x0090cc (0x1b86 bytes) 000236 1786961910.250809 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e07f): Mirror 3: 0x67c2ab97 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e07f): Mirror 3: 0x67c2ab97 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e07f: skipped 000237 1786961910.671384 write 0x013cfa thru 0x02088d (0xcb94 bytes) 000238 1786961910.687747 trunc from 0x03e07f to 0x00962f (0x34a4f bytes) 000239 1786961910.736262 punch from 0x002b8b to 0x002da5 (0x21b bytes) 000240 1786961910.748526 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x9630): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9630): Mirror 3: 0xfeb3541 (stale) CRC-32 checksum value for chunk [0, 0x9630): Mirror 1: 0x84afc955 Mirror 4: 0x84afc955 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9630: skipped 000241 1786961912.072537 read 0x0095af thru 0x00962f (0x81 bytes) 000242 1786961912.100688 mapread 0x006bc2 thru 0x00962f (0x2a6e bytes) 000243 1786961912.187365 trunc from 0x009630 to 0x0296c7 (0x20098 bytes) 000244 1786961912.261923 MIRROR_RESYNC 3 mirrors 000245 1786961913.501324 punch from 0x022c50 to 0x0270b7 (0x4468 bytes) 000246 1786961913.539405 trunc from 0x0296c8 to 0x02d56e (0x3ea7 bytes) 000247 1786961913.596372 trunc from 0x0269cb to 0x02d4fb (0x6b31 bytes) 000248 1786961913.608496 read 0x011a51 thru 0x011ea6 (0x456 bytes) 000250 1786961913.619208 mapread 0x01085d thru 0x010940 (0xe4 bytes) 000251 1786961913.625436 trunc from 0x02c2b7 to 0x02d56e (0x12b8 bytes) 000252 1786961913.632524 trunc from 0x01e00e to 0x027720 (0x9713 bytes) 000253 1786961913.649828 punch from 0x0171ba to 0x0252b2 (0xe0f9 bytes) 000254 1786961913.655799 read 0x0242fa thru 0x02d56e (0x9275 bytes) 000255 1786961913.696756 mapread 0x02d44e thru 0x02d56e (0x121 bytes) 000256 1786961913.697802 read 0x01b12a thru 0x0200f0 (0x4fc7 bytes) 000257 1786961913.698451 punch from 0x017bf3 to 0x01f69c (0x7aaa bytes) 000258 1786961913.704207 punch from 0x0060ae to 0x009c68 (0x3bbb bytes) 000259 1786961913.710028 punch from 0x010263 to 0x01ea96 (0xe834 bytes) 000260 1786961913.714518 write 0x03dee0 thru 0x03ffff (0x2120 bytes) 000261 1786961913.715719 write 0x01ac48 thru 0x01c6e0 (0x1a99 bytes) 000262 1786961913.726352 punch from 0x035a3f to 0x0388fa (0x2ebc bytes) 000263 1786961913.730813 punch from 0x03c511 to 0x03ffff (0x3aef bytes) 000264 1786961913.788258 mapwrite 0x0117c6 thru 0x017381 (0x5bbc bytes) 000265 1786961913.822554 mapread 0x0357e0 thru 0x03ffff (0xa820 bytes) 000266 1786961913.926933 punch from 0x021ca0 to 0x02ac6f (0x8fd0 bytes) 000267 1786961913.952758 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaf7b78fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaf7b78fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000268 1786961914.305015 trunc from 0x040000 to 0x0034e0 (0x3cb1f bytes) 000269 1786961914.345621 mapwrite 0x017ee1 thru 0x021295 (0x93b5 bytes) 000270 1786961914.415998 write 0x023ac6 thru 0x0320f9 (0xe634 bytes) 000271 1786961914.417908 fallocate 0x011a59 thru 0x016d5f (0x5307 bytes) 000272 1786961914.423244 trunc from 0x015686 to 0x015fac (0x927 bytes) 000273 1786961914.428570 write 0x035ec1 thru 0x039192 (0x32d2 bytes) 000274 1786961914.430167 trunc from 0x039193 to 0x019dc4 (0x1f3ce bytes) 000275 1786961914.446757 mapwrite 0x01898d thru 0x019662 (0xcd6 bytes) 000276 1786961914.462576 trunc from 0x012c38 to 0x019dc4 (0x718d bytes) 000278 1786961914.468694 fallocate 0x01f9cc thru 0x026c52 (0x7287 bytes) 000279 1786961914.475311 fallocate 0x03c4fe thru 0x03ffff (0x3b02 bytes) 000280 1786961914.483824 fallocate 0x039904 thru 0x03a58f (0xc8c bytes) 000281 1786961914.490634 trunc from 0x02ed7f to 0x03a58f (0xb811 bytes) 000282 1786961914.496665 trunc from 0x01c9b9 to 0x0284a3 (0xbaeb bytes) 000283 1786961914.502829 mapwrite 0x00ef32 thru 0x01be6e (0xcf3d bytes) 000284 1786961914.552244 mapwrite 0x02795c thru 0x027a6d (0x112 bytes) 000286 1786961914.573461 punch from 0x024564 to 0x033d96 (0xf833 bytes) 000287 1786961914.579382 punch from 0x0350f9 to 0x03a58f (0x5497 bytes) 000288 1786961914.591459 MIRROR_RESYNC 2 mirrors 000289 1786961915.319747 read 0x0222b2 thru 0x027e0e (0x5b5d bytes) 000290 1786961915.339822 trunc from 0x03a590 to 0x039915 (0xc7a bytes) 000291 1786961915.456652 MIRROR_RESYNC 2 mirrors 000292 1786961916.346133 mapwrite 0x039695 thru 0x03be39 (0x27a5 bytes) 000293 1786961916.484037 trunc from 0x03be3a to 0x034ee0 (0x6f59 bytes) 000294 1786961916.514396 write 0x005afe thru 0x00d236 (0x7739 bytes) 000295 1786961916.529977 mapwrite 0x0092fd thru 0x0131a9 (0x9ead bytes) 000296 1786961916.571451 mapread 0x00af7a thru 0x00c725 (0x17ac bytes) 000297 1786961916.572923 punch from 0x016ed9 to 0x01e94c (0x7a74 bytes) 000298 1786961916.583291 trunc from 0x034ee1 to 0x00ccb8 (0x28228 bytes) 000299 1786961916.629993 punch from 0x00058c to 0x00338c (0x2e01 bytes) 000300 1786961916.640348 write 0x012086 thru 0x01382b (0x17a6 bytes) 000301 1786961916.642318 punch from 0x012cb1 to 0x01382b (0xb7b bytes) 000302 1786961916.654981 fallocate 0x02f88f thru 0x0392b4 (0x9a26 bytes) 000303 1786961916.661219 read 0x012f6d thru 0x01382b (0x8bf bytes) 000304 1786961916.673800 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1382c): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1382c): Mirror 3: 0x7db761c (stale) CRC-32 checksum value for chunk [0, 0x1382c): Mirror 1: 0xbb52d8b8 Mirror 4: 0xbb52d8b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1382c: skipped 000305 1786961917.696641 fallocate 0x008139 thru 0x00b1e4 (0x30ac bytes) 000306 1786961917.770272 punch from 0x00b487 to 0x00ff25 (0x4a9f bytes) 000308 1786961917.776818 write 0x01f75c thru 0x02a67a (0xaf1f bytes) 000309 1786961917.789754 mapread 0x017403 thru 0x023c8e (0xc88c bytes) 000310 1786961917.810724 fallocate 0x023cc1 thru 0x024878 (0xbb8 bytes) 000311 1786961917.821248 trunc from 0x0138c1 to 0x0174f0 (0x3c30 bytes) 000312 1786961917.826645 punch from 0x01dcc8 to 0x026bb4 (0x8eed bytes) 000313 1786961917.860272 mapread 0x001da6 thru 0x00c00f (0xa26a bytes) 000314 1786961917.872414 fallocate 0x02e9f8 thru 0x03dee9 (0xf4f2 bytes) 000315 1786961917.877748 read 0x00870f thru 0x0135cb (0xaebd bytes) 000316 1786961917.890287 write 0x039ad7 thru 0x03ffff (0x6529 bytes) 000317 1786961917.900268 write 0x0201a8 thru 0x02cee7 (0xcd40 bytes) 000318 1786961917.924055 read 0x00461a thru 0x0106c8 (0xc0af bytes) 000319 1786961917.924803 read 0x02d968 thru 0x03315b (0x57f4 bytes) 000320 1786961917.941387 mapread 0x017a8a thru 0x026dbb (0xf332 bytes) 000321 1786961917.942816 trunc from 0x040000 to 0x03fb80 (0x47f bytes) 000322 1786961917.985832 mapwrite 0x02a346 thru 0x02a575 (0x230 bytes) 000323 1786961918.016591 punch from 0x03c522 to 0x03fb80 (0x365f bytes) 000324 1786961918.023545 trunc from 0x01e5c3 to 0x024f6b (0x69a9 bytes) 000325 1786961918.032409 read 0x033f0b thru 0x03aeca (0x6fc0 bytes) 000326 1786961918.037892 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fb81): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb81): Mirror 3: 0xaf361867 (stale) Mirror 4: 0x62ffdd06 (stale) CRC-32 checksum value for chunk [0, 0x3fb81): Mirror 1: 0x35da078 Mirror 5: 0x35da078 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb81: skipped 000327 1786961919.064267 trunc from 0x03fb81 to 0x00e80b (0x31375 bytes) 000328 1786961919.114531 trunc from 0x004174 to 0x00e80b (0xa698 bytes) 000330 1786961919.121539 trunc from 0x00cd60 to 0x00e80b (0x1aac bytes) 000331 1786961919.127446 punch from 0x00e2d1 to 0x00e80b (0x53b bytes) 000332 1786961919.131829 fallocate 0x029b41 thru 0x037920 (0xdde0 bytes) 000334 1786961919.138015 write 0x036cf9 thru 0x03991e (0x2c26 bytes) 000335 1786961919.143966 trunc from 0x03991f to 0x027e61 (0x11abd bytes) 000336 1786961919.166457 read 0x013022 thru 0x019036 (0x6015 bytes) 000337 1786961919.178043 mapwrite 0x030b0b thru 0x03c1c0 (0xb6b6 bytes) 000338 1786961919.259614 write 0x014a18 thru 0x01a95f (0x5f48 bytes) 000339 1786961919.274034 trunc from 0x03c1c1 to 0x01c03a (0x20186 bytes) 000340 1786961919.299715 mapread 0x0122b1 thru 0x015e54 (0x3ba4 bytes) 000341 1786961919.311488 trunc from 0x01c03b to 0x016456 (0x5be4 bytes) 000342 1786961919.329143 read 0x0162d0 thru 0x016456 (0x187 bytes) 000343 1786961919.330183 write 0x0379f7 thru 0x03e1a8 (0x67b2 bytes) 000344 1786961919.333586 mapwrite 0x0311de thru 0x03b2f4 (0xa117 bytes) 000345 1786961919.389281 punch from 0x01c4f3 to 0x024cf5 (0x8803 bytes) 000346 1786961919.396167 punch from 0x020792 to 0x02d1dc (0xca4b bytes) 000347 1786961919.405067 write 0x00fe25 thru 0x01eef5 (0xf0d1 bytes) 000348 1786961919.420292 read 0x00e16f thru 0x015c0a (0x7a9c bytes) 000350 1786961919.422271 punch from 0x037889 to 0x03e1a8 (0x6920 bytes) 000351 1786961919.436722 mapwrite 0x00a552 thru 0x0103cc (0x5e7b bytes) 000352 1786961919.468325 fallocate 0x02f457 thru 0x03a5d7 (0xb181 bytes) 000353 1786961919.476607 mapread 0x011d71 thru 0x014ae6 (0x2d76 bytes) 000354 1786961919.485489 punch from 0x021806 to 0x02c461 (0xac5c bytes) 000356 1786961919.504179 trunc from 0x03e1a9 to 0x02e4c6 (0xfce2 bytes) 000357 1786961919.532815 read 0x013b28 thru 0x0212ab (0xd784 bytes) 000358 1786961919.563687 trunc from 0x0229e6 to 0x023a23 (0x103e bytes) 000359 1786961919.581572 mapread 0x011b4d thru 0x020ae1 (0xef95 bytes) 000360 1786961919.583039 punch from 0x01ff27 to 0x02e4c6 (0xe5a0 bytes) 000361 1786961919.590317 mapwrite 0x002d3f thru 0x00bafb (0x8dbd bytes) 000362 1786961919.629339 write 0x02624a thru 0x02f81a (0x95d1 bytes) 000363 1786961919.650645 write 0x019d0e thru 0x01ce8c (0x317f bytes) 000364 1786961919.652470 read 0x000af5 thru 0x00c6cb (0xbbd7 bytes) 000365 1786961919.653293 fallocate 0x01681c thru 0x02173d (0xaf22 bytes) 000366 1786961919.681492 read 0x010cb8 thru 0x011fcb (0x1314 bytes) 000367 1786961919.682371 read 0x013a69 thru 0x01e017 (0xa5af bytes) 000368 1786961919.683063 mapread 0x02aba6 thru 0x02f81a (0x4c75 bytes) 000369 1786961919.684105 mapwrite 0x00aa97 thru 0x01a3ca (0xf934 bytes) 000370 1786961919.741320 mapwrite 0x00e38a thru 0x017141 (0x8db8 bytes) 000371 1786961919.796937 punch from 0x00d7cb to 0x01a35c (0xcb92 bytes) 000372 1786961919.805837 mapread 0x0209c1 thru 0x02dec4 (0xd504 bytes) 000373 1786961919.837991 write 0x01ae05 thru 0x029112 (0xe30e bytes) 000375 1786961919.856982 fallocate 0x00a76d thru 0x00d34b (0x2bdf bytes) 000376 1786961919.868690 MIRROR_RESYNC 4 mirrors 000377 1786961921.365832 read 0x025ff6 thru 0x029beb (0x3bf6 bytes) 000378 1786961921.387986 trunc from 0x02f81b to 0x01fbb2 (0xfc68 bytes) 000379 1786961921.430588 punch from 0x015552 to 0x01fbb2 (0xa661 bytes) 000380 1786961921.436311 fallocate 0x025b83 thru 0x031247 (0xb6c5 bytes) 000381 1786961921.445650 punch from 0x00322f to 0x00e7ac (0xb57e bytes) 000382 1786961921.450025 read 0x02b915 thru 0x031247 (0x5933 bytes) 000383 1786961921.460577 trunc from 0x031248 to 0x00f077 (0x221d0 bytes) 000384 1786961921.481788 fallocate 0x012f15 thru 0x016a15 (0x3b01 bytes) 000385 1786961921.487746 punch from 0x00f2d6 to 0x016a15 (0x7740 bytes) 000387 1786961921.493644 fallocate 0x006bca thru 0x011c39 (0xb070 bytes) 000388 1786961921.498739 mapread 0x010974 thru 0x016a15 (0x60a2 bytes) 000389 1786961921.547019 read 0x011137 thru 0x016a15 (0x58df bytes) 000390 1786961921.547745 trunc from 0x00e5e6 to 0x014b39 (0x6554 bytes) 000391 1786961921.563407 trunc from 0x016a16 to 0x005507 (0x1150e bytes) 000392 1786961921.588827 mapread 0x000e4c thru 0x005507 (0x46bc bytes) 000393 1786961921.590647 fallocate 0x00daec thru 0x010d5e (0x3273 bytes) 000394 1786961921.595814 fallocate 0x0188d5 thru 0x020311 (0x7a3d bytes) 000396 1786961921.613596 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20312): Mirror 3: 0xe3bbcc30 (stale) Mirror 5: 0xe3bbcc30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20312): Mirror 3: 0xe3bbcc30 (stale) Mirror 5: 0xe3bbcc30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20312: skipped 000397 1786961922.137573 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20312): Mirror 5: 0xe3bbcc30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20312): Mirror 5: 0xe3bbcc30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20312: skipped 000398 1786961922.454987 mapread 0x01c193 thru 0x020311 (0x417f bytes) 000399 1786961922.475243 read 0x008100 thru 0x00ee70 (0x6d71 bytes) 000400 1786961922.486531 mapwrite 0x01b8df thru 0x01d38a (0x1aac bytes) 000401 1786961922.513143 fallocate 0x01711d thru 0x02579a (0xe67e bytes) 000402 1786961922.522225 fallocate 0x0002fc thru 0x0027f5 (0x24fa bytes) 000404 1786961922.535145 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000405 1786961922.720901 fallocate 0x00be15 thru 0x00fff0 (0x41dc bytes) 000406 1786961922.732543 read 0x01a6b0 thru 0x01feba (0x580b bytes) 000407 1786961922.754631 mapread 0x00d9bc thru 0x011342 (0x3987 bytes) 000408 1786961922.777679 punch from 0x00ca8d to 0x014ed6 (0x844a bytes) 000409 1786961922.785702 trunc from 0x01104d to 0x014265 (0x3219 bytes) 000410 1786961922.790990 punch from 0x003eae to 0x00de04 (0x9f57 bytes) 000411 1786961922.798208 trunc from 0x020312 to 0x01ffc5 (0x34c bytes) 000412 1786961922.828774 mapwrite 0x012b37 thru 0x018f53 (0x641d bytes) 000413 1786961922.863174 read 0x001c44 thru 0x0101f8 (0xe5b5 bytes) 000414 1786961922.876558 trunc from 0x005e53 to 0x014ea4 (0xf052 bytes) 000415 1786961922.883756 punch from 0x0137da to 0x01d846 (0xa06d bytes) 000416 1786961922.890818 trunc from 0x01e377 to 0x01edc4 (0xa4e bytes) 000417 1786961922.896512 write 0x011fa6 thru 0x01afb7 (0x9012 bytes) 000418 1786961922.923608 mapread 0x001072 thru 0x0108dd (0xf86c bytes) 000419 1786961923.009303 fallocate 0x01283c thru 0x013927 (0x10ec bytes) 000420 1786961923.015490 trunc from 0x01ffc6 to 0x01d07f (0x2f46 bytes) 000421 1786961923.033936 mapread 0x00498b thru 0x012554 (0xdbca bytes) 000423 1786961923.036933 mapwrite 0x000e30 thru 0x009bc2 (0x8d93 bytes) 000424 1786961923.063789 punch from 0x00d0cd to 0x00e52e (0x1462 bytes) 000426 1786961923.071788 mapwrite 0x01dbf5 thru 0x02a813 (0xcc1f bytes) 000427 1786961923.164313 trunc from 0x02a814 to 0x03604a (0xb837 bytes) 000428 1786961923.196769 mapread 0x019018 thru 0x01a345 (0x132e bytes) 000429 1786961923.201159 punch from 0x02b048 to 0x03604a (0xb003 bytes) 000431 1786961923.217589 read 0x024d4c thru 0x02669d (0x1952 bytes) 000432 1786961923.218713 read 0x02272b thru 0x027136 (0x4a0c bytes) 000433 1786961923.220253 read 0x030696 thru 0x03604a (0x59b5 bytes) 000434 1786961923.242242 mapread 0x0341bb thru 0x03604a (0x1e90 bytes) 000436 1786961923.256728 MIRROR_EXTEND to 2 mirrors 000437 1786961924.179479 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000438 1786961924.338616 trunc from 0x03604b to 0x00c8c1 (0x29789 bytes) 000439 1786961924.419696 MIRROR_EXTEND to 2 mirrors 000440 1786961925.566895 fallocate 0x021c1d thru 0x024516 (0x28fa bytes) 000441 1786961925.621129 fallocate 0x025ee7 thru 0x034bda (0xecf4 bytes) 000442 1786961925.629546 punch from 0x006d2d to 0x0135a8 (0xc87c bytes) 000443 1786961925.634453 read 0x00ce3c thru 0x01355f (0x6724 bytes) 000444 1786961925.647525 mapread 0x02d594 thru 0x034bda (0x7647 bytes) 000445 1786961925.673354 trunc from 0x034bdb to 0x03c7d9 (0x7bff bytes) 000446 1786961925.716563 fallocate 0x015c1e thru 0x01a245 (0x4628 bytes) 000447 1786961925.723678 trunc from 0x03c7da to 0x03fd39 (0x3560 bytes) 000448 1786961925.755824 mapread 0x013701 thru 0x021d90 (0xe690 bytes) 000449 1786961925.803561 trunc from 0x02d594 to 0x02d88c (0x2f9 bytes) 000450 1786961925.817273 punch from 0x037755 to 0x039742 (0x1fee bytes) 000451 1786961925.856809 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fd3a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd3a): Mirror 2: 0xb6cccb15 (stale) CRC-32 checksum value for chunk [0, 0x3fd3a): Mirror 1: 0xe7e17947 Mirror 3: 0xe7e17947 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd3a: skipped 000452 1786961927.267457 write 0x01e9fd thru 0x02183c (0x2e40 bytes) 000453 1786961927.342235 write 0x02642a thru 0x0353ac (0xef83 bytes) 000454 1786961927.379826 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fd3a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd3a): Mirror 2: 0xb6cccb15 (stale) Mirror 3: 0xe7e17947 (stale) CRC-32 checksum value for chunk [0, 0x3fd3a): Mirror 1: 0x327e7d78 Mirror 4: 0x327e7d78 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd3a: skipped 000455 1786961928.861332 trunc from 0x03fd3a to 0x0335ba (0xc77f bytes) 000456 1786961928.994696 fallocate 0x0244e7 thru 0x03286d (0xe387 bytes) 000457 1786961929.019475 punch from 0x01b8e2 to 0x02b83b (0xff5a bytes) 000458 1786961929.043908 trunc from 0x0335bb to 0x038f79 (0x59bf bytes) 000459 1786961929.083431 fallocate 0x0138ff thru 0x02235e (0xea60 bytes) 000460 1786961929.092467 read 0x007f4d thru 0x010798 (0x884c bytes) 000461 1786961929.106577 trunc from 0x0170ea to 0x018865 (0x177c bytes) 000462 1786961929.112604 trunc from 0x02627b to 0x03077d (0xa503 bytes) 000463 1786961929.117843 fallocate 0x02bac6 thru 0x03733e (0xb879 bytes) 000464 1786961929.126953 punch from 0x01772b to 0x01f1d8 (0x7aae bytes) 000466 1786961929.135471 trunc from 0x02ba65 to 0x03642b (0xa9c7 bytes) 000467 1786961929.141486 write 0x022ab8 thru 0x031a71 (0xefba bytes) 000468 1786961929.161404 mapwrite 0x03968e thru 0x03ffff (0x6972 bytes) 000469 1786961929.237980 read 0x02cdf3 thru 0x033389 (0x6597 bytes) 000471 1786961929.257446 trunc from 0x040000 to 0x033ca1 (0xc35e bytes) 000472 1786961929.284501 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33ca2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ca2): Mirror 2: 0x747b137e (stale) Mirror 3: 0xaf3c1f4 (stale) Mirror 4: 0xfef7d531 (stale) CRC-32 checksum value for chunk [0, 0x33ca2): Mirror 1: 0xf53c66cb Mirror 5: 0xf53c66cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ca2: skipped 000473 1786961930.431254 mapread 0x01f010 thru 0x02cf11 (0xdf02 bytes) 000474 1786961930.484446 mapwrite 0x02e284 thru 0x038577 (0xa2f4 bytes) 000475 1786961930.629843 trunc from 0x028547 to 0x02fb58 (0x7612 bytes) 000476 1786961930.644088 mapwrite 0x002d9a thru 0x006179 (0x33e0 bytes) 000477 1786961930.684925 MIRROR_RESYNC 5 mirrors 000478 1786961932.490893 mapread 0x016ec0 thru 0x01d8ff (0x6a40 bytes) 000479 1786961932.528267 mapread 0x01c5d7 thru 0x026954 (0xa37e bytes) 000480 1786961932.536821 punch from 0x003d73 to 0x012f4c (0xf1da bytes) 000481 1786961932.608801 MIRROR_RESYNC 5 mirrors 000482 1786961934.520031 read 0x00d2c0 thru 0x012ba2 (0x58e3 bytes) 000483 1786961934.543686 MIRROR_RESYNC 5 mirrors 000484 1786961935.281250 MIRROR_EXTEND to 6 mirrors 000485 1786961936.735687 mapwrite 0x015f8f thru 0x01ebd3 (0x8c45 bytes) 000486 1786961937.016249 punch from 0x0275d3 to 0x02fd67 (0x8795 bytes) 000487 1786961937.025467 write 0x034309 thru 0x03e244 (0x9f3c bytes) 000488 1786961937.051265 mapread 0x032f2c thru 0x03b261 (0x8336 bytes) 000489 1786961937.091264 write 0x024d4c thru 0x02db5a (0x8e0f bytes) 000490 1786961937.103022 mapread 0x01a551 thru 0x01d92d (0x33dd bytes) 000491 1786961937.120437 MIRROR_RESYNC 6 mirrors 000492 1786961939.660623 read 0x0034a8 thru 0x00526f (0x1dc8 bytes) 000493 1786961939.670516 trunc from 0x03a635 to 0x03e244 (0x3c10 bytes) 000496 1786961939.721963 read 0x036205 thru 0x03e244 (0x8040 bytes) 000497 1786961939.742924 fallocate 0x03f4f9 thru 0x03ffff (0xb07 bytes) 000498 1786961939.755600 mapread 0x01b38f thru 0x01e5b9 (0x322b bytes) 000499 1786961939.767802 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x94b5c759 (stale) Mirror 3: 0x94b5c759 (stale) Mirror 4: 0x94b5c759 (stale) Mirror 6: 0x94b5c759 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x94b5c759 (stale) Mirror 3: 0x94b5c759 (stale) Mirror 4: 0x94b5c759 (stale) Mirror 6: 0x94b5c759 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000500 1786961940.324469 write 0x0020c3 thru 0x00cb64 (0xaaa2 bytes) 000501 1786961940.345769 trunc from 0x00fd72 to 0x0178f3 (0x7b82 bytes) 000502 1786961940.356348 trunc from 0x0107fa to 0x01ec78 (0xe47f bytes) 000503 1786961940.363605 trunc from 0x018d13 to 0x0210a1 (0x838f bytes) 000504 1786961940.369489 read 0x0116f1 thru 0x0160f8 (0x4a08 bytes) 000505 1786961940.380325 trunc from 0x040000 to 0x00b4f8 (0x34b07 bytes) 000506 1786961940.408627 MIRROR_RESYNC 5 mirrors 000507 1786961941.527534 fallocate 0x029157 thru 0x036357 (0xd201 bytes) 000508 1786961941.625412 MIRROR_RESYNC 5 mirrors 000509 1786961942.720519 read 0x003b05 thru 0x003e3b (0x337 bytes) 000510 1786961942.728233 write 0x021b61 thru 0x03069d (0xeb3d bytes) 000512 1786961942.764097 trunc from 0x00f39c to 0x016b7c (0x77e1 bytes) 000513 1786961942.772173 trunc from 0x03069e to 0x0105b6 (0x200e7 bytes) 000514 1786961942.800499 punch from 0x00e0bf to 0x0105b6 (0x24f8 bytes) 000515 1786961942.806704 write 0x031815 thru 0x0378ac (0x6098 bytes) 000516 1786961942.815401 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x378ad): Mirror 2: 0x1d839d1c (stale) Mirror 4: 0x1d839d1c (stale) Mirror 6: 0x1d839d1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x378ad): Mirror 2: 0x1d839d1c (stale) Mirror 4: 0x1d839d1c (stale) Mirror 6: 0x1d839d1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x378ad: skipped 000517 1786961943.264695 MIRROR_RESYNC 4 mirrors 000518 1786961944.418005 trunc from 0x0378ad to 0x02a1a0 (0xd70c bytes) 000519 1786961944.461634 read 0x028d43 thru 0x02a1a0 (0x145e bytes) 000520 1786961944.470407 fallocate 0x01fe7a thru 0x028223 (0x83aa bytes) 000522 1786961944.477569 trunc from 0x014b80 to 0x0225fe (0xda7f bytes) 000523 1786961944.483622 trunc from 0x01b824 to 0x02226b (0x6a48 bytes) 000524 1786961944.489747 trunc from 0x00f24e to 0x01ac84 (0xba37 bytes) 000525 1786961944.495736 write 0x01de5b thru 0x01ff4f (0x20f5 bytes) 000526 1786961944.507590 trunc from 0x008f7b to 0x00a253 (0x12d9 bytes) 000527 1786961944.513669 trunc from 0x02a1a1 to 0x01d3ee (0xcdb2 bytes) 000528 1786961944.548357 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d3ef): Mirror 4: 0xef5c673b (stale) Mirror 6: 0xef5c673b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d3ef): Mirror 4: 0xef5c673b (stale) Mirror 6: 0xef5c673b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d3ef: skipped 000529 1786961945.023472 punch from 0x00dfa3 to 0x01d3ee (0xf44c bytes) 000530 1786961945.033735 mapwrite 0x03a7ec thru 0x03ffff (0x5814 bytes) 000531 1786961945.149137 fallocate 0x014679 thru 0x021710 (0xd098 bytes) 000532 1786961945.163991 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb2ac3f7a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb2ac3f7a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000533 1786961945.543841 MIRROR_RESYNC 2 mirrors 000534 1786961946.568673 punch from 0x02df4c to 0x0355f3 (0x76a8 bytes) 000535 1786961946.628673 fallocate 0x00af41 thru 0x018a0c (0xdacc bytes) 000536 1786961946.634147 punch from 0x0194b0 to 0x01a0fe (0xc4f bytes) 000537 1786961946.650317 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000538 1786961946.890665 mapread 0x03b332 thru 0x03c99e (0x166d bytes) 000540 1786961946.908544 trunc from 0x040000 to 0x0110de (0x2ef21 bytes) 000541 1786961946.936071 trunc from 0x00e62c to 0x00eb54 (0x529 bytes) 000542 1786961946.943736 trunc from 0x010490 to 0x0110de (0xc4f bytes) 000544 1786961946.950268 read 0x006c7b thru 0x00e869 (0x7bef bytes) 000545 1786961946.963527 write 0x0206b1 thru 0x028042 (0x7992 bytes) 000546 1786961946.967014 punch from 0x000fe6 to 0x005360 (0x437b bytes) 000547 1786961946.973597 trunc from 0x028043 to 0x030a1c (0x89da bytes) 000548 1786961946.993666 trunc from 0x01e317 to 0x022d24 (0x4a0e bytes) 000550 1786961947.012832 read 0x011d1a thru 0x01c450 (0xa737 bytes) 000551 1786961947.026799 mapread 0x02f754 thru 0x030a1c (0x12c9 bytes) 000552 1786961947.028255 write 0x01b9e7 thru 0x029051 (0xd66b bytes) 000554 1786961947.034798 punch from 0x014786 to 0x01a076 (0x58f1 bytes) 000555 1786961947.044279 trunc from 0x030a1d to 0x00228e (0x2e78e bytes) 000557 1786961947.072340 mapwrite 0x01d0cc thru 0x025511 (0x8446 bytes) 000558 1786961947.248543 mapwrite 0x02ecbb thru 0x0365d6 (0x791c bytes) 000560 1786961947.326784 mapread 0x008855 thru 0x009233 (0x9df bytes) 000561 1786961947.338534 trunc from 0x01696e to 0x0229ce (0xc061 bytes) 000563 1786961947.346407 mapread 0x00bc6a thru 0x00c823 (0xbba bytes) 000564 1786961947.359672 fallocate 0x022398 thru 0x02f1c2 (0xce2b bytes) 000565 1786961947.366170 read 0x01f6c1 thru 0x0269d9 (0x7319 bytes) 000566 1786961947.382969 punch from 0x03279e to 0x0365d6 (0x3e39 bytes) 000567 1786961947.388684 trunc from 0x01e63e to 0x01f08f (0xa52 bytes) 000568 1786961947.395946 fallocate 0x01b5d5 thru 0x01c25f (0xc8b bytes) 000569 1786961947.404174 punch from 0x01a80c to 0x02187c (0x7071 bytes) 000570 1786961947.423695 trunc from 0x02a8d8 to 0x02e3a6 (0x3acf bytes) 000571 1786961947.451170 MIRROR_EXTEND to 2 mirrors 000572 1786961948.749031 mapread 0x017d18 thru 0x01fe1c (0x8105 bytes) 000573 1786961948.811203 MIRROR_RESYNC 2 mirrors 000574 1786961949.407046 mapread 0x00a3a7 thru 0x0168d6 (0xc530 bytes) 000575 1786961949.484643 write 0x013b10 thru 0x017176 (0x3667 bytes) 000576 1786961949.550135 fallocate 0x030269 thru 0x03a05b (0x9df3 bytes) 000577 1786961949.562250 mapread 0x009164 thru 0x0161d9 (0xd076 bytes) 000580 1786961949.603363 fallocate 0x0111d3 thru 0x018c9f (0x7acd bytes) 000581 1786961949.618787 mapread 0x011799 thru 0x019b6d (0x83d5 bytes) 000582 1786961949.620092 write 0x028a10 thru 0x030060 (0x7651 bytes) 000583 1786961949.630755 trunc from 0x00c479 to 0x01bb9c (0xf724 bytes) 000584 1786961949.644513 trunc from 0x034d0f to 0x0365d6 (0x18c8 bytes) 000585 1786961949.650789 fallocate 0x035093 thru 0x039d68 (0x4cd6 bytes) 000587 1786961949.657056 mapread 0x00694b thru 0x013250 (0xc906 bytes) 000588 1786961949.667799 trunc from 0x0365d7 to 0x039510 (0x2f3a bytes) 000589 1786961949.684839 fallocate 0x0382b4 thru 0x03ffff (0x7d4c bytes) 000590 1786961949.690349 write 0x02ee83 thru 0x0352b4 (0x6432 bytes) 000592 1786961949.698349 fallocate 0x01b74e thru 0x026635 (0xaee8 bytes) 000593 1786961949.710567 MIRROR_RESYNC 2 mirrors 000594 1786961950.748467 trunc from 0x035750 to 0x03ffff (0xa8b0 bytes) 000595 1786961950.810255 mapwrite 0x03d367 thru 0x03ffff (0x2c99 bytes) 000596 1786961950.893217 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb17852dc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7ec48ac0 Mirror 3: 0x7ec48ac0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000597 1786961952.106458 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000598 1786961952.627486 trunc from 0x03be64 to 0x03ffff (0x419c bytes) 000600 1786961952.670777 write 0x023976 thru 0x024234 (0x8bf bytes) 000601 1786961952.687510 write 0x030d6c thru 0x031c8d (0xf22 bytes) 000602 1786961952.701698 mapread 0x00ea92 thru 0x01188f (0x2dfe bytes) 000604 1786961952.735096 trunc from 0x027512 to 0x02995e (0x244d bytes) 000605 1786961952.754284 read 0x0264c5 thru 0x028c6d (0x27a9 bytes) 000606 1786961952.767690 mapread 0x036970 thru 0x03ffff (0x9690 bytes) 000607 1786961952.799116 mapread 0x013024 thru 0x01d9ce (0xa9ab bytes) 000608 1786961952.815437 trunc from 0x018d58 to 0x01eb2b (0x5dd4 bytes) 000609 1786961952.824003 fallocate 0x03a2ff thru 0x03d68a (0x338c bytes) 000610 1786961952.830603 write 0x01901f thru 0x01c2bd (0x329f bytes) 000611 1786961952.849698 mapread 0x031b6e thru 0x036d70 (0x5203 bytes) 000612 1786961952.851800 punch from 0x031ed9 to 0x034fdc (0x3104 bytes) 000613 1786961952.878530 fallocate 0x033a71 thru 0x035aaf (0x203f bytes) 000614 1786961952.886306 punch from 0x005a75 to 0x0098f4 (0x3e80 bytes) 000615 1786961952.892536 read 0x014685 thru 0x020972 (0xc2ee bytes) 000616 1786961952.929331 mapwrite 0x02ed60 thru 0x03d463 (0xe704 bytes) 000617 1786961952.997754 trunc from 0x040000 to 0x030543 (0xfabc bytes) 000618 1786961953.023518 write 0x02aa2d thru 0x03542b (0xa9ff bytes) 000619 1786961953.025444 trunc from 0x03542c to 0x039d7f (0x4954 bytes) 000620 1786961953.045072 punch from 0x027930 to 0x028bd8 (0x12a9 bytes) 000621 1786961953.051292 mapread 0x01b554 thru 0x024ef8 (0x99a5 bytes) 000622 1786961953.057116 write 0x03eca4 thru 0x03ffff (0x135c bytes) 000623 1786961953.059279 write 0x03322a thru 0x03b691 (0x8468 bytes) 000624 1786961953.072664 trunc from 0x040000 to 0x028c91 (0x1736e bytes) 000625 1786961953.097452 mapread 0x00df70 thru 0x01a530 (0xc5c1 bytes) 000626 1786961953.104696 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000627 1786961953.314021 mapwrite 0x00e87a thru 0x01cc4b (0xe3d2 bytes) 000628 1786961953.493578 mapwrite 0x03232b thru 0x032690 (0x366 bytes) 000629 1786961953.593555 trunc from 0x028f5b to 0x032690 (0x9736 bytes) 000630 1786961953.609934 fallocate 0x00a4f6 thru 0x010e8f (0x699a bytes) 000631 1786961953.623095 trunc from 0x032691 to 0x014e54 (0x1d83c bytes) 000632 1786961953.660561 mapwrite 0x01cf20 thru 0x01e2da (0x13bb bytes) 000633 1786961953.715684 write 0x02cecb thru 0x03846b (0xb5a1 bytes) 000634 1786961953.717688 trunc from 0x03846c to 0x00d6cf (0x2ad9c bytes) 000635 1786961953.745232 trunc from 0x00d6d0 to 0x02068e (0x12fbf bytes) 000636 1786961953.789109 punch from 0x0007a1 to 0x0025a8 (0x1e08 bytes) 000637 1786961953.796085 fallocate 0x021f66 thru 0x031a98 (0xfb33 bytes) 000638 1786961953.802056 write 0x010e57 thru 0x016491 (0x563b bytes) 000639 1786961953.812813 write 0x010adf thru 0x01d5b5 (0xcad7 bytes) 000640 1786961953.818843 write 0x01c9ad thru 0x028c2d (0xc281 bytes) 000641 1786961953.821577 fallocate 0x01f0cb thru 0x02874b (0x9681 bytes) 000642 1786961953.828523 trunc from 0x01b46d to 0x01e5ad (0x3141 bytes) 000643 1786961953.863598 fallocate 0x02257e thru 0x02d2f0 (0xad73 bytes) 000644 1786961953.869575 mapwrite 0x036912 thru 0x03e6e9 (0x7dd8 bytes) 000645 1786961953.922530 fallocate 0x01c2db thru 0x025ba9 (0x98cf bytes) 000646 1786961953.928409 trunc from 0x03e6ea to 0x00902c (0x356bd bytes) 000647 1786961953.944983 mapread 0x00363b thru 0x008f53 (0x5919 bytes) 000649 1786961953.959629 trunc from 0x00902d to 0x0151cb (0xc19f bytes) 000650 1786961953.985803 punch from 0x000e76 to 0x0021b5 (0x1340 bytes) 000651 1786961953.992560 read 0x0124e2 thru 0x0151cb (0x2cea bytes) 000652 1786961953.998523 trunc from 0x003151 to 0x012eed (0xfd9d bytes) 000653 1786961954.006964 punch from 0x0061b2 to 0x013d40 (0xdb8f bytes) 000654 1786961954.013775 trunc from 0x0151cc to 0x00ce44 (0x8387 bytes) 000655 1786961954.037540 read 0x0004a6 thru 0x000a33 (0x58e bytes) 000657 1786961954.049792 trunc from 0x0035f1 to 0x0056ed (0x20fd bytes) 000658 1786961954.055544 trunc from 0x00ce45 to 0x02dbaa (0x20d66 bytes) 000659 1786961954.071160 trunc from 0x0131bd to 0x01b0d3 (0x7f17 bytes) 000660 1786961954.082115 MIRROR_EXTEND to 2 mirrors 000662 1786961954.796770 punch from 0x01fb8b to 0x02756c (0x79e2 bytes) 000663 1786961954.844977 mapread 0x018050 thru 0x019eb4 (0x1e65 bytes) 000664 1786961954.858207 write 0x0392be thru 0x03f092 (0x5dd5 bytes) 000665 1786961954.864766 fallocate 0x02e29d thru 0x0300bc (0x1e20 bytes) 000666 1786961954.870055 punch from 0x03722f to 0x03a98b (0x375d bytes) 000667 1786961954.888113 trunc from 0x00968b to 0x017b73 (0xe4e9 bytes) 000668 1786961954.893216 fallocate 0x00530d thru 0x00a572 (0x5266 bytes) 000669 1786961954.904977 mapread 0x02917c thru 0x031d36 (0x8bbb bytes) 000670 1786961954.937314 trunc from 0x03f093 to 0x001aff (0x3d593 bytes) 000673 1786961955.053625 read 0x00011a thru 0x001aff (0x19e6 bytes) 000674 1786961955.074497 read 0x000e32 thru 0x001aff (0xcce bytes) 000675 1786961955.075307 mapread 0x000e21 thru 0x001aff (0xcdf bytes) 000676 1786961955.084562 read 0x000149 thru 0x00184b (0x1703 bytes) 000677 1786961955.102436 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1b00): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b00): Mirror 2: 0x74ce8394 (stale) CRC-32 checksum value for chunk [0, 0x1b00): Mirror 1: 0x74ce8394 Mirror 3: 0x74ce8394 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b00: skipped 000678 1786961956.001572 fallocate 0x018189 thru 0x027769 (0xf5e1 bytes) 000679 1786961956.083519 punch from 0x00cc08 to 0x00ec37 (0x2030 bytes) 000680 1786961956.108189 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2776a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2776a): Mirror 2: 0xa9c27dd7 (stale) Mirror 3: 0xa9c27dd7 (stale) CRC-32 checksum value for chunk [0, 0x2776a): Mirror 1: 0xa9c27dd7 Mirror 4: 0xa9c27dd7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2776a: skipped 000681 1786961957.484345 mapwrite 0x03cf7e thru 0x03e0f7 (0x117a bytes) 000682 1786961957.591673 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e0f8): Mirror 2: 0xc20d0581 (stale) Mirror 3: 0xc20d0581 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e0f8): Mirror 2: 0xc20d0581 (stale) Mirror 3: 0xc20d0581 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e0f8: skipped 000683 1786961957.955558 mapwrite 0x01de6e thru 0x020b0a (0x2c9d bytes) 000684 1786961958.005534 trunc from 0x03e0f8 to 0x00463b (0x39abc bytes) 000685 1786961958.033298 read 0x0008a8 thru 0x00463b (0x3d94 bytes) 000686 1786961958.047285 trunc from 0x00463c to 0x00b62d (0x6ff2 bytes) 000687 1786961958.060701 read 0x005b61 thru 0x00b62d (0x5acd bytes) 000688 1786961958.071757 read 0x0054be thru 0x00b62d (0x6170 bytes) 000689 1786961958.072685 fallocate 0x021143 thru 0x0232f3 (0x21b1 bytes) 000690 1786961958.078431 punch from 0x00df64 to 0x01ccdb (0xed78 bytes) 000692 1786961958.085651 fallocate 0x004e53 thru 0x0071f1 (0x239f bytes) 000694 1786961958.099434 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x232f4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x232f4): Mirror 2: 0xf6962178 (stale) Mirror 3: 0xf6962178 (stale) CRC-32 checksum value for chunk [0, 0x232f4): Mirror 1: 0xf6962178 Mirror 4: 0xf6962178 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x232f4: skipped 000696 1786961959.002360 punch from 0x007c00 to 0x00c638 (0x4a39 bytes) 000697 1786961959.081450 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x232f4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x232f4): Mirror 2: 0xf6962178 (stale) Mirror 3: 0xf6962178 (stale) Mirror 4: 0xf6962178 (stale) CRC-32 checksum value for chunk [0, 0x232f4): Mirror 1: 0xf6962178 Mirror 5: 0xf6962178 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x232f4: skipped 000698 1786961960.116296 punch from 0x0075f4 to 0x01269d (0xb0aa bytes) 000699 1786961960.157827 read 0x009cb9 thru 0x017ff3 (0xe33b bytes) 000700 1786961960.171419 trunc from 0x01e6c1 to 0x022c29 (0x4569 bytes) 000701 1786961960.181095 mapwrite 0x03c7e2 thru 0x03ffff (0x381e bytes) 000703 1786961960.259523 fallocate 0x03946d thru 0x03ffff (0x6b93 bytes) 000704 1786961960.270644 write 0x029c2d thru 0x02a465 (0x839 bytes) 000705 1786961960.287574 trunc from 0x040000 to 0x003bb2 (0x3c44d bytes) 000706 1786961960.308479 punch from 0x001cb2 to 0x003bb2 (0x1f01 bytes) 000707 1786961960.316454 mapwrite 0x01122b thru 0x01de7e (0xcc54 bytes) 000708 1786961960.406296 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1de7f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1de7f): Mirror 2: 0x3ad07484 (stale) Mirror 3: 0x3ad07484 (stale) Mirror 4: 0x3ad07484 (stale) Mirror 5: 0x3ad07484 (stale) CRC-32 checksum value for chunk [0, 0x1de7f): Mirror 1: 0xd080f521 Mirror 6: 0xd080f521 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1de7f: skipped 000709 1786961961.557961 mapread 0x005782 thru 0x00bdf1 (0x6670 bytes) 000710 1786961961.629835 mapwrite 0x01ef0c thru 0x0298da (0xa9cf bytes) 000711 1786961961.759347 read 0x02789d thru 0x0298da (0x203e bytes) 000712 1786961961.760465 mapwrite 0x03ff6e thru 0x03ffff (0x92 bytes) 000713 1786961961.816980 trunc from 0x040000 to 0x01dd1c (0x222e3 bytes) 000714 1786961961.843829 fallocate 0x002673 thru 0x00f16d (0xcafb bytes) 000715 1786961961.852453 mapwrite 0x03e890 thru 0x03ffff (0x1770 bytes) 000716 1786961961.936741 trunc from 0x040000 to 0x02e518 (0x11ae7 bytes) 000717 1786961961.980597 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e519): Mirror 2: 0x46482328 (stale) Mirror 4: 0x46482328 (stale) Mirror 5: 0x46482328 (stale) Mirror 6: 0x2951cf1d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e519): Mirror 2: 0x46482328 (stale) Mirror 4: 0x46482328 (stale) Mirror 5: 0x46482328 (stale) Mirror 6: 0x2951cf1d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e519: skipped 000718 1786961962.530142 trunc from 0x02e519 to 0x00113f (0x2d3d9 bytes) 000719 1786961962.553508 fallocate 0x00e009 thru 0x0172fa (0x92f2 bytes) 000720 1786961962.560267 punch from 0x000b39 to 0x00113f (0x607 bytes) 000721 1786961962.566485 trunc from 0x000ed4 to 0x00113f (0x26c bytes) 000722 1786961962.582320 punch from 0x000199 to 0x00113f (0xfa7 bytes) 000724 1786961962.587733 trunc from 0x000669 to 0x00113f (0xad7 bytes) 000725 1786961962.592351 fallocate 0x000993 thru 0x002b39 (0x21a7 bytes) 000726 1786961962.600130 fallocate 0x03ca1f thru 0x03ffff (0x35e1 bytes) 000727 1786961962.607232 write 0x0087b9 thru 0x008d6b (0x5b3 bytes) 000728 1786961962.612382 trunc from 0x040000 to 0x026584 (0x19a7b bytes) 000729 1786961962.624584 trunc from 0x026585 to 0x03979b (0x13217 bytes) 000730 1786961962.653593 read 0x02e19f thru 0x036530 (0x8392 bytes) 000731 1786961962.664088 mapwrite 0x01a67d thru 0x024a4a (0xa3ce bytes) 000732 1786961962.720980 mapread 0x02b697 thru 0x03979b (0xe105 bytes) 000733 1786961962.747242 trunc from 0x03979c to 0x021e1c (0x1797f bytes) 000735 1786961962.764472 fallocate 0x03605a thru 0x03ffff (0x9fa6 bytes) 000736 1786961962.771333 fallocate 0x027a57 thru 0x02f08b (0x7635 bytes) 000737 1786961962.776404 mapread 0x002785 thru 0x0050e8 (0x2964 bytes) 000739 1786961962.787353 read 0x002224 thru 0x005c6d (0x3a4a bytes) 000741 1786961962.794748 mapwrite 0x021a23 thru 0x0317c3 (0xfda1 bytes) 000742 1786961962.842913 write 0x02a631 thru 0x030176 (0x5b46 bytes) 000743 1786961962.844262 trunc from 0x040000 to 0x0349a7 (0xb658 bytes) 000744 1786961962.876906 mapread 0x02d6d7 thru 0x0349a7 (0x72d1 bytes) 000745 1786961962.878368 fallocate 0x010b19 thru 0x018028 (0x7510 bytes) 000747 1786961962.886070 fallocate 0x03d879 thru 0x03ec58 (0x13e0 bytes) 000748 1786961962.913816 MIRROR_RESYNC 5 mirrors 000750 1786961964.996379 write 0x014bca thru 0x014efb (0x332 bytes) 000751 1786961965.078489 trunc from 0x03ec59 to 0x01f460 (0x1f7f8 bytes) 000752 1786961965.153028 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f461): Mirror 4: 0xfcdad08a (stale) Mirror 5: 0xfcdad08a (stale) Mirror 6: 0xfcdad08a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f461): Mirror 4: 0xfcdad08a (stale) Mirror 5: 0xfcdad08a (stale) Mirror 6: 0xfcdad08a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f461: skipped 000753 1786961965.509660 write 0x03e505 thru 0x03ffff (0x1afb bytes) 000755 1786961965.523143 mapread 0x03cd0f thru 0x03ffff (0x32f1 bytes) 000756 1786961965.537774 trunc from 0x040000 to 0x03f814 (0x7eb bytes) 000757 1786961965.561348 read 0x0391e4 thru 0x03a27f (0x109c bytes) 000758 1786961965.571069 fallocate 0x00b7c3 thru 0x00d525 (0x1d63 bytes) 000759 1786961965.581617 write 0x037cec thru 0x03bf9c (0x42b1 bytes) 000760 1786961965.602706 read 0x01bbd0 thru 0x020322 (0x4753 bytes) 000761 1786961965.632698 fallocate 0x034d04 thru 0x03af1a (0x6217 bytes) 000762 1786961965.641442 read 0x005a2e thru 0x006094 (0x667 bytes) 000763 1786961965.642939 trunc from 0x02a521 to 0x032d96 (0x8876 bytes) 000764 1786961965.650277 trunc from 0x03f815 to 0x011a8a (0x2dd8a bytes) 000765 1786961965.680742 punch from 0x002910 to 0x00e642 (0xbd33 bytes) 000766 1786961965.686508 punch from 0x00f979 to 0x011a8a (0x2112 bytes) 000767 1786961965.693597 punch from 0x009e3f to 0x011a8a (0x7c4c bytes) 000768 1786961965.699682 trunc from 0x00fee6 to 0x011a8a (0x1ba5 bytes) 000769 1786961965.704095 punch from 0x0065f5 to 0x011a8a (0xb496 bytes) 000770 1786961965.712056 trunc from 0x011a8b to 0x0033d7 (0xe6b3 bytes) 000773 1786961965.750665 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33d8): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d8): Mirror 4: 0x692c8e4d (stale) Mirror 5: 0x692c8e4d (stale) Mirror 6: 0x692c8e4d (stale) CRC-32 checksum value for chunk [0, 0x33d8): Mirror 1: 0x692c8e4d Mirror 7: 0x692c8e4d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d8: skipped 000774 1786961966.957661 fallocate 0x02fc41 thru 0x03c0e6 (0xc4a6 bytes) 000775 1786961967.021408 mapread 0x0024e6 thru 0x0033d7 (0xef2 bytes) 000776 1786961967.040350 read 0x0023cf thru 0x0033d7 (0x1009 bytes) 000777 1786961967.041179 mapread 0x00112d thru 0x0033d7 (0x22ab bytes) 000778 1786961967.071365 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33d8): Mirror 4: 0x692c8e4d (stale) Mirror 5: 0x692c8e4d (stale) Mirror 6: 0x692c8e4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d8): Mirror 4: 0x692c8e4d (stale) Mirror 5: 0x692c8e4d (stale) Mirror 6: 0x692c8e4d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d8: skipped 000779 1786961967.535518 fallocate 0x009f13 thru 0x016384 (0xc472 bytes) 000780 1786961967.549599 write 0x0090ea thru 0x010697 (0x75ae bytes) 000781 1786961967.566186 write 0x03c2e1 thru 0x03ffa8 (0x3cc8 bytes) 000783 1786961967.569114 mapread 0x022152 thru 0x030ccb (0xeb7a bytes) 000784 1786961967.606044 mapwrite 0x0341b1 thru 0x03ffff (0xbe4f bytes) 000785 1786961967.673656 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6f732559 (stale) Mirror 5: 0x6f732559 (stale) Mirror 6: 0x6f732559 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdb47947b Mirror 7: 0xdb47947b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000786 1786961968.807717 write 0x020a62 thru 0x023610 (0x2baf bytes) 000788 1786961968.889724 MIRROR_RESYNC 5 mirrors 000789 1786961970.792326 write 0x02617b thru 0x02c3f7 (0x627d bytes) 000791 1786961970.841282 trunc from 0x040000 to 0x002837 (0x3d7c8 bytes) 000792 1786961970.870833 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2838): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2838): Mirror 4: 0xd8743e43 (stale) Mirror 5: 0xd8743e43 (stale) Mirror 6: 0xd8743e43 (stale) Mirror 7: 0xd8743e43 (stale) CRC-32 checksum value for chunk [0, 0x2838): Mirror 1: 0xd8743e43 Mirror 8: 0xd8743e43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2838: skipped 000793 1786961971.709039 punch from 0x000323 to 0x002837 (0x2515 bytes) 000794 1786961971.793752 trunc from 0x00065d to 0x002837 (0x21db bytes) 000795 1786961971.818923 trunc from 0x002838 to 0x01721d (0x149e6 bytes) 000796 1786961971.858756 write 0x034ed2 thru 0x03afe1 (0x6110 bytes) 000797 1786961971.860190 punch from 0x007ddc to 0x00e209 (0x642e bytes) 000798 1786961971.883431 mapread 0x01d21e thru 0x022d08 (0x5aeb bytes) 000799 1786961971.901424 mapread 0x01f0f0 thru 0x02d74f (0xe660 bytes) 000800 1786961971.909931 trunc from 0x0060e1 to 0x015945 (0xf865 bytes) 000801 1786961971.921926 trunc from 0x014b2c to 0x0165da (0x1aaf bytes) 000802 1786961971.931684 punch from 0x026339 to 0x026723 (0x3eb bytes) 000804 1786961971.943604 fallocate 0x03bb53 thru 0x03ffff (0x44ad bytes) 000805 1786961971.961622 write 0x02ef85 thru 0x033d66 (0x4de2 bytes) 000806 1786961971.977009 read 0x000651 thru 0x00f057 (0xea07 bytes) 000807 1786961972.012930 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x144d99e6 (stale) Mirror 6: 0x144d99e6 (stale) Mirror 7: 0x144d99e6 (stale) Mirror 8: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x144d99e6 (stale) Mirror 6: 0x144d99e6 (stale) Mirror 7: 0x144d99e6 (stale) Mirror 8: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000808 1786961972.653985 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x144d99e6 (stale) Mirror 7: 0x144d99e6 (stale) Mirror 8: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x144d99e6 (stale) Mirror 7: 0x144d99e6 (stale) Mirror 8: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000810 1786961973.070531 read 0x01397c thru 0x01a9d4 (0x7059 bytes) 000811 1786961973.082071 read 0x021830 thru 0x027cd6 (0x64a7 bytes) 000812 1786961973.096900 trunc from 0x0399d2 to 0x03ffff (0x662e bytes) 000814 1786961973.105617 trunc from 0x03efb0 to 0x03ffff (0x1050 bytes) 000815 1786961973.110846 mapwrite 0x024c45 thru 0x02f51d (0xa8d9 bytes) 000816 1786961973.135449 punch from 0x03c85d to 0x03ffff (0x37a3 bytes) 000817 1786961973.140980 trunc from 0x040000 to 0x006933 (0x396cc bytes) 000818 1786961973.165216 fallocate 0x004cf0 thru 0x00dc02 (0x8f13 bytes) 000819 1786961973.172555 mapwrite 0x00ba63 thru 0x01b222 (0xf7c0 bytes) 000820 1786961973.310395 read 0x00a33c thru 0x012b18 (0x87dd bytes) 000821 1786961973.318692 write 0x019a32 thru 0x01e7a1 (0x4d70 bytes) 000822 1786961973.322650 fallocate 0x02ccba thru 0x0338b2 (0x6bf9 bytes) 000824 1786961973.331626 write 0x017a96 thru 0x019a1c (0x1f87 bytes) 000825 1786961973.333119 read 0x000940 thru 0x00c577 (0xbc38 bytes) 000826 1786961973.334445 write 0x01dbaa thru 0x0246ff (0x6b56 bytes) 000827 1786961973.341620 trunc from 0x0338b3 to 0x013c71 (0x1fc41 bytes) 000828 1786961973.361440 punch from 0x001fe9 to 0x005255 (0x326d bytes) 000829 1786961973.367003 trunc from 0x013c72 to 0x03d7fa (0x29b89 bytes) 000830 1786961973.392760 mapwrite 0x02d0ba thru 0x032bf0 (0x5b37 bytes) 000831 1786961973.450843 trunc from 0x03d7fb to 0x0149b9 (0x28e41 bytes) 000832 1786961973.477338 trunc from 0x0149ba to 0x007d85 (0xcc34 bytes) 000833 1786961973.499439 write 0x018f3a thru 0x01c649 (0x3710 bytes) 000834 1786961973.505316 mapwrite 0x01c619 thru 0x025a0d (0x93f5 bytes) 000835 1786961973.708747 trunc from 0x00e3c1 to 0x019e02 (0xba42 bytes) 000837 1786961973.716963 write 0x02dba6 thru 0x031615 (0x3a70 bytes) 000838 1786961973.720770 read 0x02c397 thru 0x031615 (0x527f bytes) 000839 1786961973.752771 trunc from 0x028fbe to 0x02d601 (0x4644 bytes) 000840 1786961973.782063 punch from 0x02a8f7 to 0x031615 (0x6d1f bytes) 000841 1786961973.796294 read 0x00f65e thru 0x01e373 (0xed16 bytes) 000842 1786961973.798221 fallocate 0x033c06 thru 0x03cfff (0x93fa bytes) 000843 1786961973.803908 mapread 0x025425 thru 0x02966f (0x424b bytes) 000846 1786961973.836378 write 0x035666 thru 0x039b71 (0x450c bytes) 000847 1786961973.838225 fallocate 0x02a9ca thru 0x02bcad (0x12e4 bytes) 000848 1786961973.846162 fallocate 0x00b861 thru 0x00f17c (0x391c bytes) 000849 1786961973.857182 read 0x029e29 thru 0x02ee49 (0x5021 bytes) 000850 1786961973.857767 read 0x0040c6 thru 0x00f8b4 (0xb7ef bytes) 000851 1786961973.858394 write 0x008193 thru 0x012ce7 (0xab55 bytes) 000852 1786961973.863407 mapwrite 0x0148f8 thru 0x01cbd3 (0x82dc bytes) 000853 1786961973.911643 fallocate 0x001e66 thru 0x00f8a0 (0xda3b bytes) 000854 1786961973.920412 fallocate 0x03f43c thru 0x03ffff (0xbc4 bytes) 000855 1786961973.932590 mapwrite 0x034a94 thru 0x03cf71 (0x84de bytes) 000856 1786961973.989772 write 0x019310 thru 0x025dba (0xcaab bytes) 000857 1786961973.995003 trunc from 0x040000 to 0x032bd3 (0xd42c bytes) 000858 1786961974.018338 mapwrite 0x036b7b thru 0x03f039 (0x84bf bytes) 000860 1786961974.067646 fallocate 0x02b437 thru 0x036ff4 (0xbbbe bytes) 000861 1786961974.074527 read 0x0352c3 thru 0x03c8e2 (0x7620 bytes) 000862 1786961974.078982 mapread 0x02c4d6 thru 0x03b332 (0xee5d bytes) 000863 1786961974.081562 read 0x020063 thru 0x02b48e (0xb42c bytes) 000864 1786961974.082961 read 0x01d5c5 thru 0x028bbc (0xb5f8 bytes) 000865 1786961974.084343 trunc from 0x03f03a to 0x0104b1 (0x2eb88 bytes) 000867 1786961974.106665 punch from 0x003e7c to 0x005ebe (0x2043 bytes) 000868 1786961974.112740 mapread 0x00380c thru 0x0104b1 (0xcca6 bytes) 000869 1786961974.119999 write 0x00db8f thru 0x010b31 (0x2fa3 bytes) 000871 1786961974.121725 punch from 0x00bc6d to 0x010b31 (0x4ec5 bytes) 000872 1786961974.133290 trunc from 0x00613f to 0x010b31 (0xa9f3 bytes) 000873 1786961974.139132 mapwrite 0x00e7de thru 0x014b77 (0x639a bytes) 000874 1786961974.196636 trunc from 0x00f914 to 0x014b77 (0x5264 bytes) 000875 1786961974.208640 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x14b78): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14b78): Mirror 6: 0x258462e5 (stale) Mirror 7: 0x258462e5 (stale) Mirror 8: 0x9010587e (stale) CRC-32 checksum value for chunk [0, 0x14b78): Mirror 1: 0xf7b1c2f7 Mirror 9: 0xf7b1c2f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14b78: skipped 000876 1786961975.146713 read 0x004c76 thru 0x00a260 (0x55eb bytes) 000877 1786961975.201401 mapwrite 0x02e2a6 thru 0x03c7ed (0xe548 bytes) 000879 1786961975.401051 trunc from 0x010616 to 0x016815 (0x6200 bytes) 000880 1786961975.410180 fallocate 0x0375c1 thru 0x039a84 (0x24c4 bytes) 000881 1786961975.429316 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c7ee): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c7ee): Mirror 6: 0x29031397 (stale) Mirror 7: 0x29031397 (stale) Mirror 8: 0x1997edee (stale) Mirror 9: 0x710d5220 (stale) CRC-32 checksum value for chunk [0, 0x3c7ee): Mirror 1: 0xaa19e2f1 Mirror 10: 0xaa19e2f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c7ee: skipped 000883 1786961977.057576 trunc from 0x0101d8 to 0x01deb4 (0xdcdd bytes) 000884 1786961977.125150 punch from 0x03047e to 0x03324c (0x2dcf bytes) 000885 1786961977.132790 write 0x0113df thru 0x01f41f (0xe041 bytes) 000886 1786961977.153263 mapwrite 0x038cca thru 0x039990 (0xcc7 bytes) 000887 1786961977.191830 punch from 0x01e1e1 to 0x026eb9 (0x8cd9 bytes) 000888 1786961977.209145 fallocate 0x014e7b thru 0x01df71 (0x90f7 bytes) 000889 1786961977.223801 trunc from 0x02170d to 0x02acf0 (0x95e4 bytes) 000890 1786961977.232477 read 0x0341f7 thru 0x03bade (0x78e8 bytes) 000891 1786961977.242663 punch from 0x004acf to 0x00e73b (0x9c6d bytes) 000892 1786961977.249641 punch from 0x0164bc to 0x01fc7e (0x97c3 bytes) 000893 1786961977.257222 fallocate 0x01e00a thru 0x020bb2 (0x2ba9 bytes) 000894 1786961977.269070 fallocate 0x00c178 thru 0x0158cd (0x9756 bytes) 000895 1786961977.278148 mapwrite 0x014ee3 thru 0x023ef0 (0xf00e bytes) 000896 1786961977.370097 write 0x029b95 thru 0x02c0ff (0x256b bytes) 000897 1786961977.393761 read 0x0223bb thru 0x027c46 (0x588c bytes) 000898 1786961977.396086 mapread 0x008c68 thru 0x00dc9b (0x5034 bytes) 000899 1786961977.443805 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c7ee): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c7ee): Mirror 6: 0x29031397 (stale) Mirror 7: 0x29031397 (stale) Mirror 8: 0x1997edee (stale) Mirror 9: 0x710d5220 (stale) Mirror 10: 0xaa19e2f1 (stale) CRC-32 checksum value for chunk [0, 0x3c7ee): Mirror 1: 0x134ba5d Mirror 11: 0x134ba5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c7ee: skipped 000900 1786961979.013897 mapread 0x020ebc thru 0x027efe (0x7043 bytes) 000901 1786961979.071511 punch from 0x02c4bc to 0x03a6c6 (0xe20b bytes) 000902 1786961979.138748 trunc from 0x0211c2 to 0x02e12c (0xcf6b bytes) 000903 1786961979.156563 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c7ee): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c7ee): Mirror 6: 0x29031397 (stale) Mirror 7: 0x29031397 (stale) Mirror 8: 0x1997edee (stale) Mirror 9: 0x710d5220 (stale) Mirror 10: 0xaa19e2f1 (stale) Mirror 11: 0x134ba5d (stale) CRC-32 checksum value for chunk [0, 0x3c7ee): Mirror 1: 0xaf98a907 Mirror 12: 0xaf98a907 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c7ee: skipped 000905 1786961980.516484 fallocate 0x0112fb thru 0x01aed8 (0x9bde bytes) 000906 1786961980.556116 fallocate 0x02dd3f thru 0x02fb08 (0x1dca bytes) 000907 1786961980.562523 mapwrite 0x0103b6 thru 0x01387a (0x34c5 bytes) 000908 1786961980.596073 read 0x01b56f thru 0x021371 (0x5e03 bytes) 000909 1786961980.605355 mapread 0x00df24 thru 0x010216 (0x22f3 bytes) 000910 1786961980.607085 trunc from 0x03c7ee to 0x033b31 (0x8cbc bytes) 000911 1786961980.634523 punch from 0x021f2f to 0x029be9 (0x7cbb bytes) 000912 1786961980.645835 MIRROR_RESYNC 8 mirrors 000913 1786961982.910516 trunc from 0x00a388 to 0x012b95 (0x880e bytes) 000915 1786961982.975718 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33b32): Mirror 6: 0xff306050 (stale) Mirror 7: 0xff306050 (stale) Mirror 8: 0xff306050 (stale) Mirror 9: 0xff306050 (stale) Mirror 10: 0xff306050 (stale) Mirror 12: 0xff306050 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33b32): Mirror 6: 0xff306050 (stale) Mirror 7: 0xff306050 (stale) Mirror 8: 0xff306050 (stale) Mirror 9: 0xff306050 (stale) Mirror 10: 0xff306050 (stale) Mirror 12: 0xff306050 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33b32: skipped 000916 1786961983.538884 punch from 0x02efbe to 0x033b31 (0x4b74 bytes) 000917 1786961983.543792 trunc from 0x033b32 to 0x02eea4 (0x4c8d bytes) 000918 1786961983.568760 mapwrite 0x037dc3 thru 0x03ffff (0x823d bytes) 000919 1786961983.623330 trunc from 0x0287d5 to 0x02b55b (0x2d87 bytes) 000920 1786961983.628564 trunc from 0x038484 to 0x03db63 (0x56e0 bytes) 000921 1786961983.635282 write 0x02e047 thru 0x02ff3c (0x1ef6 bytes) 000922 1786961983.645280 fallocate 0x0384d9 thru 0x0384f9 (0x21 bytes) 000923 1786961983.654805 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xcd18d35 (stale) Mirror 7: 0xcd18d35 (stale) Mirror 9: 0xcd18d35 (stale) Mirror 10: 0xcd18d35 (stale) Mirror 12: 0xcd18d35 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xcd18d35 (stale) Mirror 7: 0xcd18d35 (stale) Mirror 9: 0xcd18d35 (stale) Mirror 10: 0xcd18d35 (stale) Mirror 12: 0xcd18d35 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000924 1786961984.236473 mapread 0x0297bb thru 0x0319a0 (0x81e6 bytes) 000925 1786961984.258909 mapwrite 0x020345 thru 0x02b161 (0xae1d bytes) 000926 1786961984.288513 mapwrite 0x02d0b9 thru 0x03863f (0xb587 bytes) 000927 1786961984.323763 trunc from 0x040000 to 0x016c50 (0x293af bytes) 000928 1786961984.345284 write 0x017be2 thru 0x026fad (0xf3cc bytes) 000929 1786961984.354402 MIRROR_RESYNC 6 mirrors 000930 1786961985.652274 write 0x01ebe9 thru 0x02e918 (0xfd30 bytes) 000931 1786961985.693307 mapread 0x00f822 thru 0x01249a (0x2c79 bytes) 000933 1786961985.721801 mapwrite 0x0182a7 thru 0x020f21 (0x8c7b bytes) 000934 1786961985.781302 fallocate 0x03d2f9 thru 0x03ffff (0x2d07 bytes) 000935 1786961985.795535 punch from 0x023c00 to 0x026a3b (0x2e3c bytes) 000936 1786961985.815129 mapwrite 0x017bc0 thru 0x0212e2 (0x9723 bytes) 000937 1786961985.875096 read 0x005889 thru 0x00db34 (0x82ac bytes) 000938 1786961985.911341 write 0x03ce27 thru 0x03ffff (0x31d9 bytes) 000939 1786961985.919994 mapwrite 0x0329fb thru 0x03613a (0x3740 bytes) 000940 1786961985.963098 read 0x02603f thru 0x0326e5 (0xc6a7 bytes) 000941 1786961985.972816 read 0x00f24d thru 0x01a91a (0xb6ce bytes) 000942 1786961985.973569 fallocate 0x003937 thru 0x013428 (0xfaf2 bytes) 000943 1786961985.980140 trunc from 0x040000 to 0x00356a (0x3ca95 bytes) 000944 1786961986.000447 mapwrite 0x01ae5a thru 0x023e31 (0x8fd8 bytes) 000945 1786961986.063379 mapread 0x01966b thru 0x023e31 (0xa7c7 bytes) 000946 1786961986.070446 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x23e32): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23e32): Mirror 6: 0x41c5904b (stale) Mirror 7: 0x41c5904b (stale) Mirror 9: 0x41c5904b (stale) Mirror 10: 0x41c5904b (stale) Mirror 12: 0x41c5904b (stale) CRC-32 checksum value for chunk [0, 0x23e32): Mirror 1: 0xdaa1f942 Mirror 13: 0xdaa1f942 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23e32: skipped 000947 1786961987.182059 fallocate 0x0225d9 thru 0x028bd0 (0x65f8 bytes) 000948 1786961987.252458 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x28bd1): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28bd1): Mirror 6: 0xcf7e1380 (stale) Mirror 7: 0xcf7e1380 (stale) Mirror 9: 0xcf7e1380 (stale) Mirror 10: 0xcf7e1380 (stale) Mirror 12: 0xcf7e1380 (stale) Mirror 13: 0x9e4dd331 (stale) CRC-32 checksum value for chunk [0, 0x28bd1): Mirror 1: 0x9e4dd331 Mirror 14: 0x9e4dd331 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28bd1: skipped 000949 1786961988.625694 fallocate 0x0163f4 thru 0x01f0a1 (0x8cae bytes) 000950 1786961988.684421 punch from 0x00710a to 0x00ddb2 (0x6ca9 bytes) 000951 1786961988.689430 mapwrite 0x025bd1 thru 0x032f49 (0xd379 bytes) 000952 1786961988.844419 fallocate 0x02ad0a thru 0x02e07a (0x3371 bytes) 000953 1786961988.852955 mapwrite 0x0082d3 thru 0x010351 (0x807f bytes) 000955 1786961988.899116 mapwrite 0x01959e thru 0x01b27b (0x1cde bytes) 000956 1786961988.941322 mapread 0x0197cd thru 0x020bfe (0x7432 bytes) 000957 1786961988.942380 fallocate 0x01f1d6 thru 0x028cde (0x9b09 bytes) 000958 1786961988.947574 punch from 0x016fde to 0x01f8df (0x8902 bytes) 000959 1786961988.964012 read 0x012aaa thru 0x020f81 (0xe4d8 bytes) 000961 1786961988.972823 read 0x0014bc thru 0x00e928 (0xd46d bytes) 000962 1786961988.974676 punch from 0x008a06 to 0x010873 (0x7e6e bytes) 000964 1786961988.979734 trunc from 0x032f4a to 0x00e9c4 (0x24585 bytes) 000965 1786961989.048061 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xe9c5): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe9c5): Mirror 6: 0x7e0e86bd (stale) Mirror 7: 0x7e0e86bd (stale) Mirror 9: 0x7e0e86bd (stale) Mirror 10: 0x7e0e86bd (stale) Mirror 12: 0x7e0e86bd (stale) Mirror 13: 0x7e0e86bd (stale) Mirror 14: 0x7e0e86bd (stale) CRC-32 checksum value for chunk [0, 0xe9c5): Mirror 1: 0x48185b8b Mirror 15: 0x48185b8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe9c5: skipped 000966 1786961990.463035 punch from 0x003f00 to 0x00b75a (0x785b bytes) 000967 1786961990.530921 trunc from 0x009b74 to 0x00e9c4 (0x4e51 bytes) 000968 1786961990.547397 mapread 0x000792 thru 0x00747e (0x6ced bytes) 000969 1786961990.587430 mapread 0x006544 thru 0x00760e (0x10cb bytes) 000970 1786961990.588783 punch from 0x00b238 to 0x00e9c4 (0x378d bytes) 000971 1786961990.597552 trunc from 0x00e9c5 to 0x02c3e5 (0x1da21 bytes) 000972 1786961990.664474 mapwrite 0x03c6c6 thru 0x03f27a (0x2bb5 bytes) 000973 1786961990.781654 trunc from 0x03f27b to 0x017235 (0x28045 bytes) 000974 1786961990.819600 fallocate 0x03a3f0 thru 0x03ffff (0x5c10 bytes) 000975 1786961990.829262 mapread 0x00f298 thru 0x0150b9 (0x5e22 bytes) 000976 1786961990.858529 read 0x008bdb thru 0x0091f4 (0x61a bytes) 000977 1786961990.859835 write 0x03123a thru 0x03c296 (0xb05d bytes) 000978 1786961990.873032 MIRROR_RESYNC 9 mirrors 000979 1786961992.993464 MIRROR_RESYNC 9 mirrors 000980 1786961993.699761 mapwrite 0x027cdb thru 0x032656 (0xa97c bytes) 000981 1786961993.846528 mapread 0x00fcdc thru 0x015fd6 (0x62fb bytes) 000982 1786961993.862208 write 0x02131f thru 0x0299ff (0x86e1 bytes) 000983 1786961993.865305 mapwrite 0x03ac2d thru 0x03ffff (0x53d3 bytes) 000984 1786961993.922764 mapwrite 0x0168e8 thru 0x0186dc (0x1df5 bytes) 000985 1786961993.957021 write 0x01ad20 thru 0x029aaa (0xed8b bytes) 000986 1786961993.969731 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xfaec7442 (stale) Mirror 7: 0xfaec7442 (stale) Mirror 9: 0xfaec7442 (stale) Mirror 10: 0xfaec7442 (stale) Mirror 12: 0xfaec7442 (stale) Mirror 13: 0xfaec7442 (stale) Mirror 14: 0xfaec7442 (stale) Mirror 15: 0xfaec7442 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfad122a3 Mirror 16: 0xfad122a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000987 1786961995.410906 mapread 0x034d52 thru 0x03ffff (0xb2ae bytes) 000988 1786961995.478665 trunc from 0x02afe8 to 0x0367c1 (0xb7da bytes) 000989 1786961995.539804 read 0x03e79f thru 0x03ffff (0x1861 bytes) 000990 1786961995.554554 fallocate 0x021cb2 thru 0x02b245 (0x9594 bytes) 000991 1786961995.571808 write 0x01e072 thru 0x0253a5 (0x7334 bytes) 000992 1786961995.588354 fallocate 0x02e7fd thru 0x032f73 (0x4777 bytes) 000993 1786961995.593454 trunc from 0x03054c to 0x03605d (0x5b12 bytes) 000994 1786961995.603763 trunc from 0x040000 to 0x01d882 (0x2277d bytes) 000995 1786961995.648841 punch from 0x004eda to 0x009a76 (0x4b9d bytes) 000996 1786961995.665635 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1d883): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d883): Mirror 6: 0x408ef7ca (stale) Mirror 7: 0x408ef7ca (stale) Mirror 9: 0x408ef7ca (stale) Mirror 10: 0x408ef7ca (stale) Mirror 12: 0x408ef7ca (stale) Mirror 13: 0x408ef7ca (stale) Mirror 14: 0x408ef7ca (stale) Mirror 15: 0x408ef7ca (stale) Mirror 16: 0xeea62edf (stale) CRC-32 checksum value for chunk [0, 0x1d883): Mirror 1: 0xeea62edf Mirror 17: 0xeea62edf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d883: skipped 000998 1786961997.146561 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d883): Mirror 6: 0x408ef7ca (stale) Mirror 7: 0x408ef7ca (stale) Mirror 9: 0x408ef7ca (stale) Mirror 10: 0x408ef7ca (stale) Mirror 12: 0x408ef7ca (stale) Mirror 13: 0x408ef7ca (stale) Mirror 15: 0x408ef7ca (stale) Mirror 16: 0xeea62edf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d883): Mirror 6: 0x408ef7ca (stale) Mirror 7: 0x408ef7ca (stale) Mirror 9: 0x408ef7ca (stale) Mirror 10: 0x408ef7ca (stale) Mirror 12: 0x408ef7ca (stale) Mirror 13: 0x408ef7ca (stale) Mirror 15: 0x408ef7ca (stale) Mirror 16: 0xeea62edf (stale) CRC-32 checksum value for chunk [0, 0x1d883): Mirror 1: 0xeea62edf Mirror 17: 0xeea62edf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d883: skipped 000999 1786961998.560002 trunc from 0x01a4df to 0x01d882 (0x33a4 bytes) 001000 1786961998.656720 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d883): Mirror 7: 0x408ef7ca (stale) Mirror 9: 0x408ef7ca (stale) Mirror 10: 0x408ef7ca (stale) Mirror 12: 0x408ef7ca (stale) Mirror 13: 0x408ef7ca (stale) Mirror 15: 0x408ef7ca (stale) Mirror 16: 0xeea62edf (stale) Mirror 17: 0xeea62edf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d883): Mirror 7: 0x408ef7ca (stale) Mirror 9: 0x408ef7ca (stale) Mirror 10: 0x408ef7ca (stale) Mirror 12: 0x408ef7ca (stale) Mirror 13: 0x408ef7ca (stale) Mirror 15: 0x408ef7ca (stale) Mirror 16: 0xeea62edf (stale) Mirror 17: 0xeea62edf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d883: skipped All operations completed A-OK! PASS 70a (124s) == sanity-flr test 71: check mirror extend parent fid ==== 06:20:05 (1786962005) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x1ab:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 06:20:16 (1786962016) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.252702 s, 8.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150168 s, 7.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d5:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 06:20:26 (1786962026) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre2 Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5232672 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8597600 .. Extending file size to 7683360 .. Extending file size to 8491712 .. Extending file size to 8817472 .. Extending file size to 8608128 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8692832 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8829760 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61335 61336 61337 61341 61342 done 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) 192.168.203.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:21:49 (1786962109) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre2 Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2520416 .. Extending file size to 6621056 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6421184 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6915136 .. Extending file size to 7692192 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7717376 .. Extending file size to 8275264 .. Extending file size to 8293280 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8197024 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8365408 .. Extending file size to 8277760 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8346368 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 62956 62957 62960 62961 62964 failed 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) 192.168.203.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:23:14 (1786962194) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23216 3583804 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18068 3588952 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 41284 7172756 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2032 1269272 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 23216 3583804 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18068 3588952 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 41284 7172756 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102367 s, 20.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197565 s, 10.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x189e8414 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:23:35 (1786962215) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199641 s, 10.5 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 06:23:44 (1786962224) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ba:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ed:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ed:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ee:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x2ea:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:23:51 (1786962231) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125212 s, 16.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x56:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:24:00 (1786962240) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142252 s, 14.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2eb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2eb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ec:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:24:09 (1786962249) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128318 s, 16.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ee:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:24:43 (1786962283) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg301-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884265 s, 11.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f3:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:25:35 (1786962335) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg301-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134076 s, 15.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:26:27 (1786962387) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg301-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg301-server: Pool lustre.test_204f created Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg301-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0860728 s, 24.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg301-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:27:43 (1786962463) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:27:52 (1786962472) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:28:01 (1786962481) oleg301-server: Pool lustre.test_206 created Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg301-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg301-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:28:31 (1786962511) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692097 s, 15.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824238 s, 12.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x9db2e94f Mirror 3: 0x9db2e94f PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:28:42 (1786962522) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:28:46 (1786962526) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:28:50 (1786962530) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0878679 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ab0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x15e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x15e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x15e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xb40000 Read this (4 bytes): BBBB 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:29:27 (1786962567) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0501781 s, 163 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:29:35 (1786962575) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548866 s, 19.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1dd lmm_fid: [0x200000405:0x1dd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1dd lmm_fid: [0x200000405:0x1dd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x300:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611613 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-17 06:29:36.000000000 -0400 Modify: 2026-08-17 06:29:36.000000000 -0400 Change: 2026-08-17 06:29:38.000000000 -0400 Birth: 2026-08-17 06:29:36.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:29:43 (1786962583) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296578 s, 35.4 MB/s oleg301-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg301-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg301-server: O/280000401/d29/2fd: File not found by ext2_lookup PASS 210b (25s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 06:30:08 (1786962608) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.255925 s, 160 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0248875 s, 165 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 17 06:30 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e2:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 17 06:30 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 06:30:16 (1786962616) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 06:30:25 (1786962625) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1849 sec =========== 06:30:33 (1786962633) === sanity-flr: start cleanup 06:30:35 (1786962635) === === sanity-flr: finish cleanup 06:30:43 (1786962643) ===