-----============= acceptance-small: sanity-flr ============----- Mon Aug 10 00:59:20 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg210-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 00:59:44 (1786337984) === oleg210-client.virtnet: executing check_config_client /mnt/lustre oleg210-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg210-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff907e906d1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff907e906d1800.idle_timeout=debug disable quota as required oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:00:31 (1786338031) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8253] == sanity-flr test 0a: lfs mirror create with -N option == 01:00:34 (1786338034) PASS 0a (23s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:00:56 (1786338056) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:01:00 (1786338060) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:01:06 (1786338066) PASS 0d (31s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:01:36 (1786338096) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:01:40 (1786338100) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 01:01:44 (1786338104) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff907e906d1800.stats=clear osc.lustre-OST0001-osc-ffff907e906d1800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:01:54 (1786338114) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:02:07 (1786338127) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:02:15 (1786338135) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:02:20 (1786338140) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:02:30 (1786338150) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 01:02:41 (1786338161) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:02:49 (1786338169) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:02:57 (1786338177) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:03:07 (1786338187) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:03:20 (1786338200) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:03:28 (1786338208) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.35423 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985318 s, 10.6 MB/s PASS 21 (15s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:03:44 (1786338224) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.547995 s, 11.5 MB/s PASS 22 (12s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:03:55 (1786338235) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (15s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:04:10 (1786338250) Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff907e906d1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff907e906d1800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff907e906d1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff907e906d1800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg210-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (79s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:05:29 (1786338329) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.746302 s, 14.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff907e906d1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff907e906d1800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:06:16 (1786338376) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff907e906d1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff907e906d1800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff907e906d1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff907e906d1800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg210-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (97s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:07:53 (1786338473) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.04169 s, 17.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 10 01:07 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.978s user 0m0.000s sys 0m0.163s Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff907e906d1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff907e906d1800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.587s user 0m0.001s sys 0m0.185s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff907e906d1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff907e906d1800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.390s user 0m0.000s sys 0m0.147s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg210-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg210-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff907e906d1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff907e906d1800.ost_server_uuid in FULL state after 0 sec PASS 33b (108s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:09:41 (1786338581) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:09:45 (1786338585) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:09:49 (1786338589) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 01:09:53 (1786338593) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 01:10:03 (1786338603) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.776847 s, 16.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:10:17 (1786338617) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623307 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561623 s, 18.7 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:10:25 (1786338625) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:10:35 (1786338635) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000770141 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00372891 s, 275 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00081742 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000941135 s, 1.1 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 01:10:46 (1786338646) create test files with size 15738269 .. f22371eb6f63c69bce8ae751ac968562 - fb40d8857054fa4e03374fad305beb8f - a43176eaa6feade4cb533ea652ae7102 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg210-client.virtnet /mnt/lustre (opts:) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 01:11:13 (1786338673) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:11:27 (1786338687) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:11:35 (1786338695) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127057 s, 8.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb2:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb3:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340245 s, 30.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc3:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc4:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 01:11:47 (1786338707) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0658718 s, 31.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.255695 s, 16.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.219692 s, 9.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 01:12:03 (1786338723) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 01:12:08 (1786338728) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:12:12 (1786338732) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555581 s, 18.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 01:12:21 (1786338741) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.39437 s, 8.0 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:12:31 (1786338751) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbd:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xce:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xce:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:12:44 (1786338764) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.598544 s, 17.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg210-client.virtnet /mnt/lustre (opts:) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:12:57 (1786338777) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff907e98f76800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.299979 s, 35.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg210-client.virtnet /mnt/lustre (opts:) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff907e9182d800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:13:14 (1786338794) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (50s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:14:04 (1786338844) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:14:41 (1786338881) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 01:14:44 (1786338884) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.219498 s, 14.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.138113 s, 22.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x9e99d601 (nosync,stale) Mirror 2: 0x9e99d601 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 12eb68c2c70e02b83cf3a6471013304d - mirror 1 checksum: 37fb22cd2f0f388de07a8cfde156a60b - mirror 2 checksum: 12eb68c2c70e02b83cf3a6471013304d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-10T01:14:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-10T01:14:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 12eb68c2c70e02b83cf3a6471013304d - mirror 1 checksum: 37fb22cd2f0f388de07a8cfde156a60b - mirror 2 checksum: 37fb22cd2f0f388de07a8cfde156a60b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-10T01:14:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-10T01:14:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:14:56 (1786338896) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626848 s, 16.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:15:04 (1786338904) Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 01:15:17 (1786338917) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 01:15:53 (1786338953) targets are mounted 01:15:53 (1786338953) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269865 s, 38.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } PASS 50A (70s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:16:14 (1786338974) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0749902 s, 273 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0156264 s, 1.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:16:26 (1786338986) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.048787 s, 420 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00767403 s, 2.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:16:36 (1786338996) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00872525 s, 1.9 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:16:46 (1786339006) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0353358 s, 580 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (11s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:16:57 (1786339017) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00518029 s, 791 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0373526 s, 110 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:17:06 (1786339026) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:17:28 (1786339048) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:17:58 (1786339078) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:18:22 (1786339102) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:18:48 (1786339128) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 01:18:56 (1786339136) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57625 mirror split pid 57626 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57625 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57626 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 01:20:05 (1786339205) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 9207 000001 1786339207.549400 write 0x03f42a thru 0x03ffff (0xbd6 bytes) zero_range to largest ever: 0x17d99 000002 1786339207.562529 trunc from 0x009511 to 0x017d98 (0xe888 bytes) 000003 1786339207.570780 mapread 0x03b1f8 thru 0x03c729 (0x1532 bytes) 000004 1786339207.589164 mapread 0x02a379 thru 0x03309f (0x8d27 bytes) fallocating to largest ever: 0x40000 000005 1786339207.605148 fallocate 0x03dc08 thru 0x03ffff (0x23f8 bytes) 000006 1786339207.612693 mapwrite 0x01b49a thru 0x01db0d (0x2674 bytes) 000007 1786339207.692522 trunc from 0x022c64 to 0x02dee5 (0xb282 bytes) 000008 1786339207.700609 trunc from 0x040000 to 0x03cad9 (0x3526 bytes) 000009 1786339207.730943 write 0x03552c thru 0x03ffff (0xaad4 bytes) 000010 1786339207.735121 write 0x035240 thru 0x03ffff (0xadc0 bytes) 000011 1786339207.745117 fallocate 0x01a38a thru 0x021ad5 (0x774c bytes) 000012 1786339207.759310 punch from 0x001881 to 0x0099eb (0x816b bytes) 000013 1786339207.769083 trunc from 0x019acb to 0x01ebac (0x50e2 bytes) 000014 1786339207.798706 mapread 0x00a9d3 thru 0x015630 (0xac5e bytes) 000015 1786339207.884052 fallocate 0x006732 thru 0x0081e2 (0x1ab1 bytes) 000017 1786339207.897417 write 0x01e498 thru 0x023200 (0x4d69 bytes) 000018 1786339207.919722 mapwrite 0x02aa3c thru 0x03a49d (0xfa62 bytes) 000020 1786339207.976408 punch from 0x021334 to 0x028e6a (0x7b37 bytes) 000021 1786339207.995691 trunc from 0x02aed7 to 0x02c9dd (0x1b07 bytes) 000022 1786339208.009700 punch from 0x01ffd7 to 0x02346e (0x3498 bytes) 000023 1786339208.026489 write 0x019fb4 thru 0x01d0e2 (0x312f bytes) 000024 1786339208.040666 write 0x03ad79 thru 0x03ece2 (0x3f6a bytes) 000025 1786339208.046493 trunc from 0x040000 to 0x02fdd7 (0x10228 bytes) 000026 1786339208.076902 fallocate 0x015d4a thru 0x019c35 (0x3eec bytes) 000027 1786339208.089009 fallocate 0x0284f1 thru 0x02e1c5 (0x5cd5 bytes) 000028 1786339208.097698 mapread 0x0297d9 thru 0x02a038 (0x860 bytes) 000030 1786339208.117758 trunc from 0x02fdd8 to 0x03e6e1 (0xe90a bytes) 000031 1786339208.167017 punch from 0x033b97 to 0x03b972 (0x7ddc bytes) 000032 1786339208.174497 fallocate 0x003382 thru 0x00d757 (0xa3d6 bytes) 000033 1786339208.193683 punch from 0x018093 to 0x02051a (0x8488 bytes) 000034 1786339208.221244 punch from 0x0225f3 to 0x029654 (0x7062 bytes) 000035 1786339208.231287 trunc from 0x03e6e2 to 0x03ef8b (0x8aa bytes) 000036 1786339208.267005 trunc from 0x030150 to 0x03dfe1 (0xde92 bytes) 000038 1786339208.288233 trunc from 0x03ef8c to 0x03a713 (0x4878 bytes) 000039 1786339208.314246 trunc from 0x02763d to 0x02c63e (0x5002 bytes) 000040 1786339208.321219 punch from 0x025f2d to 0x02fcd0 (0x9da4 bytes) 000041 1786339208.330747 mapwrite 0x005324 thru 0x012062 (0xcd3f bytes) 000042 1786339208.369233 mapwrite 0x014e21 thru 0x020a86 (0xbc66 bytes) 000043 1786339208.410494 trunc from 0x03a714 to 0x029448 (0x112cb bytes) 000044 1786339208.435257 mapread 0x023f94 thru 0x029448 (0x54b5 bytes) 000045 1786339208.459427 mapwrite 0x01e5f4 thru 0x023864 (0x5271 bytes) 000046 1786339208.512735 mapread 0x01097c thru 0x012e63 (0x24e8 bytes) 000047 1786339208.515241 trunc from 0x029449 to 0x03edc0 (0x15978 bytes) 000048 1786339208.552079 fallocate 0x03f6d9 thru 0x03ffff (0x927 bytes) 000049 1786339208.560551 mapwrite 0x01f3c5 thru 0x025e42 (0x6a7e bytes) 000050 1786339208.595414 punch from 0x011f2c to 0x01ef77 (0xd04c bytes) 000051 1786339208.614663 mapread 0x00d6f1 thru 0x00fcef (0x25ff bytes) 000052 1786339208.622273 mapread 0x016453 thru 0x01c84c (0x63fa bytes) 000053 1786339208.641235 trunc from 0x02aca8 to 0x02c959 (0x1cb2 bytes) 000054 1786339208.646905 write 0x025cad thru 0x034085 (0xe3d9 bytes) 000055 1786339208.654027 write 0x02cf9a thru 0x03a97b (0xd9e2 bytes) 000056 1786339208.662602 punch from 0x01f930 to 0x028be3 (0x92b4 bytes) 000059 1786339208.735551 MIRROR_EXTEND to 2 mirrors 000061 1786339210.029245 fallocate 0x00c68c thru 0x0127d7 (0x614c bytes) 000062 1786339210.063852 read 0x03a475 thru 0x03e8a4 (0x4430 bytes) 000063 1786339210.083447 fallocate 0x00b0bb thru 0x0184f3 (0xd439 bytes) 000064 1786339210.098231 mapwrite 0x0342d0 thru 0x03ffff (0xbd30 bytes) 000065 1786339210.312758 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000066 1786339210.602642 read 0x028000 thru 0x036bb9 (0xebba bytes) 000067 1786339210.634406 MIRROR_EXTEND to 2 mirrors 000068 1786339211.768607 trunc from 0x03612e to 0x03a1bb (0x408e bytes) 000069 1786339211.834001 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000071 1786339212.022072 trunc from 0x040000 to 0x006789 (0x39876 bytes) 000073 1786339212.051781 punch from 0x002abf to 0x006789 (0x3ccb bytes) 000074 1786339212.056596 fallocate 0x034402 thru 0x0384aa (0x40a9 bytes) 000076 1786339212.067044 mapwrite 0x0038c2 thru 0x011c4a (0xe389 bytes) 000078 1786339212.199656 mapwrite 0x01eaf8 thru 0x023651 (0x4b5a bytes) 000079 1786339212.322110 read 0x00291e thru 0x00cf57 (0xa63a bytes) 000080 1786339212.324947 mapwrite 0x016ffa thru 0x0171b0 (0x1b7 bytes) 000081 1786339212.354457 trunc from 0x008fe7 to 0x010ab4 (0x7ace bytes) 000082 1786339212.360492 read 0x01de15 thru 0x023651 (0x583d bytes) 000084 1786339212.368075 MIRROR_EXTEND to 2 mirrors 000085 1786339213.169625 mapread 0x00c9ed thru 0x01a7e6 (0xddfa bytes) 000086 1786339213.215381 write 0x0000a5 thru 0x00c89d (0xc7f9 bytes) 000087 1786339213.289642 mapread 0x00e8b6 thru 0x01e27e (0xf9c9 bytes) 000088 1786339213.311536 mapread 0x011fb1 thru 0x017c4f (0x5c9f bytes) 000089 1786339213.312632 punch from 0x01acb0 to 0x0217ba (0x6b0b bytes) 000090 1786339213.316892 write 0x02ea11 thru 0x0343ee (0x59de bytes) 000091 1786339213.319425 fallocate 0x01cf07 thru 0x02b76a (0xe864 bytes) 000093 1786339213.325049 trunc from 0x01e60c to 0x02e565 (0xff5a bytes) 000094 1786339213.357649 trunc from 0x0343ef to 0x015edf (0x1e50f bytes) 000095 1786339213.448746 trunc from 0x00b9a9 to 0x015edf (0xa537 bytes) 000096 1786339213.461287 mapwrite 0x026bed thru 0x02ce90 (0x62a4 bytes) 000097 1786339213.530432 fallocate 0x03a187 thru 0x03cbec (0x2a66 bytes) 000098 1786339213.538769 read 0x0140db thru 0x01effb (0xaf21 bytes) 000099 1786339213.565706 punch from 0x002a00 to 0x007a0b (0x500c bytes) 000100 1786339213.573918 write 0x011dc8 thru 0x020d81 (0xefba bytes) 000101 1786339213.584775 trunc from 0x02488c to 0x02b801 (0x6f76 bytes) 000103 1786339213.594932 mapread 0x00d4b3 thru 0x00f638 (0x2186 bytes) 000104 1786339213.618631 mapwrite 0x0284d6 thru 0x02b847 (0x3372 bytes) 000105 1786339213.672537 punch from 0x02af26 to 0x02c91f (0x19fa bytes) 000106 1786339213.682496 write 0x032a5b thru 0x03a2fa (0x78a0 bytes) 000107 1786339213.698903 trunc from 0x021a1f to 0x021d09 (0x2eb bytes) 000108 1786339213.707436 punch from 0x029c89 to 0x02fe7a (0x61f2 bytes) 000109 1786339213.715829 trunc from 0x03cbed to 0x0147af (0x2843d bytes) 000112 1786339213.753409 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000113 1786339213.947488 fallocate 0x02cecd thru 0x03a419 (0xd54d bytes) 000114 1786339213.960361 fallocate 0x012856 thru 0x012fe3 (0x78e bytes) 000115 1786339213.987169 trunc from 0x03a41a to 0x01be7c (0x1e59d bytes) 000116 1786339214.056713 fallocate 0x03ee22 thru 0x03ffff (0x11de bytes) 000117 1786339214.072490 mapwrite 0x0342bd thru 0x03c01f (0x7d63 bytes) 000118 1786339214.193906 trunc from 0x03c020 to 0x012559 (0x29ac6 bytes) 000119 1786339214.248313 write 0x0298c3 thru 0x03523f (0xb97d bytes) 000120 1786339214.251777 trunc from 0x035240 to 0x0271e0 (0xe05f bytes) 000121 1786339214.292275 trunc from 0x012f9b to 0x01f935 (0xc99b bytes) 000122 1786339214.313950 read 0x01ec78 thru 0x0271e0 (0x8569 bytes) 000123 1786339214.339847 punch from 0x00e71c to 0x0105ac (0x1e91 bytes) 000124 1786339214.355894 mapread 0x0027c1 thru 0x00a96e (0x81ae bytes) 000125 1786339214.380059 mapwrite 0x0038d9 thru 0x0054fd (0x1c25 bytes) 000126 1786339214.404300 trunc from 0x020c6e to 0x023c46 (0x2fd9 bytes) 000127 1786339214.411630 trunc from 0x00494d to 0x009682 (0x4d36 bytes) 000128 1786339214.422243 mapwrite 0x026c53 thru 0x02afde (0x438c bytes) 000129 1786339214.578530 trunc from 0x02afdf to 0x002982 (0x2865c bytes) 000130 1786339214.639206 MIRROR_EXTEND to 2 mirrors 000131 1786339215.361912 mapwrite 0x0346db thru 0x03f174 (0xaa9a bytes) 000132 1786339215.519742 mapread 0x0234df thru 0x02cedb (0x99fd bytes) 000134 1786339215.568412 trunc from 0x03ac09 to 0x03f174 (0x456c bytes) 000135 1786339215.584122 trunc from 0x03f175 to 0x00d0fb (0x32079 bytes) 000136 1786339215.633562 punch from 0x007b58 to 0x00d0fb (0x55a4 bytes) 000137 1786339215.643377 punch from 0x0035ab to 0x00cf8f (0x99e5 bytes) 000138 1786339215.658147 mapwrite 0x000465 thru 0x01041d (0xffb9 bytes) 000140 1786339215.723007 mapwrite 0x03cf74 thru 0x03ffff (0x308c bytes) 000141 1786339215.765984 mapread 0x0155f7 thru 0x017776 (0x2180 bytes) 000143 1786339215.797086 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x841ad78c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x76e357ae Mirror 3: 0x76e357ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000145 1786339216.881026 mapwrite 0x00880b thru 0x0092a3 (0xa99 bytes) 000146 1786339217.020121 read 0x015f92 thru 0x02007c (0xa0eb bytes) 000147 1786339217.038166 mapread 0x02270b thru 0x024a2d (0x2323 bytes) 000148 1786339217.048894 read 0x011458 thru 0x01cf52 (0xbafb bytes) 000149 1786339217.062206 trunc from 0x00047d to 0x006e36 (0x69ba bytes) 000150 1786339217.072311 read 0x01effc thru 0x02e54c (0xf551 bytes) 000151 1786339217.073316 trunc from 0x040000 to 0x025c7c (0x1a383 bytes) 000152 1786339217.115677 mapread 0x0088de thru 0x00c479 (0x3b9c bytes) 000153 1786339217.117106 trunc from 0x025c7d to 0x025abf (0x1bd bytes) 000154 1786339217.187528 mapwrite 0x01f4e6 thru 0x029add (0xa5f8 bytes) 000155 1786339217.318414 fallocate 0x03be5c thru 0x03d933 (0x1ad8 bytes) 000157 1786339217.352986 fallocate 0x031ad4 thru 0x03248d (0x9ba bytes) 000158 1786339217.387510 fallocate 0x01ba8b thru 0x02b155 (0xf6cb bytes) 000159 1786339217.399365 write 0x0149e5 thru 0x01967e (0x4c9a bytes) 000160 1786339217.415401 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b156): Mirror 3: 0xa46be901 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b156): Mirror 3: 0xa46be901 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b156: skipped 000162 1786339217.884535 trunc from 0x02b156 to 0x03a714 (0xf5bf bytes) 000165 1786339217.937067 mapread 0x039c00 thru 0x03a714 (0xb15 bytes) 000166 1786339217.963591 write 0x00b05d thru 0x00cf7f (0x1f23 bytes) 000168 1786339217.992956 trunc from 0x03a715 to 0x038dec (0x1928 bytes) 000169 1786339218.047952 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000170 1786339218.261913 mapwrite 0x01fbc2 thru 0x020647 (0xa86 bytes) 000171 1786339218.294487 MIRROR_EXTEND to 2 mirrors 000172 1786339219.361578 trunc from 0x038ded to 0x0003b4 (0x38a38 bytes) 000174 1786339219.432729 read 0x0002a8 thru 0x0003b4 (0x10d bytes) 000175 1786339219.444586 fallocate 0x014788 thru 0x01ad5d (0x65d6 bytes) 000176 1786339219.451754 trunc from 0x0003b5 to 0x03e354 (0x3dfa0 bytes) 000177 1786339219.487589 read 0x00b46b thru 0x011c49 (0x67df bytes) 000178 1786339219.508258 mapread 0x00c127 thru 0x0129f9 (0x68d3 bytes) 000179 1786339219.509918 mapwrite 0x01db3d thru 0x021a6f (0x3f33 bytes) 000180 1786339219.531246 punch from 0x00d644 to 0x011174 (0x3b31 bytes) 000181 1786339219.538991 fallocate 0x00ebc8 thru 0x01730f (0x8748 bytes) 000183 1786339219.547536 mapread 0x01d396 thru 0x0288b1 (0xb51c bytes) 000186 1786339219.552296 mapread 0x002b90 thru 0x00cfe3 (0xa454 bytes) 000187 1786339219.554169 punch from 0x018656 to 0x021317 (0x8cc2 bytes) 000189 1786339219.561229 write 0x00c1c1 thru 0x012c85 (0x6ac5 bytes) 000190 1786339219.564280 mapwrite 0x022f9c thru 0x02a4b2 (0x7517 bytes) 000191 1786339219.604523 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000194 1786339219.815114 write 0x01e614 thru 0x0298a2 (0xb28f bytes) 000195 1786339219.834889 read 0x02c46e thru 0x03904c (0xcbdf bytes) 000196 1786339219.847544 fallocate 0x03e4e5 thru 0x03ffff (0x1b1b bytes) 000197 1786339219.853114 read 0x036be0 thru 0x03a1ff (0x3620 bytes) 000198 1786339219.863435 punch from 0x036ae4 to 0x03ffff (0x951c bytes) 000200 1786339219.869666 mapwrite 0x032c78 thru 0x03ffff (0xd388 bytes) 000201 1786339219.944948 write 0x009a3e thru 0x00e83a (0x4dfd bytes) 000202 1786339219.946529 mapwrite 0x03bb8a thru 0x03ffff (0x4476 bytes) 000205 1786339219.973156 trunc from 0x040000 to 0x015a52 (0x2a5ad bytes) 000206 1786339219.994869 punch from 0x0133e6 to 0x015a52 (0x266d bytes) 000207 1786339220.014881 punch from 0x00bf4a to 0x00e8fe (0x29b5 bytes) 000208 1786339220.030226 write 0x02f7f0 thru 0x0301ea (0x9fb bytes) 000209 1786339220.036937 trunc from 0x00333d to 0x00fe36 (0xcafa bytes) 000210 1786339220.046739 mapread 0x012876 thru 0x0188f7 (0x6082 bytes) 000211 1786339220.099786 punch from 0x003478 to 0x00dfe9 (0xab72 bytes) 000212 1786339220.113551 trunc from 0x005edd to 0x0072a9 (0x13cd bytes) 000213 1786339220.124329 trunc from 0x0301eb to 0x00ce2e (0x233bc bytes) 000214 1786339220.160879 trunc from 0x0006a3 to 0x007536 (0x6e94 bytes) 000215 1786339220.177587 trunc from 0x004266 to 0x00c263 (0x7ffe bytes) 000216 1786339220.189411 write 0x026403 thru 0x026f87 (0xb85 bytes) 000218 1786339220.195664 mapwrite 0x02056e thru 0x02d8e3 (0xd376 bytes) 000220 1786339220.320858 MIRROR_EXTEND to 2 mirrors 000221 1786339221.222274 mapread 0x015aea thru 0x01c0e1 (0x65f8 bytes) 000222 1786339221.291484 write 0x02469d thru 0x02ee8f (0xa7f3 bytes) 000223 1786339221.396589 mapread 0x01fe82 thru 0x028a29 (0x8ba8 bytes) 000224 1786339221.415285 trunc from 0x02672c to 0x029419 (0x2cee bytes) 000225 1786339221.444396 trunc from 0x02ee90 to 0x014a4f (0x1a440 bytes) 000226 1786339221.501577 fallocate 0x007cad thru 0x012424 (0xa778 bytes) 000227 1786339221.517936 mapwrite 0x022c0b thru 0x02aabd (0x7eb3 bytes) 000228 1786339221.623078 mapwrite 0x02c4ed thru 0x02e5dc (0x20f0 bytes) 000229 1786339221.688557 fallocate 0x02a4ff thru 0x02d348 (0x2e4a bytes) 000230 1786339221.695741 fallocate 0x02f2c6 thru 0x036cac (0x79e7 bytes) 000232 1786339221.702725 mapwrite 0x02c91e thru 0x030d4c (0x442f bytes) 000233 1786339221.732354 mapwrite 0x01caf1 thru 0x029cf8 (0xd208 bytes) 000234 1786339221.760094 read 0x010090 thru 0x01c85e (0xc7cf bytes) 000235 1786339221.771946 write 0x00ae18 thru 0x00fddf (0x4fc8 bytes) 000236 1786339221.783942 trunc from 0x0006a4 to 0x009b32 (0x948f bytes) 000238 1786339221.797544 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000239 1786339221.945183 mapwrite 0x001d7f thru 0x010c2e (0xeeb0 bytes) 000240 1786339221.991091 mapread 0x034031 thru 0x036cac (0x2c7c bytes) 000241 1786339222.003286 fallocate 0x02b149 thru 0x037b2a (0xc9e2 bytes) 000242 1786339222.008558 trunc from 0x036cad to 0x03c5ea (0x593e bytes) 000244 1786339222.031824 trunc from 0x02d6e0 to 0x03bc8b (0xe5ac bytes) 000246 1786339222.041115 mapread 0x0295a9 thru 0x02e36d (0x4dc5 bytes) 000247 1786339222.054159 write 0x003d06 thru 0x00c315 (0x8610 bytes) 000248 1786339222.056130 mapwrite 0x037343 thru 0x03cf0d (0x5bcb bytes) 000249 1786339222.104412 mapwrite 0x0340c7 thru 0x03ffff (0xbf39 bytes) 000250 1786339222.159129 mapwrite 0x032f99 thru 0x03ffff (0xd067 bytes) 000251 1786339222.181001 read 0x036fd8 thru 0x03df96 (0x6fbf bytes) 000252 1786339222.181689 trunc from 0x028aaf to 0x029e48 (0x139a bytes) 000253 1786339222.191132 fallocate 0x01b14a thru 0x02940f (0xe2c6 bytes) 000254 1786339222.197158 mapread 0x00f444 thru 0x01a89a (0xb457 bytes) 000255 1786339222.228967 punch from 0x010c2e to 0x015cb9 (0x508c bytes) 000256 1786339222.244585 trunc from 0x040000 to 0x01df8f (0x22070 bytes) 000258 1786339222.268403 write 0x01712d thru 0x023b6f (0xca43 bytes) 000259 1786339222.271429 write 0x020b64 thru 0x02609d (0x553a bytes) 000260 1786339222.272876 trunc from 0x02609e to 0x0212f2 (0x4dab bytes) 000261 1786339222.298697 trunc from 0x0212f3 to 0x01172a (0xfbc8 bytes) 000262 1786339222.327341 read 0x00496f thru 0x01172a (0xcdbc bytes) 000263 1786339222.346732 fallocate 0x036bfa thru 0x038fc7 (0x23ce bytes) 000264 1786339222.360397 read 0x0036b0 thru 0x009350 (0x5ca1 bytes) 000265 1786339222.362830 read 0x00732d thru 0x00b6cc (0x43a0 bytes) 000266 1786339222.368595 trunc from 0x038fc8 to 0x03c0e7 (0x3120 bytes) 000268 1786339222.407553 trunc from 0x03c0e8 to 0x03fd6f (0x3c88 bytes) 000269 1786339222.427763 read 0x03653b thru 0x03dcb9 (0x777f bytes) 000270 1786339222.442227 mapread 0x038e3f thru 0x03ca78 (0x3c3a bytes) 000271 1786339222.443722 read 0x003867 thru 0x00d14b (0x98e5 bytes) 000272 1786339222.444621 write 0x0323a0 thru 0x03ffff (0xdc60 bytes) 000273 1786339222.447167 write 0x021e17 thru 0x02b1d5 (0x93bf bytes) 000275 1786339222.449903 mapwrite 0x022efb thru 0x0303be (0xd4c4 bytes) 000276 1786339222.474421 trunc from 0x040000 to 0x00f8bf (0x30740 bytes) 000277 1786339222.506422 mapwrite 0x038439 thru 0x03ffff (0x7bc7 bytes) 000278 1786339222.570757 read 0x035961 thru 0x03ffff (0xa69f bytes) 000279 1786339222.571528 read 0x00dd69 thru 0x01b47b (0xd713 bytes) 000280 1786339222.572230 write 0x03a4b4 thru 0x03ffff (0x5b4c bytes) 000281 1786339222.575595 trunc from 0x0082b5 to 0x015ecf (0xdc1b bytes) 000282 1786339222.588365 MIRROR_EXTEND to 2 mirrors 000283 1786339223.728363 trunc from 0x00605b to 0x013277 (0xd21d bytes) 000286 1786339223.763193 read 0x031b64 thru 0x03eb58 (0xcff5 bytes) 000287 1786339223.779493 mapwrite 0x01c18b thru 0x023858 (0x76ce bytes) 000288 1786339223.819727 trunc from 0x0076fc to 0x00bda9 (0x46ae bytes) 000290 1786339223.825748 trunc from 0x0390b8 to 0x03b5ff (0x2548 bytes) 000291 1786339223.831746 punch from 0x003fa7 to 0x00a0b9 (0x6113 bytes) 000292 1786339223.836786 fallocate 0x01d5b1 thru 0x023076 (0x5ac6 bytes) 000294 1786339223.843763 trunc from 0x040000 to 0x002455 (0x3dbaa bytes) 000295 1786339223.872563 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2456): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2456): Mirror 2: 0xeba337ff (stale) CRC-32 checksum value for chunk [0, 0x2456): Mirror 1: 0xeba337ff Mirror 3: 0xeba337ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2456: skipped 000296 1786339224.962143 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000297 1786339225.481068 write 0x03e67e thru 0x03ffff (0x1982 bytes) 000298 1786339225.587536 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000299 1786339225.806419 mapread 0x0275bb thru 0x034168 (0xcbae bytes) 000301 1786339225.839552 mapread 0x00cf34 thru 0x015fcb (0x9098 bytes) 000302 1786339225.860139 write 0x011082 thru 0x01f3e4 (0xe363 bytes) 000303 1786339225.871554 mapwrite 0x037ca0 thru 0x03ffff (0x8360 bytes) 000304 1786339225.903715 write 0x026639 thru 0x02d2c7 (0x6c8f bytes) 000305 1786339225.905880 write 0x03877b thru 0x03ffff (0x7885 bytes) 000306 1786339225.908334 trunc from 0x01fdc1 to 0x02c8bf (0xcaff bytes) 000308 1786339225.924101 punch from 0x0346e2 to 0x03c9ac (0x82cb bytes) 000309 1786339225.934044 mapwrite 0x00fef9 thru 0x01e1d0 (0xe2d8 bytes) 000311 1786339225.958331 fallocate 0x0002a6 thru 0x002884 (0x25df bytes) 000312 1786339225.964954 mapread 0x01b275 thru 0x022a34 (0x77c0 bytes) 000313 1786339225.976541 fallocate 0x012f8d thru 0x022da9 (0xfe1d bytes) 000314 1786339225.982395 trunc from 0x040000 to 0x015bd1 (0x2a42e bytes) 000315 1786339225.998733 write 0x02e394 thru 0x03cfed (0xec5a bytes) 000316 1786339226.002902 mapread 0x01a17c thru 0x023a3b (0x98c0 bytes) 000317 1786339226.017448 mapwrite 0x000519 thru 0x005502 (0x4fea bytes) 000318 1786339226.044652 mapwrite 0x00332f thru 0x00756c (0x423e bytes) 000320 1786339226.061672 read 0x00de35 thru 0x0111d6 (0x33a2 bytes) 000321 1786339226.063053 read 0x002d73 thru 0x0108b6 (0xdb44 bytes) 000323 1786339226.065480 trunc from 0x03cfee to 0x006d60 (0x3628d bytes) 000325 1786339226.085760 write 0x01c636 thru 0x02a4dc (0xdea7 bytes) 000326 1786339226.089506 punch from 0x001e3a to 0x0101f3 (0xe3ba bytes) 000327 1786339226.095729 trunc from 0x00a394 to 0x013a72 (0x96df bytes) 000328 1786339226.103651 write 0x00823d thru 0x013a80 (0xb844 bytes) 000329 1786339226.119371 read 0x01f9ff thru 0x0238ca (0x3ecc bytes) 000330 1786339226.120367 mapread 0x00fb0d thru 0x01b9be (0xbeb2 bytes) 000331 1786339226.123387 punch from 0x0039d5 to 0x00d98a (0x9fb6 bytes) 000332 1786339226.137442 punch from 0x01e658 to 0x02399f (0x5348 bytes) 000334 1786339226.143693 write 0x00a2dc thru 0x0167e8 (0xc50d bytes) 000335 1786339226.155430 trunc from 0x0001fd to 0x00de16 (0xdc1a bytes) 000337 1786339226.171101 punch from 0x00c137 to 0x013036 (0x6f00 bytes) 000339 1786339226.181551 mapread 0x0011d8 thru 0x00a73a (0x9563 bytes) 000340 1786339226.258641 punch from 0x026dd2 to 0x02a4dc (0x370b bytes) 000341 1786339226.272986 trunc from 0x02a4dd to 0x0292f6 (0x11e6 bytes) 000343 1786339226.292049 read 0x022029 thru 0x0292f6 (0x72ce bytes) 000344 1786339226.305292 trunc from 0x01205a to 0x0200b1 (0xe058 bytes) 000345 1786339226.312387 mapwrite 0x0252ff thru 0x034532 (0xf234 bytes) 000346 1786339226.404510 trunc from 0x0128c9 to 0x01b0d6 (0x880e bytes) 000347 1786339226.421745 mapwrite 0x0363c6 thru 0x036dcb (0xa06 bytes) 000348 1786339226.488569 trunc from 0x00ec17 to 0x016b0f (0x7ef9 bytes) 000349 1786339226.501945 write 0x0092d0 thru 0x014de7 (0xbb18 bytes) 000350 1786339226.526432 trunc from 0x03156a to 0x03288e (0x1325 bytes) 000352 1786339226.542424 trunc from 0x01fbd1 to 0x02932d (0x975d bytes) 000353 1786339226.557234 mapwrite 0x00db69 thru 0x0180ee (0xa586 bytes) 000354 1786339226.639587 read 0x011178 thru 0x01da47 (0xc8d0 bytes) 000355 1786339226.651640 mapwrite 0x00a436 thru 0x00c96f (0x253a bytes) 000356 1786339226.663506 trunc from 0x036dcc to 0x024e71 (0x11f5a bytes) 000357 1786339226.675651 trunc from 0x024e72 to 0x037ad1 (0x12c60 bytes) 000358 1786339226.704070 read 0x0273d2 thru 0x02e6a9 (0x72d8 bytes) 000359 1786339226.716115 fallocate 0x0145bc thru 0x0206a6 (0xc0eb bytes) 000360 1786339226.720333 punch from 0x000b92 to 0x009117 (0x8586 bytes) 000361 1786339226.724339 mapwrite 0x02e542 thru 0x030c14 (0x26d3 bytes) 000362 1786339226.736624 trunc from 0x012744 to 0x01386f (0x112c bytes) 000363 1786339226.741757 punch from 0x023f2f to 0x02c712 (0x87e4 bytes) 000364 1786339226.746659 trunc from 0x037ad2 to 0x00ebe6 (0x28eeb bytes) 000365 1786339226.776780 punch from 0x007269 to 0x00a531 (0x32c9 bytes) 000366 1786339226.781942 write 0x0299cc thru 0x0311ee (0x7823 bytes) 000367 1786339226.783694 trunc from 0x0311ef to 0x00e81d (0x229d1 bytes) 000369 1786339226.802256 trunc from 0x00060a to 0x00aa42 (0xa439 bytes) 000371 1786339226.808123 trunc from 0x00e81e to 0x01ff54 (0x11737 bytes) 000372 1786339226.827786 write 0x007c6f thru 0x00af2d (0x32bf bytes) 000374 1786339226.840585 trunc from 0x01ff55 to 0x00c135 (0x13e1f bytes) 000375 1786339226.866661 trunc from 0x00ba58 to 0x00c135 (0x6de bytes) 000376 1786339226.871173 fallocate 0x006212 thru 0x00be38 (0x5c27 bytes) 000377 1786339226.875147 read 0x00a0ee thru 0x00c135 (0x2048 bytes) 000378 1786339226.892418 MIRROR_EXTEND to 2 mirrors 000380 1786339227.836052 mapwrite 0x01b2c7 thru 0x01fee0 (0x4c1a bytes) 000381 1786339228.024338 write 0x0156a4 thru 0x01800a (0x2967 bytes) 000382 1786339228.032947 write 0x03a099 thru 0x03ffff (0x5f67 bytes) 000383 1786339228.035038 trunc from 0x040000 to 0x03dc77 (0x2388 bytes) 000384 1786339228.077889 mapwrite 0x00f482 thru 0x0154c2 (0x6041 bytes) 000385 1786339228.127670 mapwrite 0x03589b thru 0x03ffff (0xa765 bytes) 000386 1786339228.230498 MIRROR_RESYNC 2 mirrors 000387 1786339229.632316 mapwrite 0x02c6cb thru 0x034463 (0x7d99 bytes) 000388 1786339229.740228 trunc from 0x040000 to 0x0092e3 (0x36d1c bytes) 000389 1786339229.767930 write 0x0173a5 thru 0x01c131 (0x4d8d bytes) 000390 1786339229.775488 trunc from 0x014f62 to 0x017c6f (0x2d0e bytes) 000391 1786339229.809389 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1c132): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c132): Mirror 2: 0xf5a1a321 (stale) CRC-32 checksum value for chunk [0, 0x1c132): Mirror 1: 0xe485f319 Mirror 3: 0xe485f319 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c132: skipped 000392 1786339230.849302 MIRROR_RESYNC 3 mirrors 000393 1786339231.729848 trunc from 0x01c132 to 0x02be05 (0xfcd4 bytes) 000394 1786339231.789997 trunc from 0x020faa to 0x023750 (0x27a7 bytes) 000396 1786339231.799337 mapwrite 0x02f7ae thru 0x02feb0 (0x703 bytes) 000397 1786339231.856357 trunc from 0x009875 to 0x00c87e (0x300a bytes) 000398 1786339231.862721 mapwrite 0x035ff8 thru 0x03f943 (0x994c bytes) 000399 1786339231.967659 trunc from 0x01425f to 0x01508a (0xe2c bytes) 000400 1786339231.979856 trunc from 0x0131c3 to 0x01e055 (0xae93 bytes) 000401 1786339231.986218 trunc from 0x03f944 to 0x038ca3 (0x6ca0 bytes) 000402 1786339232.016003 trunc from 0x038ca4 to 0x02284e (0x16455 bytes) 000403 1786339232.035944 read 0x00621d thru 0x00c2cb (0x60af bytes) 000404 1786339232.044748 read 0x021e42 thru 0x02284e (0xa0d bytes) 000405 1786339232.066148 MIRROR_RESYNC 3 mirrors 000406 1786339232.747254 mapread 0x0062b0 thru 0x0144e7 (0xe238 bytes) 000407 1786339232.777733 fallocate 0x00c4b4 thru 0x019a56 (0xd5a3 bytes) 000408 1786339232.831631 fallocate 0x0342a8 thru 0x03a5ea (0x6343 bytes) 000409 1786339232.840593 punch from 0x011c88 to 0x015ae4 (0x3e5d bytes) 000410 1786339232.846503 trunc from 0x02284f to 0x03a218 (0x179ca bytes) 000412 1786339232.914405 mapread 0x013082 thru 0x0198dd (0x685c bytes) 000413 1786339232.925074 fallocate 0x02c44e thru 0x033cfc (0x78af bytes) 000414 1786339232.944842 trunc from 0x03a219 to 0x03aa56 (0x83e bytes) 000415 1786339232.961885 mapwrite 0x01ee75 thru 0x02298f (0x3b1b bytes) 000416 1786339232.995282 punch from 0x0083a1 to 0x00a04d (0x1cad bytes) 000417 1786339233.004582 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3aa57): Mirror 3: 0x56686d96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa57): Mirror 3: 0x56686d96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa57: skipped 000418 1786339233.253305 read 0x00803f thru 0x016544 (0xe506 bytes) 000419 1786339233.267193 trunc from 0x0188db to 0x01a064 (0x178a bytes) 000420 1786339233.272680 mapread 0x039c0c thru 0x03aa56 (0xe4b bytes) 000421 1786339233.283830 write 0x02f6f9 thru 0x033e8d (0x4795 bytes) 000422 1786339233.299669 trunc from 0x03aa57 to 0x02d146 (0xd910 bytes) 000423 1786339233.318453 write 0x00cabb thru 0x01a90a (0xde50 bytes) 000424 1786339233.324868 write 0x01d30d thru 0x01e4a2 (0x1196 bytes) 000425 1786339233.335561 read 0x00cc8c thru 0x01760c (0xa981 bytes) 000426 1786339233.340497 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000429 1786339233.481326 punch from 0x02a0fa to 0x02d146 (0x304d bytes) 000430 1786339233.492175 read 0x006b01 thru 0x010b26 (0xa026 bytes) 000431 1786339233.516439 MIRROR_EXTEND to 2 mirrors 000432 1786339234.724901 trunc from 0x02d147 to 0x03a13f (0xcff9 bytes) 000433 1786339234.817854 trunc from 0x035248 to 0x038af2 (0x38ab bytes) 000434 1786339234.830744 write 0x02f5c8 thru 0x02f673 (0xac bytes) 000435 1786339234.849009 mapwrite 0x036226 thru 0x03ffff (0x9dda bytes) 000436 1786339234.939878 read 0x006279 thru 0x007081 (0xe09 bytes) 000437 1786339234.948194 read 0x016e6d thru 0x0240aa (0xd23e bytes) 000438 1786339234.960223 write 0x013172 thru 0x0131a0 (0x2f bytes) 000439 1786339234.965481 read 0x00be09 thru 0x0100dc (0x42d4 bytes) 000440 1786339234.966846 mapwrite 0x03d4de thru 0x03ffff (0x2b22 bytes) 000441 1786339234.988280 mapread 0x0184f4 thru 0x01da52 (0x555f bytes) 000442 1786339234.989470 punch from 0x03b4a7 to 0x03ffff (0x4b59 bytes) 000443 1786339235.001123 mapread 0x038509 thru 0x03ffff (0x7af7 bytes) 000444 1786339235.025291 MIRROR_RESYNC 2 mirrors 000445 1786339235.965270 trunc from 0x01501b to 0x01901b (0x4001 bytes) 000446 1786339236.038741 write 0x03f94d thru 0x03ffff (0x6b3 bytes) 000447 1786339236.108486 write 0x022e41 thru 0x03078b (0xd94b bytes) 000448 1786339236.152539 write 0x003061 thru 0x00404f (0xfef bytes) 000450 1786339236.169129 mapread 0x0052cf thru 0x00aa6e (0x57a0 bytes) 000451 1786339236.193164 trunc from 0x040000 to 0x03378e (0xc871 bytes) 000452 1786339236.224469 trunc from 0x03378f to 0x00b3d2 (0x283bc bytes) 000453 1786339236.265574 trunc from 0x00af94 to 0x00b3d2 (0x43f bytes) 000455 1786339236.272135 mapwrite 0x01ac4b thru 0x02a031 (0xf3e7 bytes) 000456 1786339236.335159 mapwrite 0x037869 thru 0x03ffff (0x8797 bytes) 000457 1786339236.402616 fallocate 0x00adcf thru 0x01622e (0xb460 bytes) 000458 1786339236.411655 trunc from 0x007ce2 to 0x014ed3 (0xd1f2 bytes) 000459 1786339236.419189 trunc from 0x01ffda to 0x020951 (0x978 bytes) 000460 1786339236.424575 fallocate 0x031cfe thru 0x039014 (0x7317 bytes) 000461 1786339236.430717 mapread 0x01cf0b thru 0x020967 (0x3a5d bytes) 000463 1786339236.446714 MIRROR_RESYNC 2 mirrors 000464 1786339237.099919 mapread 0x02552d thru 0x034ee9 (0xf9bd bytes) 000466 1786339237.145340 MIRROR_RESYNC 2 mirrors 000468 1786339237.553442 write 0x03511e thru 0x03d368 (0x824b bytes) 000469 1786339237.653047 MIRROR_RESYNC 2 mirrors 000470 1786339238.410739 mapwrite 0x01c0a9 thru 0x01e6ae (0x2606 bytes) 000471 1786339238.539890 trunc from 0x0377c8 to 0x039a9f (0x22d8 bytes) 000472 1786339238.550362 trunc from 0x01011a to 0x01b1fe (0xb0e5 bytes) 000473 1786339238.560325 mapread 0x017b90 thru 0x019b40 (0x1fb1 bytes) 000474 1786339238.579653 write 0x003e6b thru 0x013848 (0xf9de bytes) 000475 1786339238.599776 MIRROR_RESYNC 2 mirrors 000476 1786339239.881796 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000477 1786339240.070778 punch from 0x0190e6 to 0x01a2c8 (0x11e3 bytes) 000478 1786339240.092631 trunc from 0x00bdcb to 0x01432c (0x8562 bytes) 000479 1786339240.103302 write 0x004084 thru 0x008ac4 (0x4a41 bytes) 000480 1786339240.141139 trunc from 0x040000 to 0x01bba5 (0x2445a bytes) 000483 1786339240.170959 fallocate 0x0114be thru 0x01b9b4 (0xa4f7 bytes) 000485 1786339240.201099 mapread 0x0029bd thru 0x00bd4e (0x9392 bytes) 000486 1786339240.227079 write 0x00672a thru 0x00ae95 (0x476c bytes) 000487 1786339240.233216 mapread 0x00a9bc thru 0x016d69 (0xc3ae bytes) 000489 1786339240.267729 fallocate 0x026bae thru 0x02cb81 (0x5fd4 bytes) 000490 1786339240.277963 mapwrite 0x03d5dc thru 0x03ffff (0x2a24 bytes) 000491 1786339240.346338 mapwrite 0x00ab05 thru 0x00b0a6 (0x5a2 bytes) 000492 1786339240.362290 mapread 0x02d4b4 thru 0x039a29 (0xc576 bytes) 000493 1786339240.378463 mapread 0x01dca0 thru 0x02ab32 (0xce93 bytes) 000494 1786339240.392722 read 0x0012ab thru 0x00dc13 (0xc969 bytes) 000496 1786339240.408114 mapwrite 0x038abb thru 0x03b84c (0x2d92 bytes) 000497 1786339240.423590 trunc from 0x03ff6e to 0x03ffff (0x92 bytes) 000498 1786339240.433555 MIRROR_EXTEND to 2 mirrors 000499 1786339241.113211 write 0x013fd2 thru 0x0170c7 (0x30f6 bytes) 000500 1786339241.196027 trunc from 0x024b1a to 0x02a73f (0x5c26 bytes) 000501 1786339241.205577 MIRROR_RESYNC 2 mirrors 000502 1786339242.302085 MIRROR_EXTEND to 3 mirrors 000503 1786339243.059344 read 0x01cce8 thru 0x01dff0 (0x1309 bytes) 000504 1786339243.072204 mapread 0x03e01d thru 0x03ffff (0x1fe3 bytes) 000505 1786339243.091793 trunc from 0x040000 to 0x02f8a3 (0x1075c bytes) 000506 1786339243.151641 fallocate 0x00cf32 thru 0x00ede2 (0x1eb1 bytes) 000507 1786339243.159421 trunc from 0x02f8a4 to 0x0158b6 (0x19fed bytes) 000509 1786339243.187910 write 0x0255e0 thru 0x02db9b (0x85bc bytes) 000510 1786339243.196652 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2db9c): Mirror 3: 0xa360389d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2db9c): Mirror 3: 0xa360389d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2db9c: skipped 000511 1786339243.684586 mapwrite 0x01ff81 thru 0x02ada8 (0xae28 bytes) 000512 1786339243.794486 mapread 0x00e5e7 thru 0x00fe14 (0x182e bytes) 000513 1786339243.828523 mapread 0x0286a4 thru 0x02db9b (0x54f8 bytes) 000514 1786339243.830701 mapread 0x00384b thru 0x00547a (0x1c30 bytes) 000515 1786339243.857586 fallocate 0x00e014 thru 0x00fcc4 (0x1cb1 bytes) 000516 1786339243.868064 read 0x022443 thru 0x024508 (0x20c6 bytes) 000517 1786339243.873642 write 0x036991 thru 0x03cd8c (0x63fc bytes) 000518 1786339243.880628 write 0x0241d8 thru 0x0316ff (0xd528 bytes) 000519 1786339243.909062 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000522 1786339244.148153 trunc from 0x0018c9 to 0x00b105 (0x983d bytes) 000523 1786339244.169241 read 0x032326 thru 0x032d54 (0xa2f bytes) 000524 1786339244.196872 MIRROR_EXTEND to 2 mirrors 000525 1786339245.203039 fallocate 0x00668a thru 0x01377d (0xd0f4 bytes) 000526 1786339245.248839 trunc from 0x03cd8d to 0x036a52 (0x633a bytes) 000527 1786339245.293330 mapread 0x00ae98 thru 0x018153 (0xd2bc bytes) 000528 1786339245.337541 punch from 0x00b0e5 to 0x00db43 (0x2a5f bytes) 000529 1786339245.347230 trunc from 0x02f133 to 0x036a52 (0x7920 bytes) 000530 1786339245.354533 mapread 0x035f16 thru 0x036a52 (0xb3d bytes) 000531 1786339245.391586 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000532 1786339245.576455 read 0x013990 thru 0x01805f (0x46d0 bytes) 000534 1786339245.595575 fallocate 0x02c13f thru 0x036e15 (0xacd7 bytes) 000535 1786339245.609258 write 0x015a69 thru 0x02412a (0xe6c2 bytes) 000536 1786339245.633084 read 0x023837 thru 0x0249d5 (0x119f bytes) 000537 1786339245.642083 fallocate 0x03b050 thru 0x03ffff (0x4fb0 bytes) 000538 1786339245.666306 punch from 0x008fff to 0x00aff3 (0x1ff5 bytes) 000540 1786339245.679398 trunc from 0x0239b8 to 0x025b2d (0x2176 bytes) 000541 1786339245.707655 fallocate 0x02d4ec thru 0x03a7ed (0xd302 bytes) 000542 1786339245.724499 punch from 0x033643 to 0x03ffff (0xc9bd bytes) 000543 1786339245.735021 fallocate 0x00c3eb thru 0x012cac (0x68c2 bytes) 000544 1786339245.750715 mapread 0x005008 thru 0x0123a7 (0xd3a0 bytes) 000545 1786339245.787183 fallocate 0x03afb4 thru 0x03ffff (0x504c bytes) 000547 1786339245.798424 mapwrite 0x016881 thru 0x01c736 (0x5eb6 bytes) 000548 1786339245.835776 mapwrite 0x03c070 thru 0x03ffff (0x3f90 bytes) 000549 1786339245.858397 write 0x03410e thru 0x0363b8 (0x22ab bytes) 000550 1786339245.859765 mapwrite 0x0359a8 thru 0x03ffff (0xa658 bytes) 000552 1786339245.905332 write 0x0293e4 thru 0x02ec2e (0x584b bytes) 000553 1786339245.910381 trunc from 0x0111c6 to 0x01242d (0x1268 bytes) 000554 1786339245.918423 read 0x0164ca thru 0x020a2a (0xa561 bytes) 000555 1786339245.921581 trunc from 0x040000 to 0x0196e3 (0x2691c bytes) 000556 1786339245.959517 trunc from 0x011ac0 to 0x0196e3 (0x7c24 bytes) 000557 1786339245.975230 MIRROR_EXTEND to 2 mirrors 000559 1786339246.874145 mapread 0x006fdf thru 0x00d1c0 (0x61e2 bytes) 000560 1786339246.920108 trunc from 0x0196e4 to 0x002a4d (0x16c96 bytes) 000561 1786339246.977592 mapread 0x0025a7 thru 0x002a4d (0x4a7 bytes) 000562 1786339246.988337 trunc from 0x001565 to 0x002a4d (0x14e9 bytes) 000563 1786339246.994186 mapread 0x00179f thru 0x002a4d (0x12af bytes) 000564 1786339247.005108 trunc from 0x002a4e to 0x0324f9 (0x2faac bytes) 000565 1786339247.030140 trunc from 0x0324fa to 0x03ed67 (0xc86e bytes) 000566 1786339247.052937 fallocate 0x03bc76 thru 0x03e450 (0x27db bytes) 000567 1786339247.059297 mapwrite 0x03edf2 thru 0x03ffff (0x120e bytes) 000568 1786339247.107479 trunc from 0x00ce8b to 0x01826f (0xb3e5 bytes) 000570 1786339247.116578 read 0x02f716 thru 0x03d0b0 (0xd99b bytes) 000571 1786339247.135248 fallocate 0x005627 thru 0x0119fe (0xc3d8 bytes) 000572 1786339247.142143 mapwrite 0x017f9a thru 0x026794 (0xe7fb bytes) 000573 1786339247.234274 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000574 1786339247.459615 write 0x028495 thru 0x030d58 (0x88c4 bytes) 000575 1786339247.484893 trunc from 0x040000 to 0x0398b8 (0x6747 bytes) 000576 1786339247.508492 mapread 0x02e438 thru 0x0398b8 (0xb481 bytes) 000577 1786339247.533392 trunc from 0x0398b9 to 0x011895 (0x28023 bytes) 000578 1786339247.561486 mapwrite 0x0236f0 thru 0x026b71 (0x3482 bytes) 000580 1786339247.605914 mapwrite 0x016b75 thru 0x024364 (0xd7f0 bytes) 000581 1786339247.643327 mapwrite 0x0011b0 thru 0x0106bb (0xf50c bytes) 000582 1786339247.674289 fallocate 0x01909e thru 0x01e1ee (0x5151 bytes) 000583 1786339247.679563 punch from 0x01a66e to 0x026b71 (0xc504 bytes) 000584 1786339247.685400 trunc from 0x00f010 to 0x01943a (0xa42b bytes) 000585 1786339247.691321 punch from 0x0255d2 to 0x025ea5 (0x8d4 bytes) 000587 1786339247.696855 trunc from 0x026b72 to 0x020be1 (0x5f90 bytes) 000588 1786339247.714933 mapread 0x017abe thru 0x020be1 (0x9124 bytes) 000589 1786339247.734199 write 0x03a19b thru 0x03ffff (0x5e65 bytes) 000590 1786339247.737735 mapwrite 0x00cc0d thru 0x01682b (0x9c1f bytes) 000591 1786339247.773395 fallocate 0x0079b8 thru 0x00d44c (0x5a95 bytes) 000592 1786339247.780476 trunc from 0x040000 to 0x03b580 (0x4a7f bytes) 000594 1786339247.800557 trunc from 0x002aaf to 0x00c53c (0x9a8e bytes) 000595 1786339247.807139 read 0x02d753 thru 0x034e5d (0x770b bytes) 000598 1786339247.822540 read 0x029652 thru 0x02c53f (0x2eee bytes) 000599 1786339247.832638 trunc from 0x03b581 to 0x0332ad (0x82d3 bytes) 000600 1786339247.852163 write 0x0178e6 thru 0x01f4e0 (0x7bfb bytes) 000601 1786339247.860650 MIRROR_EXTEND to 2 mirrors 000602 1786339249.052393 MIRROR_RESYNC 2 mirrors 000603 1786339249.616589 mapread 0x010c30 thru 0x01bf05 (0xb2d6 bytes) 000604 1786339249.666682 MIRROR_RESYNC 2 mirrors 000605 1786339250.272226 punch from 0x02f344 to 0x030edb (0x1b98 bytes) 000606 1786339250.315443 punch from 0x02c1a6 to 0x0332ad (0x7108 bytes) 000607 1786339250.327084 MIRROR_RESYNC 2 mirrors 000608 1786339251.400339 trunc from 0x0332ae to 0x00f326 (0x23f87 bytes) 000609 1786339251.452315 fallocate 0x0080bc thru 0x00b1d4 (0x3119 bytes) 000611 1786339251.465588 write 0x02fbf0 thru 0x039543 (0x9954 bytes) 000612 1786339251.470216 mapread 0x02aa32 thru 0x0390bd (0xe68c bytes) 000613 1786339251.501561 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000614 1786339251.687916 read 0x01a41c thru 0x028a89 (0xe66e bytes) 000615 1786339251.706688 trunc from 0x0190db to 0x019e74 (0xd9a bytes) 000616 1786339251.712494 write 0x01b640 thru 0x02b362 (0xfd23 bytes) 000617 1786339251.724672 fallocate 0x002778 thru 0x007d4b (0x55d4 bytes) 000619 1786339251.732873 write 0x02839d thru 0x036dde (0xea42 bytes) 000620 1786339251.739796 punch from 0x029a0c to 0x02c0d4 (0x26c9 bytes) 000622 1786339251.766086 read 0x01d38d thru 0x022df4 (0x5a68 bytes) 000624 1786339251.767274 mapwrite 0x00d207 thru 0x00f3de (0x21d8 bytes) 000625 1786339251.794866 mapread 0x014e07 thru 0x01ad41 (0x5f3b bytes) 000626 1786339251.795755 read 0x01a4ef thru 0x028b7b (0xe68d bytes) 000628 1786339251.796722 mapread 0x0367e6 thru 0x039543 (0x2d5e bytes) 000629 1786339251.797889 mapwrite 0x01dc1f thru 0x02bb1f (0xdf01 bytes) 000630 1786339251.827488 mapwrite 0x006fea thru 0x00abfd (0x3c14 bytes) 000631 1786339251.845837 mapread 0x01116b thru 0x019a7b (0x8911 bytes) 000632 1786339251.847230 trunc from 0x004626 to 0x00f18c (0xab67 bytes) 000633 1786339251.853144 punch from 0x008c44 to 0x008c4e (0xb bytes) 000636 1786339251.860032 trunc from 0x017950 to 0x01a99f (0x3050 bytes) 000637 1786339251.865263 punch from 0x02d8b3 to 0x031828 (0x3f76 bytes) 000638 1786339251.872704 write 0x0352e4 thru 0x03b609 (0x6326 bytes) 000639 1786339251.874399 trunc from 0x00aa17 to 0x01537e (0xa968 bytes) 000640 1786339251.880507 fallocate 0x03e27c thru 0x03ffff (0x1d84 bytes) 000642 1786339251.890688 fallocate 0x037194 thru 0x03d9c9 (0x6836 bytes) 000643 1786339251.900551 mapwrite 0x007396 thru 0x009799 (0x2404 bytes) 000644 1786339251.961784 write 0x03e38d thru 0x03ffff (0x1c73 bytes) 000645 1786339251.964081 mapread 0x00844a thru 0x012299 (0x9e50 bytes) 000647 1786339251.973097 fallocate 0x01c73f thru 0x02bef1 (0xf7b3 bytes) 000649 1786339251.993318 trunc from 0x03ac81 to 0x03ffff (0x537f bytes) 000650 1786339252.024299 fallocate 0x02018f thru 0x02c4fe (0xc370 bytes) 000651 1786339252.031996 fallocate 0x02a603 thru 0x035d3b (0xb739 bytes) 000652 1786339252.039571 punch from 0x0046fc to 0x00be14 (0x7719 bytes) 000653 1786339252.049344 read 0x019ee6 thru 0x028606 (0xe721 bytes) 000655 1786339252.052772 mapread 0x00806f thru 0x0122fb (0xa28d bytes) 000656 1786339252.085982 fallocate 0x008f5b thru 0x01616f (0xd215 bytes) 000657 1786339252.096538 trunc from 0x006091 to 0x0086bb (0x262b bytes) 000658 1786339252.103342 fallocate 0x00a42d thru 0x00b84a (0x141e bytes) 000659 1786339252.122574 write 0x01280b thru 0x01dbde (0xb3d4 bytes) 000661 1786339252.129061 punch from 0x03ed1a to 0x03ffff (0x12e6 bytes) 000662 1786339252.144074 fallocate 0x00c285 thru 0x01bebf (0xfc3b bytes) 000663 1786339252.152270 fallocate 0x00e2b9 thru 0x00f45b (0x11a3 bytes) 000665 1786339252.161291 mapread 0x025f16 thru 0x035989 (0xfa74 bytes) 000667 1786339252.166358 mapwrite 0x01eb13 thru 0x0223c4 (0x38b2 bytes) 000669 1786339252.185655 mapread 0x03ce65 thru 0x03ffff (0x319b bytes) 000671 1786339252.200838 fallocate 0x016744 thru 0x01cfa1 (0x685e bytes) 000672 1786339252.205196 trunc from 0x02bc87 to 0x032718 (0x6a92 bytes) 000673 1786339252.210446 mapwrite 0x0233de thru 0x027135 (0x3d58 bytes) 000674 1786339252.227971 fallocate 0x0212f6 thru 0x023925 (0x2630 bytes) 000675 1786339252.233524 fallocate 0x005e23 thru 0x011120 (0xb2fe bytes) 000676 1786339252.248742 write 0x01b71c thru 0x029ef9 (0xe7de bytes) 000677 1786339252.257088 write 0x012462 thru 0x012ec8 (0xa67 bytes) 000678 1786339252.262235 trunc from 0x0348d6 to 0x03a42d (0x5b58 bytes) 000679 1786339252.270514 trunc from 0x011cf8 to 0x01edf6 (0xd0ff bytes) 000680 1786339252.309138 mapread 0x033172 thru 0x039b63 (0x69f2 bytes) 000681 1786339252.349111 write 0x0151e8 thru 0x01c8bd (0x76d6 bytes) 000683 1786339252.366536 MIRROR_EXTEND to 2 mirrors 000684 1786339253.297560 punch from 0x034615 to 0x039f48 (0x5934 bytes) 000685 1786339253.327328 trunc from 0x040000 to 0x02620f (0x19df0 bytes) 000687 1786339253.357762 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000689 1786339253.533166 trunc from 0x007a2f to 0x00cfde (0x55b0 bytes) 000691 1786339253.538893 mapwrite 0x00372b thru 0x009bca (0x64a0 bytes) 000692 1786339253.576110 trunc from 0x009e22 to 0x00e23b (0x441a bytes) 000695 1786339253.584295 mapread 0x01f799 thru 0x02620f (0x6a77 bytes) 000696 1786339253.601470 trunc from 0x026210 to 0x02b297 (0x5088 bytes) 000697 1786339253.627170 read 0x006946 thru 0x0164a5 (0xfb60 bytes) 000699 1786339253.643466 read 0x023576 thru 0x025e35 (0x28c0 bytes) 000700 1786339253.644243 write 0x011450 thru 0x0174c1 (0x6072 bytes) 000701 1786339253.653651 read 0x01d895 thru 0x020792 (0x2efe bytes) 000702 1786339253.654651 trunc from 0x02b298 to 0x03388b (0x85f4 bytes) 000703 1786339253.672282 trunc from 0x03388c to 0x01c863 (0x17028 bytes) 000704 1786339253.691391 trunc from 0x01c864 to 0x004a13 (0x17e50 bytes) 000705 1786339253.726659 write 0x03e3c4 thru 0x03ffff (0x1c3c bytes) 000706 1786339253.728562 trunc from 0x040000 to 0x00a010 (0x35fef bytes) 000707 1786339253.747291 read 0x002b53 thru 0x00a010 (0x74be bytes) 000708 1786339253.758290 mapread 0x00966a thru 0x00a010 (0x9a7 bytes) 000710 1786339253.761438 punch from 0x007f69 to 0x00a010 (0x20a8 bytes) 000711 1786339253.767528 fallocate 0x008539 thru 0x0136d5 (0xb19d bytes) 000712 1786339253.773738 write 0x0049c0 thru 0x00bf58 (0x7599 bytes) 000713 1786339253.789525 fallocate 0x011606 thru 0x0181fc (0x6bf7 bytes) 000714 1786339253.796252 read 0x0019ba thru 0x00d3c1 (0xba08 bytes) 000715 1786339253.797649 read 0x004e63 thru 0x00e600 (0x979e bytes) 000716 1786339253.800026 trunc from 0x0181fd to 0x03ebef (0x269f3 bytes) 000717 1786339253.816154 punch from 0x00b013 to 0x0199f9 (0xe9e7 bytes) 000718 1786339253.834528 trunc from 0x03ebf0 to 0x03bfd4 (0x2c1b bytes) 000719 1786339253.851607 read 0x035418 thru 0x03bfd4 (0x6bbd bytes) 000720 1786339253.866562 read 0x014868 thru 0x01d4d6 (0x8c6f bytes) 000721 1786339253.867521 punch from 0x039523 to 0x039bcc (0x6aa bytes) 000722 1786339253.873330 fallocate 0x016baf thru 0x016f9a (0x3ec bytes) 000724 1786339253.879562 mapwrite 0x02d67c thru 0x035bc9 (0x854e bytes) 000725 1786339253.898779 punch from 0x033a13 to 0x03bfd4 (0x85c2 bytes) 000726 1786339253.905436 punch from 0x00a96b to 0x016b41 (0xc1d7 bytes) 000727 1786339253.911462 mapwrite 0x00cb43 thru 0x00e683 (0x1b41 bytes) 000728 1786339253.935597 trunc from 0x03bfd5 to 0x033667 (0x896d bytes) 000730 1786339253.961424 mapread 0x0219ae thru 0x029d6a (0x83bd bytes) 000731 1786339253.965338 fallocate 0x004401 thru 0x00fc9b (0xb89b bytes) 000732 1786339253.979322 trunc from 0x033668 to 0x03ba7e (0x8417 bytes) 000734 1786339254.009981 write 0x01f3d9 thru 0x02937b (0x9fa3 bytes) 000735 1786339254.012763 punch from 0x03ade1 to 0x03ba7e (0xc9e bytes) 000736 1786339254.018284 mapread 0x0068c7 thru 0x00e30a (0x7a44 bytes) 000737 1786339254.020972 trunc from 0x03ba7f to 0x012ce9 (0x28d95 bytes) 000738 1786339254.043580 mapwrite 0x034351 thru 0x03729b (0x2f4b bytes) 000739 1786339254.163386 trunc from 0x033181 to 0x03414c (0xfcc bytes) 000741 1786339254.177470 trunc from 0x03729c to 0x013394 (0x23f07 bytes) 000742 1786339254.203072 fallocate 0x025711 thru 0x031751 (0xc041 bytes) 000743 1786339254.209409 mapwrite 0x027bd6 thru 0x02e87c (0x6ca7 bytes) 000744 1786339254.309360 mapread 0x023ccc thru 0x031751 (0xda86 bytes) 000745 1786339254.349065 write 0x00de50 thru 0x0166d0 (0x8881 bytes) 000746 1786339254.366039 fallocate 0x00e795 thru 0x01969d (0xaf09 bytes) 000748 1786339254.375301 fallocate 0x005313 thru 0x011055 (0xbd43 bytes) 000749 1786339254.380382 read 0x02e8a7 thru 0x031751 (0x2eab bytes) 000750 1786339254.381202 trunc from 0x01886b to 0x019ba7 (0x133d bytes) 000752 1786339254.388427 trunc from 0x01e8c4 to 0x027833 (0x8f70 bytes) 000753 1786339254.395524 punch from 0x02ecfb to 0x02ef1c (0x222 bytes) 000754 1786339254.403394 read 0x02f978 thru 0x031751 (0x1dda bytes) 000755 1786339254.405232 punch from 0x02dc9e to 0x031751 (0x3ab4 bytes) 000756 1786339254.410980 punch from 0x029619 to 0x031751 (0x8139 bytes) 000757 1786339254.422674 write 0x02fa28 thru 0x03a432 (0xaa0b bytes) 000758 1786339254.459671 write 0x01b9ed thru 0x0297d1 (0xdde5 bytes) 000759 1786339254.472321 fallocate 0x02489a thru 0x029703 (0x4e6a bytes) 000760 1786339254.506850 trunc from 0x00baeb to 0x015eb0 (0xa3c6 bytes) 000762 1786339254.595465 punch from 0x005627 to 0x0119b0 (0xc38a bytes) 000763 1786339254.616009 write 0x030726 thru 0x03a4f6 (0x9dd1 bytes) 000764 1786339254.618606 punch from 0x0065e3 to 0x006cd2 (0x6f0 bytes) 000765 1786339254.630556 write 0x03f460 thru 0x03ffff (0xba0 bytes) 000767 1786339254.632660 punch from 0x031d77 to 0x0362e6 (0x4570 bytes) 000768 1786339254.663175 write 0x03b39f thru 0x03ffff (0x4c61 bytes) 000769 1786339254.696664 write 0x0278f6 thru 0x02cadf (0x51ea bytes) 000770 1786339254.701574 mapread 0x03a0a8 thru 0x03feab (0x5e04 bytes) 000772 1786339254.707159 fallocate 0x00665b thru 0x015c35 (0xf5db bytes) 000773 1786339254.713507 mapread 0x00a013 thru 0x018b07 (0xeaf5 bytes) 000774 1786339254.801507 read 0x006bf4 thru 0x00d1d9 (0x65e6 bytes) 000775 1786339254.811129 write 0x0124cd thru 0x01614a (0x3c7e bytes) 000776 1786339254.812621 trunc from 0x02af8e to 0x03acd6 (0xfd49 bytes) 000778 1786339254.830979 mapread 0x01f085 thru 0x0246cf (0x564b bytes) 000779 1786339254.832319 punch from 0x023602 to 0x02fb75 (0xc574 bytes) 000780 1786339254.838692 trunc from 0x040000 to 0x01d123 (0x22edc bytes) 000781 1786339254.856605 write 0x030bf2 thru 0x0396e7 (0x8af6 bytes) 000784 1786339254.861438 trunc from 0x0396e8 to 0x0161da (0x2350d bytes) 000785 1786339254.875437 trunc from 0x0112e9 to 0x01488a (0x35a2 bytes) 000786 1786339254.882087 trunc from 0x0161db to 0x03034f (0x1a175 bytes) 000787 1786339254.905591 fallocate 0x02851b thru 0x02cbf0 (0x46d6 bytes) 000788 1786339254.912037 punch from 0x00cfec to 0x0118ec (0x4901 bytes) 000790 1786339254.918735 mapread 0x01225f thru 0x013a74 (0x1816 bytes) 000791 1786339254.932482 read 0x00dcc8 thru 0x0140a0 (0x63d9 bytes) 000792 1786339254.945074 mapwrite 0x006c2b thru 0x00840f (0x17e5 bytes) 000793 1786339254.959747 punch from 0x022b7d to 0x028b26 (0x5faa bytes) 000794 1786339254.967414 MIRROR_EXTEND to 2 mirrors 000796 1786339255.752356 mapread 0x0250b5 thru 0x03034f (0xb29b bytes) 000797 1786339255.799089 mapwrite 0x017e85 thru 0x0183ab (0x527 bytes) 000798 1786339255.892775 mapread 0x01359a thru 0x017088 (0x3aef bytes) 000799 1786339255.901293 MIRROR_RESYNC 2 mirrors 000800 1786339256.713173 mapread 0x00ca98 thru 0x011d10 (0x5279 bytes) 000801 1786339256.736146 read 0x000892 thru 0x00a3aa (0x9b19 bytes) 000802 1786339256.743612 write 0x036264 thru 0x03ffff (0x9d9c bytes) 000803 1786339256.771266 read 0x03af2d thru 0x03c712 (0x17e6 bytes) 000804 1786339256.773980 punch from 0x037ae0 to 0x03b091 (0x35b2 bytes) 000806 1786339256.790346 write 0x03c8b2 thru 0x03ffff (0x374e bytes) 000807 1786339256.797378 MIRROR_RESYNC 2 mirrors 000808 1786339257.673503 mapwrite 0x01542f thru 0x01ecb9 (0x988b bytes) 000809 1786339257.872910 trunc from 0x014613 to 0x0209f6 (0xc3e4 bytes) 000810 1786339257.880044 punch from 0x003b5f to 0x00d3fe (0x98a0 bytes) 000811 1786339257.893679 mapread 0x02fbde thru 0x03f7dc (0xfbff bytes) 000812 1786339257.994589 read 0x01460a thru 0x020f64 (0xc95b bytes) 000813 1786339258.011499 punch from 0x00532e to 0x012c29 (0xd8fc bytes) 000814 1786339258.036115 mapread 0x034e59 thru 0x03e942 (0x9aea bytes) 000815 1786339258.037301 mapwrite 0x0076c1 thru 0x0085e2 (0xf22 bytes) 000816 1786339258.114982 read 0x01eee1 thru 0x02af86 (0xc0a6 bytes) 000817 1786339258.175286 mapread 0x015883 thru 0x01f7bb (0x9f39 bytes) 000820 1786339258.180775 fallocate 0x00bdf7 thru 0x016ffe (0xb208 bytes) 000821 1786339258.204464 trunc from 0x033a60 to 0x03fe59 (0xc3fa bytes) 000823 1786339258.225018 trunc from 0x040000 to 0x00fcd1 (0x3032e bytes) 000824 1786339258.272057 punch from 0x00950e to 0x00fcd1 (0x67c4 bytes) 000825 1786339258.284289 trunc from 0x00fcd2 to 0x003470 (0xc861 bytes) 000826 1786339258.328536 trunc from 0x003471 to 0x005df9 (0x2989 bytes) 000827 1786339258.364646 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x5dfa): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5dfa): Mirror 2: 0xf267222a (stale) CRC-32 checksum value for chunk [0, 0x5dfa): Mirror 1: 0x601d4806 Mirror 3: 0x601d4806 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5dfa: skipped 000828 1786339259.294531 punch from 0x002bb0 to 0x005df9 (0x324a bytes) 000829 1786339259.331684 trunc from 0x005dfa to 0x03caea (0x36cf1 bytes) 000830 1786339259.393190 write 0x002184 thru 0x00e0a9 (0xbf26 bytes) 000831 1786339259.406194 fallocate 0x005f98 thru 0x00e0b0 (0x8119 bytes) 000832 1786339259.411741 trunc from 0x03caeb to 0x031889 (0xb261 bytes) 000833 1786339259.429947 write 0x01ef7e thru 0x028db9 (0x9e3c bytes) 000834 1786339259.442635 mapread 0x00315b thru 0x009c8b (0x6b31 bytes) 000836 1786339259.445563 mapwrite 0x038c07 thru 0x03ffff (0x73f9 bytes) 000837 1786339259.507200 mapread 0x0002ac thru 0x0070f3 (0x6e48 bytes) 000838 1786339259.517073 trunc from 0x0162bb to 0x01860f (0x2355 bytes) 000839 1786339259.530771 MIRROR_RESYNC 3 mirrors 000840 1786339260.321204 MIRROR_RESYNC 3 mirrors 000841 1786339260.655894 mapread 0x00b19d thru 0x011076 (0x5eda bytes) 000842 1786339260.679967 MIRROR_RESYNC 3 mirrors 000843 1786339260.981616 read 0x00d341 thru 0x01949a (0xc15a bytes) 000845 1786339260.989615 write 0x0236db thru 0x0258de (0x2204 bytes) 000846 1786339261.045851 MIRROR_RESYNC 3 mirrors 000847 1786339261.798445 mapread 0x018278 thru 0x025dd8 (0xdb61 bytes) 000848 1786339261.829410 write 0x01abd6 thru 0x020bb5 (0x5fe0 bytes) 000849 1786339261.878051 mapread 0x027c8f thru 0x02b1c2 (0x3534 bytes) 000850 1786339261.890595 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf372559b (stale) Mirror 3: 0xf372559b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb4055a0a Mirror 4: 0xb4055a0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000852 1786339262.860551 punch from 0x026b34 to 0x02a0b4 (0x3581 bytes) 000853 1786339262.902577 trunc from 0x040000 to 0x029266 (0x16d99 bytes) 000854 1786339262.927942 trunc from 0x00c6a1 to 0x017446 (0xada6 bytes) 000855 1786339262.933577 trunc from 0x00e4a5 to 0x019f5f (0xbabb bytes) 000856 1786339262.942779 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x29267): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29267): Mirror 2: 0xab2d43de (stale) Mirror 3: 0xab2d43de (stale) Mirror 4: 0x91f78bbc (stale) CRC-32 checksum value for chunk [0, 0x29267): Mirror 1: 0xeb3fe817 Mirror 5: 0xeb3fe817 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29267: skipped 000857 1786339263.745451 mapread 0x001c13 thru 0x010830 (0xec1e bytes) 000858 1786339263.782710 punch from 0x009405 to 0x0127ec (0x93e8 bytes) 000859 1786339263.822877 trunc from 0x029267 to 0x0102e1 (0x18f85 bytes) 000860 1786339263.867308 trunc from 0x003243 to 0x0102e1 (0xd09f bytes) 000861 1786339263.878144 trunc from 0x001847 to 0x00f21c (0xd9d6 bytes) 000862 1786339263.883441 trunc from 0x0102e2 to 0x0187a1 (0x84c0 bytes) 000863 1786339263.904978 write 0x002a95 thru 0x007807 (0x4d73 bytes) 000864 1786339263.917918 MIRROR_RESYNC 5 mirrors 000865 1786339264.781229 fallocate 0x0255d1 thru 0x031305 (0xbd35 bytes) 000866 1786339264.825506 read 0x00a917 thru 0x00ac73 (0x35d bytes) 000867 1786339264.835080 fallocate 0x011b89 thru 0x01e799 (0xcc11 bytes) 000868 1786339264.841436 punch from 0x00f892 to 0x013265 (0x39d4 bytes) 000869 1786339264.851711 fallocate 0x021105 thru 0x025bed (0x4ae9 bytes) 000870 1786339264.857492 trunc from 0x031306 to 0x01fab7 (0x1184e bytes) 000871 1786339264.897525 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fab8): Mirror 3: 0x74be1e07 (stale) Mirror 4: 0x74be1e07 (stale) Mirror 5: 0x74be1e07 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fab8): Mirror 3: 0x74be1e07 (stale) Mirror 4: 0x74be1e07 (stale) Mirror 5: 0x74be1e07 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fab8: skipped 000872 1786339265.353337 mapread 0x0056c4 thru 0x010194 (0xaad1 bytes) 000873 1786339265.380450 mapwrite 0x03b696 thru 0x03ffff (0x496a bytes) 000875 1786339265.439348 read 0x0261dc thru 0x032a03 (0xc828 bytes) 000877 1786339265.450303 trunc from 0x031f03 to 0x037672 (0x5770 bytes) 000878 1786339265.456040 write 0x03f497 thru 0x03ffff (0xb69 bytes) 000879 1786339265.457326 write 0x03c16b thru 0x03ffff (0x3e95 bytes) 000880 1786339265.460340 write 0x01b355 thru 0x02232d (0x6fd9 bytes) 000881 1786339265.470405 trunc from 0x032b4a to 0x034956 (0x1e0d bytes) 000882 1786339265.475239 read 0x02d5d7 thru 0x03201e (0x4a48 bytes) 000884 1786339265.487469 mapread 0x015b03 thru 0x01aeb0 (0x53ae bytes) 000885 1786339265.488873 punch from 0x03ff73 to 0x03ffff (0x8d bytes) 000886 1786339265.511672 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9b94d653 (stale) Mirror 5: 0x9b94d653 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9b94d653 (stale) Mirror 5: 0x9b94d653 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000887 1786339265.785973 write 0x0157b8 thru 0x015901 (0x14a bytes) 000889 1786339265.797036 write 0x02f08c thru 0x0384bb (0x9430 bytes) 000890 1786339265.806942 write 0x0322a9 thru 0x03ffff (0xdd57 bytes) 000891 1786339265.813469 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9b94d653 (stale) Mirror 5: 0x9b94d653 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6f54450b Mirror 6: 0x6f54450b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000892 1786339266.676301 mapread 0x03867f thru 0x03ffff (0x7981 bytes) 000893 1786339266.710091 trunc from 0x040000 to 0x037740 (0x88bf bytes) 000894 1786339266.771373 write 0x02c06e thru 0x031130 (0x50c3 bytes) 000896 1786339266.784046 mapread 0x006960 thru 0x015c8b (0xf32c bytes) 000898 1786339266.817066 read 0x026119 thru 0x02eaa9 (0x8991 bytes) 000899 1786339266.826940 mapwrite 0x000783 thru 0x008bf5 (0x8473 bytes) 000900 1786339266.857246 MIRROR_RESYNC 4 mirrors 000901 1786339267.767040 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000902 1786339268.109643 read 0x032a87 thru 0x037740 (0x4cba bytes) 000903 1786339268.120185 mapread 0x02c206 thru 0x0301cf (0x3fca bytes) 000904 1786339268.146202 MIRROR_RESYNC 3 mirrors 000905 1786339268.591413 mapread 0x024992 thru 0x03428b (0xf8fa bytes) 000906 1786339268.617657 fallocate 0x0072a5 thru 0x00dcbf (0x6a1b bytes) 000907 1786339268.670916 fallocate 0x00a506 thru 0x00c337 (0x1e32 bytes) 000908 1786339268.677279 write 0x027810 thru 0x02c29b (0x4a8c bytes) 000909 1786339268.692919 punch from 0x011726 to 0x01c2fe (0xabd9 bytes) 000910 1786339268.703075 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37741): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37741): Mirror 5: 0xd7e3d59d (stale) Mirror 6: 0xd7e3d59d (stale) CRC-32 checksum value for chunk [0, 0x37741): Mirror 1: 0xe9ba85b4 Mirror 7: 0xe9ba85b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37741: skipped 000913 1786339269.790123 fallocate 0x03c171 thru 0x03c806 (0x696 bytes) 000914 1786339269.830914 trunc from 0x037741 to 0x016f85 (0x207bb bytes) 000915 1786339269.857133 mapread 0x0060f0 thru 0x00a0cf (0x3fe0 bytes) 000916 1786339269.881780 fallocate 0x01937e thru 0x019e8f (0xb12 bytes) 000917 1786339269.887639 mapwrite 0x02b464 thru 0x02d710 (0x22ad bytes) 000918 1786339269.935390 fallocate 0x039951 thru 0x03adcb (0x147b bytes) 000919 1786339269.942133 mapwrite 0x01c753 thru 0x02a92c (0xe1da bytes) 000920 1786339269.977254 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d711): Mirror 5: 0x6666ccf0 (stale) Mirror 6: 0x6666ccf0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d711): Mirror 5: 0x6666ccf0 (stale) Mirror 6: 0x6666ccf0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d711: skipped 000921 1786339270.279130 write 0x002b93 thru 0x00e56b (0xb9d9 bytes) 000922 1786339270.293420 trunc from 0x024a79 to 0x02571f (0xca7 bytes) 000923 1786339270.299950 punch from 0x02560d to 0x02d710 (0x8104 bytes) 000924 1786339270.305112 punch from 0x01dfb0 to 0x026d43 (0x8d94 bytes) 000925 1786339270.311435 trunc from 0x018e0e to 0x02823f (0xf432 bytes) 000926 1786339270.316345 mapwrite 0x03b13d thru 0x03ffff (0x4ec3 bytes) 000927 1786339270.367939 fallocate 0x01ad74 thru 0x01f1a3 (0x4430 bytes) 000928 1786339270.374614 fallocate 0x020f82 thru 0x026e5b (0x5eda bytes) 000929 1786339270.382421 write 0x01027e thru 0x01164c (0x13cf bytes) 000930 1786339270.393001 fallocate 0x0083fe thru 0x00ab67 (0x276a bytes) 000931 1786339270.403157 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x189ad32d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x189ad32d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000933 1786339270.680399 mapwrite 0x01d17c thru 0x02c217 (0xf09c bytes) 000934 1786339270.738459 write 0x03badc thru 0x03f40e (0x3933 bytes) 000935 1786339270.752791 mapwrite 0x01bd32 thru 0x029d20 (0xdfef bytes) 000936 1786339270.778052 MIRROR_RESYNC 2 mirrors 000937 1786339271.403057 MIRROR_EXTEND to 3 mirrors 000938 1786339272.026348 mapread 0x00f5de thru 0x0123ac (0x2dcf bytes) 000939 1786339272.051012 read 0x023b72 thru 0x02f7b6 (0xbc45 bytes) 000940 1786339272.061115 read 0x03c548 thru 0x03ffff (0x3ab8 bytes) 000941 1786339272.078569 MIRROR_RESYNC 3 mirrors 000942 1786339272.476346 trunc from 0x038564 to 0x03ffff (0x7a9c bytes) 000943 1786339272.527487 read 0x006bf3 thru 0x00d17a (0x6588 bytes) 000944 1786339272.537470 trunc from 0x040000 to 0x03a485 (0x5b7a bytes) 000945 1786339272.567305 mapread 0x00798e thru 0x014030 (0xc6a3 bytes) 000946 1786339272.587585 write 0x01fd8b thru 0x02aa4a (0xacc0 bytes) 000947 1786339272.597887 mapwrite 0x009d7a thru 0x011b04 (0x7d8b bytes) 000948 1786339272.621238 mapwrite 0x0350a2 thru 0x03ffff (0xaf5e bytes) 000949 1786339272.697834 trunc from 0x00982e to 0x019649 (0xfe1c bytes) 000950 1786339272.706509 read 0x007a13 thru 0x016d5e (0xf34c bytes) 000951 1786339272.723255 MIRROR_RESYNC 3 mirrors 000952 1786339273.696400 fallocate 0x01a493 thru 0x01b855 (0x13c3 bytes) 000953 1786339273.740221 mapread 0x004576 thru 0x011384 (0xce0f bytes) 000954 1786339273.770784 mapread 0x03d8cd thru 0x03ffff (0x2733 bytes) 000956 1786339273.782398 punch from 0x0179fd to 0x022b44 (0xb148 bytes) 000957 1786339273.789898 trunc from 0x040000 to 0x003c3e (0x3c3c1 bytes) 000958 1786339273.819124 MIRROR_RESYNC 3 mirrors 000959 1786339274.333504 mapwrite 0x021135 thru 0x0249fa (0x38c6 bytes) 000960 1786339274.409396 trunc from 0x0249fb to 0x03eea4 (0x1a4aa bytes) 000962 1786339274.429071 trunc from 0x03eea5 to 0x015539 (0x2996b bytes) 000963 1786339274.448906 trunc from 0x004ba5 to 0x013de9 (0xf245 bytes) 000964 1786339274.455025 punch from 0x006759 to 0x00ecfa (0x85a2 bytes) 000965 1786339274.459969 punch from 0x00ce5d to 0x011ca3 (0x4e47 bytes) 000966 1786339274.465219 trunc from 0x014d73 to 0x015539 (0x7c7 bytes) 000967 1786339274.470752 mapwrite 0x0050c8 thru 0x00949e (0x43d7 bytes) 000968 1786339274.497959 fallocate 0x025be2 thru 0x02d510 (0x792f bytes) 000969 1786339274.504405 mapwrite 0x0104c4 thru 0x01f451 (0xef8e bytes) 000970 1786339274.556623 mapwrite 0x01d3aa thru 0x023b28 (0x677f bytes) 000971 1786339274.597230 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d511): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d511): Mirror 6: 0x2b5d8967 (stale) Mirror 7: 0x2b5d8967 (stale) CRC-32 checksum value for chunk [0, 0x2d511): Mirror 1: 0xca1c1b34 Mirror 8: 0xca1c1b34 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d511: skipped 000972 1786339275.478029 mapwrite 0x0053c3 thru 0x014217 (0xee55 bytes) 000973 1786339275.625951 punch from 0x00a819 to 0x00d4b1 (0x2c99 bytes) 000974 1786339275.639182 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2d511): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d511): Mirror 6: 0x2b5d8967 (stale) Mirror 7: 0x2b5d8967 (stale) Mirror 8: 0xca1c1b34 (stale) CRC-32 checksum value for chunk [0, 0x2d511): Mirror 1: 0x8b60cba3 Mirror 9: 0x8b60cba3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d511: skipped 000975 1786339276.673404 trunc from 0x02d511 to 0x0208b4 (0xcc5c bytes) 000976 1786339276.746093 punch from 0x02086e to 0x0208b4 (0x47 bytes) 000977 1786339276.754225 trunc from 0x0208b5 to 0x0307e9 (0xff35 bytes) 000978 1786339276.788053 trunc from 0x0307ea to 0x01823c (0x185ad bytes) 000979 1786339276.811861 mapread 0x006296 thru 0x01590c (0xf677 bytes) 000980 1786339276.865281 punch from 0x01271a to 0x012eeb (0x7d2 bytes) 000981 1786339276.871224 trunc from 0x014aa9 to 0x01823c (0x3794 bytes) 000982 1786339276.877422 fallocate 0x00be09 thru 0x0108bd (0x4ab5 bytes) 000983 1786339276.883204 mapread 0x00d6aa thru 0x017b1b (0xa472 bytes) 000984 1786339276.891274 write 0x033ae4 thru 0x0394d3 (0x59f0 bytes) 000985 1786339276.892658 punch from 0x0017da to 0x00e12a (0xc951 bytes) 000986 1786339276.899953 mapwrite 0x01eba5 thru 0x01fbd3 (0x102f bytes) 000987 1786339276.928594 trunc from 0x0394d4 to 0x004750 (0x34d83 bytes) 000988 1786339276.956654 read 0x003ca7 thru 0x004750 (0xaaa bytes) 000989 1786339276.965744 read 0x000ad0 thru 0x002c34 (0x2165 bytes) 000990 1786339276.981723 MIRROR_RESYNC 5 mirrors 000991 1786339277.865509 punch from 0x0040d4 to 0x004750 (0x67d bytes) 000992 1786339277.901443 fallocate 0x01abc8 thru 0x01e1cf (0x3608 bytes) 000993 1786339277.908023 trunc from 0x01e1d0 to 0x0362bd (0x180ee bytes) 000994 1786339277.940036 trunc from 0x01efdf to 0x02665a (0x767c bytes) 000995 1786339277.947352 write 0x0205f5 thru 0x02320d (0x2c19 bytes) 000997 1786339277.959387 mapread 0x01fc08 thru 0x029fae (0xa3a7 bytes) 000998 1786339277.984436 trunc from 0x00bc03 to 0x011c19 (0x6017 bytes) 000999 1786339277.990071 write 0x0362a4 thru 0x03ffff (0x9d5c bytes) 001000 1786339277.997189 read 0x008dd4 thru 0x01725e (0xe48b bytes) All operations completed A-OK! PASS 70a (78s) == sanity-flr test 71: check mirror extend parent fid ==== 01:21:24 (1786339284) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x7a:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 01:21:34 (1786339294) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x298:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182842 s, 11.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x298:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x299:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927 s, 11.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x298:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x299:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x298:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x299:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x29a:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 01:21:44 (1786339304) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3717440 .. Extending file size to 4497024 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9136992 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7433888 .. Extending file size to 7724032 .. Extending file size to 8562400 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8651360 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9309120 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61256 61257 61258 61261 61262 done 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) 192.168.202.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (89s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:23:13 (1786339393) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5265216 .. Extending file size to 3147008 .. done Extending file size to 7082208 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5978624 .. Extending file size to 8484512 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6609152 .. failed Extending file size to 7332064 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8614208 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7505888 .. Extending file size to 8789792 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8780512 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62787 62788 62789 62790 62793 failed 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) 192.168.202.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:24:31 (1786339471) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1944 1269360 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18196 3588824 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 23528 3583492 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 41724 7172316 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1944 1269360 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18196 3588824 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 23528 3583492 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 41724 7172316 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1224 s, 17.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.233361 s, 9.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x22d2622e (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:24:55 (1786339495) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125941 s, 16.7 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 01:25:04 (1786339504) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x189:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cd:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cd:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2af:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:25:12 (1786339512) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0920425 s, 22.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:25:21 (1786339521) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157137 s, 13.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b3:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:25:31 (1786339531) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0940934 s, 22.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b5:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:26:06 (1786339566) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg210-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077506 s, 13.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:27:00 (1786339620) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg210-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1557 s, 13.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:27:54 (1786339674) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg210-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg210-server: Pool lustre.test_204f created Adding targets to pool oleg210-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg210-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130916 s, 16.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg210-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg210-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg210-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:29:10 (1786339750) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:29:19 (1786339759) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:29:29 (1786339769) oleg210-server: Pool lustre.test_206 created Adding targets to pool oleg210-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg210-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg210-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg210-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg210-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:29:57 (1786339797) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535514 s, 19.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483081 s, 21.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe6f5aa77 Mirror 3: 0xe6f5aa77 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:30:07 (1786339807) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:30:10 (1786339810) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:30:14 (1786339814) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0632581 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1320000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1c10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1c10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1c10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xc30000 Read this (4 bytes): BBBB 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:30:50 (1786339850) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0339695 s, 241 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:30:57 (1786339857) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680752 s, 15.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1ac lmm_fid: [0x200000405:0x1ac:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1ac lmm_fid: [0x200000405:0x1ac:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c1:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611564 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 01:30:59.000000000 -0400 Modify: 2026-08-10 01:30:59.000000000 -0400 Change: 2026-08-10 01:31:00.000000000 -0400 Birth: 2026-08-10 01:30:59.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:31:05 (1786339865) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438845 s, 23.9 MB/s oleg210-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg210-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg210-server: O/280000401/d0/2e0: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:31:27 (1786339887) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.253306 s, 162 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0248952 s, 165 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 10 01:31 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b1:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 10 01:31 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:31:36 (1786339896) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:31:44 (1786339904) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1950 sec =========== 01:31:52 (1786339912) === sanity-flr: start cleanup 01:31:54 (1786339914) === === sanity-flr: finish cleanup 01:32:02 (1786339922) ===