-----============= acceptance-small: sanity-flr ============----- Mon Aug 24 19:27:28 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg134-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 19:27:50 (1787614070) === oleg134-client.virtnet: executing check_config_client /mnt/lustre oleg134-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg134-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9443d8f05000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9443d8f05000.idle_timeout=debug disable quota as required oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 19:28:34 (1787614114) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8250] == sanity-flr test 0a: lfs mirror create with -N option == 19:28:36 (1787614116) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:28:53 (1787614133) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:28:56 (1787614136) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:29:00 (1787614140) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:29:28 (1787614168) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:29:31 (1787614171) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 19:29:35 (1787614175) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9443d8f05000.stats=clear osc.lustre-OST0001-osc-ffff9443d8f05000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:29:43 (1787614183) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:29:55 (1787614195) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:30:03 (1787614203) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:30:08 (1787614208) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:30:18 (1787614218) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 19:30:29 (1787614229) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:30:37 (1787614237) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:30:46 (1787614246) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:30:54 (1787614254) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:31:06 (1787614266) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:31:13 (1787614273) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 1.68168 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203104 s, 5.2 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:31:23 (1787614283) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.659646 s, 30.2 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:31:32 (1787614292) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:31:43 (1787614303) Stopping /mnt/lustre-ost1 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9443d8f05000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9443d8f05000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg134-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9443d8f05000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9443d8f05000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg134-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (90s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:33:13 (1787614393) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.275065 s, 7.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9443d8f05000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9443d8f05000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg134-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (53s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:34:06 (1787614446) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9443d8f05000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9443d8f05000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg134-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9443d8f05000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9443d8f05000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg134-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:35:34 (1787614534) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.89319 s, 18.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 24 19:35 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.357s user 0m0.000s sys 0m0.191s Stopping /mnt/lustre-ost1 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9443d8f05000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9443d8f05000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m1.253s user 0m0.000s sys 0m0.162s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg134-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9443d8f05000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9443d8f05000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.843s user 0m0.001s sys 0m0.175s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg134-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg134-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9443d8f05000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9443d8f05000.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:37:09 (1787614629) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:37:13 (1787614633) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:37:17 (1787614637) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 19:37:21 (1787614641) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 19:37:28 (1787614648) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.701426 s, 17.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:37:44 (1787614664) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880766 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660785 s, 15.9 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:37:52 (1787614672) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:38:01 (1787614681) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00089989 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00168372 s, 608 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00543164 s, 189 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000971011 s, 1.1 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 19:38:12 (1787614692) create test files with size 15743203 .. 2d8eb81420673b6f705af048214bc102 - 9ca8ccd90ab463105cf6405e2bd0087e - 6bf3a265ea6c13a7b55c6fdc35d572e5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg134-client.virtnet /mnt/lustre (opts:) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 19:38:39 (1787614719) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:38:57 (1787614737) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:39:04 (1787614744) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917984 s, 11.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404607 s, 25.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 19:39:15 (1787614755) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0975872 s, 21.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.372387 s, 11.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190021 s, 11.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 19:39:34 (1787614774) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 19:39:38 (1787614778) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:39:43 (1787614783) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625551 s, 16.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (11s) == sanity-flr test 44a: lfs mirror split check =========== 19:39:53 (1787614793) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.366868 s, 8.6 MB/s PASS 44a (13s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:40:06 (1787614806) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:40:16 (1787614816) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.358233 s, 29.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg134-client.virtnet /mnt/lustre (opts:) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:40:30 (1787614830) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9443d98c3800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.439214 s, 23.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg134-client.virtnet /mnt/lustre (opts:) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9443c5763800.xattr_cache=1 PASS 44e (25s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:40:56 (1787614856) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:41:41 (1787614901) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:42:20 (1787614940) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 19:42:23 (1787614943) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.372365 s, 8.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.191436 s, 16.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4c50fd8b (nosync,stale) Mirror 2: 0x4c50fd8b (nosync,stale) ** verify mirror 2 doesn't change original checksum: ac3352bfcbcddf2a8aae75313e0eb99d - mirror 1 checksum: 043c915ce588579f36f4e5b4f732b8e6 - mirror 2 checksum: ac3352bfcbcddf2a8aae75313e0eb99d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T19:42:26-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T19:42:26-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: ac3352bfcbcddf2a8aae75313e0eb99d - mirror 1 checksum: 043c915ce588579f36f4e5b4f732b8e6 - mirror 2 checksum: 043c915ce588579f36f4e5b4f732b8e6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T19:42:26-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T19:42:26-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 19:42:35 (1787614955) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489212 s, 21.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:42:43 (1787614963) Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 19:42:56 (1787614976) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 19:43:28 (1787615008) targets are mounted 19:43:28 (1787615008) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268774 s, 39.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd8:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:43:46 (1787615026) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.028528 s, 718 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0113326 s, 1.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:43:56 (1787615036) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0497646 s, 412 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0128689 s, 1.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:44:06 (1787615046) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0100778 s, 1.6 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:44:15 (1787615055) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0666441 s, 307 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:44:25 (1787615065) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0114139 s, 359 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0253126 s, 162 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:44:33 (1787615073) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:45:00 (1787615100) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:45:29 (1787615129) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x48:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:45:54 (1787615154) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:46:20 (1787615180) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 19:46:27 (1787615187) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57614 mirror split pid 57615 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57614 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57615 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 19:47:36 (1787615256) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 5258 truncating to largest ever: 0x530d 000001 1787615258.938041 trunc from 00000000 to 0x00530c (0x530d bytes) truncating to largest ever: 0x37b70 000002 1787615258.954550 trunc from 0x00530d to 0x037b6f (0x32863 bytes) 000003 1787615258.973489 write 0x00849b thru 0x00bc54 (0x37ba bytes) 000004 1787615258.988063 write 0x039386 thru 0x03d128 (0x3da3 bytes) 000005 1787615258.991495 punch from 0x030542 to 0x031f95 (0x1a54 bytes) 000006 1787615258.997465 write 0x03bc72 thru 0x03d480 (0x180f bytes) 000007 1787615258.998648 write 0x025ed1 thru 0x032377 (0xc4a7 bytes) 000008 1787615259.011338 write 0x02fbab thru 0x0369a7 (0x6dfd bytes) 000009 1787615259.016191 punch from 0x013585 to 0x013abc (0x538 bytes) 000010 1787615259.020631 write 0x023568 thru 0x02fbcf (0xc668 bytes) 000012 1787615259.027694 trunc from 0x03d481 to 0x01dabd (0x1f9c3 bytes) 000013 1787615259.050628 mapwrite 0x01ad22 thru 0x01c71d (0x19fc bytes) 000014 1787615259.089784 trunc from 0x00672b to 0x0130f1 (0xc9c7 bytes) 000015 1787615259.095383 mapwrite 0x0180be thru 0x019f92 (0x1ed5 bytes) 000016 1787615259.109052 read 0x00da3f thru 0x019fa4 (0xc566 bytes) 000017 1787615259.120665 read 0x015cca thru 0x01dabd (0x7df4 bytes) 000018 1787615259.121538 punch from 0x00950b to 0x013508 (0x9ffe bytes) 000019 1787615259.125704 write 0x005461 thru 0x00e96b (0x950b bytes) 000020 1787615259.141403 trunc from 0x01dabe to 0x000bf8 (0x1cec5 bytes) 000021 1787615259.161359 punch from 0x00033e to 0x000bf8 (0x8bb bytes) 000024 1787615259.169392 read 0x00007d thru 0x000bf8 (0xb7c bytes) 000025 1787615259.176615 mapwrite 0x0055fe thru 0x0096b7 (0x40ba bytes) 000027 1787615259.236305 mapread 0x009424 thru 0x0096b7 (0x294 bytes) 000029 1787615259.239419 write 0x00d94c thru 0x01b4d3 (0xdb88 bytes) 000030 1787615259.241797 write 0x010e65 thru 0x020b98 (0xfd34 bytes) 000031 1787615259.247688 trunc from 0x005c43 to 0x0099b6 (0x3d74 bytes) 000032 1787615259.254381 punch from 0x01613c to 0x01b80c (0x56d1 bytes) 000033 1787615259.269878 trunc from 0x020b99 to 0x02410d (0x3575 bytes) 000034 1787615259.291536 punch from 0x01d6ba to 0x02410d (0x6a54 bytes) 000035 1787615259.298534 write 0x009730 thru 0x00a7c7 (0x1098 bytes) 000036 1787615259.318444 trunc from 0x001428 to 0x00fde0 (0xe9b9 bytes) truncating to largest ever: 0x3ad66 000037 1787615259.329563 trunc from 0x02410e to 0x03ad65 (0x16c58 bytes) 000039 1787615259.353211 read 0x01d55c thru 0x02acbe (0xd763 bytes) 000040 1787615259.381649 read 0x031209 thru 0x03ad65 (0x9b5d bytes) 000041 1787615259.382990 fallocate 0x017e33 thru 0x024f11 (0xd0df bytes) 000042 1787615259.389455 mapwrite 0x00e016 thru 0x013ffb (0x5fe6 bytes) fallocating to largest ever: 0x3bd3f 000043 1787615259.418422 fallocate 0x036b9d thru 0x03bd3e (0x51a2 bytes) 000044 1787615259.427508 trunc from 0x031e2e to 0x03ad65 (0x8f38 bytes) fallocating to largest ever: 0x40000 000045 1787615259.433858 fallocate 0x03743f thru 0x03ffff (0x8bc1 bytes) 000046 1787615259.444794 mapwrite 0x00d723 thru 0x00f13c (0x1a1a bytes) 000050 1787615259.478785 MIRROR_EXTEND to 2 mirrors 000051 1787615260.160203 fallocate 0x037e16 thru 0x03de17 (0x6002 bytes) 000052 1787615260.217612 trunc from 0x040000 to 0x0046e5 (0x3b91a bytes) 000053 1787615260.296120 punch from 0x004380 to 0x0046e5 (0x366 bytes) 000054 1787615260.310589 trunc from 0x000a79 to 0x0046e5 (0x3c6d bytes) 000056 1787615260.316333 read 0x001eec thru 0x0046e5 (0x27fa bytes) 000057 1787615260.330505 mapwrite 0x030e1e thru 0x03baf5 (0xacd8 bytes) 000058 1787615260.439689 mapread 0x01b2ef thru 0x02182f (0x6541 bytes) 000059 1787615260.463732 fallocate 0x023a5e thru 0x030d6d (0xd310 bytes) 000060 1787615260.484289 trunc from 0x009045 to 0x0155c7 (0xc583 bytes) 000061 1787615260.496891 trunc from 0x03baf6 to 0x01ec53 (0x1cea2 bytes) 000062 1787615260.515990 mapwrite 0x02c9cc thru 0x037abb (0xb0f0 bytes) 000063 1787615260.576176 fallocate 0x03855c thru 0x03ef55 (0x69fa bytes) 000064 1787615260.583226 mapwrite 0x00829d thru 0x014529 (0xc28d bytes) 000065 1787615260.625796 mapread 0x02fb08 thru 0x03d670 (0xdb69 bytes) 000066 1787615260.636440 trunc from 0x03ef56 to 0x03313d (0xbe18 bytes) 000067 1787615260.658682 MIRROR_RESYNC 2 mirrors 000068 1787615261.534728 write 0x022dbb thru 0x02b77c (0x89c2 bytes) 000069 1787615261.593753 fallocate 0x025c95 thru 0x02f549 (0x98b5 bytes) 000070 1787615261.605719 trunc from 0x00f91c to 0x0143e4 (0x4ac9 bytes) 000071 1787615261.618536 trunc from 0x03313e to 0x0114d0 (0x21c6d bytes) 000072 1787615261.654743 punch from 0x00168f to 0x00b822 (0xa194 bytes) 000073 1787615261.670851 read 0x00ab58 thru 0x0114d0 (0x6979 bytes) 000074 1787615261.693378 trunc from 0x0114d1 to 0x0295a7 (0x180d7 bytes) 000075 1787615261.717517 trunc from 0x00346e to 0x00ccab (0x983e bytes) 000076 1787615261.728466 write 0x01aff4 thru 0x026f2e (0xbf3b bytes) 000077 1787615261.745604 write 0x02c316 thru 0x03af5e (0xec49 bytes) 000078 1787615261.756339 MIRROR_RESYNC 2 mirrors 000079 1787615262.864157 read 0x03adf1 thru 0x03af5e (0x16e bytes) 000081 1787615262.871839 write 0x00dc82 thru 0x00fb8c (0x1f0b bytes) 000082 1787615262.932631 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) 000084 1787615263.167469 read 0x0262e7 thru 0x028c99 (0x29b3 bytes) 000085 1787615263.174439 punch from 0x00f3cf to 0x01d1aa (0xdddc bytes) 000086 1787615263.179852 fallocate 0x025b85 thru 0x033e81 (0xe2fd bytes) 000088 1787615263.186065 fallocate 0x00ab04 thru 0x00c2db (0x17d8 bytes) 000089 1787615263.194664 fallocate 0x01c688 thru 0x02a418 (0xdd91 bytes) 000090 1787615263.202580 trunc from 0x03af5f to 0x026539 (0x14a25 bytes) 000091 1787615263.241932 punch from 0x00d4e6 to 0x01995e (0xc479 bytes) 000092 1787615263.253370 trunc from 0x013ed9 to 0x01712a (0x3252 bytes) 000093 1787615263.261432 mapread 0x008150 thru 0x014251 (0xc102 bytes) 000094 1787615263.328817 mapread 0x018186 thru 0x026539 (0xe3b4 bytes) 000095 1787615263.369532 trunc from 0x02653a to 0x02dda6 (0x786d bytes) 000096 1787615263.408262 trunc from 0x00d802 to 0x010dc5 (0x35c4 bytes) 000097 1787615263.422363 mapread 0x006224 thru 0x00905f (0x2e3c bytes) 000098 1787615263.447442 MIRROR_EXTEND to 2 mirrors 000099 1787615264.229248 read 0x016da4 thru 0x01a339 (0x3596 bytes) 000100 1787615264.241607 read 0x010d29 thru 0x017abf (0x6d97 bytes) 000101 1787615264.259546 trunc from 0x02dda7 to 0x038647 (0xa8a1 bytes) 000102 1787615264.331231 trunc from 0x038648 to 0x022d71 (0x158d6 bytes) 000103 1787615264.358706 fallocate 0x01edf7 thru 0x029e69 (0xb073 bytes) 000104 1787615264.366397 punch from 0x00994f to 0x00dbbc (0x426e bytes) 000105 1787615264.383684 write 0x031afa thru 0x035a11 (0x3f18 bytes) 000106 1787615264.386702 write 0x023641 thru 0x0332a7 (0xfc67 bytes) 000107 1787615264.405569 write 0x02b423 thru 0x02ed6f (0x394d bytes) 000108 1787615264.406836 fallocate 0x01001c thru 0x01d241 (0xd226 bytes) 000109 1787615264.414944 write 0x02133b thru 0x021475 (0x13b bytes) 000110 1787615264.424566 trunc from 0x035a12 to 0x009921 (0x2c0f0 bytes) 000111 1787615264.454371 punch from 0x005453 to 0x008fec (0x3b9a bytes) 000112 1787615264.470155 mapwrite 0x007bdd thru 0x014aee (0xcf12 bytes) 000113 1787615264.663926 read 0x007e0e thru 0x014aee (0xcce1 bytes) 000114 1787615264.664689 fallocate 0x033bd1 thru 0x0366bf (0x2aef bytes) 000115 1787615264.670951 mapwrite 0x00d6ad thru 0x01afcb (0xd91f bytes) 000116 1787615264.745353 mapwrite 0x032e6d thru 0x03924b (0x63df bytes) 000117 1787615264.834703 trunc from 0x037ec8 to 0x03924b (0x1384 bytes) 000118 1787615264.851811 read 0x00c00d thru 0x00c0ed (0xe1 bytes) 000119 1787615264.853262 mapwrite 0x017ee0 thru 0x02064e (0x876f bytes) 000121 1787615264.890740 mapread 0x0058d3 thru 0x008561 (0x2c8f bytes) 000122 1787615264.894293 punch from 0x026145 to 0x035aec (0xf9a8 bytes) 000124 1787615264.905411 mapwrite 0x003997 thru 0x00ba03 (0x806d bytes) 000125 1787615264.933502 fallocate 0x015f98 thru 0x0182e6 (0x234f bytes) 000126 1787615264.955145 write 0x02a938 thru 0x02b025 (0x6ee bytes) 000127 1787615264.973329 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) 000128 1787615265.158497 write 0x001eb1 thru 0x004580 (0x26d0 bytes) 000130 1787615265.179314 read 0x02158b thru 0x02a7a4 (0x921a bytes) 000131 1787615265.189957 mapwrite 0x003fb6 thru 0x009e89 (0x5ed4 bytes) 000132 1787615265.228409 mapread 0x025f5c thru 0x02aafe (0x4ba3 bytes) 000133 1787615265.231350 mapread 0x00d92d thru 0x01a37c (0xca50 bytes) 000134 1787615265.251906 read 0x03726b thru 0x03924b (0x1fe1 bytes) 000135 1787615265.278059 mapwrite 0x03100b thru 0x03920d (0x8203 bytes) 000136 1787615265.374025 trunc from 0x023439 to 0x02e526 (0xb0ee bytes) 000137 1787615265.385440 punch from 0x00e478 to 0x01cce0 (0xe869 bytes) 000139 1787615265.399335 read 0x00e367 thru 0x0115c9 (0x3263 bytes) 000142 1787615265.417916 write 0x0372d3 thru 0x03df53 (0x6c81 bytes) 000143 1787615265.419208 fallocate 0x03392d thru 0x03d585 (0x9c59 bytes) 000144 1787615265.423027 mapwrite 0x03ed06 thru 0x03ffff (0x12fa bytes) 000145 1787615265.457661 trunc from 0x0273d8 to 0x034c32 (0xd85b bytes) 000146 1787615265.461584 punch from 0x02361b to 0x02a23c (0x6c22 bytes) 000147 1787615265.464827 read 0x021c59 thru 0x02a440 (0x87e8 bytes) 000148 1787615265.470292 punch from 0x03e683 to 0x03ffff (0x197d bytes) 000149 1787615265.473361 mapread 0x0213c4 thru 0x027a54 (0x6691 bytes) 000150 1787615265.474256 write 0x01ee26 thru 0x027d5b (0x8f36 bytes) 000151 1787615265.475252 trunc from 0x040000 to 0x01c2d7 (0x23d28 bytes) 000152 1787615265.485807 trunc from 0x018ef2 to 0x01a3e7 (0x14f6 bytes) 000153 1787615265.488893 mapread 0x00593a thru 0x00f7d0 (0x9e97 bytes) 000154 1787615265.490206 fallocate 0x0180bf thru 0x01fcb0 (0x7bf2 bytes) 000155 1787615265.494203 fallocate 0x01b32c thru 0x021073 (0x5d48 bytes) 000156 1787615265.497789 trunc from 0x01fcb1 to 0x02c7ff (0xcb4f bytes) 000157 1787615265.506730 read 0x029e58 thru 0x02c7ff (0x29a8 bytes) 000160 1787615265.516255 read 0x00e4c6 thru 0x01c604 (0xe13f bytes) 000161 1787615265.516860 punch from 0x003900 to 0x00696c (0x306d bytes) 000162 1787615265.521228 write 0x02b446 thru 0x037c28 (0xc7e3 bytes) 000163 1787615265.522685 trunc from 0x037c29 to 0x03d2e1 (0x56b9 bytes) 000164 1787615265.534493 fallocate 0x027631 thru 0x0373b7 (0xfd87 bytes) 000165 1787615265.538259 trunc from 0x03d2e2 to 0x00362e (0x39cb3 bytes) 000166 1787615265.556820 mapread 0x001d45 thru 0x00362e (0x18ea bytes) 000167 1787615265.562576 punch from 0x0017b7 to 0x00362e (0x1e78 bytes) 000169 1787615265.569775 read 0x00098a thru 0x00362e (0x2ca5 bytes) 000171 1787615265.578883 fallocate 0x035e32 thru 0x03ffff (0xa1ce bytes) 000172 1787615265.584714 read 0x00266a thru 0x00362e (0xfc5 bytes) 000173 1787615265.585178 mapwrite 0x01fe77 thru 0x02fa58 (0xfbe2 bytes) 000174 1787615265.648682 trunc from 0x02fa59 to 0x018ea1 (0x16bb7 bytes) 000176 1787615265.666975 write 0x0063df thru 0x0142d4 (0xdef6 bytes) 000177 1787615265.668813 punch from 0x015f05 to 0x018ea1 (0x2f9d bytes) 000178 1787615265.672867 mapwrite 0x02f991 thru 0x034c60 (0x52d0 bytes) 000179 1787615265.710013 trunc from 0x034c61 to 0x0395a5 (0x4945 bytes) 000180 1787615265.725606 trunc from 0x0395a6 to 0x005c5c (0x33949 bytes) 000181 1787615265.746933 trunc from 0x002d17 to 0x005c5c (0x2f46 bytes) 000182 1787615265.756538 trunc from 0x005c5d to 0x00b6f0 (0x5a94 bytes) 000183 1787615265.776454 MIRROR_EXTEND to 2 mirrors 000184 1787615266.625389 mapread 0x000169 thru 0x00369a (0x3532 bytes) 000185 1787615266.662526 trunc from 0x000d74 to 0x00b3b1 (0xa63e bytes) 000186 1787615266.689981 read 0x001ae4 thru 0x00b6f0 (0x9c0d bytes) 000187 1787615266.700956 read 0x005206 thru 0x0073b5 (0x21b0 bytes) 000188 1787615266.701869 trunc from 0x00142b to 0x00362b (0x2201 bytes) 000191 1787615266.708248 mapread 0x001f37 thru 0x00b6f0 (0x97ba bytes) 000192 1787615266.715168 mapread 0x006e49 thru 0x00b6f0 (0x48a8 bytes) 000193 1787615266.716274 read 0x0070a9 thru 0x00b6f0 (0x4648 bytes) 000194 1787615266.718055 write 0x02757c thru 0x02b344 (0x3dc9 bytes) 000196 1787615266.729287 MIRROR_RESYNC 2 mirrors 000197 1787615267.391431 trunc from 0x006181 to 0x0142d9 (0xe159 bytes) 000198 1787615267.444875 punch from 0x008967 to 0x015a2e (0xd0c8 bytes) 000199 1787615267.457990 mapwrite 0x00ddce thru 0x017857 (0x9a8a bytes) 000200 1787615267.540983 mapwrite 0x02e6cf thru 0x035b50 (0x7482 bytes) 000201 1787615267.626416 fallocate 0x015c1a thru 0x01a39d (0x4784 bytes) 000202 1787615267.640896 mapread 0x004254 thru 0x005cf9 (0x1aa6 bytes) 000203 1787615267.655931 write 0x02c22f thru 0x03304c (0x6e1e bytes) 000204 1787615267.659419 fallocate 0x0149fe thru 0x018678 (0x3c7b bytes) 000205 1787615267.671812 trunc from 0x01d751 to 0x021962 (0x4212 bytes) 000206 1787615267.680820 read 0x01dbb8 thru 0x02bfc1 (0xe40a bytes) 000208 1787615267.720710 trunc from 0x035b51 to 0x0229ec (0x13164 bytes) 000209 1787615267.753931 trunc from 0x01e4ce to 0x021661 (0x3194 bytes) 000211 1787615267.764607 punch from 0x021e00 to 0x0229ec (0xbed bytes) 000212 1787615267.772728 trunc from 0x0229ed to 0x01a12e (0x88be bytes) 000214 1787615267.800779 trunc from 0x00b73d to 0x00fbed (0x44b1 bytes) 000215 1787615267.812628 mapread 0x00448c thru 0x009b95 (0x570a bytes) 000216 1787615267.825178 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) 000217 1787615268.005682 write 0x011938 thru 0x013fa3 (0x266c bytes) 000218 1787615268.036939 trunc from 0x019878 to 0x01a12e (0x8b7 bytes) 000219 1787615268.064203 write 0x0078ed thru 0x00f2d1 (0x79e5 bytes) 000220 1787615268.083304 mapwrite 0x0358db thru 0x03b3bc (0x5ae2 bytes) 000221 1787615268.245544 mapwrite 0x003b0d thru 0x0101db (0xc6cf bytes) 000222 1787615268.320884 punch from 0x00a1d0 to 0x00cce0 (0x2b11 bytes) 000223 1787615268.345816 fallocate 0x02b86e thru 0x0350b7 (0x984a bytes) 000224 1787615268.362217 trunc from 0x03845d to 0x03b3bc (0x2f60 bytes) 000225 1787615268.399259 trunc from 0x013b7f to 0x0162ca (0x274c bytes) 000227 1787615268.413774 read 0x010f04 thru 0x0202ee (0xf3eb bytes) 000229 1787615268.446565 mapwrite 0x01f7da thru 0x029225 (0x9a4c bytes) 000230 1787615268.539320 punch from 0x023348 to 0x028c82 (0x593b bytes) 000231 1787615268.553874 punch from 0x01818e to 0x01b8d8 (0x374b bytes) 000232 1787615268.567337 mapwrite 0x00a683 thru 0x01598a (0xb308 bytes) 000234 1787615268.650404 trunc from 0x031dd0 to 0x03b3bc (0x95ed bytes) 000235 1787615268.670405 punch from 0x008b58 to 0x0160a7 (0xd550 bytes) 000237 1787615268.697814 punch from 0x037bd3 to 0x038297 (0x6c5 bytes) 000238 1787615268.706322 trunc from 0x03b3bd to 0x0002af (0x3b10d bytes) 000239 1787615268.735925 write 0x01f068 thru 0x02eab2 (0xfa4b bytes) 000240 1787615268.737716 trunc from 0x02eab3 to 0x03943a (0xa988 bytes) 000241 1787615268.770055 mapread 0x033219 thru 0x03943a (0x6222 bytes) 000242 1787615268.785837 mapread 0x019912 thru 0x025684 (0xbd73 bytes) 000243 1787615268.800273 read 0x014c10 thru 0x01d941 (0x8d32 bytes) 000244 1787615268.802150 punch from 0x038b0c to 0x03943a (0x92f bytes) 000246 1787615268.810289 trunc from 0x013987 to 0x01cb2d (0x91a7 bytes) 000247 1787615268.825876 mapwrite 0x0391f6 thru 0x03ffff (0x6e0a bytes) 000248 1787615268.922390 read 0x01a28e thru 0x01d7d9 (0x354c bytes) 000250 1787615268.924404 punch from 0x01550a to 0x022365 (0xce5c bytes) 000252 1787615268.937928 punch from 0x018385 to 0x01cd91 (0x4a0d bytes) 000253 1787615268.943292 fallocate 0x004cc8 thru 0x01483b (0xfb74 bytes) 000254 1787615268.948387 mapwrite 0x0170ba thru 0x02086d (0x97b4 bytes) 000256 1787615268.982536 trunc from 0x040000 to 0x0321f8 (0xde07 bytes) 000257 1787615268.998881 read 0x012d19 thru 0x0167a8 (0x3a90 bytes) 000258 1787615268.999581 write 0x0200a5 thru 0x02fc89 (0xfbe5 bytes) 000259 1787615269.001835 mapwrite 0x02b5de thru 0x030174 (0x4b97 bytes) 000260 1787615269.025395 read 0x003108 thru 0x008cc9 (0x5bc2 bytes) 000261 1787615269.026501 punch from 0x0200cc to 0x026272 (0x61a7 bytes) 000262 1787615269.033490 write 0x01819b thru 0x01a5aa (0x2410 bytes) 000263 1787615269.034927 mapread 0x028e8f thru 0x0321f8 (0x936a bytes) 000265 1787615269.037903 trunc from 0x0321f9 to 0x018d5d (0x1949b bytes) 000267 1787615269.059467 write 0x00e3af thru 0x0144da (0x612c bytes) 000268 1787615269.061205 trunc from 0x018421 to 0x018d5d (0x93d bytes) 000269 1787615269.075686 trunc from 0x017daf to 0x018d5d (0xfaf bytes) 000270 1787615269.080878 trunc from 0x00cf70 to 0x010444 (0x34d5 bytes) 000271 1787615269.086901 write 0x0284d2 thru 0x033c96 (0xb7c5 bytes) 000273 1787615269.089741 read 0x0267dc thru 0x029faf (0x37d4 bytes) 000274 1787615269.103180 MIRROR_EXTEND to 2 mirrors 000275 1787615269.827298 mapread 0x01d425 thru 0x02c8f8 (0xf4d4 bytes) 000276 1787615269.876019 read 0x013794 thru 0x0155fa (0x1e67 bytes) 000278 1787615269.908928 fallocate 0x03ea46 thru 0x03ffff (0x15ba bytes) 000279 1787615269.959455 mapread 0x0096fc thru 0x01107a (0x797f bytes) 000280 1787615269.970433 trunc from 0x040000 to 0x01b5c3 (0x24a3c bytes) 000282 1787615270.027679 mapwrite 0x02ac54 thru 0x039dce (0xf17b bytes) 000283 1787615270.086259 mapwrite 0x026347 thru 0x031eb2 (0xbb6c bytes) 000284 1787615270.108223 trunc from 0x039dcf to 0x02d24a (0xcb84 bytes) 000285 1787615270.128381 fallocate 0x03fb99 thru 0x03ffff (0x467 bytes) 000286 1787615270.134370 trunc from 0x040000 to 0x024854 (0x1b7ab bytes) 000287 1787615270.156603 read 0x023c4e thru 0x024854 (0xc07 bytes) 000288 1787615270.162472 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) 000289 1787615270.329593 read 0x00f318 thru 0x01d285 (0xdf6e bytes) 000290 1787615270.340145 fallocate 0x0116d9 thru 0x01ddec (0xc714 bytes) 000291 1787615270.345472 read 0x023181 thru 0x024854 (0x16d4 bytes) 000292 1787615270.364725 write 0x0005c4 thru 0x00f82d (0xf26a bytes) 000293 1787615270.370818 write 0x02969b thru 0x02b847 (0x21ad bytes) 000294 1787615270.378133 MIRROR_EXTEND to 2 mirrors 000295 1787615270.975935 write 0x0081a6 thru 0x00ad0f (0x2b6a bytes) 000296 1787615271.040150 read 0x019a60 thru 0x01f304 (0x58a5 bytes) 000297 1787615271.053324 trunc from 0x01dfb7 to 0x026890 (0x88da bytes) 000298 1787615271.059148 mapwrite 0x007387 thru 0x00dbd4 (0x684e bytes) 000300 1787615271.102367 trunc from 0x02b848 to 0x005e1d (0x25a2a bytes) 000301 1787615271.176860 MIRROR_RESYNC 2 mirrors 000302 1787615271.997044 write 0x039700 thru 0x03ffff (0x6900 bytes) 000303 1787615272.060563 fallocate 0x0002dc thru 0x0025ef (0x2314 bytes) 000304 1787615272.069937 write 0x02c6ce thru 0x03bf72 (0xf8a5 bytes) 000305 1787615272.081229 trunc from 0x040000 to 0x038561 (0x7a9e bytes) 000306 1787615272.122051 punch from 0x0115f7 to 0x013927 (0x2331 bytes) 000307 1787615272.131747 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) 000308 1787615272.271531 mapwrite 0x0242b7 thru 0x032dc3 (0xeb0d bytes) 000310 1787615272.319242 punch from 0x00a49e to 0x017e99 (0xd9fc bytes) 000311 1787615272.323918 mapwrite 0x025fae thru 0x02d1be (0x7211 bytes) 000312 1787615272.340033 trunc from 0x038562 to 0x006e2f (0x31732 bytes) 000314 1787615272.361441 mapread 0x006e0f thru 0x006e2f (0x21 bytes) 000316 1787615272.379990 MIRROR_EXTEND to 2 mirrors 000317 1787615273.188252 read 0x002f7b thru 0x0050a2 (0x2128 bytes) 000318 1787615273.207773 trunc from 0x006e30 to 0x0017c6 (0x5669 bytes) 000320 1787615273.309643 punch from 0x0011ae to 0x0017c6 (0x619 bytes) 000321 1787615273.322179 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x17c7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17c7): Mirror 2: 0x335f59f3 (stale) CRC-32 checksum value for chunk [0, 0x17c7): Mirror 1: 0x4521204e Mirror 3: 0x4521204e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17c7: skipped 000323 1787615274.171273 trunc from 0x000049 to 0x0017c6 (0x177e bytes) 000324 1787615274.240036 trunc from 0x0017c7 to 0x0219f4 (0x2022e bytes) 000325 1787615274.282226 trunc from 0x0219f5 to 0x027d73 (0x637f bytes) 000326 1787615274.320251 write 0x03ef56 thru 0x03ffff (0x10aa bytes) 000327 1787615274.321675 trunc from 0x040000 to 0x00bf86 (0x34079 bytes) 000328 1787615274.348274 mapwrite 0x028927 thru 0x02d35e (0x4a38 bytes) 000330 1787615274.477309 trunc from 0x02d35f to 0x02ea9e (0x1740 bytes) 000331 1787615274.542305 mapread 0x014f01 thru 0x02295a (0xda5a bytes) 000332 1787615274.579965 trunc from 0x02ea9f to 0x00d449 (0x21655 bytes) 000333 1787615274.624696 read 0x00584d thru 0x00d449 (0x7bfd bytes) 000334 1787615274.655272 trunc from 0x00c1a2 to 0x00d449 (0x12a8 bytes) 000335 1787615274.685626 fallocate 0x0101a4 thru 0x01023a (0x97 bytes) 000336 1787615274.710195 trunc from 0x0003cf to 0x002e24 (0x2a56 bytes) 000337 1787615274.738713 fallocate 0x00953b thru 0x009bf1 (0x6b7 bytes) 000338 1787615274.764700 fallocate 0x01ada8 thru 0x02106b (0x62c4 bytes) 000339 1787615274.776798 mapwrite 0x02b69c thru 0x038cf7 (0xd65c bytes) 000340 1787615274.939610 punch from 0x0193e0 to 0x01f043 (0x5c64 bytes) 000341 1787615274.948917 trunc from 0x017ea0 to 0x026cbd (0xee1e bytes) 000342 1787615274.957628 read 0x007d99 thru 0x01135f (0x95c7 bytes) 000344 1787615274.975926 mapwrite 0x0329a5 thru 0x03eef2 (0xc54e bytes) 000345 1787615275.045150 trunc from 0x037a62 to 0x03eef2 (0x7491 bytes) 000346 1787615275.057949 trunc from 0x03eef3 to 0x02c32e (0x12bc4 bytes) 000347 1787615275.093966 mapwrite 0x00b23e thru 0x01a536 (0xf2f9 bytes) 000348 1787615275.144378 read 0x0166a8 thru 0x01bd59 (0x56b2 bytes) 000350 1787615275.145611 trunc from 0x02c32f to 0x01678b (0x15ba3 bytes) 000352 1787615275.173423 punch from 0x001fb2 to 0x0026da (0x729 bytes) 000353 1787615275.189965 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1678c): Mirror 3: 0xe5297750 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1678c): Mirror 3: 0xe5297750 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1678c: skipped 000354 1787615275.551266 write 0x019bd7 thru 0x022222 (0x864c bytes) 000355 1787615275.557603 mapread 0x00d61f thru 0x0142d5 (0x6cb7 bytes) 000356 1787615275.579310 punch from 0x00e2b6 to 0x01cfe7 (0xed32 bytes) 000357 1787615275.591244 fallocate 0x02cd9e thru 0x03bfc7 (0xf22a bytes) 000358 1787615275.595539 fallocate 0x0179ab thru 0x01da37 (0x608d bytes) 000359 1787615275.600828 trunc from 0x022223 to 0x02546a (0x3248 bytes) 000360 1787615275.617846 fallocate 0x025be5 thru 0x0354c3 (0xf8df bytes) 000361 1787615275.622478 trunc from 0x0354c4 to 0x0235ef (0x11ed4 bytes) 000362 1787615275.640515 trunc from 0x02341a to 0x0235ef (0x1d6 bytes) 000363 1787615275.644969 trunc from 0x0235f0 to 0x0052ab (0x1e344 bytes) 000364 1787615275.660324 write 0x012722 thru 0x014678 (0x1f57 bytes) 000366 1787615275.661854 write 0x000449 thru 0x001519 (0x10d1 bytes) 000367 1787615275.672028 punch from 0x001c33 to 0x0070c4 (0x5492 bytes) 000368 1787615275.684856 punch from 0x002117 to 0x005694 (0x357e bytes) 000370 1787615275.689941 punch from 0x010ef2 to 0x014678 (0x3787 bytes) 000371 1787615275.695415 mapwrite 0x016663 thru 0x025fab (0xf949 bytes) 000372 1787615275.746044 trunc from 0x01cc09 to 0x01d297 (0x68f bytes) 000373 1787615275.754382 fallocate 0x020327 thru 0x02e0f7 (0xddd1 bytes) 000374 1787615275.759795 punch from 0x003419 to 0x006ed8 (0x3ac0 bytes) 000375 1787615275.764428 mapread 0x0012ca thru 0x001a1d (0x754 bytes) 000377 1787615275.774921 read 0x019ca9 thru 0x020a3e (0x6d96 bytes) 000378 1787615275.779808 mapwrite 0x017ecb thru 0x01e9a1 (0x6ad7 bytes) 000379 1787615275.797039 write 0x03cb94 thru 0x03ffff (0x346c bytes) 000380 1787615275.798827 punch from 0x035f60 to 0x03d532 (0x75d3 bytes) 000381 1787615275.815406 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) 000382 1787615275.950530 mapread 0x012ba4 thru 0x017904 (0x4d61 bytes) 000383 1787615275.974885 read 0x016fa8 thru 0x023977 (0xc9d0 bytes) 000384 1787615275.981952 trunc from 0x040000 to 0x030d8d (0xf272 bytes) 000385 1787615275.998420 trunc from 0x018b19 to 0x02274a (0x9c32 bytes) 000386 1787615276.007306 mapwrite 0x0257c7 thru 0x034b99 (0xf3d3 bytes) 000387 1787615276.054657 fallocate 0x01f494 thru 0x0245ab (0x5118 bytes) 000388 1787615276.059640 mapwrite 0x011b0f thru 0x01b1c9 (0x96bb bytes) 000390 1787615276.093618 trunc from 0x034b9a to 0x00976f (0x2b42a bytes) 000391 1787615276.112055 fallocate 0x00a180 thru 0x00bfa7 (0x1e28 bytes) 000395 1787615276.120067 punch from 0x0043e4 to 0x007db7 (0x39d4 bytes) 000396 1787615276.124330 mapwrite 0x035ad2 thru 0x03c212 (0x6741 bytes) 000397 1787615276.186582 write 0x021c82 thru 0x02f35b (0xd6da bytes) 000398 1787615276.201269 read 0x01890a thru 0x025996 (0xd08d bytes) 000399 1787615276.249269 fallocate 0x03e042 thru 0x03ffff (0x1fbe bytes) 000400 1787615276.258824 trunc from 0x03c213 to 0x02bcc1 (0x10551 bytes) 000401 1787615276.301665 fallocate 0x00631c thru 0x0128a2 (0xc587 bytes) 000403 1787615276.309617 punch from 0x01ee7e to 0x02174c (0x28cf bytes) 000404 1787615276.332018 trunc from 0x01686c to 0x01bfab (0x5740 bytes) 000405 1787615276.338817 trunc from 0x01d00e to 0x0268ca (0x98bd bytes) 000406 1787615276.346935 fallocate 0x029964 thru 0x02f0fb (0x5798 bytes) 000407 1787615276.352960 mapread 0x0014e0 thru 0x010bbe (0xf6df bytes) 000408 1787615276.355851 fallocate 0x01c524 thru 0x0231e4 (0x6cc1 bytes) 000409 1787615276.361618 mapwrite 0x019e9d thru 0x027df5 (0xdf59 bytes) 000410 1787615276.528941 trunc from 0x00ae0e to 0x00f68d (0x4880 bytes) 000411 1787615276.552007 mapwrite 0x000fce thru 0x009f94 (0x8fc7 bytes) 000412 1787615276.598110 trunc from 0x002189 to 0x004758 (0x25d0 bytes) 000413 1787615276.610239 read 0x0195c2 thru 0x0271c8 (0xdc07 bytes) 000415 1787615276.611319 fallocate 0x034532 thru 0x0369bd (0x248c bytes) 000416 1787615276.627889 trunc from 0x02bcc2 to 0x03e0a9 (0x123e8 bytes) 000417 1787615276.670060 mapwrite 0x003810 thru 0x006209 (0x29fa bytes) 000419 1787615276.731503 trunc from 0x030a6b to 0x032d58 (0x22ee bytes) 000420 1787615276.747051 read 0x000dbe thru 0x007a5a (0x6c9d bytes) 000422 1787615276.751189 read 0x01b07a thru 0x021830 (0x67b7 bytes) 000423 1787615276.754397 mapread 0x019221 thru 0x0257ce (0xc5ae bytes) 000424 1787615276.758609 write 0x00e14e thru 0x01b5ea (0xd49d bytes) 000425 1787615276.773248 MIRROR_EXTEND to 2 mirrors 000426 1787615277.885008 punch from 0x005dc9 to 0x00f015 (0x924d bytes) 000427 1787615277.940741 trunc from 0x03550e to 0x0361b1 (0xca4 bytes) 000428 1787615277.947310 mapwrite 0x03e9e4 thru 0x03f894 (0xeb1 bytes) 000429 1787615278.023947 read 0x00485a thru 0x00ee7d (0xa624 bytes) 000430 1787615278.052985 mapread 0x00a5e2 thru 0x015f5f (0xb97e bytes) 000431 1787615278.073539 MIRROR_RESYNC 2 mirrors 000432 1787615278.704590 write 0x033092 thru 0x036c1a (0x3b89 bytes) 000433 1787615278.801571 mapread 0x001cb7 thru 0x00c546 (0xa890 bytes) 000434 1787615278.850416 mapread 0x0028e0 thru 0x007e2f (0x5550 bytes) 000435 1787615278.851277 trunc from 0x0259c8 to 0x02647b (0xab4 bytes) 000436 1787615278.858205 punch from 0x02d0d8 to 0x02e3fb (0x1324 bytes) 000438 1787615278.889827 trunc from 0x0041e0 to 0x0136df (0xf500 bytes) 000439 1787615278.903600 read 0x03c41a thru 0x03f894 (0x347b bytes) 000440 1787615278.922152 mapread 0x004323 thru 0x00932b (0x5009 bytes) 000441 1787615278.935611 mapread 0x01d169 thru 0x02a9d6 (0xd86e bytes) 000442 1787615278.979929 mapwrite 0x02e3bb thru 0x03caa2 (0xe6e8 bytes) 000443 1787615279.089366 write 0x00ab16 thru 0x00fe50 (0x533b bytes) 000444 1787615279.090486 read 0x026b37 thru 0x030869 (0x9d33 bytes) 000445 1787615279.091051 write 0x01505f thru 0x01dbf3 (0x8b95 bytes) 000446 1787615279.183717 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f895): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f895): Mirror 2: 0x5b89ec14 (stale) CRC-32 checksum value for chunk [0, 0x3f895): Mirror 1: 0xeb66ba88 Mirror 3: 0xeb66ba88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f895: skipped 000447 1787615280.246953 trunc from 0x03f895 to 0x0280c4 (0x177d0 bytes) 000448 1787615280.370046 punch from 0x019c37 to 0x019dd5 (0x19f bytes) 000449 1787615280.385906 trunc from 0x0280c5 to 0x00bf13 (0x1c1b1 bytes) 000450 1787615280.408058 punch from 0x00bd68 to 0x00bf13 (0x1ac bytes) 000451 1787615280.414166 read 0x000b5f thru 0x00bf13 (0xb3b5 bytes) 000453 1787615280.431365 read 0x0096a7 thru 0x00bf13 (0x286d bytes) 000454 1787615280.433300 trunc from 0x0015ee to 0x00bf13 (0xa926 bytes) 000455 1787615280.454233 mapwrite 0x02fc5b thru 0x031126 (0x14cc bytes) 000456 1787615280.534229 mapwrite 0x004472 thru 0x009359 (0x4ee8 bytes) 000458 1787615280.588851 mapwrite 0x0241ed thru 0x0258b7 (0x16cb bytes) 000459 1787615280.619340 mapwrite 0x0148ed thru 0x023aef (0xf203 bytes) 000460 1787615280.677958 trunc from 0x031127 to 0x016bcf (0x1a557 bytes) 000461 1787615280.718795 read 0x015efd thru 0x0168c9 (0x9cd bytes) 000462 1787615280.720641 mapread 0x00e2d1 thru 0x016bcf (0x88ff bytes) 000463 1787615280.746972 MIRROR_RESYNC 3 mirrors 000464 1787615282.163238 punch from 0x015804 to 0x016bcf (0x13cc bytes) 000465 1787615282.202365 MIRROR_RESYNC 3 mirrors 000466 1787615283.186687 read 0x014d46 thru 0x016bcf (0x1e8a bytes) 000467 1787615283.193033 read 0x0146dc thru 0x016bcf (0x24f4 bytes) 000468 1787615283.193646 mapread 0x0081d8 thru 0x00f95f (0x7788 bytes) 000469 1787615283.210860 read 0x00f233 thru 0x016b74 (0x7942 bytes) 000470 1787615283.212261 write 0x038d1e thru 0x03ffff (0x72e2 bytes) 000471 1787615283.242775 fallocate 0x012a45 thru 0x0189f1 (0x5fad bytes) 000472 1787615283.248913 trunc from 0x03d987 to 0x03ffff (0x2679 bytes) 000473 1787615283.282565 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xeca2be9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xeca2be9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000474 1787615283.752901 write 0x0217ed thru 0x0278df (0x60f3 bytes) 000475 1787615283.779142 read 0x018705 thru 0x02475d (0xc059 bytes) 000476 1787615283.797292 read 0x000d2e thru 0x0069fe (0x5cd1 bytes) 000477 1787615283.816839 write 0x00e4f8 thru 0x017e4a (0x9953 bytes) 000478 1787615283.818715 trunc from 0x006985 to 0x00b57b (0x4bf7 bytes) 000479 1787615283.831664 write 0x00de52 thru 0x0114c7 (0x3676 bytes) 000480 1787615283.834706 write 0x0214b1 thru 0x023117 (0x1c67 bytes) 000481 1787615283.838768 trunc from 0x040000 to 0x039349 (0x6cb6 bytes) 000482 1787615283.859820 trunc from 0x009da4 to 0x015902 (0xbb5f bytes) 000483 1787615283.893854 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3934a): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3934a): Mirror 3: 0xe409f5da (stale) CRC-32 checksum value for chunk [0, 0x3934a): Mirror 1: 0x2e9af55e Mirror 4: 0x2e9af55e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3934a: skipped 000484 1787615285.360693 mapwrite 0x001aa7 thru 0x00daf3 (0xc04d bytes) 000485 1787615285.564928 read 0x02fa40 thru 0x0301eb (0x7ac bytes) 000487 1787615285.571820 trunc from 0x018050 to 0x01baa6 (0x3a57 bytes) 000488 1787615285.581813 trunc from 0x017c57 to 0x020cb7 (0x9061 bytes) 000489 1787615285.597988 mapread 0x019719 thru 0x01aeeb (0x17d3 bytes) 000490 1787615285.609799 write 0x00b957 thru 0x01446a (0x8b14 bytes) 000491 1787615285.615634 write 0x01e2ba thru 0x02d04d (0xed94 bytes) 000492 1787615285.626836 MIRROR_RESYNC 3 mirrors 000493 1787615286.841867 punch from 0x020c55 to 0x02500b (0x43b7 bytes) 000494 1787615286.910992 mapwrite 0x03260e thru 0x032776 (0x169 bytes) 000495 1787615286.935993 mapwrite 0x00bf0c thru 0x00eed3 (0x2fc8 bytes) 000496 1787615286.992982 punch from 0x009140 to 0x016324 (0xd1e5 bytes) 000497 1787615286.999181 trunc from 0x03934a to 0x02bf88 (0xd3c1 bytes) 000498 1787615287.062136 write 0x0226d7 thru 0x02c876 (0xa1a0 bytes) 000499 1787615287.075404 fallocate 0x037bc8 thru 0x039f2e (0x2367 bytes) 000500 1787615287.090061 mapwrite 0x03ccd3 thru 0x03ffff (0x332d bytes) 000501 1787615287.196427 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc2950d77 (stale) Mirror 4: 0xc2950d77 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb3676b7f Mirror 5: 0xb3676b7f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000504 1787615288.578897 mapread 0x01ffba thru 0x0237ed (0x3834 bytes) 000505 1787615288.634737 mapwrite 0x02a1f2 thru 0x02c35b (0x216a bytes) 000506 1787615288.742536 trunc from 0x040000 to 0x021918 (0x1e6e7 bytes) 000507 1787615288.811818 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x21919): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21919): Mirror 3: 0xb483d9b6 (stale) Mirror 4: 0xb483d9b6 (stale) Mirror 5: 0x97e6832c (stale) CRC-32 checksum value for chunk [0, 0x21919): Mirror 1: 0x97e6832c Mirror 6: 0x97e6832c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21919: skipped 000508 1787615289.721991 fallocate 0x015b00 thru 0x018429 (0x292a bytes) 000509 1787615289.764671 trunc from 0x01340f to 0x015a2c (0x261e bytes) 000510 1787615289.769921 mapread 0x02034d thru 0x021918 (0x15cc bytes) 000511 1787615289.783944 read 0x017e39 thru 0x01a13e (0x2306 bytes) 000512 1787615289.790418 trunc from 0x00ac58 to 0x01696a (0xbd13 bytes) 000513 1787615289.799752 MIRROR_RESYNC 5 mirrors 000514 1787615291.232796 trunc from 0x021919 to 0x02bca9 (0xa391 bytes) 000515 1787615291.289825 punch from 0x011da3 to 0x01c8cc (0xab2a bytes) 000516 1787615291.306820 mapwrite 0x0287bc thru 0x029d80 (0x15c5 bytes) 000517 1787615291.342911 trunc from 0x018f63 to 0x027f99 (0xf037 bytes) 000518 1787615291.348535 read 0x019425 thru 0x020e8c (0x7a68 bytes) 000519 1787615291.365937 MIRROR_RESYNC 5 mirrors 000520 1787615292.425436 trunc from 0x02bcaa to 0x03fcc6 (0x1401d bytes) 000521 1787615292.485793 mapwrite 0x0013c4 thru 0x003e5e (0x2a9b bytes) 000522 1787615292.525410 mapwrite 0x00d7f0 thru 0x00dc14 (0x425 bytes) 000523 1787615292.548529 mapread 0x028508 thru 0x02e4f1 (0x5fea bytes) 000524 1787615292.571472 read 0x03a0a9 thru 0x03fcc6 (0x5c1e bytes) 000525 1787615292.588300 punch from 0x038c43 to 0x03fcc6 (0x7084 bytes) 000526 1787615292.594141 write 0x02608c thru 0x02c754 (0x66c9 bytes) 000527 1787615292.596369 write 0x01c02d thru 0x024887 (0x885b bytes) 000528 1787615292.604846 trunc from 0x03fcc7 to 0x02837c (0x1794a bytes) 000530 1787615292.640971 trunc from 0x01a46a to 0x02026e (0x5e05 bytes) 000531 1787615292.675606 read 0x00ee68 thru 0x011b6d (0x2d06 bytes) 000534 1787615292.712587 write 0x03631f thru 0x038ca2 (0x2984 bytes) 000535 1787615292.725622 fallocate 0x028a91 thru 0x0347d1 (0xbd41 bytes) 000536 1787615292.748282 fallocate 0x0161bd thru 0x01d4de (0x7322 bytes) 000537 1787615292.757103 mapwrite 0x01c866 thru 0x02bfd2 (0xf76d bytes) 000538 1787615292.852126 fallocate 0x00a974 thru 0x01a366 (0xf9f3 bytes) 000539 1787615292.863125 fallocate 0x036476 thru 0x03ed29 (0x88b4 bytes) 000540 1787615292.880276 trunc from 0x0018ef to 0x00c2a2 (0xa9b4 bytes) 000541 1787615292.888365 trunc from 0x03ece1 to 0x03ed29 (0x49 bytes) 000543 1787615292.897785 mapread 0x029b42 thru 0x032b40 (0x8fff bytes) 000544 1787615292.899933 trunc from 0x033ecf to 0x03bca7 (0x7dd9 bytes) 000545 1787615292.914884 MIRROR_RESYNC 5 mirrors 000546 1787615294.363858 fallocate 0x0111e6 thru 0x015d8d (0x4ba8 bytes) 000547 1787615294.386495 mapread 0x022487 thru 0x029e6c (0x79e6 bytes) 000548 1787615294.409172 mapwrite 0x013da6 thru 0x01954e (0x57a9 bytes) 000549 1787615294.446834 trunc from 0x03ed2a to 0x005034 (0x39cf5 bytes) 000550 1787615294.474795 fallocate 0x014279 thru 0x01ffb8 (0xbd40 bytes) 000551 1787615294.484407 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ffb9): Mirror 3: 0x4d230e19 (stale) Mirror 4: 0x4d230e19 (stale) Mirror 6: 0x4d230e19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ffb9): Mirror 3: 0x4d230e19 (stale) Mirror 4: 0x4d230e19 (stale) Mirror 6: 0x4d230e19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ffb9: skipped 000552 1787615294.806607 trunc from 0x01ffb9 to 0x037af0 (0x17b38 bytes) 000553 1787615294.824964 mapread 0x00ef00 thru 0x0134c8 (0x45c9 bytes) 000555 1787615294.847214 fallocate 0x00ecd1 thru 0x01bdb8 (0xd0e8 bytes) 000556 1787615294.859572 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37af1): Mirror 3: 0x92976023 (stale) Mirror 6: 0x92976023 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37af1): Mirror 3: 0x92976023 (stale) Mirror 6: 0x92976023 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37af1: skipped 000557 1787615295.291302 write 0x026f47 thru 0x02c8d6 (0x5990 bytes) 000558 1787615295.331442 mapread 0x0170f3 thru 0x01fb70 (0x8a7e bytes) 000559 1787615295.373779 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37af1): Mirror 6: 0x92976023 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37af1): Mirror 6: 0x92976023 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37af1: skipped 000560 1787615295.855708 punch from 0x014860 to 0x018dc0 (0x4561 bytes) 000561 1787615295.861740 fallocate 0x00e491 thru 0x0134d2 (0x5042 bytes) 000563 1787615295.870816 fallocate 0x0021e9 thru 0x005e49 (0x3c61 bytes) 000564 1787615295.875733 trunc from 0x037af1 to 0x03f853 (0x7d63 bytes) 000565 1787615295.902644 mapwrite 0x00e6cb thru 0x019dbc (0xb6f2 bytes) 000566 1787615295.953244 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) 000567 1787615296.051065 trunc from 0x0223a9 to 0x025604 (0x325c bytes) 000568 1787615296.055581 punch from 0x012c68 to 0x01af74 (0x830d bytes) 000569 1787615296.061004 mapwrite 0x02da8a thru 0x038d13 (0xb28a bytes) 000570 1787615296.095664 mapwrite 0x036b33 thru 0x03ffff (0x94cd bytes) 000571 1787615296.121582 mapwrite 0x039c27 thru 0x03cd43 (0x311d bytes) 000572 1787615296.133532 punch from 0x03ceb7 to 0x03ffff (0x3149 bytes) 000573 1787615296.144796 read 0x03b167 thru 0x03ffff (0x4e99 bytes) 000574 1787615296.153752 punch from 0x0021cf to 0x00a6a8 (0x84da bytes) 000575 1787615296.159957 trunc from 0x040000 to 0x015e7f (0x2a180 bytes) 000576 1787615296.172986 read 0x0027f7 thru 0x00bc5a (0x9464 bytes) 000577 1787615296.181693 punch from 0x0019d3 to 0x009ac1 (0x80ef bytes) 000578 1787615296.189749 punch from 0x013d6f to 0x015e7f (0x2111 bytes) 000579 1787615296.195802 punch from 0x00b94b to 0x015e7f (0xa535 bytes) 000580 1787615296.203697 mapwrite 0x02488a thru 0x02f267 (0xa9de bytes) 000581 1787615296.276734 trunc from 0x02f268 to 0x014f49 (0x1a31e bytes) 000582 1787615296.311798 punch from 0x00db1c to 0x014f49 (0x742e bytes) 000583 1787615296.319737 read 0x011cbf thru 0x014f49 (0x328b bytes) 000584 1787615296.336545 mapwrite 0x012576 thru 0x01f895 (0xd320 bytes) 000585 1787615296.414550 fallocate 0x034fea thru 0x03fb32 (0xab49 bytes) 000586 1787615296.421836 mapread 0x021c96 thru 0x022519 (0x884 bytes) 000587 1787615296.436030 read 0x00e0c0 thru 0x01d58b (0xf4cc bytes) 000588 1787615296.437909 trunc from 0x03fb33 to 0x0001d9 (0x3f959 bytes) 000589 1787615296.465965 mapread 0x000028 thru 0x0001d9 (0x1b2 bytes) 000591 1787615296.467393 punch from 0x0001bf to 0x0001d9 (0x1b bytes) 000592 1787615296.472149 trunc from 0x0001da to 0x013adb (0x13902 bytes) 000593 1787615296.488807 trunc from 0x013adc to 0x008394 (0xb747 bytes) 000594 1787615296.514835 fallocate 0x018c27 thru 0x026594 (0xd96e bytes) 000595 1787615296.529716 fallocate 0x03a6be thru 0x03ffff (0x5942 bytes) 000596 1787615296.535522 punch from 0x003934 to 0x003c05 (0x2d2 bytes) 000597 1787615296.540747 write 0x013f80 thru 0x019021 (0x50a2 bytes) 000598 1787615296.543052 punch from 0x015540 to 0x019021 (0x3ae2 bytes) 000599 1787615296.563246 trunc from 0x019022 to 0x011f93 (0x708e bytes) 000600 1787615296.579574 write 0x00ae25 thru 0x016b64 (0xbd40 bytes) 000601 1787615296.595701 punch from 0x005ed9 to 0x00b1c5 (0x52ed bytes) 000602 1787615296.618580 mapread 0x012f31 thru 0x016b64 (0x3c34 bytes) 000603 1787615296.619617 punch from 0x00e838 to 0x016b64 (0x832d bytes) 000604 1787615296.625560 trunc from 0x006583 to 0x00b308 (0x4d86 bytes) 000605 1787615296.631456 trunc from 0x016b65 to 0x035aca (0x1ef66 bytes) 000606 1787615296.658723 fallocate 0x031f05 thru 0x03fb19 (0xdc15 bytes) 000607 1787615296.665258 punch from 0x00ff78 to 0x01ac40 (0xacc9 bytes) 000608 1787615296.671934 punch from 0x01da97 to 0x02a730 (0xcc9a bytes) 000610 1787615296.677866 mapread 0x006ce6 thru 0x00ea2b (0x7d46 bytes) 000611 1787615296.723493 read 0x015782 thru 0x018999 (0x3218 bytes) 000612 1787615296.724492 trunc from 0x027110 to 0x02e7ce (0x76bf bytes) 000613 1787615296.731354 read 0x03aab5 thru 0x03d461 (0x29ad bytes) 000614 1787615296.731899 write 0x03a7eb thru 0x03ffff (0x5815 bytes) 000615 1787615296.732929 write 0x03688e thru 0x03aed2 (0x4645 bytes) 000616 1787615296.733884 fallocate 0x023f3e thru 0x02a900 (0x69c3 bytes) 000618 1787615296.742473 mapwrite 0x033925 thru 0x037a7d (0x4159 bytes) 000622 1787615296.784976 MIRROR_EXTEND to 2 mirrors 000623 1787615297.469558 punch from 0x008f26 to 0x00fdbd (0x6e98 bytes) 000624 1787615297.503407 mapread 0x000a7c thru 0x003db4 (0x3339 bytes) 000625 1787615297.528545 fallocate 0x024f74 thru 0x0267a9 (0x1836 bytes) 000626 1787615297.537160 MIRROR_RESYNC 2 mirrors 000627 1787615298.198580 trunc from 0x040000 to 0x0266d0 (0x1992f bytes) 000628 1787615298.234476 trunc from 0x0081ba to 0x0141ae (0xbff5 bytes) 000629 1787615298.239582 write 0x010c4f thru 0x01387c (0x2c2e bytes) 000630 1787615298.248494 fallocate 0x01f0c9 thru 0x02311a (0x4052 bytes) 000631 1787615298.253283 read 0x013080 thru 0x01913a (0x60bb bytes) 000632 1787615298.265523 read 0x00a541 thru 0x00f12a (0x4bea bytes) 000633 1787615298.275933 trunc from 0x0266d1 to 0x01114b (0x15585 bytes) 000634 1787615298.290484 punch from 0x00f259 to 0x01114b (0x1ef3 bytes) 000635 1787615298.297686 punch from 0x00dfdc to 0x01114b (0x3170 bytes) 000636 1787615298.301992 read 0x006f5c thru 0x01114b (0xa1f0 bytes) 000637 1787615298.308901 trunc from 0x01114c to 0x023689 (0x1253e bytes) 000638 1787615298.322519 mapwrite 0x022365 thru 0x023ad3 (0x176f bytes) 000639 1787615298.373272 mapread 0x0014e4 thru 0x00e72f (0xd24c bytes) 000640 1787615298.374570 punch from 0x018be0 to 0x023ad3 (0xaef4 bytes) 000641 1787615298.380591 mapread 0x014540 thru 0x019f60 (0x5a21 bytes) 000642 1787615298.390559 trunc from 0x0199d6 to 0x023ad3 (0xa0fe bytes) 000643 1787615298.399547 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x23ad4): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23ad4): Mirror 2: 0x32df4b10 (stale) CRC-32 checksum value for chunk [0, 0x23ad4): Mirror 1: 0x32df4b10 Mirror 3: 0x32df4b10 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23ad4: skipped 000644 1787615299.319881 read 0x0212e5 thru 0x021d3a (0xa56 bytes) 000645 1787615299.329849 mapwrite 0x0313a2 thru 0x03467d (0x32dc bytes) 000646 1787615299.485467 fallocate 0x036896 thru 0x03ffff (0x976a bytes) 000647 1787615299.492224 mapread 0x01ca58 thru 0x01fa07 (0x2fb0 bytes) 000648 1787615299.504272 trunc from 0x040000 to 0x0260c4 (0x19f3b bytes) 000649 1787615299.530445 mapread 0x009bcc thru 0x01792c (0xdd61 bytes) 000650 1787615299.546015 trunc from 0x01eb76 to 0x0260c4 (0x754f bytes) 000651 1787615299.555323 MIRROR_RESYNC 3 mirrors 000652 1787615300.207566 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) 000653 1787615300.516245 trunc from 0x01971d to 0x0260c4 (0xc9a8 bytes) 000654 1787615300.554149 read 0x007d70 thru 0x013ffe (0xc28f bytes) 000655 1787615300.567242 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) 000656 1787615300.743718 fallocate 0x00e3e1 thru 0x01bee2 (0xdb02 bytes) 000657 1787615300.752571 trunc from 0x022d9e to 0x0260c4 (0x3327 bytes) 000658 1787615300.759126 trunc from 0x0260c5 to 0x0111e1 (0x14ee3 bytes) 000659 1787615300.828390 punch from 0x00f353 to 0x0111e1 (0x1e8f bytes) 000660 1787615300.833955 mapread 0x00bc1a thru 0x0111e1 (0x55c8 bytes) 000661 1787615300.855616 write 0x01f714 thru 0x02e87d (0xf16a bytes) 000662 1787615300.857609 trunc from 0x02e87e to 0x000bcc (0x2dcb1 bytes) 000663 1787615300.892691 fallocate 0x0143ca thru 0x017f25 (0x3b5c bytes) 000664 1787615300.912671 MIRROR_EXTEND to 2 mirrors 000665 1787615301.422578 mapwrite 0x023721 thru 0x027d75 (0x4655 bytes) 000666 1787615301.507848 trunc from 0x027d76 to 0x01b8d4 (0xc4a1 bytes) 000667 1787615301.529200 punch from 0x010ea8 to 0x0140d7 (0x3230 bytes) 000668 1787615301.539703 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1b8d5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b8d5): Mirror 2: 0x8b3c4a14 (stale) CRC-32 checksum value for chunk [0, 0x1b8d5): Mirror 1: 0x8b3c4a14 Mirror 3: 0x8b3c4a14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b8d5: skipped 000669 1787615302.186916 mapwrite 0x00c565 thru 0x01b2f1 (0xed8d bytes) 000670 1787615302.304673 MIRROR_RESYNC 3 mirrors 000671 1787615303.102741 read 0x01887c thru 0x019670 (0xdf5 bytes) 000672 1787615303.111374 trunc from 0x01b8d5 to 0x0206ed (0x4e19 bytes) 000673 1787615303.155895 fallocate 0x00b850 thru 0x011215 (0x59c6 bytes) 000674 1787615303.161169 write 0x033e4b thru 0x03ab00 (0x6cb6 bytes) 000675 1787615303.162857 fallocate 0x03d48d thru 0x03d68e (0x202 bytes) 000676 1787615303.167998 read 0x011fec thru 0x0177a5 (0x57ba bytes) 000677 1787615303.178496 mapwrite 0x035758 thru 0x0395f4 (0x3e9d bytes) 000678 1787615303.194666 write 0x00db86 thru 0x01c353 (0xe7ce bytes) 000679 1787615303.200575 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ab01): Mirror 3: 0xc399d3f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab01): Mirror 3: 0xc399d3f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab01: skipped 000680 1787615303.540699 trunc from 0x03ab01 to 0x00a4e4 (0x3061c bytes) 000681 1787615303.566614 trunc from 0x00a4e5 to 0x01e86d (0x14389 bytes) 000683 1787615303.583954 write 0x0249b9 thru 0x02d6a0 (0x8ce8 bytes) 000684 1787615303.585796 punch from 0x02118a to 0x023712 (0x2589 bytes) 000685 1787615303.594728 mapread 0x0068a5 thru 0x00fd4d (0x94a9 bytes) 000687 1787615303.612967 mapwrite 0x010a7e thru 0x017863 (0x6de6 bytes) 000688 1787615303.641813 trunc from 0x00066b to 0x00c8a7 (0xc23d bytes) 000689 1787615303.647143 punch from 0x0010cf to 0x009381 (0x82b3 bytes) 000690 1787615303.658547 MIRROR_RESYNC 2 mirrors 000692 1787615304.412329 fallocate 0x00760a thru 0x00f06b (0x7a62 bytes) 000693 1787615304.462115 write 0x020d3d thru 0x02e945 (0xdc09 bytes) 000694 1787615304.471854 fallocate 0x0128fc thru 0x019b85 (0x728a bytes) 000695 1787615304.477672 mapread 0x0187f9 thru 0x01c995 (0x419d bytes) 000696 1787615304.504424 punch from 0x022b80 to 0x0247ed (0x1c6e bytes) 000697 1787615304.537722 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e946): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e946): Mirror 3: 0xb7ac27b0 (stale) CRC-32 checksum value for chunk [0, 0x2e946): Mirror 1: 0xfc107235 Mirror 4: 0xfc107235 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e946: skipped 000698 1787615305.933445 fallocate 0x01c6f5 thru 0x021053 (0x495f bytes) 000699 1787615305.978622 mapread 0x016477 thru 0x0229b0 (0xc53a bytes) 000700 1787615306.034522 mapread 0x01554c thru 0x02150b (0xbfc0 bytes) 000701 1787615306.040343 mapread 0x0055ad thru 0x00c80b (0x725f bytes) 000702 1787615306.060589 mapwrite 0x008ae8 thru 0x009e8e (0x13a7 bytes) 000703 1787615306.073387 fallocate 0x001353 thru 0x002b5a (0x1808 bytes) 000704 1787615306.078828 read 0x02237f thru 0x02cd55 (0xa9d7 bytes) 000705 1787615306.084260 mapwrite 0x036c49 thru 0x03ffff (0x93b7 bytes) 000706 1787615306.137842 trunc from 0x00f165 to 0x01a57e (0xb41a bytes) 000707 1787615306.143688 trunc from 0x001f16 to 0x00c0f5 (0xa1e0 bytes) 000708 1787615306.150246 trunc from 0x0205a5 to 0x029e51 (0x98ad bytes) 000709 1787615306.155976 trunc from 0x011881 to 0x011e27 (0x5a7 bytes) 000710 1787615306.160731 read 0x00edf7 thru 0x01c0d2 (0xd2dc bytes) 000711 1787615306.189904 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf71f253 (stale) Mirror 4: 0x885f1d6f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd9936d62 Mirror 5: 0xd9936d62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000712 1787615306.953777 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf71f253 (stale) Mirror 4: 0x885f1d6f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd9936d62 Mirror 5: 0xd9936d62 Mirror 6: 0xd9936d62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000713 1787615308.185868 mapread 0x02c197 thru 0x036298 (0xa102 bytes) 000714 1787615308.233717 write 0x02e29b thru 0x03c119 (0xde7f bytes) 000715 1787615308.297097 punch from 0x01fda1 to 0x02c047 (0xc2a7 bytes) 000716 1787615308.305066 trunc from 0x002408 to 0x00b2cc (0x8ec5 bytes) 000717 1787615308.328467 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf71f253 (stale) Mirror 4: 0x885f1d6f (stale) Mirror 5: 0xd9936d62 (stale) Mirror 6: 0xd9936d62 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x39c9c28f Mirror 7: 0x39c9c28f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000718 1787615309.670443 write 0x0240e3 thru 0x033d2b (0xfc49 bytes) 000719 1787615309.730829 mapwrite 0x035ef0 thru 0x03ffff (0xa110 bytes) 000720 1787615309.771107 mapwrite 0x00ab2d thru 0x0123b4 (0x7888 bytes) 000721 1787615309.803893 mapwrite 0x015bde thru 0x01fc67 (0xa08a bytes) 000722 1787615309.844592 write 0x003622 thru 0x008caa (0x5689 bytes) 000723 1787615309.847489 write 0x02b90d thru 0x03747d (0xbb71 bytes) 000724 1787615309.848927 trunc from 0x035ddb to 0x038652 (0x2878 bytes) 000725 1787615309.870878 mapread 0x029dfb thru 0x034021 (0xa227 bytes) 000726 1787615309.873663 read 0x034563 thru 0x03d46e (0x8f0c bytes) 000727 1787615309.881411 mapwrite 0x00b138 thru 0x013629 (0x84f2 bytes) 000728 1787615309.902844 trunc from 0x040000 to 0x01d4c4 (0x22b3b bytes) 000729 1787615309.929355 mapread 0x01131c thru 0x011900 (0x5e5 bytes) 000730 1787615309.930322 write 0x021d7a thru 0x02e9f2 (0xcc79 bytes) 000731 1787615309.934233 trunc from 0x02e9f3 to 0x00cc1e (0x21dd4 bytes) 000732 1787615309.952826 mapwrite 0x002727 thru 0x009a26 (0x7300 bytes) 000733 1787615309.977685 read 0x002024 thru 0x00cc1e (0xabfb bytes) 000734 1787615309.978322 write 0x02d82e thru 0x039d1c (0xc4ef bytes) 000735 1787615309.979906 punch from 0x0158c5 to 0x018995 (0x30d1 bytes) 000736 1787615309.986810 mapread 0x01d5ca thru 0x01eaa4 (0x14db bytes) 000737 1787615309.996407 fallocate 0x0004cf thru 0x003722 (0x3254 bytes) 000738 1787615310.001126 trunc from 0x039d1d to 0x0206e2 (0x1963a bytes) 000739 1787615310.014825 trunc from 0x012787 to 0x0196c5 (0x6f3f bytes) 000740 1787615310.021171 read 0x00c0aa thru 0x01677c (0xa6d3 bytes) 000741 1787615310.030774 mapread 0x00f37e thru 0x01f0ca (0xfd4d bytes) 000742 1787615310.036195 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x206e3): Mirror 3: 0xc961fe (stale) Mirror 4: 0xc961fe (stale) Mirror 5: 0x9993c44b (stale) Mirror 6: 0x9993c44b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x206e3): Mirror 3: 0xc961fe (stale) Mirror 4: 0xc961fe (stale) Mirror 5: 0x9993c44b (stale) Mirror 6: 0x9993c44b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x206e3: skipped 000743 1787615310.409936 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x206e3): Mirror 4: 0xc961fe (stale) Mirror 5: 0x9993c44b (stale) Mirror 6: 0x9993c44b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x206e3): Mirror 4: 0xc961fe (stale) Mirror 5: 0x9993c44b (stale) Mirror 6: 0x9993c44b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x206e3: skipped 000744 1787615310.935137 mapwrite 0x019c7e thru 0x0244cd (0xa850 bytes) 000745 1787615311.003667 mapread 0x008876 thru 0x015b81 (0xd30c bytes) 000746 1787615311.019434 mapwrite 0x006000 thru 0x00ed6f (0x8d70 bytes) 000747 1787615311.038029 trunc from 0x0244ce to 0x0310b4 (0xcbe7 bytes) 000748 1787615311.052393 punch from 0x012f45 to 0x018add (0x5b99 bytes) 000750 1787615311.058040 mapread 0x00c762 thru 0x00ea17 (0x22b6 bytes) 000751 1787615311.058959 read 0x030de6 thru 0x0310b4 (0x2cf bytes) 000752 1787615311.064640 mapwrite 0x007c7c thru 0x016107 (0xe48c bytes) 000753 1787615311.101709 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x310b5): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x310b5): Mirror 4: 0xbd3b6909 (stale) Mirror 5: 0xf3b93555 (stale) Mirror 6: 0xf3b93555 (stale) CRC-32 checksum value for chunk [0, 0x310b5): Mirror 1: 0xdf86517d Mirror 7: 0xdf86517d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x310b5: skipped 000754 1787615311.868669 mapwrite 0x03339d thru 0x038009 (0x4c6d bytes) 000755 1787615311.995326 write 0x01b567 thru 0x01e64f (0x30e9 bytes) 000756 1787615312.010273 mapread 0x00ecd8 thru 0x011b54 (0x2e7d bytes) 000757 1787615312.021367 mapwrite 0x022938 thru 0x030098 (0xd761 bytes) 000758 1787615312.055388 mapread 0x037624 thru 0x038009 (0x9e6 bytes) 000759 1787615312.056296 mapread 0x007339 thru 0x00d6de (0x63a6 bytes) 000760 1787615312.057116 trunc from 0x03800a to 0x009d13 (0x2e2f6 bytes) 000761 1787615312.081905 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9d14): Mirror 4: 0x9713a466 (stale) Mirror 6: 0x9713a466 (stale) Mirror 7: 0xa7cc1821 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9d14): Mirror 4: 0x9713a466 (stale) Mirror 6: 0x9713a466 (stale) Mirror 7: 0xa7cc1821 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9d14: skipped 000762 1787615312.384604 mapwrite 0x03c221 thru 0x03ffff (0x3ddf bytes) 000764 1787615312.437289 write 0x012e5f thru 0x01961a (0x67bc bytes) 000765 1787615312.449008 mapread 0x035325 thru 0x03dae7 (0x87c3 bytes) 000766 1787615312.463987 trunc from 0x00b33d to 0x01a02d (0xecf1 bytes) 000767 1787615312.482375 punch from 0x03b556 to 0x03ffff (0x4aaa bytes) 000768 1787615312.487241 read 0x01ed32 thru 0x02d869 (0xeb38 bytes) 000769 1787615312.497309 fallocate 0x00d98b thru 0x017479 (0x9aef bytes) 000770 1787615312.502753 mapread 0x017d5d thru 0x02595e (0xdc02 bytes) 000771 1787615312.523168 trunc from 0x040000 to 0x0324e2 (0xdb1d bytes) 000772 1787615312.544208 read 0x02a078 thru 0x0324e2 (0x846b bytes) 000773 1787615312.544898 fallocate 0x0028a2 thru 0x007991 (0x50f0 bytes) 000774 1787615312.551342 trunc from 0x01f59b to 0x021dd8 (0x283e bytes) 000775 1787615312.556489 read 0x00e500 thru 0x011d6a (0x386b bytes) 000778 1787615312.569282 fallocate 0x0366e7 thru 0x03ffff (0x9919 bytes) 000779 1787615312.574407 trunc from 0x0324e3 to 0x03ae94 (0x89b2 bytes) 000780 1787615312.592246 fallocate 0x00c351 thru 0x016414 (0xa0c4 bytes) 000781 1787615312.602213 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ae95): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae95): Mirror 4: 0xb819294a (stale) Mirror 6: 0xee8a06dd (stale) Mirror 7: 0x29acc2c9 (stale) CRC-32 checksum value for chunk [0, 0x3ae95): Mirror 1: 0xa767f6a7 Mirror 8: 0xa767f6a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae95: skipped 000782 1787615313.631666 punch from 0x038773 to 0x03ae94 (0x2722 bytes) 000783 1787615313.697506 mapread 0x03752b thru 0x03ae94 (0x396a bytes) 000784 1787615313.727773 mapread 0x00281a thru 0x00bd3b (0x9522 bytes) 000785 1787615313.737276 write 0x0288e2 thru 0x028b63 (0x282 bytes) 000786 1787615313.746316 mapwrite 0x03eb2d thru 0x03ffff (0x14d3 bytes) 000787 1787615313.883051 fallocate 0x00fb2c thru 0x018cbe (0x9193 bytes) 000789 1787615313.892682 trunc from 0x032a75 to 0x03d18a (0xa716 bytes) 000790 1787615313.905544 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x885f1d6f (stale) Mirror 6: 0xd9936d62 (stale) Mirror 7: 0xad31bf9a (stale) Mirror 8: 0xf62bac61 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6cdc469 Mirror 9: 0x6cdc469 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000791 1787615315.133900 write 0x03d49d thru 0x03e471 (0xfd5 bytes) 000792 1787615315.189865 write 0x018fae thru 0x021ce2 (0x8d35 bytes) 000794 1787615315.204534 mapread 0x036902 thru 0x03d8c3 (0x6fc2 bytes) 000795 1787615315.224395 mapwrite 0x02a45f thru 0x02c378 (0x1f1a bytes) 000796 1787615315.256942 write 0x004c42 thru 0x00de25 (0x91e4 bytes) 000797 1787615315.273990 mapwrite 0x00130d thru 0x00483d (0x3531 bytes) 000798 1787615315.298335 fallocate 0x02c856 thru 0x03a717 (0xdec2 bytes) 000799 1787615315.302969 trunc from 0x040000 to 0x01bd00 (0x242ff bytes) 000800 1787615315.321888 mapread 0x002da9 thru 0x005829 (0x2a81 bytes) 000801 1787615315.323815 punch from 0x018ccb to 0x01bd00 (0x3036 bytes) 000802 1787615315.330038 punch from 0x00a409 to 0x00d6c0 (0x32b8 bytes) 000803 1787615315.335207 fallocate 0x027fa2 thru 0x02ee2b (0x6e8a bytes) 000804 1787615315.340013 trunc from 0x0087f7 to 0x00b5dc (0x2de6 bytes) 000805 1787615315.346333 trunc from 0x02ee2c to 0x027b75 (0x72b6 bytes) 000806 1787615315.364556 mapread 0x025649 thru 0x027b75 (0x252d bytes) 000809 1787615315.378715 read 0x02096d thru 0x027b75 (0x7209 bytes) 000810 1787615315.380277 read 0x022a1f thru 0x027b75 (0x5157 bytes) 000811 1787615315.386508 MIRROR_RESYNC 6 mirrors 000812 1787615317.108964 trunc from 0x027b76 to 0x003afb (0x2407a bytes) 000813 1787615317.166153 fallocate 0x007ae1 thru 0x012ac0 (0xafe0 bytes) 000814 1787615317.175530 mapwrite 0x03d5c9 thru 0x03d917 (0x34f bytes) 000815 1787615317.256693 fallocate 0x0361fa thru 0x03b529 (0x5330 bytes) 000816 1787615317.265471 write 0x002cf7 thru 0x004fd6 (0x22e0 bytes) 000817 1787615317.279580 fallocate 0x039cad thru 0x03ffff (0x6353 bytes) 000818 1787615317.285786 mapwrite 0x00c8ad thru 0x016d45 (0xa499 bytes) 000819 1787615317.350468 mapread 0x00f8a9 thru 0x0176a9 (0x7e01 bytes) 000820 1787615317.352877 mapwrite 0x03616f thru 0x03ffff (0x9e91 bytes) 000821 1787615317.468138 read 0x00a059 thru 0x015209 (0xb1b1 bytes) 000822 1787615317.471492 fallocate 0x018a1d thru 0x027fed (0xf5d1 bytes) 000823 1787615317.481035 punch from 0x01becd to 0x022d7d (0x6eb1 bytes) 000824 1787615317.487787 mapread 0x009118 thru 0x014868 (0xb751 bytes) 000825 1787615317.490783 fallocate 0x0076d5 thru 0x00a6d0 (0x2ffc bytes) 000826 1787615317.497484 write 0x01dde2 thru 0x01f606 (0x1825 bytes) 000827 1787615317.509777 trunc from 0x040000 to 0x0105c1 (0x2fa3e bytes) 000828 1787615317.538156 write 0x037cf1 thru 0x03ffff (0x830f bytes) 000829 1787615317.540310 punch from 0x035019 to 0x036ca2 (0x1c8a bytes) 000830 1787615317.546062 read 0x027cc2 thru 0x036681 (0xe9c0 bytes) 000831 1787615317.553406 mapread 0x03a33e thru 0x03ffff (0x5cc2 bytes) 000832 1787615317.554674 write 0x03fd1b thru 0x03ffff (0x2e5 bytes) 000833 1787615317.556396 trunc from 0x010ab7 to 0x016f2a (0x6474 bytes) 000834 1787615317.562041 read 0x007314 thru 0x00dbc0 (0x68ad bytes) 000835 1787615317.562876 punch from 0x0138f6 to 0x016715 (0x2e20 bytes) 000837 1787615317.570543 trunc from 0x040000 to 0x033dcd (0xc232 bytes) 000838 1787615317.593224 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33dce): Mirror 4: 0xc1025464 (stale) Mirror 6: 0xc1025464 (stale) Mirror 7: 0xc1025464 (stale) Mirror 8: 0xc1025464 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33dce): Mirror 4: 0xc1025464 (stale) Mirror 6: 0xc1025464 (stale) Mirror 7: 0xc1025464 (stale) Mirror 8: 0xc1025464 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33dce: skipped 000839 1787615317.984925 mapwrite 0x01a74d thru 0x0206e6 (0x5f9a bytes) 000840 1787615318.019209 mapread 0x02cb98 thru 0x02e873 (0x1cdc bytes) 000841 1787615318.034523 write 0x01e4fa thru 0x023cbc (0x57c3 bytes) 000842 1787615318.038984 mapwrite 0x0105fc thru 0x015333 (0x4d38 bytes) 000843 1787615318.082613 write 0x022381 thru 0x031b98 (0xf818 bytes) 000844 1787615318.085740 punch from 0x012609 to 0x019ec2 (0x78ba bytes) 000845 1787615318.099610 fallocate 0x00232d thru 0x00d5ee (0xb2c2 bytes) 000846 1787615318.104737 mapread 0x02f375 thru 0x030496 (0x1122 bytes) 000847 1787615318.106010 mapwrite 0x003396 thru 0x00b86d (0x84d8 bytes) 000848 1787615318.139978 trunc from 0x02b703 to 0x033911 (0x820f bytes) 000849 1787615318.150726 read 0x030d15 thru 0x033dcd (0x30b9 bytes) 000850 1787615318.158414 mapread 0x014a68 thru 0x01b83d (0x6dd6 bytes) 000851 1787615318.166513 mapread 0x01f679 thru 0x02c4b6 (0xce3e bytes) 000852 1787615318.178699 trunc from 0x033dce to 0x001c97 (0x32136 bytes) 000853 1787615318.202868 write 0x01a0c7 thru 0x020500 (0x643a bytes) 000854 1787615318.204293 mapread 0x014827 thru 0x020500 (0xbcda bytes) 000855 1787615318.220391 trunc from 0x00adef to 0x00f185 (0x4397 bytes) 000857 1787615318.242546 trunc from 0x007cd8 to 0x01221f (0xa548 bytes) 000858 1787615318.262541 write 0x03bd1c thru 0x03c95b (0xc40 bytes) 000859 1787615318.264128 trunc from 0x03c95c to 0x0323a5 (0xa5b6 bytes) 000860 1787615318.306285 mapwrite 0x03b277 thru 0x03ffff (0x4d89 bytes) 000861 1787615318.377040 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde26a3aa (stale) Mirror 6: 0xde26a3aa (stale) Mirror 7: 0xde26a3aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde26a3aa (stale) Mirror 6: 0xde26a3aa (stale) Mirror 7: 0xde26a3aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000862 1787615318.735439 mapwrite 0x014e0b thru 0x0161f2 (0x13e8 bytes) 000863 1787615318.779123 trunc from 0x040000 to 0x00b715 (0x348ea bytes) 000865 1787615318.812063 fallocate 0x022d74 thru 0x02734b (0x45d8 bytes) 000866 1787615318.819480 mapread 0x00a749 thru 0x00b715 (0xfcd bytes) 000867 1787615318.830591 trunc from 0x0008c7 to 0x00b715 (0xae4f bytes) 000869 1787615318.837041 trunc from 0x004cae to 0x00b715 (0x6a68 bytes) 000870 1787615318.845196 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xb716): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb716): Mirror 4: 0x5e5c2987 (stale) Mirror 6: 0x5e5c2987 (stale) Mirror 7: 0x5e5c2987 (stale) CRC-32 checksum value for chunk [0, 0xb716): Mirror 1: 0x3dbdf26d Mirror 8: 0x3dbdf26d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb716: skipped 000871 1787615319.698327 mapwrite 0x01e343 thru 0x01fbfa (0x18b8 bytes) 000872 1787615319.800524 mapwrite 0x01d2c4 thru 0x025692 (0x83cf bytes) 000873 1787615319.867155 mapread 0x0214f4 thru 0x025692 (0x419f bytes) 000874 1787615319.868205 fallocate 0x01533c thru 0x0212ce (0xbf93 bytes) 000875 1787615319.877182 fallocate 0x030dae thru 0x034b2c (0x3d7f bytes) 000876 1787615319.887911 read 0x000ece thru 0x00710b (0x623e bytes) 000877 1787615319.899162 fallocate 0x017e0e thru 0x021ad2 (0x9cc5 bytes) 000878 1787615319.904464 trunc from 0x034b2d to 0x00db77 (0x26fb5 bytes) 000879 1787615319.915332 trunc from 0x00db78 to 0x0276ce (0x19b57 bytes) 000880 1787615319.926659 punch from 0x013b8d to 0x0205cc (0xca40 bytes) 000881 1787615319.934571 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x276cf): Mirror 4: 0x6f18d4fc (stale) Mirror 6: 0x6f18d4fc (stale) Mirror 8: 0xa855556e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x276cf): Mirror 4: 0x6f18d4fc (stale) Mirror 6: 0x6f18d4fc (stale) Mirror 8: 0xa855556e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x276cf: skipped 000882 1787615320.365638 trunc from 0x0276cf to 0x038d20 (0x11652 bytes) 000884 1787615320.402931 punch from 0x0059a1 to 0x00fe6a (0xa4ca bytes) 000885 1787615320.416903 write 0x009a99 thru 0x00bc51 (0x21b9 bytes) 000887 1787615320.430908 trunc from 0x038d21 to 0x01d224 (0x1bafc bytes) 000888 1787615320.466815 mapwrite 0x00cd56 thru 0x013a60 (0x6d0b bytes) 000889 1787615320.506849 read 0x010269 thru 0x01aad4 (0xa86c bytes) 000890 1787615320.518256 fallocate 0x00ae79 thru 0x00f817 (0x499f bytes) 000891 1787615320.524940 mapwrite 0x01a550 thru 0x027591 (0xd042 bytes) 000892 1787615320.587450 read 0x004d52 thru 0x011320 (0xc5cf bytes) 000893 1787615320.597534 mapwrite 0x03e48a thru 0x03ffff (0x1b76 bytes) 000894 1787615320.668597 read 0x025e97 thru 0x027727 (0x1891 bytes) 000895 1787615320.683199 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde26a3aa (stale) Mirror 6: 0xde26a3aa (stale) Mirror 8: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4047386a Mirror 9: 0x4047386a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000896 1787615321.729810 mapwrite 0x02fc49 thru 0x03b166 (0xb51e bytes) 000898 1787615321.881515 punch from 0x03186a to 0x039011 (0x77a8 bytes) 000899 1787615321.901969 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde26a3aa (stale) Mirror 6: 0xde26a3aa (stale) Mirror 8: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde26a3aa (stale) Mirror 6: 0xde26a3aa (stale) Mirror 8: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000900 1787615322.233669 fallocate 0x016585 thru 0x02155b (0xafd7 bytes) 000901 1787615322.242618 read 0x013643 thru 0x01eee2 (0xb8a0 bytes) 000902 1787615322.252448 trunc from 0x040000 to 0x00bb7e (0x34481 bytes) 000903 1787615322.293313 trunc from 0x00bb7f to 0x022930 (0x16db2 bytes) 000904 1787615322.317399 mapwrite 0x01f0ae thru 0x020dea (0x1d3d bytes) 000905 1787615322.351563 mapread 0x01967c thru 0x01dda8 (0x472d bytes) 000906 1787615322.366008 fallocate 0x01f1f4 thru 0x022067 (0x2e74 bytes) 000907 1787615322.375491 trunc from 0x022931 to 0x003294 (0x1f69c bytes) 000908 1787615322.408936 mapread 0x0009b9 thru 0x003294 (0x28dc bytes) 000909 1787615322.413887 fallocate 0x03200e thru 0x0381b0 (0x61a3 bytes) 000910 1787615322.420550 trunc from 0x001b53 to 0x003294 (0x1742 bytes) 000911 1787615322.427670 mapwrite 0x01747e thru 0x01a6da (0x325d bytes) 000912 1787615322.480250 mapwrite 0x01f4ac thru 0x025c95 (0x67ea bytes) 000913 1787615322.550256 trunc from 0x004d56 to 0x012680 (0xd92b bytes) 000914 1787615322.559640 mapwrite 0x014858 thru 0x0231f9 (0xe9a2 bytes) 000916 1787615322.607411 punch from 0x017f34 to 0x02465b (0xc728 bytes) 000917 1787615322.613384 write 0x03ed36 thru 0x03ffff (0x12ca bytes) 000919 1787615322.615038 punch from 0x02c909 to 0x0342a3 (0x799b bytes) 000921 1787615322.632408 punch from 0x001fb9 to 0x00a5c9 (0x8611 bytes) 000922 1787615322.638292 trunc from 0x02c896 to 0x02e39b (0x1b06 bytes) 000923 1787615322.643261 mapwrite 0x03d475 thru 0x03ffff (0x2b8b bytes) 000924 1787615322.680016 mapwrite 0x00463f thru 0x00c543 (0x7f05 bytes) 000926 1787615322.728531 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde26a3aa (stale) Mirror 6: 0xde26a3aa (stale) Mirror 8: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x21cd5261 Mirror 9: 0x21cd5261 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000927 1787615323.902495 trunc from 0x038e1a to 0x039ab0 (0xc97 bytes) 000928 1787615323.952351 mapwrite 0x0263df thru 0x031ee2 (0xbb04 bytes) 000929 1787615324.026357 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde26a3aa (stale) Mirror 6: 0xde26a3aa (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0x21cd5261 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb57f1719 Mirror 10: 0xb57f1719 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000930 1787615325.364342 mapwrite 0x017f5d thru 0x02285d (0xa901 bytes) 000931 1787615325.575667 write 0x001d2f thru 0x00ba30 (0x9d02 bytes) 000932 1787615325.604263 fallocate 0x03e9ec thru 0x03ffff (0x1614 bytes) 000933 1787615325.614220 punch from 0x034c0b to 0x0383e5 (0x37db bytes) 000934 1787615325.625587 trunc from 0x01054b to 0x019b88 (0x963e bytes) 000935 1787615325.634008 write 0x03d8f1 thru 0x03ffff (0x270f bytes) 000936 1787615325.645808 punch from 0x03636a to 0x03a814 (0x44ab bytes) 000937 1787615325.650302 trunc from 0x040000 to 0x036f7e (0x9081 bytes) 000939 1787615325.677935 mapwrite 0x011103 thru 0x018d16 (0x7c14 bytes) 000940 1787615325.748794 mapread 0x024564 thru 0x0309f0 (0xc48d bytes) 000941 1787615325.761692 write 0x0222bc thru 0x024a97 (0x27dc bytes) 000942 1787615325.763488 mapwrite 0x025b59 thru 0x02a7a2 (0x4c4a bytes) 000943 1787615325.798560 fallocate 0x03dd4a thru 0x03ffff (0x22b6 bytes) 000944 1787615325.807516 write 0x00f8a9 thru 0x0100a3 (0x7fb bytes) 000945 1787615325.815350 read 0x01d63e thru 0x02c9f3 (0xf3b6 bytes) 000946 1787615325.818286 mapread 0x004181 thru 0x00d255 (0x90d5 bytes) 000948 1787615325.825569 MIRROR_RESYNC 6 mirrors 000949 1787615327.513648 MIRROR_RESYNC 6 mirrors 000950 1787615327.928519 trunc from 0x036f7f to 0x009583 (0x2d9fb bytes) 000951 1787615328.022519 read 0x001741 thru 0x0039f6 (0x22b6 bytes) 000952 1787615328.034502 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x9584): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9584): Mirror 4: 0x9403638a (stale) Mirror 6: 0x9403638a (stale) Mirror 8: 0x9403638a (stale) Mirror 9: 0x9403638a (stale) Mirror 10: 0x9403638a (stale) CRC-32 checksum value for chunk [0, 0x9584): Mirror 1: 0x9403638a Mirror 11: 0x9403638a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9584: skipped 000953 1787615329.274117 trunc from 0x009584 to 0x009fb4 (0xa31 bytes) 000954 1787615329.440945 mapwrite 0x006aec thru 0x00e734 (0x7c49 bytes) 000955 1787615329.611597 read 0x001f57 thru 0x00be76 (0x9f20 bytes) 000956 1787615329.612360 mapwrite 0x0298e4 thru 0x034655 (0xad72 bytes) 000957 1787615329.763458 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x34656): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34656): Mirror 4: 0xe187a0c8 (stale) Mirror 6: 0xe187a0c8 (stale) Mirror 8: 0xe187a0c8 (stale) Mirror 9: 0xe187a0c8 (stale) Mirror 10: 0xe187a0c8 (stale) Mirror 11: 0x956d122a (stale) CRC-32 checksum value for chunk [0, 0x34656): Mirror 1: 0x6fb72e90 Mirror 12: 0x6fb72e90 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34656: skipped 000958 1787615331.438124 fallocate 0x0117c4 thru 0x01ff17 (0xe754 bytes) 000959 1787615331.514257 MIRROR_RESYNC 8 mirrors 000960 1787615333.640958 punch from 0x03301b to 0x034655 (0x163b bytes) 000961 1787615333.671354 read 0x019178 thru 0x0278ff (0xe788 bytes) 000962 1787615333.703467 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x34656): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34656): Mirror 4: 0x6fb72e90 (stale) Mirror 6: 0x6fb72e90 (stale) Mirror 8: 0x6fb72e90 (stale) Mirror 9: 0x6fb72e90 (stale) Mirror 10: 0x6fb72e90 (stale) Mirror 11: 0x6fb72e90 (stale) Mirror 12: 0x6fb72e90 (stale) CRC-32 checksum value for chunk [0, 0x34656): Mirror 1: 0x1b736cd6 Mirror 13: 0x1b736cd6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34656: skipped 000963 1787615335.261503 fallocate 0x0092aa thru 0x01398c (0xa6e3 bytes) 000964 1787615335.335323 punch from 0x0177da to 0x01f179 (0x79a0 bytes) 000965 1787615335.347315 trunc from 0x02e216 to 0x034655 (0x6440 bytes) 000966 1787615335.355695 trunc from 0x034656 to 0x03ff16 (0xb8c1 bytes) 000967 1787615335.403727 read 0x00a3a3 thru 0x012e7d (0x8adb bytes) 000968 1787615335.419309 mapread 0x02a33e thru 0x03355f (0x9222 bytes) 000969 1787615335.466876 mapwrite 0x00c6fa thru 0x0178aa (0xb1b1 bytes) 000970 1787615335.541601 trunc from 0x025bcb to 0x03181c (0xbc52 bytes) 000971 1787615335.550882 mapread 0x0116fb thru 0x01cda4 (0xb6aa bytes) 000972 1787615335.573110 mapread 0x00f2f9 thru 0x01cd39 (0xda41 bytes) 000973 1787615335.575650 trunc from 0x01e80c to 0x01f37c (0xb71 bytes) 000974 1787615335.590501 mapread 0x039bb6 thru 0x03d860 (0x3cab bytes) 000975 1787615335.625402 read 0x00470c thru 0x00c46a (0x7d5f bytes) 000976 1787615335.641849 fallocate 0x022476 thru 0x02f645 (0xd1d0 bytes) 000977 1787615335.654424 fallocate 0x02a9ae thru 0x02b01c (0x66f bytes) 000978 1787615335.666883 trunc from 0x03b632 to 0x03dd3a (0x2709 bytes) 000979 1787615335.685254 MIRROR_RESYNC 9 mirrors 000980 1787615338.097434 trunc from 0x03ff17 to 0x02ad33 (0x151e3 bytes) 000981 1787615338.147260 mapread 0x00d971 thru 0x018613 (0xaca3 bytes) 000982 1787615338.178346 read 0x017971 thru 0x025510 (0xdba0 bytes) 000983 1787615338.190928 fallocate 0x006ea7 thru 0x00f989 (0x8ae3 bytes) 000984 1787615338.196105 trunc from 0x02ad34 to 0x005912 (0x25421 bytes) 000985 1787615338.237245 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5913): Mirror 4: 0xcb13200d (stale) Mirror 6: 0xcb13200d (stale) Mirror 8: 0xcb13200d (stale) Mirror 9: 0xcb13200d (stale) Mirror 11: 0xcb13200d (stale) Mirror 12: 0xcb13200d (stale) Mirror 13: 0xcb13200d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5913): Mirror 4: 0xcb13200d (stale) Mirror 6: 0xcb13200d (stale) Mirror 8: 0xcb13200d (stale) Mirror 9: 0xcb13200d (stale) Mirror 11: 0xcb13200d (stale) Mirror 12: 0xcb13200d (stale) Mirror 13: 0xcb13200d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5913: skipped 000986 1787615338.702278 mapread 0x00257e thru 0x005912 (0x3395 bytes) 000987 1787615338.733110 read 0x004447 thru 0x005912 (0x14cc bytes) 000988 1787615338.733965 trunc from 0x004590 to 0x005912 (0x1383 bytes) 000989 1787615338.738930 fallocate 0x032e5f thru 0x03d8bc (0xaa5e bytes) 000990 1787615338.744661 mapread 0x00bba8 thru 0x01b0e6 (0xf53f bytes) 000992 1787615338.764614 mapread 0x003631 thru 0x00386e (0x23e bytes) 000993 1787615338.775496 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3d8bd): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d8bd): Mirror 4: 0x1d602723 (stale) Mirror 6: 0x1d602723 (stale) Mirror 8: 0x1d602723 (stale) Mirror 9: 0x1d602723 (stale) Mirror 11: 0x1d602723 (stale) Mirror 12: 0x1d602723 (stale) Mirror 13: 0x1d602723 (stale) CRC-32 checksum value for chunk [0, 0x3d8bd): Mirror 1: 0x3a5e3c88 Mirror 14: 0x3a5e3c88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d8bd: skipped 000994 1787615340.198128 mapread 0x000cfd thru 0x00a70b (0x9a0f bytes) 000995 1787615340.253234 trunc from 0x03d8bd to 0x003469 (0x3a453 bytes) 000996 1787615340.291687 punch from 0x00094a to 0x003469 (0x2b20 bytes) 000997 1787615340.317197 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x346a): Mirror 4: 0x1242989a (stale) Mirror 6: 0x1242989a (stale) Mirror 8: 0x1242989a (stale) Mirror 9: 0x1242989a (stale) Mirror 11: 0x1242989a (stale) Mirror 12: 0x1242989a (stale) Mirror 13: 0x1242989a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x346a): Mirror 4: 0x1242989a (stale) Mirror 6: 0x1242989a (stale) Mirror 8: 0x1242989a (stale) Mirror 9: 0x1242989a (stale) Mirror 11: 0x1242989a (stale) Mirror 12: 0x1242989a (stale) Mirror 13: 0x1242989a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x346a: skipped 000998 1787615340.825665 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x346a): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x346a): Mirror 4: 0x1242989a (stale) Mirror 6: 0x1242989a (stale) Mirror 8: 0x1242989a (stale) Mirror 9: 0x1242989a (stale) Mirror 11: 0x1242989a (stale) Mirror 12: 0x1242989a (stale) Mirror 13: 0x1242989a (stale) CRC-32 checksum value for chunk [0, 0x346a): Mirror 1: 0xba8d28ce Mirror 14: 0xba8d28ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x346a: skipped 000999 1787615342.319789 punch from 0x00019b to 0x0007c7 (0x62d bytes) 001000 1787615342.397969 mapwrite 0x0009f0 thru 0x00660e (0x5c1f bytes) All operations completed A-OK! PASS 70a (92s) == sanity-flr test 71: check mirror extend parent fid ==== 19:49:08 (1787615348) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x5a:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 19:49:19 (1787615359) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2de:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.239164 s, 8.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2de:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149218 s, 7.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2df:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2de:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2df:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2de:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 19:49:28 (1787615368) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4286816 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8294464 .. Extending file size to 8464352 .. Extending file size to 8570752 .. Extending file size to 9268736 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61351 61352 61353 61356 61357 done 192.168.201.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg134-client.virtnet /mnt/lustre2 (opts:) 192.168.201.134@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg134-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:50:52 (1787615452) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8021408 .. Extending file size to 7117696 .. Extending file size to 8280192 .. Extending file size to 9216288 .. Extending file size to 8537216 .. Extending file size to 9175904 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 62776 62777 62778 62781 62782 192.168.201.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg134-client.virtnet /mnt/lustre2 (opts:) 192.168.201.134@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg134-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:52:13 (1787615533) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1268832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1992 1269312 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17860 3589160 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25908 3581112 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43768 7170272 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1268832 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1992 1269312 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17860 3589160 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25908 3581112 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43768 7170272 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15422 s, 13.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172099 s, 12.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa737d2e7 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg134-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:52:35 (1787615555) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0762126 s, 27.5 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 19:52:42 (1787615562) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f5:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f5:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f6:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f8:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:52:50 (1787615570) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0713148 s, 29.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:52:59 (1787615579) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113332 s, 18.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x50:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x50:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:53:08 (1787615588) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122873 s, 17.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fa:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:53:42 (1787615622) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg134-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133576 s, 7.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:54:38 (1787615678) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg134-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0945658 s, 22.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:55:32 (1787615732) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg134-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg134-server: Pool lustre.test_204f created Adding targets to pool oleg134-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg134-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128544 s, 16.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg134-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg134-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg134-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:56:47 (1787615807) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:56:54 (1787615814) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:57:01 (1787615821) oleg134-server: Pool lustre.test_206 created Adding targets to pool oleg134-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg134-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg134-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg134-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg134-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:57:31 (1787615851) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926241 s, 11.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046076 s, 22.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4e7d4ae2 Mirror 3: 0x4e7d4ae2 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:57:43 (1787615863) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:57:47 (1787615867) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:57:50 (1787615870) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0403564 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1e50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x850000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x850000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x850000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x850000 Read this (4 bytes): BBBB 192.168.201.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg134-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:58:28 (1787615908) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00696058 s, 1.2 MB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:58:35 (1787615915) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501484 s, 20.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1e5 lmm_fid: [0x200000405:0x1e5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1e5 lmm_fid: [0x200000405:0x1e5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x304:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611621 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-24 19:58:37.000000000 -0400 Modify: 2026-08-24 19:58:37.000000000 -0400 Change: 2026-08-24 19:58:39.000000000 -0400 Birth: 2026-08-24 19:58:37.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:58:45 (1787615925) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283946 s, 36.9 MB/s oleg134-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg134-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg134-server: O/280000401/d5/305: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:59:08 (1787615948) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.237443 s, 173 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0253163 s, 162 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 24 19:59 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ea:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 24 19:59 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:59:17 (1787615957) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:59:25 (1787615965) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1925 sec =========== 19:59:34 (1787615974) === sanity-flr: start cleanup 19:59:36 (1787615976) === === sanity-flr: finish cleanup 19:59:44 (1787615984) ===