-----============= acceptance-small: sanity-flr ============----- Thu Oct 1 19:00:20 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg330-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 19:00:37 (1790895637) === oleg330-client.virtnet: executing check_config_client /mnt/lustre oleg330-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg330-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff89fc4998d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff89fc4998d000.idle_timeout=debug disable quota as required oleg330-server: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 19:01:21 (1790895681) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8224] == sanity-flr test 0a: lfs mirror create with -N option == 19:01:24 (1790895684) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:01:40 (1790895700) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:01:43 (1790895703) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:01:46 (1790895706) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:02:05 (1790895725) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:02:07 (1790895727) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 19:02:11 (1790895731) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff89fc4998d000.stats=clear osc.lustre-OST0001-osc-ffff89fc4998d000.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:02:17 (1790895737) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:02:26 (1790895746) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:02:33 (1790895753) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:02:37 (1790895757) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:02:44 (1790895764) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 19:02:52 (1790895772) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:02:58 (1790895778) PASS 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:03:04 (1790895784) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:03:10 (1790895790) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:03:19 (1790895799) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:03:25 (1790895805) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.682884 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845395 s, 12.4 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:03:33 (1790895813) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.44026 s, 40.5 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:03:40 (1790895820) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:03:49 (1790895829) Stopping /mnt/lustre-ost1 (opts:) on oleg330-server oleg330-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89fc4998d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89fc4998d000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg330-server: oleg330-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg330-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg330-server oleg330-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89fc4998d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89fc4998d000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg330-server: oleg330-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg330-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (57s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:04:46 (1790895886) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.38643 s, 21.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg330-server oleg330-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89fc4998d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89fc4998d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg330-server: oleg330-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg330-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:05:27 (1790895927) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg330-server oleg330-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89fc4998d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89fc4998d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg330-server: oleg330-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg330-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg330-server oleg330-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89fc4998d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89fc4998d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg330-server: oleg330-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg330-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (68s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:06:34 (1790895994) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.98153 s, 26.5 MB/s -rw-r--r-- 1 root root 52428800 Oct 1 19:06 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.722s user 0m0.001s sys 0m0.188s Stopping /mnt/lustre-ost1 (opts:) on oleg330-server oleg330-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89fc4998d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89fc4998d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.874s user 0m0.001s sys 0m0.181s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg330-server: oleg330-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg330-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg330-server oleg330-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89fc4998d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89fc4998d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.035s user 0m0.001s sys 0m0.170s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg330-server: oleg330-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg330-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89fc4998d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89fc4998d000.ost_server_uuid in FULL state after 0 sec PASS 33b (84s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:07:58 (1790896078) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:08:01 (1790896081) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:08:04 (1790896084) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 19:08:07 (1790896087) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 19:08:13 (1790896093) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.501952 s, 25.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:08:24 (1790896104) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417163 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433673 s, 24.2 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:08:31 (1790896111) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:08:39 (1790896119) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00212037 s, 483 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00303699 s, 337 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00097622 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00206666 s, 495 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 19:08:47 (1790896127) create test files with size 15753500 .. 114797d0a8554cd4b490504a2fe831cf - 79999a221db115146635f5992ee815f4 - 56bbbd0b8c629ea88fa2f03ac66ab99f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 19:09:08 (1790896148) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:09:25 (1790896165) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:09:31 (1790896171) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10705 s, 9.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544718 s, 19.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 19:09:41 (1790896181) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0651998 s, 32.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.2092 s, 20.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137895 s, 15.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 19:09:55 (1790896195) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 19:09:58 (1790896198) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:10:02 (1790896202) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387478 s, 27.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 19:10:10 (1790896210) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.368167 s, 8.5 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:10:19 (1790896219) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:10:29 (1790896229) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.307075 s, 34.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:10:39 (1790896239) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff89fc61681000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.312526 s, 33.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg330-client.virtnet /mnt/lustre (opts:) Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff89fc593a8800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 19:10:56 (1790896256) error lfs split: create victim file failed: File exists PASS 44f (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:11:11 (1790896271) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:11:48 (1790896308) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:12:22 (1790896342) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 19:12:25 (1790896345) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.335679 s, 9.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.157571 s, 20.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x34107926 (nosync,stale) Mirror 2: 0x34107926 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 28c8c501ff3890ade37dbbd9353659a6 - mirror 1 checksum: bcf87477a7cc8350d92a6c641e51363a - mirror 2 checksum: 28c8c501ff3890ade37dbbd9353659a6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-10-01T19:12:28-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-10-01T19:12:28-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 28c8c501ff3890ade37dbbd9353659a6 - mirror 1 checksum: bcf87477a7cc8350d92a6c641e51363a - mirror 2 checksum: bcf87477a7cc8350d92a6c641e51363a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-10-01T19:12:28-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-10-01T19:12:28-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 19:12:35 (1790896355) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478906 s, 21.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:12:41 (1790896361) Failing mds1 on oleg330-server Stopping /mnt/lustre-mds1 (opts:) on oleg330-server 19:12:47 (1790896367) shut down facet: mds1 facet_host: oleg330-server facet_failover_host: oleg330-server Failover mds1 to oleg330-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-MDT0000 19:13:17 (1790896397) targets are mounted 19:13:17 (1790896397) facet_failover done oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x8:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621422 s, 16.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000405 lmm_object_id: 0x8 lmm_fid: [0x240000405:0x8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f:0x0] } PASS 50A (54s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:13:35 (1790896415) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0400026 s, 512 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0242879 s, 843 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:13:45 (1790896425) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0439327 s, 466 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0164544 s, 1.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:13:54 (1790896434) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00753714 s, 2.2 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:14:01 (1790896441) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0485214 s, 422 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:14:09 (1790896449) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0160527 s, 255 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0237783 s, 172 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:14:16 (1790896456) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) PASS 60b (18s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:14:34 (1790896474) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:15:01 (1790896501) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:15:23 (1790896523) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:15:45 (1790896545) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 19:15:51 (1790896551) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 58235 mirror split pid 58236 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 58235 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 58236 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 19:16:57 (1790896617) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 6618 truncating to largest ever: 0x28a92 000003 1790896619.138453 trunc from 00000000 to 0x028a91 (0x28a92 bytes) 000005 1790896619.152016 read 0x013ac8 thru 0x0158b5 (0x1dee bytes) 000006 1790896619.157774 read 0x008a25 thru 0x014c94 (0xc270 bytes) 000007 1790896619.168457 fallocate 0x006cd5 thru 0x011e18 (0xb144 bytes) 000008 1790896619.172491 read 0x01fa5a thru 0x028a91 (0x9038 bytes) 000009 1790896619.173181 fallocate 0x010dad thru 0x012a1f (0x1c73 bytes) fallocating to largest ever: 0x2a14a 000010 1790896619.177791 fallocate 0x024dbc thru 0x02a149 (0x538e bytes) 000011 1790896619.181839 mapread 0x001ef4 thru 0x00c638 (0xa745 bytes) 000012 1790896619.183074 write 0x03bf19 thru 0x03ffff (0x40e7 bytes) 000013 1790896619.188042 MIRROR_EXTEND to 2 mirrors 000014 1790896619.677253 trunc from 0x040000 to 0x00a5fb (0x35a04 bytes) 000015 1790896619.742232 write 0x0260b4 thru 0x02f877 (0x97c4 bytes) 000016 1790896619.743677 punch from 0x01cd42 to 0x028850 (0xbb0f bytes) 000017 1790896619.763074 mapwrite 0x012fd6 thru 0x01672d (0x3758 bytes) 000018 1790896619.792960 trunc from 0x017f98 to 0x02204b (0xa0b4 bytes) 000020 1790896619.803334 fallocate 0x01d9a6 thru 0x020e25 (0x3480 bytes) 000021 1790896619.807956 read 0x02d630 thru 0x02f877 (0x2248 bytes) 000022 1790896619.808520 punch from 0x01b91e to 0x01ddb2 (0x2495 bytes) 000023 1790896619.821321 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2f878): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f878): Mirror 2: 0x6e5a4058 (stale) CRC-32 checksum value for chunk [0, 0x2f878): Mirror 1: 0x884e5aee Mirror 3: 0x884e5aee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f878: skipped 000024 1790896620.587009 fallocate 0x018d89 thru 0x01f0a7 (0x631f bytes) 000025 1790896620.616314 read 0x025375 thru 0x027b2b (0x27b7 bytes) 000026 1790896620.624286 mapwrite 0x03245f thru 0x0369d2 (0x4574 bytes) 000027 1790896620.730882 punch from 0x017de6 to 0x018675 (0x890 bytes) 000028 1790896620.735134 trunc from 0x0123b8 to 0x014240 (0x1e89 bytes) 000029 1790896620.744557 MIRROR_RESYNC 3 mirrors 000031 1790896621.859832 MIRROR_EXTEND to 4 mirrors zero_range to largest ever: 0x36891 000032 1790896622.901494 trunc from 0x02c94e to 0x036890 (0x9f43 bytes) 000033 1790896622.978351 punch from 0x004545 to 0x011768 (0xd224 bytes) 000034 1790896622.984850 mapread 0x016bcd thru 0x022af8 (0xbf2c bytes) 000035 1790896623.039807 write 0x00a24e thru 0x013640 (0x93f3 bytes) 000036 1790896623.067107 mapwrite 0x0214fc thru 0x0261c3 (0x4cc8 bytes) 000037 1790896623.098728 write 0x008f64 thru 0x00f760 (0x67fd bytes) 000038 1790896623.106148 trunc from 0x02605d to 0x0276b0 (0x1654 bytes) 000039 1790896623.113804 fallocate 0x028f9c thru 0x02f83c (0x68a1 bytes) 000040 1790896623.119708 mapread 0x01e5c0 thru 0x02b0b1 (0xcaf2 bytes) 000041 1790896623.129002 mapread 0x01c348 thru 0x022ee9 (0x6ba2 bytes) 000042 1790896623.130126 punch from 0x027f8e to 0x02fde2 (0x7e55 bytes) 000043 1790896623.136880 fallocate 0x01dde0 thru 0x023fb8 (0x61d9 bytes) 000044 1790896623.142544 punch from 0x02321d to 0x030b99 (0xd97d bytes) 000045 1790896623.153787 mapread 0x029d6f thru 0x0369d2 (0xcc64 bytes) 000046 1790896623.193032 mapwrite 0x0181c5 thru 0x01fa54 (0x7890 bytes) 000047 1790896623.230047 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x369d3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x369d3): Mirror 2: 0x8c37ffc6 (stale) Mirror 3: 0x8c37ffc6 (stale) Mirror 4: 0x8c37ffc6 (stale) CRC-32 checksum value for chunk [0, 0x369d3): Mirror 1: 0x6471e1bf Mirror 5: 0x6471e1bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x369d3: skipped 000048 1790896624.011922 punch from 0x00e071 to 0x01037a (0x230a bytes) 000049 1790896624.049420 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x369d3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x369d3): Mirror 2: 0x8c37ffc6 (stale) Mirror 3: 0x8c37ffc6 (stale) Mirror 4: 0x8c37ffc6 (stale) Mirror 5: 0x6471e1bf (stale) CRC-32 checksum value for chunk [0, 0x369d3): Mirror 1: 0xf575590d Mirror 6: 0xf575590d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x369d3: skipped 000050 1790896625.134711 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x369d3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x369d3): Mirror 2: 0x8c37ffc6 (stale) Mirror 3: 0x8c37ffc6 (stale) Mirror 4: 0x8c37ffc6 (stale) Mirror 5: 0x6471e1bf (stale) CRC-32 checksum value for chunk [0, 0x369d3): Mirror 1: 0xf575590d Mirror 6: 0xf575590d Mirror 7: 0xf575590d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x369d3: skipped 000051 1790896626.316772 write 0x00d172 thru 0x010fa2 (0x3e31 bytes) 000052 1790896626.355571 write 0x03969b thru 0x03ffff (0x6965 bytes) 000055 1790896626.357710 read 0x014f06 thru 0x01c4a2 (0x759d bytes) 000056 1790896626.364797 punch from 0x016eb9 to 0x021f6d (0xb0b5 bytes) 000057 1790896626.369968 punch from 0x01a05a to 0x0294c2 (0xf469 bytes) 000058 1790896626.373903 trunc from 0x040000 to 0x02e33c (0x11cc3 bytes) 000059 1790896626.391134 read 0x016c13 thru 0x022abc (0xbeaa bytes) 000060 1790896626.423493 read 0x013a9e thru 0x014beb (0x114e bytes) 000061 1790896626.424394 punch from 0x001b41 to 0x00704e (0x550e bytes) 000062 1790896626.428748 trunc from 0x026f22 to 0x02e33c (0x741b bytes) 000063 1790896626.432493 fallocate 0x0261bf thru 0x02e64c (0x848e bytes) 000064 1790896626.437823 trunc from 0x02e33d to 0x033850 (0x5514 bytes) 000066 1790896626.450666 punch from 0x031a15 to 0x033850 (0x1e3c bytes) 000067 1790896626.458287 MIRROR_RESYNC 7 mirrors 000069 1790896627.804397 fallocate 0x0166b7 thru 0x01f055 (0x899f bytes) 000070 1790896627.842415 fallocate 0x01d060 thru 0x01e446 (0x13e7 bytes) 000071 1790896627.851106 mapread 0x02472d thru 0x02655a (0x1e2e bytes) 000073 1790896627.902277 trunc from 0x024941 to 0x024f49 (0x609 bytes) 000074 1790896627.910777 trunc from 0x02eb8b to 0x033850 (0x4cc6 bytes) 000075 1790896627.915347 fallocate 0x014a31 thru 0x023bb1 (0xf181 bytes) 000076 1790896627.922357 write 0x00b1d9 thru 0x00dbcb (0x29f3 bytes) 000077 1790896627.938732 write 0x0125a0 thru 0x01d128 (0xab89 bytes) 000078 1790896627.951736 read 0x02a647 thru 0x033850 (0x920a bytes) 000079 1790896627.967962 read 0x00ddbc thru 0x0106d5 (0x291a bytes) 000080 1790896627.978222 read 0x025251 thru 0x02e434 (0x91e4 bytes) 000082 1790896627.979412 punch from 0x02bbdb to 0x02ea5d (0x2e83 bytes) 000083 1790896627.985480 read 0x01b1b7 thru 0x02a140 (0xef8a bytes) 000084 1790896627.986858 fallocate 0x028dc2 thru 0x03042c (0x766b bytes) 000085 1790896627.996114 trunc from 0x033851 to 0x027ef2 (0xb95e bytes) 000086 1790896628.031034 fallocate 0x00ebb9 thru 0x0194da (0xa922 bytes) 000087 1790896628.036869 punch from 0x02721d to 0x027ef2 (0xcd6 bytes) 000088 1790896628.041827 mapwrite 0x03feb8 thru 0x03ffff (0x148 bytes) 000089 1790896628.102278 trunc from 0x0317bc to 0x032526 (0xd6b bytes) 000090 1790896628.111994 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x28f523fd (stale) Mirror 4: 0x28f523fd (stale) Mirror 5: 0x28f523fd (stale) Mirror 6: 0x28f523fd (stale) Mirror 7: 0x28f523fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x28f523fd (stale) Mirror 4: 0x28f523fd (stale) Mirror 5: 0x28f523fd (stale) Mirror 6: 0x28f523fd (stale) Mirror 7: 0x28f523fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x3dd39 000091 1790896628.658300 trunc from 0x040000 to 0x03dd38 (0x22c7 bytes) 000092 1790896628.685936 write 0x0158e9 thru 0x0204a3 (0xabbb bytes) 000093 1790896628.699892 read 0x02c95f thru 0x031232 (0x48d4 bytes) 000094 1790896628.717073 read 0x0240c7 thru 0x0254b1 (0x13eb bytes) 000095 1790896628.744393 mapwrite 0x03ca52 thru 0x03ffff (0x35ae bytes) 000096 1790896628.835443 mapwrite 0x02b802 thru 0x037c88 (0xc487 bytes) 000097 1790896628.894444 mapwrite 0x02c8fd thru 0x03b7d8 (0xeedc bytes) 000098 1790896628.952244 write 0x019ddf thru 0x02865c (0xe87e bytes) 000099 1790896628.954304 read 0x01b358 thru 0x02b357 (0x10000 bytes) fallocating to largest ever: 0x40000 000100 1790896628.956940 fallocate 0x035324 thru 0x03ffff (0xacdc bytes) 000101 1790896628.978775 trunc from 0x040000 to 0x02ca6c (0x13593 bytes) 000102 1790896629.007701 mapwrite 0x024153 thru 0x0315eb (0xd499 bytes) 000103 1790896629.092079 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x315ec): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x315ec): Mirror 2: 0x3254cf3c (stale) Mirror 4: 0x3254cf3c (stale) Mirror 5: 0x3254cf3c (stale) Mirror 6: 0x3254cf3c (stale) Mirror 7: 0x3254cf3c (stale) CRC-32 checksum value for chunk [0, 0x315ec): Mirror 1: 0xb83cf5d9 Mirror 8: 0xb83cf5d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x315ec: skipped 000104 1790896630.050058 read 0x00f393 thru 0x01d705 (0xe373 bytes) 000105 1790896630.075450 mapread 0x00b1dd thru 0x016d68 (0xbb8c bytes) 000106 1790896630.108871 mapwrite 0x02d7c4 thru 0x038563 (0xada0 bytes) 000107 1790896630.198714 fallocate 0x02d0fb thru 0x034051 (0x6f57 bytes) 000108 1790896630.203963 trunc from 0x038564 to 0x00a92e (0x2dc35 bytes) 000109 1790896630.223892 write 0x02bfeb thru 0x0368c6 (0xa8dc bytes) 000110 1790896630.227419 trunc from 0x00220c to 0x007884 (0x5679 bytes) 000111 1790896630.234862 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x368c7): Mirror 2: 0xd68bc786 (stale) Mirror 4: 0xd68bc786 (stale) Mirror 6: 0xd68bc786 (stale) Mirror 7: 0xd68bc786 (stale) Mirror 8: 0xcad0bdc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x368c7): Mirror 2: 0xd68bc786 (stale) Mirror 4: 0xd68bc786 (stale) Mirror 6: 0xd68bc786 (stale) Mirror 7: 0xd68bc786 (stale) Mirror 8: 0xcad0bdc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x368c7: skipped 000112 1790896630.598540 trunc from 0x0266c9 to 0x027cb9 (0x15f1 bytes) 000113 1790896630.605039 write 0x02f6b3 thru 0x03051b (0xe69 bytes) 000114 1790896630.619328 trunc from 0x00d402 to 0x011a30 (0x462f bytes) 000115 1790896630.623901 read 0x006ab3 thru 0x0087c7 (0x1d15 bytes) 000117 1790896630.644041 read 0x033352 thru 0x0368c6 (0x3575 bytes) 000118 1790896630.663818 trunc from 0x0368c7 to 0x004739 (0x3218d bytes) 000120 1790896630.684912 write 0x014691 thru 0x017a4c (0x33bc bytes) 000121 1790896630.687634 punch from 0x000074 to 0x00eb0f (0xea9c bytes) 000122 1790896630.699787 trunc from 0x009384 to 0x014979 (0xb5f6 bytes) 000123 1790896630.732759 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x17a4d): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17a4d): Mirror 2: 0x3cf62b53 (stale) Mirror 4: 0x3cf62b53 (stale) Mirror 6: 0x3cf62b53 (stale) Mirror 7: 0x3cf62b53 (stale) Mirror 8: 0xa61df93a (stale) CRC-32 checksum value for chunk [0, 0x17a4d): Mirror 1: 0x50239d4d Mirror 9: 0x50239d4d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17a4d: skipped 000124 1790896631.592991 write 0x0047c3 thru 0x0111bd (0xc9fb bytes) 000125 1790896631.641650 fallocate 0x028b14 thru 0x02b6f9 (0x2be6 bytes) 000126 1790896631.646377 punch from 0x00586a to 0x01513d (0xf8d4 bytes) 000127 1790896631.673652 read 0x000231 thru 0x00d29a (0xd06a bytes) 000129 1790896631.704559 mapwrite 0x039289 thru 0x03ff90 (0x6d08 bytes) 000130 1790896631.802863 mapread 0x0248e7 thru 0x027dbd (0x34d7 bytes) 000131 1790896631.815582 mapread 0x036e0a thru 0x03b778 (0x496f bytes) 000132 1790896631.816784 read 0x035d21 thru 0x03f381 (0x9661 bytes) 000133 1790896631.817397 write 0x032bf1 thru 0x03c4a7 (0x98b7 bytes) 000134 1790896631.826487 mapwrite 0x01f27b thru 0x01f5a9 (0x32f bytes) 000135 1790896631.853808 trunc from 0x00741d to 0x00f7a4 (0x8388 bytes) 000136 1790896631.875843 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ff91): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff91): Mirror 2: 0x388cf4c7 (stale) Mirror 4: 0x388cf4c7 (stale) Mirror 6: 0x388cf4c7 (stale) Mirror 7: 0x388cf4c7 (stale) Mirror 8: 0xac4bb098 (stale) Mirror 9: 0x5cde587c (stale) CRC-32 checksum value for chunk [0, 0x3ff91): Mirror 1: 0xf95b8a3a Mirror 10: 0xf95b8a3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff91: skipped 000137 1790896632.881884 trunc from 0x03ff91 to 0x03b545 (0x4a4b bytes) 000138 1790896632.980635 fallocate 0x022864 thru 0x0238fa (0x1097 bytes) 000139 1790896632.986781 fallocate 0x013676 thru 0x021652 (0xdfdd bytes) 000140 1790896632.991947 punch from 0x0283ff to 0x02c360 (0x3f62 bytes) 000141 1790896632.996859 read 0x028561 thru 0x02ba1a (0x34ba bytes) 000142 1790896633.002699 punch from 0x02ee06 to 0x03a6cb (0xb8c6 bytes) 000143 1790896633.008929 mapwrite 0x0399f6 thru 0x03e28b (0x4896 bytes) 000144 1790896633.064139 mapread 0x0325da thru 0x038780 (0x61a7 bytes) 000145 1790896633.074929 trunc from 0x03e28c to 0x028711 (0x15b7a bytes) 000147 1790896633.094812 trunc from 0x028712 to 0x02813d (0x5d4 bytes) 000148 1790896633.114893 write 0x01ce9f thru 0x023fac (0x710e bytes) 000149 1790896633.124875 read 0x0259cd thru 0x02813d (0x2771 bytes) 000150 1790896633.136907 read 0x023f28 thru 0x02813d (0x4216 bytes) 000151 1790896633.137886 fallocate 0x0211d2 thru 0x02148b (0x2ba bytes) 000152 1790896633.142847 trunc from 0x02813e to 0x02ee15 (0x6cd8 bytes) 000153 1790896633.159177 fallocate 0x0001f5 thru 0x0089b9 (0x87c5 bytes) 000154 1790896633.165214 read 0x021aa2 thru 0x0273b0 (0x590f bytes) 000157 1790896633.172326 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ee16): Mirror 2: 0x7183a648 (stale) Mirror 4: 0x7183a648 (stale) Mirror 6: 0x7183a648 (stale) Mirror 7: 0x7183a648 (stale) Mirror 9: 0x7493ef23 (stale) Mirror 10: 0x6561089a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee16): Mirror 2: 0x7183a648 (stale) Mirror 4: 0x7183a648 (stale) Mirror 6: 0x7183a648 (stale) Mirror 7: 0x7183a648 (stale) Mirror 9: 0x7493ef23 (stale) Mirror 10: 0x6561089a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee16: skipped 000158 1790896633.638934 fallocate 0x01138a thru 0x01c47e (0xb0f5 bytes) 000159 1790896633.647158 punch from 0x00ab2e to 0x00d583 (0x2a56 bytes) 000160 1790896633.659797 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2ee16): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee16): Mirror 2: 0x7183a648 (stale) Mirror 4: 0x7183a648 (stale) Mirror 6: 0x7183a648 (stale) Mirror 7: 0x7183a648 (stale) Mirror 9: 0x7493ef23 (stale) Mirror 10: 0x6561089a (stale) CRC-32 checksum value for chunk [0, 0x2ee16): Mirror 1: 0xf08f5572 Mirror 11: 0xf08f5572 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee16: skipped 000161 1790896634.498154 fallocate 0x008a65 thru 0x0092e2 (0x87e bytes) 000162 1790896634.539700 trunc from 0x018a6f to 0x024a71 (0xc003 bytes) 000163 1790896634.545478 read 0x00b748 thru 0x014357 (0x8c10 bytes) 000164 1790896634.553425 write 0x02c881 thru 0x037e17 (0xb597 bytes) 000165 1790896634.570349 trunc from 0x016708 to 0x0202ab (0x9ba4 bytes) 000166 1790896634.575942 trunc from 0x019b96 to 0x01ce18 (0x3283 bytes) 000167 1790896634.581840 fallocate 0x0265d1 thru 0x02d356 (0x6d86 bytes) 000168 1790896634.598946 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x37e18): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e18): Mirror 2: 0xb7a24014 (stale) Mirror 4: 0xb7a24014 (stale) Mirror 6: 0xb7a24014 (stale) Mirror 7: 0xb7a24014 (stale) Mirror 9: 0x4e888299 (stale) Mirror 10: 0x32cbc2b1 (stale) Mirror 11: 0xe8be272a (stale) CRC-32 checksum value for chunk [0, 0x37e18): Mirror 1: 0x16e8c0f0 Mirror 12: 0x16e8c0f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e18: skipped 000169 1790896635.516356 punch from 0x023ac0 to 0x02a003 (0x6544 bytes) 000170 1790896635.556734 read 0x01b5a9 thru 0x025564 (0x9fbc bytes) 000171 1790896635.568707 MIRROR_RESYNC 9 mirrors 000172 1790896637.742520 mapwrite 0x014e63 thru 0x0200b9 (0xb257 bytes) 000173 1790896637.900105 write 0x00b961 thru 0x00c0f2 (0x792 bytes) 000174 1790896637.937038 MIRROR_RESYNC 9 mirrors 000175 1790896640.160908 trunc from 0x037e18 to 0x01063d (0x277da bytes) 000176 1790896640.204640 mapread 0x0037b9 thru 0x0077b2 (0x3ffa bytes) 000177 1790896640.223637 mapread 0x00c2c8 thru 0x01063d (0x4376 bytes) 000178 1790896640.224601 punch from 0x00947c to 0x00f7d9 (0x635e bytes) 000179 1790896640.237596 mapread 0x009c88 thru 0x01063d (0x69b6 bytes) 000180 1790896640.249532 mapwrite 0x008515 thru 0x016f4c (0xea38 bytes) 000181 1790896640.329938 trunc from 0x00de69 to 0x016f4c (0x90e4 bytes) 000182 1790896640.345100 write 0x00099f thru 0x006b0c (0x616e bytes) 000183 1790896640.356604 fallocate 0x028f44 thru 0x034105 (0xb1c2 bytes) 000184 1790896640.361251 punch from 0x015a31 to 0x025430 (0xfa00 bytes) 000185 1790896640.367495 trunc from 0x004539 to 0x00b1f1 (0x6cb9 bytes) 000186 1790896640.382907 trunc from 0x01fb2f to 0x02a175 (0xa647 bytes) 000187 1790896640.387669 write 0x01cd97 thru 0x01d0a2 (0x30c bytes) 000188 1790896640.396816 read 0x01b118 thru 0x02671f (0xb608 bytes) 000189 1790896640.406126 trunc from 0x034106 to 0x03e943 (0xa83e bytes) 000190 1790896640.426472 mapread 0x00ccae thru 0x00d6d1 (0xa24 bytes) 000191 1790896640.439948 mapwrite 0x00fe05 thru 0x014947 (0x4b43 bytes) 000192 1790896640.455473 mapread 0x025678 thru 0x02e4b6 (0x8e3f bytes) 000193 1790896640.466886 mapread 0x0292db thru 0x02f555 (0x627b bytes) 000195 1790896640.468596 write 0x01b61d thru 0x026c21 (0xb605 bytes) 000196 1790896640.473899 mapwrite 0x02dd29 thru 0x03349e (0x5776 bytes) 000197 1790896640.499022 write 0x0053ba thru 0x00da5b (0x86a2 bytes) 000198 1790896640.505316 mapwrite 0x013e24 thru 0x01b163 (0x7340 bytes) 000199 1790896640.529904 trunc from 0x03e944 to 0x002b56 (0x3bded bytes) 000200 1790896640.551588 mapread 0x000248 thru 0x002b56 (0x290f bytes) 000201 1790896640.552877 trunc from 0x002b57 to 0x03569c (0x32b46 bytes) 000202 1790896640.569357 mapread 0x028834 thru 0x0340ef (0xb8bc bytes) 000205 1790896640.585929 fallocate 0x009a39 thru 0x00be8a (0x2452 bytes) 000206 1790896640.590922 trunc from 0x02e1c8 to 0x033722 (0x555b bytes) 000207 1790896640.595859 trunc from 0x01d65d to 0x026a91 (0x9435 bytes) 000208 1790896640.600170 fallocate 0x036ac2 thru 0x03ffff (0x953e bytes) 000209 1790896640.605257 read 0x016be0 thru 0x023351 (0xc772 bytes) 000210 1790896640.619768 write 0x02c362 thru 0x031a57 (0x56f6 bytes) 000212 1790896640.621366 trunc from 0x03569d to 0x0276a6 (0xdff6 bytes) 000213 1790896640.646462 mapread 0x00379b thru 0x00be41 (0x86a7 bytes) 000214 1790896640.647457 trunc from 0x024919 to 0x0276a6 (0x2d8e bytes) 000215 1790896640.653638 read 0x009858 thru 0x00ac6c (0x1415 bytes) 000216 1790896640.654578 read 0x01fbc3 thru 0x0276a6 (0x7ae4 bytes) 000217 1790896640.659895 trunc from 0x0237d1 to 0x0276a6 (0x3ed6 bytes) 000218 1790896640.677314 trunc from 0x0276a7 to 0x0317a1 (0xa0fb bytes) 000219 1790896640.698055 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x317a2): Mirror 4: 0x5bd19ccc (stale) Mirror 6: 0x5bd19ccc (stale) Mirror 7: 0x5bd19ccc (stale) Mirror 9: 0x5bd19ccc (stale) Mirror 10: 0x5bd19ccc (stale) Mirror 11: 0x5bd19ccc (stale) Mirror 12: 0x5bd19ccc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x317a2): Mirror 4: 0x5bd19ccc (stale) Mirror 6: 0x5bd19ccc (stale) Mirror 7: 0x5bd19ccc (stale) Mirror 9: 0x5bd19ccc (stale) Mirror 10: 0x5bd19ccc (stale) Mirror 11: 0x5bd19ccc (stale) Mirror 12: 0x5bd19ccc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x317a2: skipped 000220 1790896641.160613 trunc from 0x005d3f to 0x009694 (0x3956 bytes) 000221 1790896641.166341 fallocate 0x03f2e9 thru 0x03ffff (0xd17 bytes) 000222 1790896641.170661 read 0x00815b thru 0x016fcd (0xee73 bytes) 000223 1790896641.179895 fallocate 0x0023c0 thru 0x004758 (0x2399 bytes) 000224 1790896641.184716 fallocate 0x00cb84 thru 0x0112eb (0x4768 bytes) 000225 1790896641.189027 trunc from 0x02af06 to 0x0317a1 (0x689c bytes) 000226 1790896641.194120 mapwrite 0x0151ab thru 0x019fb6 (0x4e0c bytes) 000227 1790896641.227903 read 0x000b19 thru 0x00fcb0 (0xf198 bytes) 000228 1790896641.241306 MIRROR_RESYNC 8 mirrors 000229 1790896642.845384 fallocate 0x0088af thru 0x00a8d3 (0x2025 bytes) 000230 1790896642.908621 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x317a2): Mirror 4: 0x9ccb0721 (stale) Mirror 6: 0x9ccb0721 (stale) Mirror 7: 0x9ccb0721 (stale) Mirror 9: 0x9ccb0721 (stale) Mirror 10: 0x9ccb0721 (stale) Mirror 11: 0x9ccb0721 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x317a2): Mirror 4: 0x9ccb0721 (stale) Mirror 6: 0x9ccb0721 (stale) Mirror 7: 0x9ccb0721 (stale) Mirror 9: 0x9ccb0721 (stale) Mirror 10: 0x9ccb0721 (stale) Mirror 11: 0x9ccb0721 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x317a2: skipped 000231 1790896643.518162 trunc from 0x0317a2 to 0x0209a6 (0x10dfb bytes) 000232 1790896643.559014 trunc from 0x01c42c to 0x0209a6 (0x457b bytes) 000233 1790896643.563803 mapread 0x018ba7 thru 0x01a72c (0x1b86 bytes) 000234 1790896643.597212 mapwrite 0x034cb9 thru 0x035f51 (0x1299 bytes) 000237 1790896643.668202 punch from 0x012629 to 0x01ddf7 (0xb7cf bytes) 000238 1790896643.683944 read 0x0022c6 thru 0x010a72 (0xe7ad bytes) 000239 1790896643.704142 trunc from 0x035f52 to 0x001e0f (0x34142 bytes) 000240 1790896643.737963 mapwrite 0x03dcb6 thru 0x03ffff (0x234a bytes) 000241 1790896643.792891 punch from 0x00dee5 to 0x00ff65 (0x2081 bytes) 000243 1790896643.811293 mapread 0x01dca3 thru 0x0205d4 (0x2932 bytes) 000244 1790896643.856221 fallocate 0x00e1c5 thru 0x01a536 (0xc372 bytes) 000245 1790896643.861167 mapread 0x02bf02 thru 0x02dc9c (0x1d9b bytes) 000246 1790896643.882805 mapread 0x014a6b thru 0x01d78f (0x8d25 bytes) 000247 1790896643.900784 trunc from 0x040000 to 0x034953 (0xb6ac bytes) 000249 1790896643.922009 punch from 0x01ba07 to 0x028b95 (0xd18f bytes) 000250 1790896643.934159 mapread 0x01273a thru 0x014c01 (0x24c8 bytes) 000251 1790896643.948769 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x34954): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34954): Mirror 4: 0x9f6286 (stale) Mirror 6: 0x9f6286 (stale) Mirror 7: 0x9f6286 (stale) Mirror 9: 0x9f6286 (stale) Mirror 10: 0x9f6286 (stale) Mirror 11: 0x9f6286 (stale) CRC-32 checksum value for chunk [0, 0x34954): Mirror 1: 0x6afa964f Mirror 12: 0x6afa964f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34954: skipped 000252 1790896644.998332 trunc from 0x034954 to 0x019088 (0x1b8cb bytes) 000253 1790896645.122954 write 0x029f76 thru 0x036fa7 (0xd032 bytes) 000255 1790896645.127515 read 0x030465 thru 0x03421e (0x3dba bytes) 000256 1790896645.128320 trunc from 0x02461f to 0x03292c (0xe30e bytes) 000257 1790896645.155617 fallocate 0x02914a thru 0x02d4b5 (0x436c bytes) 000258 1790896645.172684 fallocate 0x033659 thru 0x03ffff (0xc9a7 bytes) 000259 1790896645.179649 trunc from 0x040000 to 0x0218f4 (0x1e70b bytes) 000260 1790896645.212285 trunc from 0x0218f5 to 0x03a736 (0x18e42 bytes) 000262 1790896645.227208 read 0x01f3ae thru 0x02a006 (0xac59 bytes) 000263 1790896645.247329 mapread 0x00b0aa thru 0x018b24 (0xda7b bytes) 000264 1790896645.250385 trunc from 0x01cd69 to 0x02ac6d (0xdf05 bytes) 000265 1790896645.255930 punch from 0x02769d to 0x030972 (0x92d6 bytes) 000267 1790896645.263269 read 0x00a3c9 thru 0x018ace (0xe706 bytes) 000268 1790896645.265003 read 0x02596a thru 0x0298dc (0x3f73 bytes) 000269 1790896645.279644 read 0x01c233 thru 0x01ef7c (0x2d4a bytes) 000271 1790896645.281600 fallocate 0x01cf80 thru 0x01ee8d (0x1f0e bytes) 000272 1790896645.285755 mapread 0x009786 thru 0x012bdf (0x945a bytes) 000273 1790896645.287042 read 0x02e53c thru 0x037280 (0x8d45 bytes) 000274 1790896645.288045 trunc from 0x03a737 to 0x00867d (0x320b9 bytes) 000275 1790896645.311912 mapwrite 0x033f25 thru 0x03eed2 (0xafae bytes) 000277 1790896645.372797 mapwrite 0x02854c thru 0x033b6f (0xb624 bytes) 000278 1790896645.392872 fallocate 0x01e3f7 thru 0x0223e3 (0x3fed bytes) 000279 1790896645.398755 write 0x02695d thru 0x0320f4 (0xb798 bytes) 000280 1790896645.405697 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eed3): Mirror 4: 0x2bceb1cc (stale) Mirror 6: 0x2bceb1cc (stale) Mirror 7: 0x2bceb1cc (stale) Mirror 9: 0x2bceb1cc (stale) Mirror 11: 0x2bceb1cc (stale) Mirror 12: 0x849d11bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eed3): Mirror 4: 0x2bceb1cc (stale) Mirror 6: 0x2bceb1cc (stale) Mirror 7: 0x2bceb1cc (stale) Mirror 9: 0x2bceb1cc (stale) Mirror 11: 0x2bceb1cc (stale) Mirror 12: 0x849d11bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eed3: skipped 000281 1790896646.065155 mapread 0x00889e thru 0x00a592 (0x1cf5 bytes) 000282 1790896646.086051 write 0x020111 thru 0x029bca (0x9aba bytes) 000283 1790896646.100216 punch from 0x0268a1 to 0x0361ba (0xf91a bytes) 000284 1790896646.134919 read 0x0029e9 thru 0x00a64a (0x7c62 bytes) 000285 1790896646.147111 punch from 0x006279 to 0x00c98a (0x6712 bytes) 000286 1790896646.152616 read 0x0208af thru 0x02f4c0 (0xec12 bytes) 000287 1790896646.162891 mapread 0x030c81 thru 0x03a93b (0x9cbb bytes) 000288 1790896646.164865 mapread 0x00b165 thru 0x00e92d (0x37c9 bytes) 000289 1790896646.166148 mapread 0x018d13 thru 0x02647f (0xd76d bytes) 000290 1790896646.168113 write 0x01ef76 thru 0x027838 (0x88c3 bytes) 000291 1790896646.170869 write 0x0263b9 thru 0x026d15 (0x95d bytes) 000292 1790896646.171769 punch from 0x038d5a to 0x03c049 (0x32f0 bytes) 000293 1790896646.180359 fallocate 0x03b66e thru 0x03ffff (0x4992 bytes) 000294 1790896646.190044 trunc from 0x03eed3 to 0x03c922 (0x25b0 bytes) 000295 1790896646.230810 MIRROR_RESYNC 7 mirrors 000296 1790896648.124982 write 0x0206c7 thru 0x02b45f (0xad99 bytes) 000297 1790896648.172309 fallocate 0x03b4ee thru 0x03ffff (0x4b12 bytes) 000298 1790896648.184309 mapread 0x02fdf1 thru 0x0335b6 (0x37c6 bytes) 000299 1790896648.217140 write 0x033d17 thru 0x03592d (0x1c17 bytes) 000301 1790896648.218897 trunc from 0x037ff1 to 0x03c922 (0x4932 bytes) 000302 1790896648.226675 fallocate 0x02381d thru 0x029fe0 (0x67c4 bytes) 000303 1790896648.235826 write 0x01e4f3 thru 0x0245b5 (0x60c3 bytes) 000304 1790896648.240590 read 0x0237a6 thru 0x028c16 (0x5471 bytes) 000305 1790896648.247116 MIRROR_RESYNC 7 mirrors 000306 1790896650.060448 punch from 0x036bad to 0x03aa37 (0x3e8b bytes) 000307 1790896650.092868 mapread 0x03c273 thru 0x03c922 (0x6b0 bytes) 000308 1790896650.099132 punch from 0x006481 to 0x009b1d (0x369d bytes) 000309 1790896650.103684 write 0x0015e4 thru 0x002c63 (0x1680 bytes) 000310 1790896650.114541 mapwrite 0x01e086 thru 0x02634f (0x82ca bytes) 000311 1790896650.151924 trunc from 0x00bd9f to 0x014b50 (0x8db2 bytes) 000312 1790896650.160815 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c923): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c923): Mirror 4: 0xb8ca78c6 (stale) Mirror 6: 0xb8ca78c6 (stale) Mirror 7: 0xb8ca78c6 (stale) Mirror 9: 0xb8ca78c6 (stale) Mirror 11: 0xb8ca78c6 (stale) Mirror 12: 0xb8ca78c6 (stale) CRC-32 checksum value for chunk [0, 0x3c923): Mirror 1: 0xe69b5ba7 Mirror 13: 0xe69b5ba7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c923: skipped 000313 1790896651.071814 write 0x00719c thru 0x0162e0 (0xf145 bytes) 000314 1790896651.129525 mapwrite 0x0002ff thru 0x000fd1 (0xcd3 bytes) 000315 1790896651.153954 trunc from 0x03c923 to 0x036f3b (0x59e7 bytes) 000316 1790896651.217016 fallocate 0x0341f7 thru 0x03ffff (0xbe09 bytes) 000317 1790896651.222433 write 0x0305b9 thru 0x03ea70 (0xe4b8 bytes) 000318 1790896651.228241 fallocate 0x03d64d thru 0x03eb03 (0x14b7 bytes) 000319 1790896651.234548 trunc from 0x03ea71 to 0x01caf5 (0x21f7b bytes) 000320 1790896651.252127 write 0x0147ce thru 0x01af02 (0x6735 bytes) 000321 1790896651.258735 mapwrite 0x03fba0 thru 0x03ffff (0x460 bytes) 000323 1790896651.297284 mapread 0x00c905 thru 0x010818 (0x3f14 bytes) 000324 1790896651.298718 mapread 0x03936e thru 0x03ee38 (0x5acb bytes) 000325 1790896651.312850 read 0x02a240 thru 0x036398 (0xc159 bytes) 000326 1790896651.319750 trunc from 0x0273ef to 0x0286c0 (0x12d2 bytes) 000327 1790896651.327177 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3f38f0 (stale) Mirror 6: 0x3f38f0 (stale) Mirror 9: 0x3f38f0 (stale) Mirror 11: 0x3f38f0 (stale) Mirror 12: 0x3f38f0 (stale) Mirror 13: 0xd697370f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3f38f0 (stale) Mirror 6: 0x3f38f0 (stale) Mirror 9: 0x3f38f0 (stale) Mirror 11: 0x3f38f0 (stale) Mirror 12: 0x3f38f0 (stale) Mirror 13: 0xd697370f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000328 1790896651.781513 trunc from 0x040000 to 0x0335cd (0xca32 bytes) 000329 1790896651.802684 fallocate 0x002b73 thru 0x007ec6 (0x5354 bytes) 000331 1790896651.809280 mapread 0x02b704 thru 0x0323d5 (0x6cd2 bytes) 000332 1790896651.858004 trunc from 0x01fb0f to 0x025ec4 (0x63b6 bytes) 000333 1790896651.864099 punch from 0x0005ba to 0x00ee5f (0xe8a6 bytes) 000334 1790896651.872774 trunc from 0x0173ae to 0x01b542 (0x4195 bytes) 000335 1790896651.884030 fallocate 0x00aa9e thru 0x015c96 (0xb1f9 bytes) 000336 1790896651.893451 write 0x027f96 thru 0x02d12a (0x5195 bytes) 000338 1790896651.896082 trunc from 0x02d3c1 to 0x0335cd (0x620d bytes) 000339 1790896651.914150 read 0x02b59a thru 0x030b34 (0x559b bytes) 000340 1790896651.919781 write 0x01e7dd thru 0x027def (0x9613 bytes) 000341 1790896651.925857 write 0x03dfbe thru 0x03ffff (0x2042 bytes) 000342 1790896651.927757 punch from 0x01ad36 to 0x01bd2c (0xff7 bytes) 000343 1790896651.932628 read 0x01cdf4 thru 0x02c27f (0xf48c bytes) 000344 1790896651.953041 write 0x0287d1 thru 0x03467d (0xbead bytes) 000345 1790896651.955611 trunc from 0x040000 to 0x02ae09 (0x151f6 bytes) 000346 1790896651.976105 mapread 0x00817b thru 0x010e83 (0x8d09 bytes) 000347 1790896651.978154 write 0x0167d9 thru 0x01a4eb (0x3d13 bytes) 000348 1790896651.981146 write 0x001f00 thru 0x00810b (0x620c bytes) 000349 1790896651.985419 mapread 0x00a0c4 thru 0x00d627 (0x3564 bytes) 000350 1790896651.988095 punch from 0x002024 to 0x00e298 (0xc275 bytes) 000351 1790896652.016798 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ae0a): Mirror 6: 0xb2c750c (stale) Mirror 9: 0xb2c750c (stale) Mirror 11: 0xb2c750c (stale) Mirror 12: 0xb2c750c (stale) Mirror 13: 0xac67dd17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ae0a): Mirror 6: 0xb2c750c (stale) Mirror 9: 0xb2c750c (stale) Mirror 11: 0xb2c750c (stale) Mirror 12: 0xb2c750c (stale) Mirror 13: 0xac67dd17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ae0a: skipped 000352 1790896652.518518 write 0x004ea5 thru 0x00a09b (0x51f7 bytes) 000353 1790896652.534072 fallocate 0x01df6e thru 0x0256f2 (0x7785 bytes) 000354 1790896652.542451 read 0x02ad32 thru 0x02ae09 (0xd8 bytes) 000355 1790896652.549783 trunc from 0x0047a8 to 0x00a87c (0x60d5 bytes) 000356 1790896652.563229 read 0x00c4ca thru 0x00e6ed (0x2224 bytes) 000357 1790896652.575965 fallocate 0x009610 thru 0x019380 (0xfd71 bytes) 000358 1790896652.580702 mapwrite 0x0323d0 thru 0x036737 (0x4368 bytes) 000359 1790896652.613844 trunc from 0x0101bc to 0x01f627 (0xf46c bytes) 000360 1790896652.618867 trunc from 0x0028b1 to 0x00d5d3 (0xad23 bytes) 000361 1790896652.623613 fallocate 0x013da8 thru 0x021385 (0xd5de bytes) 000362 1790896652.627791 trunc from 0x036738 to 0x029138 (0xd5ff bytes) 000363 1790896652.639680 read 0x013551 thru 0x021ac0 (0xe570 bytes) 000364 1790896652.663028 trunc from 0x029139 to 0x0090fa (0x2003e bytes) 000365 1790896652.676878 read 0x001a91 thru 0x007627 (0x5b97 bytes) 000366 1790896652.677767 write 0x037f7a thru 0x03f013 (0x709a bytes) 000367 1790896652.680957 mapread 0x005c66 thru 0x00cb0d (0x6ea8 bytes) 000368 1790896652.690900 write 0x0110a4 thru 0x0137ec (0x2749 bytes) 000369 1790896652.691962 trunc from 0x012f7f to 0x01c903 (0x9985 bytes) 000370 1790896652.713907 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f014): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f014): Mirror 6: 0x998b3e19 (stale) Mirror 9: 0x998b3e19 (stale) Mirror 11: 0x998b3e19 (stale) Mirror 12: 0x998b3e19 (stale) Mirror 13: 0x379216c4 (stale) CRC-32 checksum value for chunk [0, 0x3f014): Mirror 1: 0x564ff99f Mirror 14: 0x564ff99f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f014: skipped 000371 1790896653.601928 mapread 0x02cfa1 thru 0x0378d5 (0xa935 bytes) 000372 1790896653.636866 mapread 0x00fb6c thru 0x015c4e (0x60e3 bytes) 000373 1790896653.656434 read 0x02c5f2 thru 0x032ed4 (0x68e3 bytes) 000375 1790896653.668668 trunc from 0x03f014 to 0x01b941 (0x236d2 bytes) 000376 1790896653.704674 write 0x00f657 thru 0x010cb1 (0x165b bytes) 000377 1790896653.713987 read 0x000957 thru 0x0034c6 (0x2b70 bytes) 000378 1790896653.728155 trunc from 0x01b942 to 0x000a8f (0x1aeb2 bytes) 000379 1790896653.748119 trunc from 0x000a90 to 0x02d234 (0x2c7a5 bytes) 000380 1790896653.800654 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d235): Mirror 6: 0x2c95cf65 (stale) Mirror 9: 0x2c95cf65 (stale) Mirror 12: 0x2c95cf65 (stale) Mirror 13: 0x7fd4bb01 (stale) Mirror 14: 0x4fa92c90 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d235): Mirror 6: 0x2c95cf65 (stale) Mirror 9: 0x2c95cf65 (stale) Mirror 12: 0x2c95cf65 (stale) Mirror 13: 0x7fd4bb01 (stale) Mirror 14: 0x4fa92c90 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d235: skipped 000381 1790896654.246667 read 0x01d312 thru 0x024435 (0x7124 bytes) 000382 1790896654.260611 mapread 0x01bb91 thru 0x027fb8 (0xc428 bytes) 000383 1790896654.277143 trunc from 0x00f9ca to 0x01e6fa (0xed31 bytes) 000384 1790896654.282109 mapread 0x00db4c thru 0x01d745 (0xfbfa bytes) 000385 1790896654.298572 mapwrite 0x021c37 thru 0x02c66b (0xaa35 bytes) 000386 1790896654.316139 trunc from 0x02d235 to 0x024af3 (0x8741 bytes) 000388 1790896654.354286 read 0x0082dd thru 0x00ec55 (0x6979 bytes) 000389 1790896654.360053 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x24af4): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24af4): Mirror 6: 0x35d914a8 (stale) Mirror 9: 0x35d914a8 (stale) Mirror 12: 0x35d914a8 (stale) Mirror 13: 0x8428261c (stale) Mirror 14: 0x2b8bf360 (stale) CRC-32 checksum value for chunk [0, 0x24af4): Mirror 1: 0x4fdfea0d Mirror 15: 0x4fdfea0d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24af4: skipped 000390 1790896655.279808 trunc from 0x024af4 to 0x00b9b3 (0x19140 bytes) 000391 1790896655.374666 punch from 0x00746c to 0x00b9b3 (0x4548 bytes) 000392 1790896655.384817 mapread 0x004eba thru 0x00b9b3 (0x6afa bytes) 000393 1790896655.406814 fallocate 0x01a332 thru 0x01ba8e (0x175d bytes) 000394 1790896655.414715 fallocate 0x01c7e8 thru 0x0274b5 (0xacce bytes) 000395 1790896655.423166 write 0x0075b8 thru 0x0114ab (0x9ef4 bytes) 000396 1790896655.425163 mapread 0x00e7b9 thru 0x0114ab (0x2cf3 bytes) 000397 1790896655.426992 write 0x03aee1 thru 0x03b771 (0x891 bytes) 000398 1790896655.431623 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b772): Mirror 6: 0xcf35c9b (stale) Mirror 9: 0xcf35c9b (stale) Mirror 12: 0xcf35c9b (stale) Mirror 13: 0x6e3ef3c (stale) Mirror 15: 0x2a894805 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b772): Mirror 6: 0xcf35c9b (stale) Mirror 9: 0xcf35c9b (stale) Mirror 12: 0xcf35c9b (stale) Mirror 13: 0x6e3ef3c (stale) Mirror 15: 0x2a894805 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b772: skipped 000400 1790896655.916880 mapread 0x032fba thru 0x03b771 (0x87b8 bytes) 000401 1790896655.954549 mapwrite 0x0308da thru 0x03dea3 (0xd5ca bytes) 000402 1790896656.030033 read 0x013913 thru 0x022b15 (0xf203 bytes) 000403 1790896656.043079 punch from 0x003fc8 to 0x00c9f3 (0x8a2c bytes) 000404 1790896656.055434 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3dea4): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dea4): Mirror 6: 0x27488997 (stale) Mirror 9: 0x27488997 (stale) Mirror 12: 0x27488997 (stale) Mirror 13: 0x77ea468b (stale) Mirror 15: 0x5be42865 (stale) CRC-32 checksum value for chunk [0, 0x3dea4): Mirror 1: 0x4b19d7c4 Mirror 16: 0x4b19d7c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dea4: skipped 000405 1790896656.890518 punch from 0x034b87 to 0x03c142 (0x75bc bytes) 000406 1790896656.945710 punch from 0x0066ac to 0x01107c (0xa9d1 bytes) 000407 1790896656.952456 write 0x035f32 thru 0x039762 (0x3831 bytes) 000408 1790896656.972761 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dea4): Mirror 9: 0x27488997 (stale) Mirror 12: 0x27488997 (stale) Mirror 13: 0x77ea468b (stale) Mirror 15: 0x5be42865 (stale) Mirror 16: 0x4b19d7c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dea4): Mirror 9: 0x27488997 (stale) Mirror 12: 0x27488997 (stale) Mirror 13: 0x77ea468b (stale) Mirror 15: 0x5be42865 (stale) Mirror 16: 0x4b19d7c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dea4: skipped 000409 1790896657.413486 fallocate 0x029e01 thru 0x034a15 (0xac15 bytes) 000410 1790896657.421572 trunc from 0x027bd5 to 0x030c28 (0x9054 bytes) 000411 1790896657.428442 punch from 0x02be38 to 0x0362ee (0xa4b7 bytes) 000412 1790896657.434633 mapread 0x01e7a1 thru 0x026426 (0x7c86 bytes) 000413 1790896657.450103 read 0x00bbbe thru 0x00bf48 (0x38b bytes) 000414 1790896657.453704 write 0x01e9c8 thru 0x026d1a (0x8353 bytes) 000415 1790896657.459729 trunc from 0x010018 to 0x01269d (0x2686 bytes) 000416 1790896657.463846 trunc from 0x0091b2 to 0x00cf7e (0x3dcd bytes) 000417 1790896657.468105 write 0x018315 thru 0x024437 (0xc123 bytes) 000418 1790896657.473762 write 0x03f74f thru 0x03ffff (0x8b1 bytes) 000419 1790896657.474769 trunc from 0x040000 to 0x03dc5c (0x23a3 bytes) 000421 1790896657.506773 punch from 0x03b3a5 to 0x03dc5c (0x28b8 bytes) 000423 1790896657.515079 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3dc5d): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc5d): Mirror 9: 0x3c5262a6 (stale) Mirror 12: 0x3c5262a6 (stale) Mirror 13: 0x44120914 (stale) Mirror 15: 0x19c57b7a (stale) Mirror 16: 0xa50a92cd (stale) CRC-32 checksum value for chunk [0, 0x3dc5d): Mirror 1: 0xf346c012 Mirror 17: 0xf346c012 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc5d: skipped 000424 1790896658.444457 MIRROR_RESYNC 7 mirrors 000426 1790896659.828751 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) 000427 1790896660.115347 read 0x0378f2 thru 0x03908b (0x179a bytes) 000429 1790896660.123909 trunc from 0x03671f to 0x038044 (0x1926 bytes) 000430 1790896660.182291 MIRROR_RESYNC 6 mirrors 000431 1790896661.787629 fallocate 0x0132cd thru 0x01b58b (0x82bf bytes) 000432 1790896661.829062 mapwrite 0x005995 thru 0x006a03 (0x106f bytes) 000433 1790896661.855978 fallocate 0x022971 thru 0x0256be (0x2d4e bytes) 000434 1790896661.861997 mapwrite 0x00a999 thru 0x018ae5 (0xe14d bytes) 000435 1790896661.898623 punch from 0x025269 to 0x02b1bb (0x5f53 bytes) 000436 1790896661.903566 fallocate 0x021742 thru 0x02d212 (0xbad1 bytes) 000437 1790896661.911568 trunc from 0x03dc5d to 0x03709e (0x6bbe bytes) 000438 1790896661.945401 trunc from 0x01bd1f to 0x01ed03 (0x2fe5 bytes) 000439 1790896661.954932 trunc from 0x003695 to 0x00b834 (0x81a0 bytes) 000440 1790896661.961637 punch from 0x014c62 to 0x017bc7 (0x2f66 bytes) 000441 1790896661.975967 mapwrite 0x02328a thru 0x02b608 (0x837f bytes) 000443 1790896662.023063 punch from 0x031aaf to 0x03709e (0x55f0 bytes) 000444 1790896662.027611 write 0x000ca7 thru 0x010584 (0xf8de bytes) 000445 1790896662.036704 trunc from 0x03709f to 0x02f5e8 (0x7ab6 bytes) 000446 1790896662.075371 mapwrite 0x00db16 thru 0x016653 (0x8b3e bytes) 000447 1790896662.117812 punch from 0x008a99 to 0x00e118 (0x5680 bytes) 000448 1790896662.126291 mapread 0x01b19b thru 0x01bd33 (0xb99 bytes) 000450 1790896662.136540 punch from 0x007859 to 0x00aa0b (0x31b3 bytes) 000451 1790896662.145168 trunc from 0x023105 to 0x024466 (0x1362 bytes) 000452 1790896662.151635 write 0x01ad91 thru 0x01c4b1 (0x1721 bytes) 000453 1790896662.152724 fallocate 0x0146e7 thru 0x0155a8 (0xec2 bytes) 000454 1790896662.158077 read 0x00dc7f thru 0x015465 (0x77e7 bytes) 000455 1790896662.172434 read 0x00faf8 thru 0x01f1e2 (0xf6eb bytes) 000456 1790896662.175777 trunc from 0x0240cf to 0x0298e7 (0x5819 bytes) 000457 1790896662.185714 trunc from 0x02f5e9 to 0x02efb1 (0x637 bytes) 000458 1790896662.205607 mapread 0x005e27 thru 0x00b694 (0x586e bytes) 000460 1790896662.222641 mapread 0x021a92 thru 0x028db3 (0x7322 bytes) 000461 1790896662.223741 mapread 0x0157f2 thru 0x0201be (0xa9cd bytes) 000462 1790896662.225923 trunc from 0x02efb2 to 0x01557c (0x19a35 bytes) 000463 1790896662.239713 read 0x00d835 thru 0x01557c (0x7d48 bytes) 000464 1790896662.244501 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1557d): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1557d): Mirror 12: 0xd027d4e6 (stale) Mirror 13: 0xd027d4e6 (stale) Mirror 15: 0xd027d4e6 (stale) Mirror 16: 0xd027d4e6 (stale) Mirror 17: 0xd027d4e6 (stale) CRC-32 checksum value for chunk [0, 0x1557d): Mirror 1: 0xa4d9f977 Mirror 18: 0xa4d9f977 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1557d: skipped 000466 1790896662.929598 trunc from 0x010ea2 to 0x01557c (0x46db bytes) 000467 1790896662.984430 punch from 0x000bdf to 0x0084ed (0x790f bytes) 000468 1790896662.991586 mapwrite 0x0261eb thru 0x0289fa (0x2810 bytes) 000469 1790896663.055823 trunc from 0x0289fb to 0x013a49 (0x14fb1 bytes) 000470 1790896663.070578 read 0x00c994 thru 0x00e5c0 (0x1c2d bytes) 000471 1790896663.079574 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x13a4a): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13a4a): Mirror 12: 0xf3862207 (stale) Mirror 13: 0xf3862207 (stale) Mirror 15: 0xf3862207 (stale) Mirror 16: 0xf3862207 (stale) Mirror 17: 0xf3862207 (stale) Mirror 18: 0x55a63ea5 (stale) CRC-32 checksum value for chunk [0, 0x13a4a): Mirror 1: 0x57a5a6e Mirror 19: 0x57a5a6e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13a4a: skipped 000472 1790896664.169813 mapread 0x008379 thru 0x009f4a (0x1bd2 bytes) 000475 1790896664.213670 trunc from 0x004d47 to 0x007800 (0x2aba bytes) 000476 1790896664.249090 write 0x01aa59 thru 0x021ec3 (0x746b bytes) 000477 1790896664.263528 MIRROR_RESYNC 8 mirrors 000478 1790896665.721860 read 0x00d4f9 thru 0x0156e1 (0x81e9 bytes) 000479 1790896665.730545 write 0x023606 thru 0x02705e (0x3a59 bytes) 000481 1790896665.778983 trunc from 0x015279 to 0x016296 (0x101e bytes) 000482 1790896665.789058 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2705f): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2705f): Mirror 12: 0x6af3c3 (stale) Mirror 13: 0x6af3c3 (stale) Mirror 15: 0x6af3c3 (stale) Mirror 16: 0x6af3c3 (stale) Mirror 17: 0x6af3c3 (stale) Mirror 18: 0x6af3c3 (stale) Mirror 19: 0x6af3c3 (stale) CRC-32 checksum value for chunk [0, 0x2705f): Mirror 1: 0x9f496854 Mirror 20: 0x9f496854 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2705f: skipped 000483 1790896666.817307 write 0x03f06a thru 0x03ffff (0xf96 bytes) 000484 1790896666.856334 trunc from 0x040000 to 0x0194e0 (0x26b1f bytes) 000485 1790896666.871553 fallocate 0x02e7b5 thru 0x0345ac (0x5df8 bytes) 000486 1790896666.878876 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x345ad): Mirror 12: 0x2c5de6e (stale) Mirror 13: 0x2c5de6e (stale) Mirror 16: 0x2c5de6e (stale) Mirror 17: 0x2c5de6e (stale) Mirror 18: 0x2c5de6e (stale) Mirror 19: 0x2c5de6e (stale) Mirror 20: 0x446ad7a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x345ad): Mirror 12: 0x2c5de6e (stale) Mirror 13: 0x2c5de6e (stale) Mirror 16: 0x2c5de6e (stale) Mirror 17: 0x2c5de6e (stale) Mirror 18: 0x2c5de6e (stale) Mirror 19: 0x2c5de6e (stale) Mirror 20: 0x446ad7a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x345ad: skipped 000487 1790896667.426388 write 0x00c77b thru 0x01780e (0xb094 bytes) 000488 1790896667.444856 write 0x02affc thru 0x036d8a (0xbd8f bytes) 000489 1790896667.455908 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36d8b): Mirror 12: 0xffe71cd5 (stale) Mirror 13: 0xffe71cd5 (stale) Mirror 16: 0xffe71cd5 (stale) Mirror 17: 0xffe71cd5 (stale) Mirror 18: 0xffe71cd5 (stale) Mirror 19: 0xffe71cd5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d8b): Mirror 12: 0xffe71cd5 (stale) Mirror 13: 0xffe71cd5 (stale) Mirror 16: 0xffe71cd5 (stale) Mirror 17: 0xffe71cd5 (stale) Mirror 18: 0xffe71cd5 (stale) Mirror 19: 0xffe71cd5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d8b: skipped 000490 1790896668.027467 mapwrite 0x027da7 thru 0x028ffa (0x1254 bytes) 000491 1790896668.063605 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x36d8b): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d8b): Mirror 12: 0xffe71cd5 (stale) Mirror 13: 0xffe71cd5 (stale) Mirror 16: 0xffe71cd5 (stale) Mirror 17: 0xffe71cd5 (stale) Mirror 18: 0xffe71cd5 (stale) Mirror 19: 0xffe71cd5 (stale) CRC-32 checksum value for chunk [0, 0x36d8b): Mirror 1: 0xce901720 Mirror 20: 0xce901720 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d8b: skipped 000492 1790896669.014724 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36d8b): Mirror 12: 0xffe71cd5 (stale) Mirror 13: 0xffe71cd5 (stale) Mirror 16: 0xffe71cd5 (stale) Mirror 18: 0xffe71cd5 (stale) Mirror 19: 0xffe71cd5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d8b): Mirror 12: 0xffe71cd5 (stale) Mirror 13: 0xffe71cd5 (stale) Mirror 16: 0xffe71cd5 (stale) Mirror 18: 0xffe71cd5 (stale) Mirror 19: 0xffe71cd5 (stale) CRC-32 checksum value for chunk [0, 0x36d8b): Mirror 1: 0xce901720 Mirror 20: 0xce901720 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d8b: skipped 000493 1790896669.902314 punch from 0x01220a to 0x012c17 (0xa0e bytes) 000495 1790896669.954421 trunc from 0x0172ec to 0x021d71 (0xaa86 bytes) 000496 1790896669.961629 fallocate 0x00edad thru 0x0168c1 (0x7b15 bytes) 000497 1790896669.968037 punch from 0x01e1cb to 0x0264ea (0x8320 bytes) 000498 1790896669.981765 trunc from 0x036d8b to 0x02b224 (0xbb66 bytes) 000499 1790896670.030055 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2b225): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b225): Mirror 12: 0x578e8e76 (stale) Mirror 13: 0x578e8e76 (stale) Mirror 16: 0x578e8e76 (stale) Mirror 18: 0x578e8e76 (stale) Mirror 19: 0x578e8e76 (stale) Mirror 20: 0xedc34ce2 (stale) CRC-32 checksum value for chunk [0, 0x2b225): Mirror 1: 0x38571296 Mirror 21: 0x38571296 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b225: skipped 000500 1790896671.200416 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b225): Mirror 12: 0x578e8e76 (stale) Mirror 13: 0x578e8e76 (stale) Mirror 16: 0x578e8e76 (stale) Mirror 19: 0x578e8e76 (stale) Mirror 20: 0xedc34ce2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b225): Mirror 12: 0x578e8e76 (stale) Mirror 13: 0x578e8e76 (stale) Mirror 16: 0x578e8e76 (stale) Mirror 19: 0x578e8e76 (stale) Mirror 20: 0xedc34ce2 (stale) CRC-32 checksum value for chunk [0, 0x2b225): Mirror 1: 0x38571296 Mirror 21: 0x38571296 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b225: skipped 000501 1790896671.967339 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b225): Mirror 12: 0x578e8e76 (stale) Mirror 16: 0x578e8e76 (stale) Mirror 19: 0x578e8e76 (stale) Mirror 20: 0xedc34ce2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b225): Mirror 12: 0x578e8e76 (stale) Mirror 16: 0x578e8e76 (stale) Mirror 19: 0x578e8e76 (stale) Mirror 20: 0xedc34ce2 (stale) CRC-32 checksum value for chunk [0, 0x2b225): Mirror 1: 0x38571296 Mirror 21: 0x38571296 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b225: skipped 000502 1790896672.449005 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b225): Mirror 12: 0x578e8e76 (stale) Mirror 16: 0x578e8e76 (stale) Mirror 19: 0x578e8e76 (stale) Mirror 20: 0xedc34ce2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b225): Mirror 12: 0x578e8e76 (stale) Mirror 16: 0x578e8e76 (stale) Mirror 19: 0x578e8e76 (stale) Mirror 20: 0xedc34ce2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b225: skipped 000503 1790896672.850261 fallocate 0x01020c thru 0x01b256 (0xb04b bytes) 000504 1790896672.870571 trunc from 0x029dcf to 0x02b224 (0x1456 bytes) 000505 1790896672.882026 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2b225): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b225): Mirror 12: 0x578e8e76 (stale) Mirror 16: 0x578e8e76 (stale) Mirror 19: 0x578e8e76 (stale) Mirror 20: 0xedc34ce2 (stale) CRC-32 checksum value for chunk [0, 0x2b225): Mirror 1: 0x4c5c525e Mirror 21: 0x4c5c525e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b225: skipped 000506 1790896673.763126 trunc from 0x02b225 to 0x02a095 (0x118f bytes) 000507 1790896673.841953 trunc from 0x02a096 to 0x02e4d4 (0x443f bytes) 000508 1790896673.866524 mapwrite 0x02682b thru 0x033b4c (0xd322 bytes) 000509 1790896673.946035 MIRROR_RESYNC 6 mirrors 000511 1790896675.402982 write 0x02f1cf thru 0x03ee12 (0xfc44 bytes) 000512 1790896675.462674 mapwrite 0x03f590 thru 0x03ffff (0xa70 bytes) 000514 1790896675.528440 fallocate 0x01221b thru 0x01d1a4 (0xaf8a bytes) 000515 1790896675.534642 punch from 0x01e8e5 to 0x01f5aa (0xcc6 bytes) 000516 1790896675.549252 fallocate 0x030bee thru 0x03a63f (0x9a52 bytes) 000517 1790896675.568156 read 0x028201 thru 0x03106e (0x8e6e bytes) 000518 1790896675.577692 mapwrite 0x03ebd2 thru 0x03ffff (0x142e bytes) 000519 1790896675.605710 trunc from 0x015015 to 0x018e73 (0x3e5f bytes) 000520 1790896675.615585 mapread 0x0008a0 thru 0x00729c (0x69fd bytes) 000521 1790896675.644801 read 0x03d828 thru 0x03ffff (0x27d8 bytes) 000522 1790896675.648270 write 0x011074 thru 0x015d6b (0x4cf8 bytes) 000523 1790896675.671321 write 0x03c251 thru 0x03ffff (0x3daf bytes) 000524 1790896675.672386 write 0x00d40b thru 0x01842e (0xb024 bytes) 000525 1790896675.677978 punch from 0x012953 to 0x013cb8 (0x1366 bytes) 000526 1790896675.715174 write 0x024345 thru 0x02a18e (0x5e4a bytes) 000527 1790896675.718139 read 0x017719 thru 0x021fe5 (0xa8cd bytes) 000528 1790896675.729892 MIRROR_RESYNC 6 mirrors 000529 1790896677.159030 write 0x021532 thru 0x022ba9 (0x1678 bytes) 000530 1790896677.204071 mapread 0x0006a4 thru 0x00266a (0x1fc7 bytes) 000532 1790896677.220701 write 0x010ba5 thru 0x011aae (0xf0a bytes) 000534 1790896677.238973 mapread 0x03eb0d thru 0x03ffff (0x14f3 bytes) 000535 1790896677.255043 trunc from 0x040000 to 0x02104d (0x1efb2 bytes) 000536 1790896677.282912 trunc from 0x02104e to 0x029a4b (0x89fe bytes) 000537 1790896677.304023 write 0x01ddab thru 0x01e683 (0x8d9 bytes) 000538 1790896677.319684 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29a4c): Mirror 12: 0x3e9a36d9 (stale) Mirror 16: 0x3e9a36d9 (stale) Mirror 20: 0x3e9a36d9 (stale) Mirror 21: 0x3e9a36d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29a4c): Mirror 12: 0x3e9a36d9 (stale) Mirror 16: 0x3e9a36d9 (stale) Mirror 20: 0x3e9a36d9 (stale) Mirror 21: 0x3e9a36d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29a4c: skipped 000539 1790896677.806982 read 0x00b0b4 thru 0x00fade (0x4a2b bytes) 000540 1790896677.818808 trunc from 0x029a4c to 0x01dcb2 (0xbd99 bytes) 000541 1790896677.837808 read 0x01aa51 thru 0x01dcb2 (0x3262 bytes) 000542 1790896677.855161 punch from 0x0106cf to 0x01b892 (0xb1c4 bytes) 000543 1790896677.861500 mapwrite 0x0259f0 thru 0x034dcd (0xf3de bytes) 000544 1790896677.906927 read 0x01c184 thru 0x0257f2 (0x966f bytes) 000545 1790896677.907635 mapwrite 0x02f50b thru 0x036758 (0x724e bytes) 000546 1790896677.979516 mapwrite 0x01b740 thru 0x01d9a2 (0x2263 bytes) 000547 1790896677.992716 fallocate 0x001751 thru 0x00618d (0x4a3d bytes) 000548 1790896677.996884 fallocate 0x00f088 thru 0x01dedf (0xee58 bytes) 000549 1790896678.002057 trunc from 0x02e4d1 to 0x02fa65 (0x1595 bytes) 000550 1790896678.007570 write 0x0006a5 thru 0x00e0ae (0xda0a bytes) 000551 1790896678.009045 mapread 0x035892 thru 0x036758 (0xec7 bytes) 000552 1790896678.009938 read 0x0322a6 thru 0x036758 (0x44b3 bytes) 000553 1790896678.010509 trunc from 0x036759 to 0x035a40 (0xd18 bytes) 000554 1790896678.037803 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35a41): Mirror 16: 0x1c95f291 (stale) Mirror 20: 0x1c95f291 (stale) Mirror 21: 0x1c95f291 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a41): Mirror 16: 0x1c95f291 (stale) Mirror 20: 0x1c95f291 (stale) Mirror 21: 0x1c95f291 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a41: skipped 000555 1790896678.389520 trunc from 0x035a41 to 0x01dd30 (0x17d10 bytes) 000556 1790896678.408427 mapread 0x002d56 thru 0x004787 (0x1a32 bytes) 000557 1790896678.425333 read 0x00cbd3 thru 0x019197 (0xc5c5 bytes) 000558 1790896678.432357 mapwrite 0x01174f thru 0x012a5e (0x1310 bytes) 000559 1790896678.449583 trunc from 0x012e06 to 0x01b406 (0x8601 bytes) 000560 1790896678.465951 mapwrite 0x023181 thru 0x02d404 (0xa284 bytes) 000561 1790896678.552767 punch from 0x017add to 0x01b576 (0x3a9a bytes) 000562 1790896678.561617 trunc from 0x02d405 to 0x013b08 (0x198fc bytes) 000563 1790896678.587378 fallocate 0x00e1da thru 0x010e5c (0x2c83 bytes) 000565 1790896678.597807 punch from 0x00b075 to 0x013b08 (0x8a94 bytes) 000566 1790896678.605525 write 0x01ed3d thru 0x01eddf (0xa3 bytes) 000567 1790896678.606459 write 0x01087e thru 0x01c109 (0xb88c bytes) 000568 1790896678.630867 read 0x0014c7 thru 0x00caf4 (0xb62e bytes) 000569 1790896678.649692 MIRROR_RESYNC 4 mirrors 000570 1790896679.636530 write 0x00a890 thru 0x00fa8a (0x51fb bytes) 000571 1790896679.671750 trunc from 0x001519 to 0x006846 (0x532e bytes) 000572 1790896679.676457 mapwrite 0x023fb5 thru 0x02cdac (0x8df8 bytes) 000573 1790896679.731534 trunc from 0x0298a4 to 0x02cdac (0x3509 bytes) 000574 1790896679.736288 mapread 0x003c1e thru 0x0049d6 (0xdb9 bytes) 000575 1790896679.744925 mapread 0x0167fc thru 0x022412 (0xbc17 bytes) 000576 1790896679.753377 mapread 0x013a13 thru 0x01b783 (0x7d71 bytes) 000577 1790896679.754734 mapwrite 0x03878d thru 0x03ec12 (0x6486 bytes) 000578 1790896679.790757 punch from 0x024bbb to 0x02e3b9 (0x97ff bytes) 000579 1790896679.795117 punch from 0x00fa0a to 0x01c365 (0xc95c bytes) 000580 1790896679.799599 trunc from 0x03ec13 to 0x009c96 (0x34f7c bytes) 000581 1790896679.811419 trunc from 0x009c97 to 0x00ef98 (0x5302 bytes) 000582 1790896679.824856 read 0x000731 thru 0x00e4a6 (0xdd76 bytes) 000583 1790896679.831987 mapwrite 0x01a507 thru 0x01f85e (0x5358 bytes) 000584 1790896679.890597 punch from 0x00135e to 0x009bdd (0x8880 bytes) 000585 1790896679.901486 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1f85f): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f85f): Mirror 16: 0xd95bb2cd (stale) Mirror 20: 0xd95bb2cd (stale) Mirror 21: 0xd95bb2cd (stale) CRC-32 checksum value for chunk [0, 0x1f85f): Mirror 1: 0x95e29350 Mirror 22: 0x95e29350 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f85f: skipped 000586 1790896680.829100 mapwrite 0x02384d thru 0x023bce (0x382 bytes) 000587 1790896680.935000 write 0x036d38 thru 0x03c49a (0x5763 bytes) 000588 1790896680.939010 read 0x038d9f thru 0x03c49a (0x36fc bytes) 000590 1790896680.944661 fallocate 0x01a4bc thru 0x0280f3 (0xdc38 bytes) 000592 1790896680.957672 fallocate 0x017e19 thru 0x027d6f (0xff57 bytes) 000594 1790896680.963347 read 0x01fdbc thru 0x0294dc (0x9721 bytes) 000595 1790896680.974344 trunc from 0x00efb2 to 0x010d5c (0x1dab bytes) 000596 1790896680.978448 mapread 0x031b50 thru 0x03c49a (0xa94b bytes) 000597 1790896680.979742 read 0x01e80b thru 0x025193 (0x6989 bytes) 000599 1790896680.980877 mapwrite 0x023525 thru 0x030eca (0xd9a6 bytes) 000600 1790896681.004929 read 0x00f09d thru 0x015808 (0x676c bytes) 000601 1790896681.013608 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c49b): Mirror 20: 0xa980d0c5 (stale) Mirror 21: 0xa980d0c5 (stale) Mirror 22: 0x30167c79 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c49b): Mirror 20: 0xa980d0c5 (stale) Mirror 21: 0xa980d0c5 (stale) Mirror 22: 0x30167c79 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c49b: skipped 000602 1790896681.386067 write 0x01b8c8 thru 0x01b92a (0x63 bytes) 000603 1790896681.401667 punch from 0x004e46 to 0x011fac (0xd167 bytes) 000604 1790896681.413341 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c49b): Mirror 21: 0xa980d0c5 (stale) Mirror 22: 0x30167c79 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c49b): Mirror 21: 0xa980d0c5 (stale) Mirror 22: 0x30167c79 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c49b: skipped 000606 1790896681.689439 trunc from 0x00cb2e to 0x00d034 (0x507 bytes) 000607 1790896681.694588 fallocate 0x017094 thru 0x0236b4 (0xc621 bytes) 000608 1790896681.698403 write 0x0256f5 thru 0x029a6e (0x437a bytes) 000609 1790896681.710853 mapread 0x038cc1 thru 0x03c49a (0x37da bytes) 000610 1790896681.735133 read 0x028bfe thru 0x02c215 (0x3618 bytes) 000611 1790896681.740539 fallocate 0x03e71d thru 0x03fb20 (0x1404 bytes) 000612 1790896681.745923 write 0x009af4 thru 0x012e5b (0x9368 bytes) 000613 1790896681.756642 mapwrite 0x03695e thru 0x03ffff (0x96a2 bytes) 000615 1790896681.797968 trunc from 0x040000 to 0x0316eb (0xe914 bytes) 000616 1790896681.812239 trunc from 0x0316ec to 0x02af93 (0x6758 bytes) 000618 1790896681.831743 write 0x038ceb thru 0x03ffff (0x7315 bytes) 000619 1790896681.834067 trunc from 0x040000 to 0x01b23a (0x24dc5 bytes) 000620 1790896681.866061 trunc from 0x01b23b to 0x03e109 (0x22ecf bytes) 000621 1790896681.911777 read 0x01069a thru 0x010a74 (0x3db bytes) 000622 1790896681.913748 trunc from 0x03e10a to 0x03a7d9 (0x3930 bytes) 000623 1790896681.944496 MIRROR_RESYNC 3 mirrors 000624 1790896682.870969 MIRROR_EXTEND to 4 mirrors 000627 1790896683.584919 mapread 0x023027 thru 0x031c8e (0xec68 bytes) 000628 1790896683.617419 mapwrite 0x034185 thru 0x03f14b (0xafc7 bytes) 000629 1790896683.712641 fallocate 0x039c4c thru 0x03ff61 (0x6316 bytes) 000630 1790896683.717800 trunc from 0x006f49 to 0x010063 (0x911b bytes) 000631 1790896683.724304 trunc from 0x021618 to 0x02c69e (0xb087 bytes) 000632 1790896683.748111 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ff62): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff62): Mirror 21: 0x539a07bf (stale) Mirror 22: 0x539a07bf (stale) Mirror 23: 0x539a07bf (stale) CRC-32 checksum value for chunk [0, 0x3ff62): Mirror 1: 0xc3c7e9ee Mirror 24: 0xc3c7e9ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff62: skipped 000633 1790896684.656263 mapread 0x005a64 thru 0x00ed8e (0x932b bytes) 000634 1790896684.690095 punch from 0x01d5f9 to 0x026065 (0x8a6d bytes) 000635 1790896684.742571 punch from 0x0255d1 to 0x02b841 (0x6271 bytes) 000636 1790896684.750336 fallocate 0x018514 thru 0x01b55a (0x3047 bytes) 000637 1790896684.761283 mapwrite 0x01b9b6 thru 0x02b100 (0xf74b bytes) 000639 1790896684.835647 write 0x01f50c thru 0x02a2f7 (0xadec bytes) 000640 1790896684.843745 trunc from 0x03ff62 to 0x02e15f (0x11e02 bytes) 000641 1790896684.860781 trunc from 0x02e160 to 0x01b549 (0x12c16 bytes) 000642 1790896684.876451 mapwrite 0x000f54 thru 0x005d3c (0x4de9 bytes) 000643 1790896684.902041 mapread 0x0159ab thru 0x019d9a (0x43f0 bytes) 000644 1790896684.903410 read 0x006a3e thru 0x006a9b (0x5e bytes) 000646 1790896684.904805 punch from 0x00d1fe to 0x017f17 (0xad1a bytes) 000647 1790896684.908850 punch from 0x0014f9 to 0x0066e5 (0x51ed bytes) 000649 1790896684.914596 read 0x012ef5 thru 0x01b549 (0x8655 bytes) 000650 1790896684.922260 mapread 0x010853 thru 0x015843 (0x4ff1 bytes) 000651 1790896684.923032 mapwrite 0x02d7a4 thru 0x03587b (0x80d8 bytes) 000653 1790896684.990812 write 0x034ba6 thru 0x03fd13 (0xb16e bytes) 000654 1790896684.997685 write 0x01e80c thru 0x029da2 (0xb597 bytes) 000656 1790896684.999787 mapwrite 0x03c8f9 thru 0x03ffff (0x3707 bytes) 000657 1790896685.074840 fallocate 0x0381f4 thru 0x03ffff (0x7e0c bytes) 000658 1790896685.081988 punch from 0x018b8a to 0x020124 (0x759b bytes) 000659 1790896685.090908 write 0x01eaab thru 0x029b8f (0xb0e5 bytes) 000660 1790896685.101049 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x53992a0d (stale) Mirror 22: 0x53992a0d (stale) Mirror 23: 0x53992a0d (stale) Mirror 24: 0x8cb2e576 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x92022e54 Mirror 25: 0x92022e54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000661 1790896685.977029 write 0x00abd7 thru 0x0145c6 (0x99f0 bytes) 000662 1790896686.045459 mapread 0x03b9c7 thru 0x03ffff (0x4639 bytes) 000663 1790896686.073709 fallocate 0x03d963 thru 0x03ffff (0x269d bytes) 000664 1790896686.090486 read 0x031f12 thru 0x032b2e (0xc1d bytes) 000665 1790896686.098403 read 0x029687 thru 0x033a08 (0xa382 bytes) 000666 1790896686.127440 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x53992a0d (stale) Mirror 22: 0x53992a0d (stale) Mirror 23: 0x53992a0d (stale) Mirror 24: 0x8cb2e576 (stale) Mirror 25: 0x92022e54 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd319b3a0 Mirror 26: 0xd319b3a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000667 1790896686.958991 trunc from 0x040000 to 0x011af6 (0x2e509 bytes) 000668 1790896687.049440 trunc from 0x006a52 to 0x00e1ed (0x779c bytes) 000669 1790896687.054264 punch from 0x011765 to 0x011af6 (0x392 bytes) 000670 1790896687.058891 mapwrite 0x00c887 thru 0x018aac (0xc226 bytes) 000671 1790896687.132432 read 0x00bf1c thru 0x00d298 (0x137d bytes) 000672 1790896687.133271 fallocate 0x02d307 thru 0x03594d (0x8647 bytes) 000673 1790896687.140854 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18aad): Mirror 21: 0x125fb40c (stale) Mirror 22: 0x125fb40c (stale) Mirror 23: 0x125fb40c (stale) Mirror 25: 0x3b21353a (stale) Mirror 26: 0x9454408d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18aad): Mirror 21: 0x125fb40c (stale) Mirror 22: 0x125fb40c (stale) Mirror 23: 0x125fb40c (stale) Mirror 25: 0x3b21353a (stale) Mirror 26: 0x9454408d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18aad: skipped 000674 1790896687.540037 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18aad): Mirror 21: 0x125fb40c (stale) Mirror 22: 0x125fb40c (stale) Mirror 25: 0x3b21353a (stale) Mirror 26: 0x9454408d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18aad): Mirror 21: 0x125fb40c (stale) Mirror 22: 0x125fb40c (stale) Mirror 25: 0x3b21353a (stale) Mirror 26: 0x9454408d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18aad: skipped 000675 1790896687.913299 read 0x00f11c thru 0x011085 (0x1f6a bytes) 000676 1790896687.918161 trunc from 0x00e182 to 0x0159fa (0x7879 bytes) 000677 1790896687.921876 trunc from 0x018aad to 0x02cb4b (0x1409f bytes) 000678 1790896687.956338 MIRROR_RESYNC 5 mirrors 000679 1790896689.211398 mapwrite 0x023af8 thru 0x032e35 (0xf33e bytes) 000680 1790896689.293215 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x32e36): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32e36): Mirror 21: 0xffc8eca0 (stale) Mirror 22: 0xffc8eca0 (stale) Mirror 25: 0xffc8eca0 (stale) Mirror 26: 0xffc8eca0 (stale) CRC-32 checksum value for chunk [0, 0x32e36): Mirror 1: 0x4a4ede87 Mirror 27: 0x4a4ede87 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32e36: skipped 000681 1790896690.211223 fallocate 0x01d225 thru 0x02d0be (0xfe9a bytes) 000682 1790896690.266718 trunc from 0x032e36 to 0x03dbc2 (0xad8d bytes) 000683 1790896690.332419 read 0x01d15a thru 0x0209ab (0x3852 bytes) 000684 1790896690.338915 trunc from 0x03dbc3 to 0x03a7a9 (0x3419 bytes) 000685 1790896690.353795 fallocate 0x0045be thru 0x0047f4 (0x237 bytes) 000686 1790896690.364978 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a7aa): Mirror 21: 0x2250d06b (stale) Mirror 22: 0x2250d06b (stale) Mirror 26: 0x2250d06b (stale) Mirror 27: 0x880e458c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a7aa): Mirror 21: 0x2250d06b (stale) Mirror 22: 0x2250d06b (stale) Mirror 26: 0x2250d06b (stale) Mirror 27: 0x880e458c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a7aa: skipped 000687 1790896690.759677 write 0x02b2c2 thru 0x0376a2 (0xc3e1 bytes) 000689 1790896690.775702 mapread 0x029268 thru 0x02faac (0x6845 bytes) 000690 1790896690.785709 punch from 0x014a35 to 0x01a9a7 (0x5f73 bytes) 000691 1790896690.793640 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a7aa): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a7aa): Mirror 21: 0x2250d06b (stale) Mirror 22: 0x2250d06b (stale) Mirror 26: 0x2250d06b (stale) Mirror 27: 0x880e458c (stale) CRC-32 checksum value for chunk [0, 0x3a7aa): Mirror 1: 0x2f62b7c Mirror 28: 0x2f62b7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a7aa: skipped 000692 1790896691.723681 punch from 0x01e60d to 0x0226f7 (0x40eb bytes) 000693 1790896691.771496 mapwrite 0x0398df thru 0x03ffff (0x6721 bytes) 000694 1790896691.874838 punch from 0x005a47 to 0x00c72b (0x6ce5 bytes) 000695 1790896691.890132 punch from 0x00c200 to 0x0117d4 (0x55d5 bytes) 000696 1790896691.899748 trunc from 0x026ef7 to 0x02d12e (0x6238 bytes) 000697 1790896691.910643 mapwrite 0x035379 thru 0x03ffff (0xac87 bytes) 000698 1790896691.932784 trunc from 0x001ee2 to 0x00a3ac (0x84cb bytes) 000699 1790896691.942442 write 0x03b6c4 thru 0x03ffff (0x493c bytes) 000700 1790896691.943622 trunc from 0x040000 to 0x03b933 (0x46cc bytes) 000701 1790896691.963514 fallocate 0x03a1e3 thru 0x03ffff (0x5e1d bytes) 000702 1790896691.978259 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b934): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b934): Mirror 21: 0x5470cc2f (stale) Mirror 22: 0x5470cc2f (stale) Mirror 26: 0x5470cc2f (stale) Mirror 27: 0xfea08caf (stale) Mirror 28: 0xa451ab03 (stale) CRC-32 checksum value for chunk [0, 0x3b934): Mirror 1: 0x58b2252c Mirror 29: 0x58b2252c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b934: skipped 000703 1790896692.748811 write 0x017d79 thru 0x01da53 (0x5cdb bytes) 000704 1790896692.805594 read 0x03296b thru 0x034a6c (0x2102 bytes) 000705 1790896692.821909 trunc from 0x033f4e to 0x03715a (0x320d bytes) 000706 1790896692.833786 trunc from 0x00c377 to 0x01578d (0x9417 bytes) 000707 1790896692.839048 trunc from 0x03b934 to 0x010850 (0x2b0e3 bytes) 000708 1790896692.874809 mapwrite 0x026356 thru 0x02da09 (0x76b4 bytes) 000709 1790896692.928869 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2da0a): Mirror 21: 0x17c64f49 (stale) Mirror 22: 0x17c64f49 (stale) Mirror 26: 0x17c64f49 (stale) Mirror 28: 0xa5960c35 (stale) Mirror 29: 0xe5c699a6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2da0a): Mirror 21: 0x17c64f49 (stale) Mirror 22: 0x17c64f49 (stale) Mirror 26: 0x17c64f49 (stale) Mirror 28: 0xa5960c35 (stale) Mirror 29: 0xe5c699a6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2da0a: skipped 000710 1790896693.289713 fallocate 0x01c107 thru 0x02413d (0x8037 bytes) 000712 1790896693.296387 read 0x027464 thru 0x02da09 (0x65a6 bytes) 000714 1790896693.307615 punch from 0x020608 to 0x02da09 (0xd402 bytes) 000715 1790896693.312060 trunc from 0x02da0a to 0x03ab83 (0xd17a bytes) 000716 1790896693.331444 trunc from 0x005f5f to 0x00b863 (0x5905 bytes) 000717 1790896693.336807 read 0x016ea4 thru 0x020a67 (0x9bc4 bytes) 000718 1790896693.354099 punch from 0x021808 to 0x02c069 (0xa862 bytes) 000719 1790896693.358629 fallocate 0x03c98c thru 0x03ffff (0x3674 bytes) 000720 1790896693.365422 trunc from 0x03ab84 to 0x01e073 (0x1cb10 bytes) 000721 1790896693.384167 fallocate 0x01f9e6 thru 0x02bc5e (0xc279 bytes) 000723 1790896693.393770 trunc from 0x027b52 to 0x02bc5e (0x410d bytes) 000724 1790896693.402454 fallocate 0x028b7c thru 0x02cc4d (0x40d2 bytes) 000725 1790896693.410854 trunc from 0x005d36 to 0x008a9b (0x2d66 bytes) 000727 1790896693.422748 write 0x035507 thru 0x03f7df (0xa2d9 bytes) 000728 1790896693.425566 fallocate 0x02ae18 thru 0x02cfff (0x21e8 bytes) 000729 1790896693.431076 trunc from 0x0213e9 to 0x02561f (0x4237 bytes) 000730 1790896693.444367 mapwrite 0x01d55b thru 0x01f402 (0x1ea8 bytes) 000731 1790896693.485013 MIRROR_RESYNC 6 mirrors 000732 1790896695.413120 MIRROR_RESYNC 6 mirrors 000733 1790896695.821843 trunc from 0x03f7e0 to 0x027b47 (0x17c98 bytes) 000734 1790896695.890791 mapread 0x01d830 thru 0x027b47 (0xa318 bytes) 000735 1790896695.917695 read 0x019b4e thru 0x01aabc (0xf6f bytes) 000737 1790896695.918735 punch from 0x00dc0b to 0x01504f (0x7445 bytes) 000739 1790896695.939531 trunc from 0x027b48 to 0x01a417 (0xd730 bytes) 000740 1790896695.973747 trunc from 0x004e89 to 0x007f82 (0x30fa bytes) 000741 1790896695.979862 write 0x00f05e thru 0x01c500 (0xd4a3 bytes) 000742 1790896695.999386 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1c501): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c501): Mirror 21: 0xb4d37504 (stale) Mirror 22: 0xb4d37504 (stale) Mirror 26: 0xb4d37504 (stale) Mirror 28: 0xb4d37504 (stale) Mirror 29: 0xb4d37504 (stale) CRC-32 checksum value for chunk [0, 0x1c501): Mirror 1: 0xdad95075 Mirror 30: 0xdad95075 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c501: skipped 000743 1790896697.006503 trunc from 0x01c501 to 0x022ec4 (0x69c4 bytes) 000744 1790896697.069719 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22ec5): Mirror 21: 0x9c5315c9 (stale) Mirror 22: 0x9c5315c9 (stale) Mirror 26: 0x9c5315c9 (stale) Mirror 28: 0x9c5315c9 (stale) Mirror 30: 0xea25dfaf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22ec5): Mirror 21: 0x9c5315c9 (stale) Mirror 22: 0x9c5315c9 (stale) Mirror 26: 0x9c5315c9 (stale) Mirror 28: 0x9c5315c9 (stale) Mirror 30: 0xea25dfaf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22ec5: skipped 000745 1790896697.548837 trunc from 0x022ec5 to 0x025c8e (0x2dca bytes) 000746 1790896697.568316 punch from 0x01927a to 0x025c8e (0xca15 bytes) 000747 1790896697.572881 read 0x011659 thru 0x018eb1 (0x7859 bytes) 000748 1790896697.582467 write 0x0296ad thru 0x02c315 (0x2c69 bytes) 000749 1790896697.583928 fallocate 0x0054ef thru 0x01004e (0xab60 bytes) 000750 1790896697.588835 trunc from 0x023d11 to 0x026a94 (0x2d84 bytes) 000751 1790896697.593370 read 0x025c08 thru 0x02c315 (0x670e bytes) 000752 1790896697.603736 trunc from 0x02c316 to 0x02c355 (0x40 bytes) 000754 1790896697.639938 mapwrite 0x029741 thru 0x03817f (0xea3f bytes) 000755 1790896697.700937 mapread 0x02cfb7 thru 0x0368bc (0x9906 bytes) 000756 1790896697.703953 fallocate 0x0291dd thru 0x0304a4 (0x72c8 bytes) 000757 1790896697.725924 trunc from 0x038180 to 0x0303aa (0x7dd5 bytes) 000758 1790896697.759714 punch from 0x018552 to 0x01f462 (0x6f11 bytes) 000759 1790896697.771744 mapwrite 0x03d4eb thru 0x03ffff (0x2b15 bytes) 000760 1790896697.821523 write 0x01915c thru 0x01bdbc (0x2c61 bytes) 000761 1790896697.834626 fallocate 0x0329b5 thru 0x036b7f (0x41cb bytes) 000762 1790896697.860155 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xe9c64e2f (stale) Mirror 22: 0xe9c64e2f (stale) Mirror 26: 0xe9c64e2f (stale) Mirror 30: 0x4436d333 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xe9c64e2f (stale) Mirror 22: 0xe9c64e2f (stale) Mirror 26: 0xe9c64e2f (stale) Mirror 30: 0x4436d333 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000763 1790896698.294928 trunc from 0x040000 to 0x00863d (0x379c2 bytes) 000764 1790896698.328977 trunc from 0x00863e to 0x0080a7 (0x596 bytes) 000765 1790896698.354041 read 0x00259c thru 0x0080a7 (0x5b0c bytes) 000767 1790896698.365678 mapwrite 0x00a1e3 thru 0x01206c (0x7e8a bytes) 000768 1790896698.450919 fallocate 0x01bada thru 0x029efe (0xe425 bytes) 000769 1790896698.466417 mapread 0x00862e thru 0x0119d1 (0x93a4 bytes) 000770 1790896698.472097 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x29eff): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 26: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29eff): Mirror 21: 0xc94dbd25 (stale) Mirror 22: 0xc94dbd25 (stale) Mirror 26: 0xc94dbd25 (stale) Mirror 30: 0x94377f0f (stale) CRC-32 checksum value for chunk [0, 0x29eff): Mirror 1: 0x7b9d05f6 Mirror 31: 0x7b9d05f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29eff: skipped 000771 1790896699.202879 read 0x028a50 thru 0x029efe (0x14af bytes) 000772 1790896699.207651 mapread 0x010a0b thru 0x0188f5 (0x7eeb bytes) 000773 1790896699.225558 read 0x02844b thru 0x029efe (0x1ab4 bytes) 000774 1790896699.226033 fallocate 0x030c20 thru 0x03d2d8 (0xc6b9 bytes) 000775 1790896699.271324 mapread 0x00dbe1 thru 0x012987 (0x4da7 bytes) 000776 1790896699.299550 trunc from 0x03d2d9 to 0x029093 (0x14245 bytes) 000777 1790896699.362727 mapwrite 0x023b91 thru 0x028f05 (0x5375 bytes) 000778 1790896699.392748 mapwrite 0x032943 thru 0x036ebb (0x4579 bytes) 000779 1790896699.503103 mapwrite 0x0109fa thru 0x01969d (0x8ca4 bytes) 000780 1790896699.564636 punch from 0x020694 to 0x020f16 (0x883 bytes) 000781 1790896699.572256 read 0x0285ae thru 0x02f242 (0x6c95 bytes) 000782 1790896699.584961 write 0x022c15 thru 0x023a30 (0xe1c bytes) 000784 1790896699.592338 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x36ebc): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 26: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ebc): Mirror 21: 0x2120af9a (stale) Mirror 22: 0x2120af9a (stale) Mirror 26: 0x2120af9a (stale) Mirror 30: 0x872e2291 (stale) Mirror 31: 0x36e1e3d1 (stale) CRC-32 checksum value for chunk [0, 0x36ebc): Mirror 1: 0x894a1741 Mirror 32: 0x894a1741 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ebc: skipped 000785 1790896700.620301 punch from 0x0356c6 to 0x036bdf (0x151a bytes) 000787 1790896700.653002 trunc from 0x017b2c to 0x0258cb (0xdda0 bytes) 000788 1790896700.662824 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x36ebc): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 26: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ebc): Mirror 21: 0x2120af9a (stale) Mirror 22: 0x2120af9a (stale) Mirror 26: 0x2120af9a (stale) Mirror 30: 0x872e2291 (stale) Mirror 31: 0x36e1e3d1 (stale) Mirror 32: 0x894a1741 (stale) CRC-32 checksum value for chunk [0, 0x36ebc): Mirror 1: 0x6f661204 Mirror 33: 0x6f661204 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ebc: skipped 000789 1790896701.486391 trunc from 0x017ba5 to 0x02724c (0xf6a8 bytes) 000790 1790896701.524564 trunc from 0x032b12 to 0x036ebb (0x43aa bytes) 000791 1790896701.529495 punch from 0x01add5 to 0x025fe3 (0xb20f bytes) 000792 1790896701.533806 mapwrite 0x0195ea thru 0x028d75 (0xf78c bytes) 000793 1790896701.579080 MIRROR_RESYNC 8 mirrors 000794 1790896704.340522 MIRROR_RESYNC 8 mirrors 000795 1790896704.824629 punch from 0x01c254 to 0x01d39b (0x1148 bytes) 000796 1790896704.868720 mapread 0x007fd2 thru 0x010c5e (0x8c8d bytes) 000797 1790896704.912458 punch from 0x004eb8 to 0x00d16b (0x82b4 bytes) 000798 1790896704.919514 MIRROR_RESYNC 8 mirrors 000799 1790896706.926649 read 0x01876d thru 0x025d4f (0xd5e3 bytes) 000800 1790896706.934611 punch from 0x03369e to 0x036ebb (0x381e bytes) 000801 1790896706.968264 write 0x005891 thru 0x009b32 (0x42a2 bytes) 000803 1790896706.975880 trunc from 0x036ebc to 0x037de5 (0xf2a bytes) 000804 1790896706.993978 write 0x03e01e thru 0x03ffff (0x1fe2 bytes) 000805 1790896706.998487 trunc from 0x040000 to 0x02d027 (0x12fd8 bytes) 000806 1790896707.019757 write 0x025c6a thru 0x029790 (0x3b27 bytes) 000807 1790896707.028847 trunc from 0x01f974 to 0x024bc4 (0x5251 bytes) 000808 1790896707.037261 punch from 0x01d124 to 0x0200c4 (0x2fa1 bytes) 000809 1790896707.045636 fallocate 0x031ccb thru 0x039cd8 (0x800e bytes) 000810 1790896707.050711 read 0x00b710 thru 0x0175c7 (0xbeb8 bytes) 000811 1790896707.074499 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d028): Mirror 21: 0xd9967c56 (stale) Mirror 22: 0xd9967c56 (stale) Mirror 26: 0xd9967c56 (stale) Mirror 30: 0xd9967c56 (stale) Mirror 31: 0xd9967c56 (stale) Mirror 33: 0xd9967c56 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d028): Mirror 21: 0xd9967c56 (stale) Mirror 22: 0xd9967c56 (stale) Mirror 26: 0xd9967c56 (stale) Mirror 30: 0xd9967c56 (stale) Mirror 31: 0xd9967c56 (stale) Mirror 33: 0xd9967c56 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d028: skipped 000813 1790896707.457586 mapread 0x023a5e thru 0x025401 (0x19a4 bytes) 000816 1790896707.477181 trunc from 0x008828 to 0x018251 (0xfa2a bytes) 000817 1790896707.482624 read 0x015183 thru 0x01ac0c (0x5a8a bytes) 000818 1790896707.491245 punch from 0x0280ce to 0x02d027 (0x4f5a bytes) 000819 1790896707.496460 mapread 0x01cae1 thru 0x0214f5 (0x4a15 bytes) 000820 1790896707.501573 fallocate 0x034203 thru 0x03ffff (0xbdfd bytes) 000821 1790896707.505650 write 0x03f571 thru 0x03ffff (0xa8f bytes) 000822 1790896707.513574 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x54bd70d0 (stale) Mirror 22: 0x54bd70d0 (stale) Mirror 30: 0x54bd70d0 (stale) Mirror 31: 0x54bd70d0 (stale) Mirror 33: 0x54bd70d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x54bd70d0 (stale) Mirror 22: 0x54bd70d0 (stale) Mirror 30: 0x54bd70d0 (stale) Mirror 31: 0x54bd70d0 (stale) Mirror 33: 0x54bd70d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000823 1790896707.946862 mapread 0x024aab thru 0x02721e (0x2774 bytes) 000824 1790896707.969272 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x54bd70d0 (stale) Mirror 22: 0x54bd70d0 (stale) Mirror 30: 0x54bd70d0 (stale) Mirror 31: 0x54bd70d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x54bd70d0 (stale) Mirror 22: 0x54bd70d0 (stale) Mirror 30: 0x54bd70d0 (stale) Mirror 31: 0x54bd70d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000825 1790896708.369911 mapwrite 0x027b83 thru 0x02a16b (0x25e9 bytes) 000826 1790896708.408684 punch from 0x00f066 to 0x016b14 (0x7aaf bytes) 000827 1790896708.415752 mapwrite 0x000dfe thru 0x00ac41 (0x9e44 bytes) 000829 1790896708.451969 mapread 0x0027bd thru 0x010545 (0xdd89 bytes) 000830 1790896708.466001 write 0x01bc86 thru 0x020f7a (0x52f5 bytes) 000831 1790896708.501865 trunc from 0x02cb88 to 0x03bf54 (0xf3cd bytes) 000833 1790896708.517434 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x54bd70d0 (stale) Mirror 22: 0x54bd70d0 (stale) Mirror 30: 0x54bd70d0 (stale) Mirror 31: 0x54bd70d0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc41caca2 Mirror 32: 0xc41caca2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000834 1790896709.570674 mapread 0x0164db thru 0x022105 (0xbc2b bytes) 000835 1790896709.600998 punch from 0x01d876 to 0x023349 (0x5ad4 bytes) 000836 1790896709.639574 trunc from 0x013f5b to 0x022498 (0xe53e bytes) 000837 1790896709.644403 mapread 0x00f0e2 thru 0x019bf5 (0xab14 bytes) 000838 1790896709.659538 mapread 0x00496e thru 0x00a48b (0x5b1e bytes) 000839 1790896709.672703 punch from 0x02be93 to 0x0376d2 (0xb840 bytes) 000842 1790896709.678180 read 0x0024a5 thru 0x01182f (0xf38b bytes) 000843 1790896709.678867 write 0x03a8f3 thru 0x03ffff (0x570d bytes) 000844 1790896709.687621 read 0x02f7e0 thru 0x03525b (0x5a7c bytes) 000845 1790896709.696619 mapwrite 0x01e762 thru 0x02d55f (0xedfe bytes) 000847 1790896709.731366 trunc from 0x040000 to 0x0005ce (0x3fa31 bytes) 000848 1790896709.756859 trunc from 0x0005cf to 0x02c8c3 (0x2c2f5 bytes) 000849 1790896709.773959 trunc from 0x00577c to 0x007146 (0x19cb bytes) 000850 1790896709.777686 punch from 0x02b25a to 0x02c7e8 (0x158f bytes) 000851 1790896709.783335 punch from 0x01349e to 0x01390e (0x471 bytes) 000852 1790896709.790669 trunc from 0x02c8c4 to 0x036194 (0x98d1 bytes) 000853 1790896709.804769 trunc from 0x02d7ee to 0x02ecad (0x14c0 bytes) 000854 1790896709.809845 write 0x005285 thru 0x00bf41 (0x6cbd bytes) 000855 1790896709.821817 MIRROR_RESYNC 6 mirrors 000856 1790896710.878449 trunc from 0x00fd24 to 0x01d093 (0xd370 bytes) 000858 1790896710.930687 MIRROR_RESYNC 6 mirrors 000859 1790896711.953078 mapread 0x02afe9 thru 0x03090e (0x5926 bytes) 000860 1790896711.967702 trunc from 0x0220b6 to 0x027ad2 (0x5a1d bytes) 000861 1790896712.010460 write 0x00422c thru 0x00df31 (0x9d06 bytes) 000862 1790896712.019160 write 0x0112e7 thru 0x0150d5 (0x3def bytes) 000863 1790896712.030121 punch from 0x019aa7 to 0x01b6c5 (0x1c1f bytes) 000864 1790896712.034828 read 0x01bd49 thru 0x029bb2 (0xde6a bytes) 000865 1790896712.046027 trunc from 0x014fb5 to 0x01d11f (0x816b bytes) 000866 1790896712.061929 fallocate 0x01fd1a thru 0x027642 (0x7929 bytes) 000867 1790896712.071934 MIRROR_RESYNC 6 mirrors 000868 1790896713.195802 fallocate 0x032a10 thru 0x03c0df (0x96d0 bytes) 000869 1790896713.226919 punch from 0x00e004 to 0x01500e (0x700b bytes) 000870 1790896713.235297 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c0e0): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c0e0): Mirror 21: 0xd30ce31 (stale) Mirror 22: 0xd30ce31 (stale) Mirror 30: 0xd30ce31 (stale) Mirror 31: 0xd30ce31 (stale) Mirror 32: 0xd30ce31 (stale) CRC-32 checksum value for chunk [0, 0x3c0e0): Mirror 1: 0x88e84fc7 Mirror 33: 0x88e84fc7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c0e0: skipped 000871 1790896713.971459 punch from 0x018ee1 to 0x01d6e9 (0x4809 bytes) 000872 1790896714.016678 trunc from 0x002088 to 0x00ee72 (0xcdeb bytes) 000873 1790896714.023274 read 0x02e7ca thru 0x031019 (0x2850 bytes) 000874 1790896714.029172 mapread 0x00849f thru 0x009804 (0x1366 bytes) 000875 1790896714.038651 fallocate 0x02acb9 thru 0x0382a6 (0xd5ee bytes) 000876 1790896714.042646 trunc from 0x03c0e0 to 0x038bdc (0x3503 bytes) 000877 1790896714.066592 trunc from 0x0029b0 to 0x008d3b (0x638c bytes) 000878 1790896714.070670 mapwrite 0x026722 thru 0x02ad1b (0x45fa bytes) 000879 1790896714.100058 read 0x028da8 thru 0x0375f4 (0xe84d bytes) 000880 1790896714.114349 MIRROR_RESYNC 7 mirrors 000881 1790896715.194651 punch from 0x0264c3 to 0x0316ef (0xb22d bytes) 000882 1790896715.229697 read 0x0211f4 thru 0x02213b (0xf48 bytes) 000883 1790896715.235574 write 0x037160 thru 0x03ffff (0x8ea0 bytes) 000884 1790896715.255528 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x9fcfcbac (stale) Mirror 22: 0x9fcfcbac (stale) Mirror 30: 0x9fcfcbac (stale) Mirror 31: 0x9fcfcbac (stale) Mirror 32: 0x9fcfcbac (stale) Mirror 33: 0x9fcfcbac (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x83933e90 Mirror 34: 0x83933e90 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000885 1790896716.074890 read 0x02f3c3 thru 0x0301e0 (0xe1e bytes) 000886 1790896716.091769 write 0x038397 thru 0x03c33b (0x3fa5 bytes) 000887 1790896716.159031 mapwrite 0x03c237 thru 0x03c834 (0x5fe bytes) 000888 1790896716.177627 write 0x01f545 thru 0x027701 (0x81bd bytes) 000889 1790896716.196545 fallocate 0x03f464 thru 0x03ffff (0xb9c bytes) 000890 1790896716.203891 mapread 0x03e842 thru 0x03ffff (0x17be bytes) 000891 1790896716.219704 mapwrite 0x00a074 thru 0x01486b (0xa7f8 bytes) 000892 1790896716.339386 mapwrite 0x02a687 thru 0x03868b (0xe005 bytes) 000894 1790896716.385255 mapread 0x027b09 thru 0x02a354 (0x284c bytes) 000895 1790896716.387378 read 0x018c59 thru 0x025e53 (0xd1fb bytes) 000896 1790896716.394155 punch from 0x01d957 to 0x023ad3 (0x617d bytes) 000897 1790896716.405745 mapwrite 0x02aab0 thru 0x039e41 (0xf392 bytes) 000898 1790896716.423669 punch from 0x02732a to 0x02c340 (0x5017 bytes) 000900 1790896716.435397 mapread 0x03d8a7 thru 0x03ffff (0x2759 bytes) 000901 1790896716.444692 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x9fcfcbac (stale) Mirror 22: 0x9fcfcbac (stale) Mirror 30: 0x9fcfcbac (stale) Mirror 31: 0x9fcfcbac (stale) Mirror 32: 0x9fcfcbac (stale) Mirror 33: 0x9fcfcbac (stale) Mirror 34: 0x83933e90 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf60e87d9 Mirror 35: 0xf60e87d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000902 1790896717.259863 trunc from 0x040000 to 0x035b58 (0xa4a7 bytes) 000903 1790896717.341964 mapwrite 0x02112b thru 0x030f38 (0xfe0e bytes) 000904 1790896717.385159 MIRROR_RESYNC 9 mirrors 000905 1790896719.091467 mapread 0x00b458 thru 0x019ef6 (0xea9f bytes) 000906 1790896719.123923 fallocate 0x00256a thru 0x006483 (0x3f1a bytes) 000907 1790896719.179625 mapwrite 0x03494e thru 0x03ffff (0xb6b2 bytes) 000908 1790896719.243527 mapread 0x01d297 thru 0x021ca0 (0x4a0a bytes) 000909 1790896719.255077 punch from 0x00afdb to 0x00da3e (0x2a64 bytes) 000910 1790896719.263243 mapwrite 0x02e2b0 thru 0x0315de (0x332f bytes) 000911 1790896719.295287 trunc from 0x040000 to 0x016287 (0x29d78 bytes) 000912 1790896719.313175 trunc from 0x016288 to 0x0253db (0xf154 bytes) 000913 1790896719.332930 read 0x010f29 thru 0x020ce5 (0xfdbd bytes) 000914 1790896719.341628 mapread 0x02153d thru 0x0253db (0x3e9f bytes) 000915 1790896719.350308 write 0x0239f5 thru 0x031820 (0xde2c bytes) 000916 1790896719.353322 trunc from 0x01e04b to 0x026d57 (0x8d0d bytes) 000918 1790896719.366403 write 0x00fdbe thru 0x0157df (0x5a22 bytes) 000919 1790896719.371173 mapread 0x02ddff thru 0x031643 (0x3845 bytes) 000920 1790896719.372277 fallocate 0x008820 thru 0x00e3f2 (0x5bd3 bytes) 000921 1790896719.380326 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x31821): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31821): Mirror 21: 0xc287b87a (stale) Mirror 22: 0xc287b87a (stale) Mirror 30: 0xc287b87a (stale) Mirror 31: 0xc287b87a (stale) Mirror 32: 0xc287b87a (stale) Mirror 33: 0xc287b87a (stale) Mirror 34: 0xc287b87a (stale) Mirror 35: 0xc287b87a (stale) CRC-32 checksum value for chunk [0, 0x31821): Mirror 1: 0x72794068 Mirror 36: 0x72794068 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31821: skipped 000922 1790896720.270846 trunc from 0x00f4c2 to 0x011939 (0x2478 bytes) 000923 1790896720.312614 mapread 0x00d431 thru 0x01b685 (0xe255 bytes) 000924 1790896720.365811 trunc from 0x004949 to 0x00caf7 (0x81af bytes) 000925 1790896720.377927 trunc from 0x0049fc to 0x00a8f2 (0x5ef7 bytes) 000926 1790896720.390250 fallocate 0x03acef thru 0x03ffff (0x5311 bytes) 000927 1790896720.394617 punch from 0x01fc2c to 0x020215 (0x5ea bytes) 000928 1790896720.403392 fallocate 0x035cb1 thru 0x03ffff (0xa34f bytes) 000929 1790896720.416795 punch from 0x017044 to 0x0233b1 (0xc36e bytes) 000930 1790896720.427131 mapwrite 0x01c13c thru 0x02513d (0x9002 bytes) 000931 1790896720.480156 trunc from 0x02824b to 0x0346e8 (0xc49e bytes) 000932 1790896720.485068 mapread 0x016980 thru 0x017c17 (0x1298 bytes) 000933 1790896720.486131 mapread 0x021735 thru 0x02211a (0x9e6 bytes) 000935 1790896720.488270 punch from 0x0257f6 to 0x02933c (0x3b47 bytes) 000936 1790896720.496279 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x7191e151 (stale) Mirror 22: 0x7191e151 (stale) Mirror 30: 0x7191e151 (stale) Mirror 31: 0x7191e151 (stale) Mirror 32: 0x7191e151 (stale) Mirror 34: 0x7191e151 (stale) Mirror 35: 0x7191e151 (stale) Mirror 36: 0x27388e8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x7191e151 (stale) Mirror 22: 0x7191e151 (stale) Mirror 30: 0x7191e151 (stale) Mirror 31: 0x7191e151 (stale) Mirror 32: 0x7191e151 (stale) Mirror 34: 0x7191e151 (stale) Mirror 35: 0x7191e151 (stale) Mirror 36: 0x27388e8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000937 1790896720.999655 punch from 0x001d52 to 0x0049fd (0x2cac bytes) 000938 1790896721.003811 mapwrite 0x0183ac thru 0x01fe29 (0x7a7e bytes) 000939 1790896721.040085 write 0x02aab0 thru 0x036cb8 (0xc209 bytes) 000940 1790896721.059390 MIRROR_RESYNC 9 mirrors 000941 1790896723.812972 punch from 0x012937 to 0x02221e (0xf8e8 bytes) 000942 1790896723.850003 trunc from 0x0370f5 to 0x0392eb (0x21f7 bytes) 000943 1790896723.856529 punch from 0x000196 to 0x007a34 (0x789f bytes) 000944 1790896723.861964 write 0x019e6d thru 0x01dc45 (0x3dd9 bytes) 000946 1790896723.877939 fallocate 0x02412b thru 0x02b48a (0x7360 bytes) 000947 1790896723.884539 write 0x0367c2 thru 0x03c284 (0x5ac3 bytes) 000948 1790896723.901350 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x18643d1a (stale) Mirror 22: 0x18643d1a (stale) Mirror 30: 0x18643d1a (stale) Mirror 31: 0x18643d1a (stale) Mirror 32: 0x18643d1a (stale) Mirror 34: 0x18643d1a (stale) Mirror 35: 0x18643d1a (stale) Mirror 36: 0x18643d1a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x87d6b667 Mirror 37: 0x87d6b667 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000949 1790896725.075966 write 0x01d996 thru 0x02d0f9 (0xf764 bytes) 000950 1790896725.136122 read 0x03db80 thru 0x03ffff (0x2480 bytes) 000951 1790896725.142698 mapwrite 0x01364d thru 0x01a1ac (0x6b60 bytes) 000952 1790896725.186436 read 0x02a766 thru 0x02f773 (0x500e bytes) 000953 1790896725.200741 read 0x003c28 thru 0x004812 (0xbeb bytes) 000954 1790896725.202475 write 0x01fa84 thru 0x021d44 (0x22c1 bytes) 000955 1790896725.204534 write 0x02a448 thru 0x02d77c (0x3335 bytes) 000956 1790896725.206759 write 0x00f9cb thru 0x014efa (0x5530 bytes) 000957 1790896725.211646 mapread 0x022123 thru 0x02f94f (0xd82d bytes) 000958 1790896725.213730 read 0x01eeb7 thru 0x0238a1 (0x49eb bytes) 000959 1790896725.215456 write 0x014a22 thru 0x016557 (0x1b36 bytes) 000960 1790896725.217431 trunc from 0x021064 to 0x02ed6b (0xdd08 bytes) 000961 1790896725.242616 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x18643d1a (stale) Mirror 22: 0x18643d1a (stale) Mirror 30: 0x18643d1a (stale) Mirror 31: 0x18643d1a (stale) Mirror 32: 0x18643d1a (stale) Mirror 34: 0x18643d1a (stale) Mirror 35: 0x18643d1a (stale) Mirror 36: 0x18643d1a (stale) Mirror 37: 0x87d6b667 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa9a00835 Mirror 38: 0xa9a00835 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000962 1790896726.124395 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x18643d1a (stale) Mirror 30: 0x18643d1a (stale) Mirror 31: 0x18643d1a (stale) Mirror 32: 0x18643d1a (stale) Mirror 34: 0x18643d1a (stale) Mirror 35: 0x18643d1a (stale) Mirror 36: 0x18643d1a (stale) Mirror 37: 0x87d6b667 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x18643d1a (stale) Mirror 30: 0x18643d1a (stale) Mirror 31: 0x18643d1a (stale) Mirror 32: 0x18643d1a (stale) Mirror 34: 0x18643d1a (stale) Mirror 35: 0x18643d1a (stale) Mirror 36: 0x18643d1a (stale) Mirror 37: 0x87d6b667 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa9a00835 Mirror 38: 0xa9a00835 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000964 1790896727.119078 write 0x022b5a thru 0x02dbc6 (0xb06d bytes) 000965 1790896727.199546 trunc from 0x040000 to 0x033476 (0xcb89 bytes) 000966 1790896727.274124 read 0x012066 thru 0x015ec1 (0x3e5c bytes) 000967 1790896727.287983 read 0x01a0e2 thru 0x021c01 (0x7b20 bytes) 000968 1790896727.315742 trunc from 0x033477 to 0x02ad72 (0x8704 bytes) 000969 1790896727.334700 punch from 0x01c637 to 0x023af1 (0x74bb bytes) 000971 1790896727.351528 trunc from 0x00b53d to 0x00be3c (0x900 bytes) 000972 1790896727.356058 fallocate 0x008f3c thru 0x017ed1 (0xef96 bytes) 000974 1790896727.361871 mapread 0x0271d0 thru 0x02ad72 (0x3ba3 bytes) 000975 1790896727.367142 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2ad73): Mirror 21: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ad73): Mirror 21: 0x7532959e (stale) Mirror 30: 0x7532959e (stale) Mirror 31: 0x7532959e (stale) Mirror 32: 0x7532959e (stale) Mirror 34: 0x7532959e (stale) Mirror 35: 0x7532959e (stale) Mirror 36: 0x7532959e (stale) Mirror 37: 0x1ab8815b (stale) Mirror 38: 0xe9fae512 (stale) CRC-32 checksum value for chunk [0, 0x2ad73): Mirror 1: 0x7fdb47da Mirror 39: 0x7fdb47da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ad73: skipped 000976 1790896728.457534 MIRROR_RESYNC 11 mirrors 000977 1790896730.468702 trunc from 0x02986d to 0x02ad72 (0x1506 bytes) 000978 1790896730.522176 trunc from 0x022909 to 0x02ad72 (0x846a bytes) 000979 1790896730.526554 mapwrite 0x01c6e4 thru 0x01f9b5 (0x32d2 bytes) 000980 1790896730.572241 fallocate 0x00f59c thru 0x01816c (0x8bd1 bytes) 000981 1790896730.579147 trunc from 0x02574c to 0x02ad72 (0x5627 bytes) 000982 1790896730.586270 punch from 0x00bf63 to 0x019f93 (0xe031 bytes) 000983 1790896730.613161 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ad73): Mirror 30: 0x7fdb47da (stale) Mirror 31: 0x7fdb47da (stale) Mirror 32: 0x7fdb47da (stale) Mirror 34: 0x7fdb47da (stale) Mirror 35: 0x7fdb47da (stale) Mirror 36: 0x7fdb47da (stale) Mirror 37: 0x7fdb47da (stale) Mirror 38: 0x7fdb47da (stale) Mirror 39: 0x7fdb47da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ad73): Mirror 30: 0x7fdb47da (stale) Mirror 31: 0x7fdb47da (stale) Mirror 32: 0x7fdb47da (stale) Mirror 34: 0x7fdb47da (stale) Mirror 35: 0x7fdb47da (stale) Mirror 36: 0x7fdb47da (stale) Mirror 37: 0x7fdb47da (stale) Mirror 38: 0x7fdb47da (stale) Mirror 39: 0x7fdb47da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ad73: skipped 000984 1790896731.148020 trunc from 0x029d99 to 0x02ad72 (0xfda bytes) 000985 1790896731.153014 punch from 0x009ad6 to 0x01758f (0xdaba bytes) 000986 1790896731.160962 write 0x005433 thru 0x0087a2 (0x3370 bytes) 000987 1790896731.175085 punch from 0x013ff7 to 0x01e958 (0xa962 bytes) 000989 1790896731.185943 punch from 0x014a0f to 0x01ebfe (0xa1f0 bytes) 000990 1790896731.192906 mapread 0x029c62 thru 0x02ad72 (0x1111 bytes) 000991 1790896731.215993 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ad73): Mirror 30: 0x7fdb47da (stale) Mirror 32: 0x7fdb47da (stale) Mirror 34: 0x7fdb47da (stale) Mirror 35: 0x7fdb47da (stale) Mirror 36: 0x7fdb47da (stale) Mirror 37: 0x7fdb47da (stale) Mirror 38: 0x7fdb47da (stale) Mirror 39: 0x7fdb47da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ad73): Mirror 30: 0x7fdb47da (stale) Mirror 32: 0x7fdb47da (stale) Mirror 34: 0x7fdb47da (stale) Mirror 35: 0x7fdb47da (stale) Mirror 36: 0x7fdb47da (stale) Mirror 37: 0x7fdb47da (stale) Mirror 38: 0x7fdb47da (stale) Mirror 39: 0x7fdb47da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ad73: skipped 000992 1790896731.728404 trunc from 0x004453 to 0x00e341 (0x9eef bytes) 000993 1790896731.737828 punch from 0x013272 to 0x01e454 (0xb1e3 bytes) 000994 1790896731.743103 mapwrite 0x03ce23 thru 0x03ffff (0x31dd bytes) 000995 1790896731.840781 trunc from 0x00dc3e to 0x018297 (0xa65a bytes) 000996 1790896731.847309 read 0x03d864 thru 0x03ffff (0x279c bytes) 000997 1790896731.848943 trunc from 0x02db57 to 0x032739 (0x4be3 bytes) 000998 1790896731.853301 trunc from 0x040000 to 0x02dd84 (0x1227b bytes) 000999 1790896731.872924 fallocate 0x025bb0 thru 0x03026f (0xa6c0 bytes) 001000 1790896731.878937 trunc from 0x008601 to 0x012830 (0xa230 bytes) All operations completed A-OK! PASS 70a (119s) == sanity-flr test 71: check mirror extend parent fid ==== 19:18:56 (1790896736) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x21b:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 72: check append on FLR file ========== 19:19:04 (1790896744) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.246973 s, 8.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918623 s, 11.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x139:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 19:19:11 (1790896751) Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6918720 .. Extending file size to 3178336 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8169280 .. Extending file size to 7206016 .. Extending file size to 8193248 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8661504 .. Extending file size to 8857248 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8926592 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8915296 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62255 62256 62257 62258 62260 done 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) 192.168.203.130@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:20:35 (1790896835) Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 Extending file size to 1918912 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7270368 .. Extending file size to 8474816 .. Extending file size to 6015616 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7439712 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8169504 .. Extending file size to 9101888 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 63822 63824 63827 63828 63834 failed 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) 192.168.203.130@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (74s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:21:49 (1790896909) Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2284 1269020 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2320 1268984 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18812 3588208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24508 3582512 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43320 7170720 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2284 1269020 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2320 1268984 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18812 3588208 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24508 3582512 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43320 7170720 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0688798 s, 30.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141605 s, 14.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x4690047b (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:22:06 (1790896926) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.153277 s, 13.7 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 19:22:12 (1790896932) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xb3:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:22:19 (1790896939) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124172 s, 16.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:22:27 (1790896947) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0563921 s, 37.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:22:34 (1790896954) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0992749 s, 21.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x153:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x153:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x154:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:23:04 (1790896984) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg330-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610262 s, 17.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x282:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x283:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x282:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x282:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x283:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x282:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x283:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:23:52 (1790897032) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg330-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x284:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x285:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0757016 s, 27.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x284:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x285:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x284:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x284:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x285:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x284:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:24:38 (1790897078) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg330-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg330-server: Pool lustre.test_204f created Adding targets to pool oleg330-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg330-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117715 s, 17.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x286:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x285:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x286:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x285:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg330-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg330-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg330-server: Pool lustre.test_204f destroyed PASS 204f (69s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:25:47 (1790897147) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:25:55 (1790897155) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:26:01 (1790897161) oleg330-server: Pool lustre.test_206 created Adding targets to pool oleg330-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg330-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg330-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg330-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg330-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:26:28 (1790897188) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769652 s, 13.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667157 s, 15.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xdd937fd0 Mirror 3: 0xdd937fd0 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:26:38 (1790897198) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:26:42 (1790897202) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:26:45 (1790897205) Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.062022 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x920000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xca0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xca0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xca0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x810000 Read this (4 bytes): BBBB 192.168.203.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg330-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:27:19 (1790897239) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0069562 s, 1.2 MB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:27:25 (1790897245) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768251 s, 13.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd2 lmm_fid: [0x200000405:0xd2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd2 lmm_fid: [0x200000405:0xd2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611346 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-01 19:27:27.000000000 -0400 Modify: 2026-10-01 19:27:27.000000000 -0400 Change: 2026-10-01 19:27:28.000000000 -0400 Birth: 2026-10-01 19:27:26.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:27:33 (1790897253) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352212 s, 29.8 MB/s oleg330-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg330-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg330-server: O/280000401/d29/15d: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:27:56 (1790897276) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.257784 s, 159 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0273403 s, 150 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 1 19:27 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xd7:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 1 19:27 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:28:03 (1790897283) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:28:10 (1790897290) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1677 sec =========== 19:28:19 (1790897299) === sanity-flr: start cleanup 19:28:21 (1790897301) === === sanity-flr: finish cleanup 19:28:29 (1790897309) ===