-----============= acceptance-small: sanity-flr ============----- Thu Jul 16 03:07:38 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg138-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 03:07:57 (1784185677) === oleg138-client.virtnet: executing check_config_client /mnt/lustre oleg138-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg138-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99cbc54ea800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99cbc54ea800.idle_timeout=debug disable quota as required oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 03:08:36 (1784185716) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8210] == sanity-flr test 0a: lfs mirror create with -N option == 03:08:39 (1784185719) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 03:08:53 (1784185733) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 03:08:57 (1784185737) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 03:09:00 (1784185740) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 03:09:24 (1784185764) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 03:09:27 (1784185767) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 03:09:30 (1784185770) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff99cbc54ea800.stats=clear osc.lustre-OST0001-osc-ffff99cbc54ea800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 03:09:37 (1784185777) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 03:09:49 (1784185789) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 03:09:58 (1784185798) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 03:10:01 (1784185801) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 03:10:09 (1784185809) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 03:10:19 (1784185819) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 03:10:27 (1784185827) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 03:10:35 (1784185835) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 03:10:42 (1784185842) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 03:10:51 (1784185851) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 03:10:57 (1784185857) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.5937 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987243 s, 10.6 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 03:11:07 (1784185867) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0461662 s, 45.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 03:11:15 (1784185875) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 03:11:25 (1784185885) Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99cbc54ea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99cbc54ea800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99cbc54ea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99cbc54ea800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg138-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (75s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 03:12:40 (1784185960) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.116541 s, 27.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99cbc54ea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99cbc54ea800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 03:13:22 (1784186002) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99cbc54ea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99cbc54ea800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99cbc54ea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99cbc54ea800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg138-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:14:49 (1784186089) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.52486 s, 20.8 MB/s -rw-r--r-- 1 root root 52428800 Jul 16 03:14 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.143s user 0m0.002s sys 0m0.180s Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99cbc54ea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99cbc54ea800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.995s user 0m0.000s sys 0m0.209s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99cbc54ea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99cbc54ea800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.588s user 0m0.002s sys 0m0.188s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg138-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99cbc54ea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99cbc54ea800.ost_server_uuid in FULL state after 0 sec PASS 33b (93s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 03:16:23 (1784186183) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 03:16:27 (1784186187) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 03:16:30 (1784186190) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 03:16:34 (1784186194) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 03:16:41 (1784186201) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.384528 s, 32.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 03:16:53 (1784186213) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277541 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290874 s, 36.0 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 03:17:00 (1784186220) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 03:17:09 (1784186229) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00111624 s, 917 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000935748 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00138384 s, 740 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00286186 s, 358 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 03:17:20 (1784186240) create test files with size 15747229 .. 79aa910ef3acf4182f16d0413c1e3e83 - 81484fa7971a366d246ce664a44a5e93 - b5757b0de1f84700f0e1d6039ba1db21 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 03:17:44 (1784186264) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 03:17:55 (1784186275) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 03:18:02 (1784186282) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632096 s, 16.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784114 s, 13.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 03:18:14 (1784186294) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0574725 s, 36.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.332706 s, 12.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131053 s, 16.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 03:18:29 (1784186309) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 03:18:33 (1784186313) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 03:18:36 (1784186316) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411827 s, 25.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 03:18:45 (1784186325) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.363957 s, 8.6 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 03:18:57 (1784186337) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 03:19:07 (1784186347) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.313568 s, 33.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 03:19:19 (1784186359) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff99cbc96ca800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.382488 s, 27.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff99cbd0f22800.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 03:19:38 (1784186378) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (45s) == sanity-flr test 46: Verify setstripe --copy option ==== 03:20:23 (1784186423) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 03:21:03 (1784186463) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 03:21:07 (1784186467) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.230085 s, 13.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.199413 s, 15.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x6c6e63a1 (nosync,stale) Mirror 2: 0x6c6e63a1 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 4f8f76fd89cfcf9095cc7e7d5b37bb90 - mirror 1 checksum: 62132d2cb5a2a1e4dcc40b2cce99e2b6 - mirror 2 checksum: 4f8f76fd89cfcf9095cc7e7d5b37bb90 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-16T03:21:09-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-16T03:21:09-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 4f8f76fd89cfcf9095cc7e7d5b37bb90 - mirror 1 checksum: 62132d2cb5a2a1e4dcc40b2cce99e2b6 - mirror 2 checksum: 62132d2cb5a2a1e4dcc40b2cce99e2b6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-16T03:21:09-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-16T03:21:09-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdc:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 03:21:16 (1784186476) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281253 s, 37.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 03:21:22 (1784186482) Failing mds1 on oleg138-server Stopping /mnt/lustre-mds1 (opts:) on oleg138-server 03:21:37 (1784186497) shut down facet: mds1 facet_host: oleg138-server facet_failover_host: oleg138-server Failover mds1 to oleg138-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 03:22:06 (1784186526) targets are mounted 03:22:06 (1784186526) facet_failover done oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258969 s, 40.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xde:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 03:22:25 (1784186545) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0824914 s, 248 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0222002 s, 923 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 03:22:36 (1784186556) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.052123 s, 393 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00975898 s, 2.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 03:22:47 (1784186567) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0133653 s, 1.2 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 03:22:55 (1784186575) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0399094 s, 513 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:23:03 (1784186583) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00748991 s, 547 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0196879 s, 208 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:23:10 (1784186590) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:23:30 (1784186610) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:23:59 (1784186639) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:24:23 (1784186663) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:24:49 (1784186689) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 03:24:57 (1784186697) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57533 mirror split pid 57534 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 57533 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 57534 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 03:26:06 (1784186766) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 6769 skipping zero length punch hole skipping zero length punch hole skipping zero size read 000006 1784186769.768218 write 0x008df4 thru 0x012e65 (0xa072 bytes) 000007 1784186769.777078 mapread 0x0094ee thru 0x00c71a (0x322d bytes) 000008 1784186769.788798 write 0x001af9 thru 0x008004 (0x650c bytes) 000009 1784186769.810554 punch from 0x004ce7 to 0x00aa10 (0x5d2a bytes) zero_range to largest ever: 0x12e66 000010 1784186769.834153 trunc from 0x00736a to 0x012e65 (0xbafc bytes) 000011 1784186769.844200 punch from 0x009a67 to 0x012e65 (0x93ff bytes) 000012 1784186769.854557 read 0x007124 thru 0x00e656 (0x7533 bytes) 000013 1784186769.898523 MIRROR_EXTEND to 2 mirrors 000015 1784186771.037473 read 0x012145 thru 0x012e65 (0xd21 bytes) 000016 1784186771.055998 trunc from 0x0036f1 to 0x00908a (0x599a bytes) 000017 1784186771.119132 trunc from 0x012e66 to 0x00b473 (0x79f2 bytes) 000018 1784186771.167293 trunc from 0x0020b0 to 0x00b473 (0x93c4 bytes) 000019 1784186771.173167 punch from 0x00b2df to 0x00b473 (0x195 bytes) fallocating to largest ever: 0x1fcd7 000021 1784186771.185295 fallocate 0x0118cd thru 0x01fcd6 (0xe40a bytes) fallocating to largest ever: 0x40000 000022 1784186771.190888 fallocate 0x03973d thru 0x03ffff (0x68c3 bytes) 000024 1784186771.210552 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000025 1784186771.409398 read 0x035a07 thru 0x03ffff (0xa5f9 bytes) 000026 1784186771.420282 mapwrite 0x017962 thru 0x02662e (0xeccd bytes) 000027 1784186771.488663 trunc from 0x040000 to 0x03ba68 (0x4597 bytes) 000028 1784186771.544088 write 0x012666 thru 0x01f670 (0xd00b bytes) 000029 1784186771.549213 fallocate 0x01a750 thru 0x020b9b (0x644c bytes) 000030 1784186771.561050 write 0x03de6e thru 0x03ffff (0x2192 bytes) 000031 1784186771.562196 fallocate 0x027e1c thru 0x032b93 (0xad78 bytes) 000032 1784186771.592206 punch from 0x020d09 to 0x02db2e (0xce26 bytes) 000034 1784186771.615410 mapread 0x038540 thru 0x039458 (0xf19 bytes) 000035 1784186771.620608 read 0x02c0db thru 0x0317f6 (0x571c bytes) 000036 1784186771.658946 punch from 0x035729 to 0x03ffff (0xa8d7 bytes) 000038 1784186771.703370 MIRROR_EXTEND to 2 mirrors 000039 1784186772.879692 fallocate 0x0138dd thru 0x017869 (0x3f8d bytes) 000040 1784186772.974117 read 0x0159fb thru 0x01616f (0x775 bytes) 000041 1784186773.027613 mapwrite 0x011e6b thru 0x019694 (0x782a bytes) 000042 1784186773.128813 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000044 1784186773.271178 mapwrite 0x038024 thru 0x03ffff (0x7fdc bytes) 000045 1784186773.299287 trunc from 0x040000 to 0x034342 (0xbcbd bytes) 000046 1784186773.314195 read 0x02282f thru 0x0242e9 (0x1abb bytes) 000047 1784186773.319234 write 0x03b300 thru 0x03c637 (0x1338 bytes) 000048 1784186773.320998 fallocate 0x03c406 thru 0x03ffff (0x3bfa bytes) 000049 1784186773.330742 trunc from 0x040000 to 0x03385e (0xc7a1 bytes) 000050 1784186773.344791 mapread 0x006579 thru 0x015726 (0xf1ae bytes) 000051 1784186773.364799 trunc from 0x03385f to 0x01320d (0x20651 bytes) 000052 1784186773.378779 write 0x0399b5 thru 0x03fb2b (0x6177 bytes) 000053 1784186773.379997 fallocate 0x01b926 thru 0x021175 (0x5850 bytes) 000055 1784186773.385776 write 0x00c859 thru 0x01af1f (0xe6c7 bytes) 000057 1784186773.392764 punch from 0x037427 to 0x0387fc (0x13d6 bytes) 000058 1784186773.398162 trunc from 0x01a126 to 0x01fec4 (0x5d9f bytes) 000059 1784186773.418799 fallocate 0x0139c8 thru 0x013c08 (0x241 bytes) 000060 1784186773.423377 read 0x0046ca thru 0x00c847 (0x817e bytes) 000062 1784186773.424994 write 0x02c0fa thru 0x02f756 (0x365d bytes) 000063 1784186773.435682 write 0x0162cd thru 0x017996 (0x16ca bytes) 000064 1784186773.436699 fallocate 0x03851e thru 0x039659 (0x113c bytes) 000065 1784186773.441463 mapread 0x00bfe9 thru 0x018757 (0xc76f bytes) 000066 1784186773.442816 mapwrite 0x037f9c thru 0x03ffff (0x8064 bytes) 000067 1784186773.484266 mapwrite 0x00db6d thru 0x00dc54 (0xe8 bytes) 000068 1784186773.506280 trunc from 0x040000 to 0x016fc4 (0x2903b bytes) 000069 1784186773.546378 fallocate 0x0396ba thru 0x03ffff (0x6946 bytes) 000071 1784186773.553680 mapwrite 0x0221af thru 0x029084 (0x6ed6 bytes) 000072 1784186773.638428 trunc from 0x040000 to 0x01bbfb (0x24404 bytes) 000073 1784186773.658068 punch from 0x00f7c3 to 0x010d1a (0x1558 bytes) 000074 1784186773.664075 mapwrite 0x02d9bf thru 0x0353c4 (0x7a06 bytes) 000075 1784186773.834319 MIRROR_EXTEND to 2 mirrors 000076 1784186774.950922 trunc from 0x0353c5 to 0x0126a9 (0x22d1b bytes) 000077 1784186775.009802 MIRROR_RESYNC 2 mirrors 000078 1784186775.837027 trunc from 0x006c58 to 0x00ea19 (0x7dc2 bytes) 000079 1784186775.913032 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000080 1784186776.264417 mapread 0x0051ed thru 0x00b79f (0x65b3 bytes) 000081 1784186776.319577 punch from 0x00f64f to 0x0114dc (0x1e8e bytes) 000083 1784186776.333561 mapread 0x00f5c9 thru 0x0126a9 (0x30e1 bytes) 000085 1784186776.356013 MIRROR_EXTEND to 2 mirrors 000086 1784186777.531746 punch from 0x004b44 to 0x007866 (0x2d23 bytes) 000087 1784186777.644887 fallocate 0x014a1b thru 0x01ef76 (0xa55c bytes) 000089 1784186777.662266 write 0x000752 thru 0x003f8d (0x383c bytes) 000090 1784186777.722194 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000091 1784186777.956094 mapwrite 0x03dd0b thru 0x03ffff (0x22f5 bytes) 000092 1784186778.027211 trunc from 0x006bc6 to 0x009e33 (0x326e bytes) 000093 1784186778.033102 mapwrite 0x0315df thru 0x03efd4 (0xd9f6 bytes) 000094 1784186778.070219 mapwrite 0x00b85f thru 0x013efb (0x869d bytes) 000095 1784186778.106967 read 0x00707e thru 0x015c19 (0xeb9c bytes) 000096 1784186778.113936 read 0x025366 thru 0x028876 (0x3511 bytes) 000097 1784186778.125897 trunc from 0x012680 to 0x01f4bf (0xce40 bytes) 000098 1784186778.134598 read 0x024265 thru 0x030964 (0xc700 bytes) 000099 1784186778.138221 fallocate 0x002828 thru 0x00b20c (0x89e5 bytes) 000100 1784186778.150037 fallocate 0x017951 thru 0x0183da (0xa8a bytes) 000101 1784186778.167805 mapread 0x02e6ba thru 0x032be0 (0x4527 bytes) 000102 1784186778.171951 trunc from 0x040000 to 0x033670 (0xc98f bytes) 000103 1784186778.196877 trunc from 0x033671 to 0x00d859 (0x25e17 bytes) 000104 1784186778.229910 punch from 0x006f50 to 0x00732a (0x3db bytes) 000105 1784186778.236069 trunc from 0x00d85a to 0x016bf7 (0x939e bytes) 000107 1784186778.282600 trunc from 0x01000d to 0x016bf7 (0x6beb bytes) 000108 1784186778.290440 mapwrite 0x00932c thru 0x015100 (0xbdd5 bytes) 000109 1784186778.358097 read 0x015f45 thru 0x016bf7 (0xcb3 bytes) 000110 1784186778.359351 write 0x039ed7 thru 0x03ffff (0x6129 bytes) 000111 1784186778.362936 trunc from 0x01cb8c to 0x029156 (0xc5cb bytes) 000112 1784186778.369182 trunc from 0x034fb2 to 0x03ffff (0xb04e bytes) 000113 1784186778.383459 write 0x03682e thru 0x036b69 (0x33c bytes) 000116 1784186778.419953 fallocate 0x012764 thru 0x020b33 (0xe3d0 bytes) 000117 1784186778.432683 punch from 0x00908e to 0x012183 (0x90f6 bytes) 000119 1784186778.450849 trunc from 0x040000 to 0x011ee6 (0x2e119 bytes) 000120 1784186778.484891 fallocate 0x005123 thru 0x0123a7 (0xd285 bytes) 000121 1784186778.499430 mapread 0x00f3e9 thru 0x011ee6 (0x2afe bytes) 000122 1784186778.532924 mapwrite 0x00d2f6 thru 0x013a1d (0x6728 bytes) 000123 1784186778.667842 read 0x002747 thru 0x00745b (0x4d15 bytes) 000124 1784186778.669003 trunc from 0x013a1e to 0x030f7b (0x1d55e bytes) 000125 1784186778.696923 MIRROR_EXTEND to 2 mirrors 000126 1784186779.921225 mapread 0x0036c0 thru 0x00a6ef (0x7030 bytes) 000127 1784186779.946798 mapwrite 0x0028da thru 0x00f118 (0xc83f bytes) 000128 1784186780.027027 read 0x01e056 thru 0x0286bc (0xa667 bytes) 000130 1784186780.051482 write 0x009335 thru 0x00fb20 (0x67ec bytes) 000131 1784186780.067022 mapread 0x027d24 thru 0x030f7b (0x9258 bytes) 000132 1784186780.082224 read 0x01c485 thru 0x0285ad (0xc129 bytes) 000133 1784186780.109491 trunc from 0x004522 to 0x00ebba (0xa699 bytes) 000134 1784186780.141538 punch from 0x017ed6 to 0x01f1f4 (0x731f bytes) 000135 1784186780.173904 mapread 0x02cffc thru 0x030f7b (0x3f80 bytes) 000136 1784186780.176070 fallocate 0x015d35 thru 0x01d73e (0x7a0a bytes) 000137 1784186780.182487 write 0x0133ab thru 0x022cf7 (0xf94d bytes) 000138 1784186780.184374 read 0x00a72c thru 0x01189a (0x716f bytes) 000139 1784186780.200678 MIRROR_RESYNC 2 mirrors 000140 1784186781.291885 read 0x02246e thru 0x022d7f (0x912 bytes) 000141 1784186781.300437 read 0x0242e5 thru 0x02bdad (0x7ac9 bytes) 000142 1784186781.312804 trunc from 0x030f7c to 0x02d436 (0x3b45 bytes) 000143 1784186781.424830 trunc from 0x002f15 to 0x00adcd (0x7eb9 bytes) 000144 1784186781.431014 trunc from 0x02d437 to 0x01fbd8 (0xd85e bytes) 000145 1784186781.454401 trunc from 0x01fbd9 to 0x007ced (0x17eeb bytes) 000146 1784186781.471974 trunc from 0x007cee to 0x03f03d (0x37350 bytes) 000147 1784186781.500506 trunc from 0x02eced to 0x0348b6 (0x5bca bytes) 000148 1784186781.506205 punch from 0x03114d to 0x0382aa (0x715e bytes) 000149 1784186781.517220 trunc from 0x010547 to 0x015f68 (0x5a22 bytes) 000150 1784186781.525454 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000151 1784186781.714031 trunc from 0x025f15 to 0x0273e1 (0x14cd bytes) 000152 1784186781.720087 trunc from 0x03ca05 to 0x03f03d (0x2639 bytes) 000153 1784186781.729857 trunc from 0x03f03e to 0x0153a2 (0x29c9b bytes) 000154 1784186781.747708 fallocate 0x02968b thru 0x02f8b3 (0x6229 bytes) 000155 1784186781.753447 mapwrite 0x006e40 thru 0x0146cd (0xd88e bytes) 000156 1784186781.797416 read 0x0035fb thru 0x00ce7e (0x9884 bytes) 000158 1784186781.802999 write 0x0008d9 thru 0x0070d7 (0x67ff bytes) 000159 1784186781.814022 MIRROR_EXTEND to 2 mirrors 000160 1784186782.275786 trunc from 0x0153a3 to 0x03d476 (0x280d4 bytes) 000161 1784186782.318824 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d477): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d477): Mirror 2: 0xedbdc5dc (stale) CRC-32 checksum value for chunk [0, 0x3d477): Mirror 1: 0xedbdc5dc Mirror 3: 0xedbdc5dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d477: skipped 000162 1784186783.027351 trunc from 0x00d031 to 0x01b3cd (0xe39d bytes) 000163 1784186783.055865 read 0x005446 thru 0x010b99 (0xb754 bytes) 000164 1784186783.063458 trunc from 0x03d477 to 0x009e91 (0x335e5 bytes) 000165 1784186783.082431 MIRROR_RESYNC 3 mirrors 000166 1784186783.339831 punch from 0x002474 to 0x009e91 (0x7a1e bytes) 000168 1784186783.365889 fallocate 0x037b24 thru 0x03ffff (0x84dc bytes) 000169 1784186783.372270 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2501668a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2501668a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000170 1784186783.511851 fallocate 0x03b5ff thru 0x03ffff (0x4a01 bytes) 000171 1784186783.515850 read 0x024e61 thru 0x02ee34 (0x9fd4 bytes) 000173 1784186783.521733 write 0x019dd3 thru 0x0216fd (0x792b bytes) 000174 1784186783.529719 trunc from 0x040000 to 0x0243e4 (0x1bc1b bytes) 000175 1784186783.553722 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000176 1784186783.654456 fallocate 0x03aff3 thru 0x03e383 (0x3391 bytes) 000177 1784186783.659525 trunc from 0x031d2a to 0x03e383 (0xc65a bytes) 000178 1784186783.665968 trunc from 0x03e384 to 0x039852 (0x4b31 bytes) 000179 1784186783.687734 trunc from 0x01a81f to 0x021700 (0x6ee2 bytes) 000180 1784186783.693013 read 0x011444 thru 0x01972c (0x82e9 bytes) 000181 1784186783.700676 mapread 0x032539 thru 0x037c87 (0x574f bytes) 000183 1784186783.712956 mapwrite 0x0162c3 thru 0x021040 (0xad7e bytes) 000186 1784186783.731773 mapwrite 0x0372f5 thru 0x038a3b (0x1747 bytes) 000187 1784186783.739464 mapread 0x02649f thru 0x02670d (0x26f bytes) 000188 1784186783.745576 punch from 0x018028 to 0x018da3 (0xd7c bytes) 000189 1784186783.748490 write 0x0322c4 thru 0x03df10 (0xbc4d bytes) 000190 1784186783.749560 read 0x00bbbb thru 0x01782e (0xbc74 bytes) 000191 1784186783.755420 trunc from 0x036c08 to 0x03df10 (0x7309 bytes) 000193 1784186783.764247 read 0x036cf0 thru 0x03df10 (0x7221 bytes) 000194 1784186783.769612 mapwrite 0x00d89b thru 0x01a4cc (0xcc32 bytes) 000197 1784186783.782347 trunc from 0x006649 to 0x014966 (0xe31e bytes) 000198 1784186783.787870 MIRROR_EXTEND to 2 mirrors 000199 1784186784.601914 mapwrite 0x00865e thru 0x00962f (0xfd2 bytes) 000200 1784186784.707185 fallocate 0x013786 thru 0x0197ed (0x6068 bytes) 000201 1784186784.718443 write 0x016f80 thru 0x024bf2 (0xdc73 bytes) 000202 1784186784.749096 punch from 0x037692 to 0x03a6d2 (0x3041 bytes) 000203 1784186784.757857 write 0x013e3c thru 0x019a45 (0x5c0a bytes) 000204 1784186784.765534 write 0x0357cb thru 0x039746 (0x3f7c bytes) 000205 1784186784.774279 fallocate 0x004d3a thru 0x00eb79 (0x9e40 bytes) 000206 1784186784.778871 trunc from 0x00a758 to 0x014ac3 (0xa36c bytes) 000209 1784186784.797610 write 0x0287c9 thru 0x03862c (0xfe64 bytes) 000210 1784186784.802282 trunc from 0x03df11 to 0x010071 (0x2de9f bytes) 000211 1784186784.831943 trunc from 0x0006a2 to 0x009105 (0x8a64 bytes) 000212 1784186784.839925 fallocate 0x01704f thru 0x0248af (0xd861 bytes) 000213 1784186784.846198 read 0x022dc9 thru 0x023bfd (0xe35 bytes) 000214 1784186784.856571 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000215 1784186784.959117 mapread 0x0142ac thru 0x01d005 (0x8d5a bytes) 000216 1784186784.989909 fallocate 0x0037cb thru 0x0127a0 (0xefd6 bytes) 000217 1784186784.994315 punch from 0x01bf60 to 0x021579 (0x561a bytes) 000218 1784186784.998953 read 0x014324 thru 0x01c6f5 (0x83d2 bytes) 000219 1784186785.003576 fallocate 0x01a3ea thru 0x02714d (0xcd64 bytes) 000220 1784186785.008516 trunc from 0x02714e to 0x037d06 (0x10bb9 bytes) 000221 1784186785.030770 MIRROR_EXTEND to 2 mirrors 000222 1784186785.553253 write 0x00c666 thru 0x018379 (0xbd14 bytes) 000223 1784186785.584232 trunc from 0x011732 to 0x014695 (0x2f64 bytes) 000224 1784186785.595116 write 0x00def3 thru 0x010ed9 (0x2fe7 bytes) 000225 1784186785.596948 punch from 0x02a47c to 0x02f7e4 (0x5369 bytes) 000226 1784186785.602718 mapwrite 0x018950 thru 0x01f2be (0x696f bytes) 000228 1784186785.652905 read 0x034513 thru 0x037d06 (0x37f4 bytes) 000229 1784186785.662761 read 0x02d3af thru 0x037d06 (0xa958 bytes) 000230 1784186785.690119 MIRROR_RESYNC 2 mirrors 000231 1784186786.569610 trunc from 0x037d07 to 0x0328e8 (0x541e bytes) 000232 1784186786.666595 trunc from 0x0328e9 to 0x02673c (0xc1ac bytes) 000233 1784186786.692248 trunc from 0x0214a5 to 0x02673c (0x5298 bytes) 000234 1784186786.701928 read 0x0207f8 thru 0x02673c (0x5f45 bytes) 000235 1784186786.719253 fallocate 0x032453 thru 0x03ffff (0xdbad bytes) 000236 1784186786.726801 read 0x021259 thru 0x02a379 (0x9121 bytes) 000237 1784186786.745903 mapwrite 0x03a6b7 thru 0x03ffff (0x5949 bytes) 000238 1784186786.776903 write 0x036e6c thru 0x038b17 (0x1cac bytes) 000239 1784186786.779184 read 0x02e703 thru 0x033361 (0x4c5f bytes) 000240 1784186786.780502 fallocate 0x01d546 thru 0x02aacc (0xd587 bytes) 000241 1784186786.786938 fallocate 0x039b5b thru 0x03ffff (0x64a5 bytes) 000242 1784186786.797313 read 0x03eebe thru 0x03ffff (0x1142 bytes) 000243 1784186786.798283 fallocate 0x02779f thru 0x0302be (0x8b20 bytes) 000244 1784186786.808394 mapread 0x00100b thru 0x009050 (0x8046 bytes) 000245 1784186786.816568 punch from 0x01dcda to 0x01e651 (0x978 bytes) 000246 1784186786.821831 fallocate 0x035b41 thru 0x03aa60 (0x4f20 bytes) 000247 1784186786.826608 punch from 0x01145b to 0x013bfd (0x27a3 bytes) 000248 1784186786.834451 trunc from 0x000a1c to 0x0102a5 (0xf88a bytes) 000249 1784186786.839302 mapwrite 0x0037e5 thru 0x0105c7 (0xcde3 bytes) 000250 1784186786.898892 trunc from 0x040000 to 0x01ed8b (0x21274 bytes) 000251 1784186786.932759 fallocate 0x00a951 thru 0x0103c7 (0x5a77 bytes) 000252 1784186786.943672 fallocate 0x027345 thru 0x032d9e (0xba5a bytes) 000253 1784186786.951405 fallocate 0x0146f0 thru 0x01c7fa (0x810b bytes) 000254 1784186786.960444 trunc from 0x032d9f to 0x00c3e3 (0x269bb bytes) 000255 1784186786.979890 fallocate 0x0324bc thru 0x03e6a2 (0xc1e7 bytes) 000256 1784186786.993967 mapread 0x014d5c thru 0x01b248 (0x64ed bytes) 000257 1784186787.084437 mapwrite 0x0098d2 thru 0x009ddf (0x50e bytes) 000258 1784186787.102785 read 0x013b30 thru 0x01d5af (0x9a80 bytes) 000259 1784186787.150766 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000260 1784186787.376100 fallocate 0x0046b8 thru 0x00e564 (0x9ead bytes) 000261 1784186787.390494 fallocate 0x02f47b thru 0x03f393 (0xff19 bytes) 000262 1784186787.403266 trunc from 0x03a2c3 to 0x03cfb1 (0x2cef bytes) 000263 1784186787.421762 mapread 0x039214 thru 0x03a96d (0x175a bytes) 000264 1784186787.461540 MIRROR_EXTEND to 2 mirrors 000265 1784186788.439330 punch from 0x03aa49 to 0x03f393 (0x494b bytes) 000266 1784186788.507528 read 0x00b50e thru 0x01b134 (0xfc27 bytes) 000268 1784186788.526154 punch from 0x02b908 to 0x03546c (0x9b65 bytes) 000269 1784186788.547699 mapwrite 0x0338dc thru 0x0349a3 (0x10c8 bytes) 000271 1784186788.616470 mapread 0x037a59 thru 0x03f393 (0x793b bytes) 000272 1784186788.636316 trunc from 0x03f394 to 0x016529 (0x28e6a bytes) 000274 1784186788.710305 fallocate 0x00dc75 thru 0x00ea14 (0xda0 bytes) 000275 1784186788.722921 trunc from 0x00109e to 0x0105f4 (0xf557 bytes) 000276 1784186788.734236 read 0x00ea7b thru 0x016529 (0x7aaf bytes) 000277 1784186788.767131 trunc from 0x010bc5 to 0x016529 (0x5965 bytes) 000278 1784186788.781203 punch from 0x014269 to 0x016529 (0x22c1 bytes) 000279 1784186788.792567 trunc from 0x00b4a8 to 0x01533d (0x9e96 bytes) 000280 1784186788.797450 mapread 0x005d58 thru 0x00a0e5 (0x438e bytes) 000281 1784186788.800231 mapwrite 0x01ea35 thru 0x027fff (0x95cb bytes) 000282 1784186788.844918 trunc from 0x028000 to 0x00f528 (0x18ad7 bytes) 000283 1784186788.863876 trunc from 0x00f529 to 0x01936c (0x9e44 bytes) 000285 1784186788.892679 read 0x0007a7 thru 0x001262 (0xabc bytes) 000287 1784186788.895908 mapwrite 0x02182c thru 0x027e4d (0x6622 bytes) 000288 1784186788.938527 mapread 0x00f22b thru 0x011c66 (0x2a3c bytes) 000289 1784186788.947501 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27e4e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27e4e): Mirror 2: 0x19dfa811 (stale) CRC-32 checksum value for chunk [0, 0x27e4e): Mirror 1: 0x92b262e1 Mirror 3: 0x92b262e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27e4e: skipped 000290 1784186789.971407 trunc from 0x027e4e to 0x035426 (0xd5d9 bytes) 000291 1784186790.024100 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35427): Mirror 2: 0x71c65c14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35427): Mirror 2: 0x71c65c14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35427: skipped 000292 1784186790.310085 write 0x0209b7 thru 0x02f9d0 (0xf01a bytes) 000293 1784186790.324571 write 0x017aae thru 0x0206d8 (0x8c2b bytes) 000294 1784186790.331379 trunc from 0x035427 to 0x01fbc1 (0x15865 bytes) 000295 1784186790.347116 fallocate 0x02f3d0 thru 0x037c2b (0x885c bytes) 000296 1784186790.352453 mapwrite 0x023d23 thru 0x02f52e (0xb80c bytes) 000297 1784186790.407789 read 0x00ad1b thru 0x018346 (0xd62c bytes) 000298 1784186790.416898 trunc from 0x02f06a to 0x02f52e (0x4c5 bytes) 000299 1784186790.422026 punch from 0x0115a1 to 0x0151bb (0x3c1b bytes) 000301 1784186790.427175 read 0x026904 thru 0x02f52e (0x8c2b bytes) 000302 1784186790.440164 fallocate 0x007674 thru 0x00bd49 (0x46d6 bytes) 000303 1784186790.445057 read 0x00dd82 thru 0x01cc50 (0xeecf bytes) 000304 1784186790.446205 fallocate 0x0313d9 thru 0x03e06f (0xcc97 bytes) 000305 1784186790.451038 fallocate 0x03cb8a thru 0x03ffff (0x3476 bytes) 000306 1784186790.455816 fallocate 0x030c02 thru 0x03b5c4 (0xa9c3 bytes) 000307 1784186790.464165 MIRROR_RESYNC 2 mirrors 000308 1784186791.038302 mapread 0x01d032 thru 0x0246f6 (0x76c5 bytes) 000309 1784186791.062810 write 0x02e1eb thru 0x037516 (0x932c bytes) 000310 1784186791.128440 trunc from 0x00701a to 0x009a68 (0x2a4f bytes) 000311 1784186791.133676 trunc from 0x040000 to 0x039a33 (0x65cc bytes) 000312 1784186791.178552 punch from 0x0030bc to 0x00a796 (0x76db bytes) 000314 1784186791.189513 fallocate 0x039268 thru 0x03ffff (0x6d98 bytes) 000315 1784186791.198702 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000316 1784186791.405609 write 0x01a60b thru 0x026062 (0xba58 bytes) 000317 1784186791.425641 trunc from 0x02fbd2 to 0x03802b (0x845a bytes) 000318 1784186791.433578 punch from 0x030aa7 to 0x03b94c (0xaea6 bytes) 000320 1784186791.443337 trunc from 0x00573b to 0x009645 (0x3f0b bytes) 000321 1784186791.455904 trunc from 0x040000 to 0x009718 (0x368e7 bytes) 000322 1784186791.479844 write 0x02e49d thru 0x03217d (0x3ce1 bytes) 000323 1784186791.485455 mapwrite 0x01c587 thru 0x02654f (0x9fc9 bytes) 000324 1784186791.544041 read 0x0028f3 thru 0x00f5ac (0xccba bytes) 000325 1784186791.574098 trunc from 0x03217e to 0x03fe6e (0xdcf1 bytes) 000327 1784186791.608573 trunc from 0x02eccd to 0x03c243 (0xd577 bytes) 000329 1784186791.616591 trunc from 0x03fe6f to 0x01ffea (0x1fe84 bytes) 000330 1784186791.643571 read 0x004fe4 thru 0x0095d0 (0x45ed bytes) 000331 1784186791.644706 mapread 0x005708 thru 0x0093b6 (0x3caf bytes) 000332 1784186791.658555 MIRROR_EXTEND to 2 mirrors 000333 1784186792.585042 write 0x02f867 thru 0x02fec3 (0x65d bytes) 000336 1784186792.619926 mapread 0x01f98b thru 0x029a36 (0xa0ac bytes) 000337 1784186792.640372 mapwrite 0x032e9c thru 0x03ffff (0xd164 bytes) 000338 1784186792.694527 mapread 0x020a7f thru 0x022ffa (0x257c bytes) 000339 1784186792.696585 write 0x00b666 thru 0x014719 (0x90b4 bytes) 000340 1784186792.708084 mapread 0x006bd9 thru 0x01141f (0xa847 bytes) 000341 1784186792.715843 trunc from 0x040000 to 0x010676 (0x2f989 bytes) 000342 1784186792.730689 punch from 0x00d99b to 0x010676 (0x2cdc bytes) 000343 1784186792.743871 trunc from 0x010677 to 0x03dc8e (0x2d618 bytes) 000345 1784186792.765005 mapread 0x0391cf thru 0x03dc8e (0x4ac0 bytes) 000346 1784186792.782213 MIRROR_RESYNC 2 mirrors 000347 1784186793.375198 mapwrite 0x0278c1 thru 0x032618 (0xad58 bytes) 000348 1784186793.465647 punch from 0x0099ae to 0x01688b (0xcede bytes) 000349 1784186793.470718 mapread 0x012b2b thru 0x01bf55 (0x942b bytes) 000350 1784186793.491465 mapread 0x00f332 thru 0x01617e (0x6e4d bytes) 000351 1784186793.493055 mapread 0x004e08 thru 0x009e39 (0x5032 bytes) 000352 1784186793.503100 trunc from 0x03dc8f to 0x034554 (0x973a bytes) 000353 1784186793.522404 fallocate 0x025a3a thru 0x03077e (0xad45 bytes) 000354 1784186793.531026 MIRROR_RESYNC 2 mirrors 000355 1784186794.137335 trunc from 0x00f3d7 to 0x01442e (0x5058 bytes) 000356 1784186794.169791 fallocate 0x02d9f5 thru 0x0321c2 (0x47ce bytes) 000357 1784186794.174159 read 0x02b44c thru 0x02fc63 (0x4818 bytes) 000358 1784186794.182077 trunc from 0x0187a9 to 0x0211e3 (0x8a3b bytes) 000359 1784186794.186600 trunc from 0x034555 to 0x01bd65 (0x187ef bytes) 000360 1784186794.206990 mapread 0x0146f5 thru 0x01bd65 (0x7671 bytes) 000361 1784186794.225692 mapwrite 0x01fc09 thru 0x0251c5 (0x55bd bytes) 000362 1784186794.271824 trunc from 0x0251c6 to 0x012c4c (0x12579 bytes) 000363 1784186794.291602 punch from 0x00527f to 0x00ac98 (0x5a1a bytes) 000364 1784186794.299401 mapread 0x010a3b thru 0x012c4c (0x2212 bytes) 000365 1784186794.300467 read 0x00d635 thru 0x012c4c (0x5618 bytes) 000366 1784186794.322150 MIRROR_RESYNC 2 mirrors 000367 1784186794.931719 fallocate 0x022ba1 thru 0x03179a (0xebfa bytes) 000368 1784186794.984176 write 0x029e20 thru 0x037680 (0xd861 bytes) 000369 1784186794.987097 write 0x0315a9 thru 0x034345 (0x2d9d bytes) 000371 1784186794.989842 mapwrite 0x027855 thru 0x02cfd7 (0x5783 bytes) 000372 1784186795.017938 fallocate 0x030580 thru 0x03f643 (0xf0c4 bytes) 000373 1784186795.024029 write 0x021d90 thru 0x02221a (0x48b bytes) 000374 1784186795.033230 punch from 0x01f917 to 0x026816 (0x6f00 bytes) 000375 1784186795.047012 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000376 1784186795.178976 punch from 0x01c299 to 0x01ca2e (0x796 bytes) 000377 1784186795.185594 write 0x00b96d thru 0x00edd6 (0x346a bytes) 000378 1784186795.200041 punch from 0x00f189 to 0x01731f (0x8197 bytes) 000379 1784186795.205958 trunc from 0x021ba4 to 0x024ff5 (0x3452 bytes) 000380 1784186795.210772 punch from 0x008138 to 0x010dc9 (0x8c92 bytes) 000381 1784186795.222570 mapread 0x01d04f thru 0x02af67 (0xdf19 bytes) 000382 1784186795.257584 mapread 0x0029b4 thru 0x004309 (0x1956 bytes) 000383 1784186795.316744 mapwrite 0x02c095 thru 0x037db2 (0xbd1e bytes) 000384 1784186795.492769 fallocate 0x02fb36 thru 0x03d3b7 (0xd882 bytes) 000385 1784186795.503597 trunc from 0x0106d8 to 0x01e03b (0xd964 bytes) 000386 1784186795.514407 fallocate 0x003698 thru 0x007214 (0x3b7d bytes) 000388 1784186795.531727 read 0x0198a6 thru 0x0221d7 (0x8932 bytes) 000390 1784186795.583057 trunc from 0x037db3 to 0x00c056 (0x2bd5c bytes) 000392 1784186795.636838 mapread 0x006591 thru 0x00b5a9 (0x5019 bytes) 000393 1784186795.640713 punch from 0x00790e to 0x00c056 (0x4749 bytes) 000394 1784186795.647785 write 0x02b5d3 thru 0x0369c4 (0xb3f2 bytes) 000397 1784186795.668398 trunc from 0x010c20 to 0x01f086 (0xe467 bytes) 000398 1784186795.672452 punch from 0x033f19 to 0x0369c4 (0x2aac bytes) 000399 1784186795.697593 fallocate 0x0074ee thru 0x00dadb (0x65ee bytes) 000400 1784186795.704461 mapwrite 0x03eefc thru 0x03ffff (0x1104 bytes) 000401 1784186795.760151 trunc from 0x040000 to 0x010055 (0x2ffaa bytes) 000402 1784186795.781158 trunc from 0x010056 to 0x036cee (0x26c99 bytes) 000403 1784186795.800400 trunc from 0x036cef to 0x01cc96 (0x1a058 bytes) 000404 1784186795.818059 mapread 0x01b051 thru 0x01cc96 (0x1c46 bytes) 000405 1784186795.836450 punch from 0x002953 to 0x008b07 (0x61b5 bytes) 000406 1784186795.849996 fallocate 0x011dde thru 0x01e688 (0xc8ab bytes) 000407 1784186795.860790 fallocate 0x00ed59 thru 0x0176df (0x8987 bytes) 000408 1784186795.870592 write 0x02bfe2 thru 0x032487 (0x64a6 bytes) 000412 1784186795.877158 write 0x02e13a thru 0x03084d (0x2714 bytes) 000413 1784186795.878610 mapwrite 0x020dbe thru 0x0271a1 (0x63e4 bytes) 000414 1784186795.925213 fallocate 0x03c24d thru 0x03d70b (0x14bf bytes) 000415 1784186795.930217 write 0x000f3c thru 0x001a18 (0xadd bytes) 000416 1784186795.936826 write 0x001127 thru 0x00dab4 (0xc98e bytes) 000418 1784186795.945075 fallocate 0x013c80 thru 0x01d721 (0x9aa2 bytes) 000420 1784186795.950183 mapread 0x02c3ec thru 0x02e13d (0x1d52 bytes) 000422 1784186795.951699 punch from 0x015772 to 0x01c8a7 (0x7136 bytes) 000423 1784186795.957285 punch from 0x0031e6 to 0x009f18 (0x6d33 bytes) 000424 1784186795.978239 trunc from 0x02372d to 0x0266d8 (0x2fac bytes) 000426 1784186795.984879 fallocate 0x02ea66 thru 0x035885 (0x6e20 bytes) 000427 1784186795.990517 mapread 0x029e37 thru 0x02f338 (0x5502 bytes) 000428 1784186796.000680 trunc from 0x00b3b7 to 0x015b7b (0xa7c5 bytes) 000429 1784186796.005711 read 0x03401f thru 0x0375b1 (0x3593 bytes) 000430 1784186796.023438 mapread 0x0355fc thru 0x03d70b (0x8110 bytes) 000431 1784186796.026003 read 0x017164 thru 0x01f8fa (0x8797 bytes) 000433 1784186796.028161 mapread 0x025518 thru 0x028312 (0x2dfb bytes) 000434 1784186796.030046 read 0x009e78 thru 0x0134fe (0x9687 bytes) 000435 1784186796.030818 fallocate 0x0264d6 thru 0x02f022 (0x8b4d bytes) 000436 1784186796.036339 mapread 0x0280db thru 0x02f4ea (0x7410 bytes) 000438 1784186796.039729 write 0x016c62 thru 0x02400d (0xd3ac bytes) 000439 1784186796.041580 trunc from 0x03d70c to 0x020006 (0x1d705 bytes) 000440 1784186796.065340 trunc from 0x020007 to 0x01eb10 (0x14f6 bytes) 000442 1784186796.079523 read 0x017c47 thru 0x01db88 (0x5f42 bytes) 000443 1784186796.080216 punch from 0x0054f1 to 0x0099d0 (0x44e0 bytes) 000444 1784186796.086120 punch from 0x007c24 to 0x0145e0 (0xc9bd bytes) 000447 1784186796.093256 mapread 0x012215 thru 0x014074 (0x1e60 bytes) 000448 1784186796.104708 write 0x03034e thru 0x03ffff (0xfcb2 bytes) 000449 1784186796.106913 mapread 0x02689a thru 0x0295d3 (0x2d3a bytes) 000450 1784186796.116111 fallocate 0x009bf3 thru 0x014794 (0xaba2 bytes) 000451 1784186796.120860 read 0x034ad2 thru 0x03ffff (0xb52e bytes) 000452 1784186796.122458 write 0x020ff9 thru 0x02261e (0x1626 bytes) 000454 1784186796.124824 write 0x017f73 thru 0x023b82 (0xbc10 bytes) 000455 1784186796.127326 trunc from 0x040000 to 0x01ccf2 (0x2330d bytes) 000456 1784186796.145872 punch from 0x00d869 to 0x0190aa (0xb842 bytes) 000460 1784186796.167676 mapwrite 0x0137f6 thru 0x014459 (0xc64 bytes) 000462 1784186796.192844 read 0x01831a thru 0x019d36 (0x1a1d bytes) 000463 1784186796.194340 write 0x001c75 thru 0x0044a8 (0x2834 bytes) 000464 1784186796.195931 trunc from 0x01ccf3 to 0x017c42 (0x50b0 bytes) 000465 1784186796.212141 read 0x007ed7 thru 0x00f7ce (0x78f8 bytes) 000466 1784186796.213579 mapwrite 0x0068ca thru 0x00cf6a (0x66a1 bytes) 000467 1784186796.231693 write 0x00ca7c thru 0x010d64 (0x42e9 bytes) 000472 1784186796.237413 read 0x005c68 thru 0x00825f (0x25f8 bytes) 000474 1784186796.239453 mapread 0x007f66 thru 0x00f237 (0x72d2 bytes) 000476 1784186796.245284 MIRROR_EXTEND to 2 mirrors 000478 1784186796.994243 trunc from 0x006342 to 0x01129e (0xaf5d bytes) 000479 1784186797.029320 trunc from 0x017c43 to 0x01e2ad (0x666b bytes) 000480 1784186797.064971 trunc from 0x01e2ae to 0x0171fe (0x70af bytes) 000481 1784186797.080399 write 0x01343f thru 0x01a52d (0x70ef bytes) 000482 1784186797.085597 mapwrite 0x03041c thru 0x031b49 (0x172e bytes) 000483 1784186797.136125 trunc from 0x031b4a to 0x011dd5 (0x1fd74 bytes) 000484 1784186797.152515 mapread 0x011681 thru 0x011dd5 (0x755 bytes) 000486 1784186797.164818 trunc from 0x011dd6 to 0x02d124 (0x1b34f bytes) 000487 1784186797.180405 trunc from 0x02d125 to 0x03137e (0x425a bytes) 000488 1784186797.197944 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000489 1784186797.305225 mapread 0x00b925 thru 0x012ddf (0x74bb bytes) 000490 1784186797.328433 read 0x002131 thru 0x00e39d (0xc26d bytes) 000492 1784186797.334667 mapread 0x00db36 thru 0x017b40 (0xa00b bytes) 000495 1784186797.345486 fallocate 0x004bf1 thru 0x00e2c3 (0x96d3 bytes) 000496 1784186797.353940 MIRROR_EXTEND to 2 mirrors 000498 1784186798.234993 trunc from 0x03137f to 0x03768e (0x6310 bytes) 000500 1784186798.324088 write 0x02f8f1 thru 0x03d678 (0xdd88 bytes) 000501 1784186798.329495 fallocate 0x010b21 thru 0x012466 (0x1946 bytes) 000502 1784186798.333262 trunc from 0x03d679 to 0x007f48 (0x35730 bytes) 000503 1784186798.350417 fallocate 0x006974 thru 0x011c4e (0xb2db bytes) 000504 1784186798.358891 mapwrite 0x020e20 thru 0x028994 (0x7b75 bytes) 000505 1784186798.422336 trunc from 0x028995 to 0x007418 (0x2157c bytes) 000506 1784186798.443704 read 0x0059d4 thru 0x007418 (0x1a45 bytes) 000507 1784186798.451080 mapwrite 0x028b31 thru 0x03488b (0xbd5b bytes) 000508 1784186798.507827 read 0x01ebd8 thru 0x01ef8d (0x3b6 bytes) 000509 1784186798.519088 trunc from 0x03488c to 0x00f9a4 (0x24ee7 bytes) 000510 1784186798.535899 fallocate 0x005b25 thru 0x00a062 (0x453e bytes) 000511 1784186798.542220 mapwrite 0x01539c thru 0x024671 (0xf2d6 bytes) 000512 1784186798.737471 read 0x01f95d thru 0x020671 (0xd15 bytes) 000513 1784186798.738213 mapwrite 0x02eca5 thru 0x038bcd (0x9f29 bytes) 000514 1784186798.804610 mapread 0x01080f thru 0x01bd6b (0xb55d bytes) 000515 1784186798.806429 write 0x026021 thru 0x02c862 (0x6842 bytes) 000516 1784186798.809033 trunc from 0x038bce to 0x0285c7 (0x10606 bytes) 000517 1784186798.824054 read 0x01223b thru 0x01bb1d (0x98e3 bytes) 000518 1784186798.825260 mapread 0x00be35 thru 0x018183 (0xc34f bytes) 000519 1784186798.827556 mapwrite 0x00fddd thru 0x0101f5 (0x419 bytes) 000520 1784186798.841398 read 0x005969 thru 0x0141da (0xe872 bytes) 000521 1784186798.845759 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x285c8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x285c8): Mirror 2: 0x41bc5d2e (stale) CRC-32 checksum value for chunk [0, 0x285c8): Mirror 1: 0x8de4b9ef Mirror 3: 0x8de4b9ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x285c8: skipped 000522 1784186799.672145 write 0x02c3d7 thru 0x035509 (0x9133 bytes) 000523 1784186799.707369 read 0x0203f9 thru 0x029c6d (0x9875 bytes) 000524 1784186799.714427 write 0x0014e3 thru 0x005687 (0x41a5 bytes) 000525 1784186799.722937 mapwrite 0x0284b0 thru 0x037423 (0xef74 bytes) 000526 1784186799.777991 punch from 0x02dbbc to 0x037423 (0x9868 bytes) 000527 1784186799.784887 fallocate 0x03e113 thru 0x03f040 (0xf2e bytes) 000528 1784186799.789674 mapwrite 0x0256d1 thru 0x0332a6 (0xdbd6 bytes) 000529 1784186799.816350 write 0x036114 thru 0x03ffff (0x9eec bytes) 000530 1784186799.822210 fallocate 0x020cf8 thru 0x022c2a (0x1f33 bytes) 000531 1784186799.827281 punch from 0x03502e to 0x036d3b (0x1d0e bytes) 000532 1784186799.838253 fallocate 0x0275e7 thru 0x027c00 (0x61a bytes) 000533 1784186799.841271 trunc from 0x0345e3 to 0x03b96d (0x738b bytes) 000534 1784186799.845522 trunc from 0x0142d9 to 0x01cde6 (0x8b0e bytes) 000535 1784186799.850340 mapread 0x03d7ee thru 0x03ffff (0x2812 bytes) 000536 1784186799.851450 trunc from 0x01014e to 0x018ea5 (0x8d58 bytes) 000537 1784186799.856270 trunc from 0x025160 to 0x026851 (0x16f2 bytes) 000538 1784186799.865982 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xda4defa3 (stale) Mirror 3: 0x952d43ed (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc9ab2a72 Mirror 4: 0xc9ab2a72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000539 1784186800.838764 trunc from 0x040000 to 0x003651 (0x3c9ae bytes) 000540 1784186800.902844 MIRROR_RESYNC 4 mirrors 000541 1784186801.544683 mapwrite 0x012816 thru 0x01b768 (0x8f53 bytes) 000542 1784186801.606511 trunc from 0x01b769 to 0x000443 (0x1b325 bytes) 000543 1784186801.639151 trunc from 0x000444 to 0x01b566 (0x1b123 bytes) 000544 1784186801.652390 mapwrite 0x034c52 thru 0x03b5ef (0x699e bytes) 000545 1784186801.692150 punch from 0x01d8b9 to 0x01ea08 (0x1150 bytes) 000546 1784186801.701602 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b5f0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b5f0): Mirror 2: 0x297156df (stale) Mirror 3: 0x297156df (stale) Mirror 4: 0x297156df (stale) CRC-32 checksum value for chunk [0, 0x3b5f0): Mirror 1: 0xa6f5e3a7 Mirror 5: 0xa6f5e3a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b5f0: skipped 000547 1784186802.590506 punch from 0x02100a to 0x025543 (0x453a bytes) 000548 1784186802.640266 fallocate 0x023388 thru 0x027d98 (0x4a11 bytes) 000549 1784186802.659873 fallocate 0x03adef thru 0x03f4cc (0x46de bytes) 000550 1784186802.670960 mapwrite 0x02f989 thru 0x0379b9 (0x8031 bytes) 000551 1784186802.709139 read 0x017397 thru 0x0198a9 (0x2513 bytes) 000552 1784186802.716841 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f4cd): Mirror 3: 0x3cb5a415 (stale) Mirror 4: 0x3cb5a415 (stale) Mirror 5: 0x581319b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f4cd): Mirror 3: 0x3cb5a415 (stale) Mirror 4: 0x3cb5a415 (stale) Mirror 5: 0x581319b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f4cd: skipped 000553 1784186803.130573 read 0x00396d thru 0x007275 (0x3909 bytes) 000554 1784186803.139571 mapread 0x02f14d thru 0x0330b4 (0x3f68 bytes) 000555 1784186803.165931 trunc from 0x008f46 to 0x013665 (0xa720 bytes) 000556 1784186803.179422 fallocate 0x02c4d0 thru 0x03ab31 (0xe662 bytes) 000557 1784186803.187361 punch from 0x011be2 to 0x014be7 (0x3006 bytes) 000558 1784186803.191889 fallocate 0x00d548 thru 0x015db4 (0x886d bytes) 000559 1784186803.196829 write 0x031d84 thru 0x03e223 (0xc4a0 bytes) 000560 1784186803.207025 mapread 0x00316c thru 0x00d1bc (0xa051 bytes) 000561 1784186803.215272 read 0x0042fb thru 0x007ec6 (0x3bcc bytes) 000563 1784186803.223992 MIRROR_RESYNC 4 mirrors 000564 1784186804.009365 write 0x0296bf thru 0x03269a (0x8fdc bytes) 000565 1784186804.092020 punch from 0x003562 to 0x00ced1 (0x9970 bytes) 000566 1784186804.100514 fallocate 0x01d92b thru 0x01e151 (0x827 bytes) 000567 1784186804.110194 fallocate 0x00ab74 thru 0x00f2f2 (0x477f bytes) 000568 1784186804.127537 trunc from 0x016532 to 0x01d867 (0x7336 bytes) 000569 1784186804.135822 mapread 0x039fc9 thru 0x03f4cc (0x5504 bytes) 000570 1784186804.171214 write 0x00fb78 thru 0x012a0a (0x2e93 bytes) 000571 1784186804.185124 trunc from 0x01d125 to 0x01fe98 (0x2d74 bytes) 000572 1784186804.207472 MIRROR_RESYNC 4 mirrors 000573 1784186805.205867 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000574 1784186805.505014 write 0x00b0f1 thru 0x017358 (0xc268 bytes) 000575 1784186805.560304 MIRROR_RESYNC 3 mirrors 000576 1784186806.224164 trunc from 0x03f4cd to 0x030094 (0xf438 bytes) 000577 1784186806.280497 mapwrite 0x03fa9e thru 0x03ffff (0x562 bytes) 000578 1784186806.320157 read 0x02e093 thru 0x0365e3 (0x8551 bytes) 000579 1784186806.333438 read 0x03133d thru 0x03ffff (0xecc3 bytes) 000580 1784186806.347254 trunc from 0x040000 to 0x009bdd (0x36422 bytes) 000581 1784186806.367276 MIRROR_RESYNC 3 mirrors 000582 1784186806.824326 trunc from 0x009bde to 0x030b4b (0x26f6e bytes) 000583 1784186806.916615 write 0x037b16 thru 0x03abf3 (0x30de bytes) 000584 1784186806.922373 trunc from 0x03abf4 to 0x019076 (0x21b7d bytes) 000585 1784186806.953535 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x19077): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19077): Mirror 3: 0x22860e7c (stale) Mirror 5: 0x22860e7c (stale) CRC-32 checksum value for chunk [0, 0x19077): Mirror 1: 0x22860e7c Mirror 6: 0x22860e7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19077: skipped 000586 1784186807.729823 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x19077): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19077): Mirror 3: 0x22860e7c (stale) Mirror 5: 0x22860e7c (stale) CRC-32 checksum value for chunk [0, 0x19077): Mirror 1: 0x22860e7c Mirror 6: 0x22860e7c Mirror 7: 0x22860e7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19077: skipped 000587 1784186808.437354 read 0x0088b4 thru 0x013338 (0xaa85 bytes) 000588 1784186808.456068 punch from 0x006117 to 0x014667 (0xe551 bytes) 000589 1784186808.494412 write 0x024fe5 thru 0x026b3f (0x1b5b bytes) 000590 1784186808.501424 write 0x037504 thru 0x03ffff (0x8afc bytes) 000591 1784186808.502982 mapread 0x018c54 thru 0x01fbb7 (0x6f64 bytes) 000592 1784186808.525663 mapwrite 0x025fe6 thru 0x03593f (0xf95a bytes) 000593 1784186808.560112 fallocate 0x03034a thru 0x032810 (0x24c7 bytes) 000595 1784186808.570805 write 0x017c08 thru 0x01c5e6 (0x49df bytes) 000597 1784186808.572591 write 0x0080e2 thru 0x0123fa (0xa319 bytes) 000598 1784186808.582875 mapread 0x037001 thru 0x038d1f (0x1d1f bytes) 000599 1784186808.584339 read 0x0155c9 thru 0x02066f (0xb0a7 bytes) 000600 1784186808.589732 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa58983ed Mirror 8: 0xa58983ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000601 1784186809.894910 read 0x007d3f thru 0x00952b (0x17ed bytes) 000602 1784186809.906812 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa58983ed Mirror 8: 0xa58983ed Mirror 9: 0xa58983ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000603 1784186811.207703 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa58983ed Mirror 8: 0xa58983ed Mirror 9: 0xa58983ed Mirror 10: 0xa58983ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000604 1784186812.689177 MIRROR_RESYNC 8 mirrors 000605 1784186814.859563 mapread 0x0243e3 thru 0x032a3f (0xe65d bytes) 000606 1784186814.888184 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000607 1784186815.266195 MIRROR_EXTEND to 8 mirrors 000608 1784186816.029552 mapread 0x035e93 thru 0x03dfbb (0x8129 bytes) 000609 1784186816.048786 fallocate 0x039a0f thru 0x03ffff (0x65f1 bytes) 000611 1784186816.090166 write 0x033c82 thru 0x03f743 (0xbac2 bytes) 000612 1784186816.105286 trunc from 0x008dfb to 0x00b35c (0x2562 bytes) 000613 1784186816.110745 mapread 0x012024 thru 0x0190f5 (0x70d2 bytes) 000614 1784186816.121010 write 0x0129e8 thru 0x01e1a4 (0xb7bd bytes) 000615 1784186816.132174 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa58983ed (stale) Mirror 6: 0xa58983ed (stale) Mirror 7: 0xa58983ed (stale) Mirror 8: 0xa58983ed (stale) Mirror 10: 0xa58983ed (stale) Mirror 11: 0xa58983ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa58983ed (stale) Mirror 6: 0xa58983ed (stale) Mirror 7: 0xa58983ed (stale) Mirror 8: 0xa58983ed (stale) Mirror 10: 0xa58983ed (stale) Mirror 11: 0xa58983ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000616 1784186816.825391 read 0x036061 thru 0x03e85e (0x87fe bytes) 000617 1784186816.854996 trunc from 0x040000 to 0x03a63e (0x59c1 bytes) 000618 1784186816.883084 mapwrite 0x03c7d0 thru 0x03ffff (0x3830 bytes) 000619 1784186816.935206 trunc from 0x040000 to 0x02823c (0x17dc3 bytes) 000620 1784186816.953086 trunc from 0x01a9ac to 0x02443d (0x9a92 bytes) 000621 1784186816.959943 write 0x01495e thru 0x020eff (0xc5a2 bytes) 000622 1784186816.969810 fallocate 0x0243d9 thru 0x031032 (0xcc5a bytes) 000623 1784186816.975152 write 0x0205f7 thru 0x02b56b (0xaf75 bytes) 000625 1784186816.978615 punch from 0x01cafd to 0x02770b (0xac0f bytes) 000626 1784186816.996117 fallocate 0x014331 thru 0x016808 (0x24d8 bytes) 000627 1784186817.004995 write 0x003556 thru 0x0121c3 (0xec6e bytes) 000628 1784186817.015203 trunc from 0x02b56c to 0x00abda (0x20991 bytes) 000629 1784186817.030154 write 0x002195 thru 0x00adf3 (0x8c5f bytes) 000630 1784186817.041686 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xadf4): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xadf4): Mirror 5: 0xc1178b42 (stale) Mirror 6: 0xc1178b42 (stale) Mirror 7: 0xc1178b42 (stale) Mirror 8: 0xc1178b42 (stale) Mirror 10: 0xc1178b42 (stale) Mirror 11: 0xc1178b42 (stale) CRC-32 checksum value for chunk [0, 0xadf4): Mirror 1: 0xaee3674f Mirror 12: 0xaee3674f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xadf4: skipped 000631 1784186817.884191 trunc from 0x001a5e to 0x007c95 (0x6238 bytes) 000632 1784186817.924372 punch from 0x001e5a to 0x0041f0 (0x2397 bytes) 000633 1784186817.929915 punch from 0x002b8f to 0x007b94 (0x5006 bytes) 000634 1784186817.935195 mapwrite 0x025d03 thru 0x02a6bd (0x49bb bytes) 000635 1784186817.988989 fallocate 0x011b61 thru 0x020869 (0xed09 bytes) 000636 1784186817.994454 punch from 0x029594 to 0x02a6bd (0x112a bytes) 000637 1784186818.000554 trunc from 0x022414 to 0x02a6bd (0x82aa bytes) 000638 1784186818.005260 read 0x019b4e thru 0x027090 (0xd543 bytes) 000639 1784186818.013374 mapwrite 0x015451 thru 0x01893d (0x34ed bytes) 000640 1784186818.041244 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a6be): Mirror 5: 0x9701496 (stale) Mirror 6: 0x9701496 (stale) Mirror 7: 0x9701496 (stale) Mirror 8: 0x9701496 (stale) Mirror 11: 0x9701496 (stale) Mirror 12: 0xda9df134 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a6be): Mirror 5: 0x9701496 (stale) Mirror 6: 0x9701496 (stale) Mirror 7: 0x9701496 (stale) Mirror 8: 0x9701496 (stale) Mirror 11: 0x9701496 (stale) Mirror 12: 0xda9df134 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a6be: skipped 000641 1784186818.383584 punch from 0x0018cd to 0x00a476 (0x8baa bytes) 000642 1784186818.389169 mapwrite 0x0353bb thru 0x03ffff (0xac45 bytes) 000643 1784186818.424593 fallocate 0x03477d thru 0x03c9bd (0x8241 bytes) 000644 1784186818.428397 fallocate 0x038668 thru 0x03ffff (0x7998 bytes) 000645 1784186818.431688 fallocate 0x006e26 thru 0x00ce6c (0x6047 bytes) 000646 1784186818.435736 write 0x017c9a thru 0x022f79 (0xb2e0 bytes) 000647 1784186818.446112 mapread 0x032f1d thru 0x03fb99 (0xcc7d bytes) 000648 1784186818.448135 trunc from 0x040000 to 0x01b5e6 (0x24a19 bytes) 000649 1784186818.466289 read 0x00c6c4 thru 0x01b086 (0xe9c3 bytes) 000650 1784186818.480258 read 0x00cafc thru 0x015733 (0x8c38 bytes) 000651 1784186818.481035 write 0x03cff4 thru 0x03ffff (0x300c bytes) 000652 1784186818.484753 read 0x00ae1f thru 0x0165ff (0xb7e1 bytes) 000653 1784186818.495764 write 0x033c16 thru 0x03ffff (0xc3ea bytes) 000654 1784186818.499144 trunc from 0x01c020 to 0x01c8bc (0x89d bytes) 000655 1784186818.504012 read 0x038aea thru 0x03ffff (0x7516 bytes) 000656 1784186818.505771 write 0x014939 thru 0x0175c7 (0x2c8f bytes) 000657 1784186818.507757 mapwrite 0x004e1c thru 0x008e66 (0x404b bytes) 000658 1784186818.526436 read 0x02fd9b thru 0x035771 (0x59d7 bytes) 000659 1784186818.527192 fallocate 0x006dbd thru 0x0133cf (0xc613 bytes) 000660 1784186818.532819 trunc from 0x01d12f to 0x022685 (0x5557 bytes) 000661 1784186818.541386 trunc from 0x040000 to 0x0042f5 (0x3bd0a bytes) 000662 1784186818.559105 read 0x000a8e thru 0x0042f5 (0x3868 bytes) 000663 1784186818.562558 mapwrite 0x00f5b0 thru 0x013ce3 (0x4734 bytes) 000664 1784186818.610055 punch from 0x0064ce to 0x00f53d (0x9070 bytes) 000666 1784186818.617157 fallocate 0x038849 thru 0x03ffff (0x77b7 bytes) 000667 1784186818.622265 write 0x01a9cd thru 0x028480 (0xdab4 bytes) 000668 1784186818.636261 mapread 0x026f9e thru 0x0333b0 (0xc413 bytes) 000669 1784186818.639272 read 0x02f56a thru 0x0367b3 (0x724a bytes) 000670 1784186818.640983 mapwrite 0x03e6bb thru 0x03ffff (0x1945 bytes) 000671 1784186818.659885 mapread 0x015a38 thru 0x02145a (0xba23 bytes) 000672 1784186818.662048 write 0x01dc02 thru 0x02a2f2 (0xc6f1 bytes) 000673 1784186818.664730 write 0x012328 thru 0x01f199 (0xce72 bytes) 000674 1784186818.667037 read 0x0334df thru 0x0357c0 (0x22e2 bytes) 000675 1784186818.668233 fallocate 0x00b1c3 thru 0x00bf1c (0xd5a bytes) 000676 1784186818.673971 trunc from 0x035325 to 0x03e5b6 (0x9292 bytes) 000677 1784186818.679239 write 0x00cc44 thru 0x00f179 (0x2536 bytes) 000678 1784186818.689254 write 0x0205fa thru 0x029f78 (0x997f bytes) 000680 1784186818.692245 mapread 0x037ace thru 0x03f516 (0x7a49 bytes) 000681 1784186818.694144 write 0x0111d6 thru 0x01296c (0x1797 bytes) 000682 1784186818.695294 trunc from 0x040000 to 0x009ec3 (0x3613c bytes) 000683 1784186818.712099 trunc from 0x009ec4 to 0x002de1 (0x70e2 bytes) 000685 1784186818.729958 mapread 0x001955 thru 0x002de1 (0x148d bytes) 000686 1784186818.731081 trunc from 0x000bd4 to 0x002de1 (0x220e bytes) 000687 1784186818.737929 trunc from 0x0006ff to 0x002de1 (0x26e3 bytes) 000688 1784186818.743815 trunc from 0x00093f to 0x002de1 (0x24a3 bytes) 000689 1784186818.757061 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2de2): Mirror 5: 0x9a1b4f64 (stale) Mirror 6: 0x9a1b4f64 (stale) Mirror 8: 0x9a1b4f64 (stale) Mirror 11: 0x9a1b4f64 (stale) Mirror 12: 0xe847cf41 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2de2): Mirror 5: 0x9a1b4f64 (stale) Mirror 6: 0x9a1b4f64 (stale) Mirror 8: 0x9a1b4f64 (stale) Mirror 11: 0x9a1b4f64 (stale) Mirror 12: 0xe847cf41 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2de2: skipped 000690 1784186819.022514 write 0x01c270 thru 0x0261fa (0x9f8b bytes) 000691 1784186819.030316 read 0x008c73 thru 0x008e64 (0x1f2 bytes) 000692 1784186819.035211 read 0x01c70c thru 0x0261fa (0x9aef bytes) 000693 1784186819.036006 mapread 0x00b515 thru 0x00b913 (0x3ff bytes) 000694 1784186819.057958 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x261fb): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x261fb): Mirror 5: 0xda048e68 (stale) Mirror 6: 0xda048e68 (stale) Mirror 8: 0xda048e68 (stale) Mirror 11: 0xda048e68 (stale) Mirror 12: 0x700b3c81 (stale) CRC-32 checksum value for chunk [0, 0x261fb): Mirror 1: 0x736f4c4f Mirror 13: 0x736f4c4f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x261fb: skipped 000695 1784186820.077212 trunc from 0x00795e to 0x00c624 (0x4cc7 bytes) 000696 1784186820.155183 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x261fb): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x261fb): Mirror 5: 0xda048e68 (stale) Mirror 6: 0xda048e68 (stale) Mirror 8: 0xda048e68 (stale) Mirror 11: 0xda048e68 (stale) Mirror 12: 0x700b3c81 (stale) Mirror 13: 0x736f4c4f (stale) CRC-32 checksum value for chunk [0, 0x261fb): Mirror 1: 0x736f4c4f Mirror 14: 0x736f4c4f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x261fb: skipped 000697 1784186821.191822 fallocate 0x032f21 thru 0x03dbd0 (0xacb0 bytes) 000698 1784186821.226374 punch from 0x023c12 to 0x030a78 (0xce67 bytes) 000699 1784186821.232085 trunc from 0x03dbd1 to 0x0151ed (0x289e3 bytes) 000700 1784186821.254923 read 0x003404 thru 0x00b990 (0x858d bytes) 000701 1784186821.262945 write 0x00ce69 thru 0x017c67 (0xadff bytes) 000702 1784186821.268080 mapwrite 0x014d65 thru 0x01ae2f (0x60cb bytes) 000703 1784186821.330081 trunc from 0x01ae30 to 0x020e07 (0x5fd8 bytes) 000704 1784186821.348282 trunc from 0x002984 to 0x010bfa (0xe277 bytes) 000705 1784186821.372333 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20e08): Mirror 5: 0xa01a7fe2 (stale) Mirror 6: 0xa01a7fe2 (stale) Mirror 11: 0xa01a7fe2 (stale) Mirror 12: 0x14916d25 (stale) Mirror 13: 0x32edcd3d (stale) Mirror 14: 0x32edcd3d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20e08): Mirror 5: 0xa01a7fe2 (stale) Mirror 6: 0xa01a7fe2 (stale) Mirror 11: 0xa01a7fe2 (stale) Mirror 12: 0x14916d25 (stale) Mirror 13: 0x32edcd3d (stale) Mirror 14: 0x32edcd3d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20e08: skipped 000706 1784186821.786212 fallocate 0x02928a thru 0x02e13f (0x4eb6 bytes) 000707 1784186821.792129 punch from 0x017799 to 0x0184bf (0xd27 bytes) 000708 1784186821.797240 trunc from 0x02e140 to 0x01aada (0x13665 bytes) 000709 1784186821.820337 write 0x00a220 thru 0x00fd3e (0x5b1f bytes) 000710 1784186821.831131 read 0x0012a0 thru 0x003bd0 (0x2931 bytes) 000711 1784186821.837521 mapread 0x01a9a1 thru 0x01aada (0x13a bytes) 000712 1784186821.843125 write 0x02a34d thru 0x02c1ff (0x1eb3 bytes) 000713 1784186821.844530 trunc from 0x02c200 to 0x007340 (0x24ebf bytes) 000714 1784186821.858315 trunc from 0x007341 to 0x0047dd (0x2b63 bytes) 000715 1784186821.878585 mapread 0x000f7b thru 0x0047dd (0x3863 bytes) 000716 1784186821.889974 mapread 0x003dcc thru 0x0047dd (0xa12 bytes) 000717 1784186821.891097 punch from 0x001542 to 0x0047dd (0x329c bytes) 000718 1784186821.896605 mapread 0x002cbc thru 0x0047dd (0x1b22 bytes) 000719 1784186821.904090 mapread 0x000611 thru 0x0047dd (0x41cd bytes) 000720 1784186821.905725 fallocate 0x00d524 thru 0x00df0e (0x9eb bytes) 000721 1784186821.911038 fallocate 0x035942 thru 0x039071 (0x3730 bytes) 000722 1784186821.916649 write 0x012601 thru 0x012f26 (0x926 bytes) 000723 1784186821.921960 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12f27): Mirror 5: 0x482d85ae (stale) Mirror 6: 0x482d85ae (stale) Mirror 11: 0x482d85ae (stale) Mirror 12: 0xc8cdecb8 (stale) Mirror 14: 0xfe9cd157 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12f27): Mirror 5: 0x482d85ae (stale) Mirror 6: 0x482d85ae (stale) Mirror 11: 0x482d85ae (stale) Mirror 12: 0xc8cdecb8 (stale) Mirror 14: 0xfe9cd157 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12f27: skipped 000724 1784186822.216076 trunc from 0x0124c3 to 0x01264b (0x189 bytes) 000725 1784186822.220897 punch from 0x005b2a to 0x010513 (0xa9ea bytes) 000726 1784186822.226575 mapread 0x00063c thru 0x00a85a (0xa21f bytes) 000727 1784186822.251003 mapwrite 0x01f84f thru 0x02dfbf (0xe771 bytes) 000728 1784186822.341912 punch from 0x02bbce to 0x02dfbf (0x23f2 bytes) 000729 1784186822.352706 fallocate 0x008514 thru 0x0129af (0xa49c bytes) 000730 1784186822.379242 trunc from 0x00474a to 0x005227 (0xade bytes) 000732 1784186822.409217 write 0x03650f thru 0x038581 (0x2073 bytes) 000733 1784186822.412185 fallocate 0x0209a1 thru 0x024650 (0x3cb0 bytes) 000735 1784186822.426330 punch from 0x01442c to 0x022fd3 (0xeba8 bytes) 000736 1784186822.435242 write 0x02e63d thru 0x035d1b (0x76df bytes) 000737 1784186822.454175 write 0x0184e9 thru 0x0262af (0xddc7 bytes) 000738 1784186822.467112 mapread 0x0349fb thru 0x036c46 (0x224c bytes) 000739 1784186822.471018 mapread 0x00a609 thru 0x0103ba (0x5db2 bytes) 000740 1784186822.477655 trunc from 0x00194a to 0x010954 (0xf00b bytes) 000742 1784186822.512094 trunc from 0x038582 to 0x0384b1 (0xd0 bytes) 000743 1784186822.537059 mapwrite 0x00bbd5 thru 0x01a30e (0xe73a bytes) 000744 1784186822.609463 trunc from 0x028d5f to 0x031329 (0x85cb bytes) 000745 1784186822.625354 trunc from 0x01f99a to 0x029f4f (0xa5b6 bytes) 000746 1784186822.629946 read 0x02ecad thru 0x030d46 (0x209a bytes) 000747 1784186822.639828 mapwrite 0x014386 thru 0x016881 (0x24fc bytes) 000749 1784186822.657295 mapwrite 0x03bdaf thru 0x03ffff (0x4251 bytes) 000750 1784186822.705949 trunc from 0x040000 to 0x03318c (0xce73 bytes) 000751 1784186822.730358 trunc from 0x03318d to 0x0053ed (0x2dd9f bytes) 000752 1784186822.751026 punch from 0x001d1b to 0x0053ed (0x36d3 bytes) 000753 1784186822.770018 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x53ee): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x53ee): Mirror 5: 0x82fdd747 (stale) Mirror 6: 0x82fdd747 (stale) Mirror 11: 0x82fdd747 (stale) Mirror 12: 0xec162d7a (stale) Mirror 14: 0x82fdd747 (stale) CRC-32 checksum value for chunk [0, 0x53ee): Mirror 1: 0x82fdd747 Mirror 15: 0x82fdd747 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x53ee: skipped 000754 1784186823.659794 fallocate 0x033d9a thru 0x03f767 (0xb9ce bytes) 000755 1784186823.692205 mapwrite 0x003c1a thru 0x00a51f (0x6906 bytes) 000756 1784186823.729262 mapread 0x035954 thru 0x036183 (0x830 bytes) 000757 1784186823.745265 trunc from 0x03f768 to 0x00446c (0x3b2fb bytes) 000758 1784186823.769834 write 0x03c379 thru 0x03ffff (0x3c87 bytes) 000759 1784186823.770876 trunc from 0x040000 to 0x002564 (0x3da9b bytes) 000760 1784186823.789083 mapwrite 0x01ec16 thru 0x021569 (0x2954 bytes) 000761 1784186823.835095 fallocate 0x02c191 thru 0x030ac3 (0x4933 bytes) 000762 1784186823.840754 trunc from 0x01a5b5 to 0x02976d (0xf1b9 bytes) 000763 1784186823.847017 punch from 0x01d71c to 0x020899 (0x317e bytes) 000764 1784186823.853265 mapwrite 0x022f12 thru 0x02ccec (0x9ddb bytes) 000765 1784186823.891753 trunc from 0x030ac4 to 0x01f0f6 (0x119cd bytes) 000766 1784186823.907155 write 0x02e0f8 thru 0x0364c8 (0x83d1 bytes) 000767 1784186823.914497 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x364c9): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364c9): Mirror 5: 0xea291f1b (stale) Mirror 6: 0xea291f1b (stale) Mirror 11: 0xea291f1b (stale) Mirror 12: 0x4c0f22f9 (stale) Mirror 14: 0xd1f61556 (stale) Mirror 15: 0xdd9e31d5 (stale) CRC-32 checksum value for chunk [0, 0x364c9): Mirror 1: 0xb91e72c6 Mirror 16: 0xb91e72c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364c9: skipped 000768 1784186824.788112 read 0x01dc50 thru 0x02afc9 (0xd37a bytes) 000770 1784186824.813482 punch from 0x0142de to 0x02098d (0xc6b0 bytes) 000771 1784186824.862226 mapread 0x009bc0 thru 0x01098f (0x6dd0 bytes) 000772 1784186824.878735 read 0x01ea8a thru 0x02df01 (0xf478 bytes) 000774 1784186824.889932 trunc from 0x0364c9 to 0x0093f9 (0x2d0cf bytes) 000775 1784186824.915563 mapread 0x00456e thru 0x0093f9 (0x4e8c bytes) 000776 1784186824.916835 mapwrite 0x00b2e4 thru 0x018864 (0xd581 bytes) 000777 1784186825.015209 read 0x007e41 thru 0x00e700 (0x68c0 bytes) 000779 1784186825.021950 read 0x00d679 thru 0x018864 (0xb1ec bytes) 000780 1784186825.022838 write 0x03a1ad thru 0x03ffff (0x5e53 bytes) 000781 1784186825.024830 mapread 0x0184c8 thru 0x024ca3 (0xc7dc bytes) 000782 1784186825.033242 fallocate 0x0343b0 thru 0x038a9c (0x46ed bytes) 000783 1784186825.037835 read 0x008dd8 thru 0x00dd02 (0x4f2b bytes) 000784 1784186825.038770 punch from 0x030ec3 to 0x0376d2 (0x6810 bytes) 000785 1784186825.043157 fallocate 0x0324b9 thru 0x03cbd5 (0xa71d bytes) 000786 1784186825.047712 read 0x03856a thru 0x03ffff (0x7a96 bytes) 000787 1784186825.048348 mapwrite 0x0045fc thru 0x013830 (0xf235 bytes) 000788 1784186825.069337 trunc from 0x00cc9f to 0x018fae (0xc310 bytes) 000789 1784186825.074109 trunc from 0x040000 to 0x02bbd5 (0x1442a bytes) 000790 1784186825.088064 fallocate 0x01da91 thru 0x02c04a (0xe5ba bytes) 000791 1784186825.094038 trunc from 0x00910c to 0x017651 (0xe546 bytes) 000792 1784186825.098900 trunc from 0x01d606 to 0x02199e (0x4399 bytes) 000793 1784186825.103929 read 0x006ca4 thru 0x0140d5 (0xd432 bytes) 000794 1784186825.113948 mapread 0x01fbe5 thru 0x02bbd5 (0xbff1 bytes) 000795 1784186825.117206 punch from 0x01a660 to 0x025640 (0xafe1 bytes) 000797 1784186825.123279 punch from 0x028c98 to 0x02bbd5 (0x2f3e bytes) 000798 1784186825.128214 mapread 0x011793 thru 0x011d37 (0x5a5 bytes) 000799 1784186825.129542 trunc from 0x002039 to 0x003e95 (0x1e5d bytes) 000800 1784186825.136367 punch from 0x01d045 to 0x01efbc (0x1f78 bytes) 000801 1784186825.143061 mapread 0x02af84 thru 0x02bbd5 (0xc52 bytes) 000802 1784186825.157252 trunc from 0x016e3f to 0x01e5f7 (0x77b9 bytes) 000803 1784186825.162613 mapread 0x01bd9f thru 0x01da72 (0x1cd4 bytes) 000804 1784186825.178113 mapread 0x022680 thru 0x022ea7 (0x828 bytes) 000805 1784186825.184334 fallocate 0x0128ab thru 0x012af6 (0x24c bytes) 000806 1784186825.192809 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2bbd6): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bbd6): Mirror 5: 0x69bc722 (stale) Mirror 6: 0x69bc722 (stale) Mirror 11: 0x69bc722 (stale) Mirror 12: 0x5df1d471 (stale) Mirror 14: 0xa256c4f (stale) Mirror 15: 0xe59690fa (stale) Mirror 16: 0xe59690fa (stale) CRC-32 checksum value for chunk [0, 0x2bbd6): Mirror 1: 0x5e83c998 Mirror 17: 0x5e83c998 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bbd6: skipped 000808 1784186825.930858 fallocate 0x031229 thru 0x03a16d (0x8f45 bytes) 000809 1784186825.969735 trunc from 0x03a16e to 0x027a42 (0x1272b bytes) 000810 1784186826.026111 punch from 0x001253 to 0x010e1d (0xfbcb bytes) 000811 1784186826.032001 punch from 0x0094a1 to 0x015393 (0xbef3 bytes) 000812 1784186826.036812 trunc from 0x0269a6 to 0x027a42 (0x109d bytes) 000813 1784186826.052506 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27a43): Mirror 5: 0xa6b8546 (stale) Mirror 6: 0xa6b8546 (stale) Mirror 11: 0xa6b8546 (stale) Mirror 12: 0x659f9453 (stale) Mirror 14: 0xdd8c9b1e (stale) Mirror 15: 0x77388ca1 (stale) Mirror 17: 0x137607ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a43): Mirror 5: 0xa6b8546 (stale) Mirror 6: 0xa6b8546 (stale) Mirror 11: 0xa6b8546 (stale) Mirror 12: 0x659f9453 (stale) Mirror 14: 0xdd8c9b1e (stale) Mirror 15: 0x77388ca1 (stale) Mirror 17: 0x137607ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a43: skipped 000814 1784186826.745512 fallocate 0x00a3dd thru 0x00c101 (0x1d25 bytes) 000816 1784186826.758228 punch from 0x00898e to 0x010d0a (0x837d bytes) 000817 1784186826.762514 write 0x025b59 thru 0x026a57 (0xeff bytes) 000818 1784186826.785953 punch from 0x0047a6 to 0x013642 (0xee9d bytes) 000819 1784186826.797074 fallocate 0x0234da thru 0x024fc6 (0x1aed bytes) 000821 1784186826.810949 trunc from 0x027a43 to 0x00cb3b (0x1af07 bytes) 000822 1784186826.836983 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xcb3c): Mirror 6: 0xf53db654 (stale) Mirror 11: 0xf53db654 (stale) Mirror 12: 0x252dca3f (stale) Mirror 14: 0xb42cae54 (stale) Mirror 15: 0xb42cae54 (stale) Mirror 17: 0x2e1a99e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcb3c): Mirror 6: 0xf53db654 (stale) Mirror 11: 0xf53db654 (stale) Mirror 12: 0x252dca3f (stale) Mirror 14: 0xb42cae54 (stale) Mirror 15: 0xb42cae54 (stale) Mirror 17: 0x2e1a99e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcb3c: skipped 000823 1784186827.202007 mapwrite 0x032a35 thru 0x036889 (0x3e55 bytes) 000824 1784186827.241689 mapwrite 0x019f51 thru 0x01f1eb (0x529b bytes) 000825 1784186827.279243 read 0x01a6e1 thru 0x027383 (0xcca3 bytes) 000826 1784186827.287521 mapwrite 0x0130fa thru 0x02064f (0xd556 bytes) 000828 1784186827.339437 trunc from 0x03688a to 0x02ffc4 (0x68c5 bytes) 000829 1784186827.358901 trunc from 0x001115 to 0x00d1db (0xc0c7 bytes) 000830 1784186827.364964 write 0x02a55f thru 0x02d0e0 (0x2b82 bytes) 000831 1784186827.371133 trunc from 0x011e75 to 0x020490 (0xe61c bytes) 000832 1784186827.376756 fallocate 0x00d328 thru 0x00eba0 (0x1879 bytes) 000833 1784186827.382016 mapwrite 0x027e3b thru 0x029cef (0x1eb5 bytes) 000834 1784186827.408968 MIRROR_RESYNC 7 mirrors 000835 1784186829.064946 mapwrite 0x02423a thru 0x027550 (0x3317 bytes) 000836 1784186829.141451 trunc from 0x02eed8 to 0x02ffc4 (0x10ed bytes) 000837 1784186829.146132 punch from 0x0080b4 to 0x0178e4 (0xf831 bytes) 000838 1784186829.155390 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2ffc5): Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffc5): Mirror 6: 0x937644ec (stale) Mirror 11: 0x937644ec (stale) Mirror 12: 0x937644ec (stale) Mirror 14: 0x937644ec (stale) Mirror 15: 0x937644ec (stale) Mirror 17: 0x937644ec (stale) CRC-32 checksum value for chunk [0, 0x2ffc5): Mirror 1: 0x42620575 Mirror 18: 0x42620575 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffc5: skipped 000839 1784186830.060826 trunc from 0x02ffc5 to 0x02073f (0xf885 bytes) 000840 1784186830.118160 trunc from 0x020740 to 0x00abf4 (0x15b4b bytes) 000841 1784186830.143063 write 0x019a52 thru 0x0296c3 (0xfc72 bytes) 000842 1784186830.145014 trunc from 0x0296c4 to 0x02880e (0xeb5 bytes) 000843 1784186830.164088 write 0x026c39 thru 0x02afd4 (0x439c bytes) 000844 1784186830.170730 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2afd5): Mirror 6: 0x71123df (stale) Mirror 11: 0x71123df (stale) Mirror 12: 0x71123df (stale) Mirror 15: 0x71123df (stale) Mirror 17: 0x71123df (stale) Mirror 18: 0x29c7d291 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2afd5): Mirror 6: 0x71123df (stale) Mirror 11: 0x71123df (stale) Mirror 12: 0x71123df (stale) Mirror 15: 0x71123df (stale) Mirror 17: 0x71123df (stale) Mirror 18: 0x29c7d291 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2afd5: skipped 000845 1784186830.635812 trunc from 0x02ac2d to 0x02afd4 (0x3a8 bytes) 000846 1784186830.648987 mapwrite 0x02695f thru 0x02c015 (0x56b7 bytes) 000847 1784186830.714053 trunc from 0x01b6a4 to 0x01fe87 (0x47e4 bytes) 000848 1784186830.720977 trunc from 0x011a72 to 0x015764 (0x3cf3 bytes) 000849 1784186830.730889 MIRROR_RESYNC 7 mirrors 000850 1784186831.923220 mapread 0x0036a6 thru 0x0100f4 (0xca4f bytes) 000851 1784186831.940913 fallocate 0x02b9f8 thru 0x0371cb (0xb7d4 bytes) 000853 1784186831.969950 trunc from 0x02c016 to 0x01e4fc (0xdb19 bytes) 000854 1784186831.991899 write 0x037c90 thru 0x03df09 (0x627a bytes) 000855 1784186831.994312 mapwrite 0x01add0 thru 0x01d779 (0x29aa bytes) 000856 1784186832.018438 trunc from 0x03df0a to 0x002bd4 (0x3b335 bytes) 000857 1784186832.043913 read 0x00129e thru 0x002bd4 (0x1937 bytes) 000858 1784186832.050093 write 0x035008 thru 0x03ed1b (0x9d14 bytes) 000859 1784186832.052557 trunc from 0x0254a9 to 0x027895 (0x23ed bytes) 000860 1784186832.063345 trunc from 0x03ed1c to 0x027fdd (0x16d3e bytes) 000861 1784186832.076907 trunc from 0x0234b6 to 0x027fdd (0x4b28 bytes) 000862 1784186832.084659 read 0x01c0e2 thru 0x01c195 (0xb4 bytes) 000863 1784186832.099784 MIRROR_RESYNC 7 mirrors 000864 1784186832.915175 trunc from 0x019f07 to 0x01c6c4 (0x27be bytes) 000865 1784186832.955519 fallocate 0x023a7b thru 0x031fa5 (0xe52b bytes) 000866 1784186832.961807 trunc from 0x023839 to 0x027fdd (0x47a5 bytes) 000867 1784186832.966280 trunc from 0x027f2c to 0x027fdd (0xb2 bytes) 000868 1784186832.973519 mapwrite 0x01dc38 thru 0x021c4b (0x4014 bytes) 000869 1784186833.021886 fallocate 0x01a0e2 thru 0x01de19 (0x3d38 bytes) 000871 1784186833.035039 trunc from 0x027fde to 0x0050ef (0x22eee bytes) 000873 1784186833.096024 fallocate 0x02e3a2 thru 0x03d89f (0xf4fe bytes) 000875 1784186833.104279 fallocate 0x007796 thru 0x00cf69 (0x57d4 bytes) 000876 1784186833.111599 read 0x00a6d7 thru 0x00cf69 (0x2893 bytes) 000877 1784186833.118255 write 0x004f51 thru 0x012ace (0xdb7e bytes) 000878 1784186833.140466 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12acf): Mirror 6: 0xc8cc3293 (stale) Mirror 11: 0xc8cc3293 (stale) Mirror 15: 0xc8cc3293 (stale) Mirror 17: 0xc8cc3293 (stale) Mirror 18: 0xc8cc3293 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12acf): Mirror 6: 0xc8cc3293 (stale) Mirror 11: 0xc8cc3293 (stale) Mirror 15: 0xc8cc3293 (stale) Mirror 17: 0xc8cc3293 (stale) Mirror 18: 0xc8cc3293 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12acf: skipped 000879 1784186833.551141 trunc from 0x012acf to 0x01caa7 (0x9fd9 bytes) 000880 1784186833.574414 fallocate 0x026858 thru 0x02781f (0xfc8 bytes) 000881 1784186833.581410 write 0x009ab4 thru 0x00b3c8 (0x1915 bytes) 000882 1784186833.589852 mapread 0x0057e1 thru 0x005a9e (0x2be bytes) 000883 1784186833.598732 MIRROR_RESYNC 6 mirrors 000884 1784186834.530317 punch from 0x01b62c to 0x01caa7 (0x147c bytes) 000886 1784186834.567822 write 0x013647 thru 0x01a654 (0x700e bytes) 000887 1784186834.577181 mapwrite 0x0216d3 thru 0x028cd8 (0x7606 bytes) 000888 1784186834.635787 fallocate 0x0070db thru 0x0122de (0xb204 bytes) 000889 1784186834.640437 mapread 0x0209aa thru 0x022b01 (0x2158 bytes) 000890 1784186834.642519 read 0x00c23f thru 0x01acb1 (0xea73 bytes) 000891 1784186834.653418 write 0x0231b9 thru 0x031c83 (0xeacb bytes) 000892 1784186834.656318 read 0x021449 thru 0x02c20e (0xadc6 bytes) 000893 1784186834.657418 punch from 0x02bb29 to 0x031c83 (0x615b bytes) 000894 1784186834.674132 fallocate 0x032dde thru 0x03750b (0x472e bytes) 000895 1784186834.679762 trunc from 0x00c004 to 0x012af5 (0x6af2 bytes) 000896 1784186834.686163 trunc from 0x03750c to 0x037dd8 (0x8cd bytes) 000897 1784186834.703183 read 0x01fcb4 thru 0x022a0e (0x2d5b bytes) 000898 1784186834.704098 mapwrite 0x02ec28 thru 0x0352d3 (0x66ac bytes) 000899 1784186834.735309 mapwrite 0x009cf9 thru 0x0151d2 (0xb4da bytes) 000900 1784186834.758867 trunc from 0x037dd9 to 0x030b32 (0x72a6 bytes) 000901 1784186834.774979 fallocate 0x01d5fc thru 0x028d01 (0xb706 bytes) 000902 1784186834.780511 write 0x025758 thru 0x028de6 (0x368f bytes) 000903 1784186834.781656 mapread 0x00abf6 thru 0x0146ac (0x9ab7 bytes) 000904 1784186834.787360 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30b33): Mirror 6: 0x79e48179 (stale) Mirror 15: 0x79e48179 (stale) Mirror 17: 0x79e48179 (stale) Mirror 18: 0x79e48179 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30b33): Mirror 6: 0x79e48179 (stale) Mirror 15: 0x79e48179 (stale) Mirror 17: 0x79e48179 (stale) Mirror 18: 0x79e48179 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30b33: skipped 000905 1784186835.107437 mapread 0x00654f thru 0x015dcd (0xf87f bytes) 000906 1784186835.134124 punch from 0x01352c to 0x013c56 (0x72b bytes) 000907 1784186835.142211 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30b33): Mirror 15: 0x79e48179 (stale) Mirror 17: 0x79e48179 (stale) Mirror 18: 0x79e48179 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30b33): Mirror 15: 0x79e48179 (stale) Mirror 17: 0x79e48179 (stale) Mirror 18: 0x79e48179 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30b33: skipped 000908 1784186835.427098 mapread 0x02aac8 thru 0x030b32 (0x606b bytes) 000909 1784186835.446326 trunc from 0x017d50 to 0x0277a3 (0xfa54 bytes) 000910 1784186835.451458 trunc from 0x030b33 to 0x02af44 (0x5bee bytes) 000911 1784186835.483043 fallocate 0x01042c thru 0x01c61e (0xc1f3 bytes) 000912 1784186835.499456 MIRROR_RESYNC 4 mirrors 000913 1784186836.429342 mapwrite 0x002fdb thru 0x01187c (0xe8a2 bytes) 000914 1784186836.611290 read 0x029ca9 thru 0x02af44 (0x129c bytes) 000915 1784186836.622033 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2af45): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2af45): Mirror 15: 0xfeef42d8 (stale) Mirror 17: 0xfeef42d8 (stale) Mirror 18: 0xfeef42d8 (stale) CRC-32 checksum value for chunk [0, 0x2af45): Mirror 1: 0xc7b30f73 Mirror 19: 0xc7b30f73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2af45: skipped 000916 1784186837.841755 read 0x008a1b thru 0x010c06 (0x81ec bytes) 000917 1784186837.897803 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2af45): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2af45): Mirror 15: 0xfeef42d8 (stale) Mirror 17: 0xfeef42d8 (stale) Mirror 18: 0xfeef42d8 (stale) CRC-32 checksum value for chunk [0, 0x2af45): Mirror 1: 0xc7b30f73 Mirror 19: 0xc7b30f73 Mirror 20: 0xc7b30f73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2af45: skipped 000918 1784186839.108505 trunc from 0x02af45 to 0x030ae4 (0x5ba0 bytes) 000919 1784186839.185064 punch from 0x029506 to 0x02ee2c (0x5927 bytes) 000920 1784186839.200095 fallocate 0x009eba thru 0x00f27f (0x53c6 bytes) 000921 1784186839.242896 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x30ae5): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30ae5): Mirror 15: 0xae67ac53 (stale) Mirror 17: 0xae67ac53 (stale) Mirror 18: 0xae67ac53 (stale) Mirror 19: 0x74039e8c (stale) Mirror 20: 0x74039e8c (stale) CRC-32 checksum value for chunk [0, 0x30ae5): Mirror 1: 0x489b595 Mirror 21: 0x489b595 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30ae5: skipped 000922 1784186841.336840 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30ae5): Mirror 15: 0xae67ac53 (stale) Mirror 17: 0xae67ac53 (stale) Mirror 18: 0xae67ac53 (stale) Mirror 19: 0x74039e8c (stale) Mirror 20: 0x74039e8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30ae5): Mirror 15: 0xae67ac53 (stale) Mirror 17: 0xae67ac53 (stale) Mirror 18: 0xae67ac53 (stale) Mirror 19: 0x74039e8c (stale) Mirror 20: 0x74039e8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30ae5: skipped 000923 1784186842.447930 mapwrite 0x001b03 thru 0x007364 (0x5862 bytes) 000924 1784186842.608739 read 0x01e3fd thru 0x02403f (0x5c43 bytes) 000925 1784186842.638223 write 0x019ebe thru 0x01e062 (0x41a5 bytes) 000926 1784186842.670909 read 0x01ac8d thru 0x021028 (0x639c bytes) 000927 1784186842.676910 mapread 0x00c050 thru 0x01af87 (0xef38 bytes) 000929 1784186842.718179 read 0x023d9a thru 0x030ae4 (0xcd4b bytes) 000930 1784186842.723253 trunc from 0x030ae5 to 0x01b427 (0x156bd bytes) 000932 1784186842.766282 read 0x018b18 thru 0x01b427 (0x2910 bytes) 000933 1784186842.769298 write 0x013156 thru 0x0209dc (0xd887 bytes) 000934 1784186842.777924 write 0x029b8d thru 0x02be93 (0x2307 bytes) 000935 1784186842.787159 read 0x012000 thru 0x015891 (0x3892 bytes) 000936 1784186842.790279 trunc from 0x02be94 to 0x01af1a (0x10f79 bytes) 000937 1784186842.855861 MIRROR_RESYNC 6 mirrors 000938 1784186844.505219 read 0x0170ea thru 0x01af1a (0x3e31 bytes) 000939 1784186844.516111 trunc from 0x002c7b to 0x00e05d (0xb3e3 bytes) 000941 1784186844.592560 mapwrite 0x00653c thru 0x006905 (0x3ca bytes) 000943 1784186844.630812 trunc from 0x01af1b to 0x004579 (0x169a1 bytes) 000944 1784186844.670154 fallocate 0x02cdfa thru 0x031cc5 (0x4ecc bytes) 000945 1784186844.685981 write 0x01e1b1 thru 0x023340 (0x5190 bytes) 000946 1784186844.687359 trunc from 0x023341 to 0x0059d4 (0x1d96c bytes) 000947 1784186844.708998 fallocate 0x01f812 thru 0x027495 (0x7c84 bytes) 000948 1784186844.721763 trunc from 0x0050f2 to 0x00c8fa (0x7809 bytes) 000949 1784186844.730319 write 0x006a1d thru 0x00f0cf (0x86b3 bytes) 000950 1784186844.747322 write 0x003add thru 0x0105bb (0xcadf bytes) 000951 1784186844.770720 trunc from 0x027496 to 0x0089f6 (0x1ea9f bytes) 000952 1784186844.803850 read 0x001a5a thru 0x0089f6 (0x6f9d bytes) 000953 1784186844.816964 mapread 0x006af4 thru 0x0089f6 (0x1f03 bytes) 000954 1784186844.819365 punch from 0x000569 to 0x0089f6 (0x848e bytes) 000955 1784186844.851693 trunc from 0x003b47 to 0x004a78 (0xf32 bytes) 000956 1784186844.863286 mapread 0x003044 thru 0x0089f6 (0x59b3 bytes) 000957 1784186844.896433 mapread 0x0072cc thru 0x0089f6 (0x172b bytes) 000958 1784186844.897431 punch from 0x000998 to 0x0089f6 (0x805f bytes) 000959 1784186844.903194 mapwrite 0x00248d thru 0x00e69d (0xc211 bytes) 000960 1784186845.039455 read 0x00bb41 thru 0x00e69d (0x2b5d bytes) 000961 1784186845.040256 fallocate 0x03fc47 thru 0x03ffff (0x3b9 bytes) 000963 1784186845.050887 trunc from 0x013aac to 0x013ca3 (0x1f8 bytes) 000964 1784186845.058717 punch from 0x01eea2 to 0x01f97c (0xadb bytes) 000965 1784186845.069204 trunc from 0x01194f to 0x020d5a (0xf40c bytes) 000966 1784186845.080986 punch from 0x00f949 to 0x016216 (0x68ce bytes) 000967 1784186845.087870 mapwrite 0x00627b thru 0x012f1a (0xcca0 bytes) 000968 1784186845.156064 punch from 0x0106c3 to 0x01542d (0x4d6b bytes) 000969 1784186845.172479 trunc from 0x040000 to 0x03ee47 (0x11b8 bytes) 000970 1784186845.234364 mapread 0x03e6a4 thru 0x03ee47 (0x7a4 bytes) 000971 1784186845.253881 trunc from 0x0388a1 to 0x03ee47 (0x65a7 bytes) 000972 1784186845.270045 read 0x008108 thru 0x017140 (0xf039 bytes) 000973 1784186845.290340 write 0x02cb97 thru 0x033f2a (0x7394 bytes) 000974 1784186845.291995 fallocate 0x016178 thru 0x01decc (0x7d55 bytes) 000975 1784186845.314937 write 0x02156f thru 0x02ae1e (0x98b0 bytes) 000976 1784186845.321315 mapread 0x01c949 thru 0x01ccd2 (0x38a bytes) 000977 1784186845.330138 mapread 0x0050e4 thru 0x00670d (0x162a bytes) 000978 1784186845.337130 trunc from 0x030a7e to 0x037c9b (0x721e bytes) 000979 1784186845.393868 trunc from 0x03ee48 to 0x033d3b (0xb10c bytes) 000980 1784186845.436845 trunc from 0x0223e2 to 0x027168 (0x4d87 bytes) 000981 1784186845.457110 MIRROR_RESYNC 6 mirrors 000982 1784186848.742665 trunc from 0x033d3c to 0x03ad59 (0x701e bytes) 000983 1784186848.826282 write 0x03697e thru 0x03ffff (0x9682 bytes) 000984 1784186848.837564 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x13ec9433 (stale) Mirror 17: 0x13ec9433 (stale) Mirror 18: 0x13ec9433 (stale) Mirror 19: 0x13ec9433 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x13ec9433 (stale) Mirror 17: 0x13ec9433 (stale) Mirror 18: 0x13ec9433 (stale) Mirror 19: 0x13ec9433 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000985 1784186849.421144 trunc from 0x040000 to 0x012345 (0x2dcba bytes) 000986 1784186849.441478 fallocate 0x00d3bf thru 0x014862 (0x74a4 bytes) 000987 1784186849.450382 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14863): Mirror 15: 0x4645b3d5 (stale) Mirror 17: 0x4645b3d5 (stale) Mirror 19: 0x4645b3d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14863): Mirror 15: 0x4645b3d5 (stale) Mirror 17: 0x4645b3d5 (stale) Mirror 19: 0x4645b3d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14863: skipped 000988 1784186849.771731 fallocate 0x01ea79 thru 0x02b946 (0xcece bytes) 000990 1784186849.784846 trunc from 0x014863 to 0x03e22e (0x299cc bytes) 000991 1784186849.804185 write 0x02bf04 thru 0x02d81c (0x1919 bytes) 000992 1784186849.814879 trunc from 0x003732 to 0x00a1d1 (0x6aa0 bytes) 000994 1784186849.822116 trunc from 0x03e22f to 0x00683e (0x379f0 bytes) 000995 1784186849.852353 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x683f): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x683f): Mirror 15: 0x3a4aa953 (stale) Mirror 17: 0x3a4aa953 (stale) Mirror 19: 0x3a4aa953 (stale) CRC-32 checksum value for chunk [0, 0x683f): Mirror 1: 0x8e33ee3 Mirror 20: 0x8e33ee3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x683f: skipped 000996 1784186850.872500 mapwrite 0x01be41 thru 0x01ec45 (0x2e05 bytes) 000997 1784186851.015594 fallocate 0x0017c1 thru 0x001f52 (0x792 bytes) 000998 1784186851.025928 fallocate 0x0358bc thru 0x036817 (0xf5c bytes) All operations completed A-OK! PASS 70a (92s) == sanity-flr test 71: check mirror extend parent fid ==== 03:27:38 (1784186858) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x19c:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 03:27:50 (1784186870) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119217 s, 17.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701542 s, 14.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13b:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 03:27:58 (1784186878) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4588864 .. Extending file size to 6889088 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7830784 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8397536 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8408096 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7880960 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8039904 .. Extending file size to 8636608 .. Extending file size to 9043744 .. Waiting 61357 61358 61361 61362 61364 done 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) 192.168.201.138@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:29:22 (1784186962) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2202272 .. Extending file size to 6974848 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8104384 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 2830240 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4375616 .. Extending file size to 5213280 .. Extending file size to 7474784 .. Extending file size to 9343872 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7705088 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8463840 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8896768 .. Waiting 62701 62702 62703 62705 62706 done 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) 192.168.201.138@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:30:42 (1784187042) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2396 1268908 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1269240 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25804 3581216 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17836 3589184 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43640 7170400 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2396 1268908 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2064 1269240 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25804 3581216 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17836 3589184 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43640 7170400 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116218 s, 18.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.275556 s, 7.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x563a1b65 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:31:04 (1784187064) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143431 s, 14.6 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 03:31:13 (1784187073) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa6:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x153:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:31:19 (1784187079) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0696332 s, 30.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x154:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:31:26 (1784187086) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110599 s, 19.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x155:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x156:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x155:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x156:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:31:35 (1784187095) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11683 s, 18.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x157:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x157:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:32:09 (1784187129) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932028 s, 11.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x224:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x227:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x226:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x224:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x227:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x226:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x225:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:33:05 (1784187185) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127992 s, 16.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x158:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x158:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (49s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:33:54 (1784187234) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg138-server: Pool lustre.test_204f created Adding targets to pool oleg138-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg138-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192812 s, 10.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x228:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x226:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x228:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x226:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg138-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg138-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg138-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:35:15 (1784187315) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:35:24 (1784187324) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:35:33 (1784187333) oleg138-server: Pool lustre.test_206 created Adding targets to pool oleg138-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg138-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg138-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg138-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg138-server: Pool lustre.test_206 destroyed PASS 206 (32s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:36:05 (1784187365) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904317 s, 11.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318646 s, 32.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xa36423be Mirror 3: 0xa36423be PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:36:16 (1784187376) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:36:21 (1784187381) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:36:26 (1784187386) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0870424 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb00000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xa20000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xa20000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xa20000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xca0000 Read this (4 bytes): BBBB 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:37:05 (1784187425) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0176942 s, 463 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:37:13 (1784187433) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745539 s, 14.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc7 lmm_fid: [0x200000405:0xc7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc7 lmm_fid: [0x200000405:0xc7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x163:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611335 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-16 03:37:15.000000000 -0400 Modify: 2026-07-16 03:37:15.000000000 -0400 Change: 2026-07-16 03:37:17.000000000 -0400 Birth: 2026-07-16 03:37:16.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:37:23 (1784187443) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482214 s, 21.7 MB/s oleg138-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg138-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg138-server: O/280000401/d4/164: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:37:48 (1784187468) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.23329 s, 176 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0190569 s, 215 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 16 03:37 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xcc:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 16 03:37 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 03:37:56 (1784187476) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 03:38:03 (1784187483) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1833 sec =========== 03:38:13 (1784187493) === sanity-flr: start cleanup 03:38:16 (1784187496) === === sanity-flr: finish cleanup 03:38:27 (1784187507) ===