-----============= acceptance-small: sanity-flr ============----- Thu Apr 30 19:52:50 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg304-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 19:53:01 (1777593181) === oleg304-client.virtnet: executing check_config_client /mnt/lustre oleg304-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg304-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9af3514cb000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9af3514cb000.idle_timeout=debug disable quota as required oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 19:53:18 (1777593198) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8206] == sanity-flr test 0a: lfs mirror create with -N option == 19:53:19 (1777593199) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:53:26 (1777593206) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:53:28 (1777593208) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:53:29 (1777593209) PASS 0d (9s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:53:38 (1777593218) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:53:40 (1777593220) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 19:53:41 (1777593221) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9af3514cb000.stats=clear osc.lustre-OST0001-osc-ffff9af3514cb000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:53:44 (1777593224) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:53:49 (1777593229) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:53:52 (1777593232) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:53:54 (1777593234) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:53:57 (1777593237) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 19:54:00 (1777593240) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:54:03 (1777593243) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:54:06 (1777593246) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:54:09 (1777593249) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:54:13 (1777593253) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:54:16 (1777593256) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.443585 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341799 s, 30.7 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:54:19 (1777593259) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.114693 s, 91.4 MB/s PASS 22 (3s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:54:22 (1777593262) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:54:26 (1777593266) Stopping /mnt/lustre-ost1 (opts:) on oleg304-server oleg304-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9af3514cb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9af3514cb000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg304-server: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg304-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg304-server oleg304-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9af3514cb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9af3514cb000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg304-server: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg304-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (32s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:54:58 (1777593298) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.120253 s, 43.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg304-server oleg304-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9af3514cb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9af3514cb000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg304-server: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg304-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (20s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:55:18 (1777593318) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg304-server oleg304-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9af3514cb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9af3514cb000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg304-server: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg304-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg304-server oleg304-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9af3514cb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9af3514cb000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg304-server: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg304-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (35s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:55:54 (1777593354) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.44031 s, 36.4 MB/s -rw-r--r-- 1 root root 52428800 Apr 30 19:55 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.469s user 0m0.002s sys 0m0.163s Stopping /mnt/lustre-ost1 (opts:) on oleg304-server oleg304-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9af3514cb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9af3514cb000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.476s user 0m0.001s sys 0m0.166s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg304-server: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg304-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg304-server oleg304-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9af3514cb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9af3514cb000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.424s user 0m0.000s sys 0m0.163s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg304-server: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg304-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9af3514cb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9af3514cb000.ost_server_uuid in FULL state after 0 sec PASS 33b (37s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:56:32 (1777593392) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:56:33 (1777593393) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:56:35 (1777593395) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 19:56:37 (1777593397) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 19:56:40 (1777593400) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.258574 s, 48.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:56:46 (1777593406) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160798 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148971 s, 70.4 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:56:49 (1777593409) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:56:53 (1777593413) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000686277 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000673106 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000721744 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000930068 s, 1.1 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 19:56:57 (1777593417) create test files with size 15731680 .. 6c486323167c9eefdd387f109e48d72a - af30bca4fca57c2eb0e68963c95d86c7 - 282b2405f0e97488ee6716e5558ff350 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg304-client.virtnet /mnt/lustre (opts:) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre PASS 37 (10s) == sanity-flr test 38: resync ============================ 19:57:07 (1777593427) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:57:13 (1777593433) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:57:16 (1777593436) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504396 s, 20.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150272 s, 69.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 19:57:20 (1777593440) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0546657 s, 38.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.172083 s, 24.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.067701 s, 31.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (7s) == sanity-flr test 42: lfs mirror verify ================= 19:57:27 (1777593447) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 19:57:29 (1777593449) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:57:30 (1777593450) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018459 s, 56.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 19:57:34 (1777593454) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.127444 s, 24.7 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:57:38 (1777593458) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:57:43 (1777593463) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.12441 s, 84.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg304-client.virtnet /mnt/lustre (opts:) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:57:56 (1777593476) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9af34997c800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.175396 s, 59.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg304-client.virtnet /mnt/lustre (opts:) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9af348d2f800.xattr_cache=1 PASS 44e (9s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:58:04 (1777593484) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (18s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:58:22 (1777593502) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (18s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:58:40 (1777593520) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 19:58:41 (1777593521) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.101443 s, 31.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0643402 s, 48.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x766587ca (stale) Mirror 2: 0x766587ca (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: f55086c7213dbd09e556d5165e3a9270 - mirror 1 checksum: df779a2e24308f3d5a9cc1c4640b1c84 - mirror 2 checksum: f55086c7213dbd09e556d5165e3a9270 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 30 19:58:43 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 30 19:58:43 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f55086c7213dbd09e556d5165e3a9270 - mirror 1 checksum: df779a2e24308f3d5a9cc1c4640b1c84 - mirror 2 checksum: df779a2e24308f3d5a9cc1c4640b1c84 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 30 19:58:43 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 30 19:58:43 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 48 (5s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 19:58:46 (1777593526) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171996 s, 61.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:58:49 (1777593529) Failing mds1 on oleg304-server Stopping /mnt/lustre-mds1 (opts:) on oleg304-server 19:58:58 (1777593538) shut down facet: mds1 facet_host: oleg304-server facet_failover_host: oleg304-server Failover mds1 to oleg304-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-MDT0000 19:59:13 (1777593553) targets are mounted 19:59:14 (1777593554) facet_failover done oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191416 s, 54.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 50A (34s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:59:23 (1777593563) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0366194 s, 559 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0116536 s, 1.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:59:27 (1777593567) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.027436 s, 746 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00635069 s, 3.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:59:32 (1777593572) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00409219 s, 4.0 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:59:34 (1777593574) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0227823 s, 899 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:59:38 (1777593578) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00983306 s, 417 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0134259 s, 305 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:59:41 (1777593581) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (9s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:59:50 (1777593590) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:00:10 (1777593610) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:00:29 (1777593629) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:00:47 (1777593647) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 20:00:50 (1777593650) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57061 mirror split pid 57062 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57061 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57062 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 20:01:53 (1777593713) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 3714 000001 1777593714.572677 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x1a9b4 000002 1777593714.708264 fallocate 0x00d0e2 thru 0x01a9b3 (0xd8d2 bytes) 000003 1777593714.721251 mapread 0x008737 thru 0x00f88a (0x7154 bytes) 000004 1777593714.738410 MIRROR_RESYNC 2 mirrors 000005 1777593714.962160 trunc from 0x01766a to 0x01a9b3 (0x334a bytes) fallocating to largest ever: 0x25dfe 000006 1777593714.991522 fallocate 0x01e196 thru 0x025dfd (0x7c68 bytes) 000007 1777593714.996611 mapread 0x000c95 thru 0x005d2f (0x509b bytes) 000008 1777593715.022367 write 0x024c81 thru 0x03003f (0xb3bf bytes) 000009 1777593715.030742 punch from 0x011387 to 0x016886 (0x5500 bytes) truncating to largest ever: 0x3b09e 000010 1777593715.034439 trunc from 0x030040 to 0x03b09d (0xb05e bytes) 000011 1777593715.057444 mapread 0x02e721 thru 0x0379c5 (0x92a5 bytes) 000012 1777593715.063576 trunc from 0x03b09e to 0x015422 (0x25c7b bytes) 000014 1777593715.071436 punch from 0x004516 to 0x007067 (0x2b52 bytes) 000015 1777593715.074051 fallocate 0x0003ad thru 0x008bd7 (0x882b bytes) 000016 1777593715.076660 trunc from 0x015423 to 0x029209 (0x13de7 bytes) 000017 1777593715.083507 trunc from 0x02006d to 0x026e15 (0x6da9 bytes) 000018 1777593715.091328 MIRROR_RESYNC 2 mirrors 000019 1777593715.292006 fallocate 0x02cf65 thru 0x03ae80 (0xdf1c bytes) 000020 1777593715.317430 mapread 0x019e18 thru 0x023911 (0x9afa bytes) 000021 1777593715.332309 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2920a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2920a): Mirror 2: 0x5f60929b (stale) CRC-32 checksum value for chunk [0, 0x2920a): Mirror 1: 0x5f60929b Mirror 3: 0x5f60929b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2920a: skipped 000022 1777593715.674989 mapread 0x005117 thru 0x00db68 (0x8a52 bytes) fallocating to largest ever: 0x40000 000023 1777593715.696837 fallocate 0x032874 thru 0x03ffff (0xd78c bytes) 000024 1777593715.717552 mapread 0x00ee88 thru 0x01862c (0x97a5 bytes) 000025 1777593715.734901 MIRROR_RESYNC 3 mirrors 000026 1777593716.004160 mapwrite 0x036ce1 thru 0x03cbb5 (0x5ed5 bytes) 000027 1777593716.067711 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cbb6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cbb6): Mirror 2: 0x9a3a406b (stale) Mirror 3: 0x9a3a406b (stale) CRC-32 checksum value for chunk [0, 0x3cbb6): Mirror 1: 0xbfde3705 Mirror 4: 0xbfde3705 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cbb6: skipped 000028 1777593716.423468 trunc from 0x003175 to 0x01241a (0xf2a6 bytes) 000029 1777593716.446205 trunc from 0x021c59 to 0x0277be (0x5b66 bytes) 000030 1777593716.450446 write 0x00d6d9 thru 0x012d88 (0x56b0 bytes) 000031 1777593716.457755 mapread 0x028c28 thru 0x02d44c (0x4825 bytes) 000032 1777593716.465631 trunc from 0x03cbb6 to 0x012c5e (0x29f57 bytes) 000033 1777593716.497687 mapread 0x00f360 thru 0x012c5e (0x38ff bytes) 000034 1777593716.498740 write 0x00636f thru 0x008af8 (0x278a bytes) 000035 1777593716.506588 mapread 0x001280 thru 0x0044ec (0x326d bytes) 000036 1777593716.507678 fallocate 0x034eb6 thru 0x03ffff (0xb14a bytes) 000037 1777593716.514901 MIRROR_RESYNC 4 mirrors 000038 1777593717.008609 MIRROR_EXTEND to 5 mirrors 000039 1777593717.357919 trunc from 0x027bef to 0x02cb2d (0x4f3f bytes) 000040 1777593717.375172 write 0x008e04 thru 0x00f9bb (0x6bb8 bytes) 000041 1777593717.383168 mapread 0x030b8c thru 0x03e190 (0xd605 bytes) 000042 1777593717.390521 read 0x005dae thru 0x0082cb (0x251e bytes) 000043 1777593717.396762 MIRROR_RESYNC 5 mirrors 000044 1777593717.794681 trunc from 0x0247e9 to 0x029a90 (0x52a8 bytes) 000045 1777593717.813270 write 0x013445 thru 0x014b95 (0x1751 bytes) 000046 1777593717.819345 punch from 0x004077 to 0x0047fe (0x788 bytes) 000047 1777593717.822217 trunc from 0x01b97e to 0x0261af (0xa832 bytes) 000048 1777593717.824823 fallocate 0x0294d8 thru 0x038283 (0xedac bytes) 000049 1777593717.827979 write 0x0193ad thru 0x023b9d (0xa7f1 bytes) 000050 1777593717.835270 trunc from 0x02f4d5 to 0x031cc2 (0x27ee bytes) 000052 1777593717.838903 trunc from 0x03be46 to 0x03ffff (0x41ba bytes) 000053 1777593717.842205 mapwrite 0x02b7b7 thru 0x0341ff (0x8a49 bytes) 000055 1777593717.863371 write 0x013c7d thru 0x020161 (0xc4e5 bytes) 000056 1777593717.864787 fallocate 0x01c155 thru 0x0227f2 (0x669e bytes) 000057 1777593717.868005 write 0x022ee0 thru 0x02d70b (0xa82c bytes) 000058 1777593717.869236 fallocate 0x02fd55 thru 0x035242 (0x54ee bytes) 000059 1777593717.871864 trunc from 0x02f024 to 0x03d586 (0xe563 bytes) 000060 1777593717.874782 mapread 0x03021a thru 0x0323a3 (0x218a bytes) 000061 1777593717.880399 read 0x01ac06 thru 0x01d582 (0x297d bytes) 000062 1777593717.883891 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf6598ee1 (stale) Mirror 4: 0xf6598ee1 (stale) Mirror 5: 0xf6598ee1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf6598ee1 (stale) Mirror 4: 0xf6598ee1 (stale) Mirror 5: 0xf6598ee1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000065 1777593718.062419 write 0x03dc45 thru 0x03ffff (0x23bb bytes) 000066 1777593718.070172 punch from 0x01f725 to 0x02480e (0x50ea bytes) 000067 1777593718.073192 mapread 0x02d921 thru 0x030be2 (0x32c2 bytes) 000068 1777593718.084569 punch from 0x00e1f7 to 0x0137dc (0x55e6 bytes) 000069 1777593718.089629 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf6598ee1 (stale) Mirror 4: 0xf6598ee1 (stale) Mirror 5: 0xf6598ee1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x16dacd94 Mirror 6: 0x16dacd94 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000070 1777593718.475987 fallocate 0x023074 thru 0x023d09 (0xc96 bytes) 000071 1777593718.496354 mapwrite 0x02a2d0 thru 0x0332b0 (0x8fe1 bytes) 000072 1777593718.520573 trunc from 0x040000 to 0x005578 (0x3aa87 bytes) 000073 1777593718.543324 write 0x009f1f thru 0x014a95 (0xab77 bytes) 000074 1777593718.545045 read 0x0073d8 thru 0x00e2e2 (0x6f0b bytes) 000075 1777593718.549532 punch from 0x000beb to 0x00c99b (0xbdb1 bytes) 000076 1777593718.559470 fallocate 0x01b563 thru 0x027fb7 (0xca55 bytes) 000077 1777593718.564188 mapwrite 0x022cb3 thru 0x025e33 (0x3181 bytes) 000078 1777593718.577209 punch from 0x00548d to 0x005613 (0x187 bytes) 000079 1777593718.579906 read 0x003182 thru 0x003fef (0xe6e bytes) 000080 1777593718.584337 read 0x0049c2 thru 0x013998 (0xefd7 bytes) 000081 1777593718.585034 write 0x00483c thru 0x00de85 (0x964a bytes) 000082 1777593718.586334 write 0x02c69f thru 0x039dbc (0xd71e bytes) 000083 1777593718.590512 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x39dbd): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39dbd): Mirror 2: 0x84d58777 (stale) Mirror 4: 0x84d58777 (stale) Mirror 5: 0x84d58777 (stale) Mirror 6: 0xc28b213d (stale) CRC-32 checksum value for chunk [0, 0x39dbd): Mirror 1: 0xf8f4e8f8 Mirror 7: 0xf8f4e8f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39dbd: skipped 000084 1777593718.997411 write 0x03d890 thru 0x03ffff (0x2770 bytes) 000086 1777593719.018062 write 0x000afb thru 0x008d3d (0x8243 bytes) 000087 1777593719.023597 trunc from 0x040000 to 0x027692 (0x1896d bytes) 000088 1777593719.048362 trunc from 0x027693 to 0x00a652 (0x1d040 bytes) 000089 1777593719.059276 trunc from 0x00a0be to 0x00a652 (0x595 bytes) 000090 1777593719.063851 trunc from 0x007b17 to 0x00a652 (0x2b3c bytes) 000091 1777593719.073970 trunc from 0x00a653 to 0x0224ca (0x17e78 bytes) 000092 1777593719.083112 read 0x00bb21 thru 0x00fa7d (0x3f5d bytes) 000093 1777593719.090065 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x224cb): Mirror 4: 0x7324a30d (stale) Mirror 5: 0x7324a30d (stale) Mirror 6: 0x9828fc4c (stale) Mirror 7: 0x90315382 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x224cb): Mirror 4: 0x7324a30d (stale) Mirror 5: 0x7324a30d (stale) Mirror 6: 0x9828fc4c (stale) Mirror 7: 0x90315382 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x224cb: skipped 000094 1777593719.254961 punch from 0x00bde9 to 0x012fca (0x71e2 bytes) 000095 1777593719.257928 trunc from 0x016ece to 0x01a14e (0x3281 bytes) 000096 1777593719.261861 trunc from 0x0224cb to 0x02edc4 (0xc8fa bytes) 000097 1777593719.279731 trunc from 0x01608d to 0x022497 (0xc40b bytes) 000098 1777593719.286159 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2edc5): Mirror 4: 0xd4ef8d47 (stale) Mirror 6: 0x4efe6d32 (stale) Mirror 7: 0x102f91c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2edc5): Mirror 4: 0xd4ef8d47 (stale) Mirror 6: 0x4efe6d32 (stale) Mirror 7: 0x102f91c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2edc5: skipped 000099 1777593719.457961 write 0x023392 thru 0x02a2e4 (0x6f53 bytes) 000100 1777593719.466909 mapwrite 0x001559 thru 0x009874 (0x831c bytes) 000101 1777593719.491056 mapread 0x025907 thru 0x02edc4 (0x94be bytes) 000102 1777593719.495678 trunc from 0x02edc5 to 0x00add6 (0x23fee bytes) 000104 1777593719.505772 trunc from 0x00add7 to 0x0157fa (0xaa24 bytes) 000105 1777593719.514706 MIRROR_RESYNC 4 mirrors 000106 1777593719.849615 read 0x001a48 thru 0x007fb0 (0x6569 bytes) 000107 1777593719.858869 MIRROR_RESYNC 4 mirrors 000108 1777593720.034826 fallocate 0x02cc49 thru 0x035912 (0x8cca bytes) 000109 1777593720.053351 mapread 0x0055f5 thru 0x011004 (0xba10 bytes) 000110 1777593720.069006 punch from 0x027da9 to 0x02cf8a (0x51e2 bytes) 000111 1777593720.072687 read 0x01cbc6 thru 0x028867 (0xbca2 bytes) 000112 1777593720.079060 punch from 0x00cb5a to 0x019f70 (0xd417 bytes) 000113 1777593720.081629 mapread 0x01e0b4 thru 0x02aa61 (0xc9ae bytes) 000114 1777593720.094145 punch from 0x012bad to 0x015164 (0x25b8 bytes) 000115 1777593720.100129 mapwrite 0x01d6df thru 0x025b75 (0x8497 bytes) 000116 1777593720.124160 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35913): Mirror 4: 0xf1245f74 (stale) Mirror 6: 0xf1245f74 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35913): Mirror 4: 0xf1245f74 (stale) Mirror 6: 0xf1245f74 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35913: skipped 000117 1777593720.310438 trunc from 0x035913 to 0x016930 (0x1efe2 bytes) 000118 1777593720.333416 mapwrite 0x023364 thru 0x030847 (0xd4e4 bytes) 000119 1777593720.371138 write 0x01a277 thru 0x0276f9 (0xd483 bytes) 000121 1777593720.376640 trunc from 0x019440 to 0x0240a6 (0xac67 bytes) 000122 1777593720.386403 read 0x022b92 thru 0x02f4d5 (0xc944 bytes) 000123 1777593720.392260 trunc from 0x030848 to 0x03dd63 (0xd51c bytes) 000124 1777593720.401551 fallocate 0x0240b2 thru 0x0293db (0x532a bytes) 000125 1777593720.404570 write 0x02fbec thru 0x03629b (0x66b0 bytes) 000126 1777593720.408344 mapread 0x02df2f thru 0x03dce4 (0xfdb6 bytes) 000127 1777593720.413136 trunc from 0x03dd64 to 0x02d7a2 (0x105c1 bytes) 000128 1777593720.428746 trunc from 0x00c837 to 0x015a38 (0x9202 bytes) 000130 1777593720.435402 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d7a3): Mirror 4: 0x7160aded (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d7a3): Mirror 4: 0x7160aded (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d7a3: skipped 000131 1777593720.562602 write 0x0045ca thru 0x004cd5 (0x70c bytes) 000133 1777593720.572710 MIRROR_RESYNC 2 mirrors 000134 1777593720.840242 write 0x017c9e thru 0x026981 (0xece4 bytes) 000135 1777593720.865881 mapwrite 0x00fb13 thru 0x011928 (0x1e16 bytes) 000136 1777593720.883418 mapread 0x002670 thru 0x00fbcf (0xd560 bytes) 000137 1777593720.890974 trunc from 0x02d7a3 to 0x00eb30 (0x1ec72 bytes) 000138 1777593720.915659 fallocate 0x009e2b thru 0x00c6e4 (0x28ba bytes) 000139 1777593720.919817 fallocate 0x033b66 thru 0x03a582 (0x6a1d bytes) 000140 1777593720.923093 mapread 0x00e3d4 thru 0x015969 (0x7596 bytes) 000141 1777593720.935781 read 0x01bb97 thru 0x01e08b (0x24f5 bytes) 000142 1777593720.939078 mapwrite 0x002bf6 thru 0x00c01d (0x9428 bytes) 000143 1777593720.956678 trunc from 0x00c245 to 0x00e9ca (0x2786 bytes) 000144 1777593720.961081 punch from 0x001a48 to 0x00c08f (0xa648 bytes) 000146 1777593720.964149 fallocate 0x000183 thru 0x004208 (0x4086 bytes) 000147 1777593720.966619 trunc from 0x024fd9 to 0x02fdbb (0xade3 bytes) 000148 1777593720.971501 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a583): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a583): Mirror 4: 0x8367a238 (stale) CRC-32 checksum value for chunk [0, 0x3a583): Mirror 1: 0x38b30de7 Mirror 5: 0x38b30de7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a583: skipped 000149 1777593721.347872 trunc from 0x02a2cd to 0x03587c (0xb5b0 bytes) 000150 1777593721.363959 write 0x00d538 thru 0x00e111 (0xbda bytes) 000151 1777593721.369713 mapwrite 0x032e6a thru 0x03eb3f (0xbcd6 bytes) 000153 1777593721.403008 fallocate 0x03b184 thru 0x03ffff (0x4e7c bytes) 000154 1777593721.406557 mapread 0x022647 thru 0x02f88f (0xd249 bytes) 000155 1777593721.414291 punch from 0x0379ee to 0x03ffff (0x8612 bytes) 000156 1777593721.419225 mapread 0x0050a9 thru 0x013290 (0xe1e8 bytes) 000157 1777593721.430697 fallocate 0x02d087 thru 0x02d3f7 (0x371 bytes) 000158 1777593721.433308 write 0x03cc77 thru 0x03ffff (0x3389 bytes) 000159 1777593721.439123 read 0x0044e0 thru 0x00d3a5 (0x8ec6 bytes) 000160 1777593721.439611 trunc from 0x011a03 to 0x01b8ef (0x9eed bytes) 000161 1777593721.443586 trunc from 0x040000 to 0x0069cb (0x39634 bytes) 000163 1777593721.455611 fallocate 0x027e6c thru 0x029189 (0x131e bytes) 000164 1777593721.458930 write 0x01fe3e thru 0x02f4d4 (0xf697 bytes) 000165 1777593721.464933 punch from 0x01f2dc to 0x022c69 (0x398e bytes) 000166 1777593721.474836 trunc from 0x02f4d5 to 0x01ca00 (0x12ad4 bytes) 000167 1777593721.483154 write 0x03d071 thru 0x03ffff (0x2f8f bytes) 000168 1777593721.484296 write 0x00a516 thru 0x012443 (0x7f2e bytes) 000169 1777593721.489594 mapwrite 0x01e93d thru 0x01f485 (0xb49 bytes) 000170 1777593721.511437 punch from 0x039930 to 0x03dcc5 (0x4396 bytes) 000171 1777593721.514353 read 0x00f1ae thru 0x01f18b (0xffde bytes) 000172 1777593721.519689 trunc from 0x040000 to 0x03551c (0xaae3 bytes) 000173 1777593721.527553 trunc from 0x03551d to 0x002f55 (0x325c7 bytes) 000175 1777593721.536580 trunc from 0x002f56 to 0x01a468 (0x17513 bytes) 000176 1777593721.544426 trunc from 0x011325 to 0x017ab6 (0x6792 bytes) 000177 1777593721.548430 read 0x014c2d thru 0x01a468 (0x583c bytes) 000179 1777593721.554700 mapwrite 0x02dc3e thru 0x038a2a (0xaded bytes) 000180 1777593721.583335 mapwrite 0x038a4d thru 0x03ffff (0x75b3 bytes) 000181 1777593721.610976 read 0x00ce1f thru 0x01203b (0x521d bytes) 000182 1777593721.611798 trunc from 0x03dad7 to 0x03e93c (0xe66 bytes) 000183 1777593721.615150 punch from 0x036e9d to 0x03c9d4 (0x5b38 bytes) 000184 1777593721.619268 mapwrite 0x020e2f thru 0x025ae1 (0x4cb3 bytes) 000185 1777593721.631956 read 0x0208ba thru 0x02fe8e (0xf5d5 bytes) 000186 1777593721.632664 mapwrite 0x0285ed thru 0x028800 (0x214 bytes) 000187 1777593721.638401 write 0x003a41 thru 0x00db3b (0xa0fb bytes) 000188 1777593721.641839 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd33b426c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd33b426c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000189 1777593721.770848 MIRROR_RESYNC 2 mirrors 000190 1777593722.062175 write 0x03e792 thru 0x03ffff (0x186e bytes) 000191 1777593722.079255 read 0x037fbd thru 0x03c544 (0x4588 bytes) 000192 1777593722.086366 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) 000193 1777593722.149029 mapwrite 0x002111 thru 0x006498 (0x4388 bytes) 000194 1777593722.179611 MIRROR_EXTEND to 2 mirrors 000195 1777593722.463022 trunc from 0x0108b9 to 0x0201c4 (0xf90c bytes) 000196 1777593722.492634 mapwrite 0x02e0d9 thru 0x0318ae (0x37d6 bytes) 000197 1777593722.511838 fallocate 0x02bd4c thru 0x0343f5 (0x86aa bytes) 000198 1777593722.516969 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa32f12b7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbbb28808 Mirror 3: 0xbbb28808 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000199 1777593722.861691 read 0x013a26 thru 0x019e81 (0x645c bytes) 000200 1777593722.879476 mapwrite 0x03a5c0 thru 0x03ffff (0x5a40 bytes) 000201 1777593722.941389 mapread 0x03f85f thru 0x03ffff (0x7a1 bytes) 000202 1777593722.942361 trunc from 0x040000 to 0x00b8a7 (0x34758 bytes) 000203 1777593722.967409 punch from 0x00a752 to 0x00b8a7 (0x1156 bytes) 000204 1777593722.972582 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb8a8): Mirror 3: 0xc02e422a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb8a8): Mirror 3: 0xc02e422a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb8a8: skipped 000205 1777593723.095926 trunc from 0x007ade to 0x00b8a7 (0x3dca bytes) 000206 1777593723.100073 write 0x00a288 thru 0x017131 (0xceaa bytes) 000207 1777593723.106106 write 0x00111d thru 0x00ce88 (0xbd6c bytes) 000208 1777593723.109643 mapwrite 0x037958 thru 0x03ae08 (0x34b1 bytes) 000209 1777593723.140594 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) 000210 1777593723.206831 punch from 0x00fc31 to 0x01c87b (0xcc4b bytes) 000211 1777593723.210117 punch from 0x011f07 to 0x014087 (0x2181 bytes) 000213 1777593723.214715 trunc from 0x02d9de to 0x03049a (0x2abd bytes) 000214 1777593723.218312 write 0x01008c thru 0x0136d1 (0x3646 bytes) 000215 1777593723.228278 mapread 0x0175a1 thru 0x01f239 (0x7c99 bytes) 000216 1777593723.233395 fallocate 0x0026d0 thru 0x0059de (0x330f bytes) 000217 1777593723.236333 mapwrite 0x01348b thru 0x01823a (0x4db0 bytes) 000220 1777593723.251637 fallocate 0x01e8da thru 0x0201f9 (0x1920 bytes) 000221 1777593723.255961 write 0x03c382 thru 0x03ffff (0x3c7e bytes) 000222 1777593723.257008 mapwrite 0x01a916 thru 0x01feaa (0x5595 bytes) 000223 1777593723.276354 trunc from 0x0280c3 to 0x02c16a (0x40a8 bytes) 000224 1777593723.280105 trunc from 0x040000 to 0x028cf7 (0x17308 bytes) 000226 1777593723.290017 trunc from 0x028cf8 to 0x03426f (0xb578 bytes) 000227 1777593723.299124 write 0x01608b thru 0x019836 (0x37ac bytes) 000228 1777593723.300034 punch from 0x019b15 to 0x01fd1f (0x620b bytes) 000229 1777593723.311641 punch from 0x02536a to 0x0278b4 (0x254b bytes) 000230 1777593723.315035 write 0x005f7d thru 0x00718d (0x1211 bytes) 000231 1777593723.320172 punch from 0x0166b2 to 0x021290 (0xabdf bytes) 000232 1777593723.323519 punch from 0x033b01 to 0x03426f (0x76f bytes) 000233 1777593723.326602 fallocate 0x00f86e thru 0x012b44 (0x32d7 bytes) 000235 1777593723.330004 mapwrite 0x028698 thru 0x03471b (0xc084 bytes) 000236 1777593723.355529 fallocate 0x03db95 thru 0x03ffff (0x246b bytes) 000237 1777593723.358420 read 0x01a9c6 thru 0x01ac72 (0x2ad bytes) 000240 1777593723.362279 punch from 0x01abec to 0x01e4e7 (0x38fc bytes) 000241 1777593723.366902 fallocate 0x03e550 thru 0x03ffff (0x1ab0 bytes) 000242 1777593723.370213 punch from 0x01d8e4 to 0x028dfd (0xb51a bytes) 000244 1777593723.373525 write 0x005f81 thru 0x00e5d5 (0x8655 bytes) 000245 1777593723.374555 read 0x024110 thru 0x026c1e (0x2b0f bytes) 000246 1777593723.379039 write 0x03dbf7 thru 0x03ffff (0x2409 bytes) 000247 1777593723.380047 read 0x003121 thru 0x006381 (0x3261 bytes) 000248 1777593723.380680 read 0x03131b thru 0x0391bd (0x7ea3 bytes) 000249 1777593723.381275 mapread 0x03a3fa thru 0x03ffff (0x5c06 bytes) 000251 1777593723.382824 punch from 0x02e60c to 0x037e8f (0x9884 bytes) 000252 1777593723.385682 trunc from 0x017643 to 0x026f5c (0xf91a bytes) 000253 1777593723.388118 mapread 0x03fc9c thru 0x03ffff (0x364 bytes) 000255 1777593723.389669 punch from 0x0240de to 0x02b8f3 (0x7816 bytes) 000256 1777593723.392425 read 0x006a2f thru 0x01177b (0xad4d bytes) 000257 1777593723.393096 punch from 0x01df70 to 0x02b0c7 (0xd158 bytes) 000258 1777593723.396109 write 0x00efc4 thru 0x010521 (0x155e bytes) 000259 1777593723.396973 punch from 0x0113a8 to 0x01cda5 (0xb9fe bytes) 000261 1777593723.400700 trunc from 0x040000 to 0x014179 (0x2be86 bytes) 000262 1777593723.407901 mapread 0x0055d0 thru 0x00e518 (0x8f49 bytes) 000263 1777593723.408853 punch from 0x0131f4 to 0x014179 (0xf86 bytes) 000265 1777593723.411785 trunc from 0x01207f to 0x014179 (0x20fb bytes) 000266 1777593723.414091 fallocate 0x010876 thru 0x02080c (0xff97 bytes) 000267 1777593723.416855 write 0x01db26 thru 0x01f5f4 (0x1acf bytes) 000269 1777593723.418273 trunc from 0x01f5f5 to 0x02b7d6 (0xc1e2 bytes) 000270 1777593723.425119 mapwrite 0x017173 thru 0x0199c6 (0x2854 bytes) 000271 1777593723.442579 trunc from 0x01bfd1 to 0x01d1e0 (0x1210 bytes) 000272 1777593723.448092 trunc from 0x02b7d7 to 0x02f585 (0x3daf bytes) 000273 1777593723.457251 trunc from 0x00dd81 to 0x00eb6a (0xdea bytes) 000274 1777593723.462591 MIRROR_EXTEND to 2 mirrors 000275 1777593723.740402 trunc from 0x0253cb to 0x02bf42 (0x6b78 bytes) 000276 1777593723.758578 write 0x0268f3 thru 0x0286d4 (0x1de2 bytes) 000277 1777593723.765540 trunc from 0x02f586 to 0x017ad7 (0x17aae bytes) 000278 1777593723.780925 trunc from 0x00e0bb to 0x014a6e (0x69b4 bytes) 000279 1777593723.783482 fallocate 0x03a7ef thru 0x03ffff (0x5811 bytes) 000280 1777593723.787938 MIRROR_RESYNC 2 mirrors 000281 1777593724.026529 read 0x01fa7a thru 0x02894d (0x8ed4 bytes) 000282 1777593724.031246 punch from 0x03b540 to 0x03ffff (0x4ac0 bytes) 000284 1777593724.048109 write 0x0039ba thru 0x00d194 (0x97db bytes) 000285 1777593724.055770 mapwrite 0x024c27 thru 0x02e2c1 (0x969b bytes) 000286 1777593724.084695 trunc from 0x040000 to 0x0165c4 (0x29a3b bytes) 000287 1777593724.114593 trunc from 0x000f35 to 0x003c05 (0x2cd1 bytes) 000288 1777593724.118788 fallocate 0x033ceb thru 0x035d39 (0x204f bytes) 000289 1777593724.122921 write 0x02b7da thru 0x02e725 (0x2f4c bytes) 000291 1777593724.124559 fallocate 0x01251f thru 0x01bd3e (0x9820 bytes) 000292 1777593724.127544 trunc from 0x0220c7 to 0x029136 (0x7070 bytes) 000293 1777593724.130833 read 0x02c433 thru 0x02d813 (0x13e1 bytes) 000294 1777593724.131462 mapread 0x01c256 thru 0x01e02c (0x1dd7 bytes) 000295 1777593724.138534 punch from 0x02d19b to 0x02dea5 (0xd0b bytes) 000297 1777593724.147797 write 0x0377a7 thru 0x03ffff (0x8859 bytes) 000299 1777593724.149766 punch from 0x020482 to 0x02b1ef (0xad6e bytes) 000300 1777593724.153023 fallocate 0x038b1d thru 0x03c28f (0x3773 bytes) 000301 1777593724.155751 punch from 0x016708 to 0x024a4c (0xe345 bytes) 000302 1777593724.158904 fallocate 0x02aa19 thru 0x039de8 (0xf3d0 bytes) 000303 1777593724.165682 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) 000304 1777593724.240832 write 0x03b2f1 thru 0x03ffff (0x4d0f bytes) 000306 1777593724.249373 trunc from 0x040000 to 0x03b171 (0x4e8e bytes) 000307 1777593724.260498 trunc from 0x03b172 to 0x030176 (0xaffb bytes) 000308 1777593724.279023 mapread 0x02307e thru 0x02cfd5 (0x9f58 bytes) 000309 1777593724.289750 write 0x00fe24 thru 0x01c5f6 (0xc7d3 bytes) 000310 1777593724.296921 fallocate 0x004704 thru 0x00b2f2 (0x6bef bytes) 000311 1777593724.299573 punch from 0x01a570 to 0x026940 (0xc3d1 bytes) 000312 1777593724.307705 mapwrite 0x001f5f thru 0x0037c9 (0x186b bytes) 000313 1777593724.321488 mapwrite 0x02a345 thru 0x02d346 (0x3002 bytes) 000314 1777593724.334100 read 0x01a3bd thru 0x020d60 (0x69a4 bytes) 000316 1777593724.339831 read 0x01d5ab thru 0x0213ab (0x3e01 bytes) 000317 1777593724.344841 punch from 0x02728e to 0x02dc85 (0x69f8 bytes) 000318 1777593724.348611 mapread 0x021f96 thru 0x027473 (0x54de bytes) 000319 1777593724.357066 MIRROR_EXTEND to 2 mirrors 000320 1777593724.642934 fallocate 0x006d82 thru 0x0075f9 (0x878 bytes) 000321 1777593724.666599 fallocate 0x004196 thru 0x01149e (0xd309 bytes) 000323 1777593724.671985 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) 000324 1777593724.727030 punch from 0x00d0ea to 0x0119eb (0x4902 bytes) 000325 1777593724.730157 punch from 0x02d876 to 0x030176 (0x2901 bytes) 000328 1777593724.735626 fallocate 0x00e805 thru 0x01be09 (0xd605 bytes) 000329 1777593724.739064 fallocate 0x01722f thru 0x019547 (0x2319 bytes) 000330 1777593724.741778 read 0x001ce0 thru 0x003ab0 (0x1dd1 bytes) 000332 1777593724.745136 write 0x03e8fd thru 0x03ffff (0x1703 bytes) 000335 1777593724.749427 trunc from 0x02865c to 0x0341ed (0xbb92 bytes) 000336 1777593724.752639 fallocate 0x02514a thru 0x033af3 (0xe9aa bytes) 000337 1777593724.755445 read 0x03f2bb thru 0x03ffff (0xd45 bytes) 000339 1777593724.756979 trunc from 0x040000 to 0x0253b8 (0x1ac47 bytes) 000340 1777593724.772392 read 0x01e909 thru 0x0253b8 (0x6ab0 bytes) 000341 1777593724.778475 mapwrite 0x034152 thru 0x03ffff (0xbeae bytes) 000342 1777593724.801028 mapwrite 0x02da01 thru 0x03d162 (0xf762 bytes) 000343 1777593724.813239 write 0x00483b thru 0x00ab97 (0x635d bytes) 000344 1777593724.814504 trunc from 0x040000 to 0x006c12 (0x393ed bytes) 000345 1777593724.824411 trunc from 0x006c13 to 0x01c6c8 (0x15ab6 bytes) 000346 1777593724.833668 read 0x017a1e thru 0x01c6c8 (0x4cab bytes) 000349 1777593724.840617 read 0x0002f8 thru 0x00d316 (0xd01f bytes) 000351 1777593724.841916 trunc from 0x00179a to 0x009cb2 (0x8519 bytes) 000352 1777593724.848350 mapwrite 0x024502 thru 0x026614 (0x2113 bytes) 000353 1777593724.873758 mapread 0x02549c thru 0x026614 (0x1179 bytes) 000354 1777593724.874695 trunc from 0x026615 to 0x019d74 (0xc8a0 bytes) 000355 1777593724.884821 fallocate 0x03052e thru 0x030dd8 (0x8ab bytes) 000357 1777593724.889941 punch from 0x013e3b to 0x015246 (0x140c bytes) 000358 1777593724.893298 mapwrite 0x03a9e8 thru 0x03ffff (0x5618 bytes) 000359 1777593724.917101 read 0x02c5d9 thru 0x03277e (0x61a6 bytes) 000360 1777593724.917683 fallocate 0x01cd01 thru 0x021143 (0x4443 bytes) 000361 1777593724.921886 write 0x016136 thru 0x01affe (0x4ec9 bytes) 000362 1777593724.927601 trunc from 0x040000 to 0x0131b3 (0x2ce4c bytes) 000364 1777593724.939876 write 0x001f39 thru 0x005bcc (0x3c94 bytes) 000366 1777593724.941934 fallocate 0x03479b thru 0x03cb5c (0x83c2 bytes) 000367 1777593724.947535 write 0x028a99 thru 0x02ff66 (0x74ce bytes) 000368 1777593724.948806 trunc from 0x01cc1d to 0x01e7c5 (0x1ba9 bytes) 000369 1777593724.951870 mapwrite 0x014076 thru 0x0227d5 (0xe760 bytes) 000370 1777593724.975079 trunc from 0x02ff67 to 0x00561c (0x2a94a bytes) 000371 1777593724.982330 write 0x030d91 thru 0x03f10d (0xe37d bytes) 000372 1777593724.983783 mapwrite 0x00bd29 thru 0x011150 (0x5428 bytes) 000373 1777593725.012335 trunc from 0x03cd5f to 0x03f10d (0x23af bytes) 000374 1777593725.015668 read 0x01a796 thru 0x022434 (0x7c9f bytes) 000375 1777593725.021786 trunc from 0x03545b to 0x03f10d (0x9cb3 bytes) 000377 1777593725.027505 mapread 0x036185 thru 0x03f10d (0x8f89 bytes) 000378 1777593725.050503 mapread 0x024174 thru 0x03095d (0xc7ea bytes) 000379 1777593725.051671 punch from 0x02a46b to 0x030fbb (0x6b51 bytes) 000381 1777593725.055140 fallocate 0x0139e7 thru 0x01fdd7 (0xc3f1 bytes) 000382 1777593725.058671 fallocate 0x031e10 thru 0x03a7f8 (0x89e9 bytes) 000383 1777593725.061195 trunc from 0x03984e to 0x03f10d (0x58c0 bytes) 000384 1777593725.063843 punch from 0x02da04 to 0x030b9c (0x3199 bytes) 000385 1777593725.067721 mapwrite 0x036d94 thru 0x0379a1 (0xc0e bytes) 000386 1777593725.076979 trunc from 0x01f7ae to 0x022ac0 (0x3313 bytes) 000388 1777593725.081627 read 0x02b634 thru 0x02e921 (0x32ee bytes) 000389 1777593725.087563 mapread 0x000f67 thru 0x009ee0 (0x8f7a bytes) 000390 1777593725.088711 punch from 0x01a4d1 to 0x025a45 (0xb575 bytes) 000392 1777593725.092623 trunc from 0x0322b7 to 0x03456e (0x22b8 bytes) 000393 1777593725.096327 mapread 0x006e43 thru 0x00ef8e (0x814c bytes) 000394 1777593725.097510 fallocate 0x01adb5 thru 0x023569 (0x87b5 bytes) 000395 1777593725.100654 write 0x017336 thru 0x020875 (0x9540 bytes) 000396 1777593725.105334 trunc from 0x0126b5 to 0x01a9f3 (0x833f bytes) 000397 1777593725.115061 punch from 0x00e143 to 0x015bd6 (0x7a94 bytes) 000398 1777593725.117956 mapwrite 0x039117 thru 0x03ffff (0x6ee9 bytes) 000401 1777593725.140143 mapwrite 0x0077fc thru 0x00fdb6 (0x85bb bytes) 000402 1777593725.156218 mapread 0x03e21d thru 0x03efb5 (0xd99 bytes) 000403 1777593725.157106 punch from 0x0159c2 to 0x01d17b (0x77ba bytes) 000404 1777593725.160586 trunc from 0x040000 to 0x0262cc (0x19d33 bytes) 000405 1777593725.168634 punch from 0x00e9cd to 0x01823d (0x9871 bytes) 000408 1777593725.173795 trunc from 0x0090d0 to 0x014fda (0xbf0b bytes) 000409 1777593725.178243 trunc from 0x001869 to 0x01112a (0xf8c2 bytes) 000410 1777593725.182227 read 0x01ed52 thru 0x0262cc (0x757b bytes) 000411 1777593725.188565 trunc from 0x0262cd to 0x01142c (0x14ea0 bytes) 000412 1777593725.195720 fallocate 0x014df0 thru 0x020aee (0xbcff bytes) 000413 1777593725.198664 punch from 0x00bd63 to 0x019523 (0xd7c1 bytes) 000414 1777593725.201121 read 0x00d3e8 thru 0x00f856 (0x246f bytes) 000417 1777593725.208801 read 0x0186f9 thru 0x020aee (0x83f6 bytes) 000418 1777593725.209469 mapread 0x00314d thru 0x00689c (0x3750 bytes) 000419 1777593725.210521 read 0x01039a thru 0x01ec1f (0xe886 bytes) 000420 1777593725.211126 trunc from 0x020aef to 0x019a4a (0x70a4 bytes) 000421 1777593725.218733 trunc from 0x001a44 to 0x005aec (0x40a9 bytes) 000422 1777593725.221981 trunc from 0x019a4b to 0x01ccd7 (0x328d bytes) 000423 1777593725.229538 trunc from 0x01ccd8 to 0x004707 (0x185d0 bytes) 000424 1777593725.236978 punch from 0x003740 to 0x004707 (0xfc8 bytes) 000425 1777593725.239567 write 0x00cb06 thru 0x00fa31 (0x2f2c bytes) 000426 1777593725.240623 mapread 0x00dcac thru 0x00fa31 (0x1d86 bytes) 000427 1777593725.241505 mapwrite 0x033315 thru 0x03d33b (0xa027 bytes) 000428 1777593725.264392 trunc from 0x03d33c to 0x0313e2 (0xbf59 bytes) 000430 1777593725.272819 punch from 0x01f2ee to 0x02c42b (0xd13e bytes) 000432 1777593725.278178 mapwrite 0x001281 thru 0x00c7e7 (0xb567 bytes) 000434 1777593725.292275 write 0x00ec57 thru 0x010997 (0x1d41 bytes) 000435 1777593725.293163 punch from 0x012ec8 to 0x01b0fa (0x8233 bytes) 000436 1777593725.295667 trunc from 0x0313e3 to 0x02ba3c (0x59a6 bytes) 000437 1777593725.307232 MIRROR_EXTEND to 2 mirrors 000438 1777593725.577624 read 0x0039a9 thru 0x009250 (0x58a8 bytes) 000439 1777593725.585610 mapwrite 0x039c74 thru 0x03b744 (0x1ad1 bytes) 000441 1777593725.650937 write 0x03d117 thru 0x03fcfc (0x2be6 bytes) 000442 1777593725.651865 fallocate 0x027ecd thru 0x02ede4 (0x6f18 bytes) 000443 1777593725.655440 mapread 0x0297b7 thru 0x02b647 (0x1e91 bytes) 000444 1777593725.665561 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fcfd): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fcfd): Mirror 2: 0x9159552a (stale) CRC-32 checksum value for chunk [0, 0x3fcfd): Mirror 1: 0x99041ac8 Mirror 3: 0x99041ac8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fcfd: skipped 000445 1777593726.046039 trunc from 0x017a5d to 0x02179d (0x9d41 bytes) 000446 1777593726.066821 mapread 0x00f9e1 thru 0x013a24 (0x4044 bytes) 000447 1777593726.081994 mapread 0x0001ec thru 0x003c05 (0x3a1a bytes) 000448 1777593726.090581 read 0x02d174 thru 0x0343b8 (0x7245 bytes) 000449 1777593726.096635 mapread 0x02d71a thru 0x031508 (0x3def bytes) 000450 1777593726.097656 mapwrite 0x0330c2 thru 0x03f4b6 (0xc3f5 bytes) 000451 1777593726.118867 punch from 0x021aec to 0x026b14 (0x5029 bytes) 000452 1777593726.122932 fallocate 0x03a9b2 thru 0x03fb05 (0x5154 bytes) 000453 1777593726.125966 read 0x015e10 thru 0x02528e (0xf47f bytes) 000455 1777593726.133807 punch from 0x02062f to 0x02f8de (0xf2b0 bytes) 000457 1777593726.137730 trunc from 0x000273 to 0x00ee39 (0xebc7 bytes) 000458 1777593726.142418 read 0x01c981 thru 0x01e87a (0x1efa bytes) 000459 1777593726.142958 read 0x03f7d3 thru 0x03fcfc (0x52a bytes) 000460 1777593726.143506 mapread 0x020792 thru 0x02dd61 (0xd5d0 bytes) 000461 1777593726.150299 write 0x02624f thru 0x03271a (0xc4cc bytes) 000462 1777593726.156094 mapwrite 0x01031b thru 0x01c2e9 (0xbfcf bytes) 000463 1777593726.168128 fallocate 0x033e45 thru 0x03c8bf (0x8a7b bytes) 000464 1777593726.170916 fallocate 0x038b5c thru 0x03d4c9 (0x496e bytes) 000465 1777593726.173720 trunc from 0x03fcfd to 0x00b842 (0x344ba bytes) 000466 1777593726.194191 mapwrite 0x034b41 thru 0x03ffff (0xb4bf bytes) 000468 1777593726.254748 mapwrite 0x008b46 thru 0x009800 (0xcbb bytes) 000469 1777593726.266184 punch from 0x00f982 to 0x01145d (0x1adc bytes) 000470 1777593726.269262 fallocate 0x002783 thru 0x007941 (0x51bf bytes) 000471 1777593726.273396 trunc from 0x040000 to 0x00040a (0x3fbf5 bytes) 000473 1777593726.287838 punch from 0x000395 to 0x00040a (0x76 bytes) 000474 1777593726.294674 MIRROR_RESYNC 3 mirrors 000475 1777593726.526534 mapread 0x000143 thru 0x00040a (0x2c8 bytes) 000476 1777593726.531327 punch from 0x0000c7 to 0x00040a (0x344 bytes) 000477 1777593726.554688 MIRROR_RESYNC 3 mirrors 000478 1777593726.764863 mapread 0x0000a6 thru 0x00040a (0x365 bytes) 000479 1777593726.768974 fallocate 0x03fb39 thru 0x03ffff (0x4c7 bytes) 000480 1777593726.783995 mapread 0x000025 thru 0x00040a (0x3e6 bytes) 000481 1777593726.786957 trunc from 0x00005e to 0x00040a (0x3ad bytes) 000482 1777593726.789301 punch from 0x000020 to 0x00040a (0x3eb bytes) 000483 1777593726.791624 mapread 0x000010 thru 0x00040a (0x3fb bytes) 000484 1777593726.794398 read 0x000093 thru 0x00040a (0x378 bytes) 000485 1777593726.794991 trunc from 0x00040b to 0x03f3aa (0x3efa0 bytes) 000486 1777593726.809603 mapread 0x018dff thru 0x02679b (0xd99d bytes) 000487 1777593726.821848 trunc from 0x00ac8c to 0x017d95 (0xd10a bytes) 000488 1777593726.824367 mapwrite 0x007c16 thru 0x00da76 (0x5e61 bytes) 000489 1777593726.840055 punch from 0x01d800 to 0x02517d (0x797e bytes) 000490 1777593726.843544 mapread 0x003258 thru 0x00f9c3 (0xc76c bytes) 000491 1777593726.844772 write 0x016e75 thru 0x0176bb (0x847 bytes) 000492 1777593726.848476 write 0x018da3 thru 0x01a197 (0x13f5 bytes) 000493 1777593726.849390 trunc from 0x00aad1 to 0x00b197 (0x6c7 bytes) 000494 1777593726.854657 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f3ab): Mirror 2: 0xf1b3a2bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f3ab): Mirror 2: 0xf1b3a2bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f3ab: skipped 000495 1777593726.976021 punch from 0x0250d1 to 0x03401c (0xef4c bytes) 000496 1777593726.979271 fallocate 0x02bbaf thru 0x037052 (0xb4a4 bytes) 000497 1777593726.983164 write 0x00be0e thru 0x01557c (0x976f bytes) 000498 1777593726.992839 mapwrite 0x003b7b thru 0x00de28 (0xa2ae bytes) 000499 1777593727.008315 read 0x014ed8 thru 0x020eed (0xc016 bytes) 000500 1777593727.014438 write 0x0110cc thru 0x015e35 (0x4d6a bytes) 000501 1777593727.015737 read 0x00f7f6 thru 0x01ee49 (0xf654 bytes) 000502 1777593727.016293 write 0x03faf4 thru 0x03ffff (0x50c bytes) 000503 1777593727.021363 fallocate 0x00d432 thru 0x0189e4 (0xb5b3 bytes) 000504 1777593727.026150 mapwrite 0x02eece thru 0x03ae40 (0xbf73 bytes) 000506 1777593727.039885 trunc from 0x031c94 to 0x03be27 (0xa194 bytes) 000507 1777593727.043953 write 0x0382b5 thru 0x03ffff (0x7d4b bytes) 000508 1777593727.047551 write 0x004572 thru 0x00f9cd (0xb45c bytes) 000509 1777593727.048672 trunc from 0x02a01d to 0x02e247 (0x422b bytes) 000510 1777593727.052991 trunc from 0x0113d8 to 0x01ae80 (0x9aa9 bytes) 000511 1777593727.057021 trunc from 0x009ae4 to 0x010842 (0x6d5f bytes) 000512 1777593727.068433 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfca72743 Mirror 3: 0xfca72743 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000513 1777593727.463667 fallocate 0x037d68 thru 0x03caa2 (0x4d3b bytes) 000514 1777593727.483628 fallocate 0x013e57 thru 0x01f481 (0xb62b bytes) 000515 1777593727.487819 punch from 0x015ff3 to 0x0233f2 (0xd400 bytes) 000516 1777593727.492166 mapread 0x002081 thru 0x0060ac (0x402c bytes) 000517 1777593727.505576 mapwrite 0x011ba4 thru 0x021b2c (0xff89 bytes) 000518 1777593727.531720 write 0x01bade thru 0x0271a6 (0xb6c9 bytes) 000519 1777593727.538028 trunc from 0x00acc1 to 0x00c9b8 (0x1cf8 bytes) 000520 1777593727.542308 trunc from 0x040000 to 0x014459 (0x2bba6 bytes) 000521 1777593727.563277 trunc from 0x01445a to 0x02a6ef (0x16296 bytes) 000522 1777593727.574294 mapread 0x005a48 thru 0x012b4e (0xd107 bytes) 000523 1777593727.579324 trunc from 0x02801e to 0x02a6ef (0x26d2 bytes) 000524 1777593727.582527 punch from 0x01962d to 0x026d21 (0xd6f5 bytes) 000526 1777593727.586235 read 0x026d75 thru 0x02a6ef (0x397b bytes) 000527 1777593727.590833 read 0x006649 thru 0x007caf (0x1667 bytes) 000528 1777593727.591525 write 0x01252c thru 0x01eac3 (0xc598 bytes) 000529 1777593727.596122 write 0x016081 thru 0x022514 (0xc494 bytes) 000530 1777593727.600543 punch from 0x00de74 to 0x00f6eb (0x1878 bytes) 000531 1777593727.603097 fallocate 0x0051c3 thru 0x012532 (0xd370 bytes) 000532 1777593727.605916 mapread 0x004f72 thru 0x009f3d (0x4fcc bytes) 000533 1777593727.609647 MIRROR_RESYNC 3 mirrors 000534 1777593728.033423 MIRROR_RESYNC 3 mirrors 000535 1777593728.195809 MIRROR_RESYNC 3 mirrors 000536 1777593728.371731 mapread 0x015ce5 thru 0x018a08 (0x2d24 bytes) 000537 1777593728.382556 fallocate 0x02d794 thru 0x02fea1 (0x270e bytes) 000539 1777593728.399911 read 0x0168a8 thru 0x01ba5f (0x51b8 bytes) 000540 1777593728.408036 punch from 0x0148dc to 0x017adc (0x3201 bytes) 000541 1777593728.411969 mapwrite 0x014243 thru 0x017946 (0x3704 bytes) 000542 1777593728.425204 mapread 0x014012 thru 0x023a0f (0xf9fe bytes) 000543 1777593728.430951 mapread 0x0256fe thru 0x0285dd (0x2ee0 bytes) 000544 1777593728.435570 mapread 0x0237ca thru 0x02a6ef (0x6f26 bytes) 000545 1777593728.436587 mapwrite 0x03a6e9 thru 0x03ffff (0x5917 bytes) 000546 1777593728.476737 trunc from 0x00fbca to 0x01c904 (0xcd3b bytes) 000547 1777593728.482568 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc17c89c2 (stale) Mirror 3: 0xc17c89c2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5146845a Mirror 4: 0x5146845a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000548 1777593728.842731 trunc from 0x0125fb to 0x01aa86 (0x848c bytes) 000550 1777593728.860111 trunc from 0x040000 to 0x00c9b7 (0x33648 bytes) 000551 1777593728.891236 trunc from 0x00c9b8 to 0x020474 (0x13abd bytes) 000552 1777593728.904350 read 0x01fa61 thru 0x020474 (0xa14 bytes) 000553 1777593728.908522 mapread 0x00caae thru 0x012c29 (0x617c bytes) 000554 1777593728.923169 trunc from 0x020475 to 0x001b43 (0x1e931 bytes) 000555 1777593728.931285 fallocate 0x03b60d thru 0x03ffff (0x49f3 bytes) 000558 1777593728.939273 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc17c89c2 (stale) Mirror 3: 0xc17c89c2 (stale) Mirror 4: 0x5146845a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 5: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000559 1777593729.338244 MIRROR_RESYNC 5 mirrors 000560 1777593729.768943 trunc from 0x040000 to 0x0288a4 (0x1775b bytes) 000561 1777593729.793981 trunc from 0x0191b3 to 0x021b6d (0x89bb bytes) 000562 1777593729.799236 MIRROR_RESYNC 5 mirrors 000563 1777593730.204549 write 0x026f7b thru 0x02ac22 (0x3ca8 bytes) 000564 1777593730.222069 mapread 0x025da0 thru 0x02ac22 (0x4e83 bytes) 000565 1777593730.225478 mapwrite 0x01dc86 thru 0x02c718 (0xea93 bytes) 000566 1777593730.254355 mapread 0x016152 thru 0x01cdac (0x6c5b bytes) 000567 1777593730.255517 read 0x005e61 thru 0x00dbd5 (0x7d75 bytes) 000568 1777593730.261334 write 0x01b575 thru 0x0258d1 (0xa35d bytes) 000569 1777593730.263407 mapread 0x0095ab thru 0x012489 (0x8edf bytes) 000570 1777593730.269852 fallocate 0x00b306 thru 0x016fba (0xbcb5 bytes) 000571 1777593730.276272 write 0x004d73 thru 0x00a6a3 (0x5931 bytes) 000572 1777593730.282392 fallocate 0x00f28f thru 0x01289d (0x360f bytes) 000574 1777593730.287355 fallocate 0x00ecaa thru 0x015f4b (0x72a2 bytes) 000575 1777593730.291254 write 0x032e8e thru 0x03f71a (0xc88d bytes) 000576 1777593730.292581 trunc from 0x00a41c to 0x010676 (0x625b bytes) 000577 1777593730.305787 trunc from 0x01ff30 to 0x0283e9 (0x84ba bytes) 000578 1777593730.311408 trunc from 0x01beff to 0x01f8ee (0x39f0 bytes) 000579 1777593730.315446 fallocate 0x022824 thru 0x02343b (0xc18 bytes) 000580 1777593730.321979 MIRROR_RESYNC 5 mirrors 000581 1777593731.127636 fallocate 0x02ab2c thru 0x036be7 (0xc0bc bytes) 000582 1777593731.146704 write 0x0040ff thru 0x010aa1 (0xc9a3 bytes) 000583 1777593731.153437 fallocate 0x03332f thru 0x034024 (0xcf6 bytes) 000584 1777593731.157080 trunc from 0x01ccc6 to 0x027470 (0xa7ab bytes) 000585 1777593731.160347 fallocate 0x00a31e thru 0x00e41f (0x4102 bytes) 000586 1777593731.166542 mapwrite 0x034f9a thru 0x03742e (0x2495 bytes) 000587 1777593731.186444 write 0x00b389 thru 0x0132ea (0x7f62 bytes) 000588 1777593731.190437 punch from 0x000191 to 0x00612c (0x5f9c bytes) 000589 1777593731.193810 fallocate 0x00f75e thru 0x01a385 (0xac28 bytes) 000590 1777593731.198434 write 0x031bfb thru 0x03ffff (0xe405 bytes) 000591 1777593731.199750 write 0x01c671 thru 0x020505 (0x3e95 bytes) 000592 1777593731.205927 mapwrite 0x03338c thru 0x03ffff (0xcc74 bytes) 000594 1777593731.220586 write 0x01fe93 thru 0x02396b (0x3ad9 bytes) 000595 1777593731.221787 mapread 0x0393a6 thru 0x039ebe (0xb19 bytes) 000596 1777593731.223773 trunc from 0x040000 to 0x022194 (0x1de6b bytes) 000597 1777593731.244906 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22195): Mirror 2: 0x2782a7ae (stale) Mirror 4: 0x2782a7ae (stale) Mirror 5: 0x2782a7ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22195): Mirror 2: 0x2782a7ae (stale) Mirror 4: 0x2782a7ae (stale) Mirror 5: 0x2782a7ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22195: skipped 000598 1777593731.409864 MIRROR_RESYNC 4 mirrors 000599 1777593731.774533 write 0x02f400 thru 0x02f893 (0x494 bytes) 000600 1777593731.788578 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f894): Mirror 4: 0x8bee81e5 (stale) Mirror 5: 0x8bee81e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f894): Mirror 4: 0x8bee81e5 (stale) Mirror 5: 0x8bee81e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f894: skipped 000601 1777593731.931951 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f894): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f894): Mirror 4: 0x8bee81e5 (stale) Mirror 5: 0x8bee81e5 (stale) CRC-32 checksum value for chunk [0, 0x2f894): Mirror 1: 0xd090783d Mirror 6: 0xd090783d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f894: skipped 000602 1777593732.342996 punch from 0x02a8bd to 0x02e8df (0x4023 bytes) 000603 1777593732.362558 trunc from 0x02f894 to 0x024965 (0xaf2e bytes) 000604 1777593732.375855 read 0x010480 thru 0x013e46 (0x39c7 bytes) 000605 1777593732.379835 mapread 0x0204b9 thru 0x024965 (0x44ad bytes) 000606 1777593732.393958 punch from 0x01c2d9 to 0x024965 (0x868d bytes) 000607 1777593732.399272 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24966): Mirror 4: 0xef654ee8 (stale) Mirror 6: 0xef654ee8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24966): Mirror 4: 0xef654ee8 (stale) Mirror 6: 0xef654ee8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24966: skipped 000608 1777593732.531757 mapwrite 0x00e30b thru 0x018fba (0xacb0 bytes) 000609 1777593732.551229 punch from 0x00d5c3 to 0x00fa33 (0x2471 bytes) 000610 1777593732.554168 punch from 0x003ef2 to 0x00c717 (0x8826 bytes) 000611 1777593732.557726 read 0x004816 thru 0x011904 (0xd0ef bytes) 000612 1777593732.563728 fallocate 0x002e31 thru 0x00ca1f (0x9bef bytes) 000613 1777593732.568013 mapwrite 0x0197cf thru 0x019b7c (0x3ae bytes) 000614 1777593732.580945 write 0x03fa78 thru 0x03ffff (0x588 bytes) 000615 1777593732.584009 write 0x03a751 thru 0x03c444 (0x1cf4 bytes) 000616 1777593732.588689 write 0x006ca6 thru 0x01360c (0xc967 bytes) 000617 1777593732.590181 trunc from 0x040000 to 0x024253 (0x1bdac bytes) 000618 1777593732.600864 punch from 0x00b2a7 to 0x01544f (0xa1a9 bytes) 000619 1777593732.611324 fallocate 0x00db16 thru 0x012f0c (0x53f7 bytes) 000620 1777593732.614787 read 0x0027b1 thru 0x00810d (0x595d bytes) 000621 1777593732.620319 fallocate 0x0069d6 thru 0x010083 (0x96ae bytes) 000622 1777593732.623114 write 0x033a4c thru 0x033ef5 (0x4aa bytes) 000623 1777593732.624160 mapwrite 0x029ca1 thru 0x036fbe (0xd31e bytes) 000624 1777593732.653150 mapwrite 0x01e3d5 thru 0x026260 (0x7e8c bytes) 000625 1777593732.664188 read 0x0078b2 thru 0x00898a (0x10d9 bytes) 000626 1777593732.664948 write 0x00f228 thru 0x011d91 (0x2b6a bytes) 000627 1777593732.666269 trunc from 0x0160f7 to 0x024a1c (0xe926 bytes) 000628 1777593732.669536 trunc from 0x036fbf to 0x00e6f8 (0x288c6 bytes) 000629 1777593732.678737 punch from 0x00c2ff to 0x00e6f8 (0x23fa bytes) 000630 1777593732.681949 mapwrite 0x0116e9 thru 0x01d730 (0xc048 bytes) 000631 1777593732.712762 punch from 0x0130da to 0x01d730 (0xa657 bytes) 000632 1777593732.721339 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1d731): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d731): Mirror 4: 0x720f122 (stale) Mirror 6: 0x720f122 (stale) CRC-32 checksum value for chunk [0, 0x1d731): Mirror 1: 0x7070a6d7 Mirror 7: 0x7070a6d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d731: skipped 000633 1777593733.116379 read 0x004364 thru 0x00f1a9 (0xae46 bytes) 000634 1777593733.124170 punch from 0x001772 to 0x0023b0 (0xc3f bytes) 000635 1777593733.141440 mapwrite 0x03ccc5 thru 0x03f971 (0x2cad bytes) 000636 1777593733.181483 read 0x020c40 thru 0x0289ce (0x7d8f bytes) 000637 1777593733.188202 read 0x036790 thru 0x03e2b5 (0x7b26 bytes) 000638 1777593733.188901 fallocate 0x00229e thru 0x00a591 (0x82f4 bytes) 000639 1777593733.192165 mapwrite 0x026595 thru 0x0306fc (0xa168 bytes) 000640 1777593733.201328 fallocate 0x03613a thru 0x03b6a6 (0x556d bytes) 000641 1777593733.203474 trunc from 0x00db30 to 0x01aa8a (0xcf5b bytes) 000642 1777593733.205615 mapwrite 0x01afc4 thru 0x02745f (0xc49c bytes) 000643 1777593733.218855 trunc from 0x02cf04 to 0x03a1c6 (0xd2c3 bytes) 000644 1777593733.221214 fallocate 0x0253e5 thru 0x02c467 (0x7083 bytes) 000645 1777593733.223702 read 0x00bf09 thru 0x01a848 (0xe940 bytes) 000646 1777593733.224344 mapread 0x0031bb thru 0x010fd5 (0xde1b bytes) 000647 1777593733.225444 read 0x004620 thru 0x00e1ab (0x9b8c bytes) 000648 1777593733.228214 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f972): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f972): Mirror 4: 0x1cbfd0b6 (stale) Mirror 6: 0xa9e6ce78 (stale) Mirror 7: 0x219bc141 (stale) CRC-32 checksum value for chunk [0, 0x3f972): Mirror 1: 0x778630d1 Mirror 8: 0x778630d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f972: skipped 000649 1777593733.564867 fallocate 0x002d6a thru 0x00b87f (0x8b16 bytes) 000650 1777593733.580362 mapwrite 0x023964 thru 0x02ac68 (0x7305 bytes) 000651 1777593733.598329 mapwrite 0x030642 thru 0x03ea42 (0xe401 bytes) 000652 1777593733.621418 read 0x0326a0 thru 0x035823 (0x3184 bytes) 000653 1777593733.622071 read 0x03a8a5 thru 0x03f971 (0x50cd bytes) 000654 1777593733.622692 read 0x0009dc thru 0x00ddb1 (0xd3d6 bytes) 000656 1777593733.629645 trunc from 0x03f972 to 0x029783 (0x161ee bytes) 000657 1777593733.647790 mapread 0x01fa22 thru 0x024c14 (0x51f3 bytes) 000658 1777593733.648876 mapwrite 0x001f35 thru 0x0087ea (0x68b6 bytes) 000660 1777593733.658073 mapread 0x00c47f thru 0x0179e1 (0xb563 bytes) 000663 1777593733.660098 punch from 0x00bc35 to 0x019b9e (0xdf6a bytes) 000664 1777593733.663387 fallocate 0x012a7f thru 0x0226ba (0xfc3c bytes) 000666 1777593733.666142 mapwrite 0x00b1ff thru 0x015da9 (0xabab bytes) 000667 1777593733.691149 read 0x0145df thru 0x020c7e (0xc6a0 bytes) 000668 1777593733.694615 fallocate 0x01552c thru 0x01665d (0x1132 bytes) 000669 1777593733.697317 trunc from 0x029784 to 0x03684a (0xd0c7 bytes) 000670 1777593733.703741 mapwrite 0x0157f7 thru 0x01fcde (0xa4e8 bytes) 000671 1777593733.713933 trunc from 0x025f8c to 0x035947 (0xf9bc bytes) 000672 1777593733.717872 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3684b): Mirror 4: 0x42c6e4bc (stale) Mirror 6: 0xa9e6768e (stale) Mirror 7: 0x9ff4ff93 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3684b): Mirror 4: 0x42c6e4bc (stale) Mirror 6: 0xa9e6768e (stale) Mirror 7: 0x9ff4ff93 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3684b: skipped 000673 1777593733.835975 trunc from 0x011528 to 0x014078 (0x2b51 bytes) 000675 1777593733.839233 trunc from 0x03684b to 0x00e763 (0x280e7 bytes) 000676 1777593733.849586 fallocate 0x00a57c thru 0x010a0d (0x6492 bytes) 000677 1777593733.852419 write 0x01fa36 thru 0x029538 (0x9b03 bytes) 000679 1777593733.854134 write 0x01d2c2 thru 0x029e62 (0xcba1 bytes) 000680 1777593733.857435 write 0x008fe7 thru 0x00bff2 (0x300c bytes) 000681 1777593733.863217 punch from 0x021f76 to 0x029e62 (0x7eed bytes) 000682 1777593733.870755 trunc from 0x029e63 to 0x00a95a (0x1f508 bytes) 000683 1777593733.881369 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xa95b): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa95b): Mirror 4: 0x4c45cf32 (stale) Mirror 6: 0x4c45cf32 (stale) Mirror 7: 0x48c58689 (stale) CRC-32 checksum value for chunk [0, 0xa95b): Mirror 1: 0x218af4d3 Mirror 8: 0x218af4d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa95b: skipped 000684 1777593734.157591 punch from 0x0011e6 to 0x00a95a (0x9775 bytes) 000687 1777593734.175089 fallocate 0x00cbd6 thru 0x00ef19 (0x2344 bytes) 000688 1777593734.177889 fallocate 0x02dad3 thru 0x03a36e (0xc89c bytes) 000690 1777593734.181434 write 0x015944 thru 0x02021f (0xa8dc bytes) 000691 1777593734.185661 fallocate 0x01fe44 thru 0x0213f2 (0x15af bytes) 000692 1777593734.192620 MIRROR_RESYNC 5 mirrors 000693 1777593734.655091 trunc from 0x00a45c to 0x00cc93 (0x2838 bytes) 000694 1777593734.671922 punch from 0x005df8 to 0x00ffb1 (0xa1ba bytes) 000696 1777593734.677526 read 0x0030e3 thru 0x00b870 (0x878e bytes) 000697 1777593734.683754 trunc from 0x020220 to 0x01c5dd (0x3c42 bytes) 000698 1777593734.697658 trunc from 0x00c42b to 0x010fa9 (0x4b7f bytes) 000699 1777593734.706410 MIRROR_RESYNC 5 mirrors 000700 1777593735.087042 trunc from 0x01c5de to 0x032530 (0x15f53 bytes) 000701 1777593735.108642 mapwrite 0x00d4f4 thru 0x015531 (0x803e bytes) 000702 1777593735.125238 trunc from 0x032531 to 0x0376b6 (0x5186 bytes) 000703 1777593735.133752 punch from 0x036125 to 0x0376b6 (0x1592 bytes) 000704 1777593735.136571 punch from 0x015256 to 0x015780 (0x52b bytes) 000705 1777593735.139412 write 0x01be31 thru 0x01e53c (0x270c bytes) 000706 1777593735.143851 write 0x01fd0c thru 0x02d255 (0xd54a bytes) 000707 1777593735.148048 mapwrite 0x018c6b thru 0x01a85b (0x1bf1 bytes) 000708 1777593735.160985 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x376b7): Mirror 4: 0xf807ba81 (stale) Mirror 7: 0xf807ba81 (stale) Mirror 8: 0xf807ba81 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376b7): Mirror 4: 0xf807ba81 (stale) Mirror 7: 0xf807ba81 (stale) Mirror 8: 0xf807ba81 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376b7: skipped 000709 1777593735.304833 trunc from 0x0376b7 to 0x020108 (0x175ae bytes) 000710 1777593735.315694 fallocate 0x02b820 thru 0x032e4d (0x762e bytes) 000711 1777593735.319511 write 0x000d4a thru 0x00a255 (0x950c bytes) 000712 1777593735.327328 trunc from 0x001141 to 0x007422 (0x62e2 bytes) 000713 1777593735.337110 write 0x0376c5 thru 0x03ffff (0x893b bytes) 000714 1777593735.338384 mapwrite 0x00cf54 thru 0x010b88 (0x3c35 bytes) 000715 1777593735.362373 mapread 0x022ddd thru 0x02d8fc (0xab20 bytes) 000716 1777593735.372942 fallocate 0x019a06 thru 0x02853d (0xeb38 bytes) 000717 1777593735.375733 trunc from 0x018cdc to 0x0272b1 (0xe5d6 bytes) 000718 1777593735.378381 mapread 0x01350f thru 0x0198b9 (0x63ab bytes) 000720 1777593735.383616 write 0x01effd thru 0x02d3df (0xe3e3 bytes) 000721 1777593735.387264 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5ea8f9fa (stale) Mirror 8: 0x5ea8f9fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5ea8f9fa (stale) Mirror 8: 0x5ea8f9fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000722 1777593735.543052 mapread 0x0384b1 thru 0x03ffff (0x7b4f bytes) 000723 1777593735.563755 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5ea8f9fa (stale) Mirror 8: 0x5ea8f9fa (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xee3a996 Mirror 9: 0xee3a996 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000724 1777593735.932262 write 0x02bb23 thru 0x0385c2 (0xcaa0 bytes) 000726 1777593735.961497 punch from 0x031e35 to 0x0363fc (0x45c8 bytes) 000727 1777593735.968423 mapwrite 0x01210f thru 0x0192ce (0x71c0 bytes) 000728 1777593735.983614 mapwrite 0x0129ee thru 0x017b08 (0x511b bytes) 000729 1777593735.992419 punch from 0x03a606 to 0x03ffff (0x59fa bytes) 000730 1777593735.995377 trunc from 0x00c857 to 0x012230 (0x59da bytes) 000731 1777593735.998744 fallocate 0x01c5d7 thru 0x02c2f5 (0xfd1f bytes) 000732 1777593736.003515 fallocate 0x00979c thru 0x012b01 (0x9366 bytes) 000733 1777593736.010142 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5ea8f9fa (stale) Mirror 8: 0x5ea8f9fa (stale) Mirror 9: 0xee3a996 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7654794a Mirror 10: 0x7654794a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000734 1777593736.539957 write 0x01d399 thru 0x02812b (0xad93 bytes) 000735 1777593736.565611 write 0x02d583 thru 0x02df95 (0xa13 bytes) 000736 1777593736.570164 fallocate 0x0267b5 thru 0x02b594 (0x4de0 bytes) 000737 1777593736.573176 mapread 0x029b16 thru 0x038639 (0xeb24 bytes) 000738 1777593736.578950 read 0x03f2fd thru 0x03ffff (0xd03 bytes) 000739 1777593736.584692 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5ea8f9fa (stale) Mirror 8: 0x5ea8f9fa (stale) Mirror 9: 0xee3a996 (stale) Mirror 10: 0x7654794a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbe2e4dda Mirror 11: 0xbe2e4dda lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000740 1777593737.079812 mapread 0x001a52 thru 0x001d2f (0x2de bytes) 000741 1777593737.092381 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x5ea8f9fa (stale) Mirror 9: 0xee3a996 (stale) Mirror 10: 0x7654794a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x5ea8f9fa (stale) Mirror 9: 0xee3a996 (stale) Mirror 10: 0x7654794a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbe2e4dda Mirror 11: 0xbe2e4dda lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000743 1777593737.316264 punch from 0x0003e7 to 0x00599f (0x55b9 bytes) 000744 1777593737.340623 trunc from 0x0217f6 to 0x0223bf (0xbca bytes) 000745 1777593737.345943 write 0x007bd0 thru 0x008907 (0xd38 bytes) 000746 1777593737.355635 fallocate 0x026b6d thru 0x0324da (0xb96e bytes) 000747 1777593737.362353 mapread 0x039536 thru 0x03b788 (0x2253 bytes) 000750 1777593737.372878 MIRROR_RESYNC 5 mirrors 000751 1777593738.399625 trunc from 0x040000 to 0x03a4ae (0x5b51 bytes) 000752 1777593738.420327 MIRROR_RESYNC 5 mirrors 000753 1777593739.198832 trunc from 0x03a4af to 0x026822 (0x13c8c bytes) 000754 1777593739.224340 trunc from 0x026823 to 0x022034 (0x47ee bytes) 000756 1777593739.238237 MIRROR_RESYNC 5 mirrors 000757 1777593739.939074 trunc from 0x000d29 to 0x00eaac (0xdd84 bytes) 000758 1777593739.959028 trunc from 0x011168 to 0x016807 (0x56a0 bytes) 000759 1777593739.962189 trunc from 0x0149b7 to 0x01ee7e (0xa4c8 bytes) 000760 1777593739.965110 mapread 0x013614 thru 0x013fc0 (0x9ad bytes) 000761 1777593739.968745 write 0x038848 thru 0x03ffff (0x77b8 bytes) 000762 1777593739.970176 read 0x006eb5 thru 0x012b43 (0xbc8f bytes) 000763 1777593739.977342 mapwrite 0x025de1 thru 0x02a3c5 (0x45e5 bytes) 000764 1777593739.999200 trunc from 0x00d72d to 0x017af0 (0xa3c4 bytes) 000766 1777593740.007040 trunc from 0x00eac2 to 0x016a76 (0x7fb5 bytes) 000767 1777593740.009786 punch from 0x029f86 to 0x02e0e8 (0x4163 bytes) 000768 1777593740.016404 fallocate 0x000478 thru 0x009bce (0x9757 bytes) 000769 1777593740.024476 fallocate 0x023a2a thru 0x026f60 (0x3537 bytes) 000770 1777593740.030904 punch from 0x01dd48 to 0x025b1c (0x7dd5 bytes) 000771 1777593740.036314 write 0x01178d thru 0x019185 (0x79f9 bytes) 000772 1777593740.045125 write 0x01d4e9 thru 0x02c089 (0xeba1 bytes) 000773 1777593740.057801 trunc from 0x040000 to 0x01232f (0x2dcd0 bytes) 000774 1777593740.073363 punch from 0x00b8fe to 0x00fb38 (0x423b bytes) 000775 1777593740.078410 trunc from 0x000819 to 0x009f56 (0x973e bytes) 000776 1777593740.082109 mapwrite 0x012c03 thru 0x01d9c0 (0xadbe bytes) 000777 1777593740.113675 trunc from 0x0189a0 to 0x01d9c0 (0x5021 bytes) 000778 1777593740.117157 trunc from 0x0084ae to 0x0087e9 (0x33c bytes) 000779 1777593740.122723 MIRROR_RESYNC 5 mirrors 000780 1777593740.569741 mapwrite 0x015e35 thru 0x024f53 (0xf11f bytes) 000781 1777593740.635857 read 0x015fe7 thru 0x024f53 (0xef6d bytes) 000783 1777593740.636764 trunc from 0x024f54 to 0x02a203 (0x52b0 bytes) 000784 1777593740.649740 fallocate 0x00c0c1 thru 0x00e4ce (0x240e bytes) 000785 1777593740.656371 MIRROR_RESYNC 5 mirrors 000786 1777593741.092285 mapwrite 0x03487b thru 0x03bf7a (0x7700 bytes) 000788 1777593741.145107 mapwrite 0x007133 thru 0x008070 (0xf3e bytes) 000790 1777593741.158953 punch from 0x02dd44 to 0x03835b (0xa618 bytes) 000791 1777593741.165006 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bf7b): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf7b): Mirror 8: 0x1746fbaa (stale) Mirror 9: 0x1746fbaa (stale) Mirror 10: 0x1746fbaa (stale) Mirror 11: 0x1746fbaa (stale) CRC-32 checksum value for chunk [0, 0x3bf7b): Mirror 1: 0x4e77494a Mirror 12: 0x4e77494a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf7b: skipped 000792 1777593741.525323 mapwrite 0x00ee09 thru 0x01898f (0x9b87 bytes) 000793 1777593741.583410 trunc from 0x03bf7b to 0x03dbe4 (0x1c6a bytes) 000794 1777593741.600440 write 0x0385a0 thru 0x03ffff (0x7a60 bytes) 000795 1777593741.606611 trunc from 0x0235ce to 0x0306db (0xd10e bytes) 000796 1777593741.608978 mapwrite 0x003efc thru 0x008b0f (0x4c14 bytes) 000797 1777593741.623156 MIRROR_RESYNC 6 mirrors 000798 1777593742.150137 punch from 0x01da5b to 0x027b47 (0xa0ed bytes) 000799 1777593742.161674 punch from 0x014fbb to 0x01df7a (0x8fc0 bytes) 000800 1777593742.163851 write 0x01f1b5 thru 0x02469e (0x54ea bytes) 000801 1777593742.167600 write 0x02bbb3 thru 0x034f4f (0x939d bytes) 000802 1777593742.173546 trunc from 0x0296b3 to 0x035af6 (0xc444 bytes) 000803 1777593742.180561 mapwrite 0x026923 thru 0x02960a (0x2ce8 bytes) 000804 1777593742.192679 trunc from 0x040000 to 0x01b9c1 (0x2463e bytes) 000805 1777593742.205329 trunc from 0x01649e to 0x01754f (0x10b2 bytes) 000806 1777593742.207909 trunc from 0x0199e6 to 0x01b9c1 (0x1fdc bytes) 000807 1777593742.210724 fallocate 0x025f7e thru 0x02bf9f (0x6022 bytes) 000808 1777593742.213805 mapwrite 0x0122d0 thru 0x02021f (0xdf50 bytes) 000809 1777593742.241781 punch from 0x00d58a to 0x01cd58 (0xf7cf bytes) 000810 1777593742.244849 mapread 0x009773 thru 0x00ed23 (0x55b1 bytes) 000811 1777593742.250038 trunc from 0x00d65f to 0x013a78 (0x641a bytes) 000812 1777593742.252176 mapread 0x014cd9 thru 0x02021f (0xb547 bytes) 000813 1777593742.256864 trunc from 0x017fd8 to 0x018f40 (0xf69 bytes) 000814 1777593742.259294 write 0x011ade thru 0x0145ae (0x2ad1 bytes) 000816 1777593742.260599 read 0x003361 thru 0x00f9b0 (0xc650 bytes) 000817 1777593742.261184 write 0x00b19e thru 0x01331b (0x817e bytes) 000818 1777593742.262368 fallocate 0x00e617 thru 0x00fc10 (0x15fa bytes) 000819 1777593742.265202 write 0x0278e7 thru 0x028237 (0x951 bytes) 000820 1777593742.266186 punch from 0x00fe89 to 0x0102b0 (0x428 bytes) 000821 1777593742.273525 trunc from 0x021844 to 0x021abb (0x278 bytes) 000822 1777593742.276141 trunc from 0x028238 to 0x00e7c8 (0x19a6f bytes) 000823 1777593742.284424 write 0x01f4d8 thru 0x02600b (0x6b34 bytes) 000824 1777593742.285603 read 0x01a47f thru 0x021b00 (0x7682 bytes) 000825 1777593742.289940 mapwrite 0x03b61f thru 0x03ffff (0x49e1 bytes) 000826 1777593742.313588 mapread 0x00bb54 thru 0x01aea9 (0xf356 bytes) 000827 1777593742.315140 mapwrite 0x02e4c7 thru 0x02ec41 (0x77b bytes) 000828 1777593742.323826 punch from 0x01e77b to 0x0216ac (0x2f32 bytes) 000830 1777593742.328445 fallocate 0x017b04 thru 0x0194cd (0x19ca bytes) 000831 1777593742.331784 punch from 0x036b46 to 0x03ffff (0x94ba bytes) 000832 1777593742.334569 fallocate 0x000df6 thru 0x00fa30 (0xec3b bytes) 000833 1777593742.337785 read 0x01a866 thru 0x023187 (0x8922 bytes) 000834 1777593742.345086 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xcc506424 (stale) Mirror 9: 0xcc506424 (stale) Mirror 10: 0xcc506424 (stale) Mirror 12: 0xcc506424 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xcc506424 (stale) Mirror 9: 0xcc506424 (stale) Mirror 10: 0xcc506424 (stale) Mirror 12: 0xcc506424 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000835 1777593742.498116 trunc from 0x040000 to 0x0007ff (0x3f800 bytes) 000837 1777593742.507528 read 0x000616 thru 0x0007ff (0x1ea bytes) 000838 1777593742.510379 trunc from 0x00019b to 0x0007ff (0x665 bytes) 000839 1777593742.513371 read 0x000541 thru 0x0007ff (0x2bf bytes) 000840 1777593742.520089 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x800): Mirror 9: 0xf1e8ba9e (stale) Mirror 10: 0xf1e8ba9e (stale) Mirror 12: 0xf1e8ba9e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x800): Mirror 9: 0xf1e8ba9e (stale) Mirror 10: 0xf1e8ba9e (stale) Mirror 12: 0xf1e8ba9e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x800: skipped 000841 1777593742.636992 mapwrite 0x0125d0 thru 0x020499 (0xdeca bytes) 000842 1777593742.669485 write 0x029689 thru 0x03804f (0xe9c7 bytes) 000843 1777593742.671336 write 0x010414 thru 0x014252 (0x3e3f bytes) 000844 1777593742.672390 trunc from 0x024035 to 0x02bd88 (0x7d54 bytes) 000845 1777593742.679120 write 0x027250 thru 0x029714 (0x24c5 bytes) 000847 1777593742.684357 trunc from 0x02c7a9 to 0x034f06 (0x875e bytes) 000849 1777593742.687725 fallocate 0x00782a thru 0x015c33 (0xe40a bytes) 000850 1777593742.693642 mapread 0x0243b7 thru 0x02973c (0x5386 bytes) 000851 1777593742.697033 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38050): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38050): Mirror 9: 0xecb01dfe (stale) Mirror 10: 0xecb01dfe (stale) Mirror 12: 0xecb01dfe (stale) CRC-32 checksum value for chunk [0, 0x38050): Mirror 1: 0x10c2e5ae Mirror 13: 0x10c2e5ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38050: skipped 000852 1777593742.990193 fallocate 0x00349b thru 0x00767d (0x41e3 bytes) 000853 1777593743.001850 trunc from 0x038050 to 0x03894f (0x900 bytes) 000854 1777593743.015293 trunc from 0x0093e7 to 0x00ceed (0x3b07 bytes) 000855 1777593743.018116 mapwrite 0x01083f thru 0x0191b2 (0x8974 bytes) 000856 1777593743.035089 trunc from 0x038950 to 0x039a5d (0x110e bytes) 000857 1777593743.042238 fallocate 0x038f1d thru 0x0395bd (0x6a1 bytes) 000858 1777593743.045648 mapwrite 0x012457 thru 0x01b54c (0x90f6 bytes) 000859 1777593743.057203 mapwrite 0x026aea thru 0x027766 (0xc7d bytes) 000860 1777593743.068313 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39a5e): Mirror 10: 0xdacda81e (stale) Mirror 12: 0xdacda81e (stale) Mirror 13: 0x4bf6491 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a5e): Mirror 10: 0xdacda81e (stale) Mirror 12: 0xdacda81e (stale) Mirror 13: 0x4bf6491 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a5e: skipped 000861 1777593743.200305 trunc from 0x00683b to 0x0159b9 (0xf17f bytes) 000862 1777593743.203110 mapwrite 0x026c7a thru 0x033843 (0xcbca bytes) 000863 1777593743.221879 read 0x02d59a thru 0x039a5d (0xc4c4 bytes) 000864 1777593743.222581 write 0x03a0f8 thru 0x03d00f (0x2f18 bytes) 000865 1777593743.225241 fallocate 0x00cd18 thru 0x019053 (0xc33c bytes) 000866 1777593743.228009 mapwrite 0x02cf98 thru 0x03026e (0x32d7 bytes) 000867 1777593743.235176 trunc from 0x03d010 to 0x030d6d (0xc2a2 bytes) 000868 1777593743.240987 read 0x011fe1 thru 0x017a79 (0x5a99 bytes) 000869 1777593743.245386 mapread 0x01a51c thru 0x0201be (0x5ca3 bytes) 000870 1777593743.248156 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30d6e): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d6e): Mirror 10: 0x29cb7c48 (stale) Mirror 12: 0x29cb7c48 (stale) Mirror 13: 0xac90a852 (stale) CRC-32 checksum value for chunk [0, 0x30d6e): Mirror 1: 0xa42d6146 Mirror 14: 0xa42d6146 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d6e: skipped 000872 1777593743.505301 punch from 0x0209ed to 0x024025 (0x3639 bytes) 000873 1777593743.519790 trunc from 0x030d6e to 0x02cbaf (0x41be bytes) 000874 1777593743.534134 punch from 0x00a18e to 0x0194a2 (0xf315 bytes) 000875 1777593743.536664 mapwrite 0x02c6dd thru 0x03b1a9 (0xeacd bytes) 000876 1777593743.563262 trunc from 0x01b54a to 0x02532f (0x9de6 bytes) 000877 1777593743.565551 write 0x00efe2 thru 0x014722 (0x5741 bytes) 000878 1777593743.570087 mapwrite 0x02113a thru 0x02e7cf (0xd696 bytes) 000879 1777593743.585230 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b1aa): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b1aa): Mirror 10: 0x2640a7bf (stale) Mirror 12: 0x2640a7bf (stale) Mirror 13: 0xa0992d14 (stale) Mirror 14: 0xeb612da0 (stale) CRC-32 checksum value for chunk [0, 0x3b1aa): Mirror 1: 0x4cdf09b6 Mirror 15: 0x4cdf09b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b1aa: skipped 000880 1777593743.870639 mapread 0x037e4b thru 0x03b1a9 (0x335f bytes) 000881 1777593743.884756 mapread 0x006811 thru 0x008a3a (0x222a bytes) 000882 1777593743.894859 write 0x028e71 thru 0x02fcc6 (0x6e56 bytes) 000884 1777593743.915179 punch from 0x02e85c to 0x03349f (0x4c44 bytes) 000885 1777593743.920932 mapread 0x0135b5 thru 0x013bb3 (0x5ff bytes) 000887 1777593743.925623 mapread 0x0301d2 thru 0x03b1a9 (0xafd8 bytes) 000888 1777593743.932066 read 0x010b6a thru 0x016f6f (0x6406 bytes) 000889 1777593743.932612 write 0x02becb thru 0x03a9c5 (0xeafb bytes) 000890 1777593743.935938 MIRROR_RESYNC 6 mirrors 000891 1777593744.370980 mapread 0x0358ed thru 0x03b1a9 (0x58bd bytes) 000892 1777593744.376303 mapread 0x02e4a4 thru 0x035691 (0x71ee bytes) 000893 1777593744.379766 read 0x020b9e thru 0x02c0c2 (0xb525 bytes) 000894 1777593744.382422 trunc from 0x03b1aa to 0x02c974 (0xe835 bytes) 000896 1777593744.394889 trunc from 0x02c975 to 0x02719b (0x57d9 bytes) 000897 1777593744.401123 punch from 0x007ceb to 0x010385 (0x869b bytes) 000898 1777593744.403343 mapread 0x004b00 thru 0x013f22 (0xf423 bytes) 000899 1777593744.409984 punch from 0x00640c to 0x00aed9 (0x4ace bytes) 000900 1777593744.412047 trunc from 0x02719c to 0x0231de (0x3fbd bytes) 000901 1777593744.416872 read 0x009610 thru 0x011e10 (0x8801 bytes) 000902 1777593744.420334 punch from 0x01b0e5 to 0x021679 (0x6595 bytes) 000903 1777593744.422390 fallocate 0x011b36 thru 0x01b994 (0x9e5f bytes) 000904 1777593744.424416 punch from 0x01911c to 0x0231de (0xa0c3 bytes) 000905 1777593744.426208 punch from 0x01ef01 to 0x0231de (0x42de bytes) 000906 1777593744.428046 trunc from 0x0231df to 0x01c17a (0x7064 bytes) 000907 1777593744.433095 write 0x03acad thru 0x03c6dc (0x1a30 bytes) 000908 1777593744.434230 fallocate 0x008be2 thru 0x0111e1 (0x8600 bytes) 000909 1777593744.436728 mapread 0x010230 thru 0x0107ba (0x58b bytes) 000910 1777593744.437608 punch from 0x026cf4 to 0x0323fa (0xb707 bytes) 000911 1777593744.439686 trunc from 0x03c6dd to 0x014f22 (0x277ba bytes) 000912 1777593744.446079 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x14f23): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14f23): Mirror 10: 0x1785cfe6 (stale) Mirror 12: 0x1785cfe6 (stale) Mirror 13: 0x1785cfe6 (stale) Mirror 14: 0x1785cfe6 (stale) Mirror 15: 0x1785cfe6 (stale) CRC-32 checksum value for chunk [0, 0x14f23): Mirror 1: 0x515931a1 Mirror 16: 0x515931a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14f23: skipped 000913 1777593744.711759 read 0x003059 thru 0x006a9d (0x3a45 bytes) 000914 1777593744.715869 punch from 0x013235 to 0x014f22 (0x1cee bytes) 000915 1777593744.727782 trunc from 0x014f23 to 0x0163ef (0x14cd bytes) 000916 1777593744.745078 trunc from 0x00d075 to 0x013050 (0x5fdc bytes) 000917 1777593744.747137 read 0x00bb89 thru 0x00c84a (0xcc2 bytes) 000918 1777593744.752533 MIRROR_RESYNC 7 mirrors 000919 1777593745.146086 trunc from 0x003f65 to 0x00704b (0x30e7 bytes) 000920 1777593745.157508 trunc from 0x0163f0 to 0x01698c (0x59d bytes) 000921 1777593745.163299 trunc from 0x0062d1 to 0x00817a (0x1eaa bytes) 000922 1777593745.165659 trunc from 0x00a64d to 0x015ba4 (0xb558 bytes) 000923 1777593745.168051 read 0x005bbe thru 0x0133f8 (0xd83b bytes) 000924 1777593745.171830 punch from 0x0083b2 to 0x00c836 (0x4485 bytes) 000925 1777593745.173962 write 0x038cb9 thru 0x03cddf (0x4127 bytes) 000926 1777593745.175079 trunc from 0x03cde0 to 0x026bf4 (0x161eb bytes) 000927 1777593745.181747 fallocate 0x01d1b1 thru 0x02c3cf (0xf21f bytes) 000928 1777593745.184332 trunc from 0x02adf3 to 0x02c3cf (0x15dd bytes) 000929 1777593745.188357 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c3d0): Mirror 10: 0xac2ed4f9 (stale) Mirror 12: 0xac2ed4f9 (stale) Mirror 13: 0xac2ed4f9 (stale) Mirror 15: 0xac2ed4f9 (stale) Mirror 16: 0xac2ed4f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c3d0): Mirror 10: 0xac2ed4f9 (stale) Mirror 12: 0xac2ed4f9 (stale) Mirror 13: 0xac2ed4f9 (stale) Mirror 15: 0xac2ed4f9 (stale) Mirror 16: 0xac2ed4f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c3d0: skipped 000930 1777593745.339900 trunc from 0x015e61 to 0x0234f1 (0xd691 bytes) 000931 1777593745.342082 trunc from 0x02c3d0 to 0x0265bc (0x5e13 bytes) 000932 1777593745.348534 mapread 0x009527 thru 0x013e56 (0xa930 bytes) 000934 1777593745.355725 punch from 0x00cc40 to 0x01866e (0xba2f bytes) 000935 1777593745.357788 trunc from 0x0265bd to 0x0252e4 (0x12d8 bytes) 000936 1777593745.362997 punch from 0x000e6b to 0x008253 (0x73e9 bytes) 000937 1777593745.365299 mapwrite 0x003839 thru 0x0089b0 (0x5178 bytes) 000938 1777593745.375718 punch from 0x00069d to 0x001798 (0x10fc bytes) 000939 1777593745.377817 trunc from 0x01eee2 to 0x0252e4 (0x6403 bytes) 000940 1777593745.379972 trunc from 0x01f46c to 0x0252e4 (0x5e79 bytes) 000941 1777593745.381705 mapwrite 0x039709 thru 0x03caa0 (0x3398 bytes) 000942 1777593745.396245 read 0x00cfe6 thru 0x010ab3 (0x3ace bytes) 000943 1777593745.399100 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3caa1): Mirror 10: 0xdb278f30 (stale) Mirror 13: 0xdb278f30 (stale) Mirror 15: 0xdb278f30 (stale) Mirror 16: 0xdb278f30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3caa1): Mirror 10: 0xdb278f30 (stale) Mirror 13: 0xdb278f30 (stale) Mirror 15: 0xdb278f30 (stale) Mirror 16: 0xdb278f30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3caa1: skipped 000944 1777593745.524373 write 0x019e11 thru 0x0207d7 (0x69c7 bytes) 000945 1777593745.530520 mapread 0x01e731 thru 0x020d01 (0x25d1 bytes) 000946 1777593745.533546 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3caa1): Mirror 13: 0xdb278f30 (stale) Mirror 15: 0xdb278f30 (stale) Mirror 16: 0xdb278f30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3caa1): Mirror 13: 0xdb278f30 (stale) Mirror 15: 0xdb278f30 (stale) Mirror 16: 0xdb278f30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3caa1: skipped 000947 1777593745.654425 MIRROR_RESYNC 4 mirrors 000948 1777593745.978936 punch from 0x03ca88 to 0x03caa0 (0x19 bytes) 000949 1777593745.991498 read 0x01ec3c thru 0x01fe0d (0x11d2 bytes) 000950 1777593745.993889 mapwrite 0x015b6c thru 0x020a63 (0xaef8 bytes) 000951 1777593746.010518 read 0x03666d thru 0x03caa0 (0x6434 bytes) 000952 1777593746.014057 trunc from 0x03caa1 to 0x0311de (0xb8c2 bytes) 000953 1777593746.025672 trunc from 0x0011e0 to 0x00cd10 (0xbb31 bytes) 000954 1777593746.029157 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x311df): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x311df): Mirror 13: 0xfbdf9e0b (stale) Mirror 15: 0xfbdf9e0b (stale) Mirror 16: 0xfbdf9e0b (stale) CRC-32 checksum value for chunk [0, 0x311df): Mirror 1: 0x53bd5a30 Mirror 17: 0x53bd5a30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x311df: skipped 000955 1777593746.301603 write 0x00c9b4 thru 0x00e8d9 (0x1f26 bytes) 000956 1777593746.329245 read 0x025a12 thru 0x02ca8b (0x707a bytes) 000957 1777593746.333428 fallocate 0x026b10 thru 0x02d005 (0x64f6 bytes) 000958 1777593746.335713 fallocate 0x00a7ab thru 0x012666 (0x7ebc bytes) 000959 1777593746.338414 read 0x025afe thru 0x02c2bc (0x67bf bytes) 000960 1777593746.341226 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x311df): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x311df): Mirror 13: 0xfbdf9e0b (stale) Mirror 15: 0xfbdf9e0b (stale) Mirror 16: 0xfbdf9e0b (stale) Mirror 17: 0x53bd5a30 (stale) CRC-32 checksum value for chunk [0, 0x311df): Mirror 1: 0xfaae5703 Mirror 18: 0xfaae5703 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x311df: skipped 000961 1777593746.646404 trunc from 0x00f485 to 0x01544f (0x5fcb bytes) 000962 1777593746.663883 trunc from 0x010be3 to 0x01e84b (0xdc69 bytes) 000963 1777593746.666777 trunc from 0x0311df to 0x01c6e7 (0x14af7 bytes) 000964 1777593746.681836 mapread 0x012376 thru 0x016cb2 (0x493d bytes) 000965 1777593746.689706 read 0x00e8f4 thru 0x0170f0 (0x87fd bytes) 000966 1777593746.690159 mapread 0x006557 thru 0x015bd5 (0xf67f bytes) 000967 1777593746.694847 trunc from 0x01c6e8 to 0x00e1b4 (0xe533 bytes) 000968 1777593746.700396 mapwrite 0x0164ac thru 0x016c15 (0x76a bytes) 000969 1777593746.714536 mapread 0x013503 thru 0x016c15 (0x3713 bytes) 000970 1777593746.715421 write 0x01e52e thru 0x02a9ea (0xc4bd bytes) 000971 1777593746.716729 fallocate 0x03323c thru 0x03ffff (0xcdc4 bytes) 000972 1777593746.719277 trunc from 0x03dd07 to 0x03ffff (0x22f9 bytes) 000973 1777593746.721253 mapwrite 0x02f2e9 thru 0x039856 (0xa56e bytes) 000974 1777593746.735069 punch from 0x001430 to 0x00698d (0x555e bytes) 000975 1777593746.736967 write 0x00b243 thru 0x00d49a (0x2258 bytes) 000976 1777593746.737869 write 0x03b350 thru 0x03ffff (0x4cb0 bytes) 000977 1777593746.738818 mapread 0x024647 thru 0x0315e4 (0xcf9e bytes) 000978 1777593746.739931 fallocate 0x01eca1 thru 0x02b7fe (0xcb5e bytes) 000979 1777593746.743656 MIRROR_RESYNC 6 mirrors 000980 1777593747.350707 MIRROR_EXTEND to 7 mirrors 000981 1777593747.614883 trunc from 0x040000 to 0x00c690 (0x3396f bytes) 000982 1777593747.632408 trunc from 0x00c691 to 0x0383a7 (0x2bd17 bytes) 000983 1777593747.639323 write 0x02ca65 thru 0x02eed4 (0x2470 bytes) 000984 1777593747.644108 read 0x00a4a2 thru 0x01184a (0x73a9 bytes) 000985 1777593747.649078 trunc from 0x0383a8 to 0x039e71 (0x1aca bytes) 000986 1777593747.655843 fallocate 0x02bd49 thru 0x038a0e (0xccc6 bytes) 000987 1777593747.658158 fallocate 0x02d714 thru 0x031b55 (0x4442 bytes) 000988 1777593747.660395 mapread 0x0125f0 thru 0x0194bf (0x6ed0 bytes) 000989 1777593747.664399 trunc from 0x0066e9 to 0x00d053 (0x696b bytes) 000990 1777593747.667290 trunc from 0x01adcb to 0x023810 (0x8a46 bytes) 000991 1777593747.669941 mapread 0x0017fc thru 0x00bdb7 (0xa5bc bytes) 000992 1777593747.675111 fallocate 0x03a249 thru 0x03ffff (0x5db7 bytes) 000993 1777593747.678969 fallocate 0x02b714 thru 0x02f2a2 (0x3b8f bytes) 000994 1777593747.681974 write 0x0032a2 thru 0x012539 (0xf298 bytes) 000996 1777593747.683976 mapwrite 0x01e382 thru 0x0232b0 (0x4f2f bytes) 000998 1777593747.698308 trunc from 0x039e72 to 0x000e2b (0x39046 bytes) 000999 1777593747.705829 write 0x022ab4 thru 0x02c80a (0x9d57 bytes) 001000 1777593747.707201 fallocate 0x011d6d thru 0x01b76c (0x9a00 bytes) All operations completed A-OK! PASS 70a (36s) == sanity-flr test 71: check mirror extend parent fid ==== 20:02:29 (1777593749) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x4fe:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (4s) == sanity-flr test 72: check append on FLR file ========== 20:02:33 (1777593753) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.084982 s, 24.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037099 s, 28.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 20:02:36 (1777593756) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre2 Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3778432 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1469024 .. done Extending file size to 4542272 .. Extending file size to 6032224 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6196864 .. Extending file size to 6620224 .. failed Extending file size to 7195904 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6217600 .. Extending file size to 8171264 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8365248 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9064128 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8636832 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9195744 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9218624 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62779 62780 62782 62784 62786 done 192.168.203.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre2 (opts:) 192.168.203.104@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (68s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:03:44 (1777593824) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre2 Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5958848 .. Extending file size to 3238304 .. Extending file size to 8912928 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9019840 .. Extending file size to 6360704 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8064128 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9113632 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8710272 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8849888 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8858592 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8973760 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 65784 65785 65787 65789 65791 192.168.203.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre2 (opts:) 192.168.203.104@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:04:48 (1777593888) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2512 1268792 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2636 1268668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24652 3582368 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18784 3588236 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43436 7170604 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2512 1268792 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2636 1268668 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24652 3582368 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18784 3588236 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43436 7170604 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0406306 s, 51.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0552774 s, 37.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x431c17eb (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:04:55 (1777593895) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0337897 s, 62.1 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 20:04:57 (1777593897) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x63:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:05:00 (1777593900) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0254797 s, 82.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x698:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x698:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x699:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:05:02 (1777593902) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0212903 s, 98.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:05:05 (1777593905) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0297509 s, 70.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x699:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x69a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x699:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x69a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x69b:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:05:30 (1777593930) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg304-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200684 s, 52.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } PASS 204d (33s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:06:02 (1777593962) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg304-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0249086 s, 84.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:06:38 (1777593998) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg304-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg304-server: Pool lustre.test_204f created Adding targets to pool oleg304-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg304-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.030352 s, 69.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x69a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x69c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x69a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x69c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg304-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg304-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg304-server: Pool lustre.test_204f destroyed PASS 204f (50s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:07:28 (1777594048) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:07:31 (1777594051) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:07:33 (1777594053) oleg304-server: Pool lustre.test_206 created Adding targets to pool oleg304-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg304-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg304-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg304-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg304-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:07:48 (1777594068) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254392 s, 41.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107173 s, 97.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd6795045 Mirror 3: 0xd6795045 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:07:51 (1777594071) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:07:52 (1777594072) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:07:53 (1777594073) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0136567 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x12a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xaa0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xaa0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xaa0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x11c0000 Read this (4 bytes): BBBB 192.168.203.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre2 (opts:) PASS 209a (17s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:08:10 (1777594090) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00651294 s, 1.3 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:08:13 (1777594093) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157645 s, 66.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611266 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-30 20:08:13.000000000 -0400 Modify: 2026-04-30 20:08:13.000000000 -0400 Change: 2026-04-30 20:08:14.000000000 -0400 Birth: 2026-04-30 20:08:14.000000000 -0400 PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:08:15 (1777594095) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139332 s, 75.3 MB/s oleg304-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg304-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg304-server: O/280000401/d29/13d: File not found by ext2_lookup PASS 210b (12s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:08:27 (1777594107) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0410139 s, 999 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00518515 s, 790 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 30 20:08 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x87:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 30 20:08 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:08:30 (1777594110) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:08:32 (1777594112) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 944 sec ============ 20:08:35 (1777594115) === sanity-flr: start cleanup 20:08:35 (1777594115) === === sanity-flr: finish cleanup 20:08:37 (1777594117) ===