-----============= acceptance-small: sanity-flr ============----- Thu Jun 11 23:34:50 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg234-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 23:35:11 (1781235311) === oleg234-client.virtnet: executing check_config_client /mnt/lustre oleg234-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg234-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ff190d71000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ff190d71000.idle_timeout=debug disable quota as required oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 23:35:51 (1781235351) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8209] == sanity-flr test 0a: lfs mirror create with -N option == 23:35:54 (1781235354) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:36:11 (1781235371) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:36:15 (1781235375) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:36:18 (1781235378) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:36:42 (1781235402) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:36:47 (1781235407) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 23:36:51 (1781235411) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8ff190d71000.stats=clear osc.lustre-OST0001-osc-ffff8ff190d71000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:37:00 (1781235420) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--compress|-Z] [--compress-chunk=SIZE] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:37:13 (1781235433) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:37:24 (1781235444) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 23:37:28 (1781235448) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:37:37 (1781235457) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 23:37:47 (1781235467) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:37:54 (1781235474) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:38:02 (1781235482) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:38:09 (1781235489) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:38:21 (1781235501) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:38:27 (1781235507) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224279 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105323 s, 10.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:38:37 (1781235517) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.197039 s, 26.6 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 23:38:48 (1781235528) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:39:01 (1781235541) Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ff190d71000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ff190d71000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ff190d71000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ff190d71000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (70s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:40:11 (1781235611) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.141561 s, 29.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ff190d71000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ff190d71000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (38s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:40:49 (1781235649) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ff190d71000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ff190d71000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ff190d71000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ff190d71000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (72s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:42:01 (1781235721) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.36184 s, 15.6 MB/s -rw-r--r-- 1 root root 52428800 Jun 11 23:42 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.310s user 0m0.001s sys 0m0.173s Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ff190d71000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ff190d71000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.349s user 0m0.001s sys 0m0.205s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ff190d71000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ff190d71000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.214s user 0m0.001s sys 0m0.178s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg234-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ff190d71000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ff190d71000.ost_server_uuid in FULL state after 0 sec PASS 33b (93s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:43:34 (1781235814) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:43:39 (1781235819) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:43:42 (1781235822) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 23:43:46 (1781235826) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 23:43:54 (1781235834) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.555366 s, 22.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:44:08 (1781235848) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316137 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425533 s, 24.6 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:44:16 (1781235856) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:44:24 (1781235864) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101268 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00125756 s, 814 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00083236 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000980545 s, 1.0 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 23:44:34 (1781235874) create test files with size 15754524 .. 8f0168536e06cc7ca830db102433129a - 04e5300f7c89f8fce1b22c6f056bcd4f - 63b7a6e4c0438fefd879178ca499924a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre PASS 37 (19s) == sanity-flr test 38: resync ============================ 23:44:53 (1781235893) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:45:07 (1781235907) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:45:15 (1781235915) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844873 s, 12.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860489 s, 12.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 23:45:26 (1781235926) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15752 s, 13.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.358626 s, 11.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.177418 s, 11.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 23:45:41 (1781235941) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 23:45:45 (1781235945) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:45:50 (1781235950) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377303 s, 27.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (14s) == sanity-flr test 43c: read prefer uncompressed mirror == 23:46:05 (1781235965) llite.lustre-ffff8ff1923c5000.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (9s) == sanity-flr test 44a: lfs mirror split check =========== 23:46:14 (1781235974) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.440805 s, 7.1 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:46:26 (1781235986) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:46:36 (1781235996) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.210482 s, 49.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:46:46 (1781236006) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8ff1b1e41800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.319227 s, 32.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8ff1919f3800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:47:02 (1781236022) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:47:42 (1781236062) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:48:23 (1781236103) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 23:48:28 (1781236108) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.271269 s, 11.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.171154 s, 18.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8efaaaa4 (nosync,stale) Mirror 2: 0x8efaaaa4 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 7b2f1aafc4c62846365036ec9053f64d - mirror 1 checksum: adf17e7a1f012b7e2e05e26b0ec18861 - mirror 2 checksum: 7b2f1aafc4c62846365036ec9053f64d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 23:48:31 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 23:48:31 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7b2f1aafc4c62846365036ec9053f64d - mirror 1 checksum: adf17e7a1f012b7e2e05e26b0ec18861 - mirror 2 checksum: adf17e7a1f012b7e2e05e26b0ec18861 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 23:48:31 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 23:48:31 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 23:48:39 (1781236119) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572336 s, 18.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:48:47 (1781236127) Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 23:48:59 (1781236139) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 23:49:23 (1781236163) targets are mounted 23:49:23 (1781236163) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632839 s, 16.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 50A (52s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:49:39 (1781236179) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0411532 s, 498 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0162961 s, 1.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:49:48 (1781236188) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0664871 s, 308 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0110517 s, 1.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:49:59 (1781236199) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0247464 s, 662 kB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:50:09 (1781236209) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0652977 s, 314 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:50:18 (1781236218) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00667843 s, 613 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0400459 s, 102 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:50:26 (1781236226) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:50:50 (1781236250) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:51:18 (1781236278) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:51:43 (1781236303) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:52:09 (1781236329) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 23:52:18 (1781236338) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57774 mirror split pid 57775 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5016: 57774 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5016: 57775 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 23:53:28 (1781236408) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 6410 skipping zero length zero range skipping zero length punch hole skipping zero size read truncating to largest ever: 0x38b73 000008 1781236410.644886 trunc from 00000000 to 0x038b72 (0x38b73 bytes) 000009 1781236410.666704 mapwrite 0x03cc2e thru 0x03ffff (0x33d2 bytes) 000010 1781236410.699364 fallocate 0x0182bc thru 0x0224e7 (0xa22c bytes) 000011 1781236410.707691 read 0x03827e thru 0x03bb33 (0x38b6 bytes) 000013 1781236410.709863 read 0x01bb8c thru 0x01e843 (0x2cb8 bytes) 000014 1781236410.720675 mapwrite 0x03c657 thru 0x03f156 (0x2b00 bytes) fallocating to largest ever: 0x3ec18 000015 1781236410.731200 fallocate 0x03ca60 thru 0x03ec17 (0x21b8 bytes) 000016 1781236410.734949 mapread 0x01056b thru 0x013ad3 (0x3569 bytes) 000017 1781236410.736389 mapwrite 0x00fb4e thru 0x01f6dd (0xfb90 bytes) 000018 1781236410.755466 trunc from 0x002476 to 0x00e0b5 (0xbc40 bytes) 000019 1781236410.759817 mapread 0x00d151 thru 0x01a307 (0xd1b7 bytes) 000020 1781236410.767353 write 0x01d79d thru 0x026a11 (0x9275 bytes) 000021 1781236410.768720 write 0x01d3e1 thru 0x02c774 (0xf394 bytes) 000022 1781236410.769906 mapwrite 0x012b1d thru 0x02251d (0xfa01 bytes) 000023 1781236410.784760 fallocate 0x02739d thru 0x029842 (0x24a6 bytes) 000025 1781236410.788771 trunc from 0x040000 to 0x01879f (0x27860 bytes) 000027 1781236410.801067 trunc from 0x0115f2 to 0x012a41 (0x1450 bytes) 000028 1781236410.805056 mapread 0x0078ca thru 0x0105c6 (0x8cfd bytes) 000029 1781236410.826085 mapread 0x014b5e thru 0x01879f (0x3c42 bytes) 000030 1781236410.827812 mapread 0x00eecf thru 0x00f05e (0x190 bytes) 000032 1781236410.829289 fallocate 0x00f549 thru 0x0197db (0xa293 bytes) 000033 1781236410.833272 read 0x00fae3 thru 0x0197db (0x9cf9 bytes) 000034 1781236410.840305 mapread 0x00d39e thru 0x0179fe (0xa661 bytes) 000035 1781236410.841636 punch from 0x014a80 to 0x0197db (0x4d5c bytes) 000036 1781236410.845869 trunc from 0x00136a to 0x00c9dd (0xb674 bytes) 000037 1781236410.849662 trunc from 0x0197dc to 0x02de76 (0x1469b bytes) 000040 1781236410.861360 trunc from 0x02de77 to 0x010355 (0x1db21 bytes) 000041 1781236410.873272 write 0x032fee thru 0x03948b (0x649e bytes) 000042 1781236410.875333 trunc from 0x005900 to 0x01520b (0xf90c bytes) 000044 1781236410.886918 MIRROR_EXTEND to 2 mirrors 000046 1781236411.487776 trunc from 0x014b70 to 0x015e67 (0x12f8 bytes) 000047 1781236411.539495 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3948c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3948c): Mirror 2: 0x1858e588 (stale) CRC-32 checksum value for chunk [0, 0x3948c): Mirror 1: 0x1858e588 Mirror 3: 0x1858e588 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3948c: skipped 000048 1781236412.432373 read 0x018f23 thru 0x01ba15 (0x2af3 bytes) 000049 1781236412.450577 MIRROR_RESYNC 3 mirrors 000050 1781236413.169288 mapwrite 0x0288c1 thru 0x029d95 (0x14d5 bytes) 000051 1781236413.276337 punch from 0x026b21 to 0x0346c4 (0xdba4 bytes) 000053 1781236413.293288 read 0x001846 thru 0x00bab2 (0xa26d bytes) 000054 1781236413.349821 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3948c): Mirror 2: 0x1858e588 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3948c): Mirror 2: 0x1858e588 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3948c: skipped 000055 1781236414.019814 punch from 0x00ef1a to 0x01bfb1 (0xd098 bytes) 000057 1781236414.037842 fallocate 0x023035 thru 0x02f080 (0xc04c bytes) 000058 1781236414.045673 write 0x01cb27 thru 0x02a97f (0xde59 bytes) 000060 1781236414.065962 mapread 0x02f933 thru 0x034862 (0x4f30 bytes) 000061 1781236414.078225 trunc from 0x028cbe to 0x038951 (0xfc94 bytes) 000062 1781236414.104090 punch from 0x027fbb to 0x032a2b (0xaa71 bytes) 000063 1781236414.110747 write 0x0116ae thru 0x01bf1f (0xa872 bytes) 000064 1781236414.124775 trunc from 0x03948c to 0x004039 (0x35452 bytes) 000065 1781236414.146311 punch from 0x003330 to 0x004039 (0xd0a bytes) 000066 1781236414.156648 MIRROR_RESYNC 2 mirrors 000067 1781236414.722009 trunc from 0x0030d8 to 0x004039 (0xf62 bytes) 000068 1781236414.762763 trunc from 0x00403a to 0x008e01 (0x4dc8 bytes) 000069 1781236414.786746 mapwrite 0x02ac60 thru 0x02e63d (0x39de bytes) 000070 1781236414.847056 mapwrite 0x031f34 thru 0x03710c (0x51d9 bytes) 000071 1781236414.929819 mapwrite 0x009dba thru 0x015ab2 (0xbcf9 bytes) 000072 1781236415.015197 mapwrite 0x02c5a1 thru 0x0301c4 (0x3c24 bytes) 000073 1781236415.043059 trunc from 0x024b21 to 0x0314f0 (0xc9d0 bytes) 000074 1781236415.057049 MIRROR_RESYNC 2 mirrors 000075 1781236415.785132 fallocate 0x02903a thru 0x033a98 (0xaa5f bytes) 000076 1781236415.829690 read 0x016583 thru 0x01944b (0x2ec9 bytes) 000077 1781236415.837801 mapread 0x0074d7 thru 0x012085 (0xabaf bytes) 000078 1781236415.866909 write 0x0275bd thru 0x027d10 (0x754 bytes) 000079 1781236415.875879 punch from 0x01a342 to 0x01d00d (0x2ccc bytes) fallocating to largest ever: 0x40000 000081 1781236415.881473 fallocate 0x03e321 thru 0x03ffff (0x1cdf bytes) 000082 1781236415.888403 trunc from 0x004cd9 to 0x00f8ad (0xabd5 bytes) 000083 1781236415.894818 trunc from 0x03710d to 0x00166b (0x35aa1 bytes) 000085 1781236415.952896 trunc from 0x00166c to 0x03cc5e (0x3b5f3 bytes) 000086 1781236415.971879 trunc from 0x00c700 to 0x0140d8 (0x79d9 bytes) 000087 1781236415.985297 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3cc5f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc5f): Mirror 2: 0x4c6a72ba (stale) CRC-32 checksum value for chunk [0, 0x3cc5f): Mirror 1: 0xce7ef66c Mirror 3: 0xce7ef66c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc5f: skipped 000088 1781236417.138300 fallocate 0x00e645 thru 0x01149e (0x2e5a bytes) 000089 1781236417.195030 read 0x0141c0 thru 0x01d69f (0x94e0 bytes) 000090 1781236417.216676 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cc5f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc5f): Mirror 2: 0x4c6a72ba (stale) Mirror 3: 0xce7ef66c (stale) CRC-32 checksum value for chunk [0, 0x3cc5f): Mirror 1: 0xce7ef66c Mirror 4: 0xce7ef66c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc5f: skipped 000091 1781236418.255064 mapread 0x037a4d thru 0x03cc5e (0x5212 bytes) 000092 1781236418.295556 fallocate 0x03881c thru 0x03fc9b (0x7480 bytes) 000093 1781236418.353043 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cc5f): Mirror 3: 0xce7ef66c (stale) Mirror 4: 0xce7ef66c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc5f): Mirror 3: 0xce7ef66c (stale) Mirror 4: 0xce7ef66c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc5f: skipped 000094 1781236418.754288 punch from 0x02d8e3 to 0x032b35 (0x5253 bytes) 000095 1781236418.759575 trunc from 0x03cc5f to 0x013181 (0x29add bytes) 000096 1781236418.781581 fallocate 0x0074cf thru 0x00bd3d (0x486f bytes) 000098 1781236418.796262 MIRROR_RESYNC 3 mirrors 000100 1781236419.472047 trunc from 0x013182 to 0x02fad3 (0x1c952 bytes) 000102 1781236419.618615 punch from 0x02c581 to 0x02fad3 (0x3553 bytes) 000103 1781236419.627674 write 0x01b4c8 thru 0x029236 (0xdd6f bytes) 000104 1781236419.644735 mapread 0x01bec9 thru 0x023faf (0x80e7 bytes) 000105 1781236419.646616 write 0x02b751 thru 0x0367d6 (0xb086 bytes) 000106 1781236419.658917 fallocate 0x00920a thru 0x01896d (0xf764 bytes) 000107 1781236419.663994 read 0x01d1ad thru 0x01f4b9 (0x230d bytes) 000109 1781236419.666265 mapwrite 0x010e79 thru 0x01bbc8 (0xad50 bytes) 000110 1781236419.706010 mapwrite 0x032c0e thru 0x037619 (0x4a0c bytes) 000111 1781236419.776160 mapread 0x011837 thru 0x012062 (0x82c bytes) 000112 1781236419.776970 mapwrite 0x03156f thru 0x033b2a (0x25bc bytes) 000113 1781236419.790186 fallocate 0x02fbd7 thru 0x03cfe0 (0xd40a bytes) 000114 1781236419.796488 trunc from 0x006a6b to 0x00a3e0 (0x3976 bytes) 000115 1781236419.805272 mapread 0x02364f thru 0x02f71b (0xc0cd bytes) 000116 1781236419.809700 fallocate 0x029bfb thru 0x02e87e (0x4c84 bytes) 000117 1781236419.815274 write 0x02f487 thru 0x03579a (0x6314 bytes) 000118 1781236419.817209 punch from 0x030d35 to 0x037619 (0x68e5 bytes) 000119 1781236419.830827 mapwrite 0x02273f thru 0x02eb88 (0xc44a bytes) 000121 1781236419.898620 trunc from 0x03761a to 0x035488 (0x2191 bytes) 000122 1781236419.918463 trunc from 0x035489 to 0x00f41f (0x26069 bytes) 000124 1781236419.941872 mapwrite 0x01ac6d thru 0x01f309 (0x469d bytes) 000125 1781236420.009995 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1f30a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f30a): Mirror 3: 0x7f6ae00f (stale) Mirror 4: 0x7f6ae00f (stale) CRC-32 checksum value for chunk [0, 0x1f30a): Mirror 1: 0x81051eaa Mirror 5: 0x81051eaa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f30a: skipped 000126 1781236421.272101 read 0x003a6f thru 0x009820 (0x5db2 bytes) 000127 1781236421.290371 read 0x019e2b thru 0x01f309 (0x54df bytes) 000128 1781236421.317743 MIRROR_RESYNC 4 mirrors 000129 1781236422.210695 mapwrite 0x020a56 thru 0x0306fa (0xfca5 bytes) 000130 1781236422.375947 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x306fb): Mirror 4: 0x302dc4a8 (stale) Mirror 5: 0x302dc4a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x306fb): Mirror 4: 0x302dc4a8 (stale) Mirror 5: 0x302dc4a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x306fb: skipped 000131 1781236422.901054 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x306fb): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x306fb): Mirror 4: 0x302dc4a8 (stale) Mirror 5: 0x302dc4a8 (stale) CRC-32 checksum value for chunk [0, 0x306fb): Mirror 1: 0xe35ebf93 Mirror 6: 0xe35ebf93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x306fb: skipped 000132 1781236424.074144 mapread 0x01be21 thru 0x022c07 (0x6de7 bytes) 000134 1781236424.117987 punch from 0x01431f to 0x01f2b5 (0xaf97 bytes) 000135 1781236424.186180 read 0x0164c9 thru 0x01b5ec (0x5124 bytes) 000136 1781236424.201123 punch from 0x018c8e to 0x01ff16 (0x7289 bytes) 000137 1781236424.205795 mapwrite 0x03a6ef thru 0x03b702 (0x1014 bytes) 000138 1781236424.277501 trunc from 0x03b703 to 0x01eb61 (0x1cba1 bytes) 000139 1781236424.298494 read 0x0115be thru 0x019411 (0x7e54 bytes) 000140 1781236424.353724 fallocate 0x00bb5e thru 0x014279 (0x871c bytes) 000141 1781236424.360488 trunc from 0x01eb62 to 0x02b8e1 (0xcd80 bytes) 000142 1781236424.375947 trunc from 0x0237cc to 0x025fd5 (0x280a bytes) 000143 1781236424.385373 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b8e2): Mirror 5: 0x2f48549f (stale) Mirror 6: 0xb544dfc3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b8e2): Mirror 5: 0x2f48549f (stale) Mirror 6: 0xb544dfc3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b8e2: skipped 000144 1781236424.703182 write 0x001df7 thru 0x00d041 (0xb24b bytes) 000145 1781236424.718885 trunc from 0x02b8e2 to 0x024317 (0x75ca bytes) 000146 1781236424.744253 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24318): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24318): Mirror 5: 0xe24d5e5 (stale) Mirror 6: 0x366c03bf (stale) CRC-32 checksum value for chunk [0, 0x24318): Mirror 1: 0x34f3450a Mirror 7: 0x34f3450a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24318: skipped 000147 1781236425.963357 trunc from 0x024318 to 0x004fa4 (0x1f373 bytes) 000149 1781236426.037083 write 0x01456c thru 0x01c00b (0x7aa0 bytes) 000150 1781236426.040122 write 0x007c36 thru 0x013337 (0xb702 bytes) 000151 1781236426.053326 write 0x00bbd1 thru 0x010de1 (0x5211 bytes) 000152 1781236426.054642 mapread 0x002c8e thru 0x00a817 (0x7b8a bytes) 000153 1781236426.066862 punch from 0x019ad0 to 0x01c00b (0x253c bytes) 000154 1781236426.085967 write 0x012857 thru 0x01cd7d (0xa527 bytes) 000155 1781236426.094071 mapwrite 0x00db48 thru 0x00e9ae (0xe67 bytes) 000156 1781236426.122395 trunc from 0x01cd7e to 0x010690 (0xc6ed bytes) 000157 1781236426.169994 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10691): Mirror 6: 0x3d85ad85 (stale) Mirror 7: 0xd97e296a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10691): Mirror 6: 0x3d85ad85 (stale) Mirror 7: 0xd97e296a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10691: skipped 000158 1781236426.651901 read 0x00d72f thru 0x010690 (0x2f62 bytes) 000159 1781236426.661501 read 0x00b805 thru 0x010690 (0x4e8c bytes) 000160 1781236426.674868 mapread 0x00db62 thru 0x010690 (0x2b2f bytes) 000161 1781236426.676409 mapread 0x0056e3 thru 0x010690 (0xafae bytes) 000162 1781236426.688691 trunc from 0x010691 to 0x01cf0b (0xc87b bytes) 000163 1781236426.719400 read 0x00adc5 thru 0x00e336 (0x3572 bytes) 000164 1781236426.721309 read 0x012518 thru 0x0139bf (0x14a8 bytes) 000165 1781236426.741079 MIRROR_RESYNC 3 mirrors 000166 1781236427.937037 punch from 0x0076e9 to 0x010661 (0x8f79 bytes) 000167 1781236427.988338 trunc from 0x01407d to 0x01cf0b (0x8e8f bytes) 000168 1781236427.994907 write 0x00a0f5 thru 0x00cf84 (0x2e90 bytes) 000169 1781236428.008185 fallocate 0x0152c0 thru 0x021614 (0xc355 bytes) 000170 1781236428.019280 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21615): Mirror 6: 0xdf13728 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21615): Mirror 6: 0xdf13728 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21615: skipped 000172 1781236428.432346 mapread 0x014373 thru 0x01567c (0x130a bytes) 000173 1781236428.444637 punch from 0x009f72 to 0x00eb36 (0x4bc5 bytes) 000174 1781236428.450488 trunc from 0x021615 to 0x01d20b (0x4409 bytes) 000175 1781236428.480409 MIRROR_RESYNC 2 mirrors 000176 1781236429.647068 trunc from 0x01d20c to 0x010ee1 (0xc32a bytes) 000177 1781236429.722599 trunc from 0x010ee2 to 0x022194 (0x112b3 bytes) 000178 1781236429.749833 read 0x01f934 thru 0x022194 (0x2861 bytes) 000179 1781236429.765370 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22195): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22195): Mirror 6: 0xe37b155a (stale) CRC-32 checksum value for chunk [0, 0x22195): Mirror 1: 0xe37b155a Mirror 7: 0xe37b155a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22195: skipped 000181 1781236431.190450 read 0x01449c thru 0x01a82f (0x6394 bytes) 000182 1781236431.216314 punch from 0x00c81b to 0x017e4c (0xb632 bytes) 000183 1781236431.296472 trunc from 0x022195 to 0x005639 (0x1cb5b bytes) 000184 1781236431.332967 mapwrite 0x023c7e thru 0x02592b (0x1cae bytes) 000185 1781236431.478677 trunc from 0x02592c to 0x01fd99 (0x5b92 bytes) 000186 1781236431.510049 read 0x012f96 thru 0x01fd99 (0xce04 bytes) 000189 1781236431.571254 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1fd9a): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fd9a): Mirror 6: 0xbddea14d (stale) Mirror 7: 0xbddea14d (stale) CRC-32 checksum value for chunk [0, 0x1fd9a): Mirror 1: 0x9fa68e0 Mirror 8: 0x9fa68e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fd9a: skipped 000190 1781236432.825875 trunc from 0x01fd9a to 0x0345c8 (0x1482f bytes) 000191 1781236432.882858 write 0x01aeda thru 0x01cefa (0x2021 bytes) 000193 1781236432.895588 write 0x0296a8 thru 0x02dc6e (0x45c7 bytes) 000194 1781236432.908505 read 0x031340 thru 0x0345c8 (0x3289 bytes) 000195 1781236432.920072 read 0x02d35c thru 0x0345c8 (0x726d bytes) 000196 1781236432.921081 read 0x02a288 thru 0x02c57b (0x22f4 bytes) 000199 1781236432.926758 trunc from 0x0345c9 to 0x0121d9 (0x223ef bytes) 000200 1781236432.968811 mapread 0x00b2b8 thru 0x00c90b (0x1654 bytes) 000201 1781236432.990249 MIRROR_RESYNC 4 mirrors 000202 1781236434.294593 trunc from 0x0121da to 0x00452f (0xdcaa bytes) 000203 1781236434.359428 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4530): Mirror 6: 0xcd043aed (stale) Mirror 7: 0xcd043aed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4530): Mirror 6: 0xcd043aed (stale) Mirror 7: 0xcd043aed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4530: skipped 000204 1781236434.792487 read 0x00214f thru 0x00452f (0x23e1 bytes) 000205 1781236434.818960 write 0x029cbd thru 0x034ac0 (0xae04 bytes) 000206 1781236434.831067 mapread 0x01f6d1 thru 0x02135b (0x1c8b bytes) 000207 1781236434.850107 trunc from 0x0333ee to 0x034ac0 (0x16d3 bytes) 000208 1781236434.864128 read 0x01ccd0 thru 0x022ec4 (0x61f5 bytes) 000210 1781236434.868018 mapread 0x01342b thru 0x021589 (0xe15f bytes) 000212 1781236434.923724 mapread 0x0297c0 thru 0x032e71 (0x96b2 bytes) 000213 1781236434.931294 mapread 0x0164a6 thru 0x01e6ac (0x8207 bytes) 000214 1781236434.953880 MIRROR_RESYNC 3 mirrors 000216 1781236436.089566 read 0x02e580 thru 0x030c4a (0x26cb bytes) 000217 1781236436.094578 trunc from 0x034ac1 to 0x01831f (0x1c7a1 bytes) 000218 1781236436.142845 MIRROR_RESYNC 3 mirrors 000219 1781236437.381755 write 0x03fcda thru 0x03ffff (0x326 bytes) 000220 1781236437.445483 write 0x028f51 thru 0x032f00 (0x9fb0 bytes) 000221 1781236437.470983 fallocate 0x031251 thru 0x03ffff (0xedaf bytes) 000222 1781236437.484694 mapread 0x003edf thru 0x00d04b (0x916d bytes) 000223 1781236437.522662 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x7386824b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x7386824b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000224 1781236437.935273 write 0x015ab6 thru 0x01ee23 (0x936e bytes) 000225 1781236437.992757 read 0x031e52 thru 0x03d0bb (0xb26a bytes) 000226 1781236438.017169 mapread 0x028b0d thru 0x02b8ea (0x2dde bytes) 000227 1781236438.027911 fallocate 0x028254 thru 0x0374f6 (0xf2a3 bytes) 000228 1781236438.056171 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) 000229 1781236438.330061 trunc from 0x007002 to 0x007fbe (0xfbd bytes) 000230 1781236438.336865 write 0x002b91 thru 0x00e4d7 (0xb947 bytes) 000231 1781236438.360000 mapread 0x039aef thru 0x03d24b (0x375d bytes) 000232 1781236438.386429 punch from 0x023739 to 0x025b76 (0x243e bytes) 000233 1781236438.395095 read 0x030442 thru 0x03d2f4 (0xceb3 bytes) 000235 1781236438.415068 read 0x02f76d thru 0x03709d (0x7931 bytes) 000236 1781236438.428341 read 0x030602 thru 0x03ae40 (0xa83f bytes) 000238 1781236438.431362 mapwrite 0x037592 thru 0x03ffff (0x8a6e bytes) 000239 1781236438.474264 trunc from 0x034415 to 0x037774 (0x3360 bytes) 000240 1781236438.485445 mapread 0x01aa3e thru 0x024d55 (0xa318 bytes) 000241 1781236438.486579 mapread 0x025581 thru 0x02cd78 (0x77f8 bytes) 000242 1781236438.504424 punch from 0x02893e to 0x02ee10 (0x64d3 bytes) 000243 1781236438.515230 trunc from 0x005511 to 0x012567 (0xd057 bytes) 000244 1781236438.530245 fallocate 0x019ec5 thru 0x027ebc (0xdff8 bytes) 000245 1781236438.542216 mapread 0x02f10d thru 0x03e539 (0xf42d bytes) 000246 1781236438.567011 punch from 0x01ce70 to 0x020a6a (0x3bfb bytes) 000247 1781236438.588809 fallocate 0x017266 thru 0x0247ac (0xd547 bytes) 000250 1781236438.608709 trunc from 0x025750 to 0x034cd6 (0xf587 bytes) 000251 1781236438.615212 write 0x0239b1 thru 0x025cb7 (0x2307 bytes) 000252 1781236438.631009 fallocate 0x02d58c thru 0x03251f (0x4f94 bytes) 000253 1781236438.639287 fallocate 0x034f4a thru 0x03ffff (0xb0b6 bytes) 000254 1781236438.647705 trunc from 0x040000 to 0x0109ea (0x2f615 bytes) 000256 1781236438.687303 write 0x03764d thru 0x03c630 (0x4fe4 bytes) 000257 1781236438.689305 trunc from 0x03c631 to 0x03951a (0x3116 bytes) 000258 1781236438.711189 write 0x032877 thru 0x03ffff (0xd789 bytes) 000259 1781236438.728568 punch from 0x013780 to 0x019acc (0x634d bytes) 000260 1781236438.745623 fallocate 0x018bad thru 0x01ec6c (0x60c0 bytes) 000262 1781236438.760569 trunc from 0x040000 to 0x038575 (0x7a8a bytes) 000263 1781236438.810628 fallocate 0x03e9e4 thru 0x03ffff (0x161c bytes) 000264 1781236438.824742 fallocate 0x0242b8 thru 0x02e886 (0xa5cf bytes) 000265 1781236438.831868 read 0x01809d thru 0x01ffe0 (0x7f44 bytes) 000266 1781236438.852317 mapread 0x031fec thru 0x03846a (0x647f bytes) 000267 1781236438.858862 punch from 0x00c7d7 to 0x0133ac (0x6bd6 bytes) 000268 1781236438.873262 mapwrite 0x00f40d thru 0x00f496 (0x8a bytes) 000269 1781236438.902436 fallocate 0x027d40 thru 0x02e8f9 (0x6bba bytes) 000270 1781236438.914065 fallocate 0x032689 thru 0x03adfb (0x8773 bytes) 000271 1781236438.925613 mapread 0x00932c thru 0x014663 (0xb338 bytes) 000272 1781236438.969347 trunc from 0x0175e4 to 0x01fcf8 (0x8715 bytes) 000273 1781236438.977581 trunc from 0x038576 to 0x009c2a (0x2e94b bytes) 000274 1781236439.027031 MIRROR_EXTEND to 2 mirrors 000275 1781236439.914395 punch from 0x006adc to 0x009c2a (0x314f bytes) 000276 1781236439.993461 mapwrite 0x0003de thru 0x00a26c (0x9e8f bytes) 000277 1781236440.173648 fallocate 0x01b3f9 thru 0x01da81 (0x2689 bytes) 000278 1781236440.184295 punch from 0x005173 to 0x005f57 (0xde5 bytes) 000279 1781236440.197684 write 0x0189c8 thru 0x02282c (0x9e65 bytes) 000280 1781236440.202028 trunc from 0x010df9 to 0x01cc82 (0xbe8a bytes) 000281 1781236440.236844 mapwrite 0x01d39b thru 0x01e2ee (0xf54 bytes) 000282 1781236440.262405 mapread 0x01aabe thru 0x02282c (0x7d6f bytes) 000283 1781236440.307136 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) 000284 1781236440.612050 trunc from 0x00de58 to 0x018657 (0xa800 bytes) 000287 1781236440.622249 trunc from 0x01c134 to 0x02282c (0x66f9 bytes) 000288 1781236440.630468 trunc from 0x008142 to 0x00ffd3 (0x7e92 bytes) 000289 1781236440.642651 fallocate 0x0274b7 thru 0x031aec (0xa636 bytes) 000290 1781236440.665184 MIRROR_EXTEND to 2 mirrors 000291 1781236441.779260 fallocate 0x01c972 thru 0x02918d (0xc81c bytes) 000292 1781236441.818721 mapwrite 0x026bd9 thru 0x02b666 (0x4a8e bytes) 000293 1781236441.860563 fallocate 0x033661 thru 0x03ffff (0xc99f bytes) 000294 1781236441.867584 trunc from 0x01a78a to 0x01b9ce (0x1245 bytes) 000295 1781236441.874242 fallocate 0x00e09d thru 0x01c1be (0xe122 bytes) 000296 1781236441.882354 read 0x013dee thru 0x01cfb3 (0x91c6 bytes) 000297 1781236441.896729 punch from 0x0398ad to 0x03facf (0x6223 bytes) 000298 1781236441.913148 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3df11100 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdd601ab5 Mirror 3: 0xdd601ab5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000299 1781236443.468802 mapread 0x03f187 thru 0x03ffff (0xe79 bytes) 000300 1781236443.476877 read 0x02f28e thru 0x0391b6 (0x9f29 bytes) 000301 1781236443.513401 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) 000302 1781236443.984441 punch from 0x00f58a to 0x01b161 (0xbbd8 bytes) 000303 1781236444.065041 fallocate 0x03944b thru 0x03f9f9 (0x65af bytes) 000305 1781236444.070666 trunc from 0x040000 to 0x012079 (0x2df86 bytes) 000306 1781236444.120888 mapwrite 0x029fae thru 0x038742 (0xe795 bytes) 000308 1781236444.263707 fallocate 0x01d283 thru 0x01e5e4 (0x1362 bytes) 000309 1781236444.270661 write 0x032d37 thru 0x0380bb (0x5385 bytes) 000310 1781236444.271634 read 0x034635 thru 0x038742 (0x410e bytes) 000311 1781236444.273811 punch from 0x0251f7 to 0x02d172 (0x7f7c bytes) 000312 1781236444.282872 fallocate 0x012062 thru 0x018497 (0x6436 bytes) 000313 1781236444.289277 read 0x00d0de thru 0x015270 (0x8193 bytes) 000314 1781236444.305282 fallocate 0x027f60 thru 0x033c04 (0xbca5 bytes) 000315 1781236444.311786 read 0x000ace thru 0x00f20d (0xe740 bytes) 000316 1781236444.312583 write 0x02bae7 thru 0x033ed2 (0x83ec bytes) 000317 1781236444.314499 write 0x038bfe thru 0x03ffff (0x7402 bytes) 000318 1781236444.316595 trunc from 0x040000 to 0x004225 (0x3bdda bytes) 000319 1781236444.348975 trunc from 0x004226 to 0x038091 (0x33e6c bytes) 000320 1781236444.411182 trunc from 0x028f48 to 0x03128d (0x8346 bytes) 000321 1781236444.423715 write 0x00a0d2 thru 0x00cb0f (0x2a3e bytes) 000322 1781236444.447203 write 0x01e586 thru 0x020e2a (0x28a5 bytes) 000324 1781236444.461369 trunc from 0x0130d3 to 0x01aed4 (0x7e02 bytes) 000325 1781236444.479700 punch from 0x010df8 to 0x013c5e (0x2e67 bytes) 000326 1781236444.498286 punch from 0x03353e to 0x0370b3 (0x3b76 bytes) 000327 1781236444.505258 write 0x0373a9 thru 0x03ffff (0x8c57 bytes) 000328 1781236444.519768 MIRROR_RESYNC 2 mirrors 000329 1781236445.287951 trunc from 0x0170de to 0x0237b0 (0xc6d3 bytes) 000330 1781236445.376434 read 0x01ab3f thru 0x0234b1 (0x8973 bytes) 000331 1781236445.422236 MIRROR_RESYNC 2 mirrors 000333 1781236446.287256 trunc from 0x040000 to 0x00d88a (0x32775 bytes) 000334 1781236446.390998 trunc from 0x00d88b to 0x032dd5 (0x2554b bytes) 000335 1781236446.466648 mapread 0x015f56 thru 0x01ca91 (0x6b3c bytes) 000336 1781236446.518480 trunc from 0x032dd6 to 0x03b067 (0x8292 bytes) 000339 1781236446.567278 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) 000340 1781236446.781095 read 0x0184d1 thru 0x025310 (0xce40 bytes) 000341 1781236446.792006 trunc from 0x03b068 to 0x021c1d (0x1944a bytes) 000342 1781236446.861354 write 0x03f99e thru 0x03ffff (0x662 bytes) 000343 1781236446.865369 mapwrite 0x02fc2f thru 0x033b91 (0x3f63 bytes) 000344 1781236446.938576 fallocate 0x024227 thru 0x02beea (0x7cc4 bytes) 000345 1781236446.947497 fallocate 0x030c19 thru 0x036d9b (0x6183 bytes) 000346 1781236446.956406 punch from 0x01fc4a to 0x024b95 (0x4f4c bytes) 000347 1781236446.964418 mapread 0x01e938 thru 0x02658f (0x7c58 bytes) 000348 1781236446.983329 write 0x0320a8 thru 0x0393ef (0x7348 bytes) 000349 1781236446.985541 mapread 0x0336d4 thru 0x038f4f (0x587c bytes) 000351 1781236446.990142 trunc from 0x01f650 to 0x02e112 (0xeac3 bytes) 000352 1781236446.996578 trunc from 0x040000 to 0x03aac2 (0x553d bytes) 000354 1781236447.028381 mapwrite 0x031177 thru 0x0358cf (0x4759 bytes) 000355 1781236447.050307 trunc from 0x01cefc to 0x01fae1 (0x2be6 bytes) 000358 1781236447.059261 write 0x03bc17 thru 0x03ffff (0x43e9 bytes) 000359 1781236447.060705 trunc from 0x03244c to 0x03ffff (0xdbb4 bytes) 000360 1781236447.076972 write 0x033de3 thru 0x03e7da (0xa9f8 bytes) 000361 1781236447.097478 mapwrite 0x006276 thru 0x01558a (0xf315 bytes) 000362 1781236447.147811 mapwrite 0x01ef3a thru 0x026389 (0x7450 bytes) 000363 1781236447.178440 trunc from 0x019f10 to 0x029c86 (0xfd77 bytes) 000365 1781236447.186610 trunc from 0x0110f7 to 0x0187bc (0x76c6 bytes) 000366 1781236447.192190 mapwrite 0x0352cc thru 0x03fd2b (0xaa60 bytes) 000367 1781236447.225478 fallocate 0x03b938 thru 0x03fdb8 (0x4481 bytes) 000368 1781236447.235646 read 0x006c47 thru 0x009dd2 (0x318c bytes) 000370 1781236447.238966 write 0x002ea4 thru 0x00de1b (0xaf78 bytes) 000371 1781236447.241408 trunc from 0x0343f2 to 0x03ffff (0xbc0e bytes) 000372 1781236447.247116 punch from 0x00865b to 0x0088f1 (0x297 bytes) 000373 1781236447.265687 read 0x02985b thru 0x0380aa (0xe850 bytes) 000374 1781236447.297567 mapread 0x0353b1 thru 0x03ffff (0xac4f bytes) 000375 1781236447.299297 punch from 0x0192b4 to 0x01c8e2 (0x362f bytes) 000376 1781236447.305740 write 0x022969 thru 0x032947 (0xffdf bytes) 000377 1781236447.308386 write 0x00824f thru 0x00df1a (0x5ccc bytes) 000378 1781236447.310638 read 0x01b045 thru 0x028836 (0xd7f2 bytes) 000379 1781236447.318529 trunc from 0x040000 to 0x0294ae (0x16b51 bytes) 000382 1781236447.354413 write 0x0110d9 thru 0x01c2d0 (0xb1f8 bytes) 000383 1781236447.361374 trunc from 0x0294af to 0x004204 (0x252aa bytes) 000384 1781236447.401207 mapwrite 0x035852 thru 0x03de9a (0x8649 bytes) 000385 1781236447.504362 fallocate 0x02e444 thru 0x03cdce (0xe98b bytes) 000386 1781236447.511662 mapread 0x02293c thru 0x031752 (0xee17 bytes) 000387 1781236447.598789 trunc from 0x03de9b to 0x036a0c (0x748e bytes) 000388 1781236447.617459 punch from 0x02c068 to 0x02e4d3 (0x246c bytes) 000389 1781236447.623444 trunc from 0x036a0d to 0x00df1a (0x28af2 bytes) 000390 1781236447.644231 mapread 0x002123 thru 0x003fdb (0x1eb9 bytes) 000391 1781236447.645031 trunc from 0x008335 to 0x00861d (0x2e9 bytes) 000392 1781236447.649616 trunc from 0x00df1b to 0x030039 (0x2211f bytes) 000393 1781236447.664013 trunc from 0x03003a to 0x02409b (0xbf9e bytes) 000394 1781236447.694833 read 0x011686 thru 0x014327 (0x2ca2 bytes) 000395 1781236447.714944 trunc from 0x02409c to 0x03a3dc (0x16341 bytes) 000397 1781236447.740457 MIRROR_EXTEND to 2 mirrors 000398 1781236448.552712 read 0x000096 thru 0x001598 (0x1503 bytes) 000399 1781236448.558198 mapwrite 0x01e711 thru 0x02c192 (0xda82 bytes) 000400 1781236448.681257 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) 000401 1781236448.868298 write 0x015d48 thru 0x01fc92 (0x9f4b bytes) 000402 1781236448.913702 mapread 0x01db43 thru 0x02caa9 (0xef67 bytes) 000403 1781236448.944407 fallocate 0x01ff04 thru 0x01ff60 (0x5d bytes) 000404 1781236448.956024 fallocate 0x03fa53 thru 0x03ff62 (0x510 bytes) 000405 1781236448.962278 mapwrite 0x03873e thru 0x03b890 (0x3153 bytes) 000406 1781236448.995611 mapread 0x02ff9a thru 0x03e3a0 (0xe407 bytes) 000407 1781236448.997880 fallocate 0x00b63d thru 0x016522 (0xaee6 bytes) 000408 1781236449.004378 fallocate 0x024c66 thru 0x025e0d (0x11a8 bytes) 000409 1781236449.011171 mapread 0x02bd63 thru 0x032043 (0x62e1 bytes) 000413 1781236449.019635 punch from 0x031af9 to 0x0358a4 (0x3dac bytes) 000414 1781236449.027228 trunc from 0x03ff63 to 0x003483 (0x3cadf bytes) 000415 1781236449.049857 punch from 0x000a4e to 0x003483 (0x2a36 bytes) 000416 1781236449.060221 trunc from 0x003484 to 0x025145 (0x21cc2 bytes) 000418 1781236449.099099 write 0x0134c8 thru 0x01e59c (0xb0d5 bytes) 000419 1781236449.126935 write 0x02e3be thru 0x03aa12 (0xc655 bytes) 000420 1781236449.131444 trunc from 0x03aa13 to 0x003bf4 (0x36e1e bytes) 000421 1781236449.179179 trunc from 0x002f42 to 0x003bf4 (0xcb3 bytes) 000422 1781236449.190311 write 0x014ff4 thru 0x018a49 (0x3a56 bytes) 000423 1781236449.191724 write 0x019585 thru 0x0239fc (0xa478 bytes) 000425 1781236449.201088 trunc from 0x0239fd to 0x004e5e (0x1eb9e bytes) 000426 1781236449.246782 punch from 0x004d83 to 0x004e5e (0xdc bytes) 000427 1781236449.258993 mapread 0x004710 thru 0x004e5e (0x74f bytes) 000428 1781236449.273631 read 0x000b0c thru 0x004837 (0x3d2c bytes) 000429 1781236449.286011 mapwrite 0x0275da thru 0x031d16 (0xa73d bytes) 000430 1781236449.406167 punch from 0x00716a to 0x015ee1 (0xed78 bytes) 000431 1781236449.412215 write 0x01cabb thru 0x022a0b (0x5f51 bytes) 000432 1781236449.428427 read 0x018c52 thru 0x01ead1 (0x5e80 bytes) 000433 1781236449.429063 write 0x026377 thru 0x02b687 (0x5311 bytes) 000434 1781236449.429810 fallocate 0x02e25b thru 0x03b8f5 (0xd69b bytes) 000436 1781236449.439348 write 0x02345a thru 0x030092 (0xcc39 bytes) 000438 1781236449.452427 write 0x00824a thru 0x00842f (0x1e6 bytes) 000439 1781236449.484402 read 0x0182ae thru 0x01d530 (0x5283 bytes) 000440 1781236449.485095 trunc from 0x01ef0a to 0x028aa2 (0x9b99 bytes) 000441 1781236449.506618 mapwrite 0x0336a7 thru 0x03ffff (0xc959 bytes) 000442 1781236449.587957 mapread 0x01ec77 thru 0x0249c1 (0x5d4b bytes) 000443 1781236449.594503 MIRROR_EXTEND to 2 mirrors 000444 1781236450.436712 mapread 0x00f6f9 thru 0x0155d6 (0x5ede bytes) 000445 1781236450.460914 read 0x009e7c thru 0x0161c3 (0xc348 bytes) 000446 1781236450.468814 fallocate 0x024515 thru 0x02f10f (0xabfb bytes) 000447 1781236450.506428 punch from 0x03b1dc to 0x03ffff (0x4e24 bytes) 000448 1781236450.518096 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) 000449 1781236450.667020 fallocate 0x02ccd0 thru 0x03afae (0xe2df bytes) 000450 1781236450.678323 punch from 0x027748 to 0x027e36 (0x6ef bytes) 000451 1781236450.688728 mapread 0x0087dd thru 0x01731e (0xeb42 bytes) 000452 1781236450.736227 punch from 0x035179 to 0x036323 (0x11ab bytes) 000453 1781236450.748256 mapread 0x026c0a thru 0x02aa99 (0x3e90 bytes) 000454 1781236450.767107 fallocate 0x01207e thru 0x0212fb (0xf27e bytes) 000455 1781236450.777691 trunc from 0x040000 to 0x017207 (0x28df8 bytes) 000456 1781236450.807149 mapwrite 0x02bd81 thru 0x02e108 (0x2388 bytes) 000457 1781236450.849704 trunc from 0x02e109 to 0x00bcc0 (0x22448 bytes) 000458 1781236450.875308 punch from 0x008b42 to 0x00bcc0 (0x317f bytes) 000459 1781236450.889673 write 0x031076 thru 0x0335a8 (0x2533 bytes) 000460 1781236450.891641 read 0x00036f thru 0x0022a8 (0x1f3a bytes) 000461 1781236450.906235 punch from 0x026468 to 0x02dbd9 (0x7772 bytes) 000462 1781236450.915899 mapread 0x027841 thru 0x02db43 (0x6303 bytes) 000463 1781236450.941338 punch from 0x0028da to 0x0062f6 (0x3a1d bytes) 000464 1781236450.948355 trunc from 0x00e061 to 0x01a0bb (0xc05b bytes) 000465 1781236450.966320 write 0x033321 thru 0x033644 (0x324 bytes) 000466 1781236450.968285 mapwrite 0x009555 thru 0x012e24 (0x98d0 bytes) 000467 1781236451.035707 fallocate 0x01186f thru 0x01410a (0x289c bytes) 000468 1781236451.049298 mapread 0x005f66 thru 0x011ae1 (0xbb7c bytes) 000469 1781236451.052093 fallocate 0x00323f thru 0x006b43 (0x3905 bytes) 000470 1781236451.059563 mapwrite 0x02d464 thru 0x03669e (0x923b bytes) 000472 1781236451.109230 read 0x02c51e thru 0x03669e (0xa181 bytes) 000473 1781236451.109958 fallocate 0x034eef thru 0x03b3d3 (0x64e5 bytes) 000475 1781236451.119483 read 0x02d7be thru 0x03669e (0x8ee1 bytes) 000476 1781236451.121194 mapwrite 0x0183ff thru 0x02761d (0xf21f bytes) 000477 1781236451.156486 trunc from 0x03669f to 0x017a89 (0x1ec15 bytes) 000479 1781236451.187785 trunc from 0x017a8a to 0x01246d (0x561c bytes) 000480 1781236451.208270 trunc from 0x01246e to 0x00d758 (0x4d15 bytes) 000481 1781236451.230274 mapread 0x004124 thru 0x00d758 (0x9635 bytes) 000483 1781236451.235949 trunc from 0x00d759 to 0x0335cc (0x25e74 bytes) 000484 1781236451.255665 punch from 0x00e324 to 0x012bde (0x48bb bytes) 000485 1781236451.265810 mapwrite 0x023340 thru 0x02dfbd (0xac7e bytes) 000486 1781236451.404146 read 0x02b046 thru 0x0335cc (0x8587 bytes) 000487 1781236451.445486 trunc from 0x0008ca to 0x00a47c (0x9bb3 bytes) 000488 1781236451.462249 mapwrite 0x00edf0 thru 0x014ef5 (0x6106 bytes) 000489 1781236451.509611 trunc from 0x00ab50 to 0x00e4b7 (0x3968 bytes) 000490 1781236451.517300 read 0x007f9d thru 0x01412a (0xc18e bytes) 000491 1781236451.552575 write 0x0247d1 thru 0x028213 (0x3a43 bytes) 000492 1781236451.559667 mapread 0x00a4c5 thru 0x01486b (0xa3a7 bytes) 000493 1781236451.565086 trunc from 0x0335cd to 0x02305b (0x10571 bytes) 000495 1781236451.611098 write 0x0055d1 thru 0x014c74 (0xf6a4 bytes) 000496 1781236451.615297 trunc from 0x02305c to 0x020e08 (0x2253 bytes) 000497 1781236451.654219 trunc from 0x020e09 to 0x02c1a3 (0xb39b bytes) 000498 1781236451.693909 punch from 0x01af39 to 0x01b5f7 (0x6bf bytes) 000499 1781236451.701747 punch from 0x0283ba to 0x02c1a3 (0x3dea bytes) 000500 1781236451.714813 read 0x008764 thru 0x00bcb5 (0x3552 bytes) 000501 1781236451.715464 mapread 0x003f52 thru 0x00906f (0x511e bytes) 000503 1781236451.730957 mapwrite 0x00391f thru 0x008230 (0x4912 bytes) 000505 1781236451.782486 read 0x0166de thru 0x0181f3 (0x1b16 bytes) 000506 1781236451.783201 trunc from 0x000517 to 0x00ea31 (0xe51b bytes) 000507 1781236451.794269 punch from 0x027728 to 0x02c1a3 (0x4a7c bytes) 000508 1781236451.810402 trunc from 0x02947b to 0x02c1a3 (0x2d29 bytes) 000509 1781236451.820401 trunc from 0x028f1c to 0x02c1a3 (0x3288 bytes) 000510 1781236451.832122 mapwrite 0x028542 thru 0x02d42d (0x4eec bytes) 000511 1781236451.911431 write 0x00a7fc thru 0x0180df (0xd8e4 bytes) 000512 1781236451.915635 mapwrite 0x005143 thru 0x009ce7 (0x4ba5 bytes) 000513 1781236451.960966 read 0x003b23 thru 0x011424 (0xd902 bytes) 000514 1781236451.964214 fallocate 0x005ec2 thru 0x00afcb (0x510a bytes) 000515 1781236451.997227 MIRROR_EXTEND to 2 mirrors 000516 1781236453.130333 mapwrite 0x002a77 thru 0x00d547 (0xaad1 bytes) 000517 1781236453.310603 trunc from 0x02d42e to 0x02608c (0x73a1 bytes) 000518 1781236453.350416 write 0x03418b thru 0x035b43 (0x19b9 bytes) 000519 1781236453.359224 trunc from 0x035b44 to 0x00daa0 (0x280a3 bytes) 000520 1781236453.393774 punch from 0x004398 to 0x00daa0 (0x9709 bytes) 000521 1781236453.434329 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xdaa1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdaa1): Mirror 2: 0x7955177d (stale) CRC-32 checksum value for chunk [0, 0xdaa1): Mirror 1: 0xfad2d5b0 Mirror 3: 0xfad2d5b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdaa1: skipped 000522 1781236454.607214 fallocate 0x010609 thru 0x01c729 (0xc121 bytes) 000523 1781236454.664685 trunc from 0x01c72a to 0x0319cc (0x152a3 bytes) 000525 1781236454.712980 write 0x03b3ad thru 0x03ee0f (0x3a63 bytes) 000526 1781236454.714296 punch from 0x03143d to 0x03a606 (0x91ca bytes) 000527 1781236454.726187 mapwrite 0x0000a0 thru 0x009c7d (0x9bde bytes) 000528 1781236454.783866 read 0x01a916 thru 0x0278f0 (0xcfdb bytes) 000529 1781236454.795469 trunc from 0x03ee10 to 0x02ade6 (0x14029 bytes) 000531 1781236454.813843 write 0x02f3e7 thru 0x0315ae (0x21c8 bytes) 000533 1781236454.815647 fallocate 0x03eea5 thru 0x03ef20 (0x7c bytes) 000534 1781236454.821779 write 0x035ffd thru 0x03e909 (0x890d bytes) 000535 1781236454.837712 read 0x00a5d9 thru 0x013969 (0x9391 bytes) 000536 1781236454.853170 trunc from 0x03ef21 to 0x00c7fa (0x32726 bytes) 000537 1781236454.874630 write 0x026b82 thru 0x036830 (0xfcaf bytes) 000539 1781236454.879721 mapread 0x0298ad thru 0x036830 (0xcf84 bytes) 000540 1781236454.884071 trunc from 0x01c5a3 to 0x02669f (0xa0fd bytes) 000542 1781236454.905253 mapread 0x00062b thru 0x007d53 (0x7729 bytes) 000544 1781236454.908645 trunc from 0x0197de to 0x025688 (0xbeab bytes) 000545 1781236454.914891 read 0x020f8f thru 0x022868 (0x18da bytes) 000546 1781236454.961075 trunc from 0x021b79 to 0x02e9f0 (0xce78 bytes) 000547 1781236454.967210 read 0x01efb8 thru 0x02862a (0x9673 bytes) 000548 1781236454.987270 trunc from 0x036831 to 0x026da0 (0xfa90 bytes) 000549 1781236455.017375 MIRROR_RESYNC 3 mirrors 000550 1781236455.822688 write 0x014b0c thru 0x0170d8 (0x25cd bytes) 000551 1781236455.888241 write 0x009b6c thru 0x01595a (0xbdef bytes) 000552 1781236455.909735 MIRROR_RESYNC 3 mirrors 000553 1781236457.001732 write 0x024321 thru 0x03359c (0xf27c bytes) 000554 1781236457.165690 MIRROR_RESYNC 3 mirrors 000555 1781236458.102852 mapread 0x005eb9 thru 0x0155c1 (0xf709 bytes) 000556 1781236458.131674 trunc from 0x02a505 to 0x02c769 (0x2265 bytes) 000557 1781236458.178542 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3359d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3359d): Mirror 2: 0xda7c986d (stale) Mirror 3: 0xda7c986d (stale) CRC-32 checksum value for chunk [0, 0x3359d): Mirror 1: 0x256aeaf7 Mirror 4: 0x256aeaf7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3359d: skipped 000558 1781236459.439483 write 0x00005e thru 0x00f740 (0xf6e3 bytes) 000559 1781236459.559095 mapwrite 0x01353c thru 0x01dd75 (0xa83a bytes) 000560 1781236459.702680 read 0x010359 thru 0x01f8e1 (0xf589 bytes) 000561 1781236459.707015 mapread 0x02d21a thru 0x02f090 (0x1e77 bytes) 000562 1781236459.728271 punch from 0x0005a5 to 0x00d389 (0xcde5 bytes) 000563 1781236459.735261 mapread 0x01fc92 thru 0x026283 (0x65f2 bytes) 000564 1781236459.739034 mapwrite 0x030f28 thru 0x0319a8 (0xa81 bytes) 000565 1781236459.753676 trunc from 0x03359d to 0x00473c (0x2ee60 bytes) 000566 1781236459.783061 mapread 0x003f67 thru 0x00473c (0x7d6 bytes) 000567 1781236459.791288 fallocate 0x01d6fd thru 0x02a95c (0xd260 bytes) 000568 1781236459.803273 mapwrite 0x0045cb thru 0x013a48 (0xf47e bytes) 000569 1781236459.892917 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2a95d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a95d): Mirror 2: 0xf58840c2 (stale) Mirror 3: 0xf58840c2 (stale) Mirror 4: 0x314acde3 (stale) CRC-32 checksum value for chunk [0, 0x2a95d): Mirror 1: 0x10218227 Mirror 5: 0x10218227 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a95d: skipped 000570 1781236461.035441 read 0x00d933 thru 0x016e9a (0x9568 bytes) 000571 1781236461.049361 read 0x017dfb thru 0x01a6a7 (0x28ad bytes) 000572 1781236461.060980 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a95d): Mirror 2: 0xf58840c2 (stale) Mirror 4: 0x314acde3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a95d): Mirror 2: 0xf58840c2 (stale) Mirror 4: 0x314acde3 (stale) CRC-32 checksum value for chunk [0, 0x2a95d): Mirror 1: 0x10218227 Mirror 5: 0x10218227 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a95d: skipped 000573 1781236461.714928 write 0x0208cf thru 0x02df1f (0xd651 bytes) 000574 1781236461.866377 mapread 0x00bebb thru 0x0113dd (0x5523 bytes) 000575 1781236461.903886 fallocate 0x03b551 thru 0x03ffff (0x4aaf bytes) 000576 1781236461.913247 mapread 0x002f26 thru 0x00fd51 (0xce2c bytes) 000577 1781236461.939623 fallocate 0x0197bf thru 0x02148f (0x7cd1 bytes) 000578 1781236461.949795 write 0x023cb3 thru 0x02b877 (0x7bc5 bytes) 000579 1781236461.976353 MIRROR_RESYNC 4 mirrors 000580 1781236463.361204 read 0x00de6b thru 0x018681 (0xa817 bytes) 000581 1781236463.379297 MIRROR_EXTEND to 5 mirrors 000582 1781236464.223578 trunc from 0x02df20 to 0x034c8c (0x6d6d bytes) 000583 1781236464.288329 write 0x029664 thru 0x02ff07 (0x68a4 bytes) 000584 1781236464.303337 mapread 0x0013d9 thru 0x00272a (0x1352 bytes) 000585 1781236464.316936 mapwrite 0x011758 thru 0x0153c1 (0x3c6a bytes) 000586 1781236464.358877 trunc from 0x034c8d to 0x03d4d3 (0x8847 bytes) 000587 1781236464.404414 write 0x039506 thru 0x03ffff (0x6afa bytes) 000588 1781236464.432069 fallocate 0x005479 thru 0x011dee (0xc976 bytes) 000590 1781236464.446374 punch from 0x008687 to 0x010c70 (0x85ea bytes) 000591 1781236464.457422 trunc from 0x003c53 to 0x009977 (0x5d25 bytes) 000592 1781236464.464262 read 0x03e09b thru 0x03ffff (0x1f65 bytes) 000594 1781236464.466494 trunc from 0x0070e7 to 0x00babc (0x49d6 bytes) 000595 1781236464.473336 trunc from 0x040000 to 0x037e6e (0x8191 bytes) 000596 1781236464.504481 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37e6f): Mirror 4: 0x2b482a36 (stale) Mirror 5: 0x2b482a36 (stale) Mirror 6: 0x2b482a36 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e6f): Mirror 4: 0x2b482a36 (stale) Mirror 5: 0x2b482a36 (stale) Mirror 6: 0x2b482a36 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e6f: skipped 000597 1781236465.014000 trunc from 0x037e6f to 0x00d767 (0x2a707 bytes) 000598 1781236465.044324 fallocate 0x03899e thru 0x03ffff (0x7662 bytes) 000599 1781236465.051840 trunc from 0x004953 to 0x00c1c7 (0x7875 bytes) 000600 1781236465.059889 trunc from 0x003e63 to 0x00d767 (0x9905 bytes) 000601 1781236465.077059 mapread 0x001dda thru 0x00d767 (0xb98e bytes) 000602 1781236465.133125 read 0x00aeb7 thru 0x00d767 (0x28b1 bytes) 000603 1781236465.141121 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xd768): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd768): Mirror 4: 0x95cbae (stale) Mirror 5: 0x95cbae (stale) Mirror 6: 0x95cbae (stale) CRC-32 checksum value for chunk [0, 0xd768): Mirror 1: 0xbc955a3 Mirror 7: 0xbc955a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd768: skipped 000604 1781236465.947574 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd768): Mirror 5: 0x95cbae (stale) Mirror 6: 0x95cbae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd768): Mirror 5: 0x95cbae (stale) Mirror 6: 0x95cbae (stale) CRC-32 checksum value for chunk [0, 0xd768): Mirror 1: 0xbc955a3 Mirror 7: 0xbc955a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd768: skipped 000605 1781236466.552651 write 0x01de95 thru 0x020938 (0x2aa4 bytes) 000606 1781236466.643303 fallocate 0x019ac8 thru 0x020917 (0x6e50 bytes) 000607 1781236466.656464 trunc from 0x0083ed to 0x0086d7 (0x2eb bytes) 000609 1781236466.668603 fallocate 0x021f65 thru 0x02be7f (0x9f1b bytes) 000610 1781236466.674025 mapread 0x01a277 thru 0x020938 (0x66c2 bytes) 000612 1781236466.733591 punch from 0x0187bb to 0x020938 (0x817e bytes) 000613 1781236466.773400 read 0x0077e4 thru 0x00c8fe (0x511b bytes) 000614 1781236466.794745 trunc from 0x020939 to 0x0276a9 (0x6d71 bytes) 000615 1781236466.872333 write 0x01e6b5 thru 0x022a94 (0x43e0 bytes) 000616 1781236466.911845 mapread 0x01606c thru 0x024692 (0xe627 bytes) 000617 1781236466.926850 punch from 0x00d8fb to 0x010cdb (0x33e1 bytes) 000618 1781236466.946361 write 0x0382d5 thru 0x03a5ae (0x22da bytes) 000619 1781236466.947698 trunc from 0x03a5af to 0x01a6cc (0x1fee2 bytes) 000620 1781236466.992160 read 0x009854 thru 0x015d34 (0xc4e1 bytes) 000621 1781236467.013891 read 0x013e9f thru 0x01a6cc (0x682e bytes) 000624 1781236467.021605 trunc from 0x0092ae to 0x00a181 (0xed4 bytes) 000625 1781236467.042689 mapwrite 0x026623 thru 0x0298ca (0x32a8 bytes) 000626 1781236467.164033 trunc from 0x0086f5 to 0x00eb21 (0x642d bytes) 000627 1781236467.169450 trunc from 0x0298cb to 0x02295c (0x6f6e bytes) 000628 1781236467.192580 mapread 0x00032b thru 0x00bf6e (0xbc44 bytes) 000629 1781236467.217028 read 0x015083 thru 0x01d89e (0x881c bytes) 000630 1781236467.217796 mapread 0x0016f9 thru 0x005d70 (0x4678 bytes) 000631 1781236467.226046 MIRROR_RESYNC 4 mirrors 000632 1781236468.257316 fallocate 0x001752 thru 0x00c86e (0xb11d bytes) 000633 1781236468.354254 fallocate 0x001d41 thru 0x007ef1 (0x61b1 bytes) 000635 1781236468.366148 fallocate 0x03a290 thru 0x03ffff (0x5d70 bytes) 000636 1781236468.379364 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb37ec821 (stale) Mirror 7: 0xb37ec821 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb37ec821 (stale) Mirror 7: 0xb37ec821 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000637 1781236468.826101 read 0x03e6c3 thru 0x03ffff (0x193d bytes) 000638 1781236468.832127 write 0x00a8d2 thru 0x00e55c (0x3c8b bytes) 000639 1781236468.846202 write 0x02c56d thru 0x03139d (0x4e31 bytes) 000640 1781236468.857162 trunc from 0x02ddf8 to 0x033353 (0x555c bytes) 000642 1781236468.875733 punch from 0x011b06 to 0x0134b6 (0x19b1 bytes) 000643 1781236468.880953 trunc from 0x0078f8 to 0x01332b (0xba34 bytes) 000644 1781236468.885711 punch from 0x012431 to 0x020735 (0xe305 bytes) 000645 1781236468.889713 write 0x00f91a thru 0x01c3ca (0xcab1 bytes) 000646 1781236468.908048 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb37ec821 (stale) Mirror 7: 0xb37ec821 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf5153b6 Mirror 8: 0xf5153b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000647 1781236469.830283 mapread 0x002c24 thru 0x01035b (0xd738 bytes) 000648 1781236469.856417 trunc from 0x013ecf to 0x019395 (0x54c7 bytes) 000649 1781236469.902594 mapread 0x0172b0 thru 0x0178e7 (0x638 bytes) 000650 1781236469.914448 trunc from 0x040000 to 0x022ce1 (0x1d31e bytes) 000651 1781236469.948347 mapread 0x011bc1 thru 0x01f4a3 (0xd8e3 bytes) 000652 1781236469.951222 punch from 0x004ecc to 0x008c44 (0x3d79 bytes) 000653 1781236469.958312 fallocate 0x0097b4 thru 0x00ba84 (0x22d1 bytes) 000654 1781236469.965256 read 0x00c8ac thru 0x017f55 (0xb6aa bytes) 000655 1781236469.974402 trunc from 0x0188b5 to 0x022ce1 (0xa42d bytes) 000656 1781236469.981132 trunc from 0x012e30 to 0x015f79 (0x314a bytes) 000657 1781236469.987500 mapwrite 0x0350c2 thru 0x03ffff (0xaf3e bytes) 000658 1781236470.034619 trunc from 0x000808 to 0x00ae65 (0xa65e bytes) 000660 1781236470.047051 fallocate 0x035ce1 thru 0x03b7dc (0x5afc bytes) 000661 1781236470.052462 mapwrite 0x004b2f thru 0x0144cd (0xf99f bytes) 000662 1781236470.105129 trunc from 0x040000 to 0x03026e (0xfd91 bytes) 000663 1781236470.126827 mapwrite 0x028653 thru 0x0288eb (0x299 bytes) 000664 1781236470.149931 mapwrite 0x03d601 thru 0x03ffff (0x29ff bytes) 000665 1781236470.196432 read 0x03ca06 thru 0x03ffff (0x35fa bytes) 000666 1781236470.198558 fallocate 0x029418 thru 0x02fd74 (0x695d bytes) 000667 1781236470.207141 write 0x032f55 thru 0x03a051 (0x70fd bytes) 000670 1781236470.222316 mapread 0x01ae79 thru 0x020d43 (0x5ecb bytes) 000671 1781236470.224472 punch from 0x03888f to 0x03f4e7 (0x6c59 bytes) 000672 1781236470.257708 trunc from 0x009c22 to 0x017f71 (0xe350 bytes) 000674 1781236470.268791 trunc from 0x03bc92 to 0x03ffff (0x436e bytes) 000675 1781236470.281075 trunc from 0x01e220 to 0x02564a (0x742b bytes) 000676 1781236470.287071 read 0x01d600 thru 0x020a34 (0x3435 bytes) 000677 1781236470.320968 mapwrite 0x0366d1 thru 0x03ffff (0x992f bytes) 000678 1781236470.358166 fallocate 0x01bf6d thru 0x0211c2 (0x5256 bytes) 000680 1781236470.372200 trunc from 0x030cd0 to 0x03c3d3 (0xb704 bytes) 000682 1781236470.386078 fallocate 0x0237ee thru 0x02a4fe (0x6d11 bytes) 000683 1781236470.397305 fallocate 0x038c18 thru 0x03ffff (0x73e8 bytes) 000686 1781236470.407393 write 0x03ec38 thru 0x03ffff (0x13c8 bytes) 000687 1781236470.408462 mapread 0x000148 thru 0x00c215 (0xc0ce bytes) 000688 1781236470.421608 MIRROR_RESYNC 4 mirrors 000690 1781236471.681500 punch from 0x02ff55 to 0x0364bc (0x6568 bytes) 000691 1781236471.768928 MIRROR_RESYNC 4 mirrors 000692 1781236473.316600 trunc from 0x040000 to 0x03de37 (0x21c8 bytes) 000693 1781236473.400695 fallocate 0x03a986 thru 0x03ffff (0x567a bytes) 000694 1781236473.407393 read 0x01c006 thru 0x01decc (0x1ec7 bytes) 000695 1781236473.421281 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3de38): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de38): Mirror 6: 0xe0bb4c01 (stale) Mirror 7: 0xe0bb4c01 (stale) Mirror 8: 0xe0bb4c01 (stale) CRC-32 checksum value for chunk [0, 0x3de38): Mirror 1: 0xe0bb4c01 Mirror 9: 0xe0bb4c01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de38: skipped 000696 1781236474.577376 write 0x01b586 thru 0x01d38f (0x1e0a bytes) 000697 1781236474.633185 mapwrite 0x005ddc thru 0x00bf7a (0x619f bytes) 000698 1781236474.670256 trunc from 0x037bd8 to 0x03de37 (0x6260 bytes) 000699 1781236474.676575 read 0x02eeb3 thru 0x03de37 (0xef85 bytes) 000700 1781236474.693795 mapwrite 0x030a82 thru 0x0313ec (0x96b bytes) 000701 1781236474.713844 trunc from 0x03de38 to 0x01987d (0x245ba bytes) 000702 1781236474.751931 fallocate 0x0333bf thru 0x03ffff (0xcc41 bytes) 000703 1781236474.757678 punch from 0x007ae6 to 0x00d46d (0x5988 bytes) 000704 1781236474.764983 punch from 0x005191 to 0x00ba0a (0x687a bytes) 000705 1781236474.776410 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1987e): Mirror 6: 0xf5e4b910 (stale) Mirror 7: 0xf5e4b910 (stale) Mirror 8: 0xf5e4b910 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1987e): Mirror 6: 0xf5e4b910 (stale) Mirror 7: 0xf5e4b910 (stale) Mirror 8: 0xf5e4b910 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1987e: skipped 000706 1781236475.127295 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1987e): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1987e): Mirror 6: 0xf5e4b910 (stale) Mirror 7: 0xf5e4b910 (stale) Mirror 8: 0xf5e4b910 (stale) CRC-32 checksum value for chunk [0, 0x1987e): Mirror 1: 0xbb8ae663 Mirror 9: 0xbb8ae663 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1987e: skipped 000707 1781236476.235434 MIRROR_RESYNC 5 mirrors 000709 1781236477.433661 mapwrite 0x03d180 thru 0x03ffff (0x2e80 bytes) 000710 1781236477.612077 trunc from 0x040000 to 0x020aa6 (0x1f559 bytes) 000711 1781236477.660616 mapwrite 0x033fd3 thru 0x038606 (0x4634 bytes) 000712 1781236477.735320 mapwrite 0x003061 thru 0x0067b5 (0x3755 bytes) 000713 1781236477.788758 trunc from 0x026110 to 0x02c6b5 (0x65a6 bytes) 000714 1781236477.798248 mapwrite 0x00745c thru 0x010291 (0x8e36 bytes) 000715 1781236477.822279 read 0x0325fc thru 0x033fbb (0x19c0 bytes) 000716 1781236477.823145 fallocate 0x000c94 thru 0x0056de (0x4a4b bytes) 000717 1781236477.838263 trunc from 0x038607 to 0x023d39 (0x148cd bytes) 000718 1781236477.884033 trunc from 0x023d3a to 0x0182cc (0xba6d bytes) 000719 1781236477.927396 trunc from 0x00cae6 to 0x014cc7 (0x81e2 bytes) 000720 1781236477.933817 mapread 0x014e0a thru 0x0182cc (0x34c3 bytes) 000721 1781236477.945234 mapread 0x00e06a thru 0x015ff2 (0x7f89 bytes) 000722 1781236477.948321 trunc from 0x001433 to 0x00c45c (0xb02a bytes) 000723 1781236477.959455 punch from 0x007bf9 to 0x0123e2 (0xa7ea bytes) 000724 1781236477.969943 punch from 0x00d34f to 0x00f91b (0x25cd bytes) 000725 1781236477.977300 mapread 0x006063 thru 0x00fa40 (0x99de bytes) 000726 1781236478.019343 trunc from 0x0182cd to 0x0016a1 (0x16c2b bytes) 000727 1781236478.041362 mapwrite 0x000cfe thru 0x00e164 (0xd467 bytes) 000728 1781236478.097784 trunc from 0x00c5ef to 0x00e164 (0x1b76 bytes) 000729 1781236478.107165 trunc from 0x00e165 to 0x025eea (0x17d86 bytes) 000731 1781236478.120469 write 0x005eb7 thru 0x00808e (0x21d8 bytes) 000732 1781236478.121743 punch from 0x0206c9 to 0x025eea (0x5822 bytes) 000734 1781236478.129394 write 0x0338a2 thru 0x03ffff (0xc75e bytes) 000735 1781236478.132313 fallocate 0x005fb3 thru 0x00e34c (0x839a bytes) 000737 1781236478.140173 trunc from 0x040000 to 0x022551 (0x1daae bytes) 000738 1781236478.154583 read 0x016579 thru 0x020539 (0x9fc1 bytes) 000740 1781236478.180406 fallocate 0x024528 thru 0x034102 (0xfbdb bytes) 000741 1781236478.194412 trunc from 0x026aa1 to 0x02b2d3 (0x4833 bytes) 000743 1781236478.201841 fallocate 0x0310d8 thru 0x03f537 (0xe460 bytes) 000744 1781236478.209826 trunc from 0x03f538 to 0x019290 (0x262a7 bytes) 000745 1781236478.236210 trunc from 0x00aa30 to 0x0118c0 (0x6e91 bytes) 000746 1781236478.244439 punch from 0x0014a7 to 0x00e2c3 (0xce1d bytes) 000747 1781236478.261379 trunc from 0x00f11b to 0x016e91 (0x7d77 bytes) 000748 1781236478.271467 trunc from 0x0147bd to 0x019290 (0x4ad4 bytes) 000749 1781236478.277457 trunc from 0x005875 to 0x00d278 (0x7a04 bytes) 000750 1781236478.283620 trunc from 0x009200 to 0x00afaa (0x1dab bytes) 000751 1781236478.289913 mapwrite 0x0161a0 thru 0x018f85 (0x2de6 bytes) 000752 1781236478.320882 write 0x017d04 thru 0x0209b1 (0x8cae bytes) 000753 1781236478.324392 mapread 0x0054b0 thru 0x01407b (0xebcc bytes) 000754 1781236478.441126 fallocate 0x02fbb4 thru 0x0372a6 (0x76f3 bytes) 000755 1781236478.448131 punch from 0x006dd4 to 0x011523 (0xa750 bytes) 000756 1781236478.455373 trunc from 0x00eb73 to 0x01c964 (0xddf2 bytes) 000757 1781236478.472681 trunc from 0x0209b2 to 0x021073 (0x6c2 bytes) 000758 1781236478.492396 fallocate 0x03f874 thru 0x03ffff (0x78c bytes) 000759 1781236478.516198 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4d152dc4 (stale) Mirror 7: 0x4d152dc4 (stale) Mirror 8: 0x4d152dc4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4d152dc4 (stale) Mirror 7: 0x4d152dc4 (stale) Mirror 8: 0x4d152dc4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000760 1781236479.038938 mapwrite 0x016e6e thru 0x024d9d (0xdf30 bytes) 000761 1781236479.130486 fallocate 0x02f3a0 thru 0x03375a (0x43bb bytes) 000762 1781236479.138009 punch from 0x0163d1 to 0x023cad (0xd8dd bytes) 000763 1781236479.145177 punch from 0x03bd08 to 0x03ffff (0x42f8 bytes) 000764 1781236479.151738 fallocate 0x026a3e thru 0x0302ef (0x98b2 bytes) 000766 1781236479.158788 read 0x0010d4 thru 0x00f521 (0xe44e bytes) 000767 1781236479.169359 read 0x000c07 thru 0x006e73 (0x626d bytes) 000768 1781236479.206923 punch from 0x03923a to 0x03ffff (0x6dc6 bytes) 000769 1781236479.213171 punch from 0x00e88d to 0x01bb4c (0xd2c0 bytes) 000770 1781236479.219793 punch from 0x02c3e9 to 0x02d06d (0xc85 bytes) 000771 1781236479.226609 trunc from 0x03f749 to 0x03ffff (0x8b7 bytes) 000772 1781236479.232182 trunc from 0x040000 to 0x01c192 (0x23e6d bytes) 000773 1781236479.263141 trunc from 0x00d8a3 to 0x01852d (0xac8b bytes) 000776 1781236479.276671 MIRROR_RESYNC 4 mirrors 000777 1781236480.325476 MIRROR_RESYNC 4 mirrors 000778 1781236480.794494 trunc from 0x01c193 to 0x013e1e (0x8374 bytes) 000779 1781236480.882580 read 0x009c84 thru 0x0116e8 (0x7a65 bytes) 000780 1781236480.894975 trunc from 0x013e1f to 0x03b751 (0x27933 bytes) 000781 1781236480.915122 read 0x01334e thru 0x01aaa2 (0x7755 bytes) 000782 1781236480.940940 trunc from 0x03b752 to 0x028f87 (0x127ca bytes) 000783 1781236480.964347 trunc from 0x018411 to 0x01d7b8 (0x53a8 bytes) 000784 1781236480.969645 mapread 0x006be2 thru 0x010634 (0x9a53 bytes) 000785 1781236480.971589 mapwrite 0x0067de thru 0x015790 (0xefb3 bytes) 000788 1781236481.011895 mapwrite 0x038610 thru 0x03f3d2 (0x6dc3 bytes) 000789 1781236481.093192 trunc from 0x03caec to 0x03f3d2 (0x28e7 bytes) 000791 1781236481.110002 write 0x00babc thru 0x0169ae (0xaef3 bytes) 000792 1781236481.117369 trunc from 0x03f3d3 to 0x03e49b (0xf37 bytes) 000793 1781236481.157724 mapwrite 0x0224d1 thru 0x02d4b4 (0xafe4 bytes) 000794 1781236481.244145 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e49c): Mirror 7: 0x40ba381a (stale) Mirror 8: 0x40ba381a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e49c): Mirror 7: 0x40ba381a (stale) Mirror 8: 0x40ba381a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e49c: skipped 000795 1781236481.783661 read 0x00c220 thru 0x00d1c6 (0xfa7 bytes) 000796 1781236481.798615 read 0x0020e2 thru 0x00bebd (0x9ddc bytes) 000797 1781236481.841171 write 0x03bc96 thru 0x03ffff (0x436a bytes) 000798 1781236481.868206 mapwrite 0x00eff9 thru 0x01e599 (0xf5a1 bytes) 000799 1781236481.904945 read 0x01dad8 thru 0x01febc (0x23e5 bytes) 000800 1781236481.905789 trunc from 0x040000 to 0x00777f (0x38880 bytes) 000801 1781236481.948385 trunc from 0x007780 to 0x004f85 (0x27fa bytes) 000802 1781236481.993237 read 0x001da5 thru 0x00399d (0x1bf9 bytes) 000803 1781236481.993955 write 0x022511 thru 0x02ff13 (0xda03 bytes) 000804 1781236482.019343 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ff14): Mirror 8: 0x571e7d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ff14): Mirror 8: 0x571e7d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ff14: skipped 000805 1781236482.339367 mapwrite 0x00f891 thru 0x016577 (0x6ce7 bytes) 000806 1781236482.410939 write 0x032bfa thru 0x037056 (0x445d bytes) 000807 1781236482.434455 MIRROR_RESYNC 2 mirrors 000808 1781236483.621037 trunc from 0x037057 to 0x00bd67 (0x2b2ef bytes) 000809 1781236483.686465 fallocate 0x032bfa thru 0x033fca (0x13d1 bytes) 000810 1781236483.692642 mapwrite 0x035cb6 thru 0x0398e4 (0x3c2f bytes) 000811 1781236483.773779 fallocate 0x008905 thru 0x00cd78 (0x4474 bytes) 000812 1781236483.779635 punch from 0x013718 to 0x0139e4 (0x2cd bytes) 000813 1781236483.786847 mapwrite 0x010635 thru 0x01c9fc (0xc3c8 bytes) 000814 1781236483.842831 mapwrite 0x01ec09 thru 0x02ea5d (0xfe55 bytes) 000815 1781236483.916194 mapread 0x0070c7 thru 0x016e9d (0xfdd7 bytes) 000816 1781236483.931388 mapread 0x0362c7 thru 0x0398e4 (0x361e bytes) 000817 1781236483.933120 mapread 0x012457 thru 0x01d923 (0xb4cd bytes) 000818 1781236483.939400 write 0x00ba9e thru 0x014af3 (0x9056 bytes) 000819 1781236483.948589 fallocate 0x023fe3 thru 0x0301b4 (0xc1d2 bytes) 000821 1781236483.963750 write 0x0063c0 thru 0x0099fd (0x363e bytes) 000822 1781236483.971332 trunc from 0x034e93 to 0x034f4e (0xbc bytes) 000823 1781236483.978463 trunc from 0x0398e5 to 0x00d53a (0x2c3aa bytes) 000824 1781236483.999647 write 0x0259e8 thru 0x02b83a (0x5e53 bytes) 000825 1781236484.003603 trunc from 0x02b83b to 0x017875 (0x13fc5 bytes) 000827 1781236484.022903 read 0x001496 thru 0x004d5e (0x38c9 bytes) 000828 1781236484.024390 mapwrite 0x02d815 thru 0x03c4b5 (0xeca1 bytes) 000829 1781236484.096796 fallocate 0x01b814 thru 0x024f31 (0x971e bytes) 000830 1781236484.103919 read 0x016481 thru 0x0197fe (0x337e bytes) 000831 1781236484.117018 trunc from 0x03c4b6 to 0x02fd61 (0xc754 bytes) 000832 1781236484.139995 mapwrite 0x020daf thru 0x02b93b (0xab8d bytes) 000835 1781236484.196628 fallocate 0x02045c thru 0x027428 (0x6fcd bytes) 000836 1781236484.209419 trunc from 0x005692 to 0x00c964 (0x72d3 bytes) 000837 1781236484.219433 trunc from 0x02fd62 to 0x025f42 (0x9e1f bytes) 000838 1781236484.241217 mapread 0x0166be thru 0x01831e (0x1c61 bytes) 000839 1781236484.243259 write 0x02095c thru 0x023efe (0x35a3 bytes) 000841 1781236484.254598 punch from 0x00fa9a to 0x01d874 (0xdddb bytes) 000842 1781236484.262333 read 0x00b14e thru 0x0135f3 (0x84a6 bytes) 000843 1781236484.312830 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x25f43): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25f43): Mirror 8: 0x16f8aa5e (stale) CRC-32 checksum value for chunk [0, 0x25f43): Mirror 1: 0x8e85d34b Mirror 9: 0x8e85d34b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25f43: skipped 000844 1781236485.459180 mapread 0x0095a7 thru 0x00e92a (0x5384 bytes) 000847 1781236485.485427 punch from 0x010226 to 0x0140bd (0x3e98 bytes) 000848 1781236485.573194 write 0x00223b thru 0x00258e (0x354 bytes) 000850 1781236485.628806 read 0x022ff7 thru 0x025f42 (0x2f4c bytes) 000851 1781236485.638361 punch from 0x017c8c to 0x025874 (0xdbe9 bytes) 000852 1781236485.649562 write 0x037744 thru 0x03ffff (0x88bc bytes) 000854 1781236485.654431 mapwrite 0x02ef4a thru 0x031069 (0x2120 bytes) 000855 1781236485.695126 mapwrite 0x01ec59 thru 0x02d280 (0xe628 bytes) 000856 1781236485.769034 trunc from 0x040000 to 0x011522 (0x2eadd bytes) 000857 1781236485.867932 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x11523): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11523): Mirror 8: 0xb19d4f8b (stale) Mirror 9: 0x55b15259 (stale) CRC-32 checksum value for chunk [0, 0x11523): Mirror 1: 0x584e2b24 Mirror 10: 0x584e2b24 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11523: skipped 000858 1781236486.865311 write 0x02489f thru 0x02fee4 (0xb646 bytes) 000859 1781236486.911208 read 0x00e349 thru 0x0150b2 (0x6d6a bytes) 000860 1781236486.925027 read 0x0156f6 thru 0x0243ca (0xecd5 bytes) 000861 1781236486.938571 MIRROR_RESYNC 4 mirrors 000862 1781236488.265374 trunc from 0x02fee5 to 0x010173 (0x1fd71 bytes) 000863 1781236488.323278 mapwrite 0x0004c0 thru 0x000a55 (0x596 bytes) 000864 1781236488.351391 trunc from 0x010174 to 0x000aec (0xf687 bytes) 000865 1781236488.382099 mapwrite 0x0296f5 thru 0x02c7d0 (0x30dc bytes) 000867 1781236488.433984 read 0x024a58 thru 0x02c7d0 (0x7d79 bytes) 000868 1781236488.444590 MIRROR_RESYNC 4 mirrors 000869 1781236489.330599 mapread 0x017e08 thru 0x02582c (0xda25 bytes) 000870 1781236489.359787 fallocate 0x006ddc thru 0x00dce3 (0x6f08 bytes) 000871 1781236489.397565 write 0x029525 thru 0x02b7cb (0x22a7 bytes) 000873 1781236489.407382 mapread 0x0291df thru 0x02c7d0 (0x35f2 bytes) 000874 1781236489.409362 trunc from 0x02c7d1 to 0x027a46 (0x4d8a bytes) 000875 1781236489.457502 punch from 0x01ada2 to 0x0207d1 (0x5a30 bytes) 000876 1781236489.463201 mapread 0x0181b1 thru 0x01f4f5 (0x7345 bytes) 000877 1781236489.476047 mapread 0x00cc65 thru 0x010886 (0x3c22 bytes) 000878 1781236489.487285 trunc from 0x01521d to 0x016227 (0x100b bytes) 000879 1781236489.492297 trunc from 0x027a47 to 0x02e446 (0x6a00 bytes) 000880 1781236489.513172 punch from 0x0168ed to 0x01ce69 (0x657d bytes) 000882 1781236489.524686 mapwrite 0x033f55 thru 0x035be5 (0x1c91 bytes) 000883 1781236489.577337 trunc from 0x035be6 to 0x032a04 (0x31e1 bytes) 000884 1781236489.596862 punch from 0x007ca0 to 0x00873b (0xa9c bytes) 000886 1781236489.624345 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32a05): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32a05): Mirror 8: 0xb75cbb82 (stale) Mirror 9: 0xb75cbb82 (stale) Mirror 10: 0xb75cbb82 (stale) CRC-32 checksum value for chunk [0, 0x32a05): Mirror 1: 0xbb36df47 Mirror 11: 0xbb36df47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32a05: skipped 000887 1781236491.004875 read 0x0107ce thru 0x01875b (0x7f8e bytes) 000888 1781236491.024193 fallocate 0x02f4f6 thru 0x03d3c5 (0xded0 bytes) 000889 1781236491.103001 trunc from 0x00ed1d to 0x00f5c7 (0x8ab bytes) 000891 1781236491.113861 punch from 0x0076db to 0x0153a4 (0xdcca bytes) 000892 1781236491.120155 trunc from 0x032a05 to 0x03fed1 (0xd4cd bytes) 000893 1781236491.169213 MIRROR_RESYNC 5 mirrors 000894 1781236492.242216 MIRROR_RESYNC 5 mirrors 000896 1781236492.998720 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) 000897 1781236493.582162 mapwrite 0x03b6b1 thru 0x03ffff (0x494f bytes) 000898 1781236493.721562 punch from 0x0217e6 to 0x0295de (0x7df9 bytes) 000899 1781236493.734211 read 0x02db81 thru 0x02f714 (0x1b94 bytes) 000900 1781236493.748793 punch from 0x0204c1 to 0x020fd8 (0xb18 bytes) 000901 1781236493.760462 mapread 0x02d377 thru 0x034d68 (0x79f2 bytes) 000902 1781236493.772238 read 0x01be62 thru 0x027096 (0xb235 bytes) 000903 1781236493.789774 trunc from 0x02daf5 to 0x034867 (0x6d73 bytes) 000904 1781236493.809878 MIRROR_RESYNC 4 mirrors 000905 1781236494.779934 trunc from 0x040000 to 0x019958 (0x266a7 bytes) 000906 1781236494.863312 mapwrite 0x007536 thru 0x00c367 (0x4e32 bytes) 000907 1781236494.960722 mapread 0x003821 thru 0x004be3 (0x13c3 bytes) 000908 1781236494.962529 trunc from 0x019959 to 0x025997 (0xc03f bytes) 000909 1781236494.996134 mapread 0x024f8b thru 0x025997 (0xa0d bytes) 000910 1781236495.013230 read 0x009eec thru 0x00a21e (0x333 bytes) 000911 1781236495.013935 read 0x0048f3 thru 0x004c35 (0x343 bytes) 000912 1781236495.018185 trunc from 0x01b11b to 0x01c7be (0x16a4 bytes) 000913 1781236495.031396 punch from 0x011d34 to 0x015a71 (0x3d3e bytes) 000914 1781236495.051647 mapread 0x01497a thru 0x01a5b7 (0x5c3e bytes) 000915 1781236495.089375 read 0x00c259 thru 0x01c0bd (0xfe65 bytes) 000916 1781236495.096074 fallocate 0x0048dc thru 0x010c6d (0xc392 bytes) 000917 1781236495.120334 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25998): Mirror 8: 0x8eb2126c (stale) Mirror 10: 0x8eb2126c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25998): Mirror 8: 0x8eb2126c (stale) Mirror 10: 0x8eb2126c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25998: skipped 000918 1781236495.594574 MIRROR_RESYNC 3 mirrors 000919 1781236496.687445 mapread 0x00acd3 thru 0x00f244 (0x4572 bytes) 000920 1781236496.718260 punch from 0x0059f0 to 0x008d29 (0x333a bytes) 000921 1781236496.780885 MIRROR_RESYNC 3 mirrors 000922 1781236497.610203 punch from 0x0258e2 to 0x025997 (0xb6 bytes) 000923 1781236497.662931 MIRROR_RESYNC 3 mirrors 000925 1781236498.414951 mapwrite 0x01fd07 thru 0x02fa75 (0xfd6f bytes) 000927 1781236498.588395 write 0x03044a thru 0x033bac (0x3763 bytes) 000928 1781236498.592824 fallocate 0x00caab thru 0x00f377 (0x28cd bytes) 000929 1781236498.598814 trunc from 0x019ec8 to 0x021a85 (0x7bbe bytes) 000930 1781236498.604787 read 0x00ae74 thru 0x00d3e5 (0x2572 bytes) 000931 1781236498.611976 mapwrite 0x02bb29 thru 0x02fae2 (0x3fba bytes) 000932 1781236498.638911 mapread 0x0321e5 thru 0x033bac (0x19c8 bytes) 000933 1781236498.640003 trunc from 0x005c91 to 0x010987 (0xacf7 bytes) 000934 1781236498.645267 trunc from 0x01430e to 0x023220 (0xef13 bytes) 000935 1781236498.657310 read 0x00ff19 thru 0x012e26 (0x2f0e bytes) 000937 1781236498.677814 write 0x0167b0 thru 0x0257ce (0xf01f bytes) 000938 1781236498.681015 mapwrite 0x01ee3d thru 0x026ac5 (0x7c89 bytes) 000939 1781236498.713055 trunc from 0x033bad to 0x0175ea (0x1c5c2 bytes) 000940 1781236498.735460 write 0x03e1f1 thru 0x03ffff (0x1e0f bytes) 000941 1781236498.737489 fallocate 0x003c41 thru 0x0105ad (0xc96d bytes) 000942 1781236498.743890 write 0x0240a4 thru 0x03333c (0xf299 bytes) 000943 1781236498.758770 trunc from 0x024040 to 0x026f88 (0x2f49 bytes) 000944 1781236498.775368 read 0x03cbb3 thru 0x03ffff (0x344d bytes) 000945 1781236498.784101 MIRROR_RESYNC 3 mirrors 000946 1781236500.330128 write 0x00aaf8 thru 0x00b5da (0xae3 bytes) 000947 1781236500.447236 trunc from 0x040000 to 0x01880a (0x277f5 bytes) 000948 1781236500.484250 punch from 0x00d390 to 0x00d6c8 (0x339 bytes) 000949 1781236500.497165 punch from 0x00f709 to 0x01880a (0x9102 bytes) 000950 1781236500.509322 fallocate 0x01867c thru 0x020ed2 (0x8857 bytes) 000951 1781236500.558088 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x20ed3): Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20ed3): Mirror 8: 0x1acaf262 (stale) Mirror 10: 0x1acaf262 (stale) CRC-32 checksum value for chunk [0, 0x20ed3): Mirror 1: 0x74f62712 Mirror 11: 0x74f62712 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20ed3: skipped 000952 1781236501.882637 write 0x026086 thru 0x02a0a6 (0x4021 bytes) 000953 1781236501.940057 trunc from 0x020a51 to 0x028b1c (0x80cc bytes) 000954 1781236501.957209 punch from 0x01ffcd to 0x0207dd (0x811 bytes) 000955 1781236501.963441 punch from 0x01bb25 to 0x029d17 (0xe1f3 bytes) 000956 1781236501.970099 trunc from 0x000193 to 0x007650 (0x74be bytes) 000957 1781236501.977073 trunc from 0x024f5f to 0x026534 (0x15d6 bytes) 000958 1781236501.983204 fallocate 0x02f2e6 thru 0x03e680 (0xf39b bytes) 000959 1781236501.988165 write 0x02cc86 thru 0x03b97c (0xecf7 bytes) 000960 1781236502.000088 mapread 0x021229 thru 0x02e271 (0xd049 bytes) 000961 1781236502.020032 mapread 0x02f3bc thru 0x02ff38 (0xb7d bytes) 000962 1781236502.022899 trunc from 0x006ef2 to 0x0121fd (0xb30c bytes) 000964 1781236502.029893 punch from 0x0183e4 to 0x0207d1 (0x83ee bytes) 000966 1781236502.037982 fallocate 0x0319c2 thru 0x036e63 (0x54a2 bytes) 000969 1781236502.046165 read 0x024003 thru 0x02fb6d (0xbb6b bytes) 000970 1781236502.047207 mapread 0x037182 thru 0x03e680 (0x74ff bytes) 000971 1781236502.054790 mapread 0x03a704 thru 0x03e680 (0x3f7d bytes) 000972 1781236502.056274 read 0x008be8 thru 0x0108bd (0x7cd6 bytes) 000973 1781236502.070140 punch from 0x00ffb4 to 0x016fd5 (0x7022 bytes) 000974 1781236502.076029 mapread 0x038a09 thru 0x039988 (0xf80 bytes) 000975 1781236502.081635 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e681): Mirror 8: 0xdd6df5ad (stale) Mirror 10: 0xdd6df5ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e681): Mirror 8: 0xdd6df5ad (stale) Mirror 10: 0xdd6df5ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e681: skipped 000976 1781236502.436756 trunc from 0x0081a0 to 0x01517f (0xcfe0 bytes) 000977 1781236502.442287 write 0x03cc05 thru 0x03ffff (0x33fb bytes) 000979 1781236502.458070 trunc from 0x02d8da to 0x03855d (0xac84 bytes) 000980 1781236502.468953 fallocate 0x02ab3e thru 0x032baa (0x806d bytes) 000981 1781236502.475639 mapread 0x03c901 thru 0x03ebba (0x22ba bytes) 000982 1781236502.477754 trunc from 0x0033b7 to 0x009ea3 (0x6aed bytes) 000983 1781236502.491368 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x3616a2a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x3616a2a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000984 1781236502.840200 mapread 0x03c813 thru 0x03c927 (0x115 bytes) 000985 1781236502.849835 write 0x0369c7 thru 0x03aabb (0x40f5 bytes) 000986 1781236502.883691 read 0x030894 thru 0x0318df (0x104c bytes) 000987 1781236502.893638 write 0x03c221 thru 0x03fd68 (0x3b48 bytes) 000989 1781236502.907930 mapread 0x02277b thru 0x02bc88 (0x950e bytes) 000990 1781236502.941788 fallocate 0x03d7c5 thru 0x03ffff (0x283b bytes) 000991 1781236502.947614 trunc from 0x011f41 to 0x01cba2 (0xac62 bytes) 000992 1781236502.953031 punch from 0x010075 to 0x0157ef (0x577b bytes) 000993 1781236502.958747 trunc from 0x040000 to 0x03e3dc (0x1c23 bytes) 000994 1781236502.976943 read 0x00d747 thru 0x00d7f0 (0xaa bytes) 000996 1781236502.991189 mapwrite 0x021fb4 thru 0x023659 (0x16a6 bytes) 000997 1781236503.010934 read 0x0040bf thru 0x00fc6c (0xbbae bytes) 000998 1781236503.012873 trunc from 0x03e3dd to 0x0350b8 (0x9324 bytes) 000999 1781236503.028941 read 0x034db5 thru 0x0350b8 (0x304 bytes) 001000 1781236503.031151 trunc from 0x022c60 to 0x030c34 (0xdfd5 bytes) All operations completed A-OK! PASS 70a (101s) == sanity-flr test 71: check mirror extend parent fid ==== 23:55:09 (1781236509) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1d4:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 23:55:21 (1781236521) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.298675 s, 7.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129623 s, 8.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 23:55:30 (1781236530) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2097888 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8309344 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2359200 .. failed Extending file size to 2362368 .. Extending file size to 2704736 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8531584 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8963808 .. Extending file size to 8568416 .. Extending file size to 8970016 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8983872 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61470 61471 61472 61477 61478 done 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) 192.168.202.134@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:56:54 (1781236614) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3140320 .. Extending file size to 2939360 .. Extending file size to 5204416 .. Extending file size to 6120256 .. Extending file size to 4585632 .. Extending file size to 6430496 .. Extending file size to 6710656 .. Extending file size to 7494720 .. done Extending file size to 7831648 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8125760 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9178272 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8370848 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8578368 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9142624 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 63240 63241 63242 63246 63247 failed 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) 192.168.202.134@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:58:15 (1781236695) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2192 1269112 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18920 3588100 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24760 3582260 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43680 7170360 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2192 1269112 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18920 3588100 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24760 3582260 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43680 7170360 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0959536 s, 21.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114736 s, 18.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x492fd1c6 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:58:36 (1781236716) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0749099 s, 28.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 23:58:42 (1781236722) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x129:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:58:49 (1781236729) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.098844 s, 21.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:58:57 (1781236737) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0780584 s, 26.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12b:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:59:04 (1781236744) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111575 s, 18.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:59:38 (1781236778) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997918 s, 10.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20c:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 00:00:33 (1781236833) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117063 s, 17.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 00:01:24 (1781236884) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg234-server: Pool lustre.test_204f created Adding targets to pool oleg234-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg234-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116003 s, 18.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg234-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg234-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg234-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 00:02:44 (1781236964) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 00:02:52 (1781236972) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 00:03:00 (1781236980) oleg234-server: Pool lustre.test_206 created Adding targets to pool oleg234-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg234-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg234-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg234-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg234-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 00:03:29 (1781237009) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808834 s, 13.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455886 s, 23.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x334429e3 Mirror 3: 0x334429e3 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 00:03:41 (1781237021) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 00:03:45 (1781237025) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 00:03:49 (1781237029) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0387395 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1680000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1030000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1030000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1030000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2550000 Read this (4 bytes): BBBB 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 00:04:33 (1781237073) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.027858 s, 294 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 00:04:41 (1781237081) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434388 s, 24.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5b lmm_fid: [0x200000405:0x5b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5b lmm_fid: [0x200000405:0x5b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611227 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-12 00:04:43.000000000 -0400 Modify: 2026-06-12 00:04:43.000000000 -0400 Change: 2026-06-12 00:04:44.000000000 -0400 Birth: 2026-06-12 00:04:42.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 00:04:51 (1781237091) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238221 s, 44.0 MB/s oleg234-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg234-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg234-server: O/280000401/d26/13a: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 00:05:14 (1781237114) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.244016 s, 168 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0289067 s, 142 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 12 00:05 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 12 00:05 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 00:05:22 (1781237122) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 00:05:31 (1781237131) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1847 sec =========== 00:05:39 (1781237139) === sanity-flr: start cleanup 00:05:41 (1781237141) === === sanity-flr: finish cleanup 00:05:48 (1781237148) ===