-----============= acceptance-small: sanity-flr ============----- Sat Feb 7 01:47:05 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg116-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:47:11 (1770446831) === oleg116-client.virtnet: executing check_config_client /mnt/lustre oleg116-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg116-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f89c3d64800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f89c3d64800.idle_timeout=debug disable quota as required oleg116-server: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:47:27 (1770446847) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8165] == sanity-flr test 0a: lfs mirror create with -N option == 01:47:28 (1770446848) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:47:34 (1770446854) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:47:35 (1770446855) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:47:37 (1770446857) PASS 0d (8s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:47:46 (1770446866) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:47:47 (1770446867) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 01:47:48 (1770446868) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f89c3d64800.stats=clear osc.lustre-OST0001-osc-ffff9f89c3d64800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:47:51 (1770446871) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:47:54 (1770446874) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:47:56 (1770446876) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:47:58 (1770446878) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 01:48:00 (1770446880) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:48:03 (1770446883) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:48:06 (1770446886) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:48:08 (1770446888) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:48:12 (1770446892) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:48:15 (1770446895) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.160721 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267343 s, 39.2 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:48:18 (1770446898) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.0740268 s, 127 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:48:21 (1770446901) Stopping /mnt/lustre-ost1 (opts:) on oleg116-server oleg116-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f89c3d64800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f89c3d64800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg116-server: oleg116-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg116-server oleg116-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f89c3d64800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f89c3d64800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg116-server: oleg116-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg116-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (31s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:48:52 (1770446932) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.172006 s, 48.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg116-server oleg116-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f89c3d64800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f89c3d64800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg116-server: oleg116-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (14s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:49:06 (1770446946) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg116-server oleg116-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f89c3d64800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f89c3d64800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg116-server: oleg116-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg116-server oleg116-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f89c3d64800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f89c3d64800.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg116-server: oleg116-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg116-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (28s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:49:34 (1770446974) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.968374 s, 54.1 MB/s -rw-r--r-- 1 root root 52428800 Feb 7 01:49 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.258s user 0m0.000s sys 0m0.156s Stopping /mnt/lustre-ost1 (opts:) on oleg116-server oleg116-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f89c3d64800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f89c3d64800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.288s user 0m0.000s sys 0m0.152s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg116-server: oleg116-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg116-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg116-server oleg116-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f89c3d64800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f89c3d64800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.261s user 0m0.000s sys 0m0.124s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg116-server: oleg116-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg116-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg116-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f89c3d64800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f89c3d64800.ost_server_uuid in FULL state after 0 sec PASS 33b (30s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:50:04 (1770447004) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:50:05 (1770447005) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:50:07 (1770447007) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 01:50:08 (1770447008) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 01:50:11 (1770447011) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.205127 s, 61.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:50:16 (1770447016) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212903 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179774 s, 58.3 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:50:19 (1770447019) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:50:23 (1770447023) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000859993 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000901981 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00084985 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000831841 s, 1.2 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 01:50:27 (1770447027) create test files with size 15754039 .. 2f31f544b07f14d56c501502982032f5 - bc82e8a676452811fe50056a499a71be - 8577462692ad0dea4427e8f6307d77a8 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg116-client.virtnet /mnt/lustre (opts:) Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 01:50:34 (1770447034) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:50:39 (1770447039) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:50:42 (1770447042) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309184 s, 33.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103488 s, 101 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 01:50:44 (1770447044) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0451207 s, 46.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.133196 s, 31.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0457251 s, 45.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 01:50:50 (1770447050) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 01:50:52 (1770447052) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:50:53 (1770447053) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209168 s, 50.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 01:50:57 (1770447057) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.102117 s, 30.8 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:51:00 (1770447060) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4c:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4c:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:51:04 (1770447064) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0840031 s, 125 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg116-client.virtnet /mnt/lustre (opts:) Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:51:19 (1770447079) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9f89d18ab800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0897668 s, 117 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg116-client.virtnet /mnt/lustre (opts:) Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9f89d81f1800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:51:23 (1770447083) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:51:34 (1770447094) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (11s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:51:45 (1770447105) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 01:51:46 (1770447106) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0692845 s, 45.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0656004 s, 48.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x35c40118 (stale) Mirror 2: 0x35c40118 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 386f7c0aa26d65c46ad033253fda0f76 - mirror 1 checksum: 2b45b826d47b4b5ba34712a1d1212711 - mirror 2 checksum: 386f7c0aa26d65c46ad033253fda0f76 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat Feb 7 01:51:47 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat Feb 7 01:51:47 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 386f7c0aa26d65c46ad033253fda0f76 - mirror 1 checksum: 2b45b826d47b4b5ba34712a1d1212711 - mirror 2 checksum: 2b45b826d47b4b5ba34712a1d1212711 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat Feb 7 01:51:47 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat Feb 7 01:51:47 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:51:49 (1770447109) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154337 s, 67.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:51:52 (1770447112) Failing mds1 on oleg116-server Stopping /mnt/lustre-mds1 (opts:) on oleg116-server 01:52:00 (1770447120) shut down facet: mds1 facet_host: oleg116-server facet_failover_host: oleg116-server Failover mds1 to oleg116-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 01:52:14 (1770447134) targets are mounted 01:52:14 (1770447134) facet_failover done oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119856 s, 87.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x57:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:52:23 (1770447143) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00916379 s, 2.2 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00405295 s, 5.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:52:26 (1770447146) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00989389 s, 2.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00351264 s, 5.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:52:29 (1770447149) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00188382 s, 8.7 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:52:32 (1770447152) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00940146 s, 2.2 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:52:34 (1770447154) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00189975 s, 2.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00958122 s, 428 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:52:37 (1770447157) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:52:43 (1770447163) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:53:02 (1770447182) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:53:21 (1770447201) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:53:40 (1770447220) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 01:53:43 (1770447223) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 55804 mirror split pid 55806 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 55804 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 55806 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 01:54:46 (1770447286) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 7287 fallocating to largest ever: 0x40000 000001 1770447287.107454 fallocate 0x03199c thru 0x03ffff (0xe664 bytes) skipping zero size read 000005 1770447287.113701 mapwrite 0x03a529 thru 0x03ffff (0x5ad7 bytes) 000007 1770447287.146950 mapread 0x0128d5 thru 0x016822 (0x3f4e bytes) 000008 1770447287.151226 trunc from 0x040000 to 0x02cb1d (0x134e2 bytes) 000009 1770447287.156763 mapread 0x00323f thru 0x009ec6 (0x6c88 bytes) 000010 1770447287.162828 trunc from 0x01bf74 to 0x02b059 (0xf0e6 bytes) 000011 1770447287.166016 mapread 0x020445 thru 0x02cb1d (0xc6d9 bytes) 000014 1770447287.177244 MIRROR_EXTEND to 2 mirrors 000015 1770447287.396808 read 0x0029f0 thru 0x0106e6 (0xdcf7 bytes) 000018 1770447287.400206 write 0x0085d7 thru 0x013da7 (0xb7d1 bytes) 000019 1770447287.416497 mapwrite 0x035395 thru 0x03ffff (0xac6b bytes) 000020 1770447287.445385 punch from 0x00da8d to 0x01c23b (0xe7af bytes) 000021 1770447287.447909 read 0x005de5 thru 0x00eff5 (0x9211 bytes) 000022 1770447287.452391 mapwrite 0x02e7e6 thru 0x02edf6 (0x611 bytes) 000023 1770447287.459536 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc903be6a Mirror 3: 0xc903be6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000025 1770447287.740019 mapwrite 0x01bf5a thru 0x01d9dc (0x1a83 bytes) 000026 1770447287.785253 mapread 0x03ce1e thru 0x03ffff (0x31e2 bytes) 000027 1770447287.789313 trunc from 0x040000 to 0x0115c7 (0x2ea38 bytes) 000028 1770447287.821889 read 0x00599c thru 0x00c70e (0x6d73 bytes) 000030 1770447287.829071 punch from 0x00ff3e to 0x0115c7 (0x168a bytes) 000032 1770447287.832069 write 0x02c17a thru 0x0374b6 (0xb33d bytes) 000033 1770447287.833500 punch from 0x01ae02 to 0x01bd5f (0xf5e bytes) 000034 1770447287.836441 punch from 0x011ebc to 0x016d05 (0x4e4a bytes) 000035 1770447287.838812 trunc from 0x0374b7 to 0x02af23 (0xc593 bytes) 000036 1770447287.847038 MIRROR_RESYNC 3 mirrors 000037 1770447288.140545 mapread 0x01e968 thru 0x020828 (0x1ec1 bytes) 000038 1770447288.145972 write 0x02ed53 thru 0x035694 (0x6942 bytes) 000039 1770447288.157088 MIRROR_RESYNC 3 mirrors 000040 1770447288.476791 write 0x025dad thru 0x02b258 (0x54ac bytes) 000041 1770447288.494409 mapwrite 0x012987 thru 0x01ed00 (0xc37a bytes) 000042 1770447288.514727 fallocate 0x018568 thru 0x0224a0 (0x9f39 bytes) 000043 1770447288.519484 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35695): Mirror 3: 0x8a1c19ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35695): Mirror 3: 0x8a1c19ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35695: skipped 000044 1770447288.666316 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) 000045 1770447288.730892 fallocate 0x0082ce thru 0x010283 (0x7fb6 bytes) 000046 1770447288.735886 punch from 0x030dec to 0x035694 (0x48a9 bytes) 000048 1770447288.738940 punch from 0x0354a0 to 0x035694 (0x1f5 bytes) 000049 1770447288.742694 fallocate 0x021b2f thru 0x023ee0 (0x23b2 bytes) 000050 1770447288.746263 trunc from 0x035695 to 0x00e524 (0x27170 bytes) 000051 1770447288.755852 trunc from 0x00e525 to 0x03d065 (0x2eb41 bytes) 000052 1770447288.768092 read 0x01c454 thru 0x0270dc (0xac89 bytes) 000053 1770447288.774782 write 0x0106d2 thru 0x0124a6 (0x1dd5 bytes) 000054 1770447288.780461 fallocate 0x008518 thru 0x00e4e8 (0x5fd1 bytes) 000055 1770447288.783202 mapread 0x002833 thru 0x009336 (0x6b04 bytes) 000057 1770447288.790324 fallocate 0x023730 thru 0x025718 (0x1fe9 bytes) 000059 1770447288.793498 trunc from 0x00f2b5 to 0x018208 (0x8f54 bytes) 000060 1770447288.802227 trunc from 0x00bfac to 0x011245 (0x529a bytes) 000062 1770447288.805939 trunc from 0x03d066 to 0x000ab6 (0x3c5af bytes) 000063 1770447288.811401 mapwrite 0x0224f8 thru 0x02f3d6 (0xcedf bytes) 000064 1770447288.842880 MIRROR_EXTEND to 2 mirrors 000067 1770447289.078774 read 0x0153e1 thru 0x02279d (0xd3bd bytes) 000068 1770447289.081837 read 0x019e88 thru 0x01c346 (0x24bf bytes) 000069 1770447289.082174 fallocate 0x015ddf thru 0x0188ba (0x2adc bytes) 000070 1770447289.100997 fallocate 0x011635 thru 0x0156b6 (0x4082 bytes) 000071 1770447289.104125 trunc from 0x00db16 to 0x0106ec (0x2bd7 bytes) 000072 1770447289.106432 fallocate 0x031a27 thru 0x038799 (0x6d73 bytes) 000073 1770447289.108632 write 0x02ba9f thru 0x035a18 (0x9f7a bytes) 000075 1770447289.117544 MIRROR_RESYNC 2 mirrors 000077 1770447289.298142 mapwrite 0x014830 thru 0x022497 (0xdc68 bytes) 000078 1770447289.339994 write 0x028f8a thru 0x0388db (0xf952 bytes) 000079 1770447289.345941 trunc from 0x020fc0 to 0x029f23 (0x8f64 bytes) 000080 1770447289.352229 mapread 0x00fb15 thru 0x012c5a (0x3146 bytes) 000083 1770447289.353734 fallocate 0x00427d thru 0x009cf5 (0x5a79 bytes) 000085 1770447289.357277 mapwrite 0x02c484 thru 0x03947c (0xcff9 bytes) 000086 1770447289.384914 read 0x02bb1f thru 0x036cf0 (0xb1d2 bytes) 000087 1770447289.385613 write 0x032d3f thru 0x03ffff (0xd2c1 bytes) 000088 1770447289.387249 punch from 0x024cf7 to 0x0253e8 (0x6f2 bytes) 000089 1770447289.391173 fallocate 0x033db0 thru 0x03da8e (0x9cdf bytes) 000090 1770447289.394156 punch from 0x0035cc to 0x012ab3 (0xf4e8 bytes) 000091 1770447289.396857 write 0x000996 thru 0x000c17 (0x282 bytes) 000092 1770447289.402777 mapwrite 0x01896b thru 0x01d7c5 (0x4e5b bytes) 000093 1770447289.413384 write 0x0087f0 thru 0x008e00 (0x611 bytes) 000094 1770447289.414228 mapread 0x032415 thru 0x035086 (0x2c72 bytes) 000095 1770447289.414981 fallocate 0x018583 thru 0x01b848 (0x32c6 bytes) 000096 1770447289.420433 mapread 0x00894c thru 0x015173 (0xc828 bytes) 000097 1770447289.426419 mapread 0x002d26 thru 0x00a6e8 (0x79c3 bytes) 000098 1770447289.427261 write 0x0278f1 thru 0x02e671 (0x6d81 bytes) 000099 1770447289.433471 punch from 0x019b4b to 0x021e59 (0x830f bytes) 000101 1770447289.436253 punch from 0x0175d1 to 0x021cf3 (0xa723 bytes) 000102 1770447289.438147 mapread 0x03c984 thru 0x03ffff (0x367c bytes) 000103 1770447289.438699 fallocate 0x003a1d thru 0x00b582 (0x7b66 bytes) 000104 1770447289.440556 read 0x0076b0 thru 0x011ffc (0xa94d bytes) 000105 1770447289.441003 trunc from 0x040000 to 0x010e97 (0x2f168 bytes) 000106 1770447289.446325 punch from 0x002b1d to 0x00a356 (0x783a bytes) 000107 1770447289.452359 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x10e98): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10e98): Mirror 2: 0x63de995c (stale) CRC-32 checksum value for chunk [0, 0x10e98): Mirror 1: 0xc2406d75 Mirror 3: 0xc2406d75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10e98: skipped 000108 1770447289.758248 write 0x02ba00 thru 0x03546b (0x9a6c bytes) 000109 1770447289.768526 write 0x017a38 thru 0x027679 (0xfc42 bytes) 000110 1770447289.771971 trunc from 0x01b2b5 to 0x02585f (0xa5ab bytes) 000111 1770447289.776510 trunc from 0x025349 to 0x02e0a5 (0x8d5d bytes) 000112 1770447289.778215 read 0x0260a5 thru 0x029504 (0x3460 bytes) 000114 1770447289.780783 read 0x01eb6e thru 0x02ae63 (0xc2f6 bytes) 000115 1770447289.784777 mapread 0x033d14 thru 0x03546b (0x1758 bytes) 000116 1770447289.785511 mapread 0x023588 thru 0x030eb2 (0xd92b bytes) 000117 1770447289.786134 trunc from 0x03546c to 0x002ade (0x3298d bytes) 000118 1770447289.792358 trunc from 0x002adf to 0x01b5ff (0x18b21 bytes) 000119 1770447289.796876 trunc from 0x012c1b to 0x01afc1 (0x83a7 bytes) 000121 1770447289.801318 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b600): Mirror 2: 0x9bb4f430 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b600): Mirror 2: 0x9bb4f430 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b600: skipped 000122 1770447289.919045 read 0x0081b3 thru 0x01507f (0xcecd bytes) 000123 1770447289.923201 write 0x01ba1a thru 0x026591 (0xab78 bytes) 000124 1770447289.930939 mapread 0x006fc8 thru 0x00f756 (0x878f bytes) 000125 1770447289.936924 punch from 0x010446 to 0x0176ec (0x72a7 bytes) 000126 1770447289.939271 punch from 0x02251b to 0x026591 (0x4077 bytes) 000127 1770447289.944553 write 0x035348 thru 0x03ffff (0xacb8 bytes) 000129 1770447289.946160 mapwrite 0x005ab5 thru 0x015a2e (0xff7a bytes) 000130 1770447289.958239 fallocate 0x01f1f8 thru 0x024d3e (0x5b47 bytes) 000131 1770447289.960430 trunc from 0x040000 to 0x00b5f2 (0x34a0d bytes) 000132 1770447289.966235 trunc from 0x00836a to 0x00b5f2 (0x3289 bytes) 000133 1770447289.970610 MIRROR_RESYNC 2 mirrors 000134 1770447290.217319 write 0x0371d9 thru 0x039edd (0x2d05 bytes) 000135 1770447290.235296 read 0x00fa79 thru 0x017b04 (0x808c bytes) 000136 1770447290.243430 trunc from 0x039ede to 0x0111e0 (0x28cfd bytes) 000137 1770447290.250909 trunc from 0x0111e1 to 0x03a9d0 (0x297f0 bytes) 000138 1770447290.255639 trunc from 0x03a9d1 to 0x023ef8 (0x16ad8 bytes) 000141 1770447290.260762 punch from 0x01c35d to 0x01d67f (0x1323 bytes) 000142 1770447290.262581 mapread 0x00ed60 thru 0x01d38c (0xe62d bytes) 000144 1770447290.273686 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) 000145 1770447290.319263 trunc from 0x023ca6 to 0x023ef8 (0x253 bytes) 000146 1770447290.321498 trunc from 0x0230f0 to 0x023ef8 (0xe09 bytes) 000148 1770447290.323778 punch from 0x01cd27 to 0x023ef8 (0x71d2 bytes) 000149 1770447290.325853 trunc from 0x023ef9 to 0x03260b (0xe713 bytes) 000150 1770447290.335202 trunc from 0x03260c to 0x000dae (0x3185d bytes) 000151 1770447290.340223 mapread 0x000bfa thru 0x000dae (0x1b5 bytes) 000152 1770447290.342503 mapwrite 0x0125eb thru 0x014ee5 (0x28fb bytes) 000154 1770447290.361782 mapwrite 0x00b358 thru 0x010e83 (0x5b2c bytes) 000156 1770447290.375404 trunc from 0x00ed7b to 0x014ee5 (0x616b bytes) 000157 1770447290.381709 read 0x007280 thru 0x0113c3 (0xa144 bytes) 000158 1770447290.387195 punch from 0x00af8f to 0x01267e (0x76f0 bytes) 000159 1770447290.391002 fallocate 0x01f9dc thru 0x02f955 (0xff7a bytes) 000160 1770447290.394906 punch from 0x016576 to 0x024b11 (0xe59c bytes) 000161 1770447290.399890 MIRROR_EXTEND to 2 mirrors 000162 1770447290.595048 fallocate 0x019bcd thru 0x01accb (0x10ff bytes) 000163 1770447290.607039 mapwrite 0x014a3d thru 0x01ffa0 (0xb564 bytes) 000164 1770447290.625918 mapread 0x01ab34 thru 0x01d8f1 (0x2dbe bytes) 000165 1770447290.626621 write 0x033f57 thru 0x03d852 (0x98fc bytes) 000166 1770447290.630542 punch from 0x035930 to 0x03919a (0x386b bytes) 000167 1770447290.637872 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) 000168 1770447290.690460 trunc from 0x0210ce to 0x02fdb2 (0xece5 bytes) 000169 1770447290.692549 write 0x02a4df thru 0x030c61 (0x6783 bytes) 000171 1770447290.698777 read 0x02e872 thru 0x039713 (0xaea2 bytes) 000172 1770447290.702423 write 0x03f911 thru 0x03ffff (0x6ef bytes) 000173 1770447290.703407 punch from 0x02daa1 to 0x02e695 (0xbf5 bytes) 000174 1770447290.716993 punch from 0x02b5df to 0x02c0df (0xb01 bytes) 000175 1770447290.719207 trunc from 0x014b09 to 0x016407 (0x18ff bytes) 000176 1770447290.721405 fallocate 0x0242dd thru 0x031adf (0xd803 bytes) 000177 1770447290.724266 trunc from 0x040000 to 0x01c807 (0x237f8 bytes) 000178 1770447290.736210 punch from 0x00e730 to 0x01c807 (0xe0d8 bytes) 000179 1770447290.742327 fallocate 0x030115 thru 0x03d6ce (0xd5ba bytes) 000180 1770447290.747954 write 0x023257 thru 0x02369e (0x448 bytes) 000181 1770447290.753706 trunc from 0x03d6cf to 0x009304 (0x343ca bytes) 000182 1770447290.760074 mapread 0x003d4d thru 0x005228 (0x14dc bytes) 000183 1770447290.763711 fallocate 0x03f2bd thru 0x03ffff (0xd43 bytes) 000184 1770447290.766166 trunc from 0x017e35 to 0x0243b6 (0xc582 bytes) 000185 1770447290.768178 trunc from 0x040000 to 0x015661 (0x2a99e bytes) 000186 1770447290.774117 trunc from 0x015662 to 0x01ecb8 (0x9657 bytes) 000189 1770447290.779913 trunc from 0x01ecb9 to 0x005c0c (0x190ac bytes) 000190 1770447290.785833 mapwrite 0x02136a thru 0x0271e5 (0x5e7c bytes) 000192 1770447290.802917 trunc from 0x0271e6 to 0x030bdc (0x99f7 bytes) 000193 1770447290.807908 mapwrite 0x013847 thru 0x0208e2 (0xd09c bytes) 000194 1770447290.817262 read 0x006db1 thru 0x013420 (0xc670 bytes) 000197 1770447290.824643 trunc from 0x030bdd to 0x01bb69 (0x15073 bytes) 000199 1770447290.830217 mapwrite 0x01f3d7 thru 0x022859 (0x3483 bytes) 000201 1770447290.845064 write 0x02cf7f thru 0x031224 (0x42a6 bytes) 000202 1770447290.846039 read 0x026749 thru 0x031224 (0xaadc bytes) 000203 1770447290.849013 mapread 0x00ce12 thru 0x0146d7 (0x78c6 bytes) 000204 1770447290.849724 trunc from 0x00c191 to 0x0193d1 (0xd241 bytes) 000205 1770447290.852079 fallocate 0x03f1c5 thru 0x03ffff (0xe3b bytes) 000206 1770447290.854402 mapread 0x0209f9 thru 0x027569 (0x6b71 bytes) 000207 1770447290.855321 mapread 0x00923a thru 0x013cc6 (0xaa8d bytes) 000208 1770447290.867810 fallocate 0x03e86d thru 0x03ffff (0x1793 bytes) 000209 1770447290.869830 fallocate 0x03c357 thru 0x03ffff (0x3ca9 bytes) 000210 1770447290.871688 mapwrite 0x00660a thru 0x007817 (0x120e bytes) 000211 1770447290.877896 mapread 0x03bf18 thru 0x03ffff (0x40e8 bytes) 000212 1770447290.878594 write 0x0058bf thru 0x007218 (0x195a bytes) 000213 1770447290.879507 trunc from 0x040000 to 0x01f3fe (0x20c01 bytes) 000214 1770447290.885536 write 0x02a1f5 thru 0x02cf97 (0x2da3 bytes) 000215 1770447290.886460 write 0x01f4df thru 0x025184 (0x5ca6 bytes) 000216 1770447290.890544 read 0x021da3 thru 0x027c7f (0x5edd bytes) 000217 1770447290.894296 trunc from 0x02cf98 to 0x01a9d9 (0x125be bytes) 000219 1770447290.904063 write 0x03d576 thru 0x03ffff (0x2a8a bytes) 000220 1770447290.905207 write 0x03834b thru 0x03ffff (0x7cb5 bytes) 000221 1770447290.909286 fallocate 0x007ba3 thru 0x014ba0 (0xcffe bytes) 000222 1770447290.911619 punch from 0x019235 to 0x01fe6f (0x6c3b bytes) 000223 1770447290.913545 read 0x027d81 thru 0x034bcb (0xce4b bytes) 000225 1770447290.917161 fallocate 0x025f92 thru 0x03223a (0xc2a9 bytes) 000226 1770447290.919122 trunc from 0x02d482 to 0x02f981 (0x2500 bytes) 000227 1770447290.920910 trunc from 0x040000 to 0x0210ba (0x1ef45 bytes) 000228 1770447290.926029 mapwrite 0x018e7d thru 0x01e9b2 (0x5b36 bytes) 000229 1770447290.933831 mapread 0x00133f thru 0x002f9e (0x1c60 bytes) 000230 1770447290.934548 write 0x01fecd thru 0x02a7bd (0xa8f1 bytes) 000231 1770447290.935584 read 0x02556c thru 0x02a7bd (0x5252 bytes) 000232 1770447290.936010 punch from 0x01ecd3 to 0x026e1b (0x8149 bytes) 000233 1770447290.941836 read 0x01e314 thru 0x023322 (0x500f bytes) 000234 1770447290.945517 write 0x02b81a thru 0x038a6f (0xd256 bytes) 000235 1770447290.947016 mapread 0x036394 thru 0x038a6f (0x26dc bytes) 000236 1770447290.947619 fallocate 0x03a77d thru 0x03ffff (0x5883 bytes) 000237 1770447290.949936 trunc from 0x03ecd3 to 0x03ffff (0x132d bytes) 000238 1770447290.951849 mapread 0x028ee4 thru 0x029eeb (0x1008 bytes) 000239 1770447290.952708 trunc from 0x00c4ce to 0x015440 (0x8f73 bytes) 000240 1770447290.955011 trunc from 0x040000 to 0x002819 (0x3d7e6 bytes) 000241 1770447290.962330 MIRROR_EXTEND to 2 mirrors 000242 1770447291.182494 read 0x000f1c thru 0x002819 (0x18fe bytes) 000243 1770447291.184719 trunc from 0x00281a to 0x031173 (0x2e95a bytes) 000244 1770447291.197473 fallocate 0x021724 thru 0x02feb5 (0xe792 bytes) 000245 1770447291.201060 MIRROR_RESYNC 2 mirrors 000246 1770447291.385280 trunc from 0x025a49 to 0x02d0d1 (0x7689 bytes) 000247 1770447291.399990 write 0x00926c thru 0x011b67 (0x88fc bytes) 000248 1770447291.403923 mapread 0x00de38 thru 0x01d227 (0xf3f0 bytes) 000249 1770447291.412795 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) 000251 1770447291.471430 punch from 0x0287db to 0x031173 (0x8999 bytes) 000252 1770447291.476545 mapread 0x019c9a thru 0x02018b (0x64f2 bytes) 000253 1770447291.489968 trunc from 0x031174 to 0x029ad6 (0x769d bytes) 000254 1770447291.496127 mapread 0x012ab5 thru 0x01aae8 (0x8034 bytes) 000255 1770447291.499197 read 0x003e14 thru 0x008cfa (0x4ee7 bytes) 000257 1770447291.502758 trunc from 0x00bab2 to 0x01ae59 (0xf3a8 bytes) 000258 1770447291.505038 write 0x023477 thru 0x02f7c6 (0xc350 bytes) 000260 1770447291.506329 trunc from 0x02f7c7 to 0x018140 (0x17686 bytes) 000261 1770447291.510935 read 0x017a55 thru 0x018140 (0x6ec bytes) 000262 1770447291.513922 trunc from 0x00cfde to 0x00daa1 (0xac4 bytes) 000263 1770447291.515610 fallocate 0x00fa73 thru 0x01188d (0x1e1b bytes) 000264 1770447291.517423 punch from 0x0086c3 to 0x00879d (0xdb bytes) 000265 1770447291.518938 trunc from 0x00ef4a to 0x018140 (0x91f7 bytes) 000267 1770447291.521119 trunc from 0x018141 to 0x03d2a1 (0x25161 bytes) 000268 1770447291.526151 fallocate 0x026f17 thru 0x0365f8 (0xf6e2 bytes) 000270 1770447291.528463 read 0x01102f thru 0x018afa (0x7acc bytes) 000271 1770447291.532158 trunc from 0x039efd to 0x03aefc (0x1000 bytes) 000273 1770447291.534602 mapread 0x01ab1a thru 0x029245 (0xe72c bytes) 000274 1770447291.535490 fallocate 0x004034 thru 0x007720 (0x36ed bytes) 000276 1770447291.538194 fallocate 0x00b08f thru 0x013cd0 (0x8c42 bytes) 000277 1770447291.540097 mapwrite 0x010606 thru 0x011327 (0xd22 bytes) 000278 1770447291.546959 trunc from 0x004986 to 0x00ddf4 (0x946f bytes) 000279 1770447291.549476 mapread 0x00a6e4 thru 0x019512 (0xee2f bytes) 000280 1770447291.559181 mapread 0x01eb8a thru 0x029f04 (0xb37b bytes) 000281 1770447291.559751 fallocate 0x00f1f8 thru 0x011cf1 (0x2afa bytes) 000282 1770447291.561707 fallocate 0x0140ae thru 0x019f9d (0x5ef0 bytes) 000283 1770447291.563428 mapread 0x039cf2 thru 0x03d2a1 (0x35b0 bytes) 000284 1770447291.568902 read 0x0097a8 thru 0x00cebf (0x3718 bytes) 000285 1770447291.572083 fallocate 0x034ce4 thru 0x03e696 (0x99b3 bytes) 000286 1770447291.574561 mapwrite 0x01e9c0 thru 0x01eca7 (0x2e8 bytes) 000287 1770447291.583671 punch from 0x0079f6 to 0x014b7d (0xd188 bytes) 000288 1770447291.586541 write 0x03fff2 thru 0x03ffff (0xe bytes) 000289 1770447291.587429 punch from 0x0100f6 to 0x0101be (0xc9 bytes) 000290 1770447291.589498 write 0x01b3a2 thru 0x01d119 (0x1d78 bytes) 000291 1770447291.590321 write 0x00bbd6 thru 0x016be7 (0xb012 bytes) 000292 1770447291.593998 trunc from 0x040000 to 0x02df20 (0x120df bytes) 000293 1770447291.599073 fallocate 0x01d074 thru 0x026ebb (0x9e48 bytes) 000294 1770447291.600936 punch from 0x014e07 to 0x01bad4 (0x6cce bytes) 000295 1770447291.605952 read 0x00c887 thru 0x018f51 (0xc6cb bytes) 000296 1770447291.609069 mapread 0x0186ae thru 0x027aa7 (0xf3fa bytes) 000298 1770447291.610244 fallocate 0x00b9cf thru 0x00d0ab (0x16dd bytes) 000299 1770447291.612268 write 0x02db77 thru 0x03a369 (0xc7f3 bytes) 000300 1770447291.613375 mapwrite 0x031c87 thru 0x035931 (0x3cab bytes) 000301 1770447291.619129 trunc from 0x03a36a to 0x021c52 (0x18717 bytes) 000302 1770447291.623462 mapwrite 0x0128db thru 0x017fbf (0x56e5 bytes) 000303 1770447291.629845 fallocate 0x0219f7 thru 0x026e17 (0x5421 bytes) 000304 1770447291.631873 fallocate 0x008b18 thru 0x017b04 (0xefed bytes) 000306 1770447291.634272 punch from 0x000490 to 0x0087ce (0x833f bytes) 000308 1770447291.637208 mapwrite 0x02dd30 thru 0x0358ae (0x7b7f bytes) 000309 1770447291.663018 fallocate 0x01684e thru 0x025bcc (0xf37f bytes) 000310 1770447291.665613 mapread 0x01a078 thru 0x029f74 (0xfefd bytes) 000311 1770447291.678010 punch from 0x022c2e to 0x02fec8 (0xd29b bytes) 000312 1770447291.680477 read 0x01b807 thru 0x01f605 (0x3dff bytes) 000313 1770447291.681057 punch from 0x000db4 to 0x010c6d (0xfeba bytes) 000314 1770447291.683682 read 0x012662 thru 0x01e5b5 (0xbf54 bytes) 000315 1770447291.684168 fallocate 0x03d30b thru 0x03e86d (0x1563 bytes) 000316 1770447291.686380 write 0x025d3f thru 0x0316c9 (0xb98b bytes) 000317 1770447291.691108 mapread 0x02b2ae thru 0x035401 (0xa154 bytes) 000319 1770447291.692299 punch from 0x00ed92 to 0x01a45f (0xb6ce bytes) 000320 1770447291.695006 trunc from 0x01facf to 0x027e02 (0x8334 bytes) 000321 1770447291.701757 mapwrite 0x0044b5 thru 0x00bffa (0x7b46 bytes) 000322 1770447291.709257 trunc from 0x0358af to 0x009838 (0x2c076 bytes) 000323 1770447291.714833 mapwrite 0x01d24f thru 0x02b154 (0xdf06 bytes) 000324 1770447291.768759 trunc from 0x012f96 to 0x01edbf (0xbe2a bytes) 000325 1770447291.771293 mapwrite 0x039879 thru 0x03dff0 (0x4778 bytes) 000326 1770447291.785624 write 0x02c4f3 thru 0x02c8b2 (0x3c0 bytes) 000327 1770447291.787431 punch from 0x015961 to 0x016b98 (0x1238 bytes) 000328 1770447291.788988 write 0x01e646 thru 0x02cdd7 (0xe792 bytes) 000329 1770447291.790872 read 0x036b75 thru 0x03b226 (0x46b2 bytes) 000330 1770447291.791332 read 0x011449 thru 0x0172f1 (0x5ea9 bytes) 000331 1770447291.794344 punch from 0x0311fc to 0x03b67c (0xa481 bytes) 000332 1770447291.796251 mapread 0x02732a thru 0x02ca9f (0x5776 bytes) 000333 1770447291.797043 mapread 0x007af3 thru 0x00dad4 (0x5fe2 bytes) 000334 1770447291.797665 write 0x029b00 thru 0x0367e4 (0xcce5 bytes) 000335 1770447291.800267 fallocate 0x033a9a thru 0x03ffff (0xc566 bytes) 000337 1770447291.803171 write 0x028ce9 thru 0x035c9b (0xcfb3 bytes) 000338 1770447291.804228 fallocate 0x021aeb thru 0x02849a (0x69b0 bytes) 000339 1770447291.806208 trunc from 0x040000 to 0x029eb7 (0x16148 bytes) 000340 1770447291.811543 write 0x01ff46 thru 0x025d40 (0x5dfb bytes) 000341 1770447291.812321 mapwrite 0x038ed0 thru 0x03ffff (0x7130 bytes) 000343 1770447291.833961 mapwrite 0x024375 thru 0x027720 (0x33ac bytes) 000344 1770447291.838674 fallocate 0x02be00 thru 0x0303b7 (0x45b8 bytes) 000345 1770447291.840311 mapwrite 0x035dc8 thru 0x03ffff (0xa238 bytes) 000346 1770447291.852613 mapread 0x02d784 thru 0x02d90f (0x18c bytes) 000347 1770447291.854985 punch from 0x01237f to 0x01eba5 (0xc827 bytes) 000348 1770447291.858398 MIRROR_EXTEND to 2 mirrors 000349 1770447292.062837 mapread 0x02e960 thru 0x030461 (0x1b02 bytes) 000350 1770447292.068266 read 0x028a48 thru 0x02de6d (0x5426 bytes) 000351 1770447292.069826 write 0x0123fd thru 0x01882f (0x6433 bytes) 000352 1770447292.084458 read 0x032f38 thru 0x03b2e2 (0x83ab bytes) 000353 1770447292.089605 mapread 0x00dab8 thru 0x01b968 (0xdeb1 bytes) 000354 1770447292.090288 write 0x0265d1 thru 0x027c8a (0x16ba bytes) 000355 1770447292.090927 punch from 0x028b55 to 0x030c09 (0x80b5 bytes) 000356 1770447292.092824 fallocate 0x01b4a2 thru 0x029ed4 (0xea33 bytes) 000357 1770447292.094348 read 0x030374 thru 0x031439 (0x10c6 bytes) 000358 1770447292.096833 punch from 0x0126f1 to 0x01a99e (0x82ae bytes) 000359 1770447292.102488 write 0x00a8dc thru 0x018c8d (0xe3b2 bytes) 000360 1770447292.105076 trunc from 0x040000 to 0x01fc31 (0x203ce bytes) 000361 1770447292.117269 MIRROR_RESYNC 2 mirrors 000362 1770447292.315200 read 0x018fb2 thru 0x01fc31 (0x6c80 bytes) 000363 1770447292.321956 read 0x00ebb5 thru 0x011bcc (0x3018 bytes) 000364 1770447292.326075 trunc from 0x01fc32 to 0x02f877 (0xfc46 bytes) 000365 1770447292.350146 trunc from 0x01f43d to 0x021605 (0x21c9 bytes) 000367 1770447292.357090 mapwrite 0x0070b8 thru 0x00d550 (0x6499 bytes) 000368 1770447292.377776 trunc from 0x021369 to 0x024b29 (0x37c1 bytes) 000369 1770447292.380440 trunc from 0x023583 to 0x028817 (0x5295 bytes) 000370 1770447292.383025 mapread 0x006826 thru 0x00c1bc (0x5997 bytes) 000371 1770447292.386170 trunc from 0x02f878 to 0x015558 (0x1a31f bytes) 000372 1770447292.400209 trunc from 0x0150de to 0x015558 (0x47b bytes) 000373 1770447292.402850 trunc from 0x013f0a to 0x01468c (0x783 bytes) 000374 1770447292.408969 trunc from 0x015559 to 0x0252e9 (0xfd91 bytes) 000375 1770447292.414894 mapread 0x00f2e5 thru 0x01e9de (0xf6fa bytes) 000376 1770447292.438861 trunc from 0x0252ea to 0x01a0aa (0xb23f bytes) 000377 1770447292.443740 trunc from 0x01124f to 0x019254 (0x8006 bytes) 000379 1770447292.446876 punch from 0x013c47 to 0x015c04 (0x1fbe bytes) 000380 1770447292.449317 trunc from 0x010224 to 0x014f96 (0x4d73 bytes) 000381 1770447292.451627 fallocate 0x00ea4b thru 0x0189f5 (0x9fab bytes) 000382 1770447292.453827 read 0x002fd8 thru 0x009388 (0x63b1 bytes) 000383 1770447292.457558 mapwrite 0x03dc61 thru 0x03ffff (0x239f bytes) 000384 1770447292.472936 trunc from 0x040000 to 0x00a3a0 (0x35c5f bytes) 000385 1770447292.478754 mapread 0x006d9a thru 0x00a3a0 (0x3607 bytes) 000386 1770447292.479504 mapread 0x003802 thru 0x00a3a0 (0x6b9f bytes) 000387 1770447292.481890 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) 000388 1770447292.521000 write 0x003223 thru 0x0077b4 (0x4592 bytes) 000389 1770447292.525059 read 0x00323f thru 0x00a3a0 (0x7162 bytes) 000391 1770447292.527463 trunc from 0x0008d9 to 0x003b22 (0x324a bytes) 000393 1770447292.533121 fallocate 0x01ad5a thru 0x023c48 (0x8eef bytes) 000394 1770447292.535420 read 0x0187fb thru 0x01e7c7 (0x5fcd bytes) 000395 1770447292.539009 trunc from 0x01eb2c to 0x023c48 (0x511d bytes) 000397 1770447292.540835 write 0x01786a thru 0x022948 (0xb0df bytes) 000398 1770447292.543211 mapread 0x008b31 thru 0x00a7db (0x1cab bytes) 000399 1770447292.544013 mapwrite 0x023ca9 thru 0x031d5e (0xe0b6 bytes) 000400 1770447292.558015 trunc from 0x02f6db to 0x031d5e (0x2684 bytes) 000401 1770447292.559684 trunc from 0x031d5f to 0x01fea0 (0x11ebe bytes) 000402 1770447292.563358 mapwrite 0x02cbd7 thru 0x03c820 (0xfc4a bytes) 000403 1770447292.579818 trunc from 0x03c821 to 0x011726 (0x2b0fa bytes) 000404 1770447292.583215 write 0x014a33 thru 0x01545d (0xa2b bytes) 000405 1770447292.583757 mapwrite 0x00bebe thru 0x0125cf (0x6712 bytes) 000406 1770447292.590570 mapwrite 0x034d32 thru 0x03ffff (0xb2ce bytes) 000407 1770447292.612238 write 0x017921 thru 0x02694f (0xf02f bytes) 000408 1770447292.616008 fallocate 0x0049a4 thru 0x00a40f (0x5a6c bytes) 000409 1770447292.617923 mapread 0x036c0c thru 0x03b63f (0x4a34 bytes) 000410 1770447292.618514 punch from 0x00abe1 to 0x01595d (0xad7d bytes) 000411 1770447292.620205 read 0x01f937 thru 0x028249 (0x8913 bytes) 000412 1770447292.623501 mapwrite 0x00c16e thru 0x010cd5 (0x4b68 bytes) 000413 1770447292.629008 punch from 0x0369ac to 0x03759c (0xbf1 bytes) 000414 1770447292.630891 read 0x000ead thru 0x00e7de (0xd932 bytes) 000415 1770447292.631506 punch from 0x027e7c to 0x03483c (0xc9c1 bytes) 000416 1770447292.633329 write 0x00cac5 thru 0x015967 (0x8ea3 bytes) 000417 1770447292.634282 mapread 0x03fc31 thru 0x03ffff (0x3cf bytes) 000418 1770447292.636533 MIRROR_EXTEND to 2 mirrors 000419 1770447292.844160 punch from 0x017e73 to 0x01a13b (0x22c9 bytes) 000420 1770447292.853581 mapwrite 0x01b1e0 thru 0x02645b (0xb27c bytes) 000421 1770447292.866771 read 0x03e387 thru 0x03ffff (0x1c79 bytes) 000422 1770447292.868859 mapread 0x03ec95 thru 0x03ffff (0x136b bytes) 000423 1770447292.869384 punch from 0x01a47e to 0x026e25 (0xc9a8 bytes) 000424 1770447292.872145 MIRROR_RESYNC 2 mirrors 000425 1770447293.100504 trunc from 0x040000 to 0x00256c (0x3da93 bytes) 000426 1770447293.110945 mapwrite 0x024c91 thru 0x02892a (0x3c9a bytes) 000427 1770447293.125801 trunc from 0x02892b to 0x001a0d (0x26f1d bytes) 000428 1770447293.130788 read 0x000048 thru 0x001a0d (0x19c6 bytes) 000429 1770447293.132951 trunc from 0x001705 to 0x001a0d (0x309 bytes) 000430 1770447293.134484 mapwrite 0x006b1e thru 0x006f70 (0x453 bytes) 000431 1770447293.145163 trunc from 0x006f71 to 0x009cf2 (0x2d82 bytes) 000432 1770447293.148877 mapread 0x005adc thru 0x009cf2 (0x4217 bytes) 000433 1770447293.151025 mapread 0x0024a9 thru 0x009cf2 (0x784a bytes) 000434 1770447293.151533 read 0x0031ef thru 0x009cf2 (0x6b04 bytes) 000435 1770447293.152054 punch from 0x002110 to 0x009cf2 (0x7be3 bytes) 000436 1770447293.153436 trunc from 0x009cf3 to 0x00aabc (0xdca bytes) 000437 1770447293.157093 trunc from 0x00aabd to 0x01333d (0x8881 bytes) 000438 1770447293.160677 fallocate 0x03306a thru 0x03308b (0x22 bytes) 000439 1770447293.162211 trunc from 0x00e8b1 to 0x0151b0 (0x6900 bytes) 000440 1770447293.163554 trunc from 0x00f224 to 0x011d2b (0x2b08 bytes) 000441 1770447293.165086 fallocate 0x039b01 thru 0x03ffff (0x64ff bytes) 000442 1770447293.166707 punch from 0x00c381 to 0x010dcd (0x4a4d bytes) 000443 1770447293.168158 mapread 0x01920a thru 0x01dbf7 (0x49ee bytes) 000444 1770447293.174010 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x247ba12e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 3: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000446 1770447293.395503 mapwrite 0x01e64b thru 0x01f27f (0xc35 bytes) 000447 1770447293.427278 mapwrite 0x00deda thru 0x01c8d4 (0xe9fb bytes) 000448 1770447293.438975 mapread 0x004969 thru 0x009e6f (0x5507 bytes) 000449 1770447293.443362 MIRROR_RESYNC 3 mirrors 000450 1770447293.663274 fallocate 0x038969 thru 0x03ffff (0x7697 bytes) 000451 1770447293.672330 write 0x0205a3 thru 0x02e93b (0xe399 bytes) 000452 1770447293.675398 punch from 0x014e39 to 0x01a61b (0x57e3 bytes) 000453 1770447293.676931 punch from 0x00e510 to 0x018dce (0xa8bf bytes) 000455 1770447293.678670 mapwrite 0x00afed thru 0x00ff9b (0x4faf bytes) 000456 1770447293.685830 read 0x02c810 thru 0x030a10 (0x4201 bytes) 000457 1770447293.687382 mapread 0x000c0e thru 0x001da0 (0x1193 bytes) 000458 1770447293.689060 write 0x029f9a thru 0x036976 (0xc9dd bytes) 000459 1770447293.691092 mapread 0x0140e7 thru 0x01d2c9 (0x91e3 bytes) 000461 1770447293.694772 fallocate 0x001828 thru 0x00f4b3 (0xdc8c bytes) 000462 1770447293.697613 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3e4e036c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3e4e036c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000463 1770447293.776615 write 0x0362d3 thru 0x036c75 (0x9a3 bytes) 000464 1770447293.780818 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) 000465 1770447293.818581 mapwrite 0x0065cd thru 0x0127e4 (0xc218 bytes) 000466 1770447293.834213 fallocate 0x022559 thru 0x024e5b (0x2903 bytes) 000467 1770447293.836232 mapread 0x008455 thru 0x00ec89 (0x6835 bytes) 000468 1770447293.837091 mapread 0x014b5c thru 0x01941f (0x48c4 bytes) 000469 1770447293.837894 trunc from 0x035145 to 0x03ffff (0xaebb bytes) 000472 1770447293.841842 mapread 0x0140e5 thru 0x0217ba (0xd6d6 bytes) 000474 1770447293.846168 trunc from 0x01720d to 0x024bb9 (0xd9ad bytes) 000475 1770447293.848532 mapread 0x00c1da thru 0x01146a (0x5291 bytes) 000477 1770447293.849879 trunc from 0x0366d9 to 0x03ed8a (0x86b2 bytes) 000478 1770447293.852448 trunc from 0x040000 to 0x00c4af (0x33b50 bytes) 000479 1770447293.864343 fallocate 0x031b5b thru 0x034005 (0x24ab bytes) 000480 1770447293.866431 write 0x024cc8 thru 0x02df3a (0x9273 bytes) 000481 1770447293.869569 MIRROR_EXTEND to 2 mirrors 000484 1770447294.060813 trunc from 0x028cf9 to 0x02df3a (0x5242 bytes) 000485 1770447294.071747 mapread 0x0197ff thru 0x0200a7 (0x68a9 bytes) 000486 1770447294.077994 fallocate 0x036423 thru 0x03cd23 (0x6901 bytes) 000487 1770447294.080220 mapwrite 0x00ddb7 thru 0x017113 (0x935d bytes) 000488 1770447294.090999 trunc from 0x031129 to 0x03a51e (0x93f6 bytes) 000489 1770447294.093059 trunc from 0x03cd24 to 0x033482 (0x98a1 bytes) 000492 1770447294.108945 write 0x010a1f thru 0x0196d1 (0x8cb3 bytes) 000493 1770447294.110075 trunc from 0x011402 to 0x017f34 (0x6b33 bytes) 000494 1770447294.115093 read 0x029216 thru 0x033482 (0xa26d bytes) 000495 1770447294.118523 mapwrite 0x02c10c thru 0x02edaa (0x2c9f bytes) 000496 1770447294.124176 read 0x03315c thru 0x033482 (0x327 bytes) 000498 1770447294.126251 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33483): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33483): Mirror 2: 0x44be6cc7 (stale) CRC-32 checksum value for chunk [0, 0x33483): Mirror 1: 0x9be64139 Mirror 3: 0x9be64139 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33483: skipped 000499 1770447294.408742 trunc from 0x02b2c5 to 0x02beb5 (0xbf1 bytes) 000500 1770447294.421830 punch from 0x00acc4 to 0x00b898 (0xbd5 bytes) 000501 1770447294.423636 trunc from 0x015786 to 0x01ddea (0x8665 bytes) 000502 1770447294.425477 write 0x01a5c0 thru 0x0285b7 (0xdff8 bytes) 000503 1770447294.430572 mapread 0x02fbc0 thru 0x032f78 (0x33b9 bytes) 000504 1770447294.433438 mapwrite 0x02ff0b thru 0x0373cb (0x74c1 bytes) 000505 1770447294.453122 trunc from 0x01692a to 0x01c79a (0x5e71 bytes) 000506 1770447294.454921 mapread 0x0027a3 thru 0x004bd2 (0x2430 bytes) 000508 1770447294.458323 trunc from 0x0373cc to 0x01aa77 (0x1c954 bytes) 000509 1770447294.463230 read 0x0028c1 thru 0x00ea3e (0xc17e bytes) 000510 1770447294.463701 punch from 0x00e3f3 to 0x014779 (0x6387 bytes) 000511 1770447294.465636 fallocate 0x01fbdc thru 0x025514 (0x5939 bytes) 000512 1770447294.467680 fallocate 0x039701 thru 0x03ffff (0x68ff bytes) 000513 1770447294.469608 trunc from 0x01046d to 0x012f08 (0x2a9c bytes) 000514 1770447294.471310 mapwrite 0x02375e thru 0x02b28a (0x7b2d bytes) 000515 1770447294.479810 fallocate 0x00d22d thru 0x01896b (0xb73f bytes) 000516 1770447294.481427 mapwrite 0x019d43 thru 0x01a0ec (0x3aa bytes) 000517 1770447294.487475 write 0x014995 thru 0x0195a8 (0x4c14 bytes) 000518 1770447294.489691 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8d393c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8d393c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000519 1770447294.575079 trunc from 0x00c82b to 0x015ce9 (0x94bf bytes) 000520 1770447294.577215 read 0x02fff0 thru 0x03b394 (0xb3a5 bytes) 000521 1770447294.580039 write 0x01c9b2 thru 0x021b2e (0x517d bytes) 000522 1770447294.584209 trunc from 0x040000 to 0x03daad (0x2552 bytes) 000523 1770447294.588781 punch from 0x00fe68 to 0x018e7a (0x9013 bytes) 000524 1770447294.590496 fallocate 0x02d6db thru 0x03c0cb (0xe9f1 bytes) 000525 1770447294.592230 trunc from 0x03daae to 0x03c58b (0x1522 bytes) 000526 1770447294.597985 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) 000527 1770447294.652934 mapwrite 0x024fd0 thru 0x0342b1 (0xf2e2 bytes) 000528 1770447294.670457 mapwrite 0x0010d7 thru 0x0054a6 (0x43d0 bytes) 000531 1770447294.679775 trunc from 0x02fef1 to 0x03c58b (0xc69b bytes) 000532 1770447294.682070 mapread 0x003129 thru 0x00ad7b (0x7c53 bytes) 000533 1770447294.682946 trunc from 0x019741 to 0x01d2fb (0x3bbb bytes) 000534 1770447294.685121 write 0x0301f0 thru 0x0357ca (0x55db bytes) 000535 1770447294.689597 mapread 0x029477 thru 0x036aa3 (0xd62d bytes) 000536 1770447294.690480 mapwrite 0x005053 thru 0x009e22 (0x4dd0 bytes) 000537 1770447294.696158 mapread 0x0331da thru 0x0354c5 (0x22ec bytes) 000538 1770447294.696800 mapwrite 0x00adb0 thru 0x014f9e (0xa1ef bytes) 000539 1770447294.706282 trunc from 0x02de14 to 0x02f0f7 (0x12e4 bytes) 000540 1770447294.708413 mapread 0x00fe0e thru 0x015e60 (0x6053 bytes) 000541 1770447294.709096 fallocate 0x02162d thru 0x029b93 (0x8567 bytes) 000542 1770447294.711159 read 0x017aed thru 0x0240f0 (0xc604 bytes) 000543 1770447294.713753 mapread 0x02e40f thru 0x03b843 (0xd435 bytes) 000546 1770447294.719044 mapwrite 0x02b628 thru 0x038236 (0xcc0f bytes) 000548 1770447294.728604 fallocate 0x00af65 thru 0x0193b1 (0xe44d bytes) 000549 1770447294.731373 fallocate 0x00874e thru 0x015fde (0xd891 bytes) 000550 1770447294.733944 punch from 0x001175 to 0x0067ba (0x5646 bytes) 000552 1770447294.736877 mapread 0x012a68 thru 0x021cb0 (0xf249 bytes) 000553 1770447294.737722 mapread 0x016015 thru 0x01f8b2 (0x989e bytes) 000554 1770447294.738500 trunc from 0x00e3bd to 0x01826e (0x9eb2 bytes) 000555 1770447294.744970 punch from 0x010f61 to 0x01b7af (0xa84f bytes) 000556 1770447294.748170 write 0x032f27 thru 0x03852e (0x5608 bytes) 000557 1770447294.749123 mapread 0x007a0e thru 0x0154f3 (0xdae6 bytes) 000558 1770447294.753040 read 0x012e04 thru 0x013aa8 (0xca5 bytes) 000559 1770447294.758684 MIRROR_EXTEND to 2 mirrors 000560 1770447294.951887 MIRROR_RESYNC 2 mirrors 000561 1770447295.061578 read 0x01cc7d thru 0x02759c (0xa920 bytes) 000563 1770447295.064492 fallocate 0x026e87 thru 0x033161 (0xc2db bytes) 000564 1770447295.076016 fallocate 0x01ef62 thru 0x02127d (0x231c bytes) 000565 1770447295.079250 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) 000566 1770447295.118289 trunc from 0x02d8eb to 0x02e603 (0xd19 bytes) 000567 1770447295.120112 trunc from 0x02d4df to 0x03c58b (0xf0ad bytes) 000568 1770447295.121747 punch from 0x00a479 to 0x01a252 (0xfdda bytes) 000569 1770447295.123548 read 0x038e05 thru 0x03c58b (0x3787 bytes) 000570 1770447295.126580 trunc from 0x039bcb to 0x03a940 (0xd76 bytes) 000571 1770447295.128668 trunc from 0x03c58c to 0x038075 (0x4516 bytes) 000572 1770447295.138924 mapread 0x028739 thru 0x02a8e6 (0x21ae bytes) 000573 1770447295.144192 trunc from 0x038076 to 0x02429f (0x13dd6 bytes) 000574 1770447295.148994 punch from 0x014d92 to 0x016064 (0x12d3 bytes) 000575 1770447295.150621 trunc from 0x0242a0 to 0x005f8e (0x1e311 bytes) 000578 1770447295.155186 read 0x004875 thru 0x005f8e (0x171a bytes) 000581 1770447295.158750 punch from 0x002a68 to 0x005f8e (0x3527 bytes) 000582 1770447295.160493 punch from 0x0039c5 to 0x005f8e (0x25ca bytes) 000583 1770447295.162075 trunc from 0x00152b to 0x005f8e (0x4a64 bytes) 000584 1770447295.164030 punch from 0x0033ca to 0x005f8e (0x2bc5 bytes) 000585 1770447295.165830 trunc from 0x005f8f to 0x02230b (0x1c37d bytes) 000586 1770447295.170919 mapwrite 0x012b41 thru 0x021c61 (0xf121 bytes) 000587 1770447295.182547 trunc from 0x02230c to 0x010753 (0x11bb8 bytes) 000589 1770447295.188768 read 0x00812e thru 0x010753 (0x8626 bytes) 000590 1770447295.189182 punch from 0x007cf0 to 0x010753 (0x8a64 bytes) 000591 1770447295.191458 trunc from 0x0031de to 0x00c469 (0x928c bytes) 000592 1770447295.193273 punch from 0x00877e to 0x010753 (0x7fd6 bytes) 000593 1770447295.194962 fallocate 0x037de2 thru 0x03ffff (0x821e bytes) 000594 1770447295.196809 punch from 0x02e2a1 to 0x03361b (0x537b bytes) 000596 1770447295.198547 write 0x0312d0 thru 0x03267c (0x13ad bytes) 000597 1770447295.202784 trunc from 0x0364c0 to 0x0395c8 (0x3109 bytes) 000599 1770447295.204837 write 0x00c323 thru 0x0176fb (0xb3d9 bytes) 000600 1770447295.205668 write 0x002e2e thru 0x0084cb (0x569e bytes) 000602 1770447295.206814 write 0x02c3e4 thru 0x02efd1 (0x2bee bytes) 000603 1770447295.207578 read 0x035e70 thru 0x03ffff (0xa190 bytes) 000604 1770447295.211869 mapread 0x01f296 thru 0x02ec66 (0xf9d1 bytes) 000605 1770447295.212620 write 0x02872c thru 0x02f4bd (0x6d92 bytes) 000607 1770447295.214122 read 0x03a802 thru 0x03ffff (0x57fe bytes) 000608 1770447295.214605 mapwrite 0x008fd6 thru 0x018223 (0xf24e bytes) 000609 1770447295.222822 trunc from 0x040000 to 0x00797e (0x38681 bytes) 000610 1770447295.227814 mapwrite 0x029bd2 thru 0x0391db (0xf60a bytes) 000612 1770447295.244347 mapread 0x015c98 thru 0x01d923 (0x7c8c bytes) 000613 1770447295.244991 fallocate 0x037f61 thru 0x03d1e4 (0x5284 bytes) 000614 1770447295.246855 trunc from 0x0391dc to 0x03a168 (0xf8d bytes) 000615 1770447295.250528 trunc from 0x03a169 to 0x00aae8 (0x2f680 bytes) 000616 1770447295.254405 read 0x009c33 thru 0x00aae8 (0xeb6 bytes) 000618 1770447295.255321 trunc from 0x000555 to 0x00aae8 (0xa594 bytes) 000619 1770447295.257835 write 0x01ab6a thru 0x0275fc (0xca93 bytes) 000620 1770447295.259361 write 0x004b19 thru 0x00cc52 (0x813a bytes) 000623 1770447295.263748 trunc from 0x00c4e4 to 0x013840 (0x735d bytes) 000624 1770447295.267966 trunc from 0x018ab6 to 0x01e52a (0x5a75 bytes) 000625 1770447295.269474 mapread 0x0111a8 thru 0x01bda9 (0xac02 bytes) 000626 1770447295.277802 punch from 0x00483b to 0x0088e8 (0x40ae bytes) 000627 1770447295.279464 mapread 0x0151ef thru 0x015c87 (0xa99 bytes) 000628 1770447295.280052 punch from 0x01fcd2 to 0x021116 (0x1445 bytes) 000629 1770447295.281600 fallocate 0x014a46 thru 0x01e253 (0x980e bytes) 000630 1770447295.283347 mapwrite 0x03f55f thru 0x03ffff (0xaa1 bytes) 000631 1770447295.293760 mapwrite 0x02866d thru 0x036eb5 (0xe849 bytes) 000632 1770447295.302174 punch from 0x008477 to 0x011b28 (0x96b2 bytes) 000633 1770447295.304198 write 0x03b052 thru 0x03ffff (0x4fae bytes) 000635 1770447295.305471 trunc from 0x040000 to 0x00fdd6 (0x30229 bytes) 000636 1770447295.311229 write 0x01956d thru 0x020527 (0x6fbb bytes) 000637 1770447295.312223 trunc from 0x020528 to 0x0226ac (0x2185 bytes) 000638 1770447295.316027 mapread 0x0040e7 thru 0x00a85e (0x6778 bytes) 000639 1770447295.324619 trunc from 0x0226ad to 0x027dac (0x5700 bytes) 000640 1770447295.328938 trunc from 0x00ac62 to 0x0136b2 (0x8a51 bytes) 000641 1770447295.330564 punch from 0x015bbc to 0x02314c (0xd591 bytes) 000642 1770447295.334197 trunc from 0x0201d0 to 0x021482 (0x12b3 bytes) 000643 1770447295.335551 trunc from 0x027dad to 0x0292d0 (0x1524 bytes) 000644 1770447295.339563 mapwrite 0x030370 thru 0x0363e4 (0x6075 bytes) 000645 1770447295.358852 mapwrite 0x019c07 thru 0x01ff7b (0x6375 bytes) 000646 1770447295.371211 trunc from 0x0363e5 to 0x009483 (0x2cf61 bytes) 000647 1770447295.375822 write 0x03a5b4 thru 0x03ffff (0x5a4c bytes) 000648 1770447295.377345 mapwrite 0x011cfa thru 0x0190a7 (0x73ae bytes) 000650 1770447295.395103 read 0x00681a thru 0x013f5c (0xd743 bytes) 000651 1770447295.400157 punch from 0x007893 to 0x016578 (0xece6 bytes) 000652 1770447295.402697 read 0x0200ca thru 0x021fc6 (0x1efd bytes) 000653 1770447295.403288 mapread 0x03f287 thru 0x03ffff (0xd79 bytes) 000654 1770447295.403941 trunc from 0x040000 to 0x00b0aa (0x34f55 bytes) 000655 1770447295.408831 trunc from 0x00b0ab to 0x039e99 (0x2edef bytes) 000658 1770447295.414153 mapwrite 0x028afa thru 0x036ce8 (0xe1ef bytes) 000659 1770447295.424983 mapwrite 0x000d85 thru 0x00c1b2 (0xb42e bytes) 000660 1770447295.431761 mapwrite 0x00561d thru 0x012ad7 (0xd4bb bytes) 000662 1770447295.439289 punch from 0x024786 to 0x02cabc (0x8337 bytes) 000663 1770447295.440912 mapread 0x02fd71 thru 0x031549 (0x17d9 bytes) 000664 1770447295.441424 mapwrite 0x02dd92 thru 0x03949d (0xb70c bytes) 000665 1770447295.448290 mapread 0x007256 thru 0x0102a6 (0x9051 bytes) 000666 1770447295.448921 read 0x019e9f thru 0x01b3dc (0x153e bytes) 000667 1770447295.449295 write 0x03d037 thru 0x03ffff (0x2fc9 bytes) 000668 1770447295.450177 punch from 0x03bd3a to 0x03f686 (0x394d bytes) 000669 1770447295.453292 read 0x005e92 thru 0x015e23 (0xff92 bytes) 000670 1770447295.453757 mapwrite 0x0304d7 thru 0x03ac4e (0xa778 bytes) 000672 1770447295.463004 trunc from 0x040000 to 0x02f2a7 (0x10d58 bytes) 000674 1770447295.468214 write 0x02eb3b thru 0x03b921 (0xcde7 bytes) 000675 1770447295.469228 trunc from 0x025a07 to 0x034354 (0xe94e bytes) 000678 1770447295.473814 mapwrite 0x03408e thru 0x03ffff (0xbf72 bytes) 000679 1770447295.488925 trunc from 0x024daf to 0x02a348 (0x559a bytes) 000680 1770447295.491169 mapread 0x020caa thru 0x02168d (0x9e4 bytes) 000682 1770447295.492482 trunc from 0x040000 to 0x0224a5 (0x1db5a bytes) 000683 1770447295.497423 fallocate 0x0323c1 thru 0x03b902 (0x9542 bytes) 000684 1770447295.499258 trunc from 0x00f10e to 0x01a95e (0xb851 bytes) 000685 1770447295.500836 punch from 0x02f804 to 0x031bee (0x23eb bytes) 000686 1770447295.502151 write 0x019d4b thru 0x029500 (0xf7b6 bytes) 000687 1770447295.505356 read 0x01bde4 thru 0x0270c8 (0xb2e5 bytes) 000688 1770447295.505798 write 0x00ffa4 thru 0x01b60c (0xb669 bytes) 000689 1770447295.509828 read 0x01de6b thru 0x02ce86 (0xf01c bytes) 000691 1770447295.510853 trunc from 0x02ee52 to 0x0392c8 (0xa477 bytes) 000692 1770447295.512968 punch from 0x014766 to 0x021bdd (0xd478 bytes) 000694 1770447295.517770 punch from 0x039eef to 0x03b902 (0x1a14 bytes) 000697 1770447295.520945 trunc from 0x03b903 to 0x013d61 (0x27ba1 bytes) 000698 1770447295.528507 MIRROR_EXTEND to 2 mirrors 000699 1770447295.694936 write 0x022051 thru 0x025d47 (0x3cf7 bytes) 000700 1770447295.705865 punch from 0x002baf to 0x003756 (0xba8 bytes) 000701 1770447295.707885 trunc from 0x000817 to 0x007334 (0x6b1e bytes) 000702 1770447295.711478 MIRROR_RESYNC 2 mirrors 000703 1770447295.899039 punch from 0x000663 to 0x0025ff (0x1f9d bytes) 000704 1770447295.908322 write 0x021414 thru 0x02b713 (0xa300 bytes) 000705 1770447295.910753 write 0x0008f1 thru 0x0086bb (0x7dcb bytes) 000706 1770447295.913825 mapread 0x01ca70 thru 0x01e892 (0x1e23 bytes) 000707 1770447295.916841 read 0x00f4e8 thru 0x018d85 (0x989e bytes) 000708 1770447295.919576 punch from 0x023854 to 0x02b713 (0x7ec0 bytes) 000709 1770447295.926560 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b714): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b714): Mirror 2: 0xa62ba3b9 (stale) CRC-32 checksum value for chunk [0, 0x2b714): Mirror 1: 0xd1ad98f1 Mirror 3: 0xd1ad98f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b714: skipped 000710 1770447296.204238 trunc from 0x0058e1 to 0x00ad50 (0x5470 bytes) 000711 1770447296.213688 mapwrite 0x025e54 thru 0x02c0df (0x628c bytes) 000712 1770447296.233058 mapread 0x000a83 thru 0x00b623 (0xaba1 bytes) 000713 1770447296.236800 read 0x0130c1 thru 0x01b12e (0x806e bytes) 000714 1770447296.239844 write 0x02be4f thru 0x031f5c (0x610e bytes) 000715 1770447296.240774 punch from 0x00d5fe to 0x019bb2 (0xc5b5 bytes) 000716 1770447296.242864 trunc from 0x0053a1 to 0x00c7ee (0x744e bytes) 000718 1770447296.244819 write 0x00b39c thru 0x0136d2 (0x8337 bytes) 000719 1770447296.248668 fallocate 0x005a9d thru 0x009408 (0x396c bytes) 000720 1770447296.252103 MIRROR_RESYNC 3 mirrors 000721 1770447296.527983 read 0x01b8ba thru 0x020769 (0x4eb0 bytes) 000722 1770447296.530417 trunc from 0x031f5d to 0x0189ed (0x1956f bytes) 000723 1770447296.546789 mapread 0x015804 thru 0x0189ed (0x31ea bytes) 000724 1770447296.551906 mapread 0x013ce7 thru 0x0189ed (0x4d07 bytes) 000725 1770447296.552507 fallocate 0x015b9b thru 0x01987b (0x3ce1 bytes) 000726 1770447296.554507 read 0x0020d1 thru 0x010ce1 (0xec11 bytes) 000727 1770447296.557784 write 0x0231e5 thru 0x03209b (0xeeb7 bytes) 000728 1770447296.559221 mapread 0x00e1cb thru 0x019b97 (0xb9cd bytes) 000729 1770447296.561986 MIRROR_RESYNC 3 mirrors 000730 1770447296.772911 write 0x0153d3 thru 0x024cbc (0xf8ea bytes) 000731 1770447296.784366 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3209c): Mirror 3: 0xa2f2c9cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3209c): Mirror 3: 0xa2f2c9cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3209c: skipped 000732 1770447296.867623 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3209c): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3209c): Mirror 3: 0xa2f2c9cf (stale) CRC-32 checksum value for chunk [0, 0x3209c): Mirror 1: 0x94f6b378 Mirror 4: 0x94f6b378 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3209c: skipped 000733 1770447297.116936 punch from 0x02d934 to 0x03209b (0x4768 bytes) 000734 1770447297.126627 fallocate 0x02f436 thru 0x03275a (0x3325 bytes) 000735 1770447297.128057 write 0x03b931 thru 0x03f40c (0x3adc bytes) 000736 1770447297.129837 read 0x01a103 thru 0x025ed7 (0xbdd5 bytes) 000737 1770447297.132255 trunc from 0x03c2c2 to 0x03cb5c (0x89b bytes) 000738 1770447297.136178 read 0x02cd67 thru 0x0369ad (0x9c47 bytes) 000739 1770447297.141045 fallocate 0x001ac7 thru 0x00a483 (0x89bd bytes) 000740 1770447297.143019 fallocate 0x00d28a thru 0x018e64 (0xbbdb bytes) 000741 1770447297.144506 write 0x035236 thru 0x03c3f2 (0x71bd bytes) 000743 1770447297.145794 mapread 0x0199e6 thru 0x0240c8 (0xa6e3 bytes) 000744 1770447297.146444 fallocate 0x0226bd thru 0x029f5f (0x78a3 bytes) 000745 1770447297.148146 trunc from 0x0370f2 to 0x03f40c (0x831b bytes) 000746 1770447297.151830 fallocate 0x03c4c9 thru 0x03ffff (0x3b37 bytes) 000747 1770447297.154996 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f40d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f40d): Mirror 3: 0xccef339f (stale) Mirror 4: 0xa9f95d80 (stale) CRC-32 checksum value for chunk [0, 0x3f40d): Mirror 1: 0x39bc639c Mirror 5: 0x39bc639c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f40d: skipped 000748 1770447297.410631 mapread 0x0080c6 thru 0x00970d (0x1648 bytes) 000749 1770447297.413365 punch from 0x02df52 to 0x0366dc (0x878b bytes) 000750 1770447297.422870 trunc from 0x025f2c to 0x02738f (0x1464 bytes) 000751 1770447297.425735 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f40d): Mirror 4: 0xa9f95d80 (stale) Mirror 5: 0x39bc639c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f40d): Mirror 4: 0xa9f95d80 (stale) Mirror 5: 0x39bc639c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f40d: skipped 000752 1770447297.520053 mapread 0x01db7b thru 0x020b7b (0x3001 bytes) 000754 1770447297.526289 trunc from 0x03f40d to 0x009f80 (0x3548c bytes) 000755 1770447297.540006 MIRROR_RESYNC 3 mirrors 000756 1770447297.755186 read 0x000023 thru 0x009f80 (0x9f5e bytes) 000757 1770447297.758079 punch from 0x009edc to 0x009f80 (0xa5 bytes) 000758 1770447297.766759 mapwrite 0x0172e4 thru 0x020cbd (0x99da bytes) 000759 1770447297.785271 mapwrite 0x031ddf thru 0x03a013 (0x8235 bytes) 000760 1770447297.798708 read 0x01a540 thru 0x01ff02 (0x59c3 bytes) 000761 1770447297.799293 write 0x03e2be thru 0x03ffff (0x1d42 bytes) 000762 1770447297.800123 mapread 0x003936 thru 0x0133ea (0xfab5 bytes) 000763 1770447297.804219 punch from 0x027f6a to 0x031908 (0x999f bytes) 000764 1770447297.807854 MIRROR_RESYNC 3 mirrors 000765 1770447298.102257 MIRROR_RESYNC 3 mirrors 000766 1770447298.241182 write 0x015569 thru 0x01c2a6 (0x6d3e bytes) 000767 1770447298.251816 mapread 0x020739 thru 0x0262ed (0x5bb5 bytes) 000769 1770447298.255594 trunc from 0x02d68a to 0x02e785 (0x10fc bytes) 000770 1770447298.257389 fallocate 0x00b060 thru 0x01ae1c (0xfdbd bytes) 000771 1770447298.259096 punch from 0x016df5 to 0x020432 (0x963e bytes) 000773 1770447298.263571 write 0x013c4a thru 0x021be8 (0xdf9f bytes) 000774 1770447298.266960 punch from 0x03c76b to 0x03ffff (0x3895 bytes) 000775 1770447298.268724 mapread 0x038406 thru 0x03ffff (0x7bfa bytes) 000776 1770447298.271490 mapwrite 0x030864 thru 0x03b9fe (0xb19b bytes) 000777 1770447298.281151 write 0x0135ec thru 0x01b8e7 (0x82fc bytes) 000778 1770447298.282039 write 0x031886 thru 0x03d0d6 (0xb851 bytes) 000780 1770447298.283161 punch from 0x03ee37 to 0x03ffff (0x11c9 bytes) 000781 1770447298.285255 mapread 0x021286 thru 0x02246f (0x11ea bytes) 000782 1770447298.286055 punch from 0x008cf9 to 0x00fb10 (0x6e18 bytes) 000783 1770447298.287978 write 0x023d1c thru 0x02e6c5 (0xa9aa bytes) 000784 1770447298.289033 trunc from 0x040000 to 0x0095ce (0x36a31 bytes) 000785 1770447298.295113 mapread 0x0080a5 thru 0x0095ce (0x152a bytes) 000786 1770447298.298700 fallocate 0x006f90 thru 0x016421 (0xf492 bytes) 000787 1770447298.300943 mapread 0x001c02 thru 0x0086a3 (0x6aa2 bytes) 000789 1770447298.302160 trunc from 0x007175 to 0x0095ce (0x245a bytes) 000790 1770447298.303943 trunc from 0x0095cf to 0x01cbb6 (0x135e8 bytes) 000791 1770447298.308337 mapread 0x017288 thru 0x019c36 (0x29af bytes) 000792 1770447298.311373 fallocate 0x012902 thru 0x01a230 (0x792f bytes) 000793 1770447298.313271 mapwrite 0x02fe8f thru 0x03bfd2 (0xc144 bytes) 000794 1770447298.329393 punch from 0x009f9f to 0x016268 (0xc2ca bytes) 000795 1770447298.331162 punch from 0x016442 to 0x01ef48 (0x8b07 bytes) 000796 1770447298.332697 mapwrite 0x00eebe thru 0x019e16 (0xaf59 bytes) 000797 1770447298.344195 write 0x030570 thru 0x031bd8 (0x1669 bytes) 000798 1770447298.345211 read 0x035cae thru 0x03bfd2 (0x6325 bytes) 000799 1770447298.345776 mapwrite 0x02444b thru 0x024f8e (0xb44 bytes) 000800 1770447298.355054 trunc from 0x03bfd3 to 0x03534c (0x6c86 bytes) 000801 1770447298.362042 write 0x003e7a thru 0x00cd4c (0x8ed3 bytes) 000802 1770447298.364350 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3534d): Mirror 4: 0x9d721650 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3534d): Mirror 4: 0x9d721650 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3534d: skipped 000803 1770447298.463612 write 0x02c358 thru 0x03bfea (0xfc93 bytes) 000804 1770447298.468370 MIRROR_RESYNC 2 mirrors 000805 1770447298.675304 mapread 0x0012fc thru 0x00688c (0x5591 bytes) 000806 1770447298.680585 mapread 0x03bd2b thru 0x03bfea (0x2c0 bytes) 000807 1770447298.683081 read 0x0120aa thru 0x0147c9 (0x2720 bytes) 000808 1770447298.684528 trunc from 0x03bfeb to 0x02fc04 (0xc3e6 bytes) 000809 1770447298.695368 fallocate 0x0256cc thru 0x03273f (0xd074 bytes) 000812 1770447298.698238 mapwrite 0x031963 thru 0x03ed62 (0xd400 bytes) 000813 1770447298.714631 write 0x009fc6 thru 0x00bac4 (0x1aff bytes) 000814 1770447298.718922 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ed63): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed63): Mirror 4: 0x3e19166f (stale) CRC-32 checksum value for chunk [0, 0x3ed63): Mirror 1: 0x915842d6 Mirror 5: 0x915842d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed63: skipped 000815 1770447298.975201 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ed63): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed63): Mirror 4: 0x3e19166f (stale) CRC-32 checksum value for chunk [0, 0x3ed63): Mirror 1: 0x915842d6 Mirror 5: 0x915842d6 Mirror 6: 0x915842d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed63: skipped 000816 1770447299.243126 mapread 0x00173c thru 0x00e9c1 (0xd286 bytes) 000817 1770447299.248218 fallocate 0x03f818 thru 0x03ff8c (0x775 bytes) 000818 1770447299.258781 write 0x034073 thru 0x0371c6 (0x3154 bytes) 000819 1770447299.262607 fallocate 0x017390 thru 0x01d5b5 (0x6226 bytes) 000820 1770447299.264178 punch from 0x02d7ef to 0x034f8b (0x779d bytes) 000822 1770447299.268121 punch from 0x01734c to 0x01fa62 (0x8717 bytes) 000823 1770447299.271022 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed63): Mirror 5: 0x915842d6 (stale) Mirror 6: 0x915842d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed63): Mirror 5: 0x915842d6 (stale) Mirror 6: 0x915842d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed63: skipped 000824 1770447299.362029 write 0x028075 thru 0x031066 (0x8ff2 bytes) 000825 1770447299.366580 punch from 0x00b8f3 to 0x00f970 (0x407e bytes) 000826 1770447299.368554 write 0x00ce35 thru 0x0193f3 (0xc5bf bytes) 000827 1770447299.373578 mapread 0x00adba thru 0x013326 (0x856d bytes) 000828 1770447299.375608 MIRROR_RESYNC 3 mirrors 000829 1770447299.615944 read 0x014b4f thru 0x01de74 (0x9326 bytes) 000830 1770447299.618736 write 0x0154c3 thru 0x015dfc (0x93a bytes) 000831 1770447299.628358 trunc from 0x03ed63 to 0x006170 (0x38bf2 bytes) 000832 1770447299.639588 trunc from 0x005c1d to 0x006170 (0x554 bytes) 000833 1770447299.641180 mapwrite 0x0104c6 thru 0x01066a (0x1a5 bytes) 000834 1770447299.651945 MIRROR_RESYNC 3 mirrors 000836 1770447299.826366 trunc from 0x01066b to 0x00bc50 (0x4a1a bytes) 000837 1770447299.843762 fallocate 0x00ed22 thru 0x014682 (0x5961 bytes) 000838 1770447299.845763 mapwrite 0x01b623 thru 0x02a210 (0xebee bytes) 000839 1770447299.868032 punch from 0x01ba4f to 0x027375 (0xb927 bytes) 000840 1770447299.870390 write 0x02bfb4 thru 0x03b67a (0xf6c7 bytes) 000841 1770447299.871832 fallocate 0x016dc2 thru 0x01e7fa (0x7a39 bytes) 000842 1770447299.874384 trunc from 0x017642 to 0x02712c (0xfaeb bytes) 000844 1770447299.877195 write 0x019062 thru 0x01962a (0x5c9 bytes) 000845 1770447299.880953 mapread 0x006057 thru 0x011140 (0xb0ea bytes) 000846 1770447299.886155 punch from 0x02a9cc to 0x02c651 (0x1c86 bytes) 000847 1770447299.892096 mapread 0x007f0d thru 0x0137c7 (0xb8bb bytes) 000848 1770447299.893001 fallocate 0x03357c thru 0x03a0d4 (0x6b59 bytes) 000850 1770447299.895412 read 0x03b2ec thru 0x03b67a (0x38f bytes) 000851 1770447299.895981 fallocate 0x03d3af thru 0x03ffff (0x2c51 bytes) 000852 1770447299.898180 mapread 0x01ea68 thru 0x01fe00 (0x1399 bytes) 000853 1770447299.898924 fallocate 0x00908c thru 0x00b998 (0x290d bytes) 000854 1770447299.901279 trunc from 0x037e4f to 0x03993d (0x1aef bytes) 000855 1770447299.903437 mapwrite 0x00cf31 thru 0x01a121 (0xd1f1 bytes) 000856 1770447299.911270 trunc from 0x02a096 to 0x039f63 (0xfece bytes) 000858 1770447299.913771 write 0x02eb49 thru 0x02f45d (0x915 bytes) 000861 1770447299.918186 write 0x00f40d thru 0x01e267 (0xee5b bytes) 000862 1770447299.919629 mapwrite 0x03c9ea thru 0x03ffff (0x3616 bytes) 000863 1770447299.932755 mapwrite 0x007590 thru 0x0132d4 (0xbd45 bytes) 000864 1770447299.940076 read 0x03fa04 thru 0x03ffff (0x5fc bytes) 000865 1770447299.940608 read 0x02277f thru 0x027c6c (0x54ee bytes) 000866 1770447299.943621 fallocate 0x00ac98 thru 0x00b746 (0xaaf bytes) 000867 1770447299.945510 write 0x01c263 thru 0x02b2e9 (0xf087 bytes) 000868 1770447299.946543 fallocate 0x035405 thru 0x03ffff (0xabfb bytes) 000869 1770447299.949833 MIRROR_RESYNC 3 mirrors 000870 1770447300.179158 punch from 0x00bc72 to 0x01b24e (0xf5dd bytes) 000871 1770447300.189159 trunc from 0x040000 to 0x03911d (0x6ee2 bytes) 000872 1770447300.195182 trunc from 0x02ef79 to 0x03911d (0xa1a5 bytes) 000873 1770447300.197027 trunc from 0x03911e to 0x00492e (0x347ef bytes) 000874 1770447300.202450 read 0x003f27 thru 0x00492e (0xa08 bytes) 000875 1770447300.204963 trunc from 0x00492f to 0x009192 (0x4864 bytes) 000876 1770447300.209962 trunc from 0x00218c to 0x00373d (0x15b2 bytes) 000877 1770447300.213802 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x9193): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9193): Mirror 5: 0x78d4d89 (stale) Mirror 6: 0x78d4d89 (stale) CRC-32 checksum value for chunk [0, 0x9193): Mirror 1: 0x60dbd842 Mirror 7: 0x60dbd842 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9193: skipped 000879 1770447300.462959 write 0x01b252 thru 0x026170 (0xaf1f bytes) 000880 1770447300.474891 trunc from 0x026171 to 0x015043 (0x1112d bytes) 000882 1770447300.485165 write 0x024ae1 thru 0x030146 (0xb666 bytes) 000883 1770447300.486261 trunc from 0x030147 to 0x0038e2 (0x2c864 bytes) 000884 1770447300.490893 fallocate 0x01c501 thru 0x01e94c (0x244c bytes) 000885 1770447300.492902 trunc from 0x00237d to 0x0038e2 (0x1566 bytes) 000886 1770447300.494479 write 0x037224 thru 0x03814d (0xf2a bytes) 000887 1770447300.495391 trunc from 0x032b8f to 0x035f11 (0x3383 bytes) 000888 1770447300.497435 mapread 0x01f3de thru 0x0259bf (0x65e2 bytes) 000889 1770447300.504111 mapwrite 0x0020c5 thru 0x011219 (0xf155 bytes) 000890 1770447300.520810 trunc from 0x03814e to 0x026b00 (0x1164d bytes) 000891 1770447300.526237 read 0x005be6 thru 0x00b834 (0x5c4f bytes) 000893 1770447300.528435 MIRROR_RESYNC 4 mirrors 000894 1770447300.761228 trunc from 0x016450 to 0x0195f0 (0x31a1 bytes) 000895 1770447300.774982 read 0x01dc85 thru 0x026b00 (0x8e7c bytes) 000896 1770447300.779047 punch from 0x01c38c to 0x026b00 (0xa775 bytes) 000897 1770447300.781281 mapread 0x022a33 thru 0x026b00 (0x40ce bytes) 000898 1770447300.787826 write 0x03d13d thru 0x03ffff (0x2ec3 bytes) 000899 1770447300.788769 trunc from 0x040000 to 0x00a4f4 (0x35b0b bytes) 000900 1770447300.800865 trunc from 0x00a4f5 to 0x036981 (0x2c48d bytes) 000901 1770447300.805959 fallocate 0x0167f2 thru 0x02558a (0xed99 bytes) 000902 1770447300.808205 read 0x005a0d thru 0x00f729 (0x9d1d bytes) 000903 1770447300.812047 read 0x016a89 thru 0x019eb1 (0x3429 bytes) 000904 1770447300.812544 fallocate 0x001ba7 thru 0x003ab0 (0x1f0a bytes) 000905 1770447300.814496 punch from 0x022157 to 0x02406f (0x1f19 bytes) 000906 1770447300.816263 mapread 0x02d3dc thru 0x033666 (0x628b bytes) 000907 1770447300.816899 trunc from 0x036982 to 0x025839 (0x11148 bytes) 000908 1770447300.822261 MIRROR_RESYNC 4 mirrors 000909 1770447301.001177 mapread 0x019d5a thru 0x024283 (0xa52a bytes) 000910 1770447301.007757 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) 000911 1770447301.105149 fallocate 0x035fb5 thru 0x037893 (0x18df bytes) 000912 1770447301.117709 trunc from 0x00b26d to 0x014be1 (0x9975 bytes) 000913 1770447301.119955 read 0x00fd38 thru 0x01f5af (0xf878 bytes) 000914 1770447301.123744 trunc from 0x037894 to 0x0104ac (0x273e7 bytes) 000915 1770447301.138034 punch from 0x00a055 to 0x0104ac (0x6458 bytes) 000916 1770447301.140120 mapwrite 0x03f77d thru 0x03ffff (0x883 bytes) 000917 1770447301.151549 mapwrite 0x0088fb thru 0x01881f (0xff25 bytes) 000918 1770447301.165707 write 0x01a85a thru 0x023d7e (0x9525 bytes) 000919 1770447301.168250 trunc from 0x040000 to 0x024b33 (0x1b4cc bytes) 000920 1770447301.173526 mapread 0x00b955 thru 0x00c574 (0xc20 bytes) 000921 1770447301.174144 trunc from 0x024b34 to 0x00c7ca (0x18369 bytes) 000922 1770447301.179772 read 0x00b9b4 thru 0x00c7ca (0xe17 bytes) 000923 1770447301.180300 trunc from 0x005161 to 0x00b439 (0x62d9 bytes) 000924 1770447301.182327 mapwrite 0x01fbf3 thru 0x0260c3 (0x64d1 bytes) 000925 1770447301.196888 MIRROR_RESYNC 3 mirrors 000926 1770447301.394978 mapread 0x00724b thru 0x00cd2e (0x5ae4 bytes) 000927 1770447301.399194 punch from 0x0011d2 to 0x00fa75 (0xe8a4 bytes) 000928 1770447301.409092 write 0x03bec9 thru 0x03ffff (0x4137 bytes) 000929 1770447301.409889 write 0x02c437 thru 0x0331f4 (0x6dbe bytes) 000930 1770447301.412763 punch from 0x0174eb to 0x021fec (0xab02 bytes) 000932 1770447301.414720 fallocate 0x025173 thru 0x03310a (0xdf98 bytes) 000934 1770447301.416716 mapwrite 0x027595 thru 0x027bb0 (0x61c bytes) 000935 1770447301.421867 trunc from 0x0228a2 to 0x02365a (0xdb9 bytes) 000936 1770447301.423411 mapwrite 0x0197f6 thru 0x028713 (0xef1e bytes) 000937 1770447301.434381 mapwrite 0x02fc70 thru 0x03c591 (0xc922 bytes) 000938 1770447301.442514 fallocate 0x02fa07 thru 0x03e486 (0xea80 bytes) 000939 1770447301.444195 trunc from 0x040000 to 0x038f5e (0x70a1 bytes) 000940 1770447301.453071 fallocate 0x01120d thru 0x01dd5c (0xcb50 bytes) 000941 1770447301.454475 mapread 0x01bbd2 thru 0x01f5f7 (0x3a26 bytes) 000942 1770447301.455012 mapread 0x00621e thru 0x00784f (0x1632 bytes) 000943 1770447301.457499 mapwrite 0x02a018 thru 0x02a7d9 (0x7c2 bytes) 000946 1770447301.462185 punch from 0x0235e3 to 0x02cccc (0x96ea bytes) 000947 1770447301.463937 punch from 0x000453 to 0x00988c (0x943a bytes) 000949 1770447301.466030 punch from 0x0006a9 to 0x00bdca (0xb722 bytes) 000950 1770447301.469059 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38f5f): Mirror 6: 0x660d40d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f5f): Mirror 6: 0x660d40d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f5f: skipped 000951 1770447301.542791 punch from 0x01a18a to 0x027062 (0xced9 bytes) 000952 1770447301.544907 read 0x0311e9 thru 0x038f5e (0x7d76 bytes) 000953 1770447301.547569 read 0x01fcf4 thru 0x02ed75 (0xf082 bytes) 000954 1770447301.551499 read 0x00b498 thru 0x00f840 (0x43a9 bytes) 000955 1770447301.552070 trunc from 0x00a737 to 0x017f60 (0xd82a bytes) 000956 1770447301.553980 read 0x02f208 thru 0x030933 (0x172c bytes) 000958 1770447301.554905 trunc from 0x020ab1 to 0x026517 (0x5a67 bytes) 000959 1770447301.557192 trunc from 0x00b02c to 0x015bf1 (0xabc6 bytes) 000961 1770447301.559365 fallocate 0x0227e8 thru 0x025e0a (0x3623 bytes) 000962 1770447301.561590 mapwrite 0x01cb12 thru 0x02ad68 (0xe257 bytes) 000964 1770447301.574337 read 0x013fd9 thru 0x0148df (0x907 bytes) 000965 1770447301.574945 trunc from 0x038f5f to 0x00f810 (0x2974e bytes) 000966 1770447301.582053 write 0x012c9e thru 0x01e26e (0xb5d1 bytes) 000967 1770447301.583475 read 0x01e140 thru 0x01e26e (0x12f bytes) 000969 1770447301.584431 trunc from 0x00c76d to 0x00fc77 (0x350b bytes) 000970 1770447301.587376 punch from 0x01a17c to 0x01e26e (0x40f3 bytes) 000971 1770447301.593322 punch from 0x00db44 to 0x01ba5a (0xdf17 bytes) 000972 1770447301.595024 fallocate 0x003530 thru 0x012b25 (0xf5f6 bytes) 000973 1770447301.596701 mapwrite 0x033238 thru 0x03d5d6 (0xa39f bytes) 000974 1770447301.610316 mapread 0x029f0f thru 0x034a4c (0xab3e bytes) 000975 1770447301.611092 mapread 0x034f54 thru 0x0379d5 (0x2a82 bytes) 000976 1770447301.611759 trunc from 0x00644f to 0x00ad03 (0x48b5 bytes) 000977 1770447301.613350 punch from 0x0012db to 0x0092f2 (0x8018 bytes) 000978 1770447301.615075 punch from 0x02ac36 to 0x03a64f (0xfa1a bytes) 000979 1770447301.616688 trunc from 0x0215c3 to 0x0216f6 (0x134 bytes) 000980 1770447301.619370 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x5c2:0x0]: Invalid argument (22) 000981 1770447301.651209 trunc from 0x03d5d7 to 0x006667 (0x36f6f bytes) 000982 1770447301.655528 trunc from 0x006668 to 0x031c4b (0x2b5e4 bytes) 000983 1770447301.659183 trunc from 0x031c4c to 0x01cb1e (0x1512d bytes) 000984 1770447301.662774 fallocate 0x022ab1 thru 0x031f98 (0xf4e8 bytes) 000985 1770447301.664486 punch from 0x0086fd to 0x00a0d8 (0x19dc bytes) 000987 1770447301.666013 mapread 0x000d1d thru 0x01036c (0xf650 bytes) 000988 1770447301.671889 read 0x019667 thru 0x01cb1e (0x34b8 bytes) 000991 1770447301.674186 trunc from 0x01cb1f to 0x039382 (0x1c864 bytes) 000992 1770447301.677966 mapwrite 0x029f48 thru 0x02fa3e (0x5af7 bytes) 000994 1770447301.686238 read 0x01a3a8 thru 0x022b57 (0x87b0 bytes) 000995 1770447301.688799 mapwrite 0x00ed75 thru 0x01a09c (0xb328 bytes) 000996 1770447301.694559 fallocate 0x013d10 thru 0x01ab73 (0x6e64 bytes) 000997 1770447301.695948 read 0x00d344 thru 0x00ee64 (0x1b21 bytes) 000998 1770447301.696235 trunc from 0x039383 to 0x0188b2 (0x20ad0 bytes) 001000 1770447301.700932 read 0x009679 thru 0x00efa4 (0x592c bytes) All operations completed A-OK! PASS 70a (17s) == sanity-flr test 71: check mirror extend parent fid ==== 01:55:03 (1770447303) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000405:0x5ed:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 72: check append on FLR file ========== 01:55:06 (1770447306) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.05807 s, 36.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329287 s, 31.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 01:55:09 (1770447309) Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre2 Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2713440 .. Extending file size to 3415680 .. failed Extending file size to 5941632 .. Extending file size to 6691328 .. Extending file size to 7458368 .. Extending file size to 6896928 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8106848 .. failed Extending file size to 8793728 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8029376 .. done Extending file size to 9106176 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8400896 .. Extending file size to 8698880 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8958240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9361312 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9114016 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 62152 62153 62155 62157 62159 192.168.201.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg116-client.virtnet /mnt/lustre2 (opts:) 192.168.201.116@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg116-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:56:14 (1770447374) Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre2 Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6716576 .. Extending file size to 1299296 .. Extending file size to 2253472 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 2548416 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 3797728 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7079648 .. done Extending file size to 5124576 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5610144 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8899616 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8220000 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8887424 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8977120 .. Waiting 65748 65749 65751 65753 65754 done 192.168.201.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg116-client.virtnet /mnt/lustre2 (opts:) 192.168.201.116@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg116-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:57:21 (1770447441) Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2568 1268736 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2676 1268628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18600 3588420 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24276 3582744 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42876 7171164 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2568 1268736 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2676 1268628 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18600 3588420 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24276 3582744 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42876 7171164 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.046468 s, 45.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0893486 s, 23.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xca30778f (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg116-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:57:29 (1770447449) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0490032 s, 42.8 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 01:57:32 (1770447452) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa2:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa3:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:57:36 (1770447456) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0629291 s, 33.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7df:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:57:39 (1770447459) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0551634 s, 38.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:57:43 (1770447463) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0403588 s, 52.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } PASS 204c (26s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:58:09 (1770447489) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg116-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261263 s, 40.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7e0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7e0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7e1:0x0] } PASS 204d (38s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:58:47 (1770447527) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg116-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0428761 s, 48.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (34s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:59:21 (1770447561) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg116-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg116-server: Pool lustre.test_204f created Adding targets to pool oleg116-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg116-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0323219 s, 64.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg116-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg116-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg116-server: Pool lustre.test_204f destroyed PASS 204f (50s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:00:11 (1770447611) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:00:14 (1770447614) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:00:16 (1770447616) oleg116-server: Pool lustre.test_206 created Adding targets to pool oleg116-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg116-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg116-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg116-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg116-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:00:32 (1770447632) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028035 s, 37.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151494 s, 69.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe797d04 Mirror 3: 0xe797d04 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:00:36 (1770447636) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:00:37 (1770447637) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:00:39 (1770447639) Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0178089 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1fb0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x11d0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x11d0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x11d0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x21b0000 Read this (4 bytes): BBBB 192.168.201.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg116-client.virtnet /mnt/lustre2 (opts:) PASS 209a (18s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:00:57 (1770447657) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00482144 s, 1.7 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:01:00 (1770447660) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226824 s, 46.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x59 lmm_fid: [0x200000405:0x59:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x59 lmm_fid: [0x200000405:0x59:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611225 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-02-07 02:01:01.000000000 -0500 Modify: 2026-02-07 02:01:01.000000000 -0500 Change: 2026-02-07 02:01:01.000000000 -0500 Birth: 2026-02-07 02:01:01.000000000 -0500 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:01:04 (1770447664) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193194 s, 54.3 MB/s oleg116-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg116-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg116-server: O/280000401/d18/b2: File not found by ext2_lookup PASS 210b (16s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:01:19 (1770447679) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0412768 s, 992 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00617447 s, 663 kB/s size after second write -rw-r--r-- 1 root root 4096 Feb 7 02:01 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Feb 7 02:01 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:01:22 (1770447682) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:01:25 (1770447685) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 864 sec ============ 02:01:29 (1770447689) === sanity-flr: start cleanup 02:01:30 (1770447690) === === sanity-flr: finish cleanup 02:01:33 (1770447693) ===