-----============= acceptance-small: sanity-flr ============----- Fri Sep 4 08:58:15 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg259-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 08:58:35 (1788526715) === oleg259-client.virtnet: executing check_config_client /mnt/lustre oleg259-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg259-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff95be83191800.idle_timeout=debug osc.lustre-OST0001-osc-ffff95be83191800.idle_timeout=debug disable quota as required oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 08:59:22 (1788526762) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8213] == sanity-flr test 0a: lfs mirror create with -N option == 08:59:24 (1788526764) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:59:40 (1788526780) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:59:43 (1788526783) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:59:46 (1788526786) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:00:10 (1788526810) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:00:14 (1788526814) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 09:00:18 (1788526818) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff95be83191800.stats=clear osc.lustre-OST0001-osc-ffff95be83191800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:00:27 (1788526827) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:00:39 (1788526839) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:00:48 (1788526848) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:00:51 (1788526851) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:00:59 (1788526859) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 09:01:08 (1788526868) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:01:16 (1788526876) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:01:23 (1788526883) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:01:31 (1788526891) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:01:42 (1788526902) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:01:50 (1788526910) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 1.06891 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916418 s, 11.4 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:01:58 (1788526918) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.269141 s, 39.0 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:02:06 (1788526926) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:02:16 (1788526936) Stopping /mnt/lustre-ost1 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95be83191800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95be83191800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg259-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95be83191800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95be83191800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg259-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (73s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:03:28 (1788527008) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.187774 s, 16.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95be83191800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95be83191800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg259-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:04:09 (1788527049) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95be83191800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95be83191800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg259-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95be83191800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95be83191800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg259-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (78s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:05:27 (1788527127) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.38178 s, 22.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 4 09:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.983s user 0m0.001s sys 0m0.151s Stopping /mnt/lustre-ost1 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95be83191800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95be83191800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.943s user 0m0.001s sys 0m0.180s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg259-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95be83191800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95be83191800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.721s user 0m0.001s sys 0m0.156s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg259-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95be83191800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95be83191800.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:06:58 (1788527218) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:07:00 (1788527220) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:07:03 (1788527223) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 09:07:06 (1788527226) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 09:07:14 (1788527234) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.262718 s, 47.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:07:26 (1788527246) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262714 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228281 s, 45.9 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:07:32 (1788527252) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:07:39 (1788527259) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102803 s, 996 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00115129 s, 889 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00175274 s, 584 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00114398 s, 895 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 09:07:48 (1788527268) create test files with size 15730896 .. df6d7b222e46272bd19a0c750d0ead63 - 81cefef6cdf17193e0af4818c8a592d2 - d4900382d12c39145aaff4a893c31561 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg259-client.virtnet /mnt/lustre (opts:) Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 09:08:10 (1788527290) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:08:22 (1788527302) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:08:28 (1788527308) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456957 s, 22.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224188 s, 46.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 09:08:37 (1788527317) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109723 s, 19.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.251728 s, 16.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129876 s, 16.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 09:08:51 (1788527331) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 09:08:53 (1788527333) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:08:56 (1788527336) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191005 s, 54.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 09:09:03 (1788527343) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.338885 s, 9.3 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:09:11 (1788527351) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:09:20 (1788527360) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.245641 s, 42.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg259-client.virtnet /mnt/lustre (opts:) Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:09:33 (1788527373) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff95be88602000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.412676 s, 25.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg259-client.virtnet /mnt/lustre (opts:) Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff95be82a6c000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:09:48 (1788527388) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:10:28 (1788527428) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:11:06 (1788527466) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 09:11:09 (1788527469) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.238873 s, 13.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.158647 s, 19.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xb354b3ce (nosync,stale) Mirror 2: 0xb354b3ce (nosync,stale) ** verify mirror 2 doesn't change original checksum: b1540acdaaa3a9152e4470e92f8e57c2 - mirror 1 checksum: bd465bb1fdaf3ad7ad3ad0eeb13207f6 - mirror 2 checksum: b1540acdaaa3a9152e4470e92f8e57c2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T09:11:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T09:11:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b1540acdaaa3a9152e4470e92f8e57c2 - mirror 1 checksum: bd465bb1fdaf3ad7ad3ad0eeb13207f6 - mirror 2 checksum: bd465bb1fdaf3ad7ad3ad0eeb13207f6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T09:11:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T09:11:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:11:20 (1788527480) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439194 s, 23.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:11:27 (1788527487) Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 09:11:40 (1788527500) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 09:12:05 (1788527525) targets are mounted 09:12:05 (1788527525) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283545 s, 37.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 50A (56s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:12:23 (1788527543) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0503306 s, 407 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0411828 s, 497 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:12:33 (1788527553) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0308488 s, 664 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00585526 s, 3.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:12:42 (1788527562) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00555095 s, 3.0 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:12:50 (1788527570) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0290317 s, 705 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:12:58 (1788527578) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00616316 s, 665 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0370691 s, 110 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:13:06 (1788527586) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:13:26 (1788527606) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:13:53 (1788527633) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:14:16 (1788527656) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:14:40 (1788527680) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 09:14:48 (1788527688) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57569 mirror split pid 57570 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57569 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57570 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 09:15:56 (1788527756) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 7758 skipping zero length punch hole skipping zero size read 000003 1788527758.935807 mapwrite 0x001e42 thru 0x0114ef (0xf6ae bytes) 000005 1788527758.994541 punch from 0x00a8c0 to 0x0114ef (0x6c30 bytes) 000006 1788527758.998904 trunc from 0x00d858 to 0x01048f (0x2c38 bytes) 000007 1788527759.003725 mapread 0x00388c thru 0x0114ef (0xdc64 bytes) fallocating to largest ever: 0x40000 000008 1788527759.012974 fallocate 0x033e79 thru 0x03ffff (0xc187 bytes) 000009 1788527759.018691 punch from 0x00a937 to 0x018db1 (0xe47b bytes) 000010 1788527759.023883 write 0x039e14 thru 0x03ffff (0x61ec bytes) 000011 1788527759.028944 mapread 0x0046e8 thru 0x0127c8 (0xe0e1 bytes) 000012 1788527759.038432 write 0x023ff1 thru 0x02f31d (0xb32d bytes) 000013 1788527759.047864 mapread 0x011c23 thru 0x014388 (0x2766 bytes) 000014 1788527759.056612 mapwrite 0x03109f thru 0x03ae80 (0x9de2 bytes) 000015 1788527759.083577 read 0x02e525 thru 0x039e10 (0xb8ec bytes) 000016 1788527759.084808 mapwrite 0x00cc8d thru 0x018c5c (0xbfd0 bytes) 000017 1788527759.105624 trunc from 0x040000 to 0x028c20 (0x173df bytes) 000018 1788527759.119966 trunc from 0x028c21 to 0x0005ec (0x28634 bytes) 000019 1788527759.145981 trunc from 0x00048b to 0x0005ec (0x162 bytes) 000020 1788527759.150529 trunc from 0x000344 to 0x0005ec (0x2a9 bytes) 000021 1788527759.154647 mapwrite 0x0286b6 thru 0x02bf64 (0x38af bytes) 000022 1788527759.189862 trunc from 0x001830 to 0x00b1a2 (0x9973 bytes) 000023 1788527759.194723 trunc from 0x02bf65 to 0x0112e5 (0x1ac7f bytes) 000024 1788527759.209679 trunc from 0x00948e to 0x0112e5 (0x7e58 bytes) 000025 1788527759.213944 fallocate 0x01452c thru 0x020f33 (0xca08 bytes) 000026 1788527759.217952 mapread 0x008fdc thru 0x0112e5 (0x830a bytes) 000028 1788527759.231860 read 0x00054d thru 0x0031b8 (0x2c6c bytes) 000029 1788527759.237492 punch from 0x00de4a to 0x00ee43 (0xffa bytes) 000030 1788527759.240997 punch from 0x00e93f to 0x0112e5 (0x29a7 bytes) 000031 1788527759.245745 mapwrite 0x0287dc thru 0x030b5b (0x8380 bytes) 000032 1788527759.292807 punch from 0x01c573 to 0x0233f1 (0x6e7f bytes) 000033 1788527759.297096 read 0x016401 thru 0x01b1e2 (0x4de2 bytes) 000034 1788527759.297858 punch from 0x003c8e to 0x0104a2 (0xc815 bytes) 000035 1788527759.303790 trunc from 0x02545a to 0x030b5b (0xb702 bytes) 000036 1788527759.308224 write 0x003c04 thru 0x013b0f (0xff0c bytes) 000037 1788527759.330850 fallocate 0x00d553 thru 0x01d256 (0xfd04 bytes) 000038 1788527759.339830 MIRROR_EXTEND to 2 mirrors 000039 1788527759.782887 mapwrite 0x01a424 thru 0x01c76d (0x234a bytes) 000040 1788527759.858160 read 0x02a7ef thru 0x030b5b (0x636d bytes) 000041 1788527759.870388 write 0x03aefb thru 0x03ffff (0x5105 bytes) 000042 1788527759.875634 write 0x0041ed thru 0x0077c3 (0x35d7 bytes) 000043 1788527759.886999 write 0x01db35 thru 0x02645d (0x8929 bytes) 000044 1788527759.894372 punch from 0x03c2a3 to 0x03ffff (0x3d5d bytes) 000045 1788527759.909109 trunc from 0x03856b to 0x03cc49 (0x46df bytes) 000046 1788527759.914006 trunc from 0x040000 to 0x025b43 (0x1a4bc bytes) 000048 1788527759.965485 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x25b44): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25b44): Mirror 2: 0xd276ae88 (stale) CRC-32 checksum value for chunk [0, 0x25b44): Mirror 1: 0x2914e602 Mirror 3: 0x2914e602 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25b44: skipped 000049 1788527761.037300 mapwrite 0x0138d2 thru 0x01f7a4 (0xbed3 bytes) 000051 1788527761.193586 mapwrite 0x010f0f thru 0x019083 (0x8175 bytes) 000052 1788527761.217422 read 0x017ec6 thru 0x025b43 (0xdc7e bytes) 000053 1788527761.217988 write 0x01a4d1 thru 0x0239c0 (0x94f0 bytes) 000054 1788527761.229825 punch from 0x020d2c to 0x025383 (0x4658 bytes) 000055 1788527761.247470 trunc from 0x02385a to 0x02511c (0x18c3 bytes) 000056 1788527761.260816 read 0x007ff7 thru 0x00be22 (0x3e2c bytes) 000057 1788527761.300082 punch from 0x009663 to 0x015e6d (0xc80b bytes) 000058 1788527761.308070 trunc from 0x01f701 to 0x0223a9 (0x2ca9 bytes) 000059 1788527761.313672 trunc from 0x025b44 to 0x0058f0 (0x20253 bytes) 000060 1788527761.339008 read 0x001cb8 thru 0x00222d (0x576 bytes) 000061 1788527761.340022 punch from 0x005857 to 0x0058f0 (0x9a bytes) 000062 1788527761.345780 mapwrite 0x03cb98 thru 0x03ffff (0x3468 bytes) 000063 1788527761.472857 trunc from 0x002ae3 to 0x0109d4 (0xdef2 bytes) 000064 1788527761.489958 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4c053839 (stale) Mirror 3: 0x24c94492 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1f025718 Mirror 4: 0x1f025718 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000065 1788527762.414945 MIRROR_RESYNC 4 mirrors 000066 1788527762.911804 punch from 0x030d89 to 0x03e58d (0xd805 bytes) 000067 1788527762.945002 mapwrite 0x01ac71 thru 0x023c77 (0x9007 bytes) 000069 1788527762.974983 trunc from 0x03fdec to 0x03ffff (0x214 bytes) 000070 1788527762.979542 mapread 0x03ffee thru 0x03ffff (0x12 bytes) 000071 1788527762.987754 trunc from 0x012fcc to 0x0166ed (0x3722 bytes) 000072 1788527762.992526 write 0x037feb thru 0x03ffff (0x8015 bytes) 000073 1788527763.001163 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1f025718 (stale) Mirror 3: 0x1f025718 (stale) Mirror 4: 0x1f025718 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x771914f4 Mirror 5: 0x771914f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000074 1788527763.824994 mapwrite 0x003f2d thru 0x00a22a (0x62fe bytes) 000075 1788527763.942507 trunc from 0x03723b to 0x037fae (0xd74 bytes) 000076 1788527763.950409 write 0x00503e thru 0x00d029 (0x7fec bytes) 000077 1788527763.961964 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1f025718 (stale) Mirror 4: 0x1f025718 (stale) Mirror 5: 0x771914f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1f025718 (stale) Mirror 4: 0x1f025718 (stale) Mirror 5: 0x771914f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000078 1788527764.354875 punch from 0x02edda to 0x03cf1f (0xe146 bytes) 000079 1788527764.359862 write 0x00335b thru 0x00beca (0x8b70 bytes) 000080 1788527764.373596 read 0x016b36 thru 0x0211e8 (0xa6b3 bytes) 000081 1788527764.383042 write 0x007320 thru 0x00acd9 (0x39ba bytes) 000082 1788527764.385854 mapwrite 0x0011b2 thru 0x007d44 (0x6b93 bytes) 000083 1788527764.411844 mapread 0x01c0fc thru 0x021bf9 (0x5afe bytes) 000084 1788527764.413249 mapread 0x02c9c6 thru 0x038b5c (0xc197 bytes) 000085 1788527764.439879 trunc from 0x01867f to 0x01b50c (0x2e8e bytes) 000086 1788527764.445335 read 0x02beb5 thru 0x031dba (0x5f06 bytes) 000087 1788527764.451793 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1f025718 (stale) Mirror 4: 0x1f025718 (stale) Mirror 5: 0x771914f4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x394c8c34 Mirror 6: 0x394c8c34 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000089 1788527765.285419 trunc from 0x00216e to 0x002483 (0x316 bytes) 000090 1788527765.318497 trunc from 0x035db4 to 0x03a761 (0x49ae bytes) 000091 1788527765.328555 read 0x011d40 thru 0x01cec3 (0xb184 bytes) 000092 1788527765.335612 write 0x006db5 thru 0x012689 (0xb8d5 bytes) 000093 1788527765.346372 mapread 0x020091 thru 0x025f60 (0x5ed0 bytes) 000094 1788527765.359307 trunc from 0x005907 to 0x00be47 (0x6541 bytes) 000095 1788527765.376551 trunc from 0x040000 to 0x01a901 (0x256fe bytes) 000096 1788527765.419642 fallocate 0x01d547 thru 0x029568 (0xc022 bytes) 000097 1788527765.423574 trunc from 0x029569 to 0x01bf07 (0xd661 bytes) 000098 1788527765.444744 trunc from 0x01a52a to 0x01bf07 (0x19de bytes) 000099 1788527765.449007 read 0x001d3c thru 0x00cfdc (0xb2a1 bytes) 000101 1788527765.458053 trunc from 0x003de6 to 0x00eb79 (0xad94 bytes) 000102 1788527765.464036 trunc from 0x01bf08 to 0x00c7ef (0xf718 bytes) 000103 1788527765.485811 read 0x00b007 thru 0x00c7ef (0x17e9 bytes) 000104 1788527765.494192 write 0x00f14f thru 0x013022 (0x3ed4 bytes) 000105 1788527765.495226 trunc from 0x005a99 to 0x00d884 (0x7dec bytes) 000106 1788527765.499467 fallocate 0x03a59c thru 0x03ffff (0x5a64 bytes) 000107 1788527765.504476 read 0x02bef5 thru 0x03681e (0xa92a bytes) 000108 1788527765.513606 mapread 0x01abe6 thru 0x024548 (0x9963 bytes) 000109 1788527765.514382 trunc from 0x039bbb to 0x03c6ea (0x2b30 bytes) 000110 1788527765.521918 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1f025718 (stale) Mirror 5: 0x771914f4 (stale) Mirror 6: 0x394c8c34 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1f025718 (stale) Mirror 5: 0x771914f4 (stale) Mirror 6: 0x394c8c34 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000111 1788527765.799390 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1f025718 (stale) Mirror 5: 0x771914f4 (stale) Mirror 6: 0x394c8c34 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd653c0e5 Mirror 7: 0xd653c0e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000112 1788527766.688563 punch from 0x0191ed to 0x028662 (0xf476 bytes) 000113 1788527766.751016 trunc from 0x040000 to 0x02e989 (0x11676 bytes) 000115 1788527766.802621 punch from 0x016670 to 0x01ec7e (0x860f bytes) 000116 1788527766.826251 mapwrite 0x026b58 thru 0x031c9b (0xb144 bytes) 000117 1788527766.907822 mapread 0x007f83 thru 0x017a6e (0xfaec bytes) 000118 1788527766.979413 mapwrite 0x02ecd9 thru 0x031e2c (0x3154 bytes) 000119 1788527767.013534 read 0x00d7fc thru 0x00e047 (0x84c bytes) 000120 1788527767.014818 trunc from 0x010642 to 0x013826 (0x31e5 bytes) 000121 1788527767.021391 read 0x00e92f thru 0x014caa (0x637c bytes) 000123 1788527767.031683 punch from 0x0148c9 to 0x0223fc (0xdb34 bytes) 000124 1788527767.035720 punch from 0x02258a to 0x02e5b7 (0xc02e bytes) 000125 1788527767.039420 trunc from 0x017ab6 to 0x01ebcc (0x7117 bytes) 000126 1788527767.042232 mapwrite 0x00a225 thru 0x01197c (0x7758 bytes) 000127 1788527767.062303 punch from 0x014489 to 0x01977b (0x52f3 bytes) 000128 1788527767.074110 trunc from 0x031e2d to 0x01a3c5 (0x17a67 bytes) 000129 1788527767.094280 punch from 0x0029e3 to 0x0099bb (0x6fd9 bytes) 000130 1788527767.099774 trunc from 0x01a3c6 to 0x008316 (0x120af bytes) 000131 1788527767.122828 punch from 0x003d24 to 0x004d84 (0x1061 bytes) 000132 1788527767.129764 read 0x000b61 thru 0x008316 (0x77b6 bytes) 000133 1788527767.181008 MIRROR_RESYNC 5 mirrors 000134 1788527768.056843 mapread 0x006667 thru 0x008316 (0x1cb0 bytes) 000135 1788527768.077699 write 0x015143 thru 0x01c226 (0x70e4 bytes) 000136 1788527768.113328 mapwrite 0x01e6cc thru 0x0220a0 (0x39d5 bytes) 000137 1788527768.193403 trunc from 0x004c2c to 0x008d76 (0x414b bytes) 000138 1788527768.198114 mapread 0x00566b thru 0x00b432 (0x5dc8 bytes) 000139 1788527768.209183 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x220a1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x220a1): Mirror 4: 0x217fbecb (stale) Mirror 5: 0x217fbecb (stale) Mirror 6: 0x217fbecb (stale) Mirror 7: 0x217fbecb (stale) CRC-32 checksum value for chunk [0, 0x220a1): Mirror 1: 0x844e6799 Mirror 8: 0x844e6799 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x220a1: skipped 000141 1788527769.246576 write 0x00e97f thru 0x01838c (0x9a0e bytes) 000143 1788527769.321760 mapread 0x01950a thru 0x0220a0 (0x8b97 bytes) 000144 1788527769.340228 mapwrite 0x022646 thru 0x02f5ac (0xcf67 bytes) 000145 1788527769.418846 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f5ad): Mirror 5: 0x402d08e5 (stale) Mirror 6: 0x402d08e5 (stale) Mirror 7: 0x402d08e5 (stale) Mirror 8: 0x9536ef4f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f5ad): Mirror 5: 0x402d08e5 (stale) Mirror 6: 0x402d08e5 (stale) Mirror 7: 0x402d08e5 (stale) Mirror 8: 0x9536ef4f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f5ad: skipped 000146 1788527769.910848 trunc from 0x02f5ad to 0x015568 (0x1a044 bytes) 000147 1788527769.924090 mapread 0x00806c thru 0x015568 (0xd4fd bytes) 000148 1788527769.950434 trunc from 0x015569 to 0x01c6a2 (0x713a bytes) 000149 1788527769.965899 write 0x0119a5 thru 0x0131af (0x180b bytes) 000150 1788527769.966849 mapwrite 0x03ad2f thru 0x03ffff (0x52d1 bytes) 000151 1788527770.010637 read 0x0368f6 thru 0x03b20d (0x4918 bytes) 000152 1788527770.011984 punch from 0x0139d6 to 0x015cda (0x2305 bytes) 000153 1788527770.020304 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x18ccc9a4 (stale) Mirror 6: 0x18ccc9a4 (stale) Mirror 7: 0x18ccc9a4 (stale) Mirror 8: 0xcd0b89a2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x45c2b804 Mirror 9: 0x45c2b804 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000154 1788527770.693974 fallocate 0x019578 thru 0x01a1b1 (0xc3a bytes) 000155 1788527770.730977 trunc from 0x01f3d2 to 0x020606 (0x1235 bytes) 000156 1788527770.736702 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x18ccc9a4 (stale) Mirror 6: 0x18ccc9a4 (stale) Mirror 7: 0x18ccc9a4 (stale) Mirror 9: 0x45c2b804 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x18ccc9a4 (stale) Mirror 6: 0x18ccc9a4 (stale) Mirror 7: 0x18ccc9a4 (stale) Mirror 9: 0x45c2b804 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000157 1788527771.058862 write 0x03f24f thru 0x03ffff (0xdb1 bytes) 000158 1788527771.068320 fallocate 0x02cfa5 thru 0x03b8db (0xe937 bytes) 000159 1788527771.075433 trunc from 0x040000 to 0x0309f1 (0xf60e bytes) 000160 1788527771.090419 punch from 0x00979c to 0x014d6b (0xb5d0 bytes) 000161 1788527771.095605 mapread 0x02197b thru 0x0299c4 (0x804a bytes) 000162 1788527771.111779 punch from 0x01399b to 0x018c13 (0x5279 bytes) 000163 1788527771.115574 mapwrite 0x001cf6 thru 0x0083e6 (0x66f1 bytes) 000164 1788527771.139073 mapread 0x00bced thru 0x017504 (0xb818 bytes) 000165 1788527771.147412 write 0x03e958 thru 0x03ffff (0x16a8 bytes) 000166 1788527771.148957 punch from 0x01c454 to 0x023379 (0x6f26 bytes) 000167 1788527771.153022 trunc from 0x040000 to 0x03ea42 (0x15bd bytes) 000168 1788527771.174298 mapread 0x00fd80 thru 0x012928 (0x2ba9 bytes) 000169 1788527771.176479 trunc from 0x01a581 to 0x01d4fd (0x2f7d bytes) 000170 1788527771.184519 read 0x0353f3 thru 0x038d4c (0x395a bytes) 000171 1788527771.195838 MIRROR_RESYNC 5 mirrors 000172 1788527772.097616 trunc from 0x00a39e to 0x016883 (0xc4e6 bytes) 000174 1788527772.128058 mapwrite 0x00f5a8 thru 0x0148f2 (0x534b bytes) 000175 1788527772.162872 MIRROR_RESYNC 5 mirrors 000176 1788527773.153873 write 0x004be4 thru 0x006e4d (0x226a bytes) 000177 1788527773.197244 mapread 0x00bfb6 thru 0x016972 (0xa9bd bytes) 000178 1788527773.224704 read 0x01e2b6 thru 0x02abf7 (0xc942 bytes) 000180 1788527773.233760 fallocate 0x033e61 thru 0x03fc38 (0xbdd8 bytes) 000181 1788527773.240656 write 0x01f8a9 thru 0x02efe5 (0xf73d bytes) 000182 1788527773.246239 fallocate 0x016744 thru 0x016c92 (0x54f bytes) 000183 1788527773.250902 trunc from 0x03ea43 to 0x01dc1d (0x20e25 bytes) 000184 1788527773.299890 MIRROR_RESYNC 5 mirrors 000185 1788527774.826807 trunc from 0x01dc1e to 0x014efa (0x8d23 bytes) 000186 1788527774.909334 mapread 0x00d57f thru 0x014efa (0x797c bytes) 000187 1788527774.947110 read 0x00a329 thru 0x00d98b (0x3663 bytes) 000189 1788527774.948403 read 0x01148c thru 0x014efa (0x3a6f bytes) 000190 1788527774.964802 mapread 0x0134eb thru 0x014efa (0x1a10 bytes) 000191 1788527774.973295 punch from 0x00a98f to 0x013114 (0x8786 bytes) 000192 1788527774.993454 MIRROR_RESYNC 5 mirrors 000193 1788527776.279504 mapread 0x00f35f thru 0x014efa (0x5b9c bytes) 000194 1788527776.294626 mapwrite 0x03c152 thru 0x03ffff (0x3eae bytes) 000196 1788527776.358898 mapwrite 0x003611 thru 0x01198e (0xe37e bytes) 000197 1788527776.394783 trunc from 0x040000 to 0x00d5d2 (0x32a2d bytes) 000199 1788527776.415918 write 0x0308ef thru 0x03bc1f (0xb331 bytes) 000200 1788527776.417844 fallocate 0x010ae8 thru 0x0167e7 (0x5d00 bytes) 000202 1788527776.424391 trunc from 0x00a8a1 to 0x014542 (0x9ca2 bytes) 000203 1788527776.430012 punch from 0x037803 to 0x03bc1f (0x441d bytes) 000204 1788527776.451257 MIRROR_RESYNC 5 mirrors 000205 1788527777.377748 mapwrite 0x0126ab thru 0x0149da (0x2330 bytes) 000206 1788527777.457973 mapread 0x02b325 thru 0x03900a (0xdce6 bytes) 000207 1788527777.477012 write 0x000a47 thru 0x004738 (0x3cf2 bytes) 000208 1788527777.485343 punch from 0x02465e to 0x0313b9 (0xcd5c bytes) 000209 1788527777.490260 mapread 0x033be0 thru 0x03bc1f (0x8040 bytes) 000210 1788527777.491349 read 0x01b0de thru 0x026c5a (0xbb7d bytes) 000211 1788527777.501814 trunc from 0x01b0e2 to 0x025e0d (0xad2c bytes) 000212 1788527777.506310 write 0x034595 thru 0x03548f (0xefb bytes) 000213 1788527777.507839 mapwrite 0x01c3f2 thru 0x01d168 (0xd77 bytes) 000214 1788527777.536586 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bc20): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bc20): Mirror 5: 0x487287d6 (stale) Mirror 6: 0x487287d6 (stale) Mirror 7: 0x487287d6 (stale) Mirror 9: 0x487287d6 (stale) CRC-32 checksum value for chunk [0, 0x3bc20): Mirror 1: 0xa2a8ca89 Mirror 10: 0xa2a8ca89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bc20: skipped 000215 1788527778.668658 mapread 0x0280d6 thru 0x02f032 (0x6f5d bytes) 000216 1788527778.717058 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bc20): Mirror 5: 0x487287d6 (stale) Mirror 6: 0x487287d6 (stale) Mirror 7: 0x487287d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bc20): Mirror 5: 0x487287d6 (stale) Mirror 6: 0x487287d6 (stale) Mirror 7: 0x487287d6 (stale) CRC-32 checksum value for chunk [0, 0x3bc20): Mirror 1: 0xa2a8ca89 Mirror 10: 0xa2a8ca89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bc20: skipped 000217 1788527779.994953 fallocate 0x00bd8f thru 0x016382 (0xa5f4 bytes) 000218 1788527780.099978 mapwrite 0x02b6b3 thru 0x032baa (0x74f8 bytes) 000219 1788527780.185827 punch from 0x00c517 to 0x01b4bc (0xefa6 bytes) 000220 1788527780.195895 read 0x021293 thru 0x028892 (0x7600 bytes) 000221 1788527780.217296 mapwrite 0x028465 thru 0x02b4b8 (0x3054 bytes) 000222 1788527780.266628 fallocate 0x009a9a thru 0x01158e (0x7af5 bytes) 000223 1788527780.276884 mapwrite 0x032650 thru 0x03ffff (0xd9b0 bytes) 000224 1788527780.395876 trunc from 0x01d6be to 0x02052d (0x2e70 bytes) 000225 1788527780.403522 read 0x0314fd thru 0x03738f (0x5e93 bytes) 000226 1788527780.424774 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2345ad8b (stale) Mirror 6: 0x2345ad8b (stale) Mirror 7: 0x2345ad8b (stale) Mirror 10: 0x5b03277c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeb804096 Mirror 11: 0xeb804096 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000227 1788527781.316789 read 0x02b2cb thru 0x030e4a (0x5b80 bytes) 000228 1788527781.329958 write 0x00a82c thru 0x01a053 (0xf828 bytes) 000229 1788527781.391354 read 0x030d83 thru 0x0348f0 (0x3b6e bytes) 000230 1788527781.400836 trunc from 0x0237b6 to 0x02ea85 (0xb2d0 bytes) 000231 1788527781.405888 read 0x021241 thru 0x02c220 (0xafe0 bytes) 000232 1788527781.421720 mapwrite 0x01d7f7 thru 0x02372b (0x5f35 bytes) 000233 1788527781.443809 trunc from 0x040000 to 0x017159 (0x28ea6 bytes) 000234 1788527781.491188 write 0x00a9f1 thru 0x01822d (0xd83d bytes) 000235 1788527781.493437 punch from 0x009898 to 0x0120d7 (0x8840 bytes) 000236 1788527781.508242 read 0x012fde thru 0x01822d (0x5250 bytes) 000237 1788527781.517892 read 0x014dd7 thru 0x01822d (0x3457 bytes) 000238 1788527781.518510 punch from 0x002380 to 0x0024c6 (0x147 bytes) 000239 1788527781.528658 MIRROR_RESYNC 6 mirrors 000240 1788527782.753401 mapwrite 0x004c94 thru 0x00a30c (0x5679 bytes) 000242 1788527782.986398 trunc from 0x0171ce to 0x01822d (0x1060 bytes) 000243 1788527782.994846 write 0x006487 thru 0x00d22d (0x6da7 bytes) 000245 1788527782.997114 mapwrite 0x01f38d thru 0x029070 (0x9ce4 bytes) 000246 1788527783.120830 write 0x01baf4 thru 0x02591d (0x9e2a bytes) 000248 1788527783.133513 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x29071): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29071): Mirror 5: 0x5b00aa5a (stale) Mirror 6: 0x5b00aa5a (stale) Mirror 7: 0x5b00aa5a (stale) Mirror 10: 0x5b00aa5a (stale) Mirror 11: 0x5b00aa5a (stale) CRC-32 checksum value for chunk [0, 0x29071): Mirror 1: 0xd044bd64 Mirror 12: 0xd044bd64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29071: skipped 000249 1788527784.308105 mapwrite 0x038922 thru 0x03f16d (0x684c bytes) 000250 1788527784.528898 punch from 0x00b5d3 to 0x013d13 (0x8741 bytes) 000252 1788527784.541786 trunc from 0x02252c to 0x0268ee (0x43c3 bytes) 000253 1788527784.556925 punch from 0x00872e to 0x013e63 (0xb736 bytes) 000255 1788527784.564821 read 0x014e95 thru 0x01ba60 (0x6bcc bytes) 000256 1788527784.583269 fallocate 0x00cae8 thru 0x017cac (0xb1c5 bytes) 000257 1788527784.604236 mapread 0x00e748 thru 0x0168a3 (0x815c bytes) 000258 1788527784.626478 mapread 0x0364d5 thru 0x03f16d (0x8c99 bytes) 000259 1788527784.628921 read 0x025f1a thru 0x027b54 (0x1c3b bytes) 000260 1788527784.657536 read 0x03a4eb thru 0x03f16d (0x4c83 bytes) 000261 1788527784.658549 write 0x0203f3 thru 0x02e336 (0xdf44 bytes) 000263 1788527784.674765 trunc from 0x028b3c to 0x02d1d3 (0x4698 bytes) 000264 1788527784.701385 trunc from 0x039ad3 to 0x03b9ee (0x1f1c bytes) 000265 1788527784.711184 trunc from 0x03f16e to 0x01e5da (0x20b93 bytes) 000266 1788527784.756181 mapwrite 0x02c8c3 thru 0x03c237 (0xf975 bytes) 000267 1788527784.866385 write 0x025472 thru 0x02727e (0x1e0d bytes) 000268 1788527784.870930 mapread 0x03a2db thru 0x03c237 (0x1f5d bytes) 000269 1788527784.872735 fallocate 0x033a75 thru 0x036cfe (0x328a bytes) 000270 1788527784.885740 write 0x0272f6 thru 0x02bbca (0x48d5 bytes) 000271 1788527784.887921 punch from 0x01a14b to 0x026432 (0xc2e8 bytes) 000272 1788527784.912918 fallocate 0x00b7d1 thru 0x017935 (0xc165 bytes) 000273 1788527784.918638 fallocate 0x03ce1d thru 0x03d99c (0xb80 bytes) 000274 1788527784.925168 mapwrite 0x02765d thru 0x033be9 (0xc58d bytes) 000275 1788527784.944028 mapread 0x01220a thru 0x01cb17 (0xa90e bytes) 000277 1788527784.981651 trunc from 0x03d99d to 0x022ed0 (0x1aacc bytes) 000278 1788527785.037293 fallocate 0x0185a1 thru 0x0278f4 (0xf354 bytes) 000279 1788527785.062327 MIRROR_RESYNC 7 mirrors 000280 1788527786.647362 trunc from 0x026b6a to 0x0278f4 (0xd8b bytes) 000281 1788527786.690204 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x278f5): Mirror 5: 0xbea3f489 (stale) Mirror 6: 0xbea3f489 (stale) Mirror 7: 0xbea3f489 (stale) Mirror 11: 0xbea3f489 (stale) Mirror 12: 0xbea3f489 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x278f5): Mirror 5: 0xbea3f489 (stale) Mirror 6: 0xbea3f489 (stale) Mirror 7: 0xbea3f489 (stale) Mirror 11: 0xbea3f489 (stale) Mirror 12: 0xbea3f489 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x278f5: skipped 000282 1788527787.027497 punch from 0x0009ab to 0x00216b (0x17c1 bytes) 000283 1788527787.033504 mapread 0x010162 thru 0x011bf2 (0x1a91 bytes) 000284 1788527787.043886 mapwrite 0x02bf7e thru 0x02d534 (0x15b7 bytes) 000285 1788527787.087074 mapwrite 0x024030 thru 0x025faa (0x1f7b bytes) 000286 1788527787.099271 read 0x023a43 thru 0x029eae (0x646c bytes) 000287 1788527787.104337 fallocate 0x013eda thru 0x018c69 (0x4d90 bytes) 000288 1788527787.108562 mapread 0x0237e1 thru 0x029d78 (0x6598 bytes) 000289 1788527787.109421 trunc from 0x02d535 to 0x024f5d (0x85d7 bytes) 000290 1788527787.125578 mapwrite 0x03d62d thru 0x03ffff (0x29d3 bytes) 000291 1788527787.164763 trunc from 0x0084bc to 0x00ad24 (0x2869 bytes) 000292 1788527787.171713 mapwrite 0x028e3a thru 0x03429d (0xb464 bytes) 000293 1788527787.206635 mapwrite 0x008b3f thru 0x00f272 (0x6734 bytes) 000294 1788527787.238832 trunc from 0x01e62b to 0x02c94f (0xe325 bytes) 000295 1788527787.244346 trunc from 0x040000 to 0x030229 (0xfdd6 bytes) 000296 1788527787.260507 mapwrite 0x02e980 thru 0x0363e3 (0x7a64 bytes) 000297 1788527787.303415 mapwrite 0x0194c4 thru 0x01fc3c (0x6779 bytes) 000299 1788527787.336605 write 0x000def thru 0x00101f (0x231 bytes) 000300 1788527787.343211 mapread 0x01139f thru 0x01a27b (0x8edd bytes) 000301 1788527787.344871 write 0x01e391 thru 0x026cc3 (0x8933 bytes) 000302 1788527787.350589 mapwrite 0x0395c9 thru 0x03ffff (0x6a37 bytes) 000303 1788527787.418187 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4f61c851 (stale) Mirror 7: 0x4f61c851 (stale) Mirror 11: 0x4f61c851 (stale) Mirror 12: 0x4f61c851 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4f61c851 (stale) Mirror 7: 0x4f61c851 (stale) Mirror 11: 0x4f61c851 (stale) Mirror 12: 0x4f61c851 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000304 1788527787.872532 write 0x0221a0 thru 0x02c88b (0xa6ec bytes) 000305 1788527787.894329 read 0x00abb2 thru 0x00e82a (0x3c79 bytes) 000307 1788527787.903638 fallocate 0x002a4e thru 0x006adb (0x408e bytes) 000308 1788527787.910334 punch from 0x01ad67 to 0x027d44 (0xcfde bytes) 000309 1788527787.926818 mapread 0x02da21 thru 0x03850a (0xaaea bytes) 000310 1788527787.989693 fallocate 0x02ed7f thru 0x03119f (0x2421 bytes) 000311 1788527788.016797 write 0x016095 thru 0x024338 (0xe2a4 bytes) 000313 1788527788.051191 mapwrite 0x00707e thru 0x00f34b (0x82ce bytes) 000314 1788527788.104600 trunc from 0x02422d to 0x02f10f (0xaee3 bytes) 000315 1788527788.114934 trunc from 0x040000 to 0x034ecb (0xb134 bytes) 000316 1788527788.154178 mapread 0x0170a2 thru 0x0237f4 (0xc753 bytes) 000317 1788527788.155951 punch from 0x0180aa to 0x01f578 (0x74cf bytes) 000318 1788527788.167256 mapread 0x007ac2 thru 0x00f08b (0x75ca bytes) 000319 1788527788.170642 trunc from 0x02d7ea to 0x034ecb (0x76e2 bytes) 000320 1788527788.180113 fallocate 0x00f0be thru 0x01bd96 (0xccd9 bytes) 000323 1788527788.197897 write 0x013ebb thru 0x0229bc (0xeb02 bytes) 000324 1788527788.216108 punch from 0x02dc42 to 0x0333ec (0x57ab bytes) 000325 1788527788.223074 punch from 0x00609b to 0x00e137 (0x809d bytes) 000326 1788527788.241780 mapread 0x00b15e thru 0x010bb0 (0x5a53 bytes) 000327 1788527788.267008 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34ecc): Mirror 7: 0x8fb272e6 (stale) Mirror 11: 0x8fb272e6 (stale) Mirror 12: 0x8fb272e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34ecc): Mirror 7: 0x8fb272e6 (stale) Mirror 11: 0x8fb272e6 (stale) Mirror 12: 0x8fb272e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34ecc: skipped 000328 1788527788.708420 trunc from 0x014ba9 to 0x01de38 (0x9290 bytes) 000329 1788527788.713092 punch from 0x000e8c to 0x00e054 (0xd1c9 bytes) 000330 1788527788.720482 read 0x02de0d thru 0x034ecb (0x70bf bytes) 000331 1788527788.732524 mapread 0x009d7d thru 0x011961 (0x7be5 bytes) 000332 1788527788.755710 mapwrite 0x02c92f thru 0x03403d (0x770f bytes) 000333 1788527788.784443 mapwrite 0x02168a thru 0x02357e (0x1ef5 bytes) 000334 1788527788.805639 fallocate 0x0124f3 thru 0x01b222 (0x8d30 bytes) 000335 1788527788.810539 mapwrite 0x0242b3 thru 0x027b01 (0x384f bytes) 000336 1788527788.823869 read 0x017c92 thru 0x023748 (0xbab7 bytes) 000337 1788527788.831003 read 0x021a14 thru 0x0279c1 (0x5fae bytes) 000338 1788527788.835251 MIRROR_RESYNC 4 mirrors 000339 1788527789.975237 trunc from 0x034ecc to 0x01db31 (0x1739a bytes) 000340 1788527790.014292 punch from 0x019356 to 0x01db31 (0x47dc bytes) 000342 1788527790.021370 write 0x033194 thru 0x034205 (0x1072 bytes) 000343 1788527790.022709 fallocate 0x02c95e thru 0x02dd8f (0x1432 bytes) 000344 1788527790.027849 fallocate 0x00ef05 thru 0x014600 (0x56fc bytes) 000345 1788527790.035854 MIRROR_RESYNC 4 mirrors 000346 1788527791.109343 mapwrite 0x0012af thru 0x009153 (0x7ea5 bytes) 000347 1788527791.193762 read 0x005ef8 thru 0x013231 (0xd33a bytes) 000348 1788527791.197563 trunc from 0x00828d to 0x00ec56 (0x69ca bytes) 000349 1788527791.201498 trunc from 0x034206 to 0x037e60 (0x3c5b bytes) 000350 1788527791.213993 punch from 0x00b5a3 to 0x0113c1 (0x5e1f bytes) 000351 1788527791.217788 write 0x036eae thru 0x038aa8 (0x1bfb bytes) 000352 1788527791.220452 mapread 0x02ecc2 thru 0x038aa8 (0x9de7 bytes) 000353 1788527791.227710 fallocate 0x01322d thru 0x01ed38 (0xbb0c bytes) 000354 1788527791.232584 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38aa9): Mirror 11: 0x4ca7987b (stale) Mirror 12: 0x4ca7987b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38aa9): Mirror 11: 0x4ca7987b (stale) Mirror 12: 0x4ca7987b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38aa9: skipped 000355 1788527791.530170 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38aa9): Mirror 11: 0x4ca7987b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38aa9): Mirror 11: 0x4ca7987b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38aa9: skipped 000356 1788527791.875304 read 0x0215e9 thru 0x02fef1 (0xe909 bytes) 000357 1788527791.887883 punch from 0x0043b7 to 0x00e25d (0x9ea7 bytes) 000358 1788527791.893328 trunc from 0x003aa3 to 0x00d028 (0x9586 bytes) 000359 1788527791.898286 write 0x03adc2 thru 0x03ffff (0x523e bytes) 000360 1788527791.904355 fallocate 0x0224d1 thru 0x02f6b3 (0xd1e3 bytes) 000361 1788527791.911301 trunc from 0x00c68e to 0x011fb7 (0x592a bytes) 000362 1788527791.916722 punch from 0x00d227 to 0x013e10 (0x6bea bytes) 000363 1788527791.924315 mapwrite 0x02f9f7 thru 0x0397a0 (0x9daa bytes) 000364 1788527791.958839 read 0x020c27 thru 0x0250a6 (0x4480 bytes) 000365 1788527791.968461 trunc from 0x020fb0 to 0x0271fc (0x624d bytes) 000366 1788527791.973503 trunc from 0x040000 to 0x01b059 (0x24fa6 bytes) 000367 1788527791.987026 read 0x014432 thru 0x01b059 (0x6c28 bytes) 000368 1788527791.987745 mapwrite 0x0371c3 thru 0x03ffff (0x8e3d bytes) 000369 1788527792.015228 write 0x03dd33 thru 0x03ffff (0x22cd bytes) 000370 1788527792.016241 mapwrite 0x0336d5 thru 0x036522 (0x2e4e bytes) 000371 1788527792.030071 read 0x019b02 thru 0x025a4b (0xbf4a bytes) 000373 1788527792.030897 read 0x006347 thru 0x015ca2 (0xf95c bytes) 000374 1788527792.039069 mapwrite 0x00fa4e thru 0x01d59e (0xdb51 bytes) 000375 1788527792.053715 write 0x009e57 thru 0x00c57f (0x2729 bytes) 000376 1788527792.055025 trunc from 0x0392b8 to 0x03efa0 (0x5ce9 bytes) 000378 1788527792.060630 trunc from 0x0379b4 to 0x03ffff (0x864c bytes) 000380 1788527792.067337 fallocate 0x025148 thru 0x03314b (0xe004 bytes) 000382 1788527792.071571 read 0x024d5f thru 0x0278d3 (0x2b75 bytes) 000385 1788527792.073745 trunc from 0x040000 to 0x007b2f (0x384d0 bytes) 000386 1788527792.094087 mapread 0x0042eb thru 0x007773 (0x3489 bytes) 000387 1788527792.094904 mapwrite 0x02ce2b thru 0x02f4ac (0x2682 bytes) 000388 1788527792.128748 trunc from 0x02f4ad to 0x010d9c (0x1e710 bytes) 000390 1788527792.139157 trunc from 0x010d9d to 0x012e12 (0x2076 bytes) 000391 1788527792.149933 trunc from 0x00b0dc to 0x00d2c8 (0x21ed bytes) 000392 1788527792.152976 mapread 0x00e689 thru 0x012e12 (0x478a bytes) 000393 1788527792.163781 trunc from 0x012e13 to 0x011ef7 (0xf1b bytes) 000394 1788527792.177820 mapwrite 0x03a1b0 thru 0x03ffff (0x5e50 bytes) 000395 1788527792.223498 trunc from 0x035d2c to 0x03ffff (0xa2d4 bytes) 000396 1788527792.229901 read 0x01ea12 thru 0x02386e (0x4e5d bytes) 000397 1788527792.236355 write 0x029209 thru 0x030d9f (0x7b97 bytes) 000398 1788527792.237757 trunc from 0x040000 to 0x021304 (0x1ecfb bytes) 000399 1788527792.251956 fallocate 0x01f845 thru 0x02d822 (0xdfde bytes) 000400 1788527792.265366 write 0x014c85 thru 0x018565 (0x38e1 bytes) 000401 1788527792.269940 read 0x025c17 thru 0x02d822 (0x7c0c bytes) 000402 1788527792.277666 mapread 0x0262b2 thru 0x02d822 (0x7571 bytes) 000404 1788527792.282205 mapwrite 0x03b64f thru 0x03edf8 (0x37aa bytes) 000405 1788527792.320423 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000406 1788527792.441830 trunc from 0x03edf9 to 0x033a4e (0xb3aa bytes) 000407 1788527792.458969 trunc from 0x0118a1 to 0x013c62 (0x23c2 bytes) 000408 1788527792.462030 trunc from 0x033a4f to 0x022dc9 (0x10c85 bytes) 000409 1788527792.474276 trunc from 0x020e58 to 0x022dc9 (0x1f72 bytes) 000410 1788527792.478044 trunc from 0x01325b to 0x01e4d5 (0xb27b bytes) 000411 1788527792.480670 trunc from 0x022dca to 0x03a515 (0x1774c bytes) 000412 1788527792.492502 mapread 0x00ed6c thru 0x019866 (0xaafb bytes) 000413 1788527792.509116 mapread 0x013022 thru 0x01cbbf (0x9b9e bytes) 000415 1788527792.510436 fallocate 0x0216ef thru 0x0263e4 (0x4cf6 bytes) 000416 1788527792.513596 mapwrite 0x004be5 thru 0x00baa7 (0x6ec3 bytes) 000417 1788527792.532735 mapwrite 0x018a79 thru 0x01b9ba (0x2f42 bytes) 000418 1788527792.543426 punch from 0x025c55 to 0x026728 (0xad4 bytes) 000420 1788527792.547188 punch from 0x00ec8f to 0x01a261 (0xb5d3 bytes) 000421 1788527792.551083 read 0x014452 thru 0x02387c (0xf42b bytes) 000422 1788527792.556405 write 0x0335a7 thru 0x039d63 (0x67bd bytes) 000423 1788527792.562102 read 0x005b72 thru 0x012e0b (0xd29a bytes) 000424 1788527792.568956 read 0x002d1a thru 0x00528b (0x2572 bytes) 000425 1788527792.569568 punch from 0x036461 to 0x03a515 (0x40b5 bytes) 000428 1788527792.581017 read 0x02d2ed thru 0x034151 (0x6e65 bytes) 000429 1788527792.581494 mapread 0x03513e thru 0x03725a (0x211d bytes) 000430 1788527792.586685 mapread 0x013726 thru 0x014b02 (0x13dd bytes) 000431 1788527792.587491 punch from 0x021c2c to 0x02644f (0x4824 bytes) 000432 1788527792.594796 mapread 0x028aba thru 0x02f7a4 (0x6ceb bytes) 000434 1788527792.596083 mapwrite 0x031a55 thru 0x03ffff (0xe5ab bytes) 000435 1788527792.630419 mapread 0x00da04 thru 0x01b2ec (0xd8e9 bytes) 000436 1788527792.631186 read 0x00f5aa thru 0x010544 (0xf9b bytes) 000437 1788527792.631509 write 0x00f12f thru 0x01063f (0x1511 bytes) 000438 1788527792.632207 mapread 0x038539 thru 0x03ffff (0x7ac7 bytes) 000439 1788527792.632905 fallocate 0x030c55 thru 0x03f150 (0xe4fc bytes) 000440 1788527792.637096 trunc from 0x035a8b to 0x036d84 (0x12fa bytes) 000442 1788527792.641198 write 0x02ca36 thru 0x039fd9 (0xd5a4 bytes) 000443 1788527792.642317 trunc from 0x001156 to 0x0011f4 (0x9f bytes) 000444 1788527792.645926 trunc from 0x040000 to 0x0027c0 (0x3d83f bytes) 000445 1788527792.658158 fallocate 0x03af24 thru 0x03ffff (0x50dc bytes) 000446 1788527792.662933 punch from 0x00dec8 to 0x016931 (0x8a6a bytes) 000447 1788527792.668190 read 0x03cced thru 0x03ffff (0x3313 bytes) 000448 1788527792.678473 mapread 0x0021aa thru 0x00f2f9 (0xd150 bytes) 000451 1788527792.680078 write 0x032903 thru 0x03eb68 (0xc266 bytes) 000452 1788527792.680978 read 0x03da1c thru 0x03db13 (0xf8 bytes) 000453 1788527792.681946 mapwrite 0x033aea thru 0x03ffff (0xc516 bytes) 000454 1788527792.698294 mapread 0x032c53 thru 0x03ffff (0xd3ad bytes) 000455 1788527792.698851 read 0x0175d2 thru 0x01882c (0x125b bytes) 000456 1788527792.699301 mapwrite 0x011476 thru 0x01a72d (0x92b8 bytes) 000457 1788527792.710585 trunc from 0x040000 to 0x03cc5f (0x33a0 bytes) 000458 1788527792.731213 fallocate 0x008d3c thru 0x00f793 (0x6a58 bytes) 000459 1788527792.737287 mapwrite 0x02ce00 thru 0x033ccb (0x6ecc bytes) 000460 1788527792.757237 read 0x02afac thru 0x039227 (0xe27c bytes) 000461 1788527792.759661 read 0x018273 thru 0x01bb26 (0x38b4 bytes) 000462 1788527792.768258 MIRROR_EXTEND to 2 mirrors 000463 1788527793.303519 trunc from 0x03cc60 to 0x008053 (0x34c0c bytes) 000464 1788527793.344709 mapwrite 0x009d60 thru 0x00e417 (0x46b8 bytes) 000465 1788527793.382824 mapread 0x000244 thru 0x007572 (0x732f bytes) 000466 1788527793.388113 mapread 0x005b3b thru 0x00e0fb (0x85c1 bytes) 000467 1788527793.389439 mapread 0x007e71 thru 0x00a990 (0x2b20 bytes) 000468 1788527793.390467 punch from 0x00d151 to 0x00e417 (0x12c7 bytes) 000469 1788527793.396846 mapread 0x001917 thru 0x006a64 (0x514e bytes) 000470 1788527793.398095 punch from 0x007b85 to 0x009012 (0x148e bytes) 000471 1788527793.412251 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xe418): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe418): Mirror 2: 0x2ae4c3e3 (stale) CRC-32 checksum value for chunk [0, 0xe418): Mirror 1: 0x2323c716 Mirror 3: 0x2323c716 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe418: skipped 000472 1788527794.000054 punch from 0x00191e to 0x00419f (0x2882 bytes) 000473 1788527794.032580 mapread 0x008337 thru 0x00a209 (0x1ed3 bytes) 000474 1788527794.055205 fallocate 0x028a53 thru 0x0388dc (0xfe8a bytes) 000475 1788527794.067536 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x388dd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x388dd): Mirror 2: 0xd81116fd (stale) Mirror 3: 0x23141c46 (stale) CRC-32 checksum value for chunk [0, 0x388dd): Mirror 1: 0x375fe008 Mirror 4: 0x375fe008 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x388dd: skipped 000476 1788527794.906091 read 0x01620f thru 0x01646f (0x261 bytes) 000477 1788527794.918575 write 0x03866e thru 0x03880b (0x19e bytes) 000479 1788527794.978259 write 0x02ea75 thru 0x031eec (0x3478 bytes) 000480 1788527794.987596 mapwrite 0x00be91 thru 0x019305 (0xd475 bytes) 000481 1788527795.034473 read 0x01e690 thru 0x02cdc6 (0xe737 bytes) 000482 1788527795.060287 read 0x005f47 thru 0x00deb9 (0x7f73 bytes) 000483 1788527795.061431 punch from 0x0153f3 to 0x01debc (0x8aca bytes) 000484 1788527795.069649 trunc from 0x0388dd to 0x02859f (0x1033d bytes) 000485 1788527795.101645 read 0x02737d thru 0x027ff9 (0xc7d bytes) 000486 1788527795.103174 mapread 0x00c6ba thru 0x011ef1 (0x5838 bytes) 000487 1788527795.104373 fallocate 0x022671 thru 0x029386 (0x6d16 bytes) 000488 1788527795.110216 fallocate 0x021bb8 thru 0x028c76 (0x70bf bytes) 000489 1788527795.120234 punch from 0x019aac to 0x028ad8 (0xf02d bytes) 000490 1788527795.125433 mapwrite 0x03c1a4 thru 0x03ffff (0x3e5c bytes) 000491 1788527795.161506 trunc from 0x019499 to 0x02082f (0x7397 bytes) 000492 1788527795.167373 trunc from 0x040000 to 0x03ea79 (0x1586 bytes) 000493 1788527795.186653 mapwrite 0x0274f3 thru 0x031003 (0x9b11 bytes) 000494 1788527795.205978 write 0x02005e thru 0x020c9b (0xc3e bytes) 000495 1788527795.211286 trunc from 0x03e3f3 to 0x03ea79 (0x687 bytes) 000496 1788527795.216795 read 0x02d046 thru 0x030a27 (0x39e2 bytes) 000497 1788527795.217556 mapread 0x02eef1 thru 0x0375a5 (0x86b5 bytes) 000498 1788527795.219396 mapread 0x00f269 thru 0x00fbf5 (0x98d bytes) 000499 1788527795.221445 write 0x013b00 thru 0x01802a (0x452b bytes) 000500 1788527795.223187 mapwrite 0x0355e9 thru 0x03aaf0 (0x5508 bytes) 000502 1788527795.240717 trunc from 0x019a40 to 0x027f15 (0xe4d6 bytes) 000504 1788527795.245622 read 0x031ce9 thru 0x035aea (0x3e02 bytes) 000505 1788527795.246204 mapwrite 0x02d23b thru 0x03bcf0 (0xeab6 bytes) 000506 1788527795.264617 mapwrite 0x03ffe6 thru 0x03ffff (0x1a bytes) 000507 1788527795.299740 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x75fd636b (stale) Mirror 4: 0xe592682f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x75fd636b (stale) Mirror 4: 0xe592682f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000508 1788527795.606300 punch from 0x035079 to 0x03ebc6 (0x9b4e bytes) 000509 1788527795.611582 mapwrite 0x035cab thru 0x03ffff (0xa355 bytes) 000510 1788527795.646795 read 0x016561 thru 0x017d5b (0x17fb bytes) 000511 1788527795.651432 trunc from 0x023e91 to 0x03031f (0xc48f bytes) 000512 1788527795.656172 fallocate 0x01c00a thru 0x01eb56 (0x2b4d bytes) 000513 1788527795.660272 punch from 0x03fc30 to 0x03ffff (0x3d0 bytes) 000514 1788527795.667212 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x75fd636b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x75fd636b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000515 1788527795.941309 write 0x0237ca thru 0x029dcc (0x6603 bytes) 000516 1788527795.956073 trunc from 0x040000 to 0x01e4a5 (0x21b5a bytes) 000517 1788527795.975551 read 0x01a5c3 thru 0x01e4a5 (0x3ee3 bytes) 000518 1788527795.984262 mapwrite 0x02f43e thru 0x03d24f (0xde12 bytes) 000519 1788527796.061728 write 0x023fcd thru 0x0282e9 (0x431d bytes) 000520 1788527796.072253 punch from 0x029715 to 0x03008f (0x697b bytes) 000522 1788527796.079979 punch from 0x026e20 to 0x032e6f (0xc050 bytes) 000523 1788527796.094434 punch from 0x035cb3 to 0x03d24f (0x759d bytes) 000524 1788527796.098668 read 0x0228a4 thru 0x0251e0 (0x293d bytes) 000525 1788527796.109010 mapread 0x01c6c1 thru 0x0228ae (0x61ee bytes) 000526 1788527796.110321 punch from 0x03c695 to 0x03d24f (0xbbb bytes) 000527 1788527796.114784 write 0x038215 thru 0x03ffff (0x7deb bytes) 000528 1788527796.116254 trunc from 0x040000 to 0x003d45 (0x3c2ba bytes) 000529 1788527796.134335 trunc from 0x003d46 to 0x019a80 (0x15d3b bytes) 000530 1788527796.148407 fallocate 0x02e264 thru 0x03c9a4 (0xe741 bytes) 000531 1788527796.154347 trunc from 0x019a81 to 0x0294cf (0xfa4f bytes) 000532 1788527796.188498 MIRROR_RESYNC 2 mirrors 000533 1788527796.719504 mapwrite 0x00193b thru 0x00bdfb (0xa4c1 bytes) 000534 1788527796.827423 read 0x027bc1 thru 0x0294cf (0x190f bytes) 000535 1788527796.841351 trunc from 0x0006fe to 0x0062fe (0x5c01 bytes) 000536 1788527796.849809 mapwrite 0x000248 thru 0x0024a1 (0x225a bytes) 000537 1788527796.884315 write 0x02ca99 thru 0x02d930 (0xe98 bytes) 000539 1788527796.887099 mapwrite 0x0200d2 thru 0x028434 (0x8363 bytes) 000540 1788527796.926751 fallocate 0x031dfa thru 0x033be3 (0x1dea bytes) 000541 1788527796.934448 mapread 0x01d5e3 thru 0x028ef1 (0xb90f bytes) 000542 1788527796.937708 punch from 0x00d2bc to 0x00f883 (0x25c8 bytes) 000544 1788527796.946970 write 0x029f02 thru 0x032b51 (0x8c50 bytes) 000545 1788527796.953997 write 0x031df1 thru 0x03ffff (0xe20f bytes) 000547 1788527796.960923 fallocate 0x036a6e thru 0x039336 (0x28c9 bytes) 000548 1788527796.967068 mapwrite 0x00ef82 thru 0x0112c6 (0x2345 bytes) 000549 1788527797.009771 read 0x01457a thru 0x0176bb (0x3142 bytes) 000550 1788527797.016443 mapread 0x006572 thru 0x00fa7c (0x950b bytes) 000551 1788527797.017714 trunc from 0x040000 to 0x015502 (0x2aafd bytes) 000552 1788527797.064097 trunc from 0x00093e to 0x004bcd (0x4290 bytes) 000553 1788527797.070316 read 0x00993e thru 0x00ea5b (0x511e bytes) 000554 1788527797.071985 write 0x015377 thru 0x0242ea (0xef74 bytes) 000555 1788527797.074854 fallocate 0x0187b2 thru 0x01ceb4 (0x4703 bytes) 000557 1788527797.080426 mapwrite 0x004700 thru 0x0057ff (0x1100 bytes) 000558 1788527797.105788 write 0x026dd4 thru 0x028345 (0x1572 bytes) 000559 1788527797.107457 write 0x00ccea thru 0x016db6 (0xa0cd bytes) 000560 1788527797.110243 mapwrite 0x007517 thru 0x0076d0 (0x1ba bytes) 000561 1788527797.125305 trunc from 0x01595a to 0x02443f (0xeae6 bytes) 000562 1788527797.131498 mapwrite 0x033cf5 thru 0x03ffff (0xc30b bytes) 000563 1788527797.188398 trunc from 0x0265cd to 0x033f6f (0xd9a3 bytes) 000564 1788527797.197378 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd438866a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdeecc5fd Mirror 4: 0xdeecc5fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000566 1788527798.053167 read 0x0383e4 thru 0x03ffff (0x7c1c bytes) 000567 1788527798.076483 read 0x01525a thru 0x01ef03 (0x9caa bytes) 000569 1788527798.129451 trunc from 0x040000 to 0x00f4c0 (0x30b3f bytes) 000571 1788527798.213186 mapread 0x00cb21 thru 0x00f4c0 (0x29a0 bytes) 000573 1788527798.244856 write 0x01c785 thru 0x0265d2 (0x9e4e bytes) 000574 1788527798.249055 trunc from 0x00ecd4 to 0x012b7d (0x3eaa bytes) 000575 1788527798.258897 fallocate 0x019879 thru 0x0254f1 (0xbc79 bytes) 000577 1788527798.266302 trunc from 0x007c62 to 0x00960e (0x19ad bytes) 000578 1788527798.282614 punch from 0x0158a9 to 0x025533 (0xfc8b bytes) 000579 1788527798.297587 trunc from 0x00cdcc to 0x010fd1 (0x4206 bytes) 000580 1788527798.305061 read 0x015fda thru 0x018686 (0x26ad bytes) 000581 1788527798.328104 write 0x029c53 thru 0x03136f (0x771d bytes) 000582 1788527798.330488 write 0x033e19 thru 0x034134 (0x31c bytes) 000583 1788527798.342112 mapread 0x0273e1 thru 0x032df4 (0xba14 bytes) 000584 1788527798.358013 fallocate 0x02952b thru 0x038930 (0xf406 bytes) 000585 1788527798.368233 trunc from 0x00ad68 to 0x00b657 (0x8f0 bytes) 000586 1788527798.371969 punch from 0x033bcd to 0x034134 (0x568 bytes) 000587 1788527798.385352 mapread 0x01ebad thru 0x020e2c (0x2280 bytes) 000588 1788527798.386011 mapread 0x003c6a thru 0x00a003 (0x639a bytes) 000589 1788527798.391422 punch from 0x027361 to 0x029c32 (0x28d2 bytes) 000590 1788527798.396181 mapwrite 0x03601a thru 0x03ffff (0x9fe6 bytes) 000591 1788527798.471293 fallocate 0x003e59 thru 0x0099b8 (0x5b60 bytes) 000592 1788527798.476012 mapwrite 0x00aceb thru 0x01590e (0xac24 bytes) 000593 1788527798.514508 trunc from 0x035cb4 to 0x03a660 (0x49ad bytes) 000594 1788527798.526327 punch from 0x0126ad to 0x01ec90 (0xc5e4 bytes) 000595 1788527798.543969 trunc from 0x040000 to 0x038796 (0x7869 bytes) 000596 1788527798.566032 punch from 0x00eeba to 0x01be93 (0xcfda bytes) 000597 1788527798.570615 mapwrite 0x03d49e thru 0x03fbd9 (0x273c bytes) 000598 1788527798.594367 write 0x003553 thru 0x008eff (0x59ad bytes) 000599 1788527798.595479 punch from 0x021ada to 0x023f62 (0x2489 bytes) 000600 1788527798.600617 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fbda): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fbda): Mirror 3: 0x9d849b77 (stale) Mirror 4: 0x72afd580 (stale) CRC-32 checksum value for chunk [0, 0x3fbda): Mirror 1: 0xbcf97331 Mirror 5: 0xbcf97331 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fbda: skipped 000601 1788527799.278950 trunc from 0x03fbda to 0x00cacb (0x3310e bytes) 000602 1788527799.333556 mapwrite 0x026519 thru 0x02b5e6 (0x50ce bytes) 000603 1788527799.368759 mapwrite 0x021242 thru 0x024401 (0x31c0 bytes) 000604 1788527799.378442 mapread 0x020e02 thru 0x02b5e6 (0xa7e5 bytes) 000605 1788527799.386178 trunc from 0x024d10 to 0x029c70 (0x4f61 bytes) 000607 1788527799.389587 trunc from 0x02089f to 0x022fa1 (0x2703 bytes) 000608 1788527799.397447 trunc from 0x02b5e7 to 0x022e25 (0x87c1 bytes) 000609 1788527799.415719 mapwrite 0x03d75e thru 0x03ffff (0x28a2 bytes) 000610 1788527799.468402 trunc from 0x040000 to 0x01e483 (0x21b7c bytes) 000611 1788527799.479767 write 0x038e68 thru 0x03ffff (0x7198 bytes) 000612 1788527799.481148 punch from 0x02c7d5 to 0x0367ab (0x9fd7 bytes) 000613 1788527799.489552 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd438866a (stale) Mirror 4: 0xdeecc5fd (stale) Mirror 5: 0xd41f0143 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe22abd8 Mirror 6: 0xe22abd8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000614 1788527800.682412 read 0x017af9 thru 0x01f7c7 (0x7ccf bytes) 000615 1788527800.727245 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd438866a (stale) Mirror 4: 0xdeecc5fd (stale) Mirror 5: 0xd41f0143 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd438866a (stale) Mirror 4: 0xdeecc5fd (stale) Mirror 5: 0xd41f0143 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000616 1788527801.161520 mapread 0x02e14f thru 0x031a04 (0x38b6 bytes) 000617 1788527801.209716 fallocate 0x012abb thru 0x01724e (0x4794 bytes) 000618 1788527801.251702 fallocate 0x02f94d thru 0x0359a9 (0x605d bytes) 000619 1788527801.283612 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd438866a (stale) Mirror 5: 0xd41f0143 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd438866a (stale) Mirror 5: 0xd41f0143 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000620 1788527801.721164 MIRROR_RESYNC 3 mirrors 000622 1788527802.961720 mapwrite 0x031311 thru 0x0384ff (0x71ef bytes) 000623 1788527803.041083 trunc from 0x037cf1 to 0x03ffff (0x830f bytes) 000624 1788527803.044358 trunc from 0x034ccb to 0x03b095 (0x63cb bytes) 000625 1788527803.049015 fallocate 0x00caf2 thru 0x01528c (0x879b bytes) 000626 1788527803.052953 fallocate 0x01f04f thru 0x02de84 (0xee36 bytes) 000627 1788527803.056579 read 0x02be87 thru 0x035169 (0x92e3 bytes) 000628 1788527803.062479 punch from 0x03eae4 to 0x03ffff (0x151c bytes) 000629 1788527803.068389 punch from 0x010fd9 to 0x01a9a3 (0x99cb bytes) 000630 1788527803.073337 punch from 0x014927 to 0x01d7ac (0x8e86 bytes) 000631 1788527803.077341 write 0x03b577 thru 0x03b9c9 (0x453 bytes) 000632 1788527803.082525 mapread 0x02cf73 thru 0x0354bb (0x8549 bytes) 000634 1788527803.093258 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe22abd8 (stale) Mirror 5: 0xe22abd8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6fbb4c17 Mirror 6: 0x6fbb4c17 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000635 1788527803.896361 mapread 0x00b8d7 thru 0x00f24d (0x3977 bytes) 000636 1788527803.940564 trunc from 0x015180 to 0x02115f (0xbfe0 bytes) 000637 1788527803.961853 trunc from 0x0184a4 to 0x0209b9 (0x8516 bytes) 000638 1788527803.964630 trunc from 0x003f10 to 0x00c72e (0x881f bytes) 000639 1788527803.967577 trunc from 0x00dd33 to 0x01292e (0x4bfc bytes) 000640 1788527803.970469 read 0x027cdb thru 0x03572e (0xda54 bytes) 000641 1788527803.977784 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe22abd8 (stale) Mirror 6: 0x6fbb4c17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe22abd8 (stale) Mirror 6: 0x6fbb4c17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000642 1788527804.301191 read 0x03d289 thru 0x03ffff (0x2d77 bytes) 000643 1788527804.310032 mapread 0x02836b thru 0x0299dd (0x1673 bytes) 000644 1788527804.326953 read 0x01fc7f thru 0x02be8e (0xc210 bytes) 000645 1788527804.339378 mapread 0x03d41b thru 0x03ffff (0x2be5 bytes) 000646 1788527804.340291 trunc from 0x00bc9c to 0x019cfb (0xe060 bytes) 000647 1788527804.354086 punch from 0x022202 to 0x030bd2 (0xe9d1 bytes) 000648 1788527804.359381 trunc from 0x019aa1 to 0x02282f (0x8d8f bytes) 000649 1788527804.370053 mapwrite 0x00368b thru 0x00739c (0x3d12 bytes) 000650 1788527804.397465 MIRROR_RESYNC 3 mirrors 000651 1788527805.700025 trunc from 0x040000 to 0x03f153 (0xeac bytes) 000652 1788527805.743418 punch from 0x026ed2 to 0x0289ac (0x1adb bytes) 000653 1788527805.756294 mapwrite 0x02548b thru 0x02d4a0 (0x8016 bytes) 000654 1788527805.845897 punch from 0x001e2b to 0x00a753 (0x8929 bytes) 000655 1788527805.867250 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f154): Mirror 6: 0xf0692c5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f154): Mirror 6: 0xf0692c5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f154: skipped 000657 1788527806.149953 mapread 0x010e7d thru 0x0148ff (0x3a83 bytes) 000658 1788527806.168076 trunc from 0x00b213 to 0x011d97 (0x6b85 bytes) 000659 1788527806.171792 mapwrite 0x021f9a thru 0x023ac5 (0x1b2c bytes) 000660 1788527806.199103 mapwrite 0x0180cc thru 0x01c25d (0x4192 bytes) 000661 1788527806.234615 trunc from 0x037146 to 0x03f153 (0x800e bytes) 000662 1788527806.243755 trunc from 0x0198c1 to 0x0260e9 (0xc829 bytes) 000664 1788527806.249413 trunc from 0x03f154 to 0x033d08 (0xb44b bytes) 000665 1788527806.265399 trunc from 0x02e589 to 0x033c44 (0x56bc bytes) 000666 1788527806.286241 MIRROR_RESYNC 2 mirrors 000667 1788527806.924365 mapread 0x0325fd thru 0x032e6c (0x870 bytes) 000668 1788527806.931203 trunc from 0x033d09 to 0x039341 (0x5639 bytes) 000669 1788527806.977196 mapwrite 0x015079 thru 0x018b0d (0x3a95 bytes) 000670 1788527807.020591 punch from 0x038704 to 0x039341 (0xc3e bytes) 000671 1788527807.025658 read 0x022cf5 thru 0x027a73 (0x4d7f bytes) 000672 1788527807.036572 read 0x018bef thru 0x01e1a6 (0x55b8 bytes) 000673 1788527807.044494 MIRROR_RESYNC 2 mirrors 000674 1788527808.081555 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000675 1788527808.278362 MIRROR_EXTEND to 2 mirrors 000676 1788527809.642435 trunc from 0x00ebf7 to 0x01bf72 (0xd37c bytes) 000677 1788527809.706588 fallocate 0x002e77 thru 0x00710c (0x4296 bytes) 000678 1788527809.712856 punch from 0x00f98f to 0x01f103 (0xf775 bytes) 000679 1788527809.719297 trunc from 0x039342 to 0x037ee5 (0x145c bytes) 000680 1788527809.742112 mapread 0x02c332 thru 0x034e0c (0x8adb bytes) 000681 1788527809.779864 trunc from 0x01ee37 to 0x023f14 (0x50de bytes) 000682 1788527809.784855 fallocate 0x010a42 thru 0x01e62b (0xdbea bytes) 000683 1788527809.790981 mapwrite 0x010e48 thru 0x01a348 (0x9501 bytes) 000686 1788527809.867515 mapwrite 0x038afd thru 0x03ffff (0x7503 bytes) 000687 1788527809.903838 trunc from 0x00d01f to 0x01815a (0xb13c bytes) 000688 1788527809.908563 write 0x013aec thru 0x017fb8 (0x44cd bytes) 000689 1788527809.915522 read 0x007cae thru 0x01097b (0x8cce bytes) 000690 1788527809.929241 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000691 1788527810.050349 trunc from 0x01cdaa to 0x01ce50 (0xa7 bytes) 000692 1788527810.055435 write 0x004c2d thru 0x008c9f (0x4073 bytes) 000694 1788527810.070267 trunc from 0x03e84c to 0x03ffff (0x17b4 bytes) 000695 1788527810.075027 trunc from 0x040000 to 0x014bcb (0x2b434 bytes) 000696 1788527810.093592 trunc from 0x0002c9 to 0x00cd9b (0xcad3 bytes) 000697 1788527810.115899 read 0x00a6c8 thru 0x00bfe9 (0x1922 bytes) 000698 1788527810.121400 read 0x012bc3 thru 0x014bcb (0x2009 bytes) 000699 1788527810.129344 mapread 0x005b0c thru 0x006349 (0x83e bytes) 000701 1788527810.131463 punch from 0x00702f to 0x007f1a (0xeec bytes) 000702 1788527810.135546 mapread 0x009441 thru 0x010452 (0x7012 bytes) 000703 1788527810.137290 fallocate 0x038b9e thru 0x03f0ef (0x6552 bytes) 000704 1788527810.141393 read 0x02290d thru 0x03242b (0xfb1f bytes) 000705 1788527810.152964 mapwrite 0x037b62 thru 0x03f47b (0x791a bytes) 000706 1788527810.182318 mapread 0x0325b1 thru 0x035e30 (0x3880 bytes) 000707 1788527810.183221 trunc from 0x0319bb to 0x0388f2 (0x6f38 bytes) 000708 1788527810.190367 punch from 0x008cdc to 0x009968 (0xc8d bytes) 000709 1788527810.200362 write 0x000785 thru 0x002a32 (0x22ae bytes) 000710 1788527810.201497 fallocate 0x02458b thru 0x0302b4 (0xbd2a bytes) 000711 1788527810.206513 punch from 0x03bf0a to 0x03f47b (0x3572 bytes) 000712 1788527810.211629 trunc from 0x03f47c to 0x0237ee (0x1bc8d bytes) 000713 1788527810.249446 write 0x022453 thru 0x02bcba (0x9868 bytes) 000714 1788527810.254473 trunc from 0x02bcbb to 0x038906 (0xcc4c bytes) 000715 1788527810.280278 mapwrite 0x03ee4c thru 0x03ffff (0x11b4 bytes) 000716 1788527810.337254 trunc from 0x017659 to 0x01be48 (0x47f0 bytes) 000717 1788527810.342626 trunc from 0x010a78 to 0x018be6 (0x816f bytes) 000718 1788527810.347853 write 0x02879d thru 0x03083b (0x809f bytes) 000720 1788527810.357505 mapwrite 0x02a903 thru 0x038720 (0xde1e bytes) 000721 1788527810.420375 fallocate 0x02eb32 thru 0x0326ac (0x3b7b bytes) 000722 1788527810.425239 fallocate 0x020126 thru 0x02189d (0x1778 bytes) 000723 1788527810.429592 write 0x0067c4 thru 0x00ebea (0x8427 bytes) 000724 1788527810.430735 trunc from 0x02d3ef to 0x037be3 (0xa7f5 bytes) 000726 1788527810.436452 punch from 0x0327d6 to 0x0366b9 (0x3ee4 bytes) 000727 1788527810.440901 read 0x006eda thru 0x00c0c5 (0x51ec bytes) 000728 1788527810.441565 punch from 0x02c9ca to 0x0326aa (0x5ce1 bytes) 000729 1788527810.445896 mapread 0x036220 thru 0x03ffff (0x9de0 bytes) 000730 1788527810.459404 mapwrite 0x012183 thru 0x018d95 (0x6c13 bytes) 000732 1788527810.476394 punch from 0x03fe44 to 0x03ffff (0x1bc bytes) 000733 1788527810.482075 fallocate 0x0122b2 thru 0x01bf7e (0x9ccd bytes) 000734 1788527810.488616 mapwrite 0x00211b thru 0x011bf0 (0xfad6 bytes) 000735 1788527810.507229 trunc from 0x00030c to 0x007231 (0x6f26 bytes) 000737 1788527810.513339 write 0x00c3a1 thru 0x012e33 (0x6a93 bytes) 000738 1788527810.514588 read 0x0062c8 thru 0x010c8d (0xa9c6 bytes) 000739 1788527810.523153 punch from 0x03c827 to 0x03ffff (0x37d9 bytes) 000740 1788527810.527504 trunc from 0x040000 to 0x01f1da (0x20e25 bytes) 000741 1788527810.540494 fallocate 0x00c2b1 thru 0x0102e0 (0x4030 bytes) 000743 1788527810.548472 mapwrite 0x0041ba thru 0x0104d7 (0xc31e bytes) 000744 1788527810.567907 punch from 0x01bdb4 to 0x01f1da (0x3427 bytes) 000745 1788527810.573648 trunc from 0x01c048 to 0x01e397 (0x2350 bytes) 000746 1788527810.577471 read 0x01da02 thru 0x01e17a (0x779 bytes) 000747 1788527810.586571 mapread 0x010cbb thru 0x01c94a (0xbc90 bytes) 000748 1788527810.587714 read 0x00e201 thru 0x019786 (0xb586 bytes) 000749 1788527810.589377 mapwrite 0x026cc0 thru 0x031df4 (0xb135 bytes) 000750 1788527810.628018 trunc from 0x031df5 to 0x039cdf (0x7eeb bytes) 000751 1788527810.644855 mapread 0x019f34 thru 0x02417d (0xa24a bytes) 000752 1788527810.645786 fallocate 0x0394d7 thru 0x03ffff (0x6b29 bytes) 000753 1788527810.651995 trunc from 0x034215 to 0x03ffff (0xbdeb bytes) 000754 1788527810.658261 mapwrite 0x03bb0b thru 0x03ffff (0x44f5 bytes) 000755 1788527810.679782 mapwrite 0x00d586 thru 0x00db0b (0x586 bytes) 000756 1788527810.694200 punch from 0x019173 to 0x023fda (0xae68 bytes) 000757 1788527810.700203 read 0x032694 thru 0x0356ff (0x306c bytes) 000758 1788527810.700976 punch from 0x00fa75 to 0x01d2b4 (0xd840 bytes) 000759 1788527810.706525 mapread 0x03d460 thru 0x03ffff (0x2ba0 bytes) 000760 1788527810.708401 mapwrite 0x014a78 thru 0x019a3e (0x4fc7 bytes) 000761 1788527810.745304 mapwrite 0x01fb65 thru 0x02edf3 (0xf28f bytes) 000764 1788527810.814500 MIRROR_EXTEND to 2 mirrors 000765 1788527811.633464 write 0x019887 thru 0x01f988 (0x6102 bytes) 000766 1788527811.686269 punch from 0x009989 to 0x017789 (0xde01 bytes) 000767 1788527811.695871 MIRROR_RESYNC 2 mirrors 000768 1788527812.294509 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000770 1788527812.426331 trunc from 0x02d78c to 0x035ed6 (0x874b bytes) 000772 1788527812.442887 trunc from 0x031145 to 0x03f949 (0xe805 bytes) 000773 1788527812.448192 write 0x028385 thru 0x029186 (0xe02 bytes) 000775 1788527812.464427 mapread 0x0127d3 thru 0x019209 (0x6a37 bytes) 000776 1788527812.485985 MIRROR_EXTEND to 2 mirrors 000777 1788527813.118747 trunc from 0x009913 to 0x01370e (0x9dfc bytes) 000779 1788527813.161321 read 0x031c10 thru 0x03ecec (0xd0dd bytes) 000780 1788527813.171544 trunc from 0x00e383 to 0x019c23 (0xb8a1 bytes) 000781 1788527813.177229 read 0x01471e thru 0x0155ce (0xeb1 bytes) 000782 1788527813.191108 mapread 0x01ac25 thru 0x0276a9 (0xca85 bytes) 000783 1788527813.192518 mapread 0x030ab9 thru 0x037d98 (0x72e0 bytes) 000784 1788527813.196931 read 0x02d3c0 thru 0x02dc39 (0x87a bytes) 000785 1788527813.199462 trunc from 0x038c2c to 0x03aa92 (0x1e67 bytes) 000786 1788527813.209486 trunc from 0x040000 to 0x0186f0 (0x2790f bytes) 000787 1788527813.269336 trunc from 0x0186f1 to 0x01de10 (0x5720 bytes) 000788 1788527813.287710 trunc from 0x01de11 to 0x01a59e (0x3872 bytes) 000789 1788527813.315225 mapread 0x007b2a thru 0x014159 (0xc630 bytes) 000790 1788527813.316250 write 0x01c21b thru 0x0230f2 (0x6ed8 bytes) 000791 1788527813.320971 mapwrite 0x00d353 thru 0x0141ec (0x6e9a bytes) 000793 1788527813.349189 punch from 0x00691c to 0x01506b (0xe750 bytes) 000794 1788527813.357592 read 0x007a57 thru 0x00a37c (0x2926 bytes) 000795 1788527813.368905 mapwrite 0x0105b9 thru 0x0134f8 (0x2f40 bytes) 000796 1788527813.391421 write 0x03a992 thru 0x03f79b (0x4e0a bytes) 000797 1788527813.395332 mapwrite 0x033cc3 thru 0x03ffff (0xc33d bytes) 000798 1788527813.479279 mapread 0x02edc1 thru 0x03b7c8 (0xca08 bytes) 000800 1788527813.487923 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000801 1788527813.629866 trunc from 0x036a56 to 0x037704 (0xcaf bytes) 000803 1788527813.635901 mapwrite 0x00d3e0 thru 0x013895 (0x64b6 bytes) 000804 1788527813.675658 mapread 0x00901a thru 0x018bf5 (0xfbdc bytes) 000806 1788527813.683448 read 0x00aca6 thru 0x00bc62 (0xfbd bytes) 000807 1788527813.684161 trunc from 0x004872 to 0x00a641 (0x5dd0 bytes) 000808 1788527813.689096 trunc from 0x035f58 to 0x03c633 (0x66dc bytes) 000809 1788527813.694271 fallocate 0x02f263 thru 0x03a158 (0xaef6 bytes) 000810 1788527813.698291 mapwrite 0x00334b thru 0x00c847 (0x94fd bytes) 000811 1788527813.724900 punch from 0x034491 to 0x03a9ae (0x651e bytes) 000812 1788527813.730299 punch from 0x03f608 to 0x03ffff (0x9f8 bytes) 000813 1788527813.735476 trunc from 0x02d5e4 to 0x0394bf (0xbedc bytes) 000814 1788527813.740603 mapwrite 0x024bd2 thru 0x0265c5 (0x19f4 bytes) 000815 1788527813.761200 mapread 0x016632 thru 0x01b068 (0x4a37 bytes) 000816 1788527813.762492 write 0x03a444 thru 0x03ffff (0x5bbc bytes) 000817 1788527813.773374 trunc from 0x02d550 to 0x039130 (0xbbe1 bytes) 000818 1788527813.781529 write 0x0253e4 thru 0x02c95c (0x7579 bytes) 000819 1788527813.785601 read 0x008801 thru 0x00bdc1 (0x35c1 bytes) 000820 1788527813.792575 MIRROR_EXTEND to 2 mirrors 000821 1788527814.400357 mapwrite 0x01eb2a thru 0x01ff4e (0x1425 bytes) 000822 1788527814.485531 mapread 0x02b879 thru 0x03a551 (0xecd9 bytes) 000823 1788527814.505322 trunc from 0x040000 to 0x029785 (0x1687a bytes) 000824 1788527814.535551 mapread 0x0278ab thru 0x029785 (0x1edb bytes) 000825 1788527814.537483 trunc from 0x00e36f to 0x018115 (0x9da7 bytes) 000826 1788527814.544861 punch from 0x016ed8 to 0x01cb34 (0x5c5d bytes) 000827 1788527814.550331 mapwrite 0x02c317 thru 0x035a02 (0x96ec bytes) 000828 1788527814.601500 punch from 0x0110b6 to 0x01d651 (0xc59c bytes) 000829 1788527814.613300 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000830 1788527814.744356 mapwrite 0x02d02d thru 0x02e27d (0x1251 bytes) 000831 1788527814.772495 mapread 0x024643 thru 0x02c42d (0x7deb bytes) 000832 1788527814.796650 mapread 0x006117 thru 0x0092c5 (0x31af bytes) 000833 1788527814.805582 fallocate 0x029a49 thru 0x036384 (0xc93c bytes) 000834 1788527814.809842 trunc from 0x02641b to 0x0345a1 (0xe187 bytes) 000836 1788527814.815181 mapwrite 0x01822f thru 0x01e2ce (0x60a0 bytes) 000837 1788527814.836447 trunc from 0x035a03 to 0x025415 (0x105ed bytes) 000840 1788527814.854616 write 0x0179bc thru 0x02417d (0xc7c2 bytes) 000841 1788527814.860871 MIRROR_EXTEND to 2 mirrors 000842 1788527815.466186 trunc from 0x025416 to 0x015b3f (0xf8d6 bytes) 000843 1788527815.552448 write 0x027e9d thru 0x033ae2 (0xbc46 bytes) 000844 1788527815.558457 punch from 0x0159e3 to 0x01a2f1 (0x490f bytes) 000845 1788527815.567265 fallocate 0x00a2b0 thru 0x00a2b7 (0x8 bytes) 000846 1788527815.578484 trunc from 0x028c1f to 0x02d29d (0x467f bytes) 000847 1788527815.597864 read 0x02f8e9 thru 0x033ae2 (0x41fa bytes) 000848 1788527815.598928 trunc from 0x033ae3 to 0x023138 (0x109aa bytes) 000849 1788527815.628272 trunc from 0x007bff to 0x016acf (0xeed1 bytes) 000850 1788527815.637048 punch from 0x01a22c to 0x02019f (0x5f74 bytes) 000851 1788527815.651598 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000852 1788527815.806397 trunc from 0x0211b7 to 0x023138 (0x1f82 bytes) 000853 1788527815.814284 trunc from 0x023139 to 0x01ef65 (0x41d3 bytes) 000855 1788527815.842799 mapwrite 0x02a119 thru 0x02a614 (0x4fc bytes) 000856 1788527815.874931 read 0x01ce74 thru 0x022f55 (0x60e2 bytes) 000857 1788527815.886097 read 0x026432 thru 0x02a614 (0x41e3 bytes) 000858 1788527815.895919 trunc from 0x02a615 to 0x01451b (0x160f9 bytes) 000860 1788527815.913000 write 0x03e26d thru 0x03ead0 (0x864 bytes) 000862 1788527815.917847 mapread 0x017b02 thru 0x01b1ab (0x36aa bytes) 000863 1788527815.955722 read 0x01fc8f thru 0x02297f (0x2cf1 bytes) 000864 1788527815.963381 trunc from 0x03ba7b to 0x03ead0 (0x3056 bytes) 000865 1788527815.980088 trunc from 0x03ead1 to 0x011520 (0x2d5b0 bytes) 000866 1788527816.004159 mapwrite 0x005d5a thru 0x0088ce (0x2b75 bytes) 000867 1788527816.023754 mapread 0x0066ed thru 0x00ed73 (0x8687 bytes) 000869 1788527816.025449 trunc from 0x011521 to 0x00097f (0x10ba1 bytes) 000870 1788527816.045823 MIRROR_EXTEND to 2 mirrors 000871 1788527816.639688 read 0x000303 thru 0x00097f (0x67d bytes) 000872 1788527816.646190 trunc from 0x0004f7 to 0x00097f (0x489 bytes) 000874 1788527816.689219 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000876 1788527816.860822 write 0x0056e0 thru 0x00ccde (0x75ff bytes) 000877 1788527816.869366 mapread 0x00bbfa thru 0x00ccde (0x10e5 bytes) 000878 1788527816.871256 fallocate 0x00f0c8 thru 0x019021 (0x9f5a bytes) 000879 1788527816.881004 mapwrite 0x036e4e thru 0x036f66 (0x119 bytes) 000880 1788527816.917101 trunc from 0x036f67 to 0x00998e (0x2d5d8 bytes) 000881 1788527816.934286 write 0x003895 thru 0x0039d8 (0x144 bytes) 000882 1788527816.943313 fallocate 0x027b01 thru 0x02fb5e (0x805e bytes) 000884 1788527816.951580 punch from 0x0031a5 to 0x0088ea (0x5746 bytes) 000885 1788527816.964984 fallocate 0x02da48 thru 0x0358bf (0x7e78 bytes) 000886 1788527816.971868 fallocate 0x0132b4 thru 0x018a1b (0x5768 bytes) 000887 1788527816.977220 trunc from 0x0222f4 to 0x029f6a (0x7c77 bytes) 000888 1788527816.990573 mapread 0x014377 thru 0x02049c (0xc126 bytes) 000889 1788527817.030573 read 0x0034df thru 0x00bab1 (0x85d3 bytes) 000890 1788527817.040955 fallocate 0x0042b3 thru 0x00622c (0x1f7a bytes) 000892 1788527817.055947 mapwrite 0x034fb5 thru 0x03ac0a (0x5c56 bytes) 000893 1788527817.142966 write 0x032e84 thru 0x03ffff (0xd17c bytes) 000894 1788527817.144533 mapread 0x02838e thru 0x036174 (0xdde7 bytes) 000895 1788527817.155091 mapread 0x02d481 thru 0x03a4e3 (0xd063 bytes) 000896 1788527817.156473 fallocate 0x022cfd thru 0x02b125 (0x8429 bytes) 000897 1788527817.161539 mapwrite 0x00114c thru 0x003f10 (0x2dc5 bytes) 000898 1788527817.192887 punch from 0x004254 to 0x012519 (0xe2c6 bytes) 000899 1788527817.197869 punch from 0x0174c9 to 0x01a5ed (0x3125 bytes) 000900 1788527817.202484 write 0x01e614 thru 0x0229b8 (0x43a5 bytes) 000901 1788527817.203589 punch from 0x0320b0 to 0x03ffff (0xdf50 bytes) 000902 1788527817.212742 fallocate 0x012a9d thru 0x01e507 (0xba6b bytes) 000903 1788527817.219407 fallocate 0x006015 thru 0x009635 (0x3621 bytes) 000904 1788527817.225354 trunc from 0x01aeb0 to 0x01f5f4 (0x4745 bytes) 000905 1788527817.249428 read 0x006558 thru 0x00a29d (0x3d46 bytes) 000906 1788527817.290959 read 0x03fc3a thru 0x03ffff (0x3c6 bytes) 000907 1788527817.291801 read 0x01e44e thru 0x0251c7 (0x6d7a bytes) 000908 1788527817.292371 mapwrite 0x0347c8 thru 0x03faef (0xb328 bytes) 000909 1788527817.323514 read 0x013dd5 thru 0x0167c3 (0x29ef bytes) 000910 1788527817.324359 trunc from 0x040000 to 0x0252e2 (0x1ad1d bytes) 000911 1788527817.337773 mapread 0x0118a5 thru 0x020c5a (0xf3b6 bytes) 000914 1788527817.339687 punch from 0x014003 to 0x01602e (0x202c bytes) 000915 1788527817.345064 mapread 0x013450 thru 0x01d7f9 (0xa3aa bytes) 000916 1788527817.351208 read 0x00d71f thru 0x01257b (0x4e5d bytes) 000917 1788527817.351992 mapread 0x003c4b thru 0x005c25 (0x1fdb bytes) 000920 1788527817.353977 trunc from 0x0202c2 to 0x0252e2 (0x5021 bytes) 000922 1788527817.360434 mapread 0x00e0c1 thru 0x00e858 (0x798 bytes) 000923 1788527817.361535 trunc from 0x00d83f to 0x0188bf (0xb081 bytes) 000924 1788527817.366183 trunc from 0x01957d to 0x0252e2 (0xbd66 bytes) 000925 1788527817.371552 read 0x0168aa thru 0x0252e2 (0xea39 bytes) 000926 1788527817.389044 fallocate 0x020329 thru 0x0238d0 (0x35a8 bytes) 000927 1788527817.397567 MIRROR_EXTEND to 2 mirrors 000928 1788527818.013539 trunc from 0x004439 to 0x00bf94 (0x7b5c bytes) 000929 1788527818.066495 fallocate 0x02528f thru 0x02c7b8 (0x752a bytes) 000930 1788527818.084331 trunc from 0x016b98 to 0x017129 (0x592 bytes) 000931 1788527818.090562 fallocate 0x02ce7f thru 0x035679 (0x87fb bytes) 000932 1788527818.102353 write 0x0036a0 thru 0x00cbf8 (0x9559 bytes) 000933 1788527818.132030 MIRROR_RESYNC 2 mirrors 000934 1788527818.786243 fallocate 0x032f39 thru 0x035c93 (0x2d5b bytes) 000935 1788527818.837459 fallocate 0x00dc89 thru 0x01182d (0x3ba5 bytes) 000936 1788527818.842801 write 0x025544 thru 0x027345 (0x1e02 bytes) 000937 1788527818.873091 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000938 1788527819.054330 read 0x01ec3b thru 0x027345 (0x870b bytes) 000939 1788527819.067628 read 0x0059ee thru 0x010753 (0xad66 bytes) 000940 1788527819.076860 read 0x00263a thru 0x00e397 (0xbd5e bytes) 000941 1788527819.077908 trunc from 0x027346 to 0x028e12 (0x1acd bytes) 000942 1788527819.095415 fallocate 0x00e344 thru 0x00e992 (0x64f bytes) 000944 1788527819.100414 mapwrite 0x02611e thru 0x02a499 (0x437c bytes) 000945 1788527819.131996 trunc from 0x02a49a to 0x004701 (0x25d98 bytes) 000946 1788527819.150071 write 0x0209a4 thru 0x022b7e (0x21db bytes) 000947 1788527819.152258 mapwrite 0x00eb28 thru 0x011179 (0x2652 bytes) 000948 1788527819.181944 trunc from 0x002050 to 0x002bd6 (0xb87 bytes) 000949 1788527819.185940 write 0x011d14 thru 0x0194aa (0x7797 bytes) 000950 1788527819.190993 trunc from 0x022b7f to 0x025696 (0x2b18 bytes) 000951 1788527819.211023 write 0x024c33 thru 0x028c29 (0x3ff7 bytes) 000952 1788527819.215635 write 0x0063ba thru 0x00e752 (0x8399 bytes) 000954 1788527819.222314 trunc from 0x0035a4 to 0x008130 (0x4b8d bytes) 000955 1788527819.235300 fallocate 0x0280d7 thru 0x02fcc4 (0x7bee bytes) 000956 1788527819.239351 fallocate 0x02024b thru 0x0260dc (0x5e92 bytes) 000957 1788527819.243974 write 0x0201ef thru 0x023c9c (0x3aae bytes) 000958 1788527819.249276 write 0x000927 thru 0x008641 (0x7d1b bytes) 000959 1788527819.254332 read 0x0124d4 thru 0x01446e (0x1f9b bytes) 000960 1788527819.255362 write 0x0173af thru 0x01b11f (0x3d71 bytes) 000961 1788527819.260001 trunc from 0x008326 to 0x015167 (0xce42 bytes) 000962 1788527819.276336 punch from 0x00bb82 to 0x01a640 (0xeabf bytes) 000963 1788527819.281044 trunc from 0x028c2a to 0x03564d (0xca24 bytes) 000964 1788527819.293295 mapread 0x001b92 thru 0x0085a8 (0x6a17 bytes) 000965 1788527819.304853 fallocate 0x011ea8 thru 0x012d0e (0xe67 bytes) 000966 1788527819.309965 trunc from 0x029cf3 to 0x03564d (0xb95b bytes) 000968 1788527819.314943 mapread 0x003c31 thru 0x012634 (0xea04 bytes) 000969 1788527819.367539 punch from 0x0299b5 to 0x03564d (0xbc99 bytes) 000970 1788527819.376353 read 0x016f21 thru 0x020cd7 (0x9db7 bytes) 000971 1788527819.409525 read 0x022c12 thru 0x0276e1 (0x4ad0 bytes) 000972 1788527819.410182 read 0x0324b5 thru 0x03564d (0x3199 bytes) 000973 1788527819.416845 trunc from 0x03564e to 0x02dcef (0x795e bytes) 000975 1788527819.443203 write 0x00743d thru 0x0123b3 (0xaf77 bytes) 000977 1788527819.450520 mapread 0x01831d thru 0x018342 (0x26 bytes) 000978 1788527819.452337 fallocate 0x021fef thru 0x02f338 (0xd34a bytes) 000979 1788527819.465812 mapread 0x00eb5f thru 0x01bf9f (0xd441 bytes) 000980 1788527819.473565 fallocate 0x02986f thru 0x035b2b (0xc2bd bytes) 000981 1788527819.485298 punch from 0x00123c to 0x00d996 (0xc75b bytes) 000982 1788527819.503649 fallocate 0x009110 thru 0x018345 (0xf236 bytes) 000983 1788527819.511815 fallocate 0x0203c8 thru 0x02d3ed (0xd026 bytes) 000984 1788527819.519358 fallocate 0x02e0cb thru 0x034089 (0x5fbf bytes) 000985 1788527819.525194 read 0x017781 thru 0x01e096 (0x6916 bytes) 000986 1788527819.525888 punch from 0x01deb0 to 0x027e1b (0x9f6c bytes) 000988 1788527819.541906 fallocate 0x02b012 thru 0x0395e6 (0xe5d5 bytes) 000989 1788527819.547286 fallocate 0x025282 thru 0x033d39 (0xeab8 bytes) 000990 1788527819.556781 write 0x036bc0 thru 0x03ffff (0x9440 bytes) 000992 1788527819.559740 punch from 0x03486b to 0x037834 (0x2fca bytes) 000993 1788527819.586516 write 0x0273a4 thru 0x033927 (0xc584 bytes) 000994 1788527819.599487 trunc from 0x03279f to 0x036e3c (0x469e bytes) 000995 1788527819.618937 trunc from 0x040000 to 0x005cc8 (0x3a337 bytes) 000998 1788527819.640343 read 0x0000ad thru 0x003227 (0x317b bytes) 000999 1788527819.644147 fallocate 0x011739 thru 0x0159a0 (0x4268 bytes) 001000 1788527819.655413 mapwrite 0x0098f1 thru 0x01921e (0xf92e bytes) All operations completed A-OK! PASS 70a (69s) == sanity-flr test 71: check mirror extend parent fid ==== 09:17:05 (1788527825) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1f0:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 09:17:15 (1788527835) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.222729 s, 9.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924998 s, 11.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 09:17:24 (1788527844) Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5909152 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8353408 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6535744 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6675296 .. Extending file size to 8346720 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9364832 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8687168 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8925568 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 61384 61385 61386 61387 61388 192.168.202.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg259-client.virtnet /mnt/lustre2 (opts:) 192.168.202.159@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg259-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:18:44 (1788527924) Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5898336 .. Extending file size to 5974880 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6257248 .. Extending file size to 7696832 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7008544 .. Extending file size to 7799840 .. Extending file size to 7990976 .. Extending file size to 8945568 .. Extending file size to 9107872 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 62836 62837 62838 62840 62844 failed 192.168.202.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg259-client.virtnet /mnt/lustre2 (opts:) 192.168.202.159@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg259-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:20:06 (1788528006) Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2204 1269100 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18820 3588200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24528 3582492 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43348 7170692 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2204 1269100 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18820 3588200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24528 3582492 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43348 7170692 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.179078 s, 11.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197121 s, 10.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x15c04889 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg259-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:20:26 (1788528026) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101803 s, 20.6 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 09:20:35 (1788528035) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:20:43 (1788528043) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0569029 s, 36.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x257:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x25a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x25b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:20:50 (1788528050) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0776597 s, 27.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:20:57 (1788528057) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142994 s, 14.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:21:33 (1788528093) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg259-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136269 s, 7.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } PASS 204d (51s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:22:24 (1788528144) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg259-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164574 s, 12.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (53s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:23:17 (1788528197) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg259-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg259-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg259-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg259-server: Waiting 90s for '' Adding targets to pool oleg259-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg259-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg259-server: Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0964078 s, 21.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg259-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg259-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg259-server: oleg259-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg259-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg259-server: oleg259-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg259-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg259-server: Waiting 90s for 'foo' Updated after 2s: want 'foo' got 'foo' PASS 204f (72s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:24:29 (1788528269) PASS 205a (13s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:24:43 (1788528283) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:24:51 (1788528291) oleg259-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg259-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg259-server: Waiting 90s for '' Adding targets to pool oleg259-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg259-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg259-server: Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg259-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg259-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg259-server: oleg259-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg259-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg259-server: oleg259-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg259-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg259-server: Waiting 90s for 'foo' Updated after 2s: want 'foo' got 'foo' PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:25:16 (1788528316) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633256 s, 16.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238101 s, 44.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb06f53c5 Mirror 3: 0xb06f53c5 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:25:25 (1788528325) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:25:30 (1788528330) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:25:34 (1788528334) Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.093294 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x950000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x10f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x10f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x10f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x930000 Read this (4 bytes): BBBB 192.168.202.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg259-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:26:13 (1788528373) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0353295 s, 232 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:26:21 (1788528381) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059336 s, 17.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 09:26:22.000000000 -0400 Modify: 2026-09-04 09:26:22.000000000 -0400 Change: 2026-09-04 09:26:24.000000000 -0400 Birth: 2026-09-04 09:26:21.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:26:29 (1788528389) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434505 s, 24.1 MB/s oleg259-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg259-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg259-server: O/280000401/d29/13d: File not found by ext2_lookup PASS 210b (18s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:26:47 (1788528407) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.194918 s, 210 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0192175 s, 213 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 4 09:26 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 4 09:26 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:26:55 (1788528415) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:27:02 (1788528422) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1733 sec =========== 09:27:10 (1788528430) === sanity-flr: start cleanup 09:27:11 (1788528431) === === sanity-flr: finish cleanup 09:27:18 (1788528438) ===