-----============= acceptance-small: sanity-flr ============----- Wed Aug 13 10:40:06 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg214-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 10:40:15 (1755096015) === oleg214-client.virtnet: executing check_config_client /mnt/lustre oleg214-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg214-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff993cc7d4c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff993cc7d4c000.idle_timeout=debug disable quota as required oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:40:30 (1755096030) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8190] == sanity-flr test 0a: lfs mirror create with -N option == 10:40:31 (1755096031) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:40:38 (1755096038) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:40:40 (1755096040) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:40:41 (1755096041) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:40:51 (1755096051) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:40:53 (1755096053) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 10:40:54 (1755096054) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff993cc7d4c000.stats=clear osc.lustre-OST0001-osc-ffff993cc7d4c000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:40:58 (1755096058) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:41:02 (1755096062) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:41:07 (1755096067) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:41:08 (1755096068) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 10:41:13 (1755096073) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:41:16 (1755096076) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:41:20 (1755096080) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:41:23 (1755096083) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:41:28 (1755096088) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:41:30 (1755096090) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.370058 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472384 s, 22.2 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:41:34 (1755096094) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0954292 s, 87.9 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:41:38 (1755096098) Stopping /mnt/lustre-ost1 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff993cc7d4c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff993cc7d4c000.ost_server_uuid in DISCONN state after 2 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg214-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff993cc7d4c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff993cc7d4c000.ost_server_uuid in DISCONN state after 2 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg214-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (36s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:42:14 (1755096134) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0537097 s, 39.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff993cc7d4c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff993cc7d4c000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg214-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (17s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:42:31 (1755096151) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff993cc7d4c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff993cc7d4c000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg214-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff993cc7d4c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff993cc7d4c000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg214-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (35s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:43:06 (1755096186) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.10653 s, 47.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 13 10:43 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.351s user 0m0.001s sys 0m0.164s Stopping /mnt/lustre-ost1 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff993cc7d4c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff993cc7d4c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.350s user 0m0.000s sys 0m0.167s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg214-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff993cc7d4c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff993cc7d4c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.589s user 0m0.000s sys 0m0.177s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg214-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg214-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff993cc7d4c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff993cc7d4c000.ost_server_uuid in FULL state after 0 sec PASS 33b (36s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:43:42 (1755096222) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:43:43 (1755096223) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:43:45 (1755096225) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 10:43:47 (1755096227) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 10:43:50 (1755096230) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.293435 s, 42.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:43:57 (1755096237) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162163 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187321 s, 56.0 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:44:00 (1755096240) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:44:04 (1755096244) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00140907 s, 727 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000757079 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000779601 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000684823 s, 1.5 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 10:44:08 (1755096248) create test files with size 15742440 .. 2b838a9089677946649cd50237ed6638 - a1a021544f3e254a45d1a6703942e2ff - 341d55fec788eb0e4124dc753696bc26 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg214-client.virtnet /mnt/lustre (opts:) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre PASS 37 (9s) == sanity-flr test 38: resync ============================ 10:44:17 (1755096257) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:44:22 (1755096262) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:44:25 (1755096265) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381953 s, 27.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174825 s, 60.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 10:44:28 (1755096268) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0370706 s, 56.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.136851 s, 30.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0676391 s, 31.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 10:44:33 (1755096273) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 10:44:34 (1755096274) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:44:35 (1755096275) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130407 s, 80.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 10:44:38 (1755096278) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.10874 s, 28.9 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:44:41 (1755096281) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:44:45 (1755096285) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.108246 s, 96.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg214-client.virtnet /mnt/lustre (opts:) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (18s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:45:03 (1755096303) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff993cc7674000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.117506 s, 89.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg214-client.virtnet /mnt/lustre (opts:) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff993cd19df000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:45:08 (1755096308) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:45:23 (1755096323) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (14s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:45:37 (1755096337) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 10:45:38 (1755096338) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0888294 s, 35.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0662504 s, 47.5 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 16496bdcebf593caaa899360535b51cd - mirror 1 checksum: 043084e5d2c9406b5675488ff7e730ab - mirror 2 checksum: 16496bdcebf593caaa899360535b51cd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Aug 13 10:45:40 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Aug 13 10:45:40 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 16496bdcebf593caaa899360535b51cd - mirror 1 checksum: 043084e5d2c9406b5675488ff7e730ab - mirror 2 checksum: 043084e5d2c9406b5675488ff7e730ab - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Aug 13 10:45:40 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Aug 13 10:45:40 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 10:45:42 (1755096342) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114489 s, 91.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:45:45 (1755096345) Failing mds1 on oleg214-server Stopping /mnt/lustre-mds1 (opts:) on oleg214-server 10:45:54 (1755096354) shut down facet: mds1 facet_host: oleg214-server facet_failover_host: oleg214-server Failover mds1 to oleg214-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 10:46:09 (1755096369) targets are mounted 10:46:09 (1755096369) facet_failover done oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121126 s, 86.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } PASS 50A (32s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:46:17 (1755096377) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0141104 s, 1.5 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00639439 s, 3.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:46:21 (1755096381) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0158609 s, 1.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00460998 s, 4.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:46:24 (1755096384) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00442983 s, 3.7 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:46:27 (1755096387) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0139191 s, 1.5 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:46:30 (1755096390) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00296447 s, 1.4 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0135651 s, 302 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:46:32 (1755096392) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) PASS 60b (7s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:46:39 (1755096399) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:46:59 (1755096419) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x49:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:47:18 (1755096438) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:47:36 (1755096456) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 10:47:40 (1755096460) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 55858 mirror split pid 55859 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 55858 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 55859 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 10:48:42 (1755096522) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 6523 000003 1755096523.508488 write 0x03238e thru 0x035ec3 (0x3b36 bytes) 000004 1755096523.511839 mapwrite 0x039a10 thru 0x03ffff (0x65f0 bytes) 000005 1755096523.544904 punch from 0x03f9fc to 0x03ffff (0x604 bytes) zero_range to largest ever: 0x12beb 000006 1755096523.547412 trunc from 0x00d6e6 to 0x012bea (0x5505 bytes) 000007 1755096523.553602 MIRROR_EXTEND to 2 mirrors 000008 1755096523.751059 punch from 0x014a3d to 0x01bc15 (0x71d9 bytes) 000009 1755096523.763877 read 0x011666 thru 0x019a1f (0x83ba bytes) 000010 1755096523.767455 mapread 0x023700 thru 0x023c3a (0x53b bytes) 000011 1755096523.770434 write 0x01384c thru 0x01bbfe (0x83b3 bytes) 000012 1755096523.775329 mapwrite 0x01ffe2 thru 0x023009 (0x3028 bytes) 000013 1755096523.787277 mapread 0x036e3e thru 0x03b38b (0x454e bytes) 000014 1755096523.792018 mapwrite 0x018ad0 thru 0x028394 (0xf8c5 bytes) 000015 1755096523.807170 write 0x0146a3 thru 0x021e3d (0xd79b bytes) 000016 1755096523.808660 mapwrite 0x03a4bc thru 0x03ffff (0x5b44 bytes) fallocating to largest ever: 0x1f314 000017 1755096523.819488 fallocate 0x0132d2 thru 0x01f313 (0xc042 bytes) 000018 1755096523.822468 trunc from 0x00ea17 to 0x01c7b0 (0xdd9a bytes) 000019 1755096523.825240 mapread 0x00279e thru 0x005b2e (0x3391 bytes) 000020 1755096523.831833 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) truncating to largest ever: 0x3b0fa 000021 1755096523.882695 trunc from 0x040000 to 0x03b0f9 (0x4f06 bytes) 000022 1755096523.894304 read 0x01518c thru 0x01ef74 (0x9de9 bytes) 000023 1755096523.897439 write 0x004a4d thru 0x00abbe (0x6172 bytes) 000024 1755096523.901393 fallocate 0x032af9 thru 0x038a03 (0x5f0b bytes) 000025 1755096523.903444 fallocate 0x002cd1 thru 0x003c4e (0xf7e bytes) 000026 1755096523.907276 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3fb17 000027 1755096524.095813 trunc from 0x03b0fa to 0x03fb16 (0x4a1d bytes) 000028 1755096524.111059 mapread 0x0164d1 thru 0x018ee7 (0x2a17 bytes) 000029 1755096524.121605 MIRROR_EXTEND to 3 mirrors 000030 1755096524.314010 MIRROR_EXTEND to 4 mirrors 000031 1755096524.505782 mapwrite 0x03e422 thru 0x03ffff (0x1bde bytes) 000032 1755096524.542799 trunc from 0x040000 to 0x019fa1 (0x2605e bytes) 000033 1755096524.551167 write 0x011ec6 thru 0x0165af (0x46ea bytes) 000034 1755096524.556064 mapwrite 0x02aee5 thru 0x0348c8 (0x99e4 bytes) 000035 1755096524.581839 fallocate 0x02770b thru 0x0370f2 (0xf9e8 bytes) 000036 1755096524.584696 fallocate 0x0100bb thru 0x01ba57 (0xb99d bytes) 000037 1755096524.587518 trunc from 0x02dd4f to 0x032d22 (0x4fd4 bytes) 000038 1755096524.590297 read 0x002fdb thru 0x00b665 (0x868b bytes) 000040 1755096524.595125 trunc from 0x01f7de to 0x026f2c (0x774f bytes) 000041 1755096524.598237 mapwrite 0x03bfdf thru 0x03ffff (0x4021 bytes) 000042 1755096524.620994 write 0x02d81f thru 0x034c2c (0x740e bytes) 000043 1755096524.625332 mapread 0x03e1b5 thru 0x03ffff (0x1e4b bytes) 000044 1755096524.628397 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000045 1755096524.731544 trunc from 0x02cd2a to 0x03201c (0x52f3 bytes) 000046 1755096524.737895 mapwrite 0x000053 thru 0x000a67 (0xa15 bytes) 000047 1755096524.752792 trunc from 0x040000 to 0x00b6fd (0x34902 bytes) 000048 1755096524.761201 mapread 0x00a13c thru 0x00b6fd (0x15c2 bytes) 000050 1755096524.766732 fallocate 0x00fe2f thru 0x014b12 (0x4ce4 bytes) fallocating to largest ever: 0x40000 000051 1755096524.769618 fallocate 0x03dee5 thru 0x03ffff (0x211b bytes) 000052 1755096524.772404 mapwrite 0x032b75 thru 0x03dd1c (0xb1a8 bytes) 000053 1755096524.788381 read 0x019218 thru 0x0279f2 (0xe7db bytes) 000054 1755096524.792626 write 0x03725a thru 0x03f6e4 (0x848b bytes) 000055 1755096524.793727 write 0x01e778 thru 0x02c244 (0xdacd bytes) 000056 1755096524.794971 trunc from 0x040000 to 0x019a2e (0x265d1 bytes) 000057 1755096524.801405 trunc from 0x011211 to 0x019a2e (0x881e bytes) 000058 1755096524.803691 trunc from 0x000b6b to 0x00c87a (0xbd10 bytes) 000059 1755096524.806351 trunc from 0x019a2f to 0x0167ea (0x3244 bytes) 000060 1755096524.812760 mapread 0x00d9b3 thru 0x013921 (0x5f6f bytes) 000061 1755096524.818535 fallocate 0x01367b thru 0x01dbf8 (0xa57e bytes) 000062 1755096524.821605 trunc from 0x0167eb to 0x01af71 (0x4787 bytes) 000063 1755096524.829932 read 0x0104c6 thru 0x01772d (0x7268 bytes) 000064 1755096524.834557 mapwrite 0x02b2d2 thru 0x032ece (0x7bfd bytes) 000065 1755096524.861384 trunc from 0x032ecf to 0x03070e (0x27c0 bytes) 000066 1755096524.871471 mapwrite 0x006a22 thru 0x015724 (0xed03 bytes) 000067 1755096524.889473 mapwrite 0x0028fa thru 0x010395 (0xda9c bytes) 000068 1755096524.903450 mapwrite 0x03a0f6 thru 0x03e1c0 (0x40cb bytes) 000069 1755096524.939854 mapwrite 0x034d9e thru 0x03ffff (0xb262 bytes) 000070 1755096524.976036 read 0x010978 thru 0x012685 (0x1d0e bytes) 000071 1755096524.976617 trunc from 0x024a42 to 0x0270ad (0x266c bytes) 000072 1755096524.980943 punch from 0x008c29 to 0x015118 (0xc4f0 bytes) 000073 1755096524.986713 trunc from 0x040000 to 0x03c728 (0x38d7 bytes) 000074 1755096524.997598 MIRROR_EXTEND to 4 mirrors 000075 1755096525.315338 trunc from 0x009f1f to 0x012333 (0x8415 bytes) 000076 1755096525.342005 fallocate 0x0199fb thru 0x02229f (0x88a5 bytes) 000077 1755096525.345964 mapread 0x03c43b thru 0x03c728 (0x2ee bytes) 000078 1755096525.349767 punch from 0x011a5a to 0x019de5 (0x838c bytes) 000080 1755096525.353343 mapwrite 0x03e88a thru 0x03ffff (0x1776 bytes) 000081 1755096525.387114 read 0x03fc4f thru 0x03ffff (0x3b1 bytes) 000082 1755096525.387720 trunc from 0x040000 to 0x020ee3 (0x1f11c bytes) 000083 1755096525.401304 mapwrite 0x03942a thru 0x03f5d2 (0x61a9 bytes) 000084 1755096525.426626 trunc from 0x005ca2 to 0x007856 (0x1bb5 bytes) 000085 1755096525.430169 punch from 0x013381 to 0x015878 (0x24f8 bytes) 000086 1755096525.433992 write 0x03a0cd thru 0x03ffff (0x5f33 bytes) 000087 1755096525.437396 MIRROR_RESYNC 4 mirrors 000088 1755096525.938856 punch from 0x01908b to 0x01c337 (0x32ad bytes) 000089 1755096525.959388 write 0x008f3c thru 0x014d3f (0xbe04 bytes) 000090 1755096525.967396 trunc from 0x040000 to 0x0008b8 (0x3f747 bytes) 000091 1755096525.989830 punch from 0x0003f7 to 0x0008b8 (0x4c2 bytes) 000092 1755096525.993153 trunc from 0x00050c to 0x0008b8 (0x3ad bytes) 000093 1755096525.995772 punch from 0x0000b2 to 0x0008b8 (0x807 bytes) 000094 1755096525.998298 mapwrite 0x01969c thru 0x0205c1 (0x6f26 bytes) 000095 1755096526.024598 mapwrite 0x019e8e thru 0x0267c4 (0xc937 bytes) 000096 1755096526.057501 trunc from 0x0267c5 to 0x02abce (0x440a bytes) 000097 1755096526.066496 mapread 0x028d23 thru 0x02abce (0x1eac bytes) 000098 1755096526.072523 punch from 0x0228ba to 0x02abce (0x8315 bytes) 000099 1755096526.078759 write 0x03b105 thru 0x03ffff (0x4efb bytes) 000100 1755096526.079993 trunc from 0x040000 to 0x00cca5 (0x3335a bytes) 000101 1755096526.092161 mapread 0x0037e0 thru 0x00815c (0x497d bytes) 000102 1755096526.099535 fallocate 0x03fb0e thru 0x03ffff (0x4f2 bytes) 000103 1755096526.104033 fallocate 0x014c57 thru 0x0162c2 (0x166c bytes) 000104 1755096526.108580 punch from 0x001789 to 0x006ba8 (0x5420 bytes) 000105 1755096526.112347 fallocate 0x02967d thru 0x02cb0c (0x3490 bytes) 000106 1755096526.116256 trunc from 0x040000 to 0x022ba5 (0x1d45a bytes) 000108 1755096526.126231 write 0x0181ff thru 0x023189 (0xaf8b bytes) 000109 1755096526.132713 mapwrite 0x02694e thru 0x027f45 (0x15f8 bytes) 000111 1755096526.169128 read 0x00d163 thru 0x00ff63 (0x2e01 bytes) 000112 1755096526.172939 mapread 0x006c3e thru 0x0134ad (0xc870 bytes) 000113 1755096526.179140 write 0x036998 thru 0x03ffff (0x9668 bytes) 000114 1755096526.180467 fallocate 0x038a05 thru 0x03d54c (0x4b48 bytes) 000115 1755096526.186823 mapwrite 0x00c643 thru 0x00eff2 (0x29b0 bytes) 000116 1755096526.196706 mapwrite 0x02b310 thru 0x03152b (0x621c bytes) 000117 1755096526.215798 write 0x013911 thru 0x023877 (0xff67 bytes) 000118 1755096526.217502 punch from 0x029319 to 0x02c1a7 (0x2e8f bytes) 000119 1755096526.227110 write 0x0265aa thru 0x034e40 (0xe897 bytes) 000120 1755096526.232465 mapwrite 0x00107c thru 0x00abc6 (0x9b4b bytes) 000121 1755096526.245368 trunc from 0x040000 to 0x0186c7 (0x27938 bytes) 000122 1755096526.257311 read 0x007d7d thru 0x00a04f (0x22d3 bytes) 000124 1755096526.258964 trunc from 0x016c15 to 0x0186c7 (0x1ab3 bytes) 000125 1755096526.262559 punch from 0x00993a to 0x0152f7 (0xb9be bytes) 000126 1755096526.265759 mapwrite 0x03d6f4 thru 0x03ffff (0x290c bytes) 000127 1755096526.293965 fallocate 0x018cd4 thru 0x0225b6 (0x98e3 bytes) 000128 1755096526.299596 MIRROR_RESYNC 4 mirrors 000129 1755096526.825556 write 0x01bf55 thru 0x029662 (0xd70e bytes) 000130 1755096526.848260 punch from 0x02569a to 0x030a3a (0xb3a1 bytes) 000131 1755096526.856966 write 0x02b281 thru 0x034f25 (0x9ca5 bytes) 000132 1755096526.865298 mapread 0x00dd8a thru 0x01b0dc (0xd353 bytes) 000133 1755096526.877325 mapread 0x00197d thru 0x007ce7 (0x636b bytes) 000134 1755096526.883512 write 0x025b22 thru 0x027abb (0x1f9a bytes) 000135 1755096526.884425 trunc from 0x03d3c8 to 0x03ffff (0x2c38 bytes) 000136 1755096526.888350 trunc from 0x040000 to 0x014d93 (0x2b26c bytes) 000139 1755096526.921169 mapwrite 0x01f783 thru 0x022fca (0x3848 bytes) 000140 1755096526.954083 mapread 0x004a4b thru 0x0148a5 (0xfe5b bytes) 000142 1755096526.959136 trunc from 0x0037d7 to 0x00501d (0x1847 bytes) 000143 1755096526.965972 punch from 0x015b2d to 0x01bfb1 (0x6485 bytes) 000144 1755096526.969007 trunc from 0x022fcb to 0x00f8f5 (0x136d5 bytes) 000145 1755096526.980465 fallocate 0x000bc4 thru 0x00bde5 (0xb222 bytes) 000147 1755096526.984500 punch from 0x00a1e8 to 0x00f8f5 (0x570e bytes) 000148 1755096526.988195 read 0x0038a9 thru 0x00f43c (0xbb94 bytes) 000149 1755096526.995825 write 0x00e849 thru 0x01e51a (0xfcd2 bytes) 000150 1755096526.997438 fallocate 0x02e95d thru 0x035411 (0x6ab5 bytes) 000151 1755096527.001988 write 0x0181cb thru 0x02377a (0xb5b0 bytes) 000152 1755096527.010359 MIRROR_EXTEND to 5 mirrors 000153 1755096527.356075 trunc from 0x00c2bb to 0x00cb52 (0x898 bytes) 000154 1755096527.385728 trunc from 0x000562 to 0x006b6f (0x660e bytes) 000155 1755096527.400300 fallocate 0x031205 thru 0x03db2b (0xc927 bytes) 000156 1755096527.406919 mapread 0x007622 thru 0x014dec (0xd7cb bytes) 000157 1755096527.429187 trunc from 0x013e91 to 0x0208f0 (0xca60 bytes) 000158 1755096527.434228 punch from 0x01393e to 0x023274 (0xf937 bytes) 000160 1755096527.438152 trunc from 0x00435d to 0x00dee5 (0x9b89 bytes) 000161 1755096527.441856 trunc from 0x035412 to 0x0367a6 (0x1395 bytes) 000164 1755096527.467317 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000165 1755096527.589344 fallocate 0x01b730 thru 0x026bb0 (0xb481 bytes) 000166 1755096527.593289 read 0x02a5d3 thru 0x0367a6 (0xc1d4 bytes) 000167 1755096527.599296 punch from 0x00f4b3 to 0x017630 (0x817e bytes) 000168 1755096527.604010 mapwrite 0x02d3d2 thru 0x0333c9 (0x5ff8 bytes) 000169 1755096527.627116 mapread 0x007551 thru 0x00c2dc (0x4d8c bytes) 000170 1755096527.645225 punch from 0x0333dd to 0x0367a6 (0x33ca bytes) 000171 1755096527.650008 fallocate 0x013214 thru 0x017e32 (0x4c1f bytes) 000172 1755096527.655419 mapwrite 0x00c2ef thru 0x00cac3 (0x7d5 bytes) 000173 1755096527.664250 punch from 0x034dbc to 0x0367a6 (0x19eb bytes) 000174 1755096527.668514 fallocate 0x010586 thru 0x01c3a2 (0xbe1d bytes) 000176 1755096527.673831 trunc from 0x01d5fc to 0x021e2d (0x4832 bytes) 000177 1755096527.684934 MIRROR_EXTEND to 5 mirrors 000178 1755096528.027404 punch from 0x022c39 to 0x02cef6 (0xa2be bytes) 000179 1755096528.059454 write 0x01bde1 thru 0x0242f9 (0x8519 bytes) 000180 1755096528.076159 trunc from 0x0367a7 to 0x00f091 (0x27715 bytes) 000181 1755096528.111395 write 0x039164 thru 0x03ffff (0x6e9c bytes) 000182 1755096528.112655 punch from 0x00e883 to 0x012e92 (0x4610 bytes) 000183 1755096528.117147 mapwrite 0x01ffdf thru 0x020c0f (0xc31 bytes) 000185 1755096528.136360 punch from 0x01dbb6 to 0x02bb83 (0xdfce bytes) 000187 1755096528.144834 MIRROR_RESYNC 5 mirrors 000188 1755096528.914254 punch from 0x02d8f3 to 0x02f266 (0x1974 bytes) 000189 1755096528.955785 mapwrite 0x02daff thru 0x02ff46 (0x2448 bytes) 000190 1755096528.988232 write 0x01c6b5 thru 0x020073 (0x39bf bytes) 000191 1755096528.995109 mapread 0x022dec thru 0x030058 (0xd26d bytes) 000192 1755096529.000666 mapwrite 0x01324a thru 0x01c646 (0x93fd bytes) 000193 1755096529.026646 fallocate 0x00a02e thru 0x00f134 (0x5107 bytes) 000194 1755096529.033325 MIRROR_EXTEND to 6 mirrors 000195 1755096529.475447 punch from 0x031509 to 0x034850 (0x3348 bytes) 000196 1755096529.505817 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000197 1755096529.647883 trunc from 0x040000 to 0x03a1b2 (0x5e4d bytes) 000198 1755096529.673311 mapread 0x00525e thru 0x007532 (0x22d5 bytes) 000199 1755096529.690536 mapread 0x01d929 thru 0x020a2b (0x3103 bytes) 000201 1755096529.696949 trunc from 0x03a1b3 to 0x03099f (0x9813 bytes) 000202 1755096529.706639 trunc from 0x02526a to 0x03046f (0xb206 bytes) 000203 1755096529.712581 MIRROR_RESYNC 5 mirrors 000204 1755096530.519823 read 0x0066ce thru 0x0146ff (0xe032 bytes) 000205 1755096530.531685 trunc from 0x0309a0 to 0x011f3d (0x1ea62 bytes) 000206 1755096530.556175 read 0x009d11 thru 0x01098d (0x6c7d bytes) 000207 1755096530.561021 trunc from 0x007bf1 to 0x011f3d (0xa34d bytes) 000208 1755096530.567817 MIRROR_EXTEND to 6 mirrors 000209 1755096530.892290 MIRROR_RESYNC 6 mirrors 000211 1755096531.339262 mapwrite 0x01c5fc thru 0x0273bc (0xadc1 bytes) 000213 1755096531.432848 trunc from 0x0273bd to 0x02814c (0xd90 bytes) 000214 1755096531.446088 read 0x01cbaa thru 0x02363f (0x6a96 bytes) 000215 1755096531.446722 mapread 0x017f4a thru 0x01cb1a (0x4bd1 bytes) 000216 1755096531.447572 mapwrite 0x02ad1d thru 0x0340e3 (0x93c7 bytes) 000217 1755096531.490970 trunc from 0x0328f3 to 0x0340e3 (0x17f1 bytes) 000218 1755096531.494579 write 0x02d2bd thru 0x0385a0 (0xb2e4 bytes) 000219 1755096531.496192 punch from 0x019df3 to 0x025e5d (0xc06b bytes) 000220 1755096531.500296 trunc from 0x0385a1 to 0x025168 (0x13438 bytes) 000222 1755096531.511719 mapread 0x000f42 thru 0x01077c (0xf83b bytes) 000223 1755096531.526055 read 0x00eb13 thru 0x01855d (0x9a4b bytes) 000224 1755096531.526706 write 0x01fc42 thru 0x02db8f (0xdf4e bytes) 000225 1755096531.535827 trunc from 0x0207c0 to 0x02adb5 (0xa5f6 bytes) 000226 1755096531.549857 MIRROR_EXTEND to 7 mirrors 000227 1755096531.818207 mapwrite 0x02690f thru 0x035085 (0xe777 bytes) 000228 1755096531.908160 trunc from 0x02e712 to 0x035085 (0x6974 bytes) 000229 1755096531.912775 mapwrite 0x01c8b9 thru 0x024623 (0x7d6b bytes) 000230 1755096531.930684 read 0x01cf26 thru 0x024a02 (0x7add bytes) 000231 1755096531.931315 trunc from 0x035086 to 0x002e03 (0x32282 bytes) 000232 1755096531.946048 mapread 0x0028d7 thru 0x002e03 (0x52d bytes) 000233 1755096531.951111 read 0x001253 thru 0x002e03 (0x1bb1 bytes) 000234 1755096531.951722 trunc from 0x002e04 to 0x0292e5 (0x264e2 bytes) 000235 1755096531.962800 trunc from 0x020a4a to 0x0292e5 (0x889c bytes) 000236 1755096531.966541 write 0x00370d thru 0x00e753 (0xb047 bytes) 000238 1755096531.979124 write 0x022f24 thru 0x0326f6 (0xf7d3 bytes) 000239 1755096531.981502 punch from 0x01818f to 0x025979 (0xd7eb bytes) 000240 1755096531.994580 fallocate 0x02987e thru 0x02f861 (0x5fe4 bytes) 000242 1755096531.998483 read 0x016d60 thru 0x026038 (0xf2d9 bytes) 000243 1755096532.009220 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000244 1755096532.166563 fallocate 0x01c1f0 thru 0x02900b (0xce1c bytes) 000245 1755096532.170081 trunc from 0x0326f7 to 0x0325ac (0x14a bytes) 000246 1755096532.182937 write 0x03a9bf thru 0x03ffff (0x5641 bytes) 000247 1755096532.184383 read 0x02f143 thru 0x033697 (0x4555 bytes) 000248 1755096532.188737 fallocate 0x01e909 thru 0x02ab80 (0xc278 bytes) 000249 1755096532.191834 mapread 0x034d93 thru 0x03ffff (0xb26d bytes) 000250 1755096532.203313 MIRROR_EXTEND to 7 mirrors 000251 1755096532.566109 read 0x01d9a6 thru 0x02843e (0xaa99 bytes) 000252 1755096532.572886 read 0x03a8ed thru 0x03ffff (0x5713 bytes) 000254 1755096532.578818 punch from 0x009180 to 0x010c8f (0x7b10 bytes) 000255 1755096532.601613 MIRROR_EXTEND to 8 mirrors 000257 1755096532.894054 mapread 0x03aa08 thru 0x03abb2 (0x1ab bytes) 000258 1755096532.906277 mapread 0x01c2d5 thru 0x02b881 (0xf5ad bytes) 000259 1755096532.931026 punch from 0x02cc5c to 0x03a3de (0xd783 bytes) 000260 1755096532.951360 punch from 0x011cfa to 0x018673 (0x697a bytes) 000261 1755096532.953624 punch from 0x02fd0c to 0x03e0bd (0xe3b2 bytes) 000262 1755096532.956471 trunc from 0x02e6f6 to 0x034b9e (0x64a9 bytes) 000263 1755096532.964185 MIRROR_RESYNC 8 mirrors 000264 1755096533.999378 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000265 1755096534.169024 write 0x00fa37 thru 0x012efd (0x34c7 bytes) 000266 1755096534.194605 trunc from 0x02110e to 0x023be0 (0x2ad3 bytes) 000267 1755096534.198560 write 0x00b825 thru 0x00ff60 (0x473c bytes) 000268 1755096534.203346 trunc from 0x040000 to 0x0170a8 (0x28f57 bytes) 000269 1755096534.214479 write 0x001b69 thru 0x006187 (0x461f bytes) 000270 1755096534.220533 MIRROR_RESYNC 7 mirrors 000271 1755096534.894841 punch from 0x00fdff to 0x011383 (0x1585 bytes) 000272 1755096534.909542 write 0x02a2eb thru 0x02a4ae (0x1c4 bytes) 000273 1755096534.910693 trunc from 0x005dee to 0x00e320 (0x8533 bytes) 000274 1755096534.913851 write 0x0189bf thru 0x01f40f (0x6a51 bytes) 000275 1755096534.919692 trunc from 0x028a2b to 0x02a4ae (0x1a84 bytes) 000276 1755096534.928057 trunc from 0x004353 to 0x0089a0 (0x464e bytes) 000277 1755096534.931486 fallocate 0x03055b thru 0x039ff7 (0x9a9d bytes) 000278 1755096534.934409 mapread 0x0239a5 thru 0x026fc1 (0x361d bytes) 000279 1755096534.939101 trunc from 0x02a4af to 0x02700b (0x34a3 bytes) 000282 1755096534.950427 mapwrite 0x016929 thru 0x0264b0 (0xfb88 bytes) 000283 1755096534.971006 trunc from 0x00d72e to 0x011be7 (0x44ba bytes) 000284 1755096534.974534 read 0x00dd52 thru 0x018c79 (0xaf28 bytes) 000286 1755096534.979700 write 0x00834f thru 0x00c667 (0x4319 bytes) 000288 1755096534.985224 fallocate 0x034f41 thru 0x037bad (0x2c6d bytes) 000289 1755096534.988677 trunc from 0x037bae to 0x03562b (0x2582 bytes) 000290 1755096534.999554 trunc from 0x03562c to 0x02439d (0x1128e bytes) 000291 1755096535.009787 fallocate 0x036c4e thru 0x03acf6 (0x40a9 bytes) 000292 1755096535.013389 trunc from 0x02439e to 0x028272 (0x3ed5 bytes) 000293 1755096535.022807 punch from 0x009a55 to 0x0117c5 (0x7d71 bytes) 000294 1755096535.033097 mapread 0x016075 thru 0x022063 (0xbfef bytes) 000295 1755096535.037938 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000296 1755096535.131709 punch from 0x00a45a to 0x018a77 (0xe61e bytes) 000297 1755096535.134510 mapwrite 0x01565a thru 0x02143c (0xbde3 bytes) 000298 1755096535.160375 mapread 0x002c8f thru 0x004dbd (0x212f bytes) 000299 1755096535.165535 write 0x035507 thru 0x03ffff (0xaaf9 bytes) 000300 1755096535.169667 punch from 0x010847 to 0x011714 (0xece bytes) 000301 1755096535.172511 fallocate 0x02bd36 thru 0x02daa4 (0x1d6f bytes) 000302 1755096535.175693 trunc from 0x01bb31 to 0x02b771 (0xfc41 bytes) 000303 1755096535.179234 mapwrite 0x009b3e thru 0x0199ec (0xfeaf bytes) 000305 1755096535.198172 MIRROR_EXTEND to 7 mirrors 000306 1755096535.421279 MIRROR_RESYNC 7 mirrors 000307 1755096535.932377 trunc from 0x00b3e3 to 0x015e0a (0xaa28 bytes) 000308 1755096535.950710 write 0x0007f7 thru 0x0101ce (0xf9d8 bytes) 000309 1755096535.956577 mapwrite 0x000831 thru 0x0010be (0x88e bytes) 000310 1755096535.967380 mapread 0x01eea7 thru 0x02d096 (0xe1f0 bytes) 000311 1755096535.977592 mapread 0x024c85 thru 0x02f479 (0xa7f5 bytes) 000312 1755096535.978463 mapwrite 0x024cc6 thru 0x0346a1 (0xf9dc bytes) 000313 1755096535.993643 mapread 0x02ffdd thru 0x030f11 (0xf35 bytes) 000314 1755096535.996848 MIRROR_EXTEND to 8 mirrors 000315 1755096536.247463 trunc from 0x00ee29 to 0x011311 (0x24e9 bytes) 000316 1755096536.264956 trunc from 0x040000 to 0x00131b (0x3ece4 bytes) 000317 1755096536.276692 mapwrite 0x033612 thru 0x034884 (0x1273 bytes) 000318 1755096536.299472 read 0x0191f1 thru 0x01b179 (0x1f89 bytes) 000319 1755096536.302589 read 0x01e2c7 thru 0x02bd4a (0xda84 bytes) 000320 1755096536.307270 punch from 0x0150b7 to 0x01867d (0x35c7 bytes) 000322 1755096536.310463 mapread 0x0281b4 thru 0x02fd33 (0x7b80 bytes) 000323 1755096536.311367 trunc from 0x034885 to 0x02d739 (0x714b bytes) 000324 1755096536.319903 read 0x00c911 thru 0x016545 (0x9c35 bytes) 000325 1755096536.323741 write 0x0276c5 thru 0x02a9bb (0x32f7 bytes) 000326 1755096536.324695 trunc from 0x01ed2f to 0x022f56 (0x4228 bytes) 000327 1755096536.327277 punch from 0x00f09d to 0x01e55c (0xf4c0 bytes) 000328 1755096536.329849 mapread 0x02a0af thru 0x02d739 (0x368b bytes) 000329 1755096536.330591 mapwrite 0x00c798 thru 0x016724 (0x9f8d bytes) 000330 1755096536.360444 MIRROR_EXTEND to 9 mirrors 000331 1755096536.560003 mapwrite 0x00cf22 thru 0x018f9e (0xc07d bytes) 000332 1755096536.611929 trunc from 0x00dcb2 to 0x01d349 (0xf698 bytes) 000334 1755096536.615761 trunc from 0x00e24d to 0x010c99 (0x2a4d bytes) 000335 1755096536.619208 fallocate 0x01545d thru 0x01e571 (0x9115 bytes) 000336 1755096536.622916 trunc from 0x006719 to 0x014a66 (0xe34e bytes) 000337 1755096536.626205 write 0x035771 thru 0x03d84a (0x80da bytes) 000338 1755096536.629735 mapwrite 0x0060b5 thru 0x00ecb0 (0x8bfc bytes) 000339 1755096536.648206 fallocate 0x002576 thru 0x006f05 (0x4990 bytes) 000340 1755096536.651958 trunc from 0x03d84b to 0x006fb3 (0x36897 bytes) 000342 1755096536.673600 punch from 0x006e74 to 0x006fb3 (0x140 bytes) 000343 1755096536.676602 punch from 0x006d05 to 0x006fb3 (0x2af bytes) 000344 1755096536.679341 punch from 0x00379c to 0x006fb3 (0x3818 bytes) 000345 1755096536.682087 fallocate 0x0343c6 thru 0x03ffff (0xbc3a bytes) 000346 1755096536.684794 mapwrite 0x00d703 thru 0x01a5e6 (0xcee4 bytes) 000347 1755096536.708034 MIRROR_RESYNC 9 mirrors 000348 1755096537.690329 fallocate 0x0223f3 thru 0x02e7b8 (0xc3c6 bytes) 000349 1755096537.716088 fallocate 0x0239eb thru 0x031cec (0xe302 bytes) 000351 1755096537.722447 mapwrite 0x037ef1 thru 0x03b974 (0x3a84 bytes) 000352 1755096537.754025 write 0x02d1c2 thru 0x03828e (0xb0cd bytes) 000353 1755096537.760471 mapread 0x015183 thru 0x018ef9 (0x3d77 bytes) 000354 1755096537.774132 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000355 1755096537.899604 mapwrite 0x02b87c thru 0x03ac98 (0xf41d bytes) 000356 1755096537.943625 punch from 0x015763 to 0x0199e5 (0x4283 bytes) 000357 1755096537.958564 MIRROR_RESYNC 8 mirrors 000358 1755096539.347144 mapwrite 0x00c328 thru 0x01b27a (0xef53 bytes) 000359 1755096539.425514 fallocate 0x019323 thru 0x01ce1c (0x3afa bytes) 000360 1755096539.430077 trunc from 0x040000 to 0x03abd8 (0x5427 bytes) 000361 1755096539.444612 write 0x00d332 thru 0x019836 (0xc505 bytes) 000362 1755096539.446021 read 0x03437d thru 0x038337 (0x3fbb bytes) 000363 1755096539.453249 mapread 0x019435 thru 0x02060c (0x71d8 bytes) 000364 1755096539.459230 mapread 0x01b7b9 thru 0x01d341 (0x1b89 bytes) 000366 1755096539.460486 mapwrite 0x0114fb thru 0x019f48 (0x8a4e bytes) 000369 1755096539.475318 punch from 0x00882f to 0x00f16d (0x693f bytes) 000370 1755096539.479596 trunc from 0x03abd9 to 0x025d6d (0x14e6b bytes) 000371 1755096539.491085 MIRROR_RESYNC 8 mirrors 000372 1755096540.482115 trunc from 0x017ee8 to 0x01d894 (0x59ad bytes) 000373 1755096540.501252 write 0x00257c thru 0x00c539 (0x9fbe bytes) 000374 1755096540.507003 write 0x02015d thru 0x02a570 (0xa414 bytes) 000375 1755096540.512023 fallocate 0x0271c1 thru 0x027bc5 (0xa05 bytes) 000376 1755096540.515290 trunc from 0x01726c to 0x0269fb (0xf790 bytes) 000378 1755096540.525782 write 0x009232 thru 0x00bbb0 (0x297f bytes) 000379 1755096540.527757 fallocate 0x018b3a thru 0x01c117 (0x35de bytes) 000380 1755096540.531961 trunc from 0x013f4e to 0x014064 (0x117 bytes) 000381 1755096540.535012 write 0x0204ec thru 0x029211 (0x8d26 bytes) 000382 1755096540.542170 mapread 0x011222 thru 0x01857a (0x7359 bytes) 000384 1755096540.551041 MIRROR_EXTEND to 9 mirrors 000385 1755096540.869939 MIRROR_RESYNC 9 mirrors 000386 1755096541.692372 punch from 0x027086 to 0x027160 (0xdb bytes) 000387 1755096541.722294 punch from 0x01248a to 0x01f8e5 (0xd45c bytes) 000388 1755096541.730174 write 0x02f1ff thru 0x0367bb (0x75bd bytes) 000389 1755096541.733740 mapread 0x0339fe thru 0x0367bb (0x2dbe bytes) 000390 1755096541.735821 mapread 0x007e86 thru 0x00f870 (0x79eb bytes) 000391 1755096541.754624 mapwrite 0x035b28 thru 0x036ac1 (0xf9a bytes) 000392 1755096541.793817 trunc from 0x00d2a1 to 0x019e6f (0xcbcf bytes) 000393 1755096541.801072 fallocate 0x02438d thru 0x028acb (0x473f bytes) 000394 1755096541.807050 trunc from 0x028886 to 0x036ac1 (0xe23c bytes) 000395 1755096541.812488 mapread 0x0143f0 thru 0x01f7f5 (0xb406 bytes) 000396 1755096541.835168 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000397 1755096541.999567 write 0x0168bf thru 0x02057a (0x9cbc bytes) 000399 1755096542.019229 trunc from 0x02db24 to 0x035d5f (0x823c bytes) 000400 1755096542.024327 trunc from 0x00c28b to 0x011c73 (0x59e9 bytes) 000401 1755096542.032187 write 0x036f1d thru 0x03ffff (0x90e3 bytes) 000402 1755096542.043836 fallocate 0x03962b thru 0x03f4b1 (0x5e87 bytes) 000403 1755096542.049548 trunc from 0x040000 to 0x02367b (0x1c984 bytes) 000404 1755096542.070199 mapread 0x01382b thru 0x016f2b (0x3701 bytes) 000405 1755096542.080196 punch from 0x0094df to 0x010226 (0x6d48 bytes) 000406 1755096542.086421 punch from 0x013299 to 0x01c1ea (0x8f52 bytes) 000407 1755096542.097505 read 0x00ac16 thru 0x01894e (0xdd39 bytes) 000408 1755096542.103611 punch from 0x006e6c to 0x00bfe6 (0x517b bytes) 000409 1755096542.108598 mapread 0x0214f8 thru 0x02367b (0x2184 bytes) 000410 1755096542.116719 fallocate 0x010818 thru 0x017de9 (0x75d2 bytes) 000411 1755096542.126017 write 0x0296ce thru 0x036664 (0xcf97 bytes) 000412 1755096542.127648 trunc from 0x036665 to 0x0360c3 (0x5a1 bytes) 000416 1755096542.147370 fallocate 0x00ecee thru 0x016afa (0x7e0d bytes) 000417 1755096542.151761 write 0x02cf77 thru 0x03372f (0x67b9 bytes) 000418 1755096542.152837 write 0x005b05 thru 0x01092b (0xae27 bytes) 000419 1755096542.161172 punch from 0x0042bf to 0x013d92 (0xfad4 bytes) 000420 1755096542.173475 trunc from 0x0360c4 to 0x014bd8 (0x214eb bytes) 000421 1755096542.184336 fallocate 0x009d10 thru 0x019b12 (0xfe03 bytes) 000422 1755096542.188200 punch from 0x004e51 to 0x0114fe (0xc6ae bytes) 000423 1755096542.192022 write 0x008787 thru 0x010704 (0x7f7e bytes) 000424 1755096542.203700 read 0x00a7f9 thru 0x019b12 (0xf31a bytes) 000425 1755096542.211358 fallocate 0x022b29 thru 0x023e65 (0x133d bytes) 000426 1755096542.216195 mapread 0x00f050 thru 0x01884a (0x97fb bytes) 000428 1755096542.225035 MIRROR_EXTEND to 9 mirrors 000429 1755096542.534770 read 0x01ee27 thru 0x023e65 (0x503f bytes) 000430 1755096542.540733 read 0x009a30 thru 0x00f6fa (0x5ccb bytes) 000431 1755096542.548933 mapread 0x009305 thru 0x00adb0 (0x1aac bytes) 000432 1755096542.549563 mapwrite 0x030691 thru 0x036102 (0x5a72 bytes) 000433 1755096542.637551 mapwrite 0x009687 thru 0x012436 (0x8db0 bytes) 000434 1755096542.687034 punch from 0x023be9 to 0x025a0e (0x1e26 bytes) 000435 1755096542.692032 trunc from 0x036103 to 0x0277a6 (0xe95c bytes) 000436 1755096542.711225 MIRROR_EXTEND to 10 mirrors 000438 1755096543.010636 trunc from 0x0277a7 to 0x02df78 (0x67d2 bytes) 000439 1755096543.058079 trunc from 0x015d36 to 0x0160c2 (0x38d bytes) 000440 1755096543.068690 mapwrite 0x031bb5 thru 0x031e8f (0x2db bytes) 000441 1755096543.102037 trunc from 0x031e90 to 0x0138d2 (0x1e5bd bytes) 000442 1755096543.115579 mapwrite 0x0379fe thru 0x03ee46 (0x7449 bytes) 000443 1755096543.156884 trunc from 0x01cbc8 to 0x020c1b (0x4054 bytes) 000444 1755096543.161626 mapread 0x02fef3 thru 0x033c24 (0x3d32 bytes) 000445 1755096543.167436 fallocate 0x0258c7 thru 0x02abff (0x5339 bytes) 000446 1755096543.172001 read 0x02163b thru 0x02aba1 (0x9567 bytes) 000447 1755096543.177088 mapwrite 0x012424 thru 0x019658 (0x7235 bytes) 000448 1755096543.195506 mapread 0x0033df thru 0x01306c (0xfc8e bytes) 000449 1755096543.201315 trunc from 0x01d4e4 to 0x027693 (0xa1b0 bytes) 000451 1755096543.204970 mapwrite 0x02546c thru 0x03187c (0xc411 bytes) 000452 1755096543.226804 fallocate 0x005f33 thru 0x0151ea (0xf2b8 bytes) 000453 1755096543.230195 read 0x026211 thru 0x03137f (0xb16f bytes) 000454 1755096543.230640 trunc from 0x03ee47 to 0x027c09 (0x1723d bytes) 000455 1755096543.244617 MIRROR_EXTEND to 11 mirrors 000456 1755096543.572560 mapwrite 0x03f4a1 thru 0x03ffff (0xb5f bytes) 000457 1755096543.655092 mapwrite 0x017145 thru 0x025d2d (0xebe9 bytes) 000458 1755096543.687920 write 0x02805c thru 0x036e1e (0xedc3 bytes) 000460 1755096543.694690 read 0x002138 thru 0x00614e (0x4017 bytes) 000461 1755096543.701824 fallocate 0x029465 thru 0x0344dc (0xb078 bytes) 000462 1755096543.705578 fallocate 0x02dfd6 thru 0x033904 (0x592f bytes) 000463 1755096543.709005 trunc from 0x040000 to 0x029427 (0x16bd8 bytes) 000464 1755096543.727568 fallocate 0x02105d thru 0x030753 (0xf6f7 bytes) 000465 1755096543.732653 punch from 0x011419 to 0x011d62 (0x94a bytes) 000466 1755096543.737755 trunc from 0x029428 to 0x00f04f (0x1a3d8 bytes) 000467 1755096543.747738 fallocate 0x027aba thru 0x02c964 (0x4eab bytes) 000468 1755096543.751481 punch from 0x0273b5 to 0x02a952 (0x359e bytes) 000470 1755096543.754418 mapread 0x01d1b2 thru 0x02c640 (0xf48f bytes) 000471 1755096543.770928 write 0x002a67 thru 0x009657 (0x6bf1 bytes) 000472 1755096543.778474 punch from 0x029518 to 0x02bfee (0x2ad7 bytes) 000473 1755096543.788194 MIRROR_RESYNC 11 mirrors 000474 1755096544.567171 read 0x00ca56 thru 0x0108ef (0x3e9a bytes) 000475 1755096544.572552 mapread 0x017c54 thru 0x0278ca (0xfc77 bytes) 000476 1755096544.589522 mapwrite 0x011ed6 thru 0x012dec (0xf17 bytes) 000479 1755096544.663120 trunc from 0x021302 to 0x024dba (0x3ab9 bytes) 000481 1755096544.667557 trunc from 0x02c965 to 0x008e24 (0x23b40 bytes) 000483 1755096544.679479 trunc from 0x008e25 to 0x03abcd (0x31da9 bytes) 000485 1755096544.694317 punch from 0x008d24 to 0x00d48c (0x4769 bytes) 000486 1755096544.700906 fallocate 0x024912 thru 0x02c096 (0x7785 bytes) 000487 1755096544.705291 write 0x01db0b thru 0x02d661 (0xfb57 bytes) 000488 1755096544.714142 write 0x005c77 thru 0x006e1e (0x11a8 bytes) 000489 1755096544.719359 trunc from 0x03abce to 0x00eb8e (0x2c03f bytes) 000490 1755096544.727906 trunc from 0x00dd30 to 0x00eb8e (0xe5f bytes) 000491 1755096544.731175 trunc from 0x00eb8f to 0x020800 (0x11c72 bytes) 000492 1755096544.742707 punch from 0x003bfc to 0x008de1 (0x51e6 bytes) 000493 1755096544.750298 write 0x00e489 thru 0x01647f (0x7ff7 bytes) 000495 1755096544.761965 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000496 1755096544.914465 trunc from 0x018036 to 0x020800 (0x87cb bytes) 000497 1755096544.923220 mapread 0x010c0a thru 0x012aba (0x1eb1 bytes) 000498 1755096544.945152 trunc from 0x009790 to 0x009d57 (0x5c8 bytes) 000499 1755096544.949789 write 0x022d2e thru 0x02db7b (0xae4e bytes) 000501 1755096544.955706 mapwrite 0x0237a2 thru 0x031499 (0xdcf8 bytes) 000502 1755096545.020645 MIRROR_RESYNC 10 mirrors 000503 1755096546.320895 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000505 1755096546.532624 fallocate 0x029ef7 thru 0x02f4ab (0x55b5 bytes) 000506 1755096546.561256 trunc from 0x03149a to 0x01dc5d (0x1383c bytes) 000507 1755096546.582096 mapread 0x004de3 thru 0x0070aa (0x22c8 bytes) 000508 1755096546.602360 fallocate 0x00f314 thru 0x014317 (0x5004 bytes) 000509 1755096546.610170 read 0x005cd5 thru 0x00b880 (0x5bac bytes) 000510 1755096546.611090 read 0x001c68 thru 0x00b9a9 (0x9d42 bytes) 000511 1755096546.612102 mapread 0x002e94 thru 0x00c852 (0x99bf bytes) 000512 1755096546.613008 trunc from 0x01dc5e to 0x0353b3 (0x17756 bytes) 000513 1755096546.622631 mapread 0x0260c4 thru 0x0294df (0x341c bytes) 000514 1755096546.633360 fallocate 0x03fc11 thru 0x03ffff (0x3ef bytes) 000515 1755096546.638822 trunc from 0x040000 to 0x01c539 (0x23ac6 bytes) 000516 1755096546.653114 MIRROR_RESYNC 9 mirrors 000519 1755096547.571647 mapread 0x01c24b thru 0x01c539 (0x2ef bytes) 000520 1755096547.576465 read 0x014491 thru 0x01c539 (0x80a9 bytes) 000521 1755096547.583013 trunc from 0x01b091 to 0x01c539 (0x14a9 bytes) 000522 1755096547.608277 mapwrite 0x0143ae thru 0x01fd7c (0xb9cf bytes) 000523 1755096547.652532 mapwrite 0x00f826 thru 0x012f84 (0x375f bytes) 000524 1755096547.667157 MIRROR_EXTEND to 10 mirrors 000525 1755096547.961638 write 0x012739 thru 0x01ea60 (0xc328 bytes) 000526 1755096547.992755 read 0x00fe95 thru 0x011605 (0x1771 bytes) 000527 1755096547.997976 trunc from 0x01b294 to 0x01fd7c (0x4ae9 bytes) 000528 1755096548.009267 fallocate 0x03a28a thru 0x03ffff (0x5d76 bytes) 000530 1755096548.013328 write 0x0219f1 thru 0x02825e (0x686e bytes) 000532 1755096548.015304 trunc from 0x02825f to 0x00bdce (0x1c490 bytes) 000533 1755096548.042438 trunc from 0x00bdcf to 0x0229a7 (0x16bd9 bytes) 000534 1755096548.051702 trunc from 0x01370a to 0x01b7ab (0x80a2 bytes) 000535 1755096548.056549 trunc from 0x009c62 to 0x00dc17 (0x3fb6 bytes) 000536 1755096548.066027 MIRROR_EXTEND to 11 mirrors 000537 1755096548.400782 write 0x0166f4 thru 0x01ddfb (0x7708 bytes) 000538 1755096548.441005 punch from 0x00ba39 to 0x01a6c2 (0xec8a bytes) 000539 1755096548.453838 mapread 0x00945a thru 0x011c12 (0x87b9 bytes) 000540 1755096548.461067 fallocate 0x024ac6 thru 0x02a262 (0x579d bytes) 000541 1755096548.465482 trunc from 0x02a263 to 0x00d440 (0x1ce22 bytes) 000542 1755096548.486579 mapwrite 0x00df2a thru 0x01412f (0x6206 bytes) 000543 1755096548.529011 trunc from 0x00c6e9 to 0x00f20d (0x2b25 bytes) 000544 1755096548.533118 mapwrite 0x02d290 thru 0x03b345 (0xe0b6 bytes) 000545 1755096548.580367 write 0x00553c thru 0x011ca2 (0xc767 bytes) 000546 1755096548.587381 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000547 1755096548.743429 trunc from 0x00b032 to 0x013d3c (0x8d0b bytes) 000548 1755096548.749684 read 0x02795a thru 0x035884 (0xdf2b bytes) 000549 1755096548.755548 trunc from 0x0300e6 to 0x03b345 (0xb260 bytes) 000550 1755096548.766049 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000551 1755096548.889551 punch from 0x02cf94 to 0x031050 (0x40bd bytes) 000552 1755096548.892314 read 0x01933b thru 0x01cf0c (0x3bd2 bytes) 000553 1755096548.895999 trunc from 0x03b346 to 0x01c225 (0x1f120 bytes) 000554 1755096548.907282 trunc from 0x01c226 to 0x03ed4d (0x22b28 bytes) 000555 1755096548.916731 punch from 0x0296cc to 0x0357e1 (0xc116 bytes) 000556 1755096548.921066 punch from 0x03ec2c to 0x03ed4d (0x122 bytes) 000557 1755096548.926681 mapwrite 0x013fa4 thru 0x01921f (0x527c bytes) 000558 1755096548.953284 write 0x00ae14 thru 0x016435 (0xb622 bytes) 000559 1755096548.962343 mapread 0x00f41f thru 0x014dc1 (0x59a3 bytes) 000560 1755096548.963228 write 0x0233a2 thru 0x02f793 (0xc3f2 bytes) 000561 1755096548.975069 read 0x00158b thru 0x00451c (0x2f92 bytes) 000562 1755096548.983295 mapread 0x038ea1 thru 0x03ed4d (0x5ead bytes) 000563 1755096548.984329 write 0x017390 thru 0x020393 (0x9004 bytes) 000564 1755096548.985767 write 0x021eb8 thru 0x02cb41 (0xac8a bytes) 000565 1755096548.987640 mapread 0x00842f thru 0x00966f (0x1241 bytes) 000566 1755096548.988553 mapread 0x00386c thru 0x0107ff (0xcf94 bytes) 000567 1755096548.993210 MIRROR_RESYNC 9 mirrors 000568 1755096550.023722 trunc from 0x03ed4e to 0x01c4f4 (0x22859 bytes) 000569 1755096550.069151 read 0x003f3d thru 0x00b169 (0x722d bytes) 000570 1755096550.083177 mapread 0x010743 thru 0x013cd0 (0x358e bytes) 000571 1755096550.097741 punch from 0x00298a to 0x00915e (0x67d5 bytes) 000572 1755096550.102048 trunc from 0x01c4f5 to 0x019a8b (0x2a69 bytes) 000573 1755096550.114721 read 0x0044bc thru 0x00c874 (0x83b9 bytes) 000574 1755096550.122193 trunc from 0x013e95 to 0x019a8b (0x5bf7 bytes) 000576 1755096550.129228 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000577 1755096550.317578 trunc from 0x019a8c to 0x0302ea (0x1685f bytes) 000578 1755096550.334632 mapread 0x023de3 thru 0x026e56 (0x3074 bytes) 000579 1755096550.354353 mapread 0x02ca6e thru 0x0302ea (0x387d bytes) 000580 1755096550.359956 trunc from 0x02e673 to 0x02febf (0x184d bytes) 000582 1755096550.365338 mapwrite 0x02f48a thru 0x03b997 (0xc50e bytes) 000583 1755096550.425625 MIRROR_RESYNC 8 mirrors 000584 1755096551.624302 write 0x01288b thru 0x019410 (0x6b86 bytes) 000585 1755096551.648711 trunc from 0x03b998 to 0x03cddc (0x1445 bytes) 000586 1755096551.667257 MIRROR_RESYNC 8 mirrors 000587 1755096552.597147 trunc from 0x03cddd to 0x0165ee (0x267ee bytes) 000589 1755096552.637488 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000591 1755096552.798670 MIRROR_RESYNC 7 mirrors 000592 1755096553.441484 mapread 0x0067ce thru 0x00c815 (0x6048 bytes) 000593 1755096553.451554 mapread 0x0063df thru 0x00d020 (0x6c42 bytes) 000594 1755096553.452190 fallocate 0x003a75 thru 0x00d8bc (0x9e48 bytes) 000595 1755096553.472519 trunc from 0x00b1d9 to 0x010c14 (0x5a3c bytes) 000596 1755096553.477462 fallocate 0x01de9d thru 0x01fc23 (0x1d87 bytes) 000597 1755096553.482017 punch from 0x014ba3 to 0x0165ee (0x1a4c bytes) 000598 1755096553.486220 trunc from 0x0165ef to 0x037a64 (0x21476 bytes) 000599 1755096553.503042 MIRROR_EXTEND to 8 mirrors 000600 1755096553.854738 fallocate 0x03f6d5 thru 0x03ffff (0x92b bytes) 000601 1755096553.874722 write 0x02e893 thru 0x031570 (0x2cde bytes) 000602 1755096553.884226 read 0x01a87e thru 0x01f73e (0x4ec1 bytes) 000603 1755096553.889798 write 0x00cc7f thru 0x017977 (0xacf9 bytes) 000604 1755096553.901120 punch from 0x034fe5 to 0x03b368 (0x6384 bytes) 000605 1755096553.904438 write 0x02292f thru 0x0298f1 (0x6fc3 bytes) 000606 1755096553.915577 MIRROR_RESYNC 8 mirrors 000607 1755096555.093089 write 0x02f0d1 thru 0x03d4fd (0xe42d bytes) 000608 1755096555.119956 read 0x0017d8 thru 0x00f51f (0xdd48 bytes) 000609 1755096555.124489 punch from 0x0175e7 to 0x01e7cb (0x71e5 bytes) 000610 1755096555.127734 read 0x00166a thru 0x00c18c (0xab23 bytes) 000611 1755096555.128366 trunc from 0x040000 to 0x028177 (0x17e88 bytes) 000613 1755096555.137566 trunc from 0x028178 to 0x008e14 (0x1f363 bytes) 000614 1755096555.145473 trunc from 0x0085f3 to 0x008e14 (0x822 bytes) 000615 1755096555.151015 MIRROR_EXTEND to 9 mirrors 000616 1755096555.361448 mapread 0x008ca7 thru 0x008e14 (0x16e bytes) 000617 1755096555.364607 trunc from 0x005c47 to 0x008e14 (0x31ce bytes) 000618 1755096555.387080 fallocate 0x0115e5 thru 0x01aaf1 (0x950d bytes) 000619 1755096555.391111 trunc from 0x00366e to 0x007edf (0x4872 bytes) 000621 1755096555.396217 punch from 0x00208b to 0x008e14 (0x6d8a bytes) 000622 1755096555.401499 mapwrite 0x036b44 thru 0x03b34e (0x480b bytes) 000623 1755096555.465091 mapread 0x02792c thru 0x02813e (0x813 bytes) 000624 1755096555.487997 MIRROR_RESYNC 9 mirrors 000625 1755096555.966625 write 0x0274a8 thru 0x02a007 (0x2b60 bytes) 000626 1755096555.986135 read 0x00477b thru 0x00ba82 (0x7308 bytes) 000627 1755096555.989955 mapread 0x039224 thru 0x03b34e (0x212b bytes) 000628 1755096555.993665 read 0x0126b5 thru 0x01a45d (0x7da9 bytes) 000629 1755096555.997970 mapread 0x038a8f thru 0x03b34e (0x28c0 bytes) 000630 1755096555.998667 read 0x00142a thru 0x001e0e (0x9e5 bytes) 000631 1755096555.999194 trunc from 0x036ef0 to 0x03b34e (0x445f bytes) 000632 1755096556.001819 write 0x03499e thru 0x0390f5 (0x4758 bytes) 000633 1755096556.005402 mapread 0x024ef3 thru 0x02ee29 (0x9f37 bytes) 000634 1755096556.006281 punch from 0x020faf to 0x021e9e (0xef0 bytes) 000635 1755096556.011265 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000636 1755096556.119647 punch from 0x02a620 to 0x039187 (0xeb68 bytes) 000637 1755096556.124702 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000638 1755096556.242005 read 0x021dd9 thru 0x025e10 (0x4038 bytes) 000639 1755096556.247796 trunc from 0x03b34f to 0x036a72 (0x48dc bytes) 000642 1755096556.268453 mapread 0x00a914 thru 0x00c65c (0x1d49 bytes) 000643 1755096556.280578 mapwrite 0x006ac2 thru 0x0093e8 (0x2927 bytes) 000644 1755096556.294741 read 0x01ab17 thru 0x022b32 (0x801c bytes) 000645 1755096556.302070 read 0x008e31 thru 0x018520 (0xf6f0 bytes) 000647 1755096556.303251 trunc from 0x036a73 to 0x02ac61 (0xbe11 bytes) 000648 1755096556.313476 read 0x021417 thru 0x0225e0 (0x11ca bytes) 000649 1755096556.314192 punch from 0x0280ca to 0x02ac61 (0x2b98 bytes) 000650 1755096556.319594 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000651 1755096556.417433 fallocate 0x0051bd thru 0x00a606 (0x544a bytes) 000652 1755096556.420382 fallocate 0x00227f thru 0x0078a5 (0x5627 bytes) 000653 1755096556.423380 trunc from 0x027829 to 0x029492 (0x1c6a bytes) 000654 1755096556.426105 trunc from 0x003975 to 0x00ec4b (0xb2d7 bytes) 000655 1755096556.429797 mapwrite 0x025010 thru 0x031846 (0xc837 bytes) 000656 1755096556.471322 MIRROR_EXTEND to 7 mirrors 000657 1755096556.705553 mapwrite 0x015d2c thru 0x0224c1 (0xc796 bytes) 000658 1755096556.784285 trunc from 0x031847 to 0x003e91 (0x2d9b5 bytes) 000659 1755096556.809456 mapwrite 0x033940 thru 0x03807b (0x473c bytes) 000660 1755096556.840595 punch from 0x01b633 to 0x01dd8c (0x275a bytes) 000661 1755096556.843251 trunc from 0x003be3 to 0x00c004 (0x8422 bytes) 000662 1755096556.845604 fallocate 0x03487d thru 0x03ffff (0xb783 bytes) 000663 1755096556.850251 MIRROR_EXTEND to 8 mirrors 000665 1755096557.080815 read 0x014232 thru 0x016886 (0x2655 bytes) 000666 1755096557.089324 write 0x033169 thru 0x03e544 (0xb3dc bytes) 000668 1755096557.111949 write 0x03d488 thru 0x03ffff (0x2b78 bytes) 000669 1755096557.112993 trunc from 0x040000 to 0x01041f (0x2fbe0 bytes) 000670 1755096557.138197 MIRROR_EXTEND to 9 mirrors 000671 1755096557.386966 write 0x0002c8 thru 0x0020f6 (0x1e2f bytes) 000672 1755096557.413288 read 0x003ed9 thru 0x008e14 (0x4f3c bytes) 000674 1755096557.421353 trunc from 0x007e60 to 0x009baf (0x1d50 bytes) 000675 1755096557.424493 read 0x00c6f5 thru 0x01035e (0x3c6a bytes) 000676 1755096557.429756 mapread 0x00d732 thru 0x01041f (0x2cee bytes) 000677 1755096557.430686 trunc from 0x00c163 to 0x01041f (0x42bd bytes) 000678 1755096557.437114 MIRROR_EXTEND to 10 mirrors 000679 1755096557.680480 punch from 0x003765 to 0x01041f (0xccbb bytes) 000680 1755096557.703354 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000681 1755096557.803196 trunc from 0x001b58 to 0x00b3da (0x9883 bytes) 000682 1755096557.806027 trunc from 0x007395 to 0x01041f (0x908b bytes) 000683 1755096557.808361 read 0x00cc64 thru 0x00e705 (0x1aa2 bytes) 000684 1755096557.811670 read 0x006875 thru 0x00a01a (0x37a6 bytes) 000685 1755096557.816066 trunc from 0x00726b to 0x00c3a4 (0x513a bytes) 000686 1755096557.823205 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000688 1755096557.933442 read 0x000d29 thru 0x0073ed (0x66c5 bytes) 000689 1755096557.941108 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000691 1755096558.056333 punch from 0x008168 to 0x00dc33 (0x5acc bytes) 000692 1755096558.059289 mapwrite 0x002a30 thru 0x004730 (0x1d01 bytes) 000693 1755096558.082129 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000694 1755096558.169260 trunc from 0x0050f9 to 0x00dadb (0x89e3 bytes) 000695 1755096558.172152 write 0x027b62 thru 0x029a42 (0x1ee1 bytes) 000696 1755096558.174777 punch from 0x014951 to 0x01d665 (0x8d15 bytes) 000697 1755096558.177417 read 0x01fe00 thru 0x0259f7 (0x5bf8 bytes) 000698 1755096558.180812 mapread 0x00b4b9 thru 0x00e615 (0x315d bytes) 000699 1755096558.188447 trunc from 0x029a43 to 0x027bf1 (0x1e51 bytes) 000700 1755096558.201881 MIRROR_RESYNC 6 mirrors 000701 1755096558.573640 read 0x00891d thru 0x016ad4 (0xe1b8 bytes) 000702 1755096558.577596 punch from 0x00f342 to 0x01c372 (0xd031 bytes) 000703 1755096558.595522 mapread 0x014738 thru 0x023cc7 (0xf590 bytes) 000704 1755096558.605656 read 0x0153ba thru 0x022718 (0xd35f bytes) 000705 1755096558.608387 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000706 1755096558.712422 read 0x004405 thru 0x00ec07 (0xa803 bytes) 000707 1755096558.718341 trunc from 0x027bf2 to 0x01c7a7 (0xb44a bytes) 000708 1755096558.738631 read 0x0140ea thru 0x019660 (0x5577 bytes) 000710 1755096558.744623 trunc from 0x01162c to 0x017111 (0x5ae6 bytes) 000711 1755096558.749109 mapread 0x01685e thru 0x01c7a7 (0x5f4a bytes) 000712 1755096558.758051 mapwrite 0x01e17d thru 0x02af99 (0xce1d bytes) 000713 1755096558.787308 write 0x001d9d thru 0x00f8bc (0xdb20 bytes) 000716 1755096558.789448 write 0x0114c5 thru 0x017f64 (0x6aa0 bytes) 000717 1755096558.793479 mapwrite 0x02abf4 thru 0x039aca (0xeed7 bytes) 000718 1755096558.820213 trunc from 0x039acb to 0x01cefc (0x1cbce bytes) 000719 1755096558.829005 trunc from 0x01ca53 to 0x01cefc (0x4aa bytes) 000720 1755096558.832921 mapwrite 0x010ac8 thru 0x013801 (0x2d3a bytes) 000722 1755096558.844495 punch from 0x00a9a3 to 0x015fae (0xb60c bytes) 000723 1755096558.848652 fallocate 0x00de71 thru 0x0124f3 (0x4683 bytes) 000724 1755096558.852458 write 0x03795c thru 0x03ffff (0x86a4 bytes) 000725 1755096558.854010 fallocate 0x033a49 thru 0x03ffff (0xc5b7 bytes) 000726 1755096558.857643 punch from 0x03f297 to 0x03ffff (0xd69 bytes) 000727 1755096558.865944 mapwrite 0x03cd51 thru 0x03ffff (0x32af bytes) 000729 1755096558.877381 fallocate 0x01b240 thru 0x028e05 (0xdbc6 bytes) 000730 1755096558.881204 fallocate 0x00dbb3 thru 0x01609d (0x84eb bytes) 000731 1755096558.884071 read 0x03b301 thru 0x03ffff (0x4cff bytes) 000732 1755096558.884589 punch from 0x021eb7 to 0x026f23 (0x506d bytes) 000733 1755096558.887241 read 0x029972 thru 0x034b67 (0xb1f6 bytes) 000734 1755096558.892751 punch from 0x0112c3 to 0x012db4 (0x1af2 bytes) 000735 1755096558.895520 mapread 0x0084f7 thru 0x016aca (0xe5d4 bytes) 000736 1755096558.900902 punch from 0x01e668 to 0x02a9ac (0xc345 bytes) 000737 1755096558.904467 fallocate 0x01981b thru 0x02534d (0xbb33 bytes) 000738 1755096558.909903 MIRROR_EXTEND to 6 mirrors 000739 1755096559.177162 read 0x02eea0 thru 0x0302a3 (0x1404 bytes) 000740 1755096559.185283 punch from 0x00deaa to 0x01b564 (0xd6bb bytes) 000741 1755096559.209287 mapwrite 0x034485 thru 0x038971 (0x44ed bytes) 000742 1755096559.229683 trunc from 0x007e96 to 0x00a561 (0x26cc bytes) 000743 1755096559.232770 punch from 0x035524 to 0x03ffff (0xaadc bytes) 000744 1755096559.235721 trunc from 0x040000 to 0x015a17 (0x2a5e8 bytes) 000745 1755096559.249895 punch from 0x00942b to 0x015a17 (0xc5ed bytes) 000747 1755096559.253777 mapread 0x01429f thru 0x015a17 (0x1779 bytes) 000748 1755096559.259130 punch from 0x01336b to 0x015a17 (0x26ad bytes) 000749 1755096559.261767 trunc from 0x006e2c to 0x015296 (0xe46b bytes) 000750 1755096559.265521 punch from 0x00b9bd to 0x00f35d (0x39a1 bytes) 000751 1755096559.268859 trunc from 0x015a18 to 0x00ef0d (0x6b0a bytes) 000752 1755096559.278656 mapread 0x008cd2 thru 0x00ef0d (0x623c bytes) 000753 1755096559.283141 mapread 0x00135b thru 0x00d221 (0xbec7 bytes) 000754 1755096559.284013 mapwrite 0x03fe7c thru 0x03ffff (0x184 bytes) 000755 1755096559.306849 mapread 0x034326 thru 0x037e25 (0x3b00 bytes) 000756 1755096559.312290 read 0x011157 thru 0x01a51f (0x93c9 bytes) 000757 1755096559.318479 mapwrite 0x036f2c thru 0x03c49e (0x5573 bytes) 000759 1755096559.334659 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000760 1755096559.422290 read 0x035888 thru 0x03af99 (0x5712 bytes) 000761 1755096559.426765 read 0x02ccf4 thru 0x039842 (0xcb4f bytes) 000762 1755096559.435726 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000764 1755096559.527203 trunc from 0x004074 to 0x00bbf4 (0x7b81 bytes) 000765 1755096559.530196 read 0x002a5d thru 0x0068a7 (0x3e4b bytes) 000767 1755096559.534835 fallocate 0x034f77 thru 0x035f5f (0xfe9 bytes) 000768 1755096559.540470 MIRROR_EXTEND to 5 mirrors 000769 1755096559.769254 punch from 0x00d8da to 0x01ca4f (0xf176 bytes) 000770 1755096559.798529 write 0x029445 thru 0x02bb5a (0x2716 bytes) 000771 1755096559.808850 mapwrite 0x00b145 thru 0x00c873 (0x172f bytes) 000772 1755096559.828050 mapread 0x013a06 thru 0x01df20 (0xa51b bytes) 000773 1755096559.837069 mapread 0x02d451 thru 0x03cf94 (0xfb44 bytes) 000774 1755096559.845944 punch from 0x02b1f5 to 0x037990 (0xc79c bytes) 000775 1755096559.848920 punch from 0x013907 to 0x021d33 (0xe42d bytes) 000776 1755096559.851720 mapread 0x01de89 thru 0x028b41 (0xacb9 bytes) 000777 1755096559.857592 fallocate 0x02321e thru 0x025c25 (0x2a08 bytes) 000778 1755096559.861336 read 0x033802 thru 0x03dd41 (0xa540 bytes) 000779 1755096559.868597 mapwrite 0x037545 thru 0x0386fc (0x11b8 bytes) 000780 1755096559.876596 punch from 0x033ed0 to 0x03ffff (0xc130 bytes) 000781 1755096559.879751 read 0x0197d4 thru 0x02978f (0xffbc bytes) 000782 1755096559.882634 MIRROR_EXTEND to 6 mirrors 000783 1755096560.071407 punch from 0x009e2f to 0x00ff5d (0x612f bytes) 000784 1755096560.085609 trunc from 0x02383d to 0x02bd6b (0x852f bytes) 000785 1755096560.088052 read 0x0396aa thru 0x03ffff (0x6956 bytes) 000786 1755096560.091839 mapread 0x0187af thru 0x022320 (0x9b72 bytes) 000787 1755096560.101859 trunc from 0x040000 to 0x00be86 (0x34179 bytes) 000788 1755096560.117683 read 0x0024e8 thru 0x00a495 (0x7fae bytes) 000789 1755096560.122101 fallocate 0x03a194 thru 0x03ffff (0x5e6c bytes) 000790 1755096560.124936 write 0x01cd94 thru 0x01f56e (0x27db bytes) 000791 1755096560.129830 punch from 0x001728 to 0x010afd (0xf3d6 bytes) 000792 1755096560.132449 trunc from 0x040000 to 0x01bc22 (0x243dd bytes) 000793 1755096560.139818 trunc from 0x009a14 to 0x013a75 (0xa062 bytes) 000794 1755096560.142223 trunc from 0x01bc23 to 0x03b017 (0x1f3f5 bytes) 000795 1755096560.147564 read 0x024f99 thru 0x02d5d6 (0x863e bytes) 000796 1755096560.154323 MIRROR_RESYNC 6 mirrors 000797 1755096560.434296 mapwrite 0x024cb1 thru 0x0292c5 (0x4615 bytes) 000798 1755096560.488671 read 0x0128c0 thru 0x0150ed (0x282e bytes) 000799 1755096560.492133 trunc from 0x02f999 to 0x03923a (0x98a2 bytes) 000800 1755096560.495573 write 0x027496 thru 0x02b390 (0x3efb bytes) 000801 1755096560.496695 mapread 0x00ccca thru 0x01b70f (0xea46 bytes) 000802 1755096560.504104 trunc from 0x01e180 to 0x029306 (0xb187 bytes) 000803 1755096560.510427 mapread 0x014136 thru 0x02018c (0xc057 bytes) 000804 1755096560.515286 mapwrite 0x02cf0d thru 0x02e764 (0x1858 bytes) 000807 1755096560.528541 MIRROR_EXTEND to 7 mirrors 000808 1755096560.732296 punch from 0x034b6e to 0x03b017 (0x64aa bytes) 000809 1755096560.753665 MIRROR_EXTEND to 8 mirrors 000810 1755096560.926602 punch from 0x0036cc to 0x010f17 (0xd84c bytes) 000811 1755096560.943531 read 0x01f35d thru 0x026abf (0x7763 bytes) 000812 1755096560.949093 punch from 0x02fc2b to 0x03ad46 (0xb11c bytes) 000813 1755096560.953774 MIRROR_RESYNC 8 mirrors 000814 1755096561.451927 fallocate 0x02a50f thru 0x02cd08 (0x27fa bytes) 000815 1755096561.466930 mapwrite 0x021cd0 thru 0x023b13 (0x1e44 bytes) 000817 1755096561.484029 trunc from 0x03b018 to 0x032b66 (0x84b1 bytes) 000819 1755096561.493013 mapwrite 0x023257 thru 0x02cb02 (0x98ac bytes) 000820 1755096561.506576 write 0x02e5b1 thru 0x03d4d1 (0xef21 bytes) 000821 1755096561.508114 trunc from 0x03d4d2 to 0x02b414 (0x120bd bytes) 000822 1755096561.521956 MIRROR_EXTEND to 9 mirrors 000823 1755096561.709101 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000824 1755096561.854118 punch from 0x013366 to 0x01bcbc (0x8957 bytes) 000825 1755096561.870272 mapread 0x029bd7 thru 0x02a4f6 (0x920 bytes) 000826 1755096561.875950 trunc from 0x02b415 to 0x023fa6 (0x746e bytes) 000828 1755096561.886179 mapread 0x00653a thru 0x009911 (0x33d8 bytes) 000829 1755096561.890521 trunc from 0x023fa7 to 0x036af0 (0x12b4a bytes) 000830 1755096561.896330 mapwrite 0x009748 thru 0x00ee06 (0x56bf bytes) 000831 1755096561.906601 fallocate 0x02c4f5 thru 0x030227 (0x3d33 bytes) 000833 1755096561.909934 write 0x00c358 thru 0x015671 (0x931a bytes) 000834 1755096561.913317 fallocate 0x01982d thru 0x01d75c (0x3f30 bytes) 000835 1755096561.915776 fallocate 0x0202fe thru 0x02180c (0x150f bytes) 000836 1755096561.918269 mapwrite 0x026dd5 thru 0x02d569 (0x6795 bytes) 000837 1755096561.931250 punch from 0x027899 to 0x02902b (0x1793 bytes) 000838 1755096561.933764 write 0x03e26d thru 0x03ffff (0x1d93 bytes) 000840 1755096561.935180 mapread 0x006577 thru 0x015017 (0xeaa1 bytes) 000841 1755096561.936041 read 0x01d442 thru 0x01d7e9 (0x3a8 bytes) 000842 1755096561.938706 mapread 0x02ae41 thru 0x02f866 (0x4a26 bytes) 000843 1755096561.943725 read 0x02eafe thru 0x038bff (0xa102 bytes) 000844 1755096561.947774 write 0x00bb49 thru 0x0172ad (0xb765 bytes) 000845 1755096561.948985 trunc from 0x001be6 to 0x00a884 (0x8c9f bytes) 000847 1755096561.952221 mapread 0x01e706 thru 0x0242e2 (0x5bdd bytes) 000848 1755096561.953113 write 0x007e61 thru 0x013b2e (0xbcce bytes) 000849 1755096561.956589 mapread 0x00e833 thru 0x01ac9b (0xc469 bytes) 000850 1755096561.957492 punch from 0x03f812 to 0x03ffff (0x7ee bytes) 000851 1755096561.964632 write 0x021880 thru 0x02921f (0x79a0 bytes) 000852 1755096561.968762 MIRROR_EXTEND to 9 mirrors 000853 1755096562.209697 write 0x0147cf thru 0x023adf (0xf311 bytes) 000854 1755096562.239559 fallocate 0x005d07 thru 0x0153bb (0xf6b5 bytes) 000855 1755096562.243523 punch from 0x02d7a0 to 0x03b7d4 (0xe035 bytes) 000856 1755096562.246712 write 0x01cc7a thru 0x024e3e (0x81c5 bytes) 000857 1755096562.251048 punch from 0x01db88 to 0x0218fc (0x3d75 bytes) 000858 1755096562.258365 mapread 0x005015 thru 0x010fbb (0xbfa7 bytes) 000859 1755096562.266749 write 0x0179ad thru 0x022807 (0xae5b bytes) 000860 1755096562.268167 read 0x01d1e3 thru 0x02935d (0xc17b bytes) 000861 1755096562.268734 fallocate 0x03b462 thru 0x03ffff (0x4b9e bytes) 000862 1755096562.274729 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000863 1755096562.370379 punch from 0x03e334 to 0x03ffff (0x1ccc bytes) 000864 1755096562.372888 fallocate 0x015237 thru 0x024b3c (0xf906 bytes) 000865 1755096562.376105 mapwrite 0x000014 thru 0x00008b (0x78 bytes) 000866 1755096562.386728 trunc from 0x040000 to 0x02991c (0x166e3 bytes) 000867 1755096562.395353 mapread 0x003d02 thru 0x007e3e (0x413d bytes) 000868 1755096562.403807 read 0x02977a thru 0x02991c (0x1a3 bytes) 000869 1755096562.407926 trunc from 0x02991d to 0x022eed (0x6a2f bytes) 000870 1755096562.414940 trunc from 0x011924 to 0x01c9a3 (0xb080 bytes) 000871 1755096562.417571 read 0x01515b thru 0x022eed (0xdd93 bytes) 000872 1755096562.422167 fallocate 0x02b790 thru 0x03aec9 (0xf73a bytes) 000873 1755096562.427601 MIRROR_RESYNC 8 mirrors 000874 1755096562.967483 read 0x015882 thru 0x015c07 (0x386 bytes) 000875 1755096562.970577 trunc from 0x022eee to 0x0118a3 (0x1164a bytes) 000876 1755096562.996998 mapwrite 0x009d76 thru 0x00e566 (0x47f1 bytes) 000878 1755096563.016612 trunc from 0x0118a4 to 0x0200c9 (0xe826 bytes) 000879 1755096563.026005 trunc from 0x0200ca to 0x01f98f (0x73a bytes) 000880 1755096563.031472 mapwrite 0x037f30 thru 0x03ffff (0x80d0 bytes) 000881 1755096563.052331 read 0x01c606 thru 0x021132 (0x4b2d bytes) 000882 1755096563.057213 punch from 0x003b7b to 0x0113ff (0xd885 bytes) 000883 1755096563.060442 trunc from 0x040000 to 0x00b05f (0x34fa0 bytes) 000884 1755096563.071070 punch from 0x00aa9b to 0x00b05f (0x5c5 bytes) 000885 1755096563.073619 punch from 0x0005de to 0x008ed1 (0x88f4 bytes) 000889 1755096563.078545 fallocate 0x02135d thru 0x02da0b (0xc6af bytes) 000890 1755096563.081959 punch from 0x0079dc to 0x00b05f (0x3684 bytes) 000891 1755096563.085222 read 0x001288 thru 0x0078c1 (0x663a bytes) 000892 1755096563.093306 punch from 0x00376c to 0x00b05f (0x78f4 bytes) 000893 1755096563.096387 trunc from 0x001e9f to 0x00b05f (0x91c1 bytes) 000894 1755096563.099819 trunc from 0x00607f to 0x00b05f (0x4fe1 bytes) 000895 1755096563.103063 trunc from 0x00b060 to 0x036303 (0x2b2a4 bytes) 000896 1755096563.114079 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000897 1755096563.218831 fallocate 0x00cfb9 thru 0x01721d (0xa265 bytes) 000898 1755096563.224393 fallocate 0x033c90 thru 0x0396f5 (0x5a66 bytes) 000899 1755096563.230069 mapwrite 0x011f81 thru 0x01923e (0x72be bytes) 000900 1755096563.268973 punch from 0x01061b to 0x0123f4 (0x1dda bytes) 000901 1755096563.274949 write 0x0029f9 thru 0x007902 (0x4f0a bytes) 000902 1755096563.283646 trunc from 0x0396f6 to 0x019e40 (0x1f8b5 bytes) 000903 1755096563.292424 mapwrite 0x00513f thru 0x01329d (0xe15f bytes) 000905 1755096563.315027 punch from 0x004ad3 to 0x014959 (0xfe87 bytes) 000906 1755096563.320894 mapread 0x014cea thru 0x019e40 (0x5157 bytes) 000907 1755096563.328529 fallocate 0x0385d5 thru 0x03e644 (0x6070 bytes) 000908 1755096563.336428 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000909 1755096563.462601 punch from 0x00972d to 0x011747 (0x801b bytes) 000910 1755096563.466054 trunc from 0x002fa5 to 0x00839e (0x53fa bytes) 000911 1755096563.470031 mapread 0x002952 thru 0x00bff7 (0x96a6 bytes) 000912 1755096563.483645 fallocate 0x037e1a thru 0x03bb5a (0x3d41 bytes) 000913 1755096563.487315 mapread 0x025b86 thru 0x033bac (0xe027 bytes) 000914 1755096563.497493 write 0x006441 thru 0x0099b4 (0x3574 bytes) 000915 1755096563.500502 trunc from 0x03bb5b to 0x026fde (0x14b7c bytes) 000916 1755096563.509167 mapread 0x0022f9 thru 0x006a2a (0x4732 bytes) 000917 1755096563.509990 fallocate 0x03d0a6 thru 0x03ffff (0x2f5a bytes) 000919 1755096563.513753 fallocate 0x02019c thru 0x02dd28 (0xdb8d bytes) 000920 1755096563.518389 punch from 0x0188c9 to 0x0261fa (0xd932 bytes) 000922 1755096563.524305 punch from 0x037916 to 0x03ffff (0x86ea bytes) 000923 1755096563.527306 trunc from 0x040000 to 0x017add (0x28522 bytes) 000924 1755096563.535818 read 0x00b63b thru 0x017add (0xc4a3 bytes) 000925 1755096563.540463 punch from 0x00f69b to 0x017add (0x8443 bytes) 000926 1755096563.543485 trunc from 0x017ade to 0x00a60b (0xd4d2 bytes) 000927 1755096563.553201 fallocate 0x033fae thru 0x03ffff (0xc052 bytes) 000928 1755096563.556489 fallocate 0x00e721 thru 0x010111 (0x19f1 bytes) 000929 1755096563.559398 mapread 0x00480a thru 0x010307 (0xbafe bytes) 000931 1755096563.570280 MIRROR_RESYNC 6 mirrors 000932 1755096564.179513 trunc from 0x01d105 to 0x02c3f9 (0xf2f5 bytes) 000933 1755096564.197424 read 0x002150 thru 0x00db8f (0xba40 bytes) 000935 1755096564.202102 read 0x01114c thru 0x018211 (0x70c6 bytes) 000936 1755096564.207461 trunc from 0x0055b5 to 0x0068a1 (0x12ed bytes) 000937 1755096564.210795 read 0x033489 thru 0x03fe6a (0xc9e2 bytes) 000938 1755096564.211539 trunc from 0x040000 to 0x013d19 (0x2c2e6 bytes) 000939 1755096564.221455 trunc from 0x0121f6 to 0x013d19 (0x1b24 bytes) 000940 1755096564.223886 mapread 0x00aa21 thru 0x013d19 (0x92f9 bytes) 000941 1755096564.227745 mapread 0x0037a4 thru 0x00db97 (0xa3f4 bytes) 000942 1755096564.228491 trunc from 0x013d1a to 0x02bda3 (0x1808a bytes) 000943 1755096564.236027 trunc from 0x0137ea to 0x01ffc8 (0xc7df bytes) 000944 1755096564.238711 mapread 0x013981 thru 0x020ce3 (0xd363 bytes) 000948 1755096564.247106 mapread 0x00a684 thru 0x00c05a (0x19d7 bytes) 000949 1755096564.247934 read 0x02bbeb thru 0x02bda3 (0x1b9 bytes) 000950 1755096564.248461 mapwrite 0x015d01 thru 0x01c420 (0x6720 bytes) 000951 1755096564.265710 mapread 0x01b9f9 thru 0x024180 (0x8788 bytes) 000952 1755096564.267349 trunc from 0x029399 to 0x02bda3 (0x2a0b bytes) 000953 1755096564.271184 mapwrite 0x01fc8b thru 0x02f977 (0xfced bytes) 000954 1755096564.309308 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000955 1755096564.434752 mapread 0x02f068 thru 0x02f977 (0x910 bytes) 000956 1755096564.439597 mapread 0x02caec thru 0x02f977 (0x2e8c bytes) 000957 1755096564.447783 read 0x025313 thru 0x02f977 (0xa665 bytes) 000958 1755096564.451580 punch from 0x01f1b0 to 0x01f354 (0x1a5 bytes) 000959 1755096564.455288 mapread 0x0125b6 thru 0x017868 (0x52b3 bytes) 000960 1755096564.460740 mapwrite 0x00eb1f thru 0x019c79 (0xb15b bytes) 000961 1755096564.473988 trunc from 0x02c487 to 0x02f977 (0x34f1 bytes) 000962 1755096564.476990 mapwrite 0x00eb5e thru 0x01263a (0x3add bytes) 000963 1755096564.490324 MIRROR_RESYNC 5 mirrors 000964 1755096565.196629 write 0x02c82f thru 0x02cf3a (0x70c bytes) 000965 1755096565.216096 fallocate 0x00c83e thru 0x010127 (0x38ea bytes) 000966 1755096565.219961 write 0x031d6d thru 0x03e3f7 (0xc68b bytes) 000967 1755096565.221289 mapread 0x007453 thru 0x00aa1f (0x35cd bytes) 000968 1755096565.234275 MIRROR_RESYNC 5 mirrors 000969 1755096565.984246 punch from 0x01424d to 0x015cb1 (0x1a65 bytes) 000970 1755096566.003455 mapwrite 0x0313f3 thru 0x037a62 (0x6670 bytes) 000971 1755096566.028655 write 0x00f52c thru 0x01a812 (0xb2e7 bytes) 000972 1755096566.035212 trunc from 0x008c4f to 0x018197 (0xf549 bytes) 000973 1755096566.042577 mapwrite 0x02b188 thru 0x0315d4 (0x644d bytes) 000974 1755096566.057325 fallocate 0x003375 thru 0x00e164 (0xadf0 bytes) 000975 1755096566.060296 mapwrite 0x0091e5 thru 0x0097c5 (0x5e1 bytes) 000976 1755096566.071643 read 0x01a293 thru 0x029923 (0xf691 bytes) 000977 1755096566.078279 write 0x00b237 thru 0x019ff2 (0xedbc bytes) 000978 1755096566.079801 mapwrite 0x03419a thru 0x03ffff (0xbe66 bytes) 000979 1755096566.110158 read 0x02473b thru 0x0252fa (0xbc0 bytes) 000980 1755096566.110608 punch from 0x00f625 to 0x012362 (0x2d3e bytes) 000981 1755096566.114927 MIRROR_EXTEND to 6 mirrors 000982 1755096566.420285 fallocate 0x02868b thru 0x037236 (0xebac bytes) 000983 1755096566.441259 punch from 0x0291a4 to 0x0300c9 (0x6f26 bytes) 000984 1755096566.447133 MIRROR_RESYNC 6 mirrors 000985 1755096567.270061 write 0x00ea7e thru 0x016552 (0x7ad5 bytes) 000986 1755096567.288601 write 0x038f0e thru 0x03af65 (0x2058 bytes) 000987 1755096567.292506 mapwrite 0x0206ee thru 0x02f8df (0xf1f2 bytes) 000988 1755096567.313523 mapwrite 0x002bda thru 0x009143 (0x656a bytes) 000989 1755096567.329695 write 0x02a66d thru 0x030bb3 (0x6547 bytes) 000990 1755096567.331029 punch from 0x006b5d to 0x0081ac (0x1650 bytes) 000991 1755096567.334808 fallocate 0x008389 thru 0x0116e5 (0x935d bytes) 000992 1755096567.340513 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44a:0x0]: Invalid argument (22) 000993 1755096567.443672 read 0x011231 thru 0x021139 (0xff09 bytes) 000994 1755096567.450620 trunc from 0x03dc1a to 0x03ffff (0x23e6 bytes) 000996 1755096567.454320 write 0x03cef7 thru 0x03ffff (0x3109 bytes) 000997 1755096567.459816 read 0x0372ea thru 0x03ffff (0x8d16 bytes) 000998 1755096567.465924 read 0x005987 thru 0x00d267 (0x78e1 bytes) 000999 1755096567.466467 punch from 0x01da68 to 0x025745 (0x7cde bytes) 001000 1755096567.469600 write 0x00df20 thru 0x018376 (0xa457 bytes) All operations completed A-OK! PASS 70a (47s) == sanity-flr test 71: check mirror extend parent fid ==== 10:49:29 (1755096569) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x490:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (4s) == sanity-flr test 200a: stress test ===================== 10:49:33 (1755096573) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre2 Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1011200 .. Extending file size to 6859712 .. done Extending file size to 4315328 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7976544 .. Extending file size to 7790016 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9104032 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8840864 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8973504 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61285 61286 61288 61290 61292 done 192.168.202.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre2 (opts:) 192.168.202.114@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:50:40 (1755096640) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre2 Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6668000 .. Extending file size to 3885760 .. failed Extending file size to 6181664 .. Extending file size to 7587200 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7766272 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7865280 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7811360 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7942208 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8863936 .. Extending file size to 8288000 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9216032 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9031520 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9156320 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 64730 64731 64733 64734 64735 192.168.202.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre2 (opts:) 192.168.202.114@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:51:46 (1755096706) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2612 1285076 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2768 1284920 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14764 3592256 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22808 3584212 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37572 7176468 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2612 1285076 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2768 1284920 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14764 3592256 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22808 3584212 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37572 7176468 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0477054 s, 44.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0729962 s, 28.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:51:53 (1755096713) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0474572 s, 44.2 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 10:51:56 (1755096716) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:51:59 (1755096719) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0367818 s, 57.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x655:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x657:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x658:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:52:02 (1755096722) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0392292 s, 53.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:52:05 (1755096725) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0517204 s, 40.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:52:30 (1755096750) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg214-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389412 s, 26.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x656:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x65a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x659:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x656:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x65a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x659:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x657:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:53:05 (1755096785) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg214-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x658:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x659:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.039535 s, 53.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x658:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x659:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x65b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x658:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x659:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x65b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:53:41 (1755096821) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg214-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg214-server: Pool lustre.test_204f created Adding targets to pool oleg214-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg214-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0379636 s, 55.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x65a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x65c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x65a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x65c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg214-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg214-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg214-server: Pool lustre.test_204f destroyed PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:54:30 (1755096870) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:54:32 (1755096872) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:54:35 (1755096875) oleg214-server: Pool lustre.test_206 created Adding targets to pool oleg214-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg214-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg214-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg214-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg214-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:54:49 (1755096889) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289903 s, 36.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158231 s, 66.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xfbe3d382 Mirror 3: 0xfbe3d382 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:54:52 (1755096892) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:54:54 (1755096894) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:54:55 (1755096895) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0173084 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2070000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x9e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x9e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x9e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x910000 Read this (4 bytes): BBBB 192.168.202.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre2 (opts:) PASS 209a (18s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:55:13 (1755096913) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00447751 s, 1.8 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:55:15 (1755096915) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171572 s, 61.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7c lmm_fid: [0x200000405:0x7c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7c lmm_fid: [0x200000405:0x7c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb7:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611260 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-08-13 10:55:15.000000000 -0400 Modify: 2025-08-13 10:55:16.000000000 -0400 Change: 2025-08-13 10:55:16.000000000 -0400 Birth: 2025-08-13 10:55:16.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:55:18 (1755096918) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130495 s, 80.4 MB/s oleg214-server: debugfs 1.47.1-wc2 (08-Nov-2024) fail_loc=0x1428 Waiting for MDT destroys to complete oleg214-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg214-server: O/280000401/d24/b8: File not found by ext2_lookup PASS 210b (15s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:55:33 (1755096933) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0326172 s, 1.3 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00628875 s, 651 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 13 10:55 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x81:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 13 10:55 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 929 sec ============ 10:55:36 (1755096936) === sanity-flr: start cleanup 10:55:36 (1755096936) === === sanity-flr: finish cleanup 10:55:39 (1755096939) ===