-----============= acceptance-small: sanity-flr ============----- Fri Dec 13 12:48:43 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg434-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 12:48:47 (1734112127) === oleg434-client.virtnet: executing check_config_client /mnt/lustre oleg434-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg434-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6583000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6583000.idle_timeout=debug disable quota as required oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 12:48:55 (1734112135) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7593] == sanity-flr test 0a: lfs mirror create with -N option == 12:48:55 (1734112135) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:48:58 (1734112138) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:48:59 (1734112139) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:49:00 (1734112140) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:49:04 (1734112144) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:49:04 (1734112144) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 12:49:05 (1734112145) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6583000.stats=clear osc.lustre-OST0001-osc-ffff8800b6583000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:49:07 (1734112147) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:49:09 (1734112149) PASS 0j (1s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:49:10 (1734112150) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:49:11 (1734112151) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 12:49:13 (1734112153) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:49:14 (1734112154) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:49:16 (1734112156) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:49:18 (1734112158) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:49:20 (1734112160) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:49:21 (1734112161) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.148879 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0273661 s, 38.3 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:49:23 (1734112163) 17+0 records in 17+0 records out 17825792 bytes (18 MB) copied, 0.120964 s, 147 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:49:25 (1734112165) Stopping /mnt/lustre-ost1 (opts:) on oleg434-server oleg434-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6583000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6583000.ost_server_uuid in DISCONN state after 3 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg434-server: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg434-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg434-server oleg434-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6583000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6583000.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg434-server: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg434-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (27s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:49:52 (1734112192) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0445136 s, 70.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg434-server oleg434-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6583000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6583000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg434-server: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg434-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (10s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:50:02 (1734112202) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg434-server oleg434-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6583000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6583000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg434-server: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg434-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg434-server oleg434-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6583000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6583000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg434-server: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg434-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (15s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:50:17 (1734112217) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.764908 s, 68.5 MB/s -rw-r--r-- 1 root root 52428800 Dec 13 12:50 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.184s user 0m0.000s sys 0m0.100s Stopping /mnt/lustre-ost1 (opts:) on oleg434-server oleg434-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6583000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6583000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.193s user 0m0.000s sys 0m0.096s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg434-server: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg434-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg434-server oleg434-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6583000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6583000.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.228s user 0m0.001s sys 0m0.105s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg434-server: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg434-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg434-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6583000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6583000.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:50:38 (1734112238) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:50:39 (1734112239) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (0s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:50:40 (1734112240) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 12:50:40 (1734112240) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 12:50:42 (1734112242) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.116196 s, 108 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:50:44 (1734112244) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00959804 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00901881 s, 116 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:50:46 (1734112246) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:50:48 (1734112248) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000464345 s, 2.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000456291 s, 2.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000700634 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000495842 s, 2.1 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 12:50:50 (1734112250) create test files with size 15740135 .. b58a8bbf59d2cdf7fedcafbefc495ff6 - b5cb27a32daa49433fad2dd3234f758d - 2f0c41a04cf14ff5152652d4f4136e1e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg434-client.virtnet /mnt/lustre (opts:) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 12:50:54 (1734112254) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:50:57 (1734112257) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:50:58 (1734112258) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0144028 s, 72.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x35:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100441 s, 104 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x36:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 12:51:00 (1734112260) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0187222 s, 112 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0806868 s, 52.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0293921 s, 71.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 12:51:02 (1734112262) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 12:51:03 (1734112263) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:51:03 (1734112263) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00803017 s, 131 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 12:51:05 (1734112265) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.073995 s, 42.5 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x10:0x0]: Invalid argument (22) PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:51:07 (1734112267) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x44:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x41:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:51:09 (1734112269) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0751621 s, 140 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg434-client.virtnet /mnt/lustre (opts:) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:51:32 (1734112292) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800b66f7800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0644659 s, 163 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg434-client.virtnet /mnt/lustre (opts:) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800b6673800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:51:35 (1734112295) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:51:42 (1734112302) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:51:48 (1734112308) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) == sanity-flr test 48: Verify snapshot mirror ============ 12:51:48 (1734112308) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0607453 s, 51.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0477213 s, 65.9 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 1283f7176b08e1d066f23df7bb9e444e - mirror 1 checksum: 84d74ad67e2783de8cbe02299e8d2718 - mirror 2 checksum: 1283f7176b08e1d066f23df7bb9e444e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Dec 13 12:51:49 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Dec 13 12:51:49 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1283f7176b08e1d066f23df7bb9e444e - mirror 1 checksum: 84d74ad67e2783de8cbe02299e8d2718 - mirror 2 checksum: 84d74ad67e2783de8cbe02299e8d2718 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Dec 13 12:51:49 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Dec 13 12:51:49 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 12:51:50 (1734112310) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102845 s, 102 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 529408.. 530431: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:51:52 (1734112312) Failing mds1 on oleg434-server Stopping /mnt/lustre-mds1 (opts:) on oleg434-server 12:51:59 (1734112319) shut down facet: mds1 facet_host: oleg434-server facet_failover_host: oleg434-server Failover mds1 to oleg434-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 12:52:12 (1734112332) targets are mounted 12:52:12 (1734112332) facet_failover done oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00806004 s, 130 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:52:19 (1734112339) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00843674 s, 2.4 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00372958 s, 5.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:52:21 (1734112341) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0102416 s, 2.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00251444 s, 8.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:52:23 (1734112343) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00204438 s, 8.0 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:52:25 (1734112345) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00833323 s, 2.5 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:52:26 (1734112346) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00146604 s, 2.8 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00607111 s, 675 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:52:28 (1734112348) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:52:32 (1734112352) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:52:50 (1734112370) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x48:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:53:07 (1734112387) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:53:24 (1734112404) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 12:53:26 (1734112406) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 26256 mirror split pid 26257 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 26256 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 26257 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 12:54:27 (1734112467) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 2468 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read skipping zero size read 000003 1734112468.256098 mapwrite 0x03b15d thru 0x03c5f2 (0x1496 bytes) 000005 1734112468.269954 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x1acfc 000006 1734112468.347578 trunc from 0x03c5f3 to 0x01acfb (0x218f7 bytes) 000007 1734112468.359860 mapwrite 0x0015e3 thru 0x00ac80 (0x969e bytes) 000008 1734112468.370093 punch from 0x000561 to 0x002481 (0x1f21 bytes) 000009 1734112468.371516 punch from 0x001568 to 0x007315 (0x5dae bytes) 000010 1734112468.372819 punch from 0x013d6d to 0x01acfb (0x6f8f bytes) fallocating to largest ever: 0x29928 000011 1734112468.374341 fallocate 0x0291b3 thru 0x029927 (0x775 bytes) 000012 1734112468.376313 punch from 0x0205b6 to 0x02412a (0x3b75 bytes) 000013 1734112468.377895 mapwrite 0x00b2cf thru 0x011305 (0x6037 bytes) 000014 1734112468.386724 MIRROR_EXTEND to 3 mirrors 000016 1734112468.530111 mapread 0x02618a thru 0x029927 (0x379e bytes) 000017 1734112468.545665 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) truncating to largest ever: 0x3f98a 000019 1734112468.630056 trunc from 0x029928 to 0x03f989 (0x16062 bytes) 000020 1734112468.643142 read 0x01393d thru 0x01ae59 (0x751d bytes) 000022 1734112468.646227 fallocate 0x02e4de thru 0x03ad38 (0xc85b bytes) 000023 1734112468.647708 read 0x024c8f thru 0x02a672 (0x59e4 bytes) 000024 1734112468.651179 mapread 0x0030fa thru 0x00e2b2 (0xb1b9 bytes) 000025 1734112468.651690 punch from 0x006f6d to 0x0167b9 (0xf84d bytes) 000027 1734112468.653568 fallocate 0x011584 thru 0x012804 (0x1281 bytes) 000028 1734112468.657004 MIRROR_RESYNC 2 mirrors 000029 1734112468.746263 read 0x032be5 thru 0x03ca4d (0x9e69 bytes) 000031 1734112468.748618 mapwrite 0x00196e thru 0x002500 (0xb93 bytes) 000032 1734112468.778686 MIRROR_EXTEND to 3 mirrors 000033 1734112468.914388 fallocate 0x00b796 thru 0x017aef (0xc35a bytes) 000035 1734112468.922453 write 0x02544a thru 0x02bf1f (0x6ad6 bytes) 000036 1734112468.926214 mapwrite 0x01e69e thru 0x029b67 (0xb4ca bytes) 000037 1734112468.934601 punch from 0x01fdf0 to 0x023099 (0x32aa bytes) 000038 1734112468.935961 read 0x03efed thru 0x03f989 (0x99d bytes) 000039 1734112468.938701 MIRROR_RESYNC 3 mirrors 000040 1734112469.100686 trunc from 0x03f98a to 0x00db01 (0x31e88 bytes) 000042 1734112469.109955 trunc from 0x00db02 to 0x02d8c3 (0x1fdc2 bytes) 000043 1734112469.114863 mapread 0x010025 thru 0x0197cd (0x97a9 bytes) 000044 1734112469.119470 read 0x02cfb2 thru 0x02d8c3 (0x912 bytes) 000045 1734112469.120980 trunc from 0x02d8c4 to 0x03ba5a (0xe197 bytes) 000046 1734112469.125078 punch from 0x020434 to 0x020dff (0x9cc bytes) 000047 1734112469.126660 write 0x02f1c5 thru 0x0356e8 (0x6524 bytes) 000048 1734112469.129563 punch from 0x00548b to 0x014c63 (0xf7d9 bytes) 000049 1734112469.132545 MIRROR_RESYNC 3 mirrors 000050 1734112469.248775 mapwrite 0x004025 thru 0x005faf (0x1f8b bytes) 000051 1734112469.281589 mapwrite 0x0361a3 thru 0x03ffff (0x9e5d bytes) 000052 1734112469.300856 mapread 0x020c52 thru 0x02dcc3 (0xd072 bytes) 000053 1734112469.306572 punch from 0x015b41 to 0x01e208 (0x86c8 bytes) 000054 1734112469.308554 mapread 0x022321 thru 0x028603 (0x62e3 bytes) 000055 1734112469.308994 read 0x02c607 thru 0x036c2c (0xa626 bytes) 000056 1734112469.309354 write 0x00cea3 thru 0x0140d7 (0x7235 bytes) 000057 1734112469.311170 write 0x0311df thru 0x03c55c (0xb37e bytes) 000058 1734112469.311899 mapread 0x00abc2 thru 0x00cffa (0x2439 bytes) 000060 1734112469.312585 mapwrite 0x022119 thru 0x031671 (0xf559 bytes) 000061 1734112469.319756 punch from 0x0177be to 0x019d55 (0x2598 bytes) 000062 1734112469.321289 trunc from 0x040000 to 0x03611b (0x9ee4 bytes) 000063 1734112469.326603 read 0x01d333 thru 0x02c00e (0xecdc bytes) 000065 1734112469.329682 mapread 0x02451d thru 0x034014 (0xfaf8 bytes) 000066 1734112469.331983 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000067 1734112469.380881 fallocate 0x022dc0 thru 0x02842f (0x5670 bytes) 000068 1734112469.382334 trunc from 0x03611c to 0x028b50 (0xd5cb bytes) 000069 1734112469.386873 punch from 0x01fbc1 to 0x028b50 (0x8f90 bytes) 000070 1734112469.388404 punch from 0x0190ad to 0x023607 (0xa55b bytes) 000071 1734112469.390884 MIRROR_EXTEND to 3 mirrors 000073 1734112469.505672 mapwrite 0x02dcfa thru 0x0302b3 (0x25ba bytes) 000075 1734112469.531216 punch from 0x00b56f to 0x01a1b4 (0xec46 bytes) 000078 1734112469.533626 mapwrite 0x000b86 thru 0x00311a (0x2595 bytes) 000079 1734112469.541023 mapread 0x00b7d1 thru 0x012e1f (0x764f bytes) 000080 1734112469.543502 mapwrite 0x01c6ef thru 0x026b79 (0xa48b bytes) 000082 1734112469.552754 fallocate 0x03381c thru 0x038b23 (0x5308 bytes) 000083 1734112469.554651 mapwrite 0x018bae thru 0x024c28 (0xc07b bytes) 000085 1734112469.561230 mapread 0x018411 thru 0x0220b6 (0x9ca6 bytes) 000086 1734112469.561904 fallocate 0x026dbd thru 0x03337a (0xc5be bytes) 000088 1734112469.564311 write 0x008d86 thru 0x014abe (0xbd39 bytes) 000090 1734112469.565666 mapwrite 0x02d0a9 thru 0x038ea9 (0xbe01 bytes) 000091 1734112469.586070 write 0x03e4eb thru 0x03ffff (0x1b15 bytes) 000092 1734112469.586710 mapwrite 0x030abc thru 0x03de53 (0xd398 bytes) 000094 1734112469.596049 write 0x032059 thru 0x033cc9 (0x1c71 bytes) 000095 1734112469.598863 MIRROR_EXTEND to 4 mirrors 000096 1734112469.728302 MIRROR_RESYNC 4 mirrors 000097 1734112469.865213 MIRROR_EXTEND to 5 mirrors 000098 1734112469.985768 MIRROR_RESYNC 5 mirrors 000099 1734112470.048813 MIRROR_EXTEND to 6 mirrors 000100 1734112470.203985 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000102 1734112470.330025 punch from 0x01553e to 0x01a84f (0x5312 bytes) 000103 1734112470.339211 mapread 0x0161ac thru 0x016e72 (0xcc7 bytes) 000104 1734112470.340900 trunc from 0x040000 to 0x023637 (0x1c9c8 bytes) 000105 1734112470.349690 fallocate 0x0182fa thru 0x01d40d (0x5114 bytes) 000106 1734112470.351074 mapwrite 0x0228ab thru 0x03002c (0xd782 bytes) 000107 1734112470.368333 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000109 1734112470.418167 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000110 1734112470.462191 mapwrite 0x028485 thru 0x02eb56 (0x66d2 bytes) 000111 1734112470.471181 write 0x02ab97 thru 0x030de0 (0x624a bytes) 000112 1734112470.473084 write 0x032d06 thru 0x03bd67 (0x9062 bytes) 000114 1734112470.474005 punch from 0x036017 to 0x03bd67 (0x5d51 bytes) 000115 1734112470.477298 punch from 0x0376ef to 0x03bd67 (0x4679 bytes) 000116 1734112470.480312 MIRROR_EXTEND to 4 mirrors 000117 1734112470.598553 read 0x009345 thru 0x0101f6 (0x6eb2 bytes) 000119 1734112470.600892 punch from 0x0129e6 to 0x020d92 (0xe3ad bytes) 000120 1734112470.609286 write 0x01b7ee thru 0x025165 (0x9978 bytes) 000121 1734112470.613183 trunc from 0x03bd68 to 0x00ab1c (0x3124b bytes) 000122 1734112470.622326 punch from 0x003a3e to 0x00ab1c (0x70df bytes) 000124 1734112470.624532 trunc from 0x00ab1d to 0x02c8f4 (0x21dd8 bytes) 000126 1734112470.628976 trunc from 0x02c8f5 to 0x011014 (0x1b8e0 bytes) 000127 1734112470.634384 MIRROR_RESYNC 4 mirrors 000128 1734112470.784268 MIRROR_RESYNC 4 mirrors 000129 1734112470.823827 mapread 0x00413e thru 0x010b09 (0xc9cc bytes) 000130 1734112470.830514 mapwrite 0x01137d thru 0x0161a2 (0x4e26 bytes) 000131 1734112470.851227 fallocate 0x026615 thru 0x0350f1 (0xeadd bytes) 000133 1734112470.853062 mapread 0x00b438 thru 0x00bcf7 (0x8c0 bytes) 000134 1734112470.855513 mapread 0x000a64 thru 0x001008 (0x5a5 bytes) 000136 1734112470.857493 trunc from 0x0350f2 to 0x00ff3f (0x251b2 bytes) 000137 1734112470.862797 fallocate 0x026474 thru 0x02b60c (0x5199 bytes) 000138 1734112470.864519 trunc from 0x02b60d to 0x03dd0f (0x12703 bytes) 000139 1734112470.868834 punch from 0x037a48 to 0x03dd0f (0x62c8 bytes) 000140 1734112470.870413 fallocate 0x0153ad thru 0x017d70 (0x29c4 bytes) 000142 1734112470.873797 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000143 1734112470.921906 trunc from 0x03dd10 to 0x020b5e (0x1d1b1 bytes) 000144 1734112470.926282 write 0x031764 thru 0x03ffff (0xe89c bytes) 000145 1734112470.927364 mapwrite 0x0216f1 thru 0x02fac8 (0xe3d8 bytes) 000146 1734112470.943039 trunc from 0x040000 to 0x01fee5 (0x2011a bytes) 000147 1734112470.947412 mapread 0x0162fa thru 0x01fee5 (0x9bec bytes) 000148 1734112470.950232 write 0x02acec thru 0x037048 (0xc35d bytes) 000149 1734112470.951059 write 0x03d48a thru 0x03ffff (0x2b76 bytes) 000150 1734112470.951528 fallocate 0x02b7e9 thru 0x0337d1 (0x7fe9 bytes) 000152 1734112470.953689 mapwrite 0x010fa5 thru 0x013add (0x2b39 bytes) 000153 1734112470.967582 trunc from 0x040000 to 0x021bf8 (0x1e407 bytes) 000154 1734112470.972311 read 0x01fac5 thru 0x021bf8 (0x2134 bytes) 000155 1734112470.974060 fallocate 0x023152 thru 0x026862 (0x3711 bytes) 000156 1734112470.977891 MIRROR_EXTEND to 4 mirrors 000157 1734112471.121976 MIRROR_EXTEND to 5 mirrors 000158 1734112471.248516 write 0x01049b thru 0x01dcdb (0xd841 bytes) 000159 1734112471.260648 mapwrite 0x01c7af thru 0x01d192 (0x9e4 bytes) 000160 1734112471.266942 trunc from 0x021bf9 to 0x019bfa (0x7ffe bytes) 000161 1734112471.272016 mapwrite 0x0185b9 thru 0x019c2f (0x1677 bytes) 000162 1734112471.280312 trunc from 0x019c30 to 0x00d001 (0xcc2e bytes) 000163 1734112471.284329 trunc from 0x00d002 to 0x030446 (0x23445 bytes) 000165 1734112471.288291 read 0x0051ce thru 0x0054c7 (0x2fa bytes) 000168 1734112471.291656 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000170 1734112471.335967 read 0x01316e thru 0x01e965 (0xb7f8 bytes) 000171 1734112471.338539 fallocate 0x01fff3 thru 0x027a99 (0x7aa7 bytes) 000172 1734112471.340293 write 0x020e95 thru 0x02d95e (0xcaca bytes) 000173 1734112471.344134 write 0x0132b7 thru 0x020326 (0xd070 bytes) 000174 1734112471.344891 mapwrite 0x010784 thru 0x01cc2a (0xc4a7 bytes) 000175 1734112471.353444 mapread 0x017ab0 thru 0x024ab5 (0xd006 bytes) 000176 1734112471.353932 punch from 0x0098c6 to 0x0105c0 (0x6cfb bytes) 000177 1734112471.355305 mapwrite 0x017c6c thru 0x01ad7e (0x3113 bytes) 000178 1734112471.358859 write 0x022f48 thru 0x031e22 (0xeedb bytes) 000179 1734112471.359636 read 0x0077fc thru 0x0145f6 (0xcdfb bytes) 000180 1734112471.361976 mapwrite 0x01553e thru 0x0210ea (0xbbad bytes) 000181 1734112471.367933 trunc from 0x031e23 to 0x033dd8 (0x1fb6 bytes) 000182 1734112471.371882 write 0x020bc9 thru 0x023792 (0x2bca bytes) 000184 1734112471.372865 write 0x012b3d thru 0x0149e0 (0x1ea4 bytes) 000186 1734112471.373513 mapwrite 0x0204c4 thru 0x02ff1e (0xfa5b bytes) 000187 1734112471.380352 write 0x03bb9f thru 0x03ffff (0x4461 bytes) 000188 1734112471.380971 mapread 0x00a462 thru 0x00c624 (0x21c3 bytes) 000189 1734112471.381355 punch from 0x036fc5 to 0x03b517 (0x4553 bytes) 000190 1734112471.384620 fallocate 0x021e72 thru 0x02ad88 (0x8f17 bytes) 000191 1734112471.386105 mapwrite 0x00a57a thru 0x011e46 (0x78cd bytes) 000192 1734112471.391060 read 0x028621 thru 0x02e007 (0x59e7 bytes) 000193 1734112471.391435 mapwrite 0x02caa4 thru 0x03bc98 (0xf1f5 bytes) 000194 1734112471.399516 mapread 0x00d1e1 thru 0x017674 (0xa494 bytes) 000195 1734112471.400021 mapwrite 0x031677 thru 0x039fc0 (0x894a bytes) 000196 1734112471.404750 fallocate 0x013e9f thru 0x018bcd (0x4d2f bytes) 000200 1734112471.406756 mapread 0x02db14 thru 0x033375 (0x5862 bytes) 000201 1734112471.407167 write 0x03b7f3 thru 0x03ffff (0x480d bytes) 000202 1734112471.409015 MIRROR_RESYNC 4 mirrors 000204 1734112471.556079 write 0x03e112 thru 0x03f407 (0x12f6 bytes) 000205 1734112471.568639 mapwrite 0x03f2c7 thru 0x03ffff (0xd39 bytes) 000206 1734112471.574789 mapread 0x0048cc thru 0x005aea (0x121f bytes) 000207 1734112471.577860 mapwrite 0x0081d1 thru 0x009530 (0x1360 bytes) 000208 1734112471.582664 read 0x0169dc thru 0x0220c1 (0xb6e6 bytes) 000210 1734112471.585836 read 0x02d494 thru 0x033a13 (0x6580 bytes) 000211 1734112471.589489 read 0x0013e2 thru 0x006021 (0x4c40 bytes) 000213 1734112471.590069 write 0x03cf92 thru 0x03ffff (0x306e bytes) 000214 1734112471.590727 fallocate 0x0168f1 thru 0x018b48 (0x2258 bytes) 000215 1734112471.592633 punch from 0x022adf to 0x031f86 (0xf4a8 bytes) 000216 1734112471.594856 read 0x034a93 thru 0x03ffff (0xb56d bytes) 000217 1734112471.595339 write 0x00370b thru 0x0090e7 (0x59dd bytes) 000218 1734112471.596118 write 0x03bf12 thru 0x03ffff (0x40ee bytes) 000220 1734112471.596944 fallocate 0x0279d7 thru 0x0293d4 (0x19fe bytes) 000222 1734112471.599898 MIRROR_EXTEND to 5 mirrors 000224 1734112471.732262 mapread 0x03dcfe thru 0x03ffff (0x2302 bytes) 000227 1734112471.743846 punch from 0x03812d to 0x03ffff (0x7ed3 bytes) 000228 1734112471.754405 MIRROR_EXTEND to 6 mirrors 000229 1734112471.877203 MIRROR_EXTEND to 7 mirrors 000230 1734112471.997657 MIRROR_EXTEND to 8 mirrors 000231 1734112472.123493 write 0x02392a thru 0x02e022 (0xa6f9 bytes) 000233 1734112472.142758 mapread 0x001b67 thru 0x006329 (0x47c3 bytes) 000234 1734112472.146750 MIRROR_EXTEND to 9 mirrors 000236 1734112472.278253 read 0x00a09a thru 0x0170a4 (0xd00b bytes) 000239 1734112472.285778 fallocate 0x0150fe thru 0x01ef78 (0x9e7b bytes) 000240 1734112472.294565 read 0x0346cf thru 0x03ffff (0xb931 bytes) 000241 1734112472.297892 fallocate 0x027221 thru 0x02f399 (0x8179 bytes) 000242 1734112472.299441 write 0x0143ce thru 0x0150bc (0xcef bytes) 000243 1734112472.301089 fallocate 0x0262cf thru 0x02a428 (0x415a bytes) 000245 1734112472.302869 mapwrite 0x03322c thru 0x034f61 (0x1d36 bytes) 000246 1734112472.308289 mapread 0x0293fd thru 0x0356f7 (0xc2fb bytes) 000247 1734112472.308794 mapwrite 0x0056d3 thru 0x00d257 (0x7b85 bytes) fallocating to largest ever: 0x40000 000248 1734112472.313852 fallocate 0x03eae6 thru 0x03ffff (0x151a bytes) 000249 1734112472.315599 write 0x006247 thru 0x00c277 (0x6031 bytes) 000250 1734112472.316428 write 0x02f77e thru 0x030728 (0xfab bytes) 000251 1734112472.317174 mapwrite 0x008c89 thru 0x014000 (0xb378 bytes) 000252 1734112472.322982 mapwrite 0x00da8f thru 0x01a1a5 (0xc717 bytes) 000253 1734112472.329186 mapread 0x01b6a1 thru 0x0292fd (0xdc5d bytes) 000254 1734112472.329674 trunc from 0x040000 to 0x01ab4e (0x254b1 bytes) 000255 1734112472.338674 fallocate 0x0027a1 thru 0x00901b (0x687b bytes) 000256 1734112472.355545 MIRROR_RESYNC 9 mirrors 000257 1734112472.524544 punch from 0x01a604 to 0x01ab4e (0x54b bytes) 000260 1734112472.533407 trunc from 0x01ab4f to 0x00ecf6 (0xbe58 bytes) 000261 1734112472.541673 trunc from 0x00ecf7 to 0x03df2b (0x2f235 bytes) 000262 1734112472.546172 write 0x02a362 thru 0x02b33c (0xfdb bytes) 000263 1734112472.549092 trunc from 0x03df2c to 0x01eaf1 (0x1f43a bytes) 000264 1734112472.553412 mapread 0x014543 thru 0x01d0ff (0x8bbd bytes) 000266 1734112472.559416 trunc from 0x01eaf2 to 0x00852a (0x165c7 bytes) 000267 1734112472.564149 fallocate 0x032fab thru 0x037afd (0x4b53 bytes) 000268 1734112472.565668 mapread 0x0007b4 thru 0x00852a (0x7d77 bytes) 000270 1734112472.569602 read 0x0026b6 thru 0x00852a (0x5e75 bytes) 000273 1734112472.570415 read 0x0000dc thru 0x00852a (0x844f bytes) 000274 1734112472.570735 mapread 0x003df9 thru 0x00852a (0x4732 bytes) 000277 1734112472.571815 punch from 0x0031b9 to 0x00852a (0x5372 bytes) 000278 1734112472.575328 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000280 1734112472.624460 read 0x004beb thru 0x006a29 (0x1e3f bytes) 000281 1734112472.626157 fallocate 0x02bdb2 thru 0x030a10 (0x4c5f bytes) 000282 1734112472.627573 write 0x001bab thru 0x001f51 (0x3a7 bytes) 000284 1734112472.630498 fallocate 0x03fb0b thru 0x03ffff (0x4f5 bytes) 000285 1734112472.632399 mapread 0x03a778 thru 0x03ffff (0x5888 bytes) 000287 1734112472.636047 mapwrite 0x00dbc8 thru 0x01d019 (0xf452 bytes) 000289 1734112472.646770 read 0x015e7c thru 0x022c09 (0xcd8e bytes) 000290 1734112472.649200 write 0x019dd0 thru 0x0278ec (0xdb1d bytes) 000291 1734112472.649921 fallocate 0x00eb43 thru 0x0147bb (0x5c79 bytes) 000292 1734112472.651744 mapread 0x02f3b9 thru 0x0386cd (0x9315 bytes) 000294 1734112472.652450 punch from 0x010692 to 0x01c7ee (0xc15d bytes) 000295 1734112472.657637 MIRROR_RESYNC 8 mirrors 000296 1734112473.054130 mapread 0x03f1ef thru 0x03ffff (0xe11 bytes) 000297 1734112473.055704 write 0x017056 thru 0x024bf1 (0xdb9c bytes) 000299 1734112473.065358 punch from 0x0354aa to 0x03a792 (0x52e9 bytes) 000300 1734112473.066635 read 0x03a81f thru 0x03ffff (0x57e1 bytes) 000301 1734112473.068745 mapread 0x00429d thru 0x01126f (0xcfd3 bytes) 000302 1734112473.072926 mapread 0x035b99 thru 0x037c11 (0x2079 bytes) 000303 1734112473.075521 read 0x037772 thru 0x03d8ed (0x617c bytes) 000304 1734112473.076039 punch from 0x033c83 to 0x03c8eb (0x8c69 bytes) 000305 1734112473.077534 mapwrite 0x00471a thru 0x00c990 (0x8277 bytes) 000306 1734112473.084410 mapread 0x00f739 thru 0x0117b4 (0x207c bytes) 000307 1734112473.085022 write 0x025a5c thru 0x027ad5 (0x207a bytes) 000308 1734112473.088158 mapwrite 0x03d6ab thru 0x03e409 (0xd5f bytes) 000309 1734112473.093165 mapread 0x029948 thru 0x038150 (0xe809 bytes) 000312 1734112473.094838 punch from 0x005dce to 0x00922c (0x345f bytes) 000313 1734112473.096678 write 0x02e448 thru 0x0346df (0x6298 bytes) 000314 1734112473.097268 write 0x001b87 thru 0x00d524 (0xb99e bytes) 000315 1734112473.098300 mapread 0x033eb5 thru 0x03ffff (0xc14b bytes) 000316 1734112473.098916 trunc from 0x040000 to 0x03af49 (0x50b6 bytes) 000317 1734112473.109090 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000318 1734112473.158393 mapwrite 0x01c8b5 thru 0x02c0c7 (0xf813 bytes) 000319 1734112473.171841 mapread 0x036d4a thru 0x03af49 (0x4200 bytes) 000320 1734112473.174169 punch from 0x01b197 to 0x0225c5 (0x742f bytes) 000321 1734112473.175455 trunc from 0x03af4a to 0x030674 (0xa8d5 bytes) 000322 1734112473.180014 trunc from 0x030675 to 0x01a479 (0x161fb bytes) 000323 1734112473.183720 fallocate 0x03c1b9 thru 0x03ffff (0x3e47 bytes) 000324 1734112473.185112 fallocate 0x003b3b thru 0x00530f (0x17d5 bytes) 000325 1734112473.186498 punch from 0x00b620 to 0x00c9cc (0x13ad bytes) 000326 1734112473.189544 MIRROR_RESYNC 7 mirrors 000327 1734112473.362891 fallocate 0x0313d3 thru 0x034174 (0x2da2 bytes) 000328 1734112473.371114 mapread 0x010d1f thru 0x012761 (0x1a43 bytes) 000330 1734112473.375759 fallocate 0x0346eb thru 0x036e96 (0x27ac bytes) 000331 1734112473.377196 punch from 0x003fbc to 0x013dea (0xfe2f bytes) 000333 1734112473.378930 mapread 0x0016de thru 0x006b6d (0x5490 bytes) 000334 1734112473.381433 mapread 0x00bef1 thru 0x01a479 (0xe589 bytes) 000336 1734112473.384973 read 0x00d86d thru 0x013f55 (0x66e9 bytes) 000338 1734112473.385471 punch from 0x018e2e to 0x01a479 (0x164c bytes) 000340 1734112473.387439 fallocate 0x00205c thru 0x00d873 (0xb818 bytes) 000341 1734112473.389140 punch from 0x00b27b to 0x00c3b9 (0x113f bytes) 000343 1734112473.390628 fallocate 0x015fe2 thru 0x01b2d5 (0x52f4 bytes) 000344 1734112473.392197 trunc from 0x01a47a to 0x0259cb (0xb552 bytes) 000345 1734112473.399530 write 0x026db0 thru 0x02dff1 (0x7242 bytes) 000346 1734112473.400191 fallocate 0x02bbba thru 0x035905 (0x9d4c bytes) 000347 1734112473.401569 trunc from 0x035906 to 0x029984 (0xbf81 bytes) 000348 1734112473.405104 read 0x0034d4 thru 0x00e1ff (0xad2c bytes) 000349 1734112473.407796 punch from 0x01bff2 to 0x01e5b7 (0x25c6 bytes) 000350 1734112473.410576 MIRROR_RESYNC 7 mirrors 000351 1734112473.661829 read 0x00a65a thru 0x01454f (0x9ef6 bytes) 000353 1734112473.664486 write 0x039663 thru 0x03d1b9 (0x3b57 bytes) 000355 1734112473.671757 mapwrite 0x02ac15 thru 0x0309eb (0x5dd7 bytes) 000356 1734112473.682104 punch from 0x01ca31 to 0x02135d (0x492d bytes) 000357 1734112473.683687 fallocate 0x031bcd thru 0x03ffff (0xe433 bytes) 000358 1734112473.685123 mapwrite 0x015ff9 thru 0x01c2f5 (0x62fd bytes) 000360 1734112473.693051 fallocate 0x015f2e thru 0x01adf6 (0x4ec9 bytes) 000362 1734112473.695943 mapread 0x01e677 thru 0x029535 (0xaebf bytes) 000365 1734112473.698670 read 0x03c68c thru 0x03d1b9 (0xb2e bytes) 000366 1734112473.698942 trunc from 0x03d1ba to 0x039670 (0x3b49 bytes) 000367 1734112473.707390 mapread 0x01a22d thru 0x01b8fc (0x16d0 bytes) 000368 1734112473.707851 write 0x03c4ac thru 0x03cdc2 (0x917 bytes) 000369 1734112473.708315 fallocate 0x01286b thru 0x017bea (0x5380 bytes) 000370 1734112473.709816 trunc from 0x03cdc3 to 0x022c8e (0x1a134 bytes) 000371 1734112473.713768 read 0x0210f5 thru 0x022c8e (0x1b9a bytes) 000373 1734112473.714260 trunc from 0x022c8f to 0x000a5e (0x22230 bytes) 000375 1734112473.718289 punch from 0x000439 to 0x000a5e (0x626 bytes) 000376 1734112473.719566 mapwrite 0x0276e5 thru 0x02b4a0 (0x3dbc bytes) 000377 1734112473.731870 read 0x025ef0 thru 0x02a0b8 (0x41c9 bytes) 000378 1734112473.732217 punch from 0x01b357 to 0x023844 (0x84ee bytes) 000379 1734112473.733711 trunc from 0x02b4a1 to 0x004beb (0x268b5 bytes) 000380 1734112473.737743 mapwrite 0x01090b thru 0x01e3b1 (0xdaa7 bytes) 000381 1734112473.750562 trunc from 0x01e3b2 to 0x01ac13 (0x379e bytes) 000382 1734112473.755336 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000383 1734112473.800878 mapread 0x0006bf thru 0x000990 (0x2d2 bytes) 000384 1734112473.802796 mapwrite 0x03cc57 thru 0x03ffff (0x33a9 bytes) 000385 1734112473.816714 trunc from 0x040000 to 0x0070a6 (0x38f59 bytes) 000386 1734112473.820753 write 0x009ced thru 0x00b4f5 (0x1809 bytes) 000388 1734112473.821486 fallocate 0x017c7f thru 0x01d642 (0x59c4 bytes) 000389 1734112473.823055 mapread 0x016c32 thru 0x01d642 (0x6a11 bytes) 000393 1734112473.827131 punch from 0x008855 to 0x012715 (0x9ec1 bytes) 000394 1734112473.830059 mapwrite 0x023f0e thru 0x029109 (0x51fc bytes) 000395 1734112473.841792 read 0x0285e5 thru 0x029109 (0xb25 bytes) 000397 1734112473.842309 read 0x019039 thru 0x028acb (0xfa93 bytes) 000398 1734112473.843975 MIRROR_RESYNC 6 mirrors 000400 1734112474.004078 write 0x031c66 thru 0x0348ad (0x2c48 bytes) 000401 1734112474.012180 trunc from 0x0348ae to 0x037cf9 (0x344c bytes) 000402 1734112474.020613 mapwrite 0x0272c3 thru 0x034378 (0xd0b6 bytes) 000403 1734112474.032076 write 0x02f4ac thru 0x0327c5 (0x331a bytes) 000404 1734112474.032667 fallocate 0x01d65b thru 0x029b9a (0xc540 bytes) 000406 1734112474.034353 write 0x01d1ec thru 0x026fd6 (0x9deb bytes) 000407 1734112474.036924 trunc from 0x037cfa to 0x017060 (0x20c99 bytes) 000408 1734112474.040764 read 0x001111 thru 0x010c37 (0xfb27 bytes) 000409 1734112474.043021 fallocate 0x00848f thru 0x011b01 (0x9673 bytes) 000410 1734112474.044441 punch from 0x0094df to 0x00a9d9 (0x14fb bytes) 000411 1734112474.045669 trunc from 0x017061 to 0x03d6cb (0x2666b bytes) 000412 1734112474.049078 mapread 0x026f20 thru 0x02b680 (0x4761 bytes) 000413 1734112474.051574 read 0x026a24 thru 0x02caad (0x608a bytes) 000414 1734112474.051880 trunc from 0x03d6cc to 0x03b5e7 (0x20e4 bytes) 000415 1734112474.055336 mapwrite 0x001326 thru 0x00c26b (0xaf46 bytes) 000416 1734112474.063852 fallocate 0x012c6e thru 0x01737d (0x4710 bytes) 000417 1734112474.065344 mapwrite 0x018663 thru 0x023c6a (0xb608 bytes) 000418 1734112474.070515 punch from 0x0338c7 to 0x03b5e7 (0x7d21 bytes) 000419 1734112474.071949 fallocate 0x02d8d1 thru 0x038a25 (0xb155 bytes) 000421 1734112474.073840 mapread 0x012096 thru 0x013987 (0x18f2 bytes) 000422 1734112474.074432 fallocate 0x01e1b6 thru 0x02c234 (0xe07f bytes) 000423 1734112474.076000 punch from 0x0326cf to 0x03ad30 (0x8662 bytes) 000425 1734112474.077720 punch from 0x023293 to 0x02cc04 (0x9972 bytes) 000426 1734112474.079184 mapread 0x03a55a thru 0x03b5e7 (0x108e bytes) 000427 1734112474.082011 write 0x0182b8 thru 0x022bac (0xa8f5 bytes) 000428 1734112474.082706 read 0x00ebed thru 0x01ca8f (0xdea3 bytes) 000429 1734112474.083040 mapread 0x017a88 thru 0x0201c5 (0x873e bytes) 000430 1734112474.083483 mapwrite 0x031669 thru 0x03d199 (0xbb31 bytes) 000431 1734112474.095850 write 0x00e4a9 thru 0x00f33d (0xe95 bytes) 000433 1734112474.096571 mapwrite 0x01a3e3 thru 0x0273c6 (0xcfe4 bytes) 000434 1734112474.102267 punch from 0x0039b1 to 0x008980 (0x4fd0 bytes) 000435 1734112474.104038 read 0x002408 thru 0x006fe8 (0x4be1 bytes) 000436 1734112474.106055 punch from 0x017676 to 0x01cdcc (0x5757 bytes) 000438 1734112474.108685 MIRROR_EXTEND to 7 mirrors 000439 1734112474.215715 mapwrite 0x006522 thru 0x0083fb (0x1eda bytes) 000440 1734112474.242269 fallocate 0x0042e0 thru 0x010d2a (0xca4b bytes) 000441 1734112474.243716 mapwrite 0x018bdf thru 0x02267a (0x9a9c bytes) 000442 1734112474.252541 write 0x010e74 thru 0x012d34 (0x1ec1 bytes) 000443 1734112474.254415 trunc from 0x03d19a to 0x0074be (0x35cdb bytes) 000446 1734112474.264028 read 0x006111 thru 0x0074be (0x13ae bytes) 000447 1734112474.264507 trunc from 0x0074bf to 0x00277e (0x4d40 bytes) 000448 1734112474.268027 trunc from 0x00277f to 0x01dbf6 (0x1b478 bytes) 000449 1734112474.271557 mapwrite 0x0209cd thru 0x02e26e (0xd8a2 bytes) 000450 1734112474.285626 read 0x0122ef thru 0x01f2b9 (0xcfcb bytes) 000452 1734112474.290167 MIRROR_RESYNC 7 mirrors 000453 1734112474.455914 fallocate 0x01ba5a thru 0x027bf6 (0xc19d bytes) 000454 1734112474.464881 mapwrite 0x00de64 thru 0x012372 (0x450f bytes) 000455 1734112474.474712 write 0x027d0b thru 0x03650a (0xe800 bytes) 000458 1734112474.477983 mapread 0x017d94 thru 0x0224dc (0xa749 bytes) 000459 1734112474.480243 fallocate 0x035aef thru 0x037a28 (0x1f3a bytes) 000460 1734112474.482100 fallocate 0x0311f8 thru 0x03ac35 (0x9a3e bytes) 000461 1734112474.483798 punch from 0x021cc2 to 0x029672 (0x79b1 bytes) 000463 1734112474.487914 mapwrite 0x033e1f thru 0x037a50 (0x3c32 bytes) 000465 1734112474.493728 mapwrite 0x005cb2 thru 0x0145f9 (0xe948 bytes) 000466 1734112474.500763 fallocate 0x0335af thru 0x037138 (0x3b8a bytes) 000467 1734112474.502311 write 0x0244cd thru 0x027792 (0x32c6 bytes) 000468 1734112474.504760 mapwrite 0x02df5c thru 0x036a50 (0x8af5 bytes) 000469 1734112474.510690 trunc from 0x03ac36 to 0x01d01c (0x1dc19 bytes) 000470 1734112474.519211 punch from 0x0108b7 to 0x016324 (0x5a6e bytes) 000471 1734112474.520587 punch from 0x002f30 to 0x010137 (0xd208 bytes) 000472 1734112474.521991 trunc from 0x01d01d to 0x01b477 (0x1ba5 bytes) 000473 1734112474.525761 write 0x007303 thru 0x00a018 (0x2d16 bytes) 000474 1734112474.528862 trunc from 0x01b478 to 0x01371a (0x7d5d bytes) 000475 1734112474.532705 fallocate 0x018b45 thru 0x02000f (0x74cb bytes) 000477 1734112474.534398 trunc from 0x020010 to 0x01b9f1 (0x461e bytes) 000478 1734112474.538040 punch from 0x00cdaa to 0x00ea1f (0x1c76 bytes) 000479 1734112474.539528 mapwrite 0x004f4c thru 0x0072bf (0x2374 bytes) 000480 1734112474.545939 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000481 1734112474.593309 read 0x0187c4 thru 0x01b9f1 (0x322e bytes) 000482 1734112474.595314 trunc from 0x01b9f2 to 0x00b144 (0x108ad bytes) 000483 1734112474.599401 punch from 0x002f07 to 0x009db5 (0x6eaf bytes) 000486 1734112474.601290 mapwrite 0x035bac thru 0x0382e2 (0x2737 bytes) 000487 1734112474.614133 mapread 0x019a3a thru 0x01a762 (0xd29 bytes) 000488 1734112474.616918 fallocate 0x00e4a6 thru 0x017751 (0x92ac bytes) 000489 1734112474.618498 read 0x037cd8 thru 0x0382e2 (0x60b bytes) 000491 1734112474.618993 trunc from 0x0382e3 to 0x03f9a8 (0x76c6 bytes) 000492 1734112474.622578 punch from 0x00b757 to 0x01aff9 (0xf8a3 bytes) 000493 1734112474.623951 mapread 0x035f3c thru 0x03db70 (0x7c35 bytes) 000494 1734112474.626766 fallocate 0x02b165 thru 0x036de6 (0xbc82 bytes) 000497 1734112474.628381 fallocate 0x005ff3 thru 0x00c34f (0x635d bytes) 000498 1734112474.629839 punch from 0x00000f to 0x00281f (0x2811 bytes) 000499 1734112474.631118 mapread 0x024fcb thru 0x033570 (0xe5a6 bytes) 000500 1734112474.632769 MIRROR_EXTEND to 7 mirrors 000502 1734112474.745494 punch from 0x0350df to 0x03f7b4 (0xa6d6 bytes) 000503 1734112474.754866 MIRROR_EXTEND to 8 mirrors 000504 1734112474.872285 mapwrite 0x014293 thru 0x02148a (0xd1f8 bytes) 000505 1734112474.906283 punch from 0x02b24e to 0x03874f (0xd502 bytes) 000506 1734112474.907844 punch from 0x00928d to 0x00bca3 (0x2a17 bytes) 000507 1734112474.909245 fallocate 0x017bcf thru 0x01aa3b (0x2e6d bytes) 000508 1734112474.910725 punch from 0x00b534 to 0x01ac33 (0xf700 bytes) 000509 1734112474.912277 fallocate 0x006014 thru 0x008c54 (0x2c41 bytes) 000510 1734112474.913834 mapwrite 0x019475 thru 0x01f86d (0x63f9 bytes) 000511 1734112474.921491 mapread 0x01fe93 thru 0x02cb58 (0xccc6 bytes) 000512 1734112474.923958 punch from 0x0204c9 to 0x02837a (0x7eb2 bytes) 000513 1734112474.925409 punch from 0x021ac6 to 0x02c0cb (0xa606 bytes) 000514 1734112474.926795 write 0x0043bf thru 0x0118c7 (0xd509 bytes) 000515 1734112474.932224 mapread 0x01de74 thru 0x021f40 (0x40cd bytes) 000516 1734112474.936597 MIRROR_EXTEND to 9 mirrors 000517 1734112475.070708 fallocate 0x03f8e5 thru 0x03ffff (0x71b bytes) 000518 1734112475.078315 punch from 0x003ca0 to 0x0083a8 (0x4709 bytes) 000519 1734112475.079655 trunc from 0x040000 to 0x0168d1 (0x2972e bytes) 000521 1734112475.085803 trunc from 0x0168d2 to 0x022cca (0xc3f9 bytes) 000522 1734112475.089410 mapread 0x01307d thru 0x01582d (0x27b1 bytes) 000524 1734112475.094936 write 0x00cffe thru 0x016758 (0x975b bytes) 000526 1734112475.098053 write 0x0102ee thru 0x01ec6c (0xe97f bytes) 000527 1734112475.101038 MIRROR_EXTEND to 10 mirrors 000528 1734112475.217609 punch from 0x00fe67 to 0x015882 (0x5a1c bytes) 000529 1734112475.226778 mapwrite 0x0271ac thru 0x036709 (0xf55e bytes) 000531 1734112475.247849 read 0x00f082 thru 0x0149c6 (0x5945 bytes) 000532 1734112475.251818 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000533 1734112475.297750 mapread 0x01d76f thru 0x023278 (0x5b0a bytes) 000537 1734112475.303752 fallocate 0x01383f thru 0x015228 (0x19ea bytes) 000538 1734112475.306894 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000539 1734112475.355162 mapwrite 0x00c715 thru 0x012223 (0x5b0f bytes) 000540 1734112475.364189 mapwrite 0x037d2b thru 0x03e5dc (0x68b2 bytes) 000541 1734112475.377950 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000542 1734112475.429825 read 0x017f0c thru 0x01fe16 (0x7f0b bytes) 000543 1734112475.432181 mapread 0x039909 thru 0x03e5dc (0x4cd4 bytes) 000544 1734112475.436573 mapwrite 0x037d92 thru 0x03addd (0x304c bytes) 000545 1734112475.440442 mapwrite 0x007bfe thru 0x00b22d (0x3630 bytes) 000546 1734112475.446748 mapread 0x01b5d4 thru 0x01e5a2 (0x2fcf bytes) 000548 1734112475.447396 punch from 0x00b33e to 0x016769 (0xb42c bytes) 000549 1734112475.449998 MIRROR_RESYNC 7 mirrors 000550 1734112475.886019 mapwrite 0x02ff76 thru 0x03f752 (0xf7dd bytes) 000551 1734112475.913543 trunc from 0x03f753 to 0x0384cd (0x7285 bytes) 000552 1734112475.918640 write 0x019ee4 thru 0x027878 (0xd995 bytes) 000554 1734112475.922948 mapread 0x000cda thru 0x008449 (0x7770 bytes) 000555 1734112475.926905 trunc from 0x0384ce to 0x03b4a4 (0x2fd7 bytes) 000556 1734112475.932690 MIRROR_EXTEND to 8 mirrors 000557 1734112476.073291 mapread 0x02597c thru 0x02970e (0x3d93 bytes) 000558 1734112476.079335 MIRROR_RESYNC 8 mirrors 000559 1734112476.445435 mapread 0x01687c thru 0x01d77e (0x6f03 bytes) 000560 1734112476.450880 fallocate 0x03085c thru 0x03a444 (0x9be9 bytes) 000561 1734112476.459651 trunc from 0x03b4a5 to 0x006173 (0x35331 bytes) 000563 1734112476.464837 fallocate 0x00f59f thru 0x0138cb (0x432d bytes) 000564 1734112476.466430 fallocate 0x00e461 thru 0x013bd1 (0x5771 bytes) 000565 1734112476.467861 fallocate 0x01085f thru 0x016732 (0x5ed4 bytes) 000566 1734112476.469514 mapread 0x00075c thru 0x000b24 (0x3c9 bytes) 000567 1734112476.472203 trunc from 0x016733 to 0x00758a (0xf1a8 bytes) 000568 1734112476.476284 trunc from 0x00758b to 0x02c59a (0x25010 bytes) 000570 1734112476.481002 trunc from 0x02c59b to 0x032f10 (0x6976 bytes) 000571 1734112476.485029 trunc from 0x032f11 to 0x0045ea (0x2e926 bytes) 000572 1734112476.488863 fallocate 0x00a428 thru 0x018367 (0xdf40 bytes) 000574 1734112476.490519 fallocate 0x01ab25 thru 0x027349 (0xc825 bytes) 000575 1734112476.491891 punch from 0x00fd78 to 0x010120 (0x3a9 bytes) 000577 1734112476.493590 punch from 0x015c10 to 0x018367 (0x2758 bytes) 000578 1734112476.496556 MIRROR_RESYNC 8 mirrors 000579 1734112476.870437 punch from 0x00f11e to 0x018367 (0x924a bytes) 000580 1734112476.879834 punch from 0x001c3f to 0x00c0ad (0xa46f bytes) 000581 1734112476.882917 MIRROR_RESYNC 8 mirrors 000582 1734112477.135992 trunc from 0x018368 to 0x026b72 (0xe80b bytes) 000583 1734112477.146123 mapread 0x022939 thru 0x026b72 (0x423a bytes) 000584 1734112477.151668 mapwrite 0x0164a4 thru 0x01cb3f (0x669c bytes) 000585 1734112477.160028 punch from 0x0208d6 to 0x022eeb (0x2616 bytes) 000587 1734112477.161652 punch from 0x00a2c6 to 0x017f7a (0xdcb5 bytes) 000588 1734112477.163055 fallocate 0x035307 thru 0x0359b6 (0x6b0 bytes) 000589 1734112477.164449 fallocate 0x032f79 thru 0x03c579 (0x9601 bytes) 000590 1734112477.165981 write 0x012168 thru 0x0183f5 (0x628e bytes) 000591 1734112477.168538 mapread 0x000ed0 thru 0x0047a1 (0x38d2 bytes) 000593 1734112477.172186 read 0x00f4c3 thru 0x01ee70 (0xf9ae bytes) 000595 1734112477.173114 punch from 0x01e8d5 to 0x02da0e (0xf13a bytes) 000596 1734112477.174557 mapwrite 0x02d298 thru 0x03a050 (0xcdb9 bytes) 000597 1734112477.183783 trunc from 0x03c57a to 0x02f70e (0xce6b bytes) 000598 1734112477.189426 write 0x0208d5 thru 0x02625c (0x5988 bytes) 000599 1734112477.192898 write 0x030047 thru 0x03262e (0x25e8 bytes) 000600 1734112477.193521 read 0x02e08d thru 0x03262e (0x45a2 bytes) 000601 1734112477.193961 read 0x02cafd thru 0x0322b4 (0x57b8 bytes) 000603 1734112477.194839 mapwrite 0x007607 thru 0x00f83b (0x8235 bytes) 000604 1734112477.204267 MIRROR_RESYNC 8 mirrors 000605 1734112477.579303 write 0x013b9f thru 0x01c29a (0x86fc bytes) 000607 1734112477.590127 mapwrite 0x0124d9 thru 0x01c891 (0xa3b9 bytes) 000608 1734112477.598450 mapread 0x01ed88 thru 0x01fee2 (0x115b bytes) 000609 1734112477.601183 mapwrite 0x02ab8e thru 0x02d162 (0x25d5 bytes) 000610 1734112477.607633 mapread 0x000c55 thru 0x0060cd (0x5479 bytes) 000611 1734112477.611252 punch from 0x018aa9 to 0x023061 (0xa5b9 bytes) 000612 1734112477.613111 fallocate 0x01b553 thru 0x028e0f (0xd8bd bytes) 000613 1734112477.614552 mapread 0x0048fb thru 0x00e467 (0x9b6d bytes) 000614 1734112477.615034 punch from 0x0114a1 to 0x016ede (0x5a3e bytes) 000615 1734112477.616571 read 0x005ac0 thru 0x00b2d1 (0x5812 bytes) 000616 1734112477.616879 punch from 0x00b79f to 0x014320 (0x8b82 bytes) 000619 1734112477.619172 read 0x025ae4 thru 0x029839 (0x3d56 bytes) 000620 1734112477.621097 MIRROR_EXTEND to 9 mirrors 000621 1734112477.745821 MIRROR_RESYNC 9 mirrors 000622 1734112478.122402 mapread 0x017384 thru 0x01aee2 (0x3b5f bytes) 000623 1734112478.127177 mapread 0x00cef6 thru 0x013014 (0x611f bytes) 000624 1734112478.130631 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000625 1734112478.204524 fallocate 0x0211f2 thru 0x02bb28 (0xa937 bytes) 000627 1734112478.214741 fallocate 0x010640 thru 0x0161ad (0x5b6e bytes) 000628 1734112478.216539 read 0x0286b2 thru 0x03262e (0x9f7d bytes) 000629 1734112478.218888 mapwrite 0x016450 thru 0x02567c (0xf22d bytes) 000630 1734112478.232316 mapwrite 0x01fcfe thru 0x02322b (0x352e bytes) 000631 1734112478.237872 read 0x029b36 thru 0x02ee26 (0x52f1 bytes) 000632 1734112478.238262 write 0x03410a thru 0x03415c (0x53 bytes) 000633 1734112478.240269 write 0x02c2a6 thru 0x02fd8e (0x3ae9 bytes) 000635 1734112478.241392 trunc from 0x03415d to 0x03368b (0xad1 bytes) 000636 1734112478.251510 MIRROR_RESYNC 8 mirrors 000637 1734112478.557052 mapwrite 0x01b8c7 thru 0x01c3f5 (0xb2f bytes) 000638 1734112478.585195 mapread 0x00c421 thru 0x019d77 (0xd957 bytes) 000639 1734112478.590354 write 0x0059ba thru 0x00de7b (0x84c2 bytes) 000641 1734112478.591397 mapwrite 0x001b0a thru 0x005e3f (0x4336 bytes) 000642 1734112478.599501 write 0x013b7f thru 0x01fc63 (0xc0e5 bytes) 000645 1734112478.600803 write 0x02d3b9 thru 0x03429d (0x6ee5 bytes) 000646 1734112478.603562 fallocate 0x00ea8f thru 0x01cb40 (0xe0b2 bytes) 000647 1734112478.606415 MIRROR_EXTEND to 9 mirrors 000649 1734112478.699076 mapread 0x015a90 thru 0x01d255 (0x77c6 bytes) 000650 1734112478.705731 mapwrite 0x00fb0d thru 0x01a98c (0xae80 bytes) 000651 1734112478.748424 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000652 1734112478.807502 mapwrite 0x03ef68 thru 0x03ffff (0x1098 bytes) 000653 1734112478.824487 fallocate 0x015391 thru 0x01e60a (0x927a bytes) 000655 1734112478.826193 mapwrite 0x0074f7 thru 0x00ebb9 (0x76c3 bytes) 000656 1734112478.834744 read 0x0316f3 thru 0x037168 (0x5a76 bytes) 000657 1734112478.838991 MIRROR_RESYNC 8 mirrors 000658 1734112479.062250 punch from 0x013fa6 to 0x0224ac (0xe507 bytes) 000659 1734112479.071779 punch from 0x02d248 to 0x03b212 (0xdfcb bytes) 000660 1734112479.073504 trunc from 0x040000 to 0x014395 (0x2bc6a bytes) 000661 1734112479.081604 mapread 0x006197 thru 0x008637 (0x24a1 bytes) 000662 1734112479.086169 read 0x009237 thru 0x014395 (0xb15f bytes) 000663 1734112479.088108 trunc from 0x014396 to 0x0113bb (0x2fda bytes) 000665 1734112479.093056 fallocate 0x01c7e4 thru 0x028736 (0xbf53 bytes) 000666 1734112479.094965 trunc from 0x0113bc to 0x032564 (0x211a9 bytes) 000667 1734112479.099674 write 0x02c501 thru 0x02c549 (0x49 bytes) 000668 1734112479.102456 fallocate 0x02d197 thru 0x039ba7 (0xca11 bytes) 000669 1734112479.104297 trunc from 0x032565 to 0x025bba (0xc9aa bytes) 000670 1734112479.109627 MIRROR_RESYNC 8 mirrors 000671 1734112479.305665 punch from 0x006b98 to 0x015e72 (0xf2db bytes) 000672 1734112479.313892 fallocate 0x02f0ab thru 0x034aaa (0x5a00 bytes) 000673 1734112479.316588 MIRROR_EXTEND to 9 mirrors 000674 1734112479.422843 MIRROR_EXTEND to 10 mirrors 000675 1734112479.518751 mapread 0x005167 thru 0x00746f (0x2309 bytes) 000676 1734112479.523310 read 0x014e71 thru 0x018214 (0x33a4 bytes) 000677 1734112479.525124 read 0x02cc68 thru 0x02dca4 (0x103d bytes) 000678 1734112479.527448 mapread 0x032d77 thru 0x034aaa (0x1d34 bytes) 000679 1734112479.527791 read 0x028f3d thru 0x03038a (0x744e bytes) 000680 1734112479.527984 punch from 0x02590a to 0x0319b1 (0xc0a8 bytes) 000681 1734112479.537675 fallocate 0x01b6fb thru 0x025942 (0xa248 bytes) 000682 1734112479.539305 read 0x0281e6 thru 0x028e44 (0xc5f bytes) 000683 1734112479.542435 mapread 0x02197e thru 0x026624 (0x4ca7 bytes) 000684 1734112479.542917 punch from 0x006068 to 0x01532a (0xf2c3 bytes) 000685 1734112479.545627 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000686 1734112479.588807 fallocate 0x0006c6 thru 0x008434 (0x7d6f bytes) 000688 1734112479.590577 mapwrite 0x013a67 thru 0x0195d4 (0x5b6e bytes) 000689 1734112479.600174 fallocate 0x02b55c thru 0x03383a (0x82df bytes) 000690 1734112479.601622 mapwrite 0x009ed2 thru 0x00c1bc (0x22eb bytes) 000691 1734112479.606993 punch from 0x015766 to 0x01afb6 (0x5851 bytes) 000695 1734112479.609269 read 0x0070a6 thru 0x008402 (0x135d bytes) 000696 1734112479.609784 read 0x008f53 thru 0x017318 (0xe3c6 bytes) 000697 1734112479.612347 write 0x0372ab thru 0x0397c2 (0x2518 bytes) 000698 1734112479.615126 MIRROR_RESYNC 9 mirrors 000700 1734112480.002428 fallocate 0x002a88 thru 0x00ce93 (0xa40c bytes) 000701 1734112480.010337 mapwrite 0x03a32e thru 0x03ffff (0x5cd2 bytes) 000704 1734112480.027466 fallocate 0x000131 thru 0x00b272 (0xb142 bytes) 000705 1734112480.028845 mapread 0x00314d thru 0x00ec38 (0xbaec bytes) 000706 1734112480.032941 fallocate 0x0198ed thru 0x0267f5 (0xcf09 bytes) 000707 1734112480.034463 read 0x028b0d thru 0x0320dd (0x95d1 bytes) 000709 1734112480.038055 MIRROR_RESYNC 9 mirrors 000711 1734112480.469050 mapread 0x022950 thru 0x03220c (0xf8bd bytes) 000712 1734112480.476880 write 0x007032 thru 0x016d04 (0xfcd3 bytes) 000713 1734112480.493845 punch from 0x00e773 to 0x0181c5 (0x9a53 bytes) 000714 1734112480.498923 mapread 0x0255da thru 0x02c7a2 (0x71c9 bytes) 000715 1734112480.502628 punch from 0x02699c to 0x031b40 (0xb1a5 bytes) 000716 1734112480.504814 mapwrite 0x031942 thru 0x0340b9 (0x2778 bytes) 000717 1734112480.515854 write 0x009fe0 thru 0x01611c (0xc13d bytes) 000718 1734112480.519260 mapwrite 0x00eba0 thru 0x01763e (0x8a9f bytes) 000719 1734112480.530844 mapread 0x034202 thru 0x03b573 (0x7372 bytes) 000720 1734112480.533923 mapwrite 0x02019f thru 0x02cad2 (0xc934 bytes) 000721 1734112480.546653 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000722 1734112480.595032 mapread 0x02de28 thru 0x02e2f9 (0x4d2 bytes) 000723 1734112480.597504 read 0x0048df thru 0x007832 (0x2f54 bytes) 000724 1734112480.598968 punch from 0x032fe1 to 0x0350ed (0x210d bytes) 000725 1734112480.600397 fallocate 0x029560 thru 0x03313f (0x9be0 bytes) 000726 1734112480.601709 read 0x001212 thru 0x0032c8 (0x20b7 bytes) 000727 1734112480.605712 MIRROR_EXTEND to 9 mirrors 000728 1734112480.731899 read 0x016521 thru 0x0169a1 (0x481 bytes) 000730 1734112480.736296 trunc from 0x040000 to 0x005c6c (0x3a393 bytes) 000731 1734112480.752350 MIRROR_RESYNC 9 mirrors 000733 1734112480.898676 mapread 0x001cc4 thru 0x005c6c (0x3fa9 bytes) 000734 1734112480.903908 write 0x02634d thru 0x02b5af (0x5263 bytes) 000735 1734112480.913369 mapwrite 0x0278da thru 0x031189 (0x98b0 bytes) 000736 1734112480.936457 fallocate 0x019c43 thru 0x0228d4 (0x8c92 bytes) 000737 1734112480.938462 fallocate 0x0321e2 thru 0x034571 (0x2390 bytes) 000738 1734112480.940199 write 0x03290d thru 0x032edf (0x5d3 bytes) 000739 1734112480.942438 read 0x02dbfb thru 0x030dfd (0x3203 bytes) 000740 1734112480.942896 trunc from 0x034572 to 0x0155d6 (0x1ef9b bytes) 000741 1734112480.948046 fallocate 0x02aa5f thru 0x0355ef (0xab91 bytes) 000742 1734112480.949878 fallocate 0x01a17b thru 0x021c8e (0x7b14 bytes) 000743 1734112480.951545 punch from 0x02f049 to 0x033469 (0x4421 bytes) 000744 1734112480.952916 read 0x0272ca thru 0x03531e (0xe055 bytes) 000745 1734112480.956739 mapread 0x0239d9 thru 0x02bd4e (0x8376 bytes) 000746 1734112480.957344 read 0x0341ee thru 0x0355ef (0x1402 bytes) 000747 1734112480.959400 MIRROR_RESYNC 9 mirrors 000749 1734112481.293652 fallocate 0x039424 thru 0x039628 (0x205 bytes) 000750 1734112481.302049 mapwrite 0x0128b7 thru 0x01df6a (0xb6b4 bytes) 000751 1734112481.314810 punch from 0x02c5c4 to 0x02e8dc (0x2319 bytes) 000752 1734112481.316219 punch from 0x0073a8 to 0x011878 (0xa4d1 bytes) 000753 1734112481.317589 write 0x018bc8 thru 0x020b86 (0x7fbf bytes) 000754 1734112481.319805 MIRROR_RESYNC 9 mirrors 000755 1734112481.851611 mapwrite 0x00d7bd thru 0x019aee (0xc332 bytes) 000756 1734112481.887608 MIRROR_EXTEND to 10 mirrors 000757 1734112482.016651 mapread 0x03851d thru 0x039628 (0x110c bytes) 000758 1734112482.021632 punch from 0x011b01 to 0x021721 (0xfc21 bytes) 000759 1734112482.030870 read 0x002a18 thru 0x00caf2 (0xa0db bytes) 000760 1734112482.034624 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000762 1734112482.084544 MIRROR_EXTEND to 10 mirrors 000763 1734112482.219513 mapwrite 0x03f789 thru 0x03ffff (0x877 bytes) 000764 1734112482.235202 trunc from 0x040000 to 0x01ba91 (0x2456e bytes) 000765 1734112482.240351 punch from 0x009c3d to 0x00d496 (0x385a bytes) 000766 1734112482.241894 write 0x03edef thru 0x03ffff (0x1211 bytes) 000767 1734112482.242744 fallocate 0x02307d thru 0x029537 (0x64bb bytes) 000768 1734112482.244392 fallocate 0x01dcae thru 0x0210d5 (0x3428 bytes) 000769 1734112482.246005 fallocate 0x019dea thru 0x0268b0 (0xcac7 bytes) 000771 1734112482.248024 mapread 0x039acc thru 0x03dbb6 (0x40eb bytes) 000774 1734112482.251624 mapwrite 0x024888 thru 0x02bd4f (0x74c8 bytes) 000775 1734112482.259871 punch from 0x01fe35 to 0x023982 (0x3b4e bytes) 000778 1734112482.261833 mapread 0x00caa2 thru 0x0186c8 (0xbc27 bytes) 000779 1734112482.265651 mapread 0x01299d thru 0x016287 (0x38eb bytes) 000780 1734112482.266054 read 0x00e4f5 thru 0x01b070 (0xcb7c bytes) 000781 1734112482.266344 trunc from 0x040000 to 0x0298c5 (0x1673a bytes) 000783 1734112482.270800 trunc from 0x0298c6 to 0x01b2a0 (0xe625 bytes) 000784 1734112482.274273 fallocate 0x035a01 thru 0x03d312 (0x7912 bytes) 000786 1734112482.276188 punch from 0x01abf2 to 0x01f2a7 (0x46b6 bytes) 000787 1734112482.277473 mapwrite 0x03dea5 thru 0x03ffff (0x215b bytes) 000788 1734112482.287310 trunc from 0x040000 to 0x020c25 (0x1f3da bytes) 000789 1734112482.291924 MIRROR_RESYNC 10 mirrors 000790 1734112482.675816 trunc from 0x020c26 to 0x02c529 (0xb904 bytes) 000791 1734112482.689482 write 0x0302f5 thru 0x035ecc (0x5bd8 bytes) 000793 1734112482.691602 MIRROR_EXTEND to 11 mirrors 000794 1734112482.805639 MIRROR_EXTEND to 12 mirrors 000795 1734112482.918678 mapread 0x031833 thru 0x032f33 (0x1701 bytes) 000796 1734112482.921178 punch from 0x02a7fc to 0x02b0bf (0x8c4 bytes) 000797 1734112482.930931 MIRROR_EXTEND to 13 mirrors 000798 1734112483.060650 write 0x003bba thru 0x007e6a (0x42b1 bytes) 000799 1734112483.072664 punch from 0x029776 to 0x035695 (0xbf20 bytes) 000800 1734112483.074449 punch from 0x008a8e to 0x00d368 (0x48db bytes) 000801 1734112483.077213 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000802 1734112483.127218 punch from 0x034c3b to 0x035850 (0xc16 bytes) 000803 1734112483.128955 write 0x03ef5b thru 0x03ffff (0x10a5 bytes) 000804 1734112483.130741 mapread 0x02420f thru 0x02944e (0x5240 bytes) 000805 1734112483.135933 mapread 0x030845 thru 0x03b6b8 (0xae74 bytes) 000806 1734112483.140018 fallocate 0x01eb50 thru 0x02b05d (0xc50e bytes) 000807 1734112483.141745 punch from 0x00aaad to 0x00f02f (0x4583 bytes) 000808 1734112483.143479 mapread 0x01bed0 thru 0x01c844 (0x975 bytes) 000809 1734112483.146264 fallocate 0x033730 thru 0x03c9b8 (0x9289 bytes) 000810 1734112483.147833 mapread 0x006a9f thru 0x0140cf (0xd631 bytes) 000811 1734112483.152255 write 0x02b985 thru 0x035711 (0x9d8d bytes) 000812 1734112483.152999 mapread 0x019a8e thru 0x019c99 (0x20c bytes) 000813 1734112483.153425 mapread 0x00bac7 thru 0x01a170 (0xe6aa bytes) 000814 1734112483.153938 mapread 0x0177fa thru 0x024681 (0xce88 bytes) 000815 1734112483.154476 mapwrite 0x005b83 thru 0x014974 (0xedf2 bytes) 000816 1734112483.161976 write 0x03e8cf thru 0x03ffff (0x1731 bytes) 000818 1734112483.162739 mapread 0x01fd54 thru 0x023f60 (0x420d bytes) 000819 1734112483.163188 trunc from 0x040000 to 0x00d648 (0x329b7 bytes) 000820 1734112483.168681 mapwrite 0x028d05 thru 0x02a31b (0x1617 bytes) 000821 1734112483.180202 mapread 0x001682 thru 0x003dcf (0x274e bytes) 000822 1734112483.182904 mapwrite 0x012c7e thru 0x01847c (0x57ff bytes) 000823 1734112483.189717 mapread 0x014706 thru 0x015cda (0x15d5 bytes) 000825 1734112483.190393 mapwrite 0x0289bc thru 0x02a23d (0x1882 bytes) 000826 1734112483.193614 punch from 0x026294 to 0x02a31b (0x4088 bytes) 000830 1734112483.195645 trunc from 0x02a31c to 0x0018a4 (0x28a77 bytes) 000831 1734112483.200605 write 0x01acac thru 0x01be7c (0x11d1 bytes) 000832 1734112483.202605 MIRROR_EXTEND to 13 mirrors 000833 1734112483.315886 mapwrite 0x03f8c4 thru 0x03ffff (0x73c bytes) 000834 1734112483.333929 trunc from 0x040000 to 0x01b050 (0x24faf bytes) 000835 1734112483.339090 punch from 0x018713 to 0x01b050 (0x293e bytes) 000836 1734112483.340439 mapread 0x0030d5 thru 0x00c823 (0x974f bytes) 000837 1734112483.344785 trunc from 0x01b051 to 0x02aee9 (0xfe99 bytes) 000838 1734112483.348354 fallocate 0x00e6c1 thru 0x013a1c (0x535c bytes) 000839 1734112483.349792 fallocate 0x027d78 thru 0x0351b0 (0xd439 bytes) 000840 1734112483.351586 punch from 0x022216 to 0x032196 (0xff81 bytes) 000841 1734112483.353169 mapread 0x01dd8b thru 0x0239d6 (0x5c4c bytes) 000842 1734112483.356322 punch from 0x00529a to 0x00f219 (0x9f80 bytes) 000843 1734112483.357852 trunc from 0x0351b1 to 0x039dea (0x4c3a bytes) 000844 1734112483.362679 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000845 1734112483.412092 trunc from 0x039deb to 0x03c905 (0x2b1b bytes) 000846 1734112483.416851 write 0x03b13d thru 0x03ffff (0x4ec3 bytes) 000847 1734112483.418878 fallocate 0x004f59 thru 0x007be5 (0x2c8d bytes) 000848 1734112483.421161 mapwrite 0x03686a thru 0x0388cd (0x2064 bytes) 000849 1734112483.430854 write 0x0374be thru 0x03cf52 (0x5a95 bytes) 000851 1734112483.432207 read 0x03eaaf thru 0x03ffff (0x1551 bytes) 000852 1734112483.434586 MIRROR_RESYNC 12 mirrors 000854 1734112483.888544 write 0x034183 thru 0x03e92d (0xa7ab bytes) 000855 1734112483.899344 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000856 1734112483.946391 fallocate 0x0000a6 thru 0x00e4af (0xe40a bytes) 000857 1734112483.948210 fallocate 0x02a1b5 thru 0x02caf5 (0x2941 bytes) 000858 1734112483.949926 mapread 0x027638 thru 0x02ac0d (0x35d6 bytes) 000859 1734112483.954764 write 0x00e5b8 thru 0x016df2 (0x883b bytes) 000860 1734112483.959003 mapread 0x034267 thru 0x039583 (0x531d bytes) 000861 1734112483.963458 MIRROR_RESYNC 11 mirrors 000862 1734112484.403527 mapread 0x03a35f thru 0x03d53f (0x31e1 bytes) 000863 1734112484.409073 MIRROR_EXTEND to 12 mirrors 000864 1734112484.575196 mapwrite 0x01a48d thru 0x029e12 (0xf986 bytes) 000865 1734112484.612384 write 0x012700 thru 0x0177f7 (0x50f8 bytes) 000866 1734112484.615651 write 0x00bab1 thru 0x017ea3 (0xc3f3 bytes) 000868 1734112484.616538 trunc from 0x040000 to 0x038076 (0x7f89 bytes) 000869 1734112484.621350 trunc from 0x038077 to 0x00112e (0x36f48 bytes) 000870 1734112484.625264 trunc from 0x00112f to 0x038b6b (0x37a3d bytes) 000871 1734112484.628850 write 0x02aa05 thru 0x038241 (0xd83d bytes) 000872 1734112484.632826 write 0x02eaa4 thru 0x039a94 (0xaff1 bytes) 000873 1734112484.633444 punch from 0x01261c to 0x012cd4 (0x6b9 bytes) 000875 1734112484.634894 write 0x009fc3 thru 0x0124e9 (0x8527 bytes) 000876 1734112484.638512 mapwrite 0x03f26e thru 0x03ffff (0xd92 bytes) 000878 1734112484.650221 mapwrite 0x0172b4 thru 0x027252 (0xff9f bytes) 000879 1734112484.663736 mapwrite 0x0134b3 thru 0x017fd5 (0x4b23 bytes) 000880 1734112484.670168 write 0x00873a thru 0x014d12 (0xc5d9 bytes) 000881 1734112484.673598 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000882 1734112484.749943 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000883 1734112484.817559 fallocate 0x0023bc thru 0x004c3c (0x2881 bytes) 000884 1734112484.819769 punch from 0x020892 to 0x0267ea (0x5f59 bytes) 000886 1734112484.822132 write 0x0250bb thru 0x0255db (0x521 bytes) 000888 1734112484.826171 write 0x0001bb thru 0x00f5d9 (0xf41f bytes) 000889 1734112484.829828 read 0x0007af thru 0x00ee54 (0xe6a6 bytes) 000890 1734112484.830223 trunc from 0x040000 to 0x01450a (0x2baf5 bytes) 000891 1734112484.834687 read 0x01133d thru 0x01450a (0x31ce bytes) 000893 1734112484.837491 write 0x0074ec thru 0x00fb17 (0x862c bytes) 000894 1734112484.838357 trunc from 0x01450b to 0x0119ad (0x2b5d bytes) 000895 1734112484.842723 fallocate 0x01390d thru 0x014af2 (0x11e6 bytes) 000896 1734112484.844289 punch from 0x006aeb to 0x010a02 (0x9f18 bytes) 000897 1734112484.849672 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000898 1734112484.898255 trunc from 0x0119ae to 0x007965 (0xa048 bytes) 000899 1734112484.904932 fallocate 0x03804e thru 0x03ffff (0x7fb2 bytes) 000900 1734112484.906820 fallocate 0x013d1a thru 0x015d5b (0x2042 bytes) 000901 1734112484.908666 mapwrite 0x01babd thru 0x01bba8 (0xec bytes) 000902 1734112484.917216 mapwrite 0x0075ec thru 0x009737 (0x214c bytes) 000904 1734112484.929159 trunc from 0x040000 to 0x021fb1 (0x1e04e bytes) 000905 1734112484.936261 fallocate 0x01cb4e thru 0x01f4fc (0x29af bytes) 000906 1734112484.938946 trunc from 0x021fb2 to 0x015d8d (0xc224 bytes) 000907 1734112484.944832 write 0x032063 thru 0x03b09e (0x903c bytes) 000908 1734112484.945997 trunc from 0x03b09f to 0x014200 (0x26e9e bytes) 000909 1734112484.951942 write 0x010dfb thru 0x014a48 (0x3c4e bytes) 000911 1734112484.954794 trunc from 0x014a49 to 0x012337 (0x2711 bytes) 000912 1734112484.960266 mapread 0x00c0ab thru 0x012337 (0x628d bytes) 000913 1734112484.962324 MIRROR_EXTEND to 10 mirrors 000914 1734112485.133172 fallocate 0x02ad08 thru 0x0354cc (0xa7c5 bytes) 000915 1734112485.151187 read 0x0071ee thru 0x00be5c (0x4c6f bytes) 000917 1734112485.156199 punch from 0x00d205 to 0x012337 (0x5133 bytes) 000919 1734112485.159305 punch from 0x005360 to 0x00aa03 (0x56a4 bytes) 000920 1734112485.161959 write 0x00951c thru 0x009ba6 (0x68b bytes) 000922 1734112485.167013 mapwrite 0x02b122 thru 0x03827e (0xd15d bytes) 000923 1734112485.199282 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000924 1734112485.248199 mapread 0x016d8c thru 0x02501c (0xe291 bytes) 000925 1734112485.254208 mapread 0x030efd thru 0x032288 (0x138c bytes) 000926 1734112485.256375 write 0x01d070 thru 0x025e7a (0x8e0b bytes) 000927 1734112485.258166 read 0x00d83c thru 0x017e49 (0xa60e bytes) 000928 1734112485.260130 fallocate 0x03f21a thru 0x03ffff (0xde6 bytes) 000929 1734112485.263070 MIRROR_EXTEND to 10 mirrors 000930 1734112485.392310 punch from 0x0303c5 to 0x03827e (0x7eba bytes) 000931 1734112485.401421 read 0x02cb9f thru 0x03082c (0x3c8e bytes) 000932 1734112485.403640 mapwrite 0x005562 thru 0x009f0c (0x49ab bytes) 000933 1734112485.415212 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000934 1734112485.464696 write 0x027861 thru 0x03561e (0xddbe bytes) 000935 1734112485.469300 read 0x0332c7 thru 0x033f15 (0xc4f bytes) 000937 1734112485.470017 write 0x010543 thru 0x01d27d (0xcd3b bytes) 000938 1734112485.474028 write 0x017681 thru 0x0268ec (0xf26c bytes) 000939 1734112485.476151 mapwrite 0x01610f thru 0x01b90f (0x5801 bytes) 000942 1734112485.482708 trunc from 0x03827f to 0x00ebf5 (0x29689 bytes) 000943 1734112485.490596 mapread 0x004636 thru 0x0092cf (0x4c9a bytes) 000944 1734112485.493566 fallocate 0x02d4fe thru 0x03226b (0x4d6e bytes) 000945 1734112485.495551 punch from 0x009b25 to 0x00ebf5 (0x50d1 bytes) 000946 1734112485.497193 mapwrite 0x024049 thru 0x02cf11 (0x8ec9 bytes) 000947 1734112485.512977 read 0x02c0dc thru 0x02cf11 (0xe36 bytes) 000948 1734112485.514817 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000949 1734112485.568643 write 0x021122 thru 0x02839f (0x727e bytes) 000950 1734112485.572539 trunc from 0x02cf12 to 0x0397cc (0xc8bb bytes) 000952 1734112485.576643 fallocate 0x0124da thru 0x016d35 (0x485c bytes) 000953 1734112485.578323 punch from 0x009996 to 0x0145bc (0xac27 bytes) 000954 1734112485.579762 trunc from 0x0397cd to 0x00e0e7 (0x2b6e5 bytes) 000955 1734112485.583458 trunc from 0x00e0e8 to 0x00bf49 (0x219e bytes) 000956 1734112485.587043 mapwrite 0x02d254 thru 0x03c3c9 (0xf176 bytes) 000957 1734112485.601577 fallocate 0x0158a4 thru 0x01af74 (0x56d1 bytes) 000958 1734112485.603194 mapwrite 0x03618f thru 0x03ffff (0x9e71 bytes) 000959 1734112485.614265 trunc from 0x040000 to 0x00c6e9 (0x33916 bytes) 000960 1734112485.619186 MIRROR_EXTEND to 9 mirrors 000961 1734112485.732930 mapwrite 0x03f90b thru 0x03ff27 (0x61d bytes) 000962 1734112485.755193 mapread 0x024b64 thru 0x02d207 (0x86a4 bytes) 000963 1734112485.761413 mapread 0x00a7d1 thru 0x011254 (0x6a84 bytes) 000965 1734112485.764508 write 0x00dd71 thru 0x014d3d (0x6fcd bytes) 000968 1734112485.767195 MIRROR_EXTEND to 10 mirrors 000969 1734112485.892479 read 0x0256cf thru 0x02fb90 (0xa4c2 bytes) 000970 1734112485.897149 punch from 0x01004e to 0x01f5da (0xf58d bytes) 000972 1734112485.906788 punch from 0x0035b7 to 0x00a410 (0x6e5a bytes) 000974 1734112485.908729 mapread 0x006943 thru 0x00aef8 (0x45b6 bytes) 000975 1734112485.914340 read 0x00d47a thru 0x00dcf0 (0x877 bytes) 000976 1734112485.916127 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000977 1734112485.967120 read 0x016084 thru 0x0234c7 (0xd444 bytes) 000978 1734112485.969987 trunc from 0x03ff28 to 0x0235c2 (0x1c965 bytes) 000979 1734112485.978346 fallocate 0x03f3c8 thru 0x03ffff (0xc38 bytes) 000981 1734112485.979902 write 0x032e30 thru 0x03c94b (0x9b1c bytes) 000982 1734112485.980857 mapwrite 0x00f6f3 thru 0x017dff (0x870d bytes) 000985 1734112485.995141 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000986 1734112486.047647 trunc from 0x03c94c to 0x00eb95 (0x2ddb6 bytes) 000987 1734112486.054215 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xacf:0x0]: Invalid argument (22) 000988 1734112486.099768 mapwrite 0x037910 thru 0x03e544 (0x6c35 bytes) 000989 1734112486.113491 mapwrite 0x0212a0 thru 0x02c892 (0xb5f3 bytes) 000990 1734112486.122037 mapwrite 0x0333c3 thru 0x038e0c (0x5a4a bytes) 000991 1734112486.127055 punch from 0x015f00 to 0x018d0e (0x2e0f bytes) 000992 1734112486.128588 fallocate 0x00c7f6 thru 0x00ca78 (0x283 bytes) 000995 1734112486.130586 trunc from 0x03e545 to 0x01b669 (0x22edb bytes) 000996 1734112486.134646 mapread 0x01b154 thru 0x01b669 (0x516 bytes) 000997 1734112486.135423 read 0x00c65d thru 0x01b669 (0xf00d bytes) 000998 1734112486.138278 fallocate 0x017ca6 thru 0x01d9c5 (0x5d20 bytes) 001000 1734112486.140116 fallocate 0x022e38 thru 0x024ddc (0x1fa5 bytes) All operations completed A-OK! PASS 70a (20s) == sanity-flr test 200a: stress test ===================== 12:54:47 (1734112487) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre2 Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 603712 .. Extending file size to 1228640 .. done Extending file size to 5377760 .. Extending file size to 9011808 .. Extending file size to 7356512 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9169920 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9071744 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Waiting 7931 7932 7933 7935 7938 192.168.204.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg434-client.virtnet /mnt/lustre2 (opts:) 192.168.204.134@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg434-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:55:52 (1734112552) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre2 Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 943584 .. Extending file size to 9147584 .. failed Extending file size to 8579648 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8774944 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9192416 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9194560 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9275040 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 11116 11117 11119 11120 11123 done 192.168.204.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg434-client.virtnet /mnt/lustre2 (opts:) 192.168.204.134@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg434-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:56:57 (1734112617) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3428 1284260 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21888 3585132 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15844 3591176 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37732 7176308 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2520 1285168 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3428 1284260 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 21888 3585132 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 15844 3591176 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37732 7176308 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0610478 s, 34.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.079971 s, 26.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg434-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:57:04 (1734112624) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0279938 s, 74.9 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 12:57:06 (1734112626) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x89:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x89:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8a:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa7:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:57:08 (1734112628) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0354927 s, 59.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xf97:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xf98:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xf99:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:57:11 (1734112631) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.037746 s, 55.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:57:14 (1734112634) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0364469 s, 57.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:57:38 (1734112658) Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg434-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178451 s, 58.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xf98:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xf9b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xf9a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xf98:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xf9b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xf9a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xf99:0x0] } PASS 204d (34s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:58:12 (1734112692) Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg434-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg434-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0347646 s, 60.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:58:47 (1734112727) Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg434-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg434-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg434-server: Pool lustre.test_204f created Adding targets to pool oleg434-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg434-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.020939 s, 100 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg434-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg434-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg434-server: Pool lustre.test_204f destroyed PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:59:36 (1734112776) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:59:38 (1734112778) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:59:40 (1734112780) oleg434-server: Pool lustre.test_206 created Adding targets to pool oleg434-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg434-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg434-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg434-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg434-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:59:54 (1734112794) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178826 s, 58.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00961565 s, 109 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x90b719f2 Mirror 3: 0x90b719f2 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:59:56 (1734112796) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:59:57 (1734112797) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:59:59 (1734112799) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0116403 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1430000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x23c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x23c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x23c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xd80000 Read this (4 bytes): BBBB 192.168.204.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg434-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:00:14 (1734112814) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00285843 s, 2.9 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:00:16 (1734112816) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113044 s, 92.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7e lmm_fid: [0x200000405:0x7e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7e lmm_fid: [0x200000405:0x7e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9a:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611262 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-13 13:00:16.000000000 -0500 Modify: 2024-12-13 13:00:16.000000000 -0500 Change: 2024-12-13 13:00:17.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:00:18 (1734112818) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00937132 s, 112 MB/s oleg434-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg434-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg434-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 oleg434-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg434-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg434-server: O/280000401/d24/b8: File not found by ext2_lookup PASS 210b (11s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:00:29 (1734112829) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0275743 s, 1.5 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00472911 s, 866 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 13 13:00 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x83:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 13 13:00 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212: I/O should skip the foreign FLR layout component ========================================================== 13:00:31 (1734112831) lfs mirror create: cannot create composite file '/mnt/lustre/f212.sanity-flr': Invalid argument sanity-flr test_212: @@@@@@ FAIL: /mnt/lustre/f212.sanity-flr: create failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7229:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:4568:test_212() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7602:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7665:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7483:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:4587:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_212.*.1734112832.log rsync: chown "/tmp/testlogs/.sanity-flr.test_212.debug_log.oleg434-server.1734112832.log.K1P6Lv" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_212.dmesg.oleg434-server.1734112832.log.9ZSN3D" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 212 (2s) == sanity-flr test complete, duration 710 sec ============ 13:00:33 (1734112833) sanity-flr: FAIL: test_212 /mnt/lustre/f212.sanity-flr: create failed === sanity-flr: start cleanup 13:00:34 (1734112834) === === sanity-flr: finish cleanup 13:00:36 (1734112836) ===