-----============= acceptance-small: sanity-flr ============----- Sun Nov 24 01:11:25 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c === sanity-flr: start setup 01:11:29 (1732428689) === oleg325-client.virtnet: executing check_config_client /mnt/lustre oleg325-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg325-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88013767f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88013767f800.idle_timeout=debug disable quota as required oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:11:37 (1732428697) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7580] == sanity-flr test 0a: lfs mirror create with -N option == 01:11:38 (1732428698) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:11:41 (1732428701) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:11:42 (1732428702) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (0s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:11:42 (1732428702) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:11:47 (1732428707) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:11:47 (1732428707) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 01:11:48 (1732428708) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88013767f800.stats=clear osc.lustre-OST0001-osc-ffff88013767f800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:11:50 (1732428710) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:11:52 (1732428712) PASS 0j (1s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:11:54 (1732428714) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (0s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:11:54 (1732428714) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 01:11:56 (1732428716) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:11:58 (1732428718) PASS 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:11:59 (1732428719) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:12:01 (1732428721) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:12:03 (1732428723) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:12:05 (1732428725) 13+0 records in 13+0 records out 13631488 bytes (14 MB) copied, 0.317967 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0271845 s, 38.6 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:12:07 (1732428727) 19+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.105816 s, 188 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:12:09 (1732428729) Stopping /mnt/lustre-ost1 (opts:) on oleg325-server oleg325-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88013767f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88013767f800.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg325-server: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg325-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg325-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg325-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg325-server oleg325-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88013767f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88013767f800.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg325-server: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg325-server: *.lustre-OST0001.recovery_status status: RECOVERING oleg325-server: Waiting 1470 secs for *.lustre-OST0001.recovery_status recovery done. status: RECOVERING oleg325-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (33s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:12:42 (1732428762) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0768978 s, 68.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg325-server oleg325-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88013767f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88013767f800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg325-server: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg325-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg325-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg325-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:12:57 (1732428777) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg325-server oleg325-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88013767f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88013767f800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg325-server: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg325-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg325-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg325-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg325-server oleg325-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88013767f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88013767f800.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg325-server: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg325-server: *.lustre-OST0001.recovery_status status: RECOVERING oleg325-server: Waiting 1470 secs for *.lustre-OST0001.recovery_status recovery done. status: RECOVERING oleg325-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (28s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:13:25 (1732428805) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.693058 s, 75.6 MB/s -rw-r--r-- 1 root root 52428800 Nov 24 01:13 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.174s user 0m0.000s sys 0m0.092s Stopping /mnt/lustre-ost1 (opts:) on oleg325-server oleg325-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88013767f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88013767f800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.202s user 0m0.000s sys 0m0.102s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg325-server: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg325-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg325-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg325-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg325-server oleg325-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88013767f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88013767f800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.200s user 0m0.001s sys 0m0.103s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg325-server: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg325-server: *.lustre-OST0001.recovery_status status: RECOVERING oleg325-server: Waiting 1470 secs for *.lustre-OST0001.recovery_status recovery done. status: RECOVERING oleg325-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg325-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88013767f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88013767f800.ost_server_uuid in FULL state after 0 sec PASS 33b (32s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:13:57 (1732428837) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:13:57 (1732428837) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:13:58 (1732428838) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 01:13:59 (1732428839) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 01:14:00 (1732428840) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.114837 s, 110 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:14:03 (1732428843) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00950268 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00760084 s, 138 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:14:04 (1732428844) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:14:07 (1732428847) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000755643 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00079067 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000506955 s, 2.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00071308 s, 1.4 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 01:14:09 (1732428849) create test files with size 15752749 .. 5f49c96101bbf5dad362a21d9149cbd7 - 1351b750b785de090f63c3f3a8b1250b - 5288e6dafebcfcc4a0a163c9bce2ccf4 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg325-client.virtnet /mnt/lustre (opts:) Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 01:14:13 (1732428853) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:14:16 (1732428856) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:14:18 (1732428858) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0212667 s, 49.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00859762 s, 122 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x42:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 01:14:20 (1732428860) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.021132 s, 99.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.082411 s, 50.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0318288 s, 65.9 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 01:14:22 (1732428862) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 01:14:23 (1732428863) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:14:23 (1732428863) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00948699 s, 111 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 01:14:25 (1732428865) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0708646 s, 44.4 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x10:0x0]: Invalid argument (22) PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:14:27 (1732428867) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:14:29 (1732428869) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0690578 s, 152 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg325-client.virtnet /mnt/lustre (opts:) Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:14:31 (1732428871) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a2718800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0711171 s, 147 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg325-client.virtnet /mnt/lustre (opts:) Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff880000048800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:14:34 (1732428874) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:14:41 (1732428881) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (7s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:14:48 (1732428888) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) == sanity-flr test 48: Verify snapshot mirror ============ 01:14:48 (1732428888) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0659866 s, 47.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0472108 s, 66.6 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 47d603c05c1d03d8273807d4b43c65fc - mirror 1 checksum: e3505c98c44deea5d3cf19c59c10b8a9 - mirror 2 checksum: 47d603c05c1d03d8273807d4b43c65fc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 24 01:14:49 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 24 01:14:49 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 47d603c05c1d03d8273807d4b43c65fc - mirror 1 checksum: e3505c98c44deea5d3cf19c59c10b8a9 - mirror 2 checksum: e3505c98c44deea5d3cf19c59c10b8a9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 24 01:14:49 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 24 01:14:49 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:14:50 (1732428890) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00761388 s, 138 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 529408.. 530431: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:14:52 (1732428892) Failing mds1 on oleg325-server Stopping /mnt/lustre-mds1 (opts:) on oleg325-server 01:14:59 (1732428899) shut down Failover mds1 to oleg325-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0000 01:15:12 (1732428912) targets are mounted 01:15:12 (1732428912) facet_failover done oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012228 s, 85.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:15:19 (1732428919) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00846196 s, 2.4 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00445932 s, 4.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:15:21 (1732428921) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00920876 s, 2.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00236588 s, 8.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:15:23 (1732428923) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00213121 s, 7.7 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:15:25 (1732428925) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00906782 s, 2.3 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:15:27 (1732428927) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00189897 s, 2.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00668063 s, 613 kB/s PASS 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:15:28 (1732428928) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:15:32 (1732428932) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:15:50 (1732428950) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:16:07 (1732428967) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:16:24 (1732428984) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 01:16:26 (1732428986) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 26087 mirror split pid 26088 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4851: 26087 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4851: 26088 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 01:17:27 (1732429047) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 9047 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x2d3ed 000001 1732429047.987994 trunc from 00000000 to 0x02d3ec (0x2d3ed bytes) 000002 1732429047.991641 mapread 0x01ab0e thru 0x020ea2 (0x6395 bytes) 000003 1732429047.995982 read 0x02265c thru 0x028f88 (0x692d bytes) 000004 1732429047.997362 punch from 0x0069a1 to 0x0082df (0x193f bytes) 000005 1732429047.998669 mapwrite 0x02b4e7 thru 0x0367d6 (0xb2f0 bytes) fallocating to largest ever: 0x356db 000006 1732429048.014307 fallocate 0x0311d3 thru 0x0356da (0x4508 bytes) 000007 1732429048.015775 read 0x02aceb thru 0x02b383 (0x699 bytes) 000008 1732429048.016138 mapread 0x03575d thru 0x0367d6 (0x107a bytes) 000011 1732429048.017386 write 0x036db4 thru 0x03ffff (0x924c bytes) 000012 1732429048.018040 write 0x024a42 thru 0x027b3f (0x30fe bytes) 000013 1732429048.018504 fallocate 0x021133 thru 0x0212ae (0x17c bytes) 000015 1732429048.020203 mapread 0x02155e thru 0x0311d3 (0xfc76 bytes) 000017 1732429048.021067 write 0x0351f0 thru 0x03ffff (0xae10 bytes) 000018 1732429048.021901 mapread 0x01419a thru 0x018b4a (0x49b1 bytes) fallocating to largest ever: 0x35e32 000021 1732429048.025135 fallocate 0x03085f thru 0x035e31 (0x55d3 bytes) 000024 1732429048.027123 fallocate 0x019c9e thru 0x01cf95 (0x32f8 bytes) 000025 1732429048.028300 read 0x02a0cc thru 0x032693 (0x85c8 bytes) 000026 1732429048.028555 mapwrite 0x016e76 thru 0x01cdcd (0x5f58 bytes) 000027 1732429048.033947 mapread 0x01933d thru 0x0202eb (0x6faf bytes) 000028 1732429048.034465 trunc from 0x040000 to 0x0349f5 (0xb60a bytes) 000029 1732429048.038274 write 0x020ef7 thru 0x02cb54 (0xbc5e bytes) 000030 1732429048.039039 fallocate 0x025465 thru 0x034e56 (0xf9f2 bytes) 000031 1732429048.040340 fallocate 0x034425 thru 0x035d6b (0x1947 bytes) fallocating to largest ever: 0x3e020 000034 1732429048.042142 fallocate 0x02ff57 thru 0x03e01f (0xe0c9 bytes) 000036 1732429048.044255 mapread 0x001164 thru 0x0034a5 (0x2342 bytes) 000038 1732429048.044857 trunc from 0x03e020 to 0x00ee0e (0x2f211 bytes) 000039 1732429048.048400 mapwrite 0x005f34 thru 0x0135ef (0xd6bc bytes) 000040 1732429048.059778 mapread 0x005033 thru 0x00e1b6 (0x9184 bytes) 000043 1732429048.061471 write 0x01cedf thru 0x02bf37 (0xf059 bytes) 000044 1732429048.062677 mapwrite 0x00a6b5 thru 0x016afd (0xc449 bytes) 000045 1732429048.069647 read 0x0265b9 thru 0x02bf37 (0x597f bytes) 000048 1732429048.070940 write 0x028db6 thru 0x037ccb (0xef16 bytes) 000050 1732429048.072578 trunc from 0x037ccc to 0x0239d8 (0x142f3 bytes) 000051 1732429048.076410 punch from 0x00402e to 0x00cb58 (0x8b2b bytes) 000052 1732429048.078106 write 0x01738e thru 0x01a2ba (0x2f2d bytes) 000053 1732429048.078670 fallocate 0x00780e thru 0x0091c9 (0x19bc bytes) 000054 1732429048.080073 mapwrite 0x018c86 thru 0x018e09 (0x184 bytes) 000055 1732429048.083720 write 0x00b22f thru 0x018b1f (0xd8f1 bytes) 000056 1732429048.086257 read 0x0148d4 thru 0x02043a (0xbb67 bytes) 000057 1732429048.086696 trunc from 0x0239d9 to 0x00beb2 (0x17b26 bytes) 000058 1732429048.090630 punch from 0x00629f to 0x006f8f (0xcf1 bytes) 000059 1732429048.093344 MIRROR_EXTEND to 2 mirrors 000060 1732429048.183753 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000064 1732429048.211789 punch from 0x005166 to 0x00beb2 (0x6d4d bytes) 000065 1732429048.215215 mapwrite 0x010543 thru 0x01d56f (0xd02d bytes) 000066 1732429048.229278 mapread 0x005dc0 thru 0x005e76 (0xb7 bytes) 000067 1732429048.231652 mapread 0x00b168 thru 0x0191fd (0xe096 bytes) 000069 1732429048.232432 trunc from 0x01d570 to 0x02bc58 (0xe6e9 bytes) 000071 1732429048.236184 mapwrite 0x02ded2 thru 0x02e6a1 (0x7d0 bytes) 000072 1732429048.245114 fallocate 0x00eed3 thru 0x00f768 (0x896 bytes) 000073 1732429048.246505 write 0x02220b thru 0x027893 (0x5689 bytes) 000075 1732429048.248785 mapread 0x02dd54 thru 0x02e6a1 (0x94e bytes) 000076 1732429048.250525 MIRROR_EXTEND to 2 mirrors 000077 1732429048.352544 punch from 0x0140b9 to 0x0183ef (0x4337 bytes) 000078 1732429048.360130 punch from 0x00309b to 0x0056ee (0x2654 bytes) 000079 1732429048.361224 mapwrite 0x012e7b thru 0x014cd8 (0x1e5e bytes) 000080 1732429048.369063 fallocate 0x02df36 thru 0x039acd (0xbb98 bytes) 000081 1732429048.370344 read 0x00e81b thru 0x01a10b (0xb8f1 bytes) 000082 1732429048.370908 fallocate 0x00edfe thru 0x013c57 (0x4e5a bytes) 000083 1732429048.373322 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000084 1732429048.397927 write 0x0324ae thru 0x038de3 (0x6936 bytes) 000086 1732429048.399937 mapread 0x038ab5 thru 0x038de3 (0x32f bytes) 000087 1732429048.400400 punch from 0x012cb4 to 0x017a00 (0x4d4d bytes) 000088 1732429048.401802 punch from 0x01c811 to 0x0296b0 (0xcea0 bytes) 000089 1732429048.403045 punch from 0x02962d to 0x02b3be (0x1d92 bytes) 000093 1732429048.404959 write 0x000533 thru 0x006080 (0x5b4e bytes) 000095 1732429048.407863 write 0x008b1c thru 0x009c23 (0x1108 bytes) 000096 1732429048.410714 mapread 0x014bf4 thru 0x01c7ce (0x7bdb bytes) 000097 1732429048.414120 read 0x00e26a thru 0x01b80c (0xd5a3 bytes) 000098 1732429048.414456 write 0x01bdbe thru 0x025561 (0x97a4 bytes) 000099 1732429048.415976 trunc from 0x038de4 to 0x00fafd (0x292e6 bytes) 000100 1732429048.422762 mapread 0x004f96 thru 0x007f19 (0x2f84 bytes) 000102 1732429048.423633 write 0x03107d thru 0x03f513 (0xe497 bytes) 000103 1732429048.424573 mapread 0x005a74 thru 0x014122 (0xe6af bytes) 000104 1732429048.427617 MIRROR_EXTEND to 2 mirrors 000106 1732429048.532746 read 0x004f67 thru 0x00d11a (0x81b4 bytes) 000107 1732429048.535925 write 0x0357c1 thru 0x03ffff (0xa83f bytes) 000108 1732429048.545698 MIRROR_EXTEND to 3 mirrors 000109 1732429048.652611 trunc from 0x040000 to 0x024545 (0x1baba bytes) 000110 1732429048.664955 mapwrite 0x022e3d thru 0x022f12 (0xd6 bytes) 000111 1732429048.669725 mapwrite 0x018f8d thru 0x022b89 (0x9bfd bytes) 000112 1732429048.678694 MIRROR_EXTEND to 4 mirrors 000113 1732429048.785143 fallocate 0x016e5d thru 0x026322 (0xf4c6 bytes) 000114 1732429048.792506 read 0x00f365 thru 0x01ede6 (0xfa82 bytes) 000115 1732429048.794730 mapwrite 0x0340d4 thru 0x03e519 (0xa446 bytes) 000116 1732429048.812297 mapwrite 0x018919 thru 0x020889 (0x7f71 bytes) 000117 1732429048.819307 MIRROR_RESYNC 4 mirrors 000118 1732429049.025722 trunc from 0x03e51a to 0x0159f4 (0x28b25 bytes) 000120 1732429049.037494 fallocate 0x0179ca thru 0x01f6af (0x7ce6 bytes) 000121 1732429049.038981 trunc from 0x0159f5 to 0x015646 (0x3ae bytes) 000122 1732429049.042920 punch from 0x009af1 to 0x00c024 (0x2534 bytes) 000123 1732429049.044290 trunc from 0x015647 to 0x0247a5 (0xf15f bytes) 000124 1732429049.047544 mapwrite 0x0177f0 thru 0x025436 (0xdc47 bytes) truncating to largest ever: 0x3f54e 000125 1732429049.062169 trunc from 0x025437 to 0x03f54d (0x1a117 bytes) 000127 1732429049.065722 mapwrite 0x01f6bd thru 0x0247a0 (0x50e4 bytes) 000128 1732429049.070747 read 0x01ca05 thru 0x01e041 (0x163d bytes) 000131 1732429049.071630 mapwrite 0x017131 thru 0x0254b7 (0xe387 bytes) 000132 1732429049.078168 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000133 1732429049.116876 fallocate 0x017f54 thru 0x01ddc8 (0x5e75 bytes) 000134 1732429049.118509 punch from 0x0313da to 0x031ee1 (0xb08 bytes) 000135 1732429049.119935 read 0x0295f4 thru 0x02f749 (0x6156 bytes) 000136 1732429049.122379 write 0x032b45 thru 0x03e310 (0xb7cc bytes) fallocating to largest ever: 0x40000 000137 1732429049.126136 fallocate 0x03c630 thru 0x03ffff (0x39d0 bytes) 000138 1732429049.127499 fallocate 0x0190b6 thru 0x024104 (0xb04f bytes) 000139 1732429049.128780 read 0x014c56 thru 0x0175fc (0x29a7 bytes) 000141 1732429049.131647 write 0x036da8 thru 0x03ffff (0x9258 bytes) 000142 1732429049.132343 punch from 0x01aba0 to 0x02534f (0xa7b0 bytes) 000144 1732429049.134024 mapread 0x02e756 thru 0x03b3c7 (0xcc72 bytes) 000145 1732429049.134607 fallocate 0x034d33 thru 0x036787 (0x1a55 bytes) 000146 1732429049.135976 mapwrite 0x024bba thru 0x0332dd (0xe724 bytes) 000148 1732429049.144102 read 0x02801f thru 0x028ced (0xccf bytes) 000150 1732429049.146796 fallocate 0x00d488 thru 0x01ceb6 (0xfa2f bytes) 000151 1732429049.148216 fallocate 0x02e3b7 thru 0x03adfa (0xca44 bytes) 000152 1732429049.149506 mapwrite 0x03beca thru 0x03c558 (0x68f bytes) 000153 1732429049.152729 mapread 0x031d0b thru 0x03ce42 (0xb138 bytes) 000154 1732429049.153208 mapread 0x03721e thru 0x03fdec (0x8bcf bytes) 000155 1732429049.153726 read 0x001cff thru 0x0032f0 (0x15f2 bytes) 000156 1732429049.154108 mapwrite 0x009db2 thru 0x00b8a7 (0x1af6 bytes) 000157 1732429049.157804 fallocate 0x017d68 thru 0x024a32 (0xcccb bytes) 000158 1732429049.159332 fallocate 0x02bb9e thru 0x02dc18 (0x207b bytes) 000160 1732429049.161040 read 0x01b326 thru 0x026a8b (0xb766 bytes) 000161 1732429049.163461 mapwrite 0x002b3a thru 0x00cbde (0xa0a5 bytes) 000162 1732429049.169567 write 0x00f4d9 thru 0x011fa9 (0x2ad1 bytes) 000163 1732429049.170226 punch from 0x03d5d6 to 0x03ffff (0x2a2a bytes) 000164 1732429049.171724 fallocate 0x02c061 thru 0x03809b (0xc03b bytes) 000167 1732429049.173545 fallocate 0x00809d thru 0x016a96 (0xe9fa bytes) 000169 1732429049.175462 punch from 0x00c500 to 0x017911 (0xb412 bytes) 000170 1732429049.178382 punch from 0x01f00d to 0x026cf8 (0x7cec bytes) 000171 1732429049.179891 fallocate 0x0112ff thru 0x015bc0 (0x48c2 bytes) 000172 1732429049.181324 mapread 0x019168 thru 0x0237ba (0xa653 bytes) 000173 1732429049.186865 read 0x03ba2c thru 0x03cb2e (0x1103 bytes) 000174 1732429049.187261 fallocate 0x02d469 thru 0x03a322 (0xceba bytes) 000175 1732429049.188684 mapread 0x0108d6 thru 0x016afc (0x6227 bytes) 000177 1732429049.191360 punch from 0x01a152 to 0x01d30d (0x31bc bytes) 000178 1732429049.192883 mapwrite 0x00e007 thru 0x00ef0b (0xf05 bytes) 000179 1732429049.196281 trunc from 0x040000 to 0x014917 (0x2b6e8 bytes) 000181 1732429049.200263 read 0x01437e thru 0x014917 (0x59a bytes) 000182 1732429049.201570 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000184 1732429049.243274 fallocate 0x03fc23 thru 0x03ffff (0x3dd bytes) 000185 1732429049.244790 trunc from 0x014918 to 0x017b1c (0x3205 bytes) 000186 1732429049.249358 write 0x03a0d4 thru 0x03ffff (0x5f2c bytes) 000187 1732429049.251563 MIRROR_RESYNC 2 mirrors 000188 1732429049.345215 MIRROR_RESYNC 2 mirrors 000189 1732429049.377160 mapread 0x015ed3 thru 0x01ab5e (0x4c8c bytes) 000190 1732429049.381106 fallocate 0x0352b9 thru 0x03ffff (0xad47 bytes) 000191 1732429049.388514 trunc from 0x040000 to 0x020939 (0x1f6c6 bytes) 000192 1732429049.396765 fallocate 0x03cf0b thru 0x03ffff (0x30f5 bytes) 000193 1732429049.398094 read 0x011f42 thru 0x011f49 (0x8 bytes) 000194 1732429049.399514 read 0x00977c thru 0x00e58b (0x4e10 bytes) 000196 1732429049.402200 fallocate 0x004726 thru 0x011fdb (0xd8b6 bytes) 000197 1732429049.403532 mapread 0x00acf4 thru 0x019a5a (0xed67 bytes) 000198 1732429049.404073 punch from 0x015bbe to 0x016eea (0x132d bytes) 000199 1732429049.405329 fallocate 0x035692 thru 0x03ffff (0xa96e bytes) 000200 1732429049.406745 mapwrite 0x01e4cc thru 0x02a098 (0xbbcd bytes) 000201 1732429049.417543 write 0x020178 thru 0x03010f (0xff98 bytes) 000202 1732429049.418499 read 0x0092c2 thru 0x00fc83 (0x69c2 bytes) 000203 1732429049.418883 fallocate 0x00ef29 thru 0x014f12 (0x5fea bytes) 000204 1732429049.420341 write 0x02c610 thru 0x02f81e (0x320f bytes) 000207 1732429049.421214 write 0x00fe11 thru 0x012e75 (0x3065 bytes) 000208 1732429049.421688 mapread 0x00f906 thru 0x017094 (0x778f bytes) 000211 1732429049.424062 trunc from 0x030110 to 0x02dc68 (0x24a7 bytes) 000212 1732429049.427573 mapread 0x02a53e thru 0x02dc68 (0x372b bytes) 000213 1732429049.428135 trunc from 0x02dc69 to 0x004b77 (0x290f1 bytes) 000214 1732429049.431715 read 0x0039c7 thru 0x004b77 (0x11b1 bytes) 000216 1732429049.432456 mapwrite 0x01035b thru 0x016488 (0x612e bytes) 000218 1732429049.442614 read 0x007fa4 thru 0x00fe62 (0x7ebf bytes) 000219 1732429049.442987 write 0x02252f thru 0x02c9a9 (0xa47b bytes) 000220 1732429049.443772 trunc from 0x02c9aa to 0x036716 (0x9d6d bytes) 000221 1732429049.446843 mapwrite 0x03b114 thru 0x03ffff (0x4eec bytes) 000222 1732429049.457534 mapread 0x00b51e thru 0x0124a8 (0x6f8b bytes) 000223 1732429049.459249 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000224 1732429049.485406 trunc from 0x040000 to 0x02ca40 (0x135bf bytes) 000225 1732429049.489700 mapwrite 0x009e13 thru 0x015ddc (0xbfca bytes) 000226 1732429049.500813 punch from 0x0015ab to 0x00d4a1 (0xbef7 bytes) 000227 1732429049.502316 mapread 0x01690b thru 0x01f1cb (0x88c1 bytes) 000229 1732429049.504636 fallocate 0x0285a8 thru 0x02ebd3 (0x662c bytes) 000230 1732429049.506057 read 0x025bc4 thru 0x02ebd3 (0x9010 bytes) 000231 1732429049.508198 mapwrite 0x0040d2 thru 0x004438 (0x367 bytes) 000232 1732429049.512902 read 0x01f656 thru 0x0214b8 (0x1e63 bytes) 000233 1732429049.513168 trunc from 0x02ebd4 to 0x00706e (0x27b65 bytes) 000234 1732429049.516706 mapread 0x0034bf thru 0x00706e (0x3bb0 bytes) 000235 1732429049.517134 mapwrite 0x01d390 thru 0x01fb5f (0x27d0 bytes) 000236 1732429049.526040 trunc from 0x01fb60 to 0x0039c7 (0x1c198 bytes) 000237 1732429049.529478 trunc from 0x0039c8 to 0x03f6af (0x3bce8 bytes) 000238 1732429049.533589 MIRROR_EXTEND to 2 mirrors 000242 1732429049.624609 read 0x01a897 thru 0x02692c (0xc096 bytes) 000244 1732429049.629956 trunc from 0x03f6b0 to 0x01243d (0x2d272 bytes) 000245 1732429049.642461 read 0x0058bc thru 0x0067c9 (0xf0e bytes) 000246 1732429049.644888 trunc from 0x01243e to 0x009f14 (0x8529 bytes) 000247 1732429049.649137 write 0x03c6c3 thru 0x03ffff (0x393d bytes) 000248 1732429049.649806 trunc from 0x040000 to 0x011703 (0x2e8fc bytes) 000249 1732429049.653287 trunc from 0x011704 to 0x0368a2 (0x2519f bytes) 000250 1732429049.656653 write 0x0154cf thru 0x023bec (0xe71e bytes) 000251 1732429049.660200 write 0x02d875 thru 0x03b08d (0xd819 bytes) 000252 1732429049.661150 mapwrite 0x02c1ec thru 0x035dac (0x9bc1 bytes) 000254 1732429049.668760 MIRROR_RESYNC 2 mirrors 000255 1732429049.761151 MIRROR_EXTEND to 3 mirrors 000256 1732429049.859287 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000257 1732429049.922872 mapread 0x0037b2 thru 0x009363 (0x5bb2 bytes) 000258 1732429049.931365 mapread 0x02eb3b thru 0x03af86 (0xc44c bytes) 000259 1732429049.939323 trunc from 0x03b08e to 0x00f297 (0x2bdf6 bytes) 000260 1732429049.953604 punch from 0x005f49 to 0x00e985 (0x8a3d bytes) 000261 1732429049.955102 mapwrite 0x0177f7 thru 0x022492 (0xac9c bytes) 000262 1732429049.968948 fallocate 0x03d071 thru 0x03ffff (0x2f8f bytes) 000263 1732429049.970392 write 0x011c69 thru 0x0131e6 (0x157e bytes) 000264 1732429049.971130 read 0x013564 thru 0x01d5ba (0xa057 bytes) 000265 1732429049.971595 trunc from 0x022493 to 0x0059a2 (0x1caf0 bytes) 000266 1732429049.975385 mapread 0x001013 thru 0x0059a2 (0x4990 bytes) 000267 1732429049.977630 punch from 0x001a1f to 0x00334f (0x1931 bytes) 000268 1732429049.979924 MIRROR_EXTEND to 3 mirrors 000269 1732429050.067439 fallocate 0x00ffc0 thru 0x016685 (0x66c6 bytes) 000270 1732429050.074721 write 0x021594 thru 0x0277ca (0x6237 bytes) 000272 1732429050.076805 write 0x024f50 thru 0x02e1cd (0x927e bytes) 000273 1732429050.077575 punch from 0x016287 to 0x0171db (0xf55 bytes) 000274 1732429050.079031 read 0x012e8a thru 0x01fd9a (0xcf11 bytes) 000275 1732429050.081149 mapwrite 0x03b7e0 thru 0x03ffff (0x4820 bytes) 000276 1732429050.095138 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000277 1732429050.133274 trunc from 0x040000 to 0x006962 (0x3969d bytes) 000278 1732429050.137295 read 0x004265 thru 0x006962 (0x26fe bytes) 000279 1732429050.138693 trunc from 0x006963 to 0x02e522 (0x27bc0 bytes) 000280 1732429050.141661 fallocate 0x013468 thru 0x01d734 (0xa2cd bytes) 000282 1732429050.144278 MIRROR_EXTEND to 3 mirrors 000283 1732429050.230386 write 0x000c79 thru 0x005f5b (0x52e3 bytes) 000284 1732429050.239808 fallocate 0x039f0b thru 0x03eeb7 (0x4fad bytes) 000285 1732429050.241372 read 0x005f5b thru 0x00a6a6 (0x474c bytes) 000286 1732429050.243346 mapread 0x02c044 thru 0x02e522 (0x24df bytes) 000287 1732429050.248114 MIRROR_EXTEND to 4 mirrors 000288 1732429050.335158 punch from 0x009705 to 0x00e7bf (0x50bb bytes) 000289 1732429050.342214 punch from 0x0268ca to 0x02e1a3 (0x78da bytes) 000290 1732429050.343298 mapwrite 0x02a6b3 thru 0x036e59 (0xc7a7 bytes) 000291 1732429050.361977 mapread 0x032922 thru 0x036e59 (0x4538 bytes) 000292 1732429050.362609 mapwrite 0x018f2d thru 0x01e160 (0x5234 bytes) 000293 1732429050.368701 write 0x0146de thru 0x01cc94 (0x85b7 bytes) 000294 1732429050.369378 write 0x039021 thru 0x03ffff (0x6fdf bytes) 000295 1732429050.370099 read 0x0180e6 thru 0x0247e7 (0xc702 bytes) 000296 1732429050.371544 mapwrite 0x03156e thru 0x03f0ea (0xdb7d bytes) 000297 1732429050.377975 write 0x034132 thru 0x03f657 (0xb526 bytes) 000298 1732429050.378653 write 0x03dfec thru 0x03ffff (0x2014 bytes) 000299 1732429050.379153 fallocate 0x020342 thru 0x02bbbe (0xb87d bytes) 000301 1732429050.380878 mapwrite 0x023441 thru 0x02716c (0x3d2c bytes) 000302 1732429050.384893 punch from 0x022977 to 0x02ca84 (0xa10e bytes) 000303 1732429050.386349 read 0x0339c5 thru 0x03b841 (0x7e7d bytes) 000304 1732429050.386626 trunc from 0x040000 to 0x00e410 (0x31bef bytes) 000305 1732429050.390094 mapwrite 0x03e07c thru 0x03ffff (0x1f84 bytes) 000306 1732429050.400018 punch from 0x00cb20 to 0x014ea5 (0x8386 bytes) 000307 1732429050.401269 fallocate 0x024862 thru 0x02aa2f (0x61ce bytes) 000309 1732429050.402638 fallocate 0x00ece4 thru 0x015c0c (0x6f29 bytes) 000310 1732429050.404051 fallocate 0x01203e thru 0x01b0b6 (0x9079 bytes) 000311 1732429050.405370 fallocate 0x0085d7 thru 0x016481 (0xdeab bytes) 000312 1732429050.406743 mapwrite 0x039847 thru 0x03eda4 (0x555e bytes) 000313 1732429050.411325 mapread 0x025e55 thru 0x0269d3 (0xb7f bytes) 000314 1732429050.412864 MIRROR_RESYNC 4 mirrors 000315 1732429050.515269 trunc from 0x040000 to 0x03100b (0xeff4 bytes) 000316 1732429050.527373 trunc from 0x03100c to 0x02588d (0xb77e bytes) 000317 1732429050.531361 write 0x0379f0 thru 0x03d334 (0x5945 bytes) 000318 1732429050.532155 trunc from 0x03d335 to 0x02cc96 (0x1069e bytes) 000320 1732429050.535813 read 0x029a06 thru 0x02cc96 (0x3291 bytes) 000322 1732429050.537575 punch from 0x007144 to 0x00bf81 (0x4e3e bytes) 000323 1732429050.538693 mapread 0x01e158 thru 0x02ad93 (0xcc3c bytes) 000324 1732429050.545094 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000325 1732429050.585621 fallocate 0x02055f thru 0x02e9ad (0xe44f bytes) 000326 1732429050.587210 write 0x0365a6 thru 0x038fce (0x2a29 bytes) 000327 1732429050.589693 MIRROR_EXTEND to 4 mirrors 000328 1732429050.688077 mapread 0x008de7 thru 0x00b351 (0x256b bytes) 000329 1732429050.695320 mapwrite 0x002d18 thru 0x005d05 (0x2fee bytes) 000330 1732429050.720110 read 0x006d58 thru 0x00efef (0x8298 bytes) 000332 1732429050.720642 read 0x01efee thru 0x0213cc (0x23df bytes) 000333 1732429050.722998 trunc from 0x038fcf to 0x0347d0 (0x47fe bytes) 000334 1732429050.731399 mapwrite 0x0226b7 thru 0x025eaa (0x37f4 bytes) 000335 1732429050.735595 trunc from 0x0347d1 to 0x017c76 (0x1cb5a bytes) 000336 1732429050.738872 fallocate 0x0057bb thru 0x00f9c2 (0xa208 bytes) 000337 1732429050.741589 MIRROR_RESYNC 4 mirrors 000338 1732429050.871631 write 0x00425b thru 0x00d82d (0x95d3 bytes) 000339 1732429050.880563 mapread 0x017a32 thru 0x017c76 (0x245 bytes) 000340 1732429050.882808 write 0x0281d5 thru 0x037d5f (0xfb8b bytes) 000341 1732429050.883788 write 0x014f4b thru 0x0170bc (0x2172 bytes) 000343 1732429050.884505 read 0x006b01 thru 0x01589d (0xed9d bytes) 000344 1732429050.886220 write 0x0339d5 thru 0x037af4 (0x4120 bytes) 000345 1732429050.886925 punch from 0x032b0b to 0x037d5f (0x5255 bytes) 000346 1732429050.890493 punch from 0x01739b to 0x01a767 (0x33cd bytes) 000347 1732429050.891686 write 0x0359e1 thru 0x03ffff (0xa61f bytes) 000348 1732429050.894170 mapwrite 0x02c558 thru 0x02fb46 (0x35ef bytes) 000349 1732429050.900033 MIRROR_RESYNC 4 mirrors 000350 1732429051.019943 fallocate 0x02ec40 thru 0x03be34 (0xd1f5 bytes) 000352 1732429051.027785 read 0x012d48 thru 0x02175a (0xea13 bytes) 000353 1732429051.030039 fallocate 0x037288 thru 0x03e37e (0x70f7 bytes) 000354 1732429051.031398 read 0x008a12 thru 0x009b49 (0x1138 bytes) 000355 1732429051.035089 MIRROR_RESYNC 4 mirrors 000356 1732429051.148549 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000358 1732429051.199608 MIRROR_EXTEND to 4 mirrors 000359 1732429051.302255 read 0x023145 thru 0x02c8d8 (0x9794 bytes) 000360 1732429051.305752 punch from 0x01f7d5 to 0x0298b4 (0xa0e0 bytes) 000361 1732429051.314986 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000362 1732429051.353599 write 0x01190c thru 0x014e82 (0x3577 bytes) 000363 1732429051.356964 punch from 0x017cf1 to 0x019808 (0x1b18 bytes) 000364 1732429051.358334 mapread 0x000b51 thru 0x010217 (0xf6c7 bytes) 000365 1732429051.363196 read 0x03716b thru 0x03d644 (0x64da bytes) 000366 1732429051.365021 read 0x00cfc7 thru 0x019fb0 (0xcfea bytes) 000368 1732429051.367700 punch from 0x02d0bf to 0x0334a5 (0x63e7 bytes) 000370 1732429051.369600 fallocate 0x032f5a thru 0x03c05c (0x9103 bytes) 000371 1732429051.371005 read 0x026fb7 thru 0x02ef35 (0x7f7f bytes) 000374 1732429051.373529 trunc from 0x040000 to 0x018443 (0x27bbc bytes) 000375 1732429051.377354 mapread 0x003afb thru 0x00ec5b (0xb161 bytes) 000376 1732429051.378945 MIRROR_EXTEND to 4 mirrors 000377 1732429051.468105 read 0x001703 thru 0x001ce2 (0x5e0 bytes) 000378 1732429051.470695 write 0x038b33 thru 0x03ffff (0x74cd bytes) 000379 1732429051.478252 fallocate 0x012599 thru 0x021dac (0xf814 bytes) 000380 1732429051.479764 read 0x026a2c thru 0x0326b5 (0xbc8a bytes) 000383 1732429051.484056 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000384 1732429051.525156 punch from 0x01459c to 0x01e85c (0xa2c1 bytes) 000385 1732429051.526594 trunc from 0x040000 to 0x00b910 (0x346ef bytes) 000388 1732429051.534749 mapread 0x00792e thru 0x00b910 (0x3fe3 bytes) 000389 1732429051.539093 fallocate 0x03745f thru 0x037a38 (0x5da bytes) 000390 1732429051.540404 punch from 0x00a77c to 0x00b910 (0x1195 bytes) 000392 1732429051.541866 read 0x0055c4 thru 0x00a8f4 (0x5331 bytes) 000393 1732429051.543364 fallocate 0x031e71 thru 0x034c85 (0x2e15 bytes) 000394 1732429051.544641 write 0x03015a thru 0x03b760 (0xb607 bytes) 000395 1732429051.546848 mapwrite 0x0343f9 thru 0x03ffff (0xbc07 bytes) 000396 1732429051.556991 punch from 0x019732 to 0x025012 (0xb8e1 bytes) 000397 1732429051.558189 mapwrite 0x000afe thru 0x00a8a7 (0x9daa bytes) 000398 1732429051.564514 trunc from 0x040000 to 0x01b191 (0x24e6e bytes) 000399 1732429051.567786 mapwrite 0x03ea6d thru 0x03ffff (0x1593 bytes) 000402 1732429051.577769 mapread 0x01973e thru 0x025593 (0xbe56 bytes) 000403 1732429051.582432 punch from 0x02100b to 0x02971e (0x8714 bytes) 000405 1732429051.584008 punch from 0x0356b7 to 0x03ffff (0xa949 bytes) 000406 1732429051.585175 mapread 0x024882 thru 0x028c63 (0x43e2 bytes) 000407 1732429051.587390 mapwrite 0x00bf75 thru 0x019e24 (0xdeb0 bytes) 000408 1732429051.596469 MIRROR_RESYNC 3 mirrors 000410 1732429051.684977 punch from 0x00e014 to 0x01c42a (0xe417 bytes) 000411 1732429051.692598 mapwrite 0x02b588 thru 0x02d7f6 (0x226f bytes) 000412 1732429051.700572 mapwrite 0x025253 thru 0x02fa74 (0xa822 bytes) 000413 1732429051.707577 fallocate 0x02360a thru 0x02b362 (0x7d59 bytes) 000414 1732429051.709069 mapread 0x03b7b0 thru 0x03ffff (0x4850 bytes) 000415 1732429051.712308 MIRROR_RESYNC 3 mirrors 000416 1732429051.811027 read 0x03924b thru 0x03ffff (0x6db5 bytes) 000417 1732429051.812795 fallocate 0x01e48c thru 0x023e9d (0x5a12 bytes) 000420 1732429051.821207 write 0x012083 thru 0x01500a (0x2f88 bytes) 000422 1732429051.823703 write 0x0261a3 thru 0x0306d4 (0xa532 bytes) 000423 1732429051.826811 write 0x00a94e thru 0x0158ee (0xafa1 bytes) 000424 1732429051.829391 fallocate 0x009680 thru 0x0145ab (0xaf2c bytes) 000425 1732429051.830736 mapread 0x03eb73 thru 0x03ffff (0x148d bytes) 000426 1732429051.832818 read 0x00ae2d thru 0x00c589 (0x175d bytes) 000427 1732429051.833104 fallocate 0x02764b thru 0x032406 (0xadbc bytes) 000428 1732429051.834464 trunc from 0x040000 to 0x004009 (0x3bff6 bytes) 000429 1732429051.843008 punch from 0x0022ea to 0x004009 (0x1d20 bytes) 000430 1732429051.845193 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000431 1732429051.884123 mapwrite 0x039d14 thru 0x03ffff (0x62ec bytes) 000432 1732429051.898040 write 0x037223 thru 0x03ffff (0x8ddd bytes) 000433 1732429051.898639 punch from 0x00aa55 to 0x00cf80 (0x252c bytes) 000435 1732429051.900199 write 0x039a52 thru 0x03ffff (0x65ae bytes) 000436 1732429051.900965 mapread 0x01ae0a thru 0x02a7bc (0xf9b3 bytes) 000437 1732429051.907187 fallocate 0x014f03 thru 0x01da00 (0x8afe bytes) 000438 1732429051.910098 MIRROR_EXTEND to 3 mirrors 000439 1732429052.013488 trunc from 0x040000 to 0x018fb3 (0x2704c bytes) 000442 1732429052.026698 fallocate 0x02fe06 thru 0x037dd8 (0x7fd3 bytes) 000443 1732429052.028318 fallocate 0x017066 thru 0x0198c5 (0x2860 bytes) 000444 1732429052.029669 fallocate 0x01c29a thru 0x01e25f (0x1fc6 bytes) 000445 1732429052.030938 write 0x003689 thru 0x012056 (0xe9ce bytes) 000446 1732429052.034033 fallocate 0x0376d9 thru 0x03f25d (0x7b85 bytes) 000447 1732429052.036702 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000448 1732429052.081806 MIRROR_EXTEND to 3 mirrors 000449 1732429052.186151 punch from 0x02553c to 0x029844 (0x4309 bytes) 000451 1732429052.194598 mapwrite 0x035ec4 thru 0x03ffff (0xa13c bytes) 000453 1732429052.209169 read 0x02d31b thru 0x031829 (0x450f bytes) 000456 1732429052.211368 mapwrite 0x02d5b3 thru 0x036a83 (0x94d1 bytes) 000457 1732429052.215949 write 0x02a81a thru 0x0365e1 (0xbdc8 bytes) 000458 1732429052.218174 write 0x003755 thru 0x006b97 (0x3443 bytes) 000459 1732429052.220378 trunc from 0x040000 to 0x006783 (0x3987c bytes) 000461 1732429052.224472 read 0x00293e thru 0x006783 (0x3e46 bytes) 000462 1732429052.224851 mapwrite 0x03fad2 thru 0x03ffff (0x52e bytes) 000464 1732429052.235586 trunc from 0x040000 to 0x01f1f3 (0x20e0c bytes) 000465 1732429052.238966 mapwrite 0x00343a thru 0x00a038 (0x6bff bytes) 000466 1732429052.245026 MIRROR_RESYNC 3 mirrors 000468 1732429052.332347 trunc from 0x01f1f4 to 0x030edc (0x11ce9 bytes) 000469 1732429052.345019 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000471 1732429052.384425 punch from 0x004c92 to 0x011206 (0xc575 bytes) 000473 1732429052.386625 fallocate 0x033914 thru 0x03ffff (0xc6ec bytes) 000477 1732429052.389005 mapread 0x024867 thru 0x02b58f (0x6d29 bytes) 000478 1732429052.394243 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000479 1732429052.420269 mapread 0x02468b thru 0x02f232 (0xaba8 bytes) 000481 1732429052.426818 trunc from 0x030edd to 0x0310cb (0x1ef bytes) 000482 1732429052.431041 punch from 0x0048a5 to 0x011ffa (0xd756 bytes) 000483 1732429052.432461 trunc from 0x0310cc to 0x01dfab (0x13120 bytes) 000486 1732429052.436391 fallocate 0x00fce9 thru 0x01725f (0x7577 bytes) 000487 1732429052.437745 mapwrite 0x03c4ca thru 0x03ffff (0x3b36 bytes) 000489 1732429052.447762 fallocate 0x01d4b2 thru 0x02329a (0x5de9 bytes) 000490 1732429052.449258 fallocate 0x02062d thru 0x02edd3 (0xe7a7 bytes) 000491 1732429052.450417 mapwrite 0x006e65 thru 0x007d99 (0xf35 bytes) 000493 1732429052.457140 MIRROR_EXTEND to 2 mirrors 000494 1732429052.553259 punch from 0x026c79 to 0x02a494 (0x381c bytes) 000495 1732429052.561312 MIRROR_RESYNC 2 mirrors 000496 1732429052.669347 read 0x001045 thru 0x010e26 (0xfde2 bytes) 000497 1732429052.672652 MIRROR_EXTEND to 3 mirrors 000498 1732429052.787437 read 0x029899 thru 0x02d10a (0x3872 bytes) 000500 1732429052.792748 fallocate 0x032bd4 thru 0x03b43e (0x886b bytes) 000501 1732429052.800015 punch from 0x03750a to 0x03ffff (0x8af6 bytes) 000502 1732429052.802163 MIRROR_EXTEND to 4 mirrors 000503 1732429052.910477 read 0x01ae0e thru 0x020d9a (0x5f8d bytes) 000505 1732429052.915638 trunc from 0x040000 to 0x002009 (0x3dff6 bytes) 000506 1732429052.928582 mapread 0x00024b thru 0x002009 (0x1dbf bytes) 000507 1732429052.932115 trunc from 0x00200a to 0x0055ec (0x35e3 bytes) 000508 1732429052.936288 punch from 0x002582 to 0x0055ec (0x306b bytes) 000509 1732429052.937458 mapread 0x0015b9 thru 0x0055ec (0x4034 bytes) 000510 1732429052.939203 fallocate 0x02b9f9 thru 0x036225 (0xa82d bytes) 000511 1732429052.940609 mapread 0x0032c5 thru 0x0055ec (0x2328 bytes) 000512 1732429052.940990 read 0x000d7e thru 0x0035a9 (0x282c bytes) 000513 1732429052.941337 trunc from 0x0055ed to 0x023c15 (0x1e629 bytes) 000515 1732429052.945134 fallocate 0x018712 thru 0x020ffe (0x88ed bytes) 000516 1732429052.946473 fallocate 0x003b88 thru 0x004e06 (0x127f bytes) 000517 1732429052.947887 fallocate 0x036599 thru 0x037e52 (0x18ba bytes) 000519 1732429052.949551 trunc from 0x037e53 to 0x02b283 (0xcbcf bytes) 000520 1732429052.952982 trunc from 0x02b284 to 0x014930 (0x16953 bytes) 000521 1732429052.957291 MIRROR_EXTEND to 5 mirrors 000522 1732429053.051803 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000523 1732429053.091502 read 0x0031c4 thru 0x00d6bd (0xa4fa bytes) 000524 1732429053.093675 read 0x00bab5 thru 0x0114bd (0x5a09 bytes) 000525 1732429053.095633 punch from 0x0016d5 to 0x00fb0f (0xe43b bytes) 000526 1732429053.099898 MIRROR_RESYNC 4 mirrors 000528 1732429053.210694 fallocate 0x0060e7 thru 0x00e88f (0x87a9 bytes) 000530 1732429053.218337 write 0x010eb1 thru 0x013457 (0x25a7 bytes) 000531 1732429053.220672 read 0x0068f7 thru 0x00dcb6 (0x73c0 bytes) 000533 1732429053.223887 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000535 1732429053.265528 fallocate 0x00c260 thru 0x016e54 (0xabf5 bytes) 000536 1732429053.267032 fallocate 0x02fb1c thru 0x03076d (0xc52 bytes) 000537 1732429053.268492 mapread 0x001d08 thru 0x00d6ca (0xb9c3 bytes) 000538 1732429053.274089 trunc from 0x014931 to 0x01f4fd (0xabcd bytes) 000539 1732429053.278118 punch from 0x019f1f to 0x01f0cc (0x51ae bytes) 000540 1732429053.279392 write 0x0385f9 thru 0x039adf (0x14e7 bytes) 000541 1732429053.280122 mapread 0x01f0c6 thru 0x023855 (0x4790 bytes) 000542 1732429053.282382 mapread 0x0283dc thru 0x02fc29 (0x784e bytes) 000543 1732429053.284519 punch from 0x01f42a to 0x028c94 (0x986b bytes) 000544 1732429053.286884 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000545 1732429053.326079 trunc from 0x039ae0 to 0x034d4f (0x4d90 bytes) 000547 1732429053.331409 MIRROR_RESYNC 2 mirrors 000548 1732429053.412758 punch from 0x031c1c to 0x034d4f (0x3134 bytes) 000549 1732429053.419988 mapwrite 0x0324e7 thru 0x03c5b2 (0xa0cc bytes) 000551 1732429053.436952 punch from 0x014657 to 0x015a8f (0x1439 bytes) 000552 1732429053.438354 read 0x027dd7 thru 0x03124e (0x9478 bytes) 000553 1732429053.440394 mapread 0x003b3c thru 0x006b92 (0x3057 bytes) 000554 1732429053.444026 MIRROR_EXTEND to 3 mirrors 000556 1732429053.534660 read 0x023903 thru 0x024023 (0x721 bytes) 000558 1732429053.538952 mapwrite 0x015d93 thru 0x0246cf (0xe93d bytes) 000559 1732429053.572401 punch from 0x030da2 to 0x034641 (0x38a0 bytes) 000560 1732429053.573805 write 0x0210bf thru 0x02d993 (0xc8d5 bytes) 000561 1732429053.576693 punch from 0x03366f to 0x034e98 (0x182a bytes) 000562 1732429053.579071 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000563 1732429053.623124 mapwrite 0x00e447 thru 0x013a19 (0x55d3 bytes) 000564 1732429053.631634 fallocate 0x03ebfd thru 0x03ffff (0x1403 bytes) 000565 1732429053.633375 mapread 0x033c4d thru 0x03c5b2 (0x8966 bytes) 000566 1732429053.637433 mapwrite 0x034cd5 thru 0x03ffff (0xb32b bytes) 000567 1732429053.655318 MIRROR_EXTEND to 3 mirrors 000569 1732429053.762269 MIRROR_RESYNC 3 mirrors 000571 1732429053.847167 punch from 0x03874b to 0x03ffff (0x78b5 bytes) 000572 1732429053.854689 write 0x03bc6d thru 0x03ffff (0x4393 bytes) 000574 1732429053.857417 read 0x01b4c8 thru 0x01ec32 (0x376b bytes) 000578 1732429053.860242 write 0x02988b thru 0x02ff6d (0x66e3 bytes) 000579 1732429053.863119 read 0x00e6ed thru 0x01979a (0xb0ae bytes) 000580 1732429053.866191 trunc from 0x040000 to 0x012792 (0x2d86d bytes) 000581 1732429053.874315 trunc from 0x012793 to 0x001403 (0x1138f bytes) 000583 1732429053.877474 mapwrite 0x010c1f thru 0x018916 (0x7cf8 bytes) 000584 1732429053.896241 fallocate 0x03eb49 thru 0x03f530 (0x9e8 bytes) 000585 1732429053.897806 read 0x00a39f thru 0x016853 (0xc4b5 bytes) 000586 1732429053.900025 mapread 0x012fee thru 0x018916 (0x5929 bytes) 000587 1732429053.900549 punch from 0x013a5a to 0x01614f (0x26f6 bytes) 000588 1732429053.901876 fallocate 0x024869 thru 0x02f23d (0xa9d5 bytes) 000589 1732429053.903192 write 0x018e2d thru 0x01fb45 (0x6d19 bytes) 000591 1732429053.905036 read 0x00347b thru 0x00cdb3 (0x9939 bytes) 000592 1732429053.906580 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000593 1732429053.945180 mapread 0x0112e2 thru 0x017cc8 (0x69e7 bytes) 000594 1732429053.949428 write 0x01bca7 thru 0x01e29a (0x25f4 bytes) 000597 1732429053.952435 trunc from 0x02f23e to 0x00ccb0 (0x2258d bytes) 000599 1732429053.956482 punch from 0x00a68e to 0x00ccb0 (0x2623 bytes) 000601 1732429053.957985 punch from 0x0078f7 to 0x00a4ec (0x2bf6 bytes) 000602 1732429053.960112 MIRROR_RESYNC 2 mirrors 000603 1732429054.025876 mapread 0x00bd8e thru 0x00ccb0 (0xf23 bytes) 000604 1732429054.027200 mapwrite 0x011b93 thru 0x01901f (0x748d bytes) 000605 1732429054.047858 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000606 1732429054.073242 write 0x01c6d2 thru 0x02a19d (0xdacc bytes) 000608 1732429054.075406 trunc from 0x02a19e to 0x026a28 (0x3775 bytes) 000609 1732429054.078762 trunc from 0x026a29 to 0x015520 (0x11508 bytes) 000612 1732429054.082745 trunc from 0x015521 to 0x02173d (0xc21d bytes) 000614 1732429054.085886 mapread 0x00130d thru 0x003f47 (0x2c3b bytes) 000615 1732429054.089952 fallocate 0x002771 thru 0x00c6b4 (0x9f44 bytes) 000616 1732429054.091394 mapwrite 0x01fbec thru 0x02fb76 (0xff8b bytes) 000617 1732429054.104083 mapwrite 0x019fff thru 0x022cd8 (0x8cda bytes) 000618 1732429054.108339 fallocate 0x031299 thru 0x03549e (0x4206 bytes) 000619 1732429054.109660 fallocate 0x024eda thru 0x029217 (0x433e bytes) 000621 1732429054.111364 mapread 0x01c883 thru 0x020379 (0x3af7 bytes) 000622 1732429054.111789 mapwrite 0x0127ba thru 0x016c50 (0x4497 bytes) 000624 1732429054.117714 punch from 0x003ad4 to 0x01009e (0xc5cb bytes) 000625 1732429054.119058 mapread 0x0067ff thru 0x00d29c (0x6a9e bytes) 000626 1732429054.120999 punch from 0x026ed3 to 0x02e5e3 (0x7711 bytes) 000627 1732429054.122250 trunc from 0x02fb77 to 0x0354dc (0x5966 bytes) 000628 1732429054.125521 read 0x01b028 thru 0x01ec7f (0x3c58 bytes) 000629 1732429054.125781 punch from 0x017dcb to 0x01f6ec (0x7922 bytes) 000631 1732429054.127215 write 0x037d49 thru 0x03aba3 (0x2e5b bytes) 000632 1732429054.127688 fallocate 0x0399c9 thru 0x03aaa7 (0x10df bytes) 000638 1732429054.130138 read 0x0240c8 thru 0x0245b5 (0x4ee bytes) 000639 1732429054.130399 read 0x01b2a9 thru 0x02716d (0xbec5 bytes) 000641 1732429054.133334 trunc from 0x03aba4 to 0x005f39 (0x34c6a bytes) 000642 1732429054.136607 fallocate 0x0191f9 thru 0x024bd6 (0xb9de bytes) 000643 1732429054.137899 trunc from 0x024bd7 to 0x008220 (0x1c9b6 bytes) 000645 1732429054.141675 punch from 0x0070bb to 0x008220 (0x1166 bytes) 000647 1732429054.143261 punch from 0x000a92 to 0x00521f (0x478e bytes) 000648 1732429054.144657 fallocate 0x03af87 thru 0x03d5cd (0x2647 bytes) 000650 1732429054.146680 trunc from 0x03d5ce to 0x03cc7c (0x951 bytes) 000651 1732429054.150155 mapread 0x01dc46 thru 0x0295e0 (0xb99b bytes) 000653 1732429054.163921 read 0x00c3a9 thru 0x00ef6d (0x2bc5 bytes) 000654 1732429054.166591 read 0x01eae0 thru 0x02d04e (0xe56f bytes) 000658 1732429054.167705 mapread 0x039af3 thru 0x03cc7c (0x318a bytes) 000659 1732429054.168125 read 0x032978 thru 0x03cc7c (0xa305 bytes) 000660 1732429054.168454 punch from 0x03478b to 0x03cc7c (0x84f2 bytes) 000661 1732429054.169698 trunc from 0x03cc7d to 0x00e14b (0x2eb31 bytes) 000663 1732429054.173378 read 0x006713 thru 0x008119 (0x1a07 bytes) 000664 1732429054.173768 mapwrite 0x029344 thru 0x029974 (0x631 bytes) 000666 1732429054.183172 write 0x0054a8 thru 0x00a48b (0x4fe4 bytes) 000667 1732429054.183764 read 0x00c124 thru 0x01be08 (0xfce5 bytes) 000670 1732429054.184496 trunc from 0x029975 to 0x0138ff (0x16075 bytes) 000671 1732429054.187874 read 0x00a471 thru 0x00cc93 (0x2823 bytes) 000672 1732429054.188226 punch from 0x00412d to 0x006eb5 (0x2d89 bytes) 000673 1732429054.190953 read 0x00310c thru 0x005963 (0x2858 bytes) 000674 1732429054.193359 MIRROR_EXTEND to 2 mirrors 000675 1732429054.280857 punch from 0x010996 to 0x0138ff (0x2f6a bytes) 000676 1732429054.287568 mapwrite 0x019be8 thru 0x029051 (0xf46a bytes) 000678 1732429054.308720 mapwrite 0x02681b thru 0x02ad65 (0x454b bytes) 000680 1732429054.318036 trunc from 0x02ad66 to 0x019dd9 (0x10f8c bytes) 000682 1732429054.321985 mapwrite 0x0066d7 thru 0x013fee (0xd918 bytes) 000683 1732429054.330863 mapwrite 0x002e7a thru 0x004852 (0x19d9 bytes) 000684 1732429054.333704 mapread 0x00c3fd thru 0x0170c8 (0xaccc bytes) 000685 1732429054.334125 fallocate 0x018354 thru 0x01c4cf (0x417c bytes) 000686 1732429054.335443 mapwrite 0x017ec3 thru 0x01d1ce (0x530c bytes) 000688 1732429054.345845 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000689 1732429054.371523 write 0x038252 thru 0x03ffff (0x7dae bytes) 000690 1732429054.373240 punch from 0x01ba3a to 0x01e432 (0x29f9 bytes) 000693 1732429054.375875 fallocate 0x02063f thru 0x021e3d (0x17ff bytes) 000694 1732429054.378328 MIRROR_EXTEND to 2 mirrors 000696 1732429054.489939 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000697 1732429054.515717 fallocate 0x008811 thru 0x0116ff (0x8eef bytes) 000698 1732429054.518721 mapwrite 0x038c50 thru 0x03ffff (0x73b0 bytes) 000699 1732429054.527339 fallocate 0x03c0a9 thru 0x03ffff (0x3f57 bytes) 000700 1732429054.528573 mapread 0x00e81d thru 0x01b20f (0xc9f3 bytes) 000702 1732429054.532475 write 0x002103 thru 0x008734 (0x6632 bytes) 000705 1732429054.536128 write 0x012f55 thru 0x01a8e8 (0x7994 bytes) 000706 1732429054.536727 mapwrite 0x00a6c1 thru 0x01723f (0xcb7f bytes) 000707 1732429054.541745 mapread 0x029450 thru 0x034f38 (0xbae9 bytes) 000708 1732429054.545386 mapwrite 0x012d4d thru 0x01e5b2 (0xb866 bytes) 000710 1732429054.550499 fallocate 0x03e45c thru 0x03ffff (0x1ba4 bytes) 000711 1732429054.551910 punch from 0x010feb to 0x013f1b (0x2f31 bytes) 000712 1732429054.553177 trunc from 0x040000 to 0x03a703 (0x58fc bytes) 000714 1732429054.560016 write 0x011cea thru 0x01cf34 (0xb24b bytes) 000716 1732429054.561797 trunc from 0x03a704 to 0x009864 (0x30e9f bytes) 000717 1732429054.565624 fallocate 0x0290f5 thru 0x02e26e (0x517a bytes) 000718 1732429054.567069 write 0x0003b8 thru 0x00a91d (0xa566 bytes) 000719 1732429054.569358 mapwrite 0x010a21 thru 0x01fe86 (0xf466 bytes) 000720 1732429054.579123 mapwrite 0x001ff0 thru 0x008c33 (0x6c44 bytes) 000721 1732429054.583378 trunc from 0x02e26f to 0x00556f (0x28cff bytes) 000722 1732429054.586628 mapwrite 0x0326e6 thru 0x03bb00 (0x941b bytes) 000723 1732429054.598258 mapread 0x00cc16 thru 0x01373a (0x6b25 bytes) 000724 1732429054.600509 read 0x031cab thru 0x035d69 (0x40bf bytes) 000725 1732429054.600935 write 0x035280 thru 0x03d1f1 (0x7f72 bytes) 000726 1732429054.601749 punch from 0x01b660 to 0x01b7d9 (0x17a bytes) 000727 1732429054.603042 trunc from 0x03d1f2 to 0x017718 (0x25ad9 bytes) 000728 1732429054.606558 fallocate 0x028cc3 thru 0x035fb7 (0xd2f5 bytes) 000729 1732429054.608106 trunc from 0x035fb8 to 0x01695f (0x1f658 bytes) 000733 1732429054.612120 read 0x014327 thru 0x01695f (0x2639 bytes) 000734 1732429054.612390 mapwrite 0x037731 thru 0x03f9a6 (0x8276 bytes) 000737 1732429054.623633 fallocate 0x00cc51 thru 0x01366b (0x6a1b bytes) 000738 1732429054.626071 MIRROR_EXTEND to 2 mirrors 000739 1732429054.714739 MIRROR_EXTEND to 3 mirrors 000740 1732429054.810142 mapwrite 0x027bc3 thru 0x02dffa (0x6438 bytes) 000741 1732429054.842791 punch from 0x01b6ea to 0x021fd5 (0x68ec bytes) 000742 1732429054.844388 trunc from 0x03f9a7 to 0x00ff85 (0x2fa21 bytes) 000744 1732429054.853705 read 0x005363 thru 0x00635e (0xffc bytes) 000745 1732429054.855228 trunc from 0x00ff86 to 0x031f1e (0x21f99 bytes) 000748 1732429054.860076 MIRROR_EXTEND to 4 mirrors 000750 1732429054.954261 fallocate 0x013bb6 thru 0x01e354 (0xa79f bytes) 000751 1732429054.962159 mapwrite 0x00316a thru 0x007475 (0x430c bytes) 000752 1732429054.971359 fallocate 0x03da7a thru 0x03f739 (0x1cc0 bytes) 000753 1732429054.972872 mapwrite 0x00e886 thru 0x0146af (0x5e2a bytes) 000754 1732429054.979537 write 0x03048e thru 0x038e59 (0x89cc bytes) 000755 1732429054.983029 trunc from 0x038e5a to 0x0337c3 (0x5696 bytes) 000756 1732429054.990427 trunc from 0x0337c4 to 0x00b418 (0x283ab bytes) 000757 1732429054.994995 MIRROR_EXTEND to 5 mirrors 000759 1732429055.098082 read 0x00840c thru 0x00b418 (0x300d bytes) 000760 1732429055.102867 punch from 0x0083ec to 0x00b418 (0x302d bytes) 000761 1732429055.112312 MIRROR_EXTEND to 6 mirrors 000762 1732429055.202719 fallocate 0x03ebd3 thru 0x03ffff (0x142d bytes) 000763 1732429055.210696 punch from 0x032714 to 0x03c8c0 (0xa1ad bytes) 000765 1732429055.212596 write 0x00c3a8 thru 0x00e0c7 (0x1d20 bytes) 000766 1732429055.216665 punch from 0x025021 to 0x033350 (0xe330 bytes) 000767 1732429055.218172 mapwrite 0x00bdbe thru 0x01b731 (0xf974 bytes) 000768 1732429055.229752 trunc from 0x040000 to 0x0379d9 (0x8626 bytes) 000769 1732429055.234605 fallocate 0x00d608 thru 0x012b05 (0x54fe bytes) 000770 1732429055.236346 mapwrite 0x01a874 thru 0x01b009 (0x796 bytes) 000771 1732429055.241125 mapread 0x027a03 thru 0x029b20 (0x211e bytes) 000772 1732429055.244678 MIRROR_EXTEND to 7 mirrors 000773 1732429055.340896 mapwrite 0x019b8a thru 0x01bd8f (0x2206 bytes) 000774 1732429055.366397 read 0x031894 thru 0x0379d9 (0x6146 bytes) 000775 1732429055.368388 write 0x00d73c thru 0x013937 (0x61fc bytes) 000776 1732429055.371404 read 0x01d1ab thru 0x026dec (0x9c42 bytes) 000777 1732429055.373702 mapread 0x004b45 thru 0x007456 (0x2912 bytes) 000778 1732429055.375353 MIRROR_RESYNC 7 mirrors 000779 1732429055.599228 mapread 0x003bd2 thru 0x005510 (0x193f bytes) 000780 1732429055.603650 trunc from 0x0379da to 0x018e32 (0x1eba7 bytes) 000781 1732429055.616954 trunc from 0x018e33 to 0x015bf7 (0x323b bytes) 000782 1732429055.621245 mapread 0x00d074 thru 0x015bf7 (0x8b84 bytes) 000784 1732429055.624174 mapread 0x0133a2 thru 0x015b75 (0x27d4 bytes) 000786 1732429055.625089 fallocate 0x011f77 thru 0x021ad4 (0xfb5e bytes) 000788 1732429055.626685 mapwrite 0x009ecd thru 0x0191db (0xf30f bytes) 000789 1732429055.635411 write 0x022583 thru 0x02a0a2 (0x7b20 bytes) 000790 1732429055.636361 mapread 0x00e07e thru 0x014b14 (0x6a97 bytes) 000791 1732429055.638016 MIRROR_EXTEND to 8 mirrors 000792 1732429055.739874 trunc from 0x02a0a3 to 0x0055c0 (0x24ae2 bytes) 000794 1732429055.754767 trunc from 0x0055c1 to 0x03f133 (0x39b73 bytes) 000795 1732429055.759376 write 0x027d05 thru 0x02fb96 (0x7e92 bytes) 000796 1732429055.762018 mapread 0x039bce thru 0x03f133 (0x5566 bytes) 000798 1732429055.765691 MIRROR_EXTEND to 9 mirrors 000799 1732429055.867658 fallocate 0x036e24 thru 0x03f70b (0x88e8 bytes) 000800 1732429055.875823 mapread 0x032ccd thru 0x038bac (0x5ee0 bytes) 000801 1732429055.879929 punch from 0x01520a to 0x01c0c9 (0x6ec0 bytes) 000802 1732429055.881298 mapwrite 0x012caa thru 0x0167a4 (0x3afb bytes) 000803 1732429055.888380 fallocate 0x03d60d thru 0x03ffff (0x29f3 bytes) 000804 1732429055.889870 trunc from 0x03f70c to 0x019c60 (0x25aab bytes) 000805 1732429055.899384 mapwrite 0x008f2e thru 0x017d70 (0xee43 bytes) 000807 1732429055.908861 write 0x00beea thru 0x00f1a1 (0x32b8 bytes) 000810 1732429055.909958 punch from 0x00e8a2 to 0x01413a (0x5899 bytes) 000811 1732429055.913108 punch from 0x00c34e to 0x012143 (0x5df6 bytes) 000812 1732429055.914569 read 0x001d84 thru 0x00f733 (0xd9b0 bytes) 000813 1732429055.916332 read 0x006110 thru 0x00f05b (0x8f4c bytes) 000814 1732429055.916731 mapread 0x00a526 thru 0x019c60 (0xf73b bytes) 000816 1732429055.919283 write 0x00b5a6 thru 0x00bf08 (0x963 bytes) 000817 1732429055.919702 read 0x00d5eb thru 0x018601 (0xb017 bytes) 000818 1732429055.919995 mapwrite 0x020662 thru 0x029707 (0x90a6 bytes) 000819 1732429055.932345 mapwrite 0x00582e thru 0x0157e7 (0xffba bytes) 000820 1732429055.939271 mapread 0x0103fb thru 0x0125eb (0x21f1 bytes) 000822 1732429055.940281 read 0x02649d thru 0x029707 (0x326b bytes) 000823 1732429055.940717 read 0x028cc3 thru 0x029707 (0xa45 bytes) 000824 1732429055.942488 MIRROR_EXTEND to 10 mirrors 000825 1732429056.031878 punch from 0x0044bf to 0x00e2f6 (0x9e38 bytes) 000827 1732429056.040527 trunc from 0x029708 to 0x00c1c4 (0x1d543 bytes) 000828 1732429056.049026 fallocate 0x002ccc thru 0x0075d4 (0x4909 bytes) 000829 1732429056.050271 trunc from 0x00c1c5 to 0x00a990 (0x1834 bytes) 000830 1732429056.053478 write 0x00200d thru 0x00617d (0x4171 bytes) 000831 1732429056.055917 punch from 0x007691 to 0x00a990 (0x3300 bytes) 000832 1732429056.057050 read 0x00267b thru 0x00a990 (0x8316 bytes) 000833 1732429056.058529 fallocate 0x019ef2 thru 0x02175f (0x786e bytes) 000834 1732429056.059650 fallocate 0x009351 thru 0x016550 (0xd200 bytes) 000835 1732429056.061939 MIRROR_EXTEND to 11 mirrors 000836 1732429056.154072 fallocate 0x010ba5 thru 0x01af4d (0xa3a9 bytes) 000837 1732429056.162154 write 0x007457 thru 0x013bc7 (0xc771 bytes) 000838 1732429056.166183 MIRROR_RESYNC 11 mirrors 000839 1732429056.329048 read 0x003115 thru 0x003cd3 (0xbbf bytes) 000840 1732429056.330337 trunc from 0x013bc8 to 0x00613e (0xda89 bytes) 000841 1732429056.342702 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000843 1732429056.386454 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000845 1732429056.427886 read 0x0016a8 thru 0x00613e (0x4a97 bytes) 000846 1732429056.430032 mapread 0x00459f thru 0x00613e (0x1ba0 bytes) 000850 1732429056.431317 read 0x00382d thru 0x00613e (0x2912 bytes) 000851 1732429056.432753 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000852 1732429056.478402 punch from 0x005954 to 0x00613e (0x7eb bytes) 000853 1732429056.479846 read 0x00566b thru 0x00613e (0xad4 bytes) 000854 1732429056.481367 trunc from 0x00613f to 0x03fc01 (0x39ac3 bytes) 000855 1732429056.486258 trunc from 0x03fc02 to 0x02e836 (0x113cb bytes) 000856 1732429056.489513 mapread 0x0147eb thru 0x020e41 (0xc657 bytes) 000858 1732429056.496057 trunc from 0x02e837 to 0x0338ab (0x5075 bytes) 000859 1732429056.499521 fallocate 0x036c2b thru 0x03b87d (0x4c53 bytes) 000860 1732429056.500739 write 0x023637 thru 0x02f4c1 (0xbe8b bytes) 000861 1732429056.502733 fallocate 0x00f13a thru 0x016d07 (0x7bce bytes) 000862 1732429056.505167 MIRROR_EXTEND to 9 mirrors 000864 1732429056.609848 mapread 0x025eca thru 0x02816c (0x22a3 bytes) 000865 1732429056.616727 trunc from 0x0338ac to 0x00ee51 (0x24a5a bytes) 000866 1732429056.630792 punch from 0x0042fa to 0x00a235 (0x5f3c bytes) 000867 1732429056.632231 write 0x00ccb2 thru 0x00e214 (0x1563 bytes) 000868 1732429056.634438 mapwrite 0x0090de thru 0x00bee5 (0x2e08 bytes) 000869 1732429056.639303 mapwrite 0x013879 thru 0x01bf1f (0x86a7 bytes) 000870 1732429056.652181 mapread 0x0012f4 thru 0x00d4bb (0xc1c8 bytes) 000871 1732429056.654202 mapwrite 0x01ff4c thru 0x0298eb (0x99a0 bytes) 000872 1732429056.666784 punch from 0x019537 to 0x01a8a3 (0x136d bytes) 000873 1732429056.668215 mapread 0x00a1cd thru 0x00b212 (0x1046 bytes) 000874 1732429056.668612 write 0x00af12 thru 0x00e982 (0x3a71 bytes) 000875 1732429056.669147 fallocate 0x03d13e thru 0x03e249 (0x110c bytes) 000878 1732429056.672124 MIRROR_EXTEND to 10 mirrors 000879 1732429056.784671 mapwrite 0x0327a6 thru 0x037139 (0x4994 bytes) 000880 1732429056.818650 punch from 0x00d349 to 0x01bc63 (0xe91b bytes) 000881 1732429056.820285 punch from 0x00940f to 0x00be4e (0x2a40 bytes) 000882 1732429056.821606 mapread 0x011542 thru 0x0181dc (0x6c9b bytes) 000884 1732429056.824480 fallocate 0x028113 thru 0x03806a (0xff58 bytes) 000886 1732429056.826607 write 0x01b525 thru 0x01cfec (0x1ac8 bytes) 000887 1732429056.828256 punch from 0x0282c6 to 0x031389 (0x90c4 bytes) 000888 1732429056.829620 punch from 0x01cd24 to 0x01d131 (0x40e bytes) 000890 1732429056.832403 mapwrite 0x02e591 thru 0x0388ee (0xa35e bytes) 000891 1732429056.839237 read 0x00f22c thru 0x013de0 (0x4bb5 bytes) 000892 1732429056.840581 fallocate 0x033a98 thru 0x03b9cd (0x7f36 bytes) 000894 1732429056.843343 MIRROR_RESYNC 10 mirrors 000895 1732429057.386509 punch from 0x02a952 to 0x02bbbb (0x126a bytes) 000896 1732429057.394148 write 0x039bc9 thru 0x03ffff (0x6437 bytes) 000897 1732429057.397105 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000898 1732429057.442975 trunc from 0x040000 to 0x02f9b0 (0x1064f bytes) 000899 1732429057.447422 write 0x006206 thru 0x013ec2 (0xdcbd bytes) 000900 1732429057.450247 trunc from 0x02f9b1 to 0x00024b (0x2f765 bytes) 000901 1732429057.454043 trunc from 0x00024c to 0x021c5b (0x21a10 bytes) 000902 1732429057.457204 fallocate 0x020982 thru 0x023ffe (0x367d bytes) 000903 1732429057.458624 read 0x0047ee thru 0x00d560 (0x8d73 bytes) 000904 1732429057.460734 trunc from 0x023fff to 0x026fc1 (0x2fc3 bytes) 000906 1732429057.465438 MIRROR_RESYNC 9 mirrors 000907 1732429057.649239 MIRROR_EXTEND to 10 mirrors 000908 1732429057.764291 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000909 1732429057.833318 mapread 0x021691 thru 0x026fc1 (0x5931 bytes) 000910 1732429057.838390 MIRROR_RESYNC 9 mirrors 000912 1732429057.891438 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000913 1732429057.956576 fallocate 0x00ffb2 thru 0x013e69 (0x3eb8 bytes) 000914 1732429057.966622 trunc from 0x026fc2 to 0x022820 (0x47a1 bytes) 000915 1732429057.975166 write 0x02bda6 thru 0x02cb4d (0xda8 bytes) 000916 1732429057.975974 write 0x00d4f2 thru 0x01749b (0x9faa bytes) 000918 1732429057.979137 mapread 0x01603b thru 0x0163d8 (0x39e bytes) 000919 1732429057.979646 mapwrite 0x02c8d7 thru 0x031e37 (0x5561 bytes) 000920 1732429057.991400 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000921 1732429058.035704 trunc from 0x031e38 to 0x007c57 (0x2a1e0 bytes) 000923 1732429058.041808 MIRROR_EXTEND to 8 mirrors 000925 1732429058.143902 punch from 0x000edd to 0x0062d4 (0x53f8 bytes) 000926 1732429058.152077 punch from 0x006cd7 to 0x007c57 (0xf81 bytes) 000928 1732429058.154897 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000929 1732429058.200982 mapread 0x0041e5 thru 0x0043f7 (0x213 bytes) 000930 1732429058.202798 write 0x02ad43 thru 0x038be8 (0xdea6 bytes) 000932 1732429058.205128 mapread 0x010a14 thru 0x012487 (0x1a74 bytes) 000933 1732429058.208280 mapread 0x008811 thru 0x018041 (0xf831 bytes) 000934 1732429058.210370 trunc from 0x038be9 to 0x0248fb (0x142ed bytes) 000935 1732429058.213755 write 0x01c28f thru 0x01daea (0x185c bytes) 000937 1732429058.217720 MIRROR_EXTEND to 8 mirrors 000939 1732429058.325589 mapread 0x01c6c8 thru 0x01d585 (0xebe bytes) 000940 1732429058.330765 write 0x0123dd thru 0x0200e5 (0xdd09 bytes) 000941 1732429058.342402 trunc from 0x0248fc to 0x00b337 (0x195c4 bytes) 000942 1732429058.347013 mapread 0x0069a3 thru 0x00b337 (0x4995 bytes) 000943 1732429058.349064 read 0x0086f5 thru 0x00b337 (0x2c43 bytes) 000944 1732429058.349381 mapwrite 0x03295d thru 0x03a027 (0x76cb bytes) 000945 1732429058.360379 trunc from 0x03a028 to 0x00c5fd (0x2da2a bytes) 000946 1732429058.364750 MIRROR_RESYNC 8 mirrors 000948 1732429058.534756 mapread 0x008582 thru 0x00c5fd (0x407c bytes) 000950 1732429058.538630 fallocate 0x0389d9 thru 0x03f23e (0x6866 bytes) 000951 1732429058.546488 fallocate 0x03c7bf thru 0x03ffff (0x3841 bytes) 000952 1732429058.547895 punch from 0x01dd27 to 0x0296ee (0xb9c8 bytes) 000954 1732429058.549355 mapwrite 0x007aa5 thru 0x00a6f1 (0x2c4d bytes) 000955 1732429058.556098 trunc from 0x040000 to 0x00e7bf (0x31840 bytes) 000956 1732429058.560402 trunc from 0x00e7c0 to 0x01a26f (0xbab0 bytes) 000957 1732429058.563464 mapwrite 0x035201 thru 0x03ffff (0xadff bytes) 000958 1732429058.576266 read 0x01795c thru 0x01dc53 (0x62f8 bytes) 000960 1732429058.578537 mapread 0x01b0bc thru 0x02adbb (0xfd00 bytes) 000961 1732429058.583519 MIRROR_EXTEND to 9 mirrors 000962 1732429058.682382 read 0x03de9d thru 0x03f561 (0x16c5 bytes) 000963 1732429058.685390 write 0x01c00c thru 0x01fdc9 (0x3dbe bytes) 000964 1732429058.696092 trunc from 0x040000 to 0x007625 (0x389da bytes) 000965 1732429058.704056 punch from 0x007298 to 0x007625 (0x38e bytes) 000966 1732429058.706430 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000967 1732429058.748136 read 0x002dd1 thru 0x007625 (0x4855 bytes) 000968 1732429058.750557 write 0x0127ba thru 0x017004 (0x484b bytes) 000969 1732429058.752485 write 0x019bf1 thru 0x01e708 (0x4b18 bytes) 000970 1732429058.753050 punch from 0x005e03 to 0x00975e (0x395c bytes) 000971 1732429058.754391 read 0x0047d0 thru 0x009c05 (0x5436 bytes) 000972 1732429058.756609 read 0x011505 thru 0x01df2e (0xca2a bytes) 000973 1732429058.757016 trunc from 0x01e709 to 0x012c7a (0xba8e bytes) 000975 1732429058.760842 read 0x00c30c thru 0x00c8ad (0x5a2 bytes) 000976 1732429058.761104 write 0x003d22 thru 0x00bf8a (0x8269 bytes) 000977 1732429058.762618 MIRROR_EXTEND to 9 mirrors 000978 1732429058.857076 read 0x0037b7 thru 0x012c7a (0xf4c4 bytes) 000979 1732429058.860574 fallocate 0x03aebe thru 0x03ffff (0x5142 bytes) 000980 1732429058.870400 trunc from 0x040000 to 0x02c5ce (0x13a31 bytes) 000981 1732429058.878434 mapwrite 0x01c009 thru 0x02989c (0xd894 bytes) 000983 1732429058.890741 punch from 0x01857a to 0x01f84d (0x72d4 bytes) 000985 1732429058.892714 write 0x0288c0 thru 0x031e58 (0x9599 bytes) 000986 1732429058.893541 read 0x0033dc thru 0x00fbb9 (0xc7de bytes) 000988 1732429058.896665 mapread 0x01474f thru 0x022a44 (0xe2f6 bytes) 000989 1732429058.897204 trunc from 0x031e59 to 0x03a545 (0x86ed bytes) 000990 1732429058.900765 mapread 0x021df8 thru 0x0304ee (0xe6f7 bytes) 000991 1732429058.901399 fallocate 0x01d747 thru 0x01df6e (0x828 bytes) 000992 1732429058.902740 punch from 0x02bee1 to 0x034a9c (0x8bbc bytes) 000993 1732429058.906104 read 0x02c699 thru 0x0308dc (0x4244 bytes) 000996 1732429058.908757 punch from 0x018e5d to 0x01f7e9 (0x698d bytes) 000997 1732429058.910003 mapwrite 0x033508 thru 0x03ffff (0xcaf8 bytes) 000998 1732429058.921447 punch from 0x03f015 to 0x03ffff (0xfeb bytes) 000999 1732429058.923006 fallocate 0x03af88 thru 0x03ffff (0x5078 bytes) 001000 1732429058.924501 fallocate 0x00e605 thru 0x0172f9 (0x8cf5 bytes) All operations completed A-OK! PASS 70a (12s) == sanity-flr test 200a: stress test ===================== 01:17:39 (1732429059) Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre2 Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4124416 .. Extending file size to 7552672 .. failed Extending file size to 8555680 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8596896 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8912960 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8804288 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8939776 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8900896 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8948160 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9205792 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 8517 8518 8520 8521 8524 192.168.203.125@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg325-client.virtnet /mnt/lustre2 (opts:) 192.168.203.125@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg325-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:18:43 (1732429123) Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre2 Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5130592 .. Extending file size to 3964992 .. failed Extending file size to 5357120 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7267104 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8733824 .. failed Extending file size to 7375168 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8906592 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8189824 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8915648 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8614432 .. Extending file size to 8995936 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9141568 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9336928 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 12158 12159 12161 12163 12164 192.168.203.125@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg325-client.virtnet /mnt/lustre2 (opts:) 192.168.203.125@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg325-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:19:46 (1732429186) Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2800 1284888 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3168 1284520 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21996 3585024 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15900 3591120 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37896 7176144 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2800 1284888 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3168 1284520 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 21996 3585024 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 15900 3591120 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37896 7176144 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0324616 s, 64.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0453497 s, 46.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.125@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg325-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:19:50 (1732429190) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.023841 s, 88.0 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 01:19:52 (1732429192) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa7:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xd2:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:19:53 (1732429193) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.020723 s, 101 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1069:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x106a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x106b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:19:55 (1732429195) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0226607 s, 92.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x106a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x106b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x106c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x106a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x106b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x106c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x106d:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:19:57 (1732429197) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0212377 s, 98.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x106c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x106e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x106c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x106e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x106f:0x0] } PASS 204c (22s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:20:19 (1732429219) Waiting for MDT destroys to complete oleg325-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg325-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg325-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135986 s, 77.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x106d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1071:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1070:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x106d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1071:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1070:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x106e:0x0] } PASS 204d (33s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:20:52 (1732429252) Waiting for MDT destroys to complete oleg325-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg325-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg325-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0204062 s, 103 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (32s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:21:24 (1732429284) Waiting for MDT destroys to complete oleg325-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg325-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg325-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool oleg325-server: Pool lustre.test_204f created Adding targets to pool oleg325-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg325-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0203904 s, 103 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1072:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x106f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1072:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x106f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg325-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg325-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg325-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (46s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:22:10 (1732429330) PASS 205a (1s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:22:11 (1732429331) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:22:13 (1732429333) oleg325-server: Pool lustre.test_206 created Adding targets to pool oleg325-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg325-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg325-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg325-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg325-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:22:27 (1732429347) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0197986 s, 53.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00914898 s, 115 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x8d5da912 Mirror 3: 0x8d5da912 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:22:29 (1732429349) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:22:30 (1732429350) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (0s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:22:30 (1732429350) Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00720689 s, 0.6 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1640000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x16a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x16a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x16a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x19c0000 Read this (4 bytes): BBBB 192.168.203.125@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg325-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:22:44 (1732429364) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0028137 s, 2.9 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:22:45 (1732429365) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010341 s, 101 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc2 lmm_fid: [0x200000405:0xc2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc2 lmm_fid: [0x200000405:0xc2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xdf:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611330 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-24 01:22:46.000000000 -0500 Modify: 2024-11-24 01:22:46.000000000 -0500 Change: 2024-11-24 01:22:46.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:22:47 (1732429367) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107932 s, 97.2 MB/s oleg325-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg325-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg325-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg325-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 oleg325-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg325-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg325-server: O/280000401/d0/e0: File not found by ext2_lookup PASS 210b (12s) == sanity-flr test complete, duration 694 sec ============ 01:22:59 (1732429379) === sanity-flr: start cleanup 01:22:59 (1732429379) === === sanity-flr: finish cleanup 01:23:01 (1732429381) ===