-----============= acceptance-small: sanity-flr ============----- Fri Apr 25 18:23:06 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg430-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 18:23:13 (1745619793) === oleg430-client.virtnet: executing check_config_client /mnt/lustre oleg430-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg430-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5975800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5975800.idle_timeout=debug disable quota as required oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 18:23:24 (1745619804) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7031] == sanity-flr test 0a: lfs mirror create with -N option == 18:23:24 (1745619804) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:23:31 (1745619811) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:23:32 (1745619812) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:23:33 (1745619813) PASS 0d (11s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:23:44 (1745619824) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:23:46 (1745619826) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 18:23:47 (1745619827) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b5975800.stats=clear osc.lustre-OST0001-osc-ffff8800b5975800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:23:51 (1745619831) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:23:55 (1745619835) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:23:59 (1745619839) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:24:00 (1745619840) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 18:24:05 (1745619845) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:24:08 (1745619848) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:24:10 (1745619850) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:24:13 (1745619853) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:24:17 (1745619857) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:24:19 (1745619859) 15+0 records in 15+0 records out 15728640 bytes (16 MB) copied, 0.860946 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.104194 s, 10.1 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:24:24 (1745619864) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0587408 s, 107 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:24:28 (1745619868) Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5975800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5975800.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5975800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5975800.ost_server_uuid in DISCONN state after 1 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg430-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (28s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:24:56 (1745619896) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.218978 s, 43.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5975800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5975800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:25:11 (1745619911) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5975800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5975800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5975800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5975800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg430-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (34s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:25:45 (1745619945) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.08935 s, 48.1 MB/s -rw-r--r-- 1 root root 52428800 Apr 25 18:25 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.505s user 0m0.000s sys 0m0.135s Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5975800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5975800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.468s user 0m0.000s sys 0m0.122s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5975800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5975800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.561s user 0m0.000s sys 0m0.125s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg430-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg430-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5975800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5975800.ost_server_uuid in FULL state after 0 sec PASS 33b (33s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:26:18 (1745619978) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:26:19 (1745619979) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:26:20 (1745619980) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 18:26:21 (1745619981) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 18:26:24 (1745619984) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.227978 s, 55.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:26:29 (1745619989) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0165907 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0170866 s, 61.4 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:26:32 (1745619992) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:26:36 (1745619996) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00254686 s, 402 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00176687 s, 580 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00447677 s, 229 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00123206 s, 831 kB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 18:26:40 (1745620000) create test files with size 15729872 .. 6cc8165f92df363bed0bfeea0b9102d9 - 0cc8dedb24a16db3ccefbd5e11018025 - 7c7daab41bfa1b855f9eedec1a2ca934 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg430-client.virtnet /mnt/lustre (opts:) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre PASS 37 (9s) == sanity-flr test 38: resync ============================ 18:26:49 (1745620009) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:26:55 (1745620015) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:26:57 (1745620017) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0435691 s, 24.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128698 s, 81.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 18:27:01 (1745620021) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0543342 s, 38.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.144188 s, 29.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.102791 s, 20.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 18:27:06 (1745620026) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 18:27:07 (1745620027) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:27:09 (1745620029) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193325 s, 54.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 18:27:12 (1745620032) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:27:13 (1745620033) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:27:16 (1745620036) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.100508 s, 104 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg430-client.virtnet /mnt/lustre (opts:) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (19s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:27:35 (1745620055) llite.lustre-ffff8800aa59b800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.101945 s, 103 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg430-client.virtnet /mnt/lustre (opts:) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800aa591000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:27:41 (1745620061) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (12s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:27:53 (1745620073) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (12s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:28:05 (1745620085) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 18:28:06 (1745620086) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.106075 s, 29.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0769036 s, 40.9 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 4807b3b04592b0f6d05981f96b14ce19 - mirror 1 checksum: e00426dccbaa2a1fe8c957c3ddcb6a48 - mirror 2 checksum: 4807b3b04592b0f6d05981f96b14ce19 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 25 18:28:08 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 25 18:28:08 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 4807b3b04592b0f6d05981f96b14ce19 - mirror 1 checksum: e00426dccbaa2a1fe8c957c3ddcb6a48 - mirror 2 checksum: e00426dccbaa2a1fe8c957c3ddcb6a48 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 25 18:28:08 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 25 18:28:08 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:28:11 (1745620091) Failing mds1 on oleg430-server Stopping /mnt/lustre-mds1 (opts:) on oleg430-server 18:28:14 (1745620094) shut down facet: mds1 facet_host: oleg430-server facet_failover_host: oleg430-server Failover mds1 to oleg430-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0000 18:28:28 (1745620108) targets are mounted 18:28:28 (1745620108) facet_failover done oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00893089 s, 117 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (22s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:28:33 (1745620113) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (0s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:28:34 (1745620114) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:28:34 (1745620114) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00259518 s, 6.3 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:28:36 (1745620116) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:28:37 (1745620117) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:28:38 (1745620118) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:28:43 (1745620123) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:29:02 (1745620142) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:29:20 (1745620160) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:29:37 (1745620177) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 18:29:39 (1745620179) mirror create pid 22562 mirror split pid 22563 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4892: 22562 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4892: 22563 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 18:30:41 (1745620241) Seed set to 242 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1745620242.397660 write 0x01cced thru 0x0266a0 (0x99b4 bytes) 000004 1745620242.400541 read 0x002506 thru 0x00e9f7 (0xc4f2 bytes) truncating to largest ever: 0x13038 000005 1745620242.405447 trunc from 0x0266a1 to 0x013037 (0x13669 bytes) 000007 1745620242.417163 read 0x000c5f thru 0x002296 (0x1638 bytes) 000008 1745620242.421826 mapwrite 0x011de1 thru 0x01770c (0x592c bytes) 000009 1745620242.489912 read 0x00176f thru 0x003152 (0x19e4 bytes) 000010 1745620242.490307 read 0x0055fe thru 0x005811 (0x214 bytes) 000011 1745620242.490617 write 0x00ea8b thru 0x00fe62 (0x13d8 bytes) truncating to largest ever: 0x2cc2d 000012 1745620242.491179 trunc from 0x01770d to 0x02cc2c (0x15520 bytes) 000014 1745620242.497898 read 0x02466d thru 0x02cc2c (0x85c0 bytes) 000015 1745620242.502346 write 0x01cedc thru 0x020e41 (0x3f66 bytes) 000016 1745620242.502781 mapwrite 0x01ad3c thru 0x0221ff (0x74c4 bytes) 000017 1745620242.538187 trunc from 0x02cc2d to 0x023d2d (0x8eff bytes) 000018 1745620242.547295 mapwrite 0x02f65d thru 0x0390dd (0x9a81 bytes) 000019 1745620242.615117 MIRROR_EXTEND to 2 mirrors 000021 1745620242.807272 read 0x01f67a thru 0x02e305 (0xec8c bytes) 000022 1745620242.811123 mapwrite 0x0094ee thru 0x01513b (0xbc4e bytes) 000025 1745620242.925136 write 0x00d9c6 thru 0x017fe6 (0xa621 bytes) 000026 1745620242.926037 write 0x0238b5 thru 0x0295f4 (0x5d40 bytes) 000027 1745620242.934304 write 0x012f1a thru 0x01bdd3 (0x8eba bytes) 000028 1745620242.935095 read 0x00c76e thru 0x0101fc (0x3a8f bytes) 000030 1745620242.935771 write 0x0197c9 thru 0x022645 (0x8e7d bytes) 000032 1745620242.936930 trunc from 0x0390de to 0x029d83 (0xf35a bytes) truncating to largest ever: 0x3dece 000034 1745620242.957306 trunc from 0x029d84 to 0x03decd (0x1414a bytes) 000036 1745620242.967142 mapwrite 0x01f474 thru 0x02cc47 (0xd7d4 bytes) 000039 1745620243.033252 trunc from 0x03dece to 0x030b4a (0xd383 bytes) 000040 1745620243.048403 mapwrite 0x0132eb thru 0x01c40d (0x9123 bytes) 000041 1745620243.105016 MIRROR_EXTEND to 3 mirrors 000042 1745620243.334880 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000048 1745620243.446670 trunc from 0x030b4b to 0x03a1db (0x9691 bytes) 000049 1745620243.474104 mapwrite 0x00401c thru 0x008916 (0x48fb bytes) 000050 1745620243.541721 mapwrite 0x03f35d thru 0x03ff36 (0xbda bytes) 000052 1745620243.592422 trunc from 0x03ff37 to 0x00743c (0x38afa bytes) 000054 1745620243.600610 write 0x039f77 thru 0x03a9bf (0xa49 bytes) 000056 1745620243.602057 read 0x013cc0 thru 0x020ccf (0xd010 bytes) 000058 1745620243.612213 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000059 1745620243.690536 mapwrite 0x008202 thru 0x00d8a5 (0x56a4 bytes) 000061 1745620243.741249 write 0x03369c thru 0x039f3a (0x689f bytes) 000062 1745620243.750517 read 0x02d30b thru 0x0351dc (0x7ed2 bytes) 000063 1745620243.751703 mapwrite 0x018511 thru 0x0280f5 (0xfbe5 bytes) 000065 1745620243.802932 mapread 0x00d985 thru 0x01a75c (0xcdd8 bytes) 000067 1745620243.804290 read 0x029974 thru 0x030f23 (0x75b0 bytes) 000069 1745620243.811967 MIRROR_EXTEND to 2 mirrors 000070 1745620244.039222 read 0x011073 thru 0x01a4ff (0x948d bytes) 000071 1745620244.043998 write 0x00fcad thru 0x013b2d (0x3e81 bytes) 000072 1745620244.090230 mapread 0x0021f9 thru 0x0112e4 (0xf0ec bytes) 000073 1745620244.108776 write 0x017e59 thru 0x024efe (0xd0a6 bytes) 000074 1745620244.120919 MIRROR_EXTEND to 3 mirrors 000075 1745620244.397411 write 0x024c3c thru 0x0324d8 (0xd89d bytes) 000076 1745620244.448033 mapread 0x0248ea thru 0x027276 (0x298d bytes) 000077 1745620244.448753 write 0x02763d thru 0x02e25e (0x6c22 bytes) 000078 1745620244.449457 write 0x002787 thru 0x00560c (0x2e86 bytes) 000079 1745620244.456585 write 0x0133f5 thru 0x01d5f2 (0xa1fe bytes) 000082 1745620244.468614 read 0x004b50 thru 0x009da9 (0x525a bytes) 000087 1745620244.472290 read 0x02a160 thru 0x02d904 (0x37a5 bytes) 000088 1745620244.472742 mapread 0x01d2b5 thru 0x025ec4 (0x8c10 bytes) 000089 1745620244.480108 read 0x00405d thru 0x00bf50 (0x7ef4 bytes) 000090 1745620244.480692 trunc from 0x03a9c0 to 0x01aacf (0x1fef0 bytes) 000092 1745620244.517772 read 0x01961d thru 0x01aacf (0x14b3 bytes) 000093 1745620244.518782 mapwrite 0x002b7f thru 0x00c289 (0x970b bytes) 000094 1745620244.564548 trunc from 0x01aad0 to 0x01bd58 (0x1289 bytes) 000095 1745620244.573371 MIRROR_RESYNC 3 mirrors 000098 1745620244.807697 trunc from 0x01bd59 to 0x03446f (0x18717 bytes) 000099 1745620244.846715 mapwrite 0x022e5c thru 0x028f53 (0x60f8 bytes) 000100 1745620244.878125 mapwrite 0x03f872 thru 0x03ffff (0x78e bytes) 000102 1745620244.906856 read 0x037491 thru 0x03ffff (0x8b6f bytes) 000104 1745620244.909342 write 0x0226bb thru 0x02fdbe (0xd704 bytes) 000106 1745620244.912407 mapread 0x00a357 thru 0x010804 (0x64ae bytes) 000107 1745620244.919026 trunc from 0x040000 to 0x029ef5 (0x1610a bytes) 000108 1745620244.927129 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000109 1745620245.014598 mapwrite 0x01ef45 thru 0x021cf2 (0x2dae bytes) 000111 1745620245.052729 read 0x009783 thru 0x00d63b (0x3eb9 bytes) 000112 1745620245.056234 write 0x02f97c thru 0x031d93 (0x2418 bytes) 000113 1745620245.056915 write 0x00a096 thru 0x013958 (0x98c3 bytes) 000114 1745620245.060425 trunc from 0x031d94 to 0x0204ca (0x118c9 bytes) 000117 1745620245.068696 mapwrite 0x02ca17 thru 0x02d79a (0xd84 bytes) 000119 1745620245.113489 write 0x007cf8 thru 0x00f14d (0x7456 bytes) 000121 1745620245.120031 mapwrite 0x035266 thru 0x03ffff (0xad9a bytes) 000123 1745620245.162022 mapwrite 0x01f549 thru 0x02c9cb (0xd483 bytes) 000126 1745620245.194588 write 0x03118b thru 0x03ffff (0xee75 bytes) 000127 1745620245.195513 mapwrite 0x015b12 thru 0x0206b3 (0xaba2 bytes) 000128 1745620245.225767 trunc from 0x040000 to 0x032604 (0xd9fb bytes) 000129 1745620245.232026 trunc from 0x032605 to 0x00ceef (0x25715 bytes) 000130 1745620245.239684 MIRROR_EXTEND to 3 mirrors 000131 1745620245.394920 mapread 0x007847 thru 0x00ceef (0x56a9 bytes) 000133 1745620245.403059 trunc from 0x00cef0 to 0x02bb67 (0x1ec78 bytes) 000137 1745620245.446324 trunc from 0x02bb68 to 0x030eef (0x5388 bytes) 000138 1745620245.453739 mapread 0x010b4d thru 0x01db98 (0xd04c bytes) 000139 1745620245.464046 mapread 0x01cb5c thru 0x028c33 (0xc0d8 bytes) 000140 1745620245.471062 mapwrite 0x02273c thru 0x022ed8 (0x79d bytes) 000141 1745620245.505563 write 0x03a1ed thru 0x03ffff (0x5e13 bytes) 000145 1745620245.507426 write 0x03dca2 thru 0x03ffff (0x235e bytes) 000147 1745620245.508343 mapread 0x00c976 thru 0x014540 (0x7bcb bytes) 000148 1745620245.515656 trunc from 0x040000 to 0x03529f (0xad60 bytes) 000149 1745620245.520849 write 0x01566b thru 0x01bb44 (0x64da bytes) 000151 1745620245.521869 trunc from 0x0352a0 to 0x01823b (0x1d064 bytes) 000152 1745620245.530636 write 0x03bc83 thru 0x03ffff (0x437d bytes) 000153 1745620245.531632 mapwrite 0x0078b0 thru 0x01681d (0xef6e bytes) 000154 1745620245.563991 write 0x033008 thru 0x03f23e (0xc237 bytes) 000155 1745620245.570598 write 0x03031e thru 0x03e890 (0xe573 bytes) 000157 1745620245.572541 mapwrite 0x0289da thru 0x02d550 (0x4b77 bytes) 000158 1745620245.607752 read 0x002f0f thru 0x00d454 (0xa546 bytes) 000159 1745620245.610848 trunc from 0x040000 to 0x00be07 (0x341f8 bytes) 000160 1745620245.619981 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000161 1745620245.718985 mapwrite 0x00208a thru 0x002457 (0x3ce bytes) 000162 1745620245.746746 mapread 0x006aa6 thru 0x00b6c6 (0x4c21 bytes) 000164 1745620245.755733 mapread 0x0097fb thru 0x00be07 (0x260d bytes) 000167 1745620245.760146 MIRROR_EXTEND to 3 mirrors 000168 1745620245.982242 read 0x004f15 thru 0x00be07 (0x6ef3 bytes) 000170 1745620245.986844 trunc from 0x00be08 to 0x02786c (0x1ba65 bytes) 000172 1745620246.039995 write 0x0255d8 thru 0x02ab6d (0x5596 bytes) 000174 1745620246.049083 mapread 0x02aa83 thru 0x02ab6d (0xeb bytes) 000175 1745620246.051886 MIRROR_RESYNC 3 mirrors 000176 1745620246.314897 write 0x014fa1 thru 0x01d723 (0x8783 bytes) 000177 1745620246.359056 read 0x024e65 thru 0x02ab6d (0x5d09 bytes) 000179 1745620246.364169 mapwrite 0x0131db thru 0x0213fb (0xe221 bytes) 000181 1745620246.418358 write 0x02b257 thru 0x0349c1 (0x976b bytes) 000184 1745620246.420529 write 0x034169 thru 0x03ffff (0xbe97 bytes) 000189 1745620246.423543 trunc from 0x040000 to 0x011b21 (0x2e4de bytes) 000190 1745620246.443631 mapread 0x002b6c thru 0x0086a5 (0x5b3a bytes) 000192 1745620246.453659 mapread 0x009a8d thru 0x011b21 (0x8095 bytes) 000194 1745620246.454625 mapwrite 0x003c3b thru 0x011907 (0xdccd bytes) 000196 1745620246.495491 mapwrite 0x02fae3 thru 0x03aec7 (0xb3e5 bytes) 000197 1745620246.573454 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000198 1745620246.664840 mapwrite 0x015f72 thru 0x022039 (0xc0c8 bytes) 000199 1745620246.723894 read 0x0215af thru 0x0269c1 (0x5413 bytes) 000200 1745620246.724399 read 0x020f63 thru 0x02cf94 (0xc032 bytes) 000202 1745620246.732228 trunc from 0x03aec8 to 0x0207b6 (0x1a711 bytes) 000204 1745620246.741345 trunc from 0x0207b7 to 0x010d40 (0xfa76 bytes) 000206 1745620246.749014 mapread 0x003d8d thru 0x00cbaa (0x8e1e bytes) 000207 1745620246.749960 mapwrite 0x02f7c8 thru 0x03f772 (0xffab bytes) 000210 1745620246.848967 write 0x025f55 thru 0x0345a0 (0xe64c bytes) 000213 1745620246.854007 write 0x023e7c thru 0x031d48 (0xdecd bytes) 000216 1745620246.859479 read 0x0260a4 thru 0x02e4f3 (0x8450 bytes) 000218 1745620246.860997 mapread 0x0204a1 thru 0x0264ef (0x604f bytes) 000219 1745620246.865582 trunc from 0x03f773 to 0x03a5d8 (0x519a bytes) 000221 1745620246.875173 read 0x013645 thru 0x01f3d4 (0xbd90 bytes) 000224 1745620246.878516 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000225 1745620246.953060 trunc from 0x03a5d9 to 0x01bb89 (0x1ea4f bytes) 000228 1745620246.963655 write 0x01a5e2 thru 0x024f1d (0xa93c bytes) 000231 1745620246.968352 mapread 0x01056b thru 0x015cb1 (0x5747 bytes) 000233 1745620246.975662 read 0x009f0f thru 0x01529b (0xb38d bytes) 000236 1745620246.980887 mapread 0x022ff9 thru 0x024f1d (0x1f25 bytes) 000237 1745620246.981341 write 0x0195a5 thru 0x01f06f (0x5acb bytes) 000238 1745620246.983086 write 0x0286a2 thru 0x029dcf (0x172e bytes) 000239 1745620246.983676 trunc from 0x029dd0 to 0x009c53 (0x2017c bytes) 000240 1745620246.990968 trunc from 0x009c54 to 0x00cb6d (0x2f1a bytes) 000241 1745620246.997296 read 0x008415 thru 0x00cb6d (0x4759 bytes) 000245 1745620247.002343 write 0x029d30 thru 0x036898 (0xcb69 bytes) 000247 1745620247.003533 trunc from 0x036899 to 0x013420 (0x23478 bytes) 000248 1745620247.011260 write 0x02e71f thru 0x038d6e (0xa650 bytes) 000249 1745620247.012503 write 0x009048 thru 0x018161 (0xf11a bytes) 000250 1745620247.016552 read 0x004734 thru 0x0121f0 (0xdabd bytes) 000252 1745620247.017417 read 0x03354e thru 0x038baa (0x565d bytes) 000253 1745620247.017767 write 0x03dd3f thru 0x03ffff (0x22c1 bytes) 000255 1745620247.021581 read 0x0147dd thru 0x01a5b1 (0x5dd5 bytes) 000261 1745620247.029679 read 0x0261f6 thru 0x033ef1 (0xdcfc bytes) 000263 1745620247.031445 trunc from 0x040000 to 0x03c2a5 (0x3d5a bytes) 000264 1745620247.038564 trunc from 0x03c2a6 to 0x01459e (0x27d07 bytes) 000265 1745620247.048821 write 0x0316b9 thru 0x037e64 (0x67ac bytes) 000266 1745620247.051699 MIRROR_EXTEND to 2 mirrors 000267 1745620247.253067 MIRROR_EXTEND to 3 mirrors 000276 1745620247.460888 trunc from 0x037e65 to 0x0349a2 (0x34c2 bytes) 000278 1745620247.499049 mapread 0x008769 thru 0x00b593 (0x2e2b bytes) 000280 1745620247.509083 mapread 0x0075a2 thru 0x00cd34 (0x5793 bytes) 000281 1745620247.509599 mapwrite 0x02e36b thru 0x02f820 (0x14b6 bytes) 000285 1745620247.549233 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000286 1745620247.624851 mapwrite 0x02196b thru 0x025382 (0x3a18 bytes) 000288 1745620247.663466 mapread 0x02da26 thru 0x02e485 (0xa60 bytes) 000289 1745620247.669160 read 0x01384d thru 0x01a4c0 (0x6c74 bytes) 000291 1745620247.673517 trunc from 0x0349a3 to 0x01e195 (0x1680d bytes) truncating to largest ever: 0x3f20a 000292 1745620247.679488 trunc from 0x01e196 to 0x03f209 (0x21074 bytes) 000293 1745620247.687011 mapwrite 0x021e67 thru 0x029afd (0x7c97 bytes) 000296 1745620247.727099 MIRROR_EXTEND to 3 mirrors 000297 1745620247.899598 mapread 0x030a7f thru 0x03f209 (0xe78b bytes) 000299 1745620247.908749 trunc from 0x03f20a to 0x02069a (0x1eb6f bytes) 000300 1745620247.955013 read 0x01eca1 thru 0x02069a (0x19fa bytes) 000301 1745620247.960165 MIRROR_RESYNC 3 mirrors 000302 1745620248.195164 write 0x030f49 thru 0x0328ad (0x1965 bytes) 000303 1745620248.221674 read 0x00a81e thru 0x01a749 (0xff2c bytes) 000304 1745620248.225969 read 0x00d2e5 thru 0x0185aa (0xb2c6 bytes) 000305 1745620248.228199 MIRROR_EXTEND to 4 mirrors 000307 1745620248.418570 MIRROR_RESYNC 4 mirrors 000309 1745620248.619515 trunc from 0x0328ae to 0x020af7 (0x11db6 bytes) 000310 1745620248.660809 read 0x005cf4 thru 0x0092c9 (0x35d6 bytes) 000314 1745620248.665232 mapread 0x002ee5 thru 0x01228b (0xf3a7 bytes) 000316 1745620248.676189 trunc from 0x020af8 to 0x005545 (0x1b5b2 bytes) 000318 1745620248.685596 mapwrite 0x02d380 thru 0x03851c (0xb19d bytes) 000321 1745620248.731296 MIRROR_EXTEND to 5 mirrors 000323 1745620248.887702 mapwrite 0x006e69 thru 0x00bc9d (0x4e35 bytes) 000327 1745620248.957921 mapread 0x0207fa thru 0x029858 (0x905f bytes) 000328 1745620248.968135 trunc from 0x03851d to 0x014496 (0x24086 bytes) 000329 1745620248.980686 write 0x01d566 thru 0x024b50 (0x75eb bytes) 000331 1745620248.982923 trunc from 0x024b51 to 0x027014 (0x24c4 bytes) 000333 1745620248.990461 write 0x0311f7 thru 0x038abb (0x78c5 bytes) 000336 1745620248.991670 mapwrite 0x03d54d thru 0x03ffff (0x2ab3 bytes) 000340 1745620249.027703 MIRROR_RESYNC 5 mirrors 000342 1745620249.282595 mapread 0x034561 thru 0x03d5d3 (0x9073 bytes) 000343 1745620249.289087 write 0x00ec08 thru 0x01278c (0x3b85 bytes) 000345 1745620249.311516 mapread 0x03dbb7 thru 0x03ffff (0x2449 bytes) 000346 1745620249.315149 write 0x001a8d thru 0x00ec13 (0xd187 bytes) 000347 1745620249.318695 mapread 0x02c0c0 thru 0x034efc (0x8e3d bytes) 000348 1745620249.326237 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000349 1745620249.409695 mapread 0x004bc0 thru 0x009db4 (0x51f5 bytes) 000350 1745620249.418135 mapwrite 0x0120e1 thru 0x01f4f1 (0xd411 bytes) 000351 1745620249.453944 mapwrite 0x00d413 thru 0x010d88 (0x3976 bytes) 000352 1745620249.477540 write 0x0115d9 thru 0x01ce3f (0xb867 bytes) 000353 1745620249.481274 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000355 1745620249.563680 trunc from 0x040000 to 0x0043be (0x3bc41 bytes) 000357 1745620249.572754 trunc from 0x0043bf to 0x006778 (0x23ba bytes) 000359 1745620249.578528 mapwrite 0x007aa6 thru 0x012c07 (0xb162 bytes) 000360 1745620249.625106 mapwrite 0x004b10 thru 0x00f02d (0xa51e bytes) 000362 1745620249.648263 trunc from 0x012c08 to 0x02c34b (0x19744 bytes) 000363 1745620249.654650 read 0x011066 thru 0x0134ee (0x2489 bytes) 000364 1745620249.657454 read 0x012d1c thru 0x021bb4 (0xee99 bytes) 000365 1745620249.661303 trunc from 0x02c34c to 0x01b267 (0x110e4 bytes) 000366 1745620249.671015 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000367 1745620249.734548 write 0x038509 thru 0x03ffff (0x7af7 bytes) 000368 1745620249.735481 mapwrite 0x036e16 thru 0x03d8ec (0x6ad7 bytes) 000369 1745620249.765992 mapwrite 0x012fca thru 0x01ee8a (0xbec1 bytes) 000372 1745620249.802230 mapwrite 0x00870b thru 0x0161f8 (0xdaee bytes) 000377 1745620249.834180 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000378 1745620249.879883 mapwrite 0x01be57 thru 0x023d8e (0x7f38 bytes) 000383 1745620249.915376 mapwrite 0x0326bd thru 0x03464b (0x1f8f bytes) 000384 1745620249.937846 mapread 0x005b14 thru 0x0066c3 (0xbb0 bytes) 000389 1745620249.945165 trunc from 0x040000 to 0x00d7f9 (0x32806 bytes) 000391 1745620249.952277 mapwrite 0x0007d0 thru 0x00b8a0 (0xb0d1 bytes) 000392 1745620249.978318 write 0x029fe7 thru 0x0357a2 (0xb7bc bytes) 000393 1745620249.979402 mapwrite 0x03285b thru 0x03dd53 (0xb4f9 bytes) 000396 1745620250.011643 mapwrite 0x02f039 thru 0x035073 (0x603b bytes) 000397 1745620250.035182 mapread 0x034b0e thru 0x0358fa (0xded bytes) 000398 1745620250.035688 write 0x006b68 thru 0x014bde (0xe077 bytes) 000399 1745620250.038855 trunc from 0x03dd54 to 0x01996b (0x243e8 bytes) truncating to largest ever: 0x3f520 000403 1745620250.045847 trunc from 0x01996c to 0x03f51f (0x25bb4 bytes) 000404 1745620250.050768 mapread 0x00ece5 thru 0x01e762 (0xfa7e bytes) 000405 1745620250.058058 read 0x02b9c3 thru 0x02ec5b (0x3299 bytes) 000407 1745620250.061165 read 0x002b85 thru 0x01155f (0xe9db bytes) 000410 1745620250.062212 mapwrite 0x003e66 thru 0x00db0d (0x9ca8 bytes) 000411 1745620250.089970 mapwrite 0x03fe29 thru 0x03ffff (0x1d7 bytes) 000414 1745620250.128086 write 0x03401c thru 0x035ca1 (0x1c86 bytes) 000416 1745620250.128899 mapread 0x01a261 thru 0x027f0b (0xdcab bytes) 000420 1745620250.130419 trunc from 0x040000 to 0x03326a (0xcd95 bytes) 000423 1745620250.136741 mapread 0x02eafc thru 0x030a30 (0x1f35 bytes) 000425 1745620250.137728 mapread 0x00921c thru 0x014ebf (0xbca4 bytes) 000427 1745620250.138685 mapwrite 0x023971 thru 0x027cd9 (0x4369 bytes) 000428 1745620250.161091 mapread 0x0226c5 thru 0x02399e (0x12da bytes) 000430 1745620250.161696 mapread 0x02f253 thru 0x031a1e (0x27cc bytes) 000431 1745620250.162067 read 0x016c1a thru 0x01886e (0x1c55 bytes) 000432 1745620250.162654 mapwrite 0x02e057 thru 0x02f8f9 (0x18a3 bytes) 000435 1745620250.181472 read 0x01138f thru 0x011897 (0x509 bytes) 000436 1745620250.182027 mapread 0x01ace2 thru 0x02a7f7 (0xfb16 bytes) 000437 1745620250.182633 trunc from 0x03326b to 0x0068e0 (0x2c98a bytes) 000438 1745620250.190103 mapread 0x0008a4 thru 0x0068e0 (0x603d bytes) 000439 1745620250.191145 mapread 0x00396d thru 0x0054a9 (0x1b3d bytes) 000440 1745620250.191729 write 0x03e39e thru 0x03fd9e (0x1a01 bytes) 000441 1745620250.192547 mapwrite 0x020729 thru 0x02f9ca (0xf2a2 bytes) 000442 1745620250.231939 mapread 0x007869 thru 0x00d17f (0x5917 bytes) 000444 1745620250.233212 write 0x005ebc thru 0x009028 (0x316d bytes) 000445 1745620250.233893 mapread 0x032cf9 thru 0x03fc14 (0xcf1c bytes) 000446 1745620250.234450 mapread 0x009abe thru 0x00ed6e (0x52b1 bytes) 000447 1745620250.235008 trunc from 0x03fd9f to 0x01c6f9 (0x236a5 bytes) 000451 1745620250.250706 mapread 0x005310 thru 0x00bd39 (0x6a2a bytes) 000453 1745620250.251726 mapread 0x01288d thru 0x01886a (0x5fde bytes) 000455 1745620250.252661 mapwrite 0x035dcf thru 0x03b762 (0x5994 bytes) 000459 1745620250.301954 mapread 0x035eb4 thru 0x03b762 (0x58af bytes) 000461 1745620250.303686 mapread 0x02feed thru 0x03b762 (0xb876 bytes) 000464 1745620250.316047 mapread 0x005c40 thru 0x0061d1 (0x592 bytes) 000469 1745620250.317779 trunc from 0x03b763 to 0x005f30 (0x35832 bytes) 000470 1745620250.327305 write 0x004255 thru 0x00cf48 (0x8cf4 bytes) 000471 1745620250.328227 write 0x0108de thru 0x0115d5 (0xcf8 bytes) 000472 1745620250.331598 MIRROR_EXTEND to 2 mirrors 000473 1745620250.518250 MIRROR_EXTEND to 3 mirrors 000474 1745620250.676371 mapread 0x0004aa thru 0x008d4a (0x88a1 bytes) 000477 1745620250.685419 trunc from 0x0115d6 to 0x01b027 (0x9a52 bytes) 000481 1745620250.729073 trunc from 0x01b028 to 0x025927 (0xa900 bytes) 000485 1745620250.736744 mapwrite 0x01f4d8 thru 0x02b77e (0xc2a7 bytes) 000486 1745620250.771739 mapread 0x01e1bc thru 0x027901 (0x9746 bytes) 000488 1745620250.772878 write 0x024a43 thru 0x028b4a (0x4108 bytes) 000491 1745620250.774104 trunc from 0x02b77f to 0x00d3fd (0x1e381 bytes) 000495 1745620250.782355 read 0x00ce60 thru 0x00d3fd (0x59e bytes) 000499 1745620250.786229 read 0x00745d thru 0x00b5ba (0x415e bytes) 000500 1745620250.791467 write 0x01a9de thru 0x022119 (0x773c bytes) 000501 1745620250.792313 read 0x00a14e thru 0x00dde0 (0x3c93 bytes) 000502 1745620250.792647 trunc from 0x02211a to 0x004c92 (0x1d487 bytes) 000503 1745620250.800638 mapwrite 0x038baa thru 0x03ffff (0x7456 bytes) 000504 1745620250.842699 write 0x008d06 thru 0x009e6a (0x1165 bytes) 000505 1745620250.843334 read 0x009cf1 thru 0x00afaf (0x12bf bytes) 000506 1745620250.843611 trunc from 0x040000 to 0x02aed9 (0x15126 bytes) 000509 1745620250.849473 read 0x01e484 thru 0x020359 (0x1ed6 bytes) 000510 1745620250.849924 mapread 0x006747 thru 0x00a187 (0x3a41 bytes) 000511 1745620250.850571 read 0x01b667 thru 0x024487 (0x8e21 bytes) 000513 1745620250.853920 MIRROR_EXTEND to 4 mirrors 000515 1745620251.039033 trunc from 0x02aeda to 0x000f09 (0x29fd0 bytes) 000516 1745620251.075406 mapread 0x000747 thru 0x000f09 (0x7c3 bytes) 000517 1745620251.077910 mapwrite 0x036990 thru 0x03f445 (0x8ab6 bytes) 000518 1745620251.124942 trunc from 0x03f446 to 0x00d12e (0x32317 bytes) 000519 1745620251.131987 write 0x021284 thru 0x02978b (0x8508 bytes) 000521 1745620251.133146 read 0x002bfd thru 0x006580 (0x3984 bytes) 000522 1745620251.136277 trunc from 0x02978c to 0x021ea8 (0x78e3 bytes) 000523 1745620251.142071 trunc from 0x021ea9 to 0x03ec91 (0x1cde9 bytes) 000524 1745620251.148396 trunc from 0x03ec92 to 0x006808 (0x38489 bytes) 000526 1745620251.156988 MIRROR_RESYNC 4 mirrors 000528 1745620251.407886 MIRROR_EXTEND to 5 mirrors 000531 1745620251.567497 write 0x000a4a thru 0x00a002 (0x95b9 bytes) 000532 1745620251.599708 mapread 0x00755d thru 0x009c01 (0x26a5 bytes) 000533 1745620251.603401 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000537 1745620251.691382 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000539 1745620251.768703 mapwrite 0x025bb4 thru 0x0316c9 (0xbb16 bytes) 000540 1745620251.814587 write 0x035761 thru 0x03ffff (0xa89f bytes) 000543 1745620251.816983 mapread 0x014a71 thru 0x01e0c9 (0x9659 bytes) 000544 1745620251.826387 mapwrite 0x03649c thru 0x03ffff (0x9b64 bytes) 000545 1745620251.856242 mapread 0x02f40b thru 0x03c092 (0xcc88 bytes) 000547 1745620251.860245 write 0x032055 thru 0x033f28 (0x1ed4 bytes) 000549 1745620251.861550 trunc from 0x040000 to 0x032042 (0xdfbd bytes) 000550 1745620251.869695 mapread 0x02ef48 thru 0x032042 (0x30fb bytes) 000553 1745620251.871414 mapread 0x02962f thru 0x032042 (0x8a14 bytes) 000555 1745620251.872636 mapwrite 0x01f494 thru 0x02b862 (0xc3cf bytes) 000556 1745620251.896400 write 0x00865e thru 0x017035 (0xe9d8 bytes) 000561 1745620251.903911 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000563 1745620251.985906 trunc from 0x032043 to 0x0064f8 (0x2bb4a bytes) 000564 1745620251.995168 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000565 1745620252.051600 write 0x0354b9 thru 0x038517 (0x305f bytes) 000566 1745620252.052404 write 0x011d30 thru 0x0136ca (0x199b bytes) 000567 1745620252.055499 write 0x015e56 thru 0x0237f5 (0xd9a0 bytes) 000568 1745620252.061366 mapread 0x0159ed thru 0x0217a2 (0xbdb6 bytes) 000569 1745620252.062025 read 0x00f3a6 thru 0x0158c6 (0x6521 bytes) 000570 1745620252.062418 read 0x02f4b7 thru 0x038517 (0x9061 bytes) 000572 1745620252.067649 read 0x026a9c thru 0x030da5 (0xa30a bytes) 000573 1745620252.075132 MIRROR_EXTEND to 2 mirrors 000577 1745620252.260389 trunc from 0x038518 to 0x022c1b (0x158fc bytes) 000580 1745620252.298360 mapread 0x020a18 thru 0x022c1b (0x2204 bytes) 000581 1745620252.304737 mapread 0x020971 thru 0x022c1b (0x22ab bytes) 000582 1745620252.307211 MIRROR_RESYNC 2 mirrors 000584 1745620252.457506 mapread 0x003088 thru 0x0103ed (0xd366 bytes) 000585 1745620252.468434 MIRROR_RESYNC 2 mirrors 000588 1745620252.521595 mapwrite 0x035f1e thru 0x036339 (0x41c bytes) 000589 1745620252.581371 write 0x036ac3 thru 0x03990b (0x2e49 bytes) 000592 1745620252.583551 mapwrite 0x0288c6 thru 0x02a84b (0x1f86 bytes) 000593 1745620252.615391 write 0x002109 thru 0x003c0d (0x1b05 bytes) 000594 1745620252.619126 mapread 0x0173ec thru 0x01daa5 (0x66ba bytes) 000597 1745620252.625853 mapread 0x02475a thru 0x02fc92 (0xb539 bytes) 000599 1745620252.626520 mapwrite 0x010c06 thru 0x01f1bc (0xe5b7 bytes) 000600 1745620252.657941 write 0x038426 thru 0x03e50c (0x60e7 bytes) 000602 1745620252.661077 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000604 1745620252.717042 trunc from 0x03e50d to 0x00283c (0x3bcd0 bytes) 000605 1745620252.726749 mapread 0x00028d thru 0x00283c (0x25b0 bytes) 000607 1745620252.735397 mapwrite 0x00451f thru 0x010910 (0xc3f2 bytes) 000608 1745620252.777389 write 0x0182be thru 0x01b0fd (0x2e40 bytes) 000610 1745620252.780054 MIRROR_EXTEND to 2 mirrors 000612 1745620252.945612 read 0x000a4c thru 0x00e4e1 (0xda96 bytes) 000613 1745620252.949319 mapread 0x010fea thru 0x019e0b (0x8e22 bytes) 000614 1745620252.956515 mapread 0x00726c thru 0x009896 (0x262b bytes) 000615 1745620252.956982 mapwrite 0x030188 thru 0x03b11b (0xaf94 bytes) 000616 1745620253.033638 mapread 0x019218 thru 0x028e65 (0xfc4e bytes) 000618 1745620253.042004 mapwrite 0x0317a4 thru 0x03ffff (0xe85c bytes) 000619 1745620253.094629 write 0x018393 thru 0x01857a (0x1e8 bytes) 000621 1745620253.102054 MIRROR_EXTEND to 3 mirrors 000623 1745620253.324267 write 0x01cbd5 thru 0x02335f (0x678b bytes) 000624 1745620253.354875 mapread 0x01e62e thru 0x0226cd (0x40a0 bytes) 000625 1745620253.355633 trunc from 0x040000 to 0x0249e0 (0x1b61f bytes) 000627 1745620253.373012 mapwrite 0x03fd68 thru 0x03ffff (0x298 bytes) 000628 1745620253.420360 MIRROR_RESYNC 3 mirrors 000629 1745620253.663474 mapwrite 0x02cbc4 thru 0x036b6e (0x9fab bytes) 000630 1745620253.746317 MIRROR_EXTEND to 4 mirrors 000631 1745620253.947841 mapwrite 0x00123c thru 0x00a769 (0x952e bytes) 000632 1745620254.051433 MIRROR_EXTEND to 5 mirrors 000634 1745620254.288252 trunc from 0x040000 to 0x005ef7 (0x3a108 bytes) 000635 1745620254.321673 read 0x000b65 thru 0x005ef7 (0x5393 bytes) 000636 1745620254.324559 mapread 0x00415c thru 0x005ef7 (0x1d9c bytes) 000638 1745620254.325477 read 0x001e6b thru 0x005ef7 (0x408d bytes) 000639 1745620254.325968 trunc from 0x005ef8 to 0x037e12 (0x31f1b bytes) 000641 1745620254.342035 MIRROR_EXTEND to 6 mirrors 000642 1745620254.538009 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000643 1745620254.653274 mapread 0x02f706 thru 0x036ea4 (0x779f bytes) 000645 1745620254.667748 mapwrite 0x01a5f2 thru 0x022d6b (0x877a bytes) 000646 1745620254.750435 read 0x00ab7c thru 0x0192eb (0xe770 bytes) 000647 1745620254.754585 mapread 0x02c348 thru 0x037e12 (0xbacb bytes) 000648 1745620254.766508 write 0x02a779 thru 0x02f654 (0x4edc bytes) 000649 1745620254.768498 write 0x026b20 thru 0x03099c (0x9e7d bytes) 000650 1745620254.777704 write 0x03b499 thru 0x03ffff (0x4b67 bytes) 000653 1745620254.784232 MIRROR_EXTEND to 6 mirrors 000654 1745620255.008712 mapread 0x03fba7 thru 0x03ffff (0x459 bytes) 000655 1745620255.011147 write 0x00935c thru 0x010a9c (0x7741 bytes) 000657 1745620255.039527 mapwrite 0x024cbd thru 0x03121a (0xc55e bytes) 000659 1745620255.081441 trunc from 0x040000 to 0x02594d (0x1a6b2 bytes) 000660 1745620255.097036 trunc from 0x02594e to 0x01f5b8 (0x6395 bytes) 000663 1745620255.107614 MIRROR_RESYNC 6 mirrors 000665 1745620255.482953 mapwrite 0x03feaa thru 0x03ffff (0x156 bytes) 000669 1745620255.548402 mapread 0x038f48 thru 0x03ffff (0x70b8 bytes) 000670 1745620255.555573 trunc from 0x040000 to 0x00f39d (0x30c62 bytes) 000671 1745620255.563666 write 0x037412 thru 0x038cee (0x18dd bytes) 000672 1745620255.564518 mapread 0x02da8b thru 0x02f1d8 (0x174e bytes) 000674 1745620255.568449 mapwrite 0x0278f5 thru 0x033eaf (0xc5bb bytes) 000677 1745620255.608579 MIRROR_EXTEND to 7 mirrors 000681 1745620255.778650 mapread 0x0052f6 thru 0x005a85 (0x790 bytes) 000682 1745620255.783304 mapread 0x0156cb thru 0x0250ec (0xfa22 bytes) 000683 1745620255.796374 trunc from 0x038cef to 0x02d879 (0xb475 bytes) 000684 1745620255.841861 write 0x0291aa thru 0x03693e (0xd795 bytes) 000685 1745620255.846126 trunc from 0x03693f to 0x0341f7 (0x2747 bytes) 000686 1745620255.854650 write 0x0361a0 thru 0x03c350 (0x61b1 bytes) 000688 1745620255.856134 read 0x00cb96 thru 0x00d689 (0xaf4 bytes) 000689 1745620255.858586 read 0x015b9c thru 0x0226f9 (0xcb5e bytes) 000691 1745620255.863679 read 0x002a4f thru 0x007d98 (0x534a bytes) 000692 1745620255.868096 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000693 1745620255.949380 read 0x027941 thru 0x027ddc (0x49c bytes) 000694 1745620255.952972 trunc from 0x03c351 to 0x032343 (0xa00d bytes) 000696 1745620255.959648 mapwrite 0x032db5 thru 0x03b288 (0x84d4 bytes) 000697 1745620256.005661 mapread 0x00c515 thru 0x010577 (0x4063 bytes) 000698 1745620256.009729 read 0x026e3d thru 0x02d82c (0x69f0 bytes) 000699 1745620256.016196 write 0x01fa4d thru 0x02cab6 (0xd06a bytes) 000703 1745620256.018881 write 0x0246ab thru 0x02ef6f (0xa8c5 bytes) 000704 1745620256.019710 mapwrite 0x03e954 thru 0x03ffff (0x16ac bytes) 000706 1745620256.060713 read 0x039596 thru 0x03ffff (0x6a6a bytes) 000707 1745620256.061189 read 0x02ace5 thru 0x037c97 (0xcfb3 bytes) 000709 1745620256.062131 trunc from 0x040000 to 0x001e8e (0x3e171 bytes) 000710 1745620256.069996 mapread 0x0000d7 thru 0x001e8e (0x1db8 bytes) 000711 1745620256.070578 write 0x035322 thru 0x03ffff (0xacde bytes) 000716 1745620256.072635 read 0x01cbb8 thru 0x020da2 (0x41eb bytes) 000717 1745620256.077740 mapwrite 0x025686 thru 0x034969 (0xf2e4 bytes) 000720 1745620256.116597 mapwrite 0x00f2e3 thru 0x00fdc6 (0xae4 bytes) 000722 1745620256.137460 read 0x0294b5 thru 0x02f40e (0x5f5a bytes) 000724 1745620256.138173 write 0x02a2bd thru 0x02ab83 (0x8c7 bytes) 000725 1745620256.138762 mapread 0x00ba08 thru 0x01971e (0xdd17 bytes) 000727 1745620256.139681 mapread 0x019d35 thru 0x01bfeb (0x22b7 bytes) 000728 1745620256.140238 read 0x03fb20 thru 0x03ffff (0x4e0 bytes) 000732 1745620256.141581 mapwrite 0x021b0d thru 0x02a378 (0x886c bytes) 000733 1745620256.171657 MIRROR_EXTEND to 7 mirrors 000736 1745620256.362292 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000737 1745620256.450505 trunc from 0x040000 to 0x00c8f8 (0x33707 bytes) 000738 1745620256.494957 trunc from 0x00c8f9 to 0x012e3d (0x6545 bytes) 000740 1745620256.505608 mapread 0x001eea thru 0x00a049 (0x8160 bytes) 000741 1745620256.520676 MIRROR_RESYNC 6 mirrors 000744 1745620256.857928 mapwrite 0x00b007 thru 0x011540 (0x653a bytes) 000746 1745620256.929472 write 0x01b48a thru 0x026406 (0xaf7d bytes) 000747 1745620256.930823 read 0x00f7a8 thru 0x01e895 (0xf0ee bytes) 000751 1745620256.935159 mapwrite 0x035dde thru 0x038cd3 (0x2ef6 bytes) 000752 1745620256.987190 write 0x0380cf thru 0x03fea6 (0x7dd8 bytes) 000753 1745620256.988374 mapwrite 0x00606c thru 0x00e81c (0x87b1 bytes) 000755 1745620257.023109 mapwrite 0x00b7df thru 0x0127b3 (0x6fd5 bytes) 000756 1745620257.052691 read 0x012972 thru 0x01f81f (0xceae bytes) 000757 1745620257.053130 trunc from 0x03fea7 to 0x018261 (0x27c45 bytes) 000758 1745620257.060071 mapread 0x00c62b thru 0x016e6d (0xa843 bytes) 000761 1745620257.061643 mapread 0x0027be thru 0x007162 (0x49a5 bytes) 000763 1745620257.068660 MIRROR_EXTEND to 7 mirrors 000764 1745620257.250413 MIRROR_EXTEND to 8 mirrors 000767 1745620257.469216 mapwrite 0x01c835 thru 0x01c9bd (0x189 bytes) 000768 1745620257.548414 mapread 0x00b4a7 thru 0x0198d5 (0xe42f bytes) 000769 1745620257.563219 write 0x01a1f0 thru 0x01a478 (0x289 bytes) 000770 1745620257.565746 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000771 1745620257.649274 trunc from 0x01c9be to 0x03a7d1 (0x1de14 bytes) 000773 1745620257.657683 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000777 1745620257.741840 mapwrite 0x015f0d thru 0x0160a8 (0x19c bytes) 000778 1745620257.772545 read 0x0257f5 thru 0x02a534 (0x4d40 bytes) 000780 1745620257.776170 trunc from 0x03a7d2 to 0x03e943 (0x4172 bytes) 000781 1745620257.782292 trunc from 0x03e944 to 0x0063f5 (0x3854e bytes) 000782 1745620257.788628 mapwrite 0x030cc5 thru 0x0388f7 (0x7c33 bytes) 000783 1745620257.835533 MIRROR_RESYNC 6 mirrors 000784 1745620258.188214 write 0x03ea1d thru 0x03ffff (0x15e3 bytes) 000793 1745620258.221767 mapread 0x01f387 thru 0x023215 (0x3e8f bytes) 000796 1745620258.231694 mapread 0x01673a thru 0x01ca78 (0x633f bytes) 000798 1745620258.241747 write 0x037948 thru 0x03ffff (0x86b8 bytes) 000800 1745620258.249121 trunc from 0x040000 to 0x000ec3 (0x3f13c bytes) 000801 1745620258.257334 read 0x0005bb thru 0x000ec3 (0x909 bytes) 000804 1745620258.260130 trunc from 0x000ec4 to 0x010cc5 (0xfe02 bytes) 000806 1745620258.268229 write 0x01ef58 thru 0x022691 (0x373a bytes) 000807 1745620258.269024 mapread 0x020d8e thru 0x02224f (0x14c2 bytes) 000809 1745620258.269883 mapread 0x003dd7 thru 0x00e3b3 (0xa5dd bytes) 000810 1745620258.276975 mapread 0x022547 thru 0x022691 (0x14b bytes) 000811 1745620258.277621 trunc from 0x022692 to 0x025786 (0x30f5 bytes) 000812 1745620258.284440 mapread 0x0106a3 thru 0x01d20c (0xcb6a bytes) 000813 1745620258.288659 trunc from 0x025787 to 0x009db1 (0x1b9d5 bytes) 000817 1745620258.296506 trunc from 0x009db2 to 0x032917 (0x28b66 bytes) 000818 1745620258.302433 write 0x00c501 thru 0x016d23 (0xa823 bytes) 000820 1745620258.310049 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000821 1745620258.395668 mapwrite 0x0114d6 thru 0x014eab (0x39d6 bytes) 000822 1745620258.436161 read 0x0241f3 thru 0x0326bd (0xe4cb bytes) 000824 1745620258.443093 mapwrite 0x00eaef thru 0x0175e8 (0x8afa bytes) 000825 1745620258.473999 mapwrite 0x034682 thru 0x035bea (0x1569 bytes) 000827 1745620258.521366 MIRROR_RESYNC 5 mirrors 000828 1745620258.825749 read 0x02373a thru 0x02b984 (0x824b bytes) 000829 1745620258.829733 read 0x00a38e thru 0x016629 (0xc29c bytes) 000831 1745620258.835392 mapwrite 0x032b32 thru 0x03ffff (0xd4ce bytes) 000833 1745620258.922957 trunc from 0x040000 to 0x028701 (0x178fe bytes) 000834 1745620258.932212 write 0x00a4f7 thru 0x010601 (0x610b bytes) 000835 1745620258.939332 mapread 0x00c49d thru 0x0158c1 (0x9425 bytes) 000837 1745620258.954401 MIRROR_EXTEND to 6 mirrors 000838 1745620259.148168 write 0x016c2b thru 0x019a49 (0x2e1f bytes) 000842 1745620259.187899 MIRROR_RESYNC 6 mirrors 000844 1745620259.518359 mapread 0x0077e7 thru 0x013ee8 (0xc702 bytes) 000845 1745620259.529116 write 0x00667a thru 0x013903 (0xd28a bytes) 000846 1745620259.570855 read 0x015abc thru 0x025572 (0xfab7 bytes) 000847 1745620259.576675 read 0x00bdbf thru 0x0131f2 (0x7434 bytes) 000848 1745620259.577346 write 0x018a9a thru 0x018cb1 (0x218 bytes) 000849 1745620259.578123 mapread 0x01bb3d thru 0x01d22a (0x16ee bytes) 000851 1745620259.579130 read 0x007d0c thru 0x00dcb8 (0x5fad bytes) 000852 1745620259.579559 mapread 0x019018 thru 0x024b45 (0xbb2e bytes) 000853 1745620259.580149 read 0x00af73 thru 0x00d319 (0x23a7 bytes) 000854 1745620259.580478 read 0x026b7c thru 0x027e4d (0x12d2 bytes) 000856 1745620259.581050 write 0x0180e1 thru 0x024087 (0xbfa7 bytes) 000858 1745620259.582086 read 0x014c36 thru 0x019001 (0x43cc bytes) 000859 1745620259.586298 read 0x01c351 thru 0x0242a3 (0x7f53 bytes) 000861 1745620259.588221 trunc from 0x028702 to 0x0306aa (0x7fa9 bytes) 000863 1745620259.595940 mapread 0x006b00 thru 0x00d5a9 (0x6aaa bytes) 000865 1745620259.596598 mapread 0x015f1f thru 0x01c184 (0x6266 bytes) 000866 1745620259.597097 read 0x022fd1 thru 0x0295c2 (0x65f2 bytes) 000867 1745620259.601512 trunc from 0x0306ab to 0x020d48 (0xf962 bytes) 000869 1745620259.609762 trunc from 0x020d49 to 0x011f7d (0xedcb bytes) 000870 1745620259.615094 write 0x01e662 thru 0x02afb0 (0xc94f bytes) 000871 1745620259.616709 trunc from 0x02afb1 to 0x020b30 (0xa480 bytes) 000873 1745620259.631562 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000874 1745620259.728127 mapwrite 0x00484c thru 0x009ddd (0x5592 bytes) 000876 1745620259.764469 write 0x01a4b3 thru 0x02987f (0xf3cd bytes) 000878 1745620259.770813 mapread 0x017c4b thru 0x025ec4 (0xe27a bytes) 000879 1745620259.771662 mapread 0x01a290 thru 0x020135 (0x5ea6 bytes) 000880 1745620259.774041 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000881 1745620259.864671 trunc from 0x029880 to 0x0047de (0x250a1 bytes) 000882 1745620259.873305 mapwrite 0x02d572 thru 0x037990 (0xa41f bytes) 000884 1745620259.928986 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000889 1745620260.010911 write 0x03620f thru 0x03ffff (0x9df1 bytes) 000891 1745620260.013832 read 0x02f7bb thru 0x03ae66 (0xb6ac bytes) 000894 1745620260.018392 trunc from 0x040000 to 0x01f1c0 (0x20e3f bytes) 000895 1745620260.024630 read 0x000a90 thru 0x001364 (0x8d5 bytes) 000896 1745620260.029728 mapread 0x0057a3 thru 0x01083a (0xb098 bytes) 000897 1745620260.031505 mapwrite 0x011d1d thru 0x013cf9 (0x1fdd bytes) 000898 1745620260.055196 read 0x01445a thru 0x016cfd (0x28a4 bytes) 000899 1745620260.055751 write 0x00f8d0 thru 0x01c738 (0xce69 bytes) 000901 1745620260.057060 mapwrite 0x00cf10 thru 0x019592 (0xc683 bytes) 000902 1745620260.086089 read 0x016efc thru 0x01bd0d (0x4e12 bytes) 000904 1745620260.086880 trunc from 0x01f1c1 to 0x02ca01 (0xd841 bytes) 000906 1745620260.092440 mapwrite 0x02380c thru 0x026012 (0x2807 bytes) 000909 1745620260.130910 trunc from 0x02ca02 to 0x03a955 (0xdf54 bytes) 000912 1745620260.138616 trunc from 0x03a956 to 0x01ad72 (0x1fbe3 bytes) 000915 1745620260.145580 mapread 0x0185d7 thru 0x01ad72 (0x279c bytes) 000916 1745620260.147203 read 0x00f47e thru 0x01ad72 (0xb8f5 bytes) 000917 1745620260.149151 mapread 0x0096a4 thru 0x011038 (0x7995 bytes) 000924 1745620260.151856 trunc from 0x01ad73 to 0x03247a (0x17708 bytes) 000926 1745620260.159755 mapwrite 0x022924 thru 0x02728c (0x4969 bytes) 000927 1745620260.200312 read 0x01f7ab thru 0x02186c (0x20c2 bytes) 000928 1745620260.201013 read 0x02f12e thru 0x03247a (0x334d bytes) 000930 1745620260.202110 read 0x02f88d thru 0x03247a (0x2bee bytes) 000933 1745620260.203138 mapwrite 0x0364b7 thru 0x03ffff (0x9b49 bytes) 000934 1745620260.247567 read 0x00d1b1 thru 0x01ce6d (0xfcbd bytes) 000935 1745620260.248084 trunc from 0x040000 to 0x03a678 (0x5987 bytes) 000936 1745620260.256452 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000937 1745620260.328180 read 0x0335e7 thru 0x03a678 (0x7092 bytes) 000939 1745620260.335518 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) 000941 1745620260.386500 write 0x005b4b thru 0x0104a8 (0xa95e bytes) 000944 1745620260.393160 trunc from 0x03a679 to 0x028100 (0x12578 bytes) 000945 1745620260.399554 write 0x009b3d thru 0x017ef2 (0xe3b6 bytes) 000949 1745620260.403352 write 0x037c45 thru 0x03ffff (0x83bb bytes) 000951 1745620260.404527 mapwrite 0x036d9d thru 0x03bd6b (0x4fcf bytes) 000953 1745620260.439737 read 0x01b3fd thru 0x0276a8 (0xc2ac bytes) 000954 1745620260.446221 mapwrite 0x0353ac thru 0x036c45 (0x189a bytes) 000955 1745620260.472185 read 0x0157b7 thru 0x01bab3 (0x62fd bytes) 000956 1745620260.475831 mapwrite 0x03a77d thru 0x03ffff (0x5883 bytes) 000957 1745620260.502529 write 0x023b3a thru 0x02c473 (0x893a bytes) 000958 1745620260.503377 mapread 0x00d033 thru 0x01b642 (0xe610 bytes) 000960 1745620260.504259 mapread 0x019430 thru 0x01a5ad (0x117e bytes) 000961 1745620260.504754 write 0x007c96 thru 0x00e431 (0x679c bytes) 000963 1745620260.506084 mapwrite 0x01eeee thru 0x02acc1 (0xbdd4 bytes) 000965 1745620260.535252 mapwrite 0x0267d1 thru 0x02dae2 (0x7312 bytes) 000968 1745620260.561362 mapread 0x01429f thru 0x01c1f4 (0x7f56 bytes) 000969 1745620260.561980 mapwrite 0x01dc6a thru 0x024ea8 (0x723f bytes) 000974 1745620260.590462 write 0x034f4e thru 0x039eef (0x4fa2 bytes) 000976 1745620260.591768 mapread 0x030cb9 thru 0x0352e6 (0x462e bytes) 000977 1745620260.592407 mapwrite 0x01e225 thru 0x0215de (0x33ba bytes) 000979 1745620260.614628 mapwrite 0x004ce8 thru 0x011622 (0xc93b bytes) 000980 1745620260.637940 trunc from 0x040000 to 0x001762 (0x3e89d bytes) 000985 1745620260.648057 MIRROR_EXTEND to 2 mirrors 000986 1745620260.794290 mapread 0x000d70 thru 0x001762 (0x9f3 bytes) 000990 1745620260.797641 mapwrite 0x030181 thru 0x03cb3d (0xc9bd bytes) 000991 1745620260.867608 read 0x01e452 thru 0x02211f (0x3cce bytes) 000993 1745620260.871083 write 0x034c9a thru 0x03e382 (0x96e9 bytes) 000995 1745620260.874242 MIRROR_RESYNC 2 mirrors 000996 1745620261.023127 mapread 0x024101 thru 0x028ff6 (0x4ef6 bytes) 000997 1745620261.031608 read 0x00c39b thru 0x0162c2 (0x9f28 bytes) 000998 1745620261.037016 MIRROR_EXTEND to 3 mirrors 000999 1745620261.228438 read 0x0000ce thru 0x00e72a (0xe65d bytes) 001000 1745620261.233924 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ed:0x0]: Invalid argument (22) All operations completed A-OK! PASS 70a (21s) == sanity-flr test 71: check mirror extend parent fid ==== 18:31:02 (1745620262) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 18:31:03 (1745620263) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7684192 .. failed Extending file size to 3529600 .. Extending file size to 6031456 .. Extending file size to 7434272 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9212832 .. done Extending file size to 8376288 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8498880 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8604672 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8762112 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8840352 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 26768 26770 26771 26772 26773 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) 192.168.204.130@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:32:08 (1745620328) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3272512 .. Extending file size to 7443776 .. done Extending file size to 3767968 .. Extending file size to 5444384 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7941984 .. failed Extending file size to 6390080 .. Extending file size to 6656640 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8490432 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8016416 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8777344 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9343648 .. Extending file size to 8619072 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8642752 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8753056 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9318688 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9328928 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 30066 30068 30069 30071 30073 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) 192.168.204.130@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:33:12 (1745620392) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4608 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4608 2203776 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0401157 s, 52.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.097946 s, 21.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:33:17 (1745620397) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0325205 s, 64.5 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 18:33:19 (1745620399) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x534:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x603:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x603:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x604:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x5fe:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:33:21 (1745620401) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0491769 s, 42.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x605:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x600:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:33:23 (1745620403) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0459617 s, 45.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x606:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x607:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x601:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x606:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x607:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x601:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x602:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:33:25 (1745620405) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.037202 s, 56.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x608:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x603:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x608:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x603:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x604:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:33:49 (1745620429) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg430-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023471 s, 44.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x609:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x606:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x605:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x609:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x606:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x605:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x60a:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:34:32 (1745620472) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg430-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x607:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x608:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0423231 s, 49.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x607:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x608:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x60b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x607:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x608:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x60b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:35:17 (1745620517) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg430-server mds-ost sync done. Creating new pool test_204f oleg430-server: Pool lustre.test_204f created Adding targets to pool oleg430-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg430-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.135439 s, 15.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x609:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x60c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x609:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x60c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg430-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg430-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg430-server: Pool lustre.test_204f destroyed PASS 204f (59s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:36:16 (1745620576) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:36:19 (1745620579) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:36:21 (1745620581) oleg430-server: Pool lustre.test_206 created Adding targets to pool oleg430-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg430-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg430-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg430-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg430-server: Pool lustre.test_206 destroyed PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:36:37 (1745620597) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0252195 s, 41.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0203661 s, 51.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x9ae61a9a Mirror 3: 0x9ae61a9a PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:36:40 (1745620600) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:36:41 (1745620601) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:36:43 (1745620603) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0722518 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1b50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x930000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x930000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x930000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xd60000 Read this (4 bytes): BBBB 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) PASS 209a (19s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:37:02 (1745620622) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00365342 s, 2.2 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:37:05 (1745620625) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0248243 s, 42.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x559 lmm_fid: [0x200000404:0x559:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x559 lmm_fid: [0x200000404:0x559:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x616:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835289 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-25 18:37:06.000000000 -0400 Modify: 2025-04-25 18:37:06.000000000 -0400 Change: 2025-04-25 18:37:07.000000000 -0400 Birth: - PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:37:10 (1745620630) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:37:11 (1745620631) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.41659 s, 98.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.055544 s, 73.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 25 18:37 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x55b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 25 18:37 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test complete, duration 850 sec ============ 18:37:16 (1745620636) === sanity-flr: start cleanup 18:37:16 (1745620636) === === sanity-flr: finish cleanup 18:37:19 (1745620639) ===