-----============= acceptance-small: sanity-flr ============----- Fri Dec 13 12:48:27 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg131-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 12:48:32 (1734112112) === oleg131-client.virtnet: executing check_config_client /mnt/lustre oleg131-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg131-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b58f5000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b58f5000.idle_timeout=debug disable quota as required oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 12:48:39 (1734112119) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f6968] == sanity-flr test 0a: lfs mirror create with -N option == 12:48:40 (1734112120) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:48:43 (1734112123) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:48:44 (1734112124) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:48:45 (1734112125) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:48:50 (1734112130) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:48:50 (1734112130) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 12:48:51 (1734112131) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b58f5000.stats=clear osc.lustre-OST0001-osc-ffff8800b58f5000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:48:53 (1734112133) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:48:55 (1734112135) PASS 0j (1s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:48:57 (1734112137) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:48:57 (1734112137) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 12:48:59 (1734112139) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:49:01 (1734112141) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (0s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:49:01 (1734112141) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:49:03 (1734112143) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:49:05 (1734112145) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:49:07 (1734112147) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.210848 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0348188 s, 30.1 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:49:09 (1734112149) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.0524934 s, 220 MB/s PASS 22 (1s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:49:10 (1734112150) Stopping /mnt/lustre-ost1 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58f5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58f5000.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg131-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b58f5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58f5000.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg131-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (19s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:49:29 (1734112169) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0387699 s, 54.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58f5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58f5000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg131-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:49:38 (1734112178) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58f5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58f5000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg131-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b58f5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58f5000.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg131-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (20s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:49:58 (1734112198) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.698672 s, 75.0 MB/s -rw-r--r-- 1 root root 52428800 Dec 13 12:49 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.179s user 0m0.001s sys 0m0.092s Stopping /mnt/lustre-ost1 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58f5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58f5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.173s user 0m0.001s sys 0m0.098s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg131-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b58f5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58f5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.180s user 0m0.003s sys 0m0.103s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg131-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg131-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b58f5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58f5000.ost_server_uuid in FULL state after 0 sec PASS 33b (17s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:50:15 (1734112215) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:50:16 (1734112216) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (0s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:50:17 (1734112217) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) == sanity-flr test 35: allow to write to mirrored files == 12:50:17 (1734112217) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 12:50:19 (1734112219) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.084065 s, 150 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:50:21 (1734112221) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00704475 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00640351 s, 164 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:50:23 (1734112223) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:50:25 (1734112225) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000788927 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000588366 s, 1.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000570405 s, 1.8 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000478675 s, 2.1 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 12:50:27 (1734112227) create test files with size 15760773 .. 1ae92d913148a83234c9b82a8c0fc516 - 2c105e67c011d2746629666511f0a0ca - 1b19565b46b2abec10e7c4469f24436b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 12:50:32 (1734112232) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:50:34 (1734112234) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:50:36 (1734112236) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0189611 s, 55.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00640544 s, 164 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (1s) == sanity-flr test 41: lfs mirror resync check =========== 12:50:37 (1734112237) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0260701 s, 80.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0682127 s, 61.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0332463 s, 63.1 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 12:50:40 (1734112240) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) == sanity-flr test 43a: mirror pick on write ============= 12:50:40 (1734112240) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:50:41 (1734112241) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00717604 s, 146 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 12:50:43 (1734112243) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (0s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:50:43 (1734112243) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x13:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:50:45 (1734112245) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0531385 s, 197 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (25s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:51:10 (1734112270) llite.lustre-ffff88012b284000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0644708 s, 163 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800b5839000.xattr_cache=1 PASS 44e (2s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:51:12 (1734112272) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (8s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:51:20 (1734112280) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:51:26 (1734112286) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) == sanity-flr test 48: Verify snapshot mirror ============ 12:51:26 (1734112286) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0475012 s, 66.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0363092 s, 86.6 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 92c5ebe903821e738c755ca6f4aa641a - mirror 1 checksum: 691a7dec67306e6a3f7d15e338a6fb26 - mirror 2 checksum: 92c5ebe903821e738c755ca6f4aa641a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Dec 13 12:51:27 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Dec 13 12:51:27 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 92c5ebe903821e738c755ca6f4aa641a - mirror 1 checksum: 691a7dec67306e6a3f7d15e338a6fb26 - mirror 2 checksum: 691a7dec67306e6a3f7d15e338a6fb26 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Dec 13 12:51:27 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Dec 13 12:51:27 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (2s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:51:29 (1734112289) Failing mds1 on oleg131-server Stopping /mnt/lustre-mds1 (opts:) on oleg131-server 12:51:30 (1734112290) shut down facet: mds1 facet_host: oleg131-server facet_failover_host: oleg131-server Failover mds1 to oleg131-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 12:51:43 (1734112303) targets are mounted 12:51:43 (1734112303) facet_failover done oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00711302 s, 147 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (19s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:51:48 (1734112308) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (0s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:51:48 (1734112308) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:51:49 (1734112309) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00185954 s, 8.8 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:51:51 (1734112311) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:51:52 (1734112312) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (0s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:51:52 (1734112312) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:51:56 (1734112316) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:52:14 (1734112334) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:52:31 (1734112351) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:52:48 (1734112368) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 12:52:49 (1734112369) mirror create pid 22396 mirror split pid 22397 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 22396 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 22397 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 12:53:51 (1734112431) Seed set to 2431 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1734112432.033944 mapwrite 0x02b973 thru 0x03a8f4 (0xef82 bytes) 000003 1734112432.097102 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x5df1 000004 1734112432.226263 trunc from 0x03a8f5 to 0x005df0 (0x34b04 bytes) 000006 1734112432.257968 mapwrite 0x003755 thru 0x003a83 (0x32f bytes) 000007 1734112432.284247 mapread 0x000213 thru 0x001098 (0xe86 bytes) 000008 1734112432.289585 mapread 0x00518d thru 0x005df0 (0xc64 bytes) 000009 1734112432.289969 mapread 0x0043b3 thru 0x005df0 (0x1a3e bytes) truncating to largest ever: 0x13510 000010 1734112432.290332 trunc from 0x005df1 to 0x01350f (0xd71f bytes) 000013 1734112432.297568 mapread 0x00d0c9 thru 0x0127ca (0x5702 bytes) 000015 1734112432.303105 mapread 0x002365 thru 0x00d0e4 (0xad80 bytes) 000018 1734112432.306701 mapread 0x00ba3b thru 0x01350f (0x7ad5 bytes) 000020 1734112432.307294 write 0x002629 thru 0x00a8d0 (0x82a8 bytes) 000021 1734112432.307824 mapread 0x009181 thru 0x01350f (0xa38f bytes) 000023 1734112432.308412 mapwrite 0x01dba7 thru 0x02d054 (0xf4ae bytes) 000024 1734112432.348027 mapwrite 0x02d11b thru 0x03ba2a (0xe910 bytes) 000025 1734112432.384169 mapwrite 0x033388 thru 0x03e382 (0xaffb bytes) truncating to largest ever: 0x2ee1d 000026 1734112432.419692 trunc from 0x03e383 to 0x02ee1c (0xf566 bytes) 000027 1734112432.425030 write 0x01038d thru 0x01cbca (0xc83e bytes) 000034 1734112432.427532 write 0x008bbc thru 0x014ddf (0xc224 bytes) 000036 1734112432.430267 MIRROR_RESYNC 2 mirrors 000038 1734112432.554581 write 0x021fb7 thru 0x02ada2 (0x8dec bytes) 000039 1734112432.577570 read 0x015f6b thru 0x025d8c (0xfe22 bytes) 000040 1734112432.583107 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000042 1734112432.627863 write 0x00b7bf thru 0x00b8b2 (0xf4 bytes) 000044 1734112432.630201 read 0x01162d thru 0x0160fe (0x4ad2 bytes) 000045 1734112432.633597 mapwrite 0x027453 thru 0x032fc0 (0xbb6e bytes) 000047 1734112432.678219 mapread 0x02d77f thru 0x032fc0 (0x5842 bytes) 000048 1734112432.678659 read 0x02b575 thru 0x03142b (0x5eb7 bytes) truncating to largest ever: 0x33239 000051 1734112432.679328 trunc from 0x032fc1 to 0x033238 (0x278 bytes) 000054 1734112432.684775 write 0x02b9b8 thru 0x03230c (0x6955 bytes) 000055 1734112432.685286 mapread 0x006109 thru 0x00e873 (0x876b bytes) 000056 1734112432.690463 mapwrite 0x025219 thru 0x028d7c (0x3b64 bytes) 000057 1734112432.712793 read 0x005ce5 thru 0x014f4f (0xf26b bytes) 000060 1734112432.713763 trunc from 0x033239 to 0x01a38d (0x18eab bytes) 000062 1734112432.719000 read 0x002852 thru 0x0057e4 (0x2f93 bytes) 000063 1734112432.719277 write 0x0234de thru 0x029630 (0x6153 bytes) 000066 1734112432.720239 write 0x000972 thru 0x008df2 (0x8481 bytes) 000068 1734112432.720942 trunc from 0x029631 to 0x01d0de (0xc552 bytes) 000069 1734112432.725986 read 0x0015c6 thru 0x009f04 (0x893f bytes) 000073 1734112432.727770 mapread 0x011465 thru 0x01d0de (0xbc7a bytes) 000077 1734112432.731145 read 0x00438a thru 0x006538 (0x21af bytes) 000079 1734112432.731748 mapread 0x004b7c thru 0x00e154 (0x95d9 bytes) 000086 1734112432.733678 mapwrite 0x004174 thru 0x00f379 (0xb206 bytes) 000087 1734112432.758567 read 0x0153bc thru 0x017280 (0x1ec5 bytes) 000089 1734112432.759059 write 0x015db9 thru 0x02577b (0xf9c3 bytes) 000091 1734112432.759975 mapread 0x00f81c thru 0x01c80c (0xcff1 bytes) 000094 1734112432.760955 read 0x01aba7 thru 0x021c84 (0x70de bytes) 000095 1734112432.761203 mapwrite 0x00d8be thru 0x0105f3 (0x2d36 bytes) 000096 1734112432.785218 mapread 0x0075c6 thru 0x0155a3 (0xdfde bytes) 000098 1734112432.786050 mapread 0x00b79c thru 0x014273 (0x8ad8 bytes) 000099 1734112432.786429 read 0x022041 thru 0x02577b (0x373b bytes) 000100 1734112432.786679 mapwrite 0x03faf0 thru 0x03ffff (0x510 bytes) 000102 1734112432.813195 read 0x019ae8 thru 0x0232f5 (0x980e bytes) 000103 1734112432.813518 read 0x00e82f thru 0x01d282 (0xea54 bytes) 000104 1734112432.813786 mapread 0x00bda4 thru 0x01285a (0x6ab7 bytes) 000105 1734112432.814201 read 0x00dcfb thru 0x0196fc (0xba02 bytes) 000106 1734112432.814527 trunc from 0x040000 to 0x0147e9 (0x2b816 bytes) 000107 1734112432.819795 mapwrite 0x02c356 thru 0x038f46 (0xcbf1 bytes) 000108 1734112432.854980 read 0x005108 thru 0x01018c (0xb085 bytes) 000112 1734112432.855894 read 0x0388a0 thru 0x038f46 (0x6a7 bytes) 000113 1734112432.856131 mapread 0x02079f thru 0x02d84c (0xd0ae bytes) 000115 1734112432.856758 mapwrite 0x00874c thru 0x00d070 (0x4925 bytes) 000120 1734112432.879149 read 0x029ef9 thru 0x0370b5 (0xd1bd bytes) 000121 1734112432.879465 mapwrite 0x036cf2 thru 0x03fea8 (0x91b7 bytes) 000122 1734112432.908795 read 0x007a42 thru 0x014ceb (0xd2aa bytes) 000123 1734112432.909125 mapread 0x0322e4 thru 0x0385d5 (0x62f2 bytes) 000124 1734112432.909516 mapwrite 0x0293ad thru 0x0363fd (0xd051 bytes) 000125 1734112432.936628 mapread 0x026c5b thru 0x02b63e (0x49e4 bytes) 000126 1734112432.937056 trunc from 0x03fea9 to 0x002e02 (0x3d0a6 bytes) 000127 1734112432.942965 read 0x001573 thru 0x002e02 (0x1890 bytes) 000131 1734112432.944007 read 0x001ef5 thru 0x002e02 (0xf0e bytes) 000133 1734112432.946149 MIRROR_EXTEND to 2 mirrors 000134 1734112433.078334 read 0x0010ae thru 0x002e02 (0x1d55 bytes) 000137 1734112433.088211 write 0x01e674 thru 0x0266a8 (0x8035 bytes) 000138 1734112433.112698 MIRROR_EXTEND to 3 mirrors truncating to largest ever: 0x3a588 000140 1734112433.285232 trunc from 0x0266a9 to 0x03a587 (0x13edf bytes) 000141 1734112433.309369 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000142 1734112433.362810 trunc from 0x03a588 to 0x0110e3 (0x294a4 bytes) 000146 1734112433.368588 trunc from 0x0110e4 to 0x01155d (0x47a bytes) 000147 1734112433.373218 mapwrite 0x010496 thru 0x0129ee (0x2559 bytes) 000148 1734112433.405539 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000149 1734112433.443125 mapread 0x005eaf thru 0x011a06 (0xbb58 bytes) 000151 1734112433.452053 read 0x006b7e thru 0x0129ee (0xbe71 bytes) 000153 1734112433.454077 MIRROR_EXTEND to 2 mirrors 000154 1734112433.587916 read 0x009343 thru 0x00d6f3 (0x43b1 bytes) 000155 1734112433.592712 trunc from 0x0129ef to 0x002277 (0x10777 bytes) 000158 1734112433.614839 read 0x001693 thru 0x002277 (0xbe5 bytes) truncating to largest ever: 0x3f5fc 000159 1734112433.617006 trunc from 0x002278 to 0x03f5fb (0x3d384 bytes) 000160 1734112433.623453 trunc from 0x03f5fc to 0x018259 (0x273a2 bytes) 000162 1734112433.628367 read 0x009e2b thru 0x00e9e5 (0x4bbb bytes) 000163 1734112433.631907 mapread 0x00e48e thru 0x018259 (0x9dcc bytes) 000165 1734112433.632580 write 0x022dc1 thru 0x026151 (0x3391 bytes) 000167 1734112433.633270 mapread 0x00020f thru 0x00818f (0x7f81 bytes) 000168 1734112433.635329 MIRROR_EXTEND to 3 mirrors 000169 1734112433.774457 MIRROR_EXTEND to 4 mirrors 000170 1734112433.904031 trunc from 0x026152 to 0x02410b (0x2046 bytes) 000171 1734112433.927739 MIRROR_RESYNC 4 mirrors 000173 1734112434.080618 mapwrite 0x00b57a thru 0x00eb71 (0x35f8 bytes) 000174 1734112434.135581 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000175 1734112434.201290 MIRROR_EXTEND to 4 mirrors 000176 1734112434.344074 write 0x020668 thru 0x02b662 (0xaffb bytes) 000177 1734112434.366129 write 0x006fc0 thru 0x00719e (0x1df bytes) 000178 1734112434.372405 MIRROR_EXTEND to 5 mirrors 000179 1734112434.516991 mapread 0x0232f4 thru 0x02b662 (0x836f bytes) 000181 1734112434.528587 mapread 0x009f49 thru 0x013567 (0x961f bytes) 000182 1734112434.537096 mapwrite 0x037f08 thru 0x03ffff (0x80f8 bytes) 000183 1734112434.584917 MIRROR_RESYNC 5 mirrors 000184 1734112434.793312 mapwrite 0x001879 thru 0x01031a (0xeaa2 bytes) 000185 1734112434.861926 trunc from 0x040000 to 0x0034af (0x3cb50 bytes) 000186 1734112434.874627 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000189 1734112434.931754 write 0x02902f thru 0x02ff77 (0x6f49 bytes) 000190 1734112434.932484 mapwrite 0x02d292 thru 0x035926 (0x8695 bytes) 000191 1734112434.966519 trunc from 0x035927 to 0x00a682 (0x2b2a4 bytes) 000194 1734112434.975127 write 0x0043b7 thru 0x0113d7 (0xd021 bytes) 000197 1734112434.980988 MIRROR_EXTEND to 5 mirrors 000200 1734112435.129817 mapread 0x009823 thru 0x00c175 (0x2953 bytes) 000201 1734112435.141068 read 0x00e0ba thru 0x0113d7 (0x331e bytes) 000202 1734112435.143836 read 0x00807b thru 0x0113d7 (0x935d bytes) 000205 1734112435.151424 write 0x008226 thru 0x013511 (0xb2ec bytes) 000206 1734112435.174569 trunc from 0x013512 to 0x01b67f (0x816e bytes) 000207 1734112435.182917 MIRROR_EXTEND to 6 mirrors 000208 1734112435.340998 read 0x0180b8 thru 0x01b67f (0x35c8 bytes) 000209 1734112435.345505 read 0x006118 thru 0x00c4a1 (0x638a bytes) 000211 1734112435.353596 trunc from 0x01b680 to 0x019ef2 (0x178d bytes) 000212 1734112435.376372 mapwrite 0x013f34 thru 0x01e8b4 (0xa981 bytes) 000213 1734112435.418299 read 0x00b594 thru 0x00ee69 (0x38d6 bytes) 000215 1734112435.423411 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000217 1734112435.472668 read 0x0066d5 thru 0x006c95 (0x5c1 bytes) 000218 1734112435.474335 mapread 0x018104 thru 0x01ac9d (0x2b9a bytes) 000220 1734112435.479323 read 0x0056a2 thru 0x009224 (0x3b83 bytes) 000223 1734112435.482175 write 0x00252c thru 0x00bc97 (0x976c bytes) 000224 1734112435.482729 mapread 0x01783a thru 0x01dc1e (0x63e5 bytes) 000227 1734112435.483559 read 0x00920d thru 0x016445 (0xd239 bytes) 000228 1734112435.485257 MIRROR_RESYNC 5 mirrors 000229 1734112435.683779 write 0x011421 thru 0x0152d2 (0x3eb2 bytes) 000230 1734112435.704941 trunc from 0x01e8b5 to 0x00f3ec (0xf4c8 bytes) 000231 1734112435.717878 mapread 0x0079dc thru 0x00f3ec (0x7a11 bytes) 000232 1734112435.726584 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000234 1734112435.790028 MIRROR_RESYNC 4 mirrors 000235 1734112435.957315 trunc from 0x00f3ed to 0x00af64 (0x4488 bytes) 000236 1734112435.983951 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000237 1734112436.039151 mapwrite 0x0283b0 thru 0x036e2f (0xea80 bytes) 000238 1734112436.076097 read 0x009358 thru 0x0103f1 (0x709a bytes) 000240 1734112436.079560 read 0x02bc82 thru 0x02dbc6 (0x1f45 bytes) 000241 1734112436.079843 write 0x00a57c thru 0x01371b (0x91a0 bytes) 000242 1734112436.082866 write 0x00d119 thru 0x01c050 (0xef38 bytes) 000246 1734112436.087476 MIRROR_RESYNC 3 mirrors 000247 1734112436.237751 mapread 0x017526 thru 0x01fc9a (0x8775 bytes) 000249 1734112436.244888 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000251 1734112436.301362 write 0x0121a5 thru 0x02041c (0xe278 bytes) 000254 1734112436.322371 trunc from 0x036e30 to 0x00cad6 (0x2a359 bytes) 000255 1734112436.334555 MIRROR_RESYNC 2 mirrors 000256 1734112436.443470 mapread 0x004923 thru 0x00cad6 (0x81b4 bytes) 000258 1734112436.449028 mapread 0x00ae1d thru 0x00cad6 (0x1cba bytes) 000259 1734112436.449276 write 0x02f159 thru 0x03894d (0x97f5 bytes) 000262 1734112436.466335 write 0x009c66 thru 0x00a9e6 (0xd81 bytes) 000263 1734112436.469608 trunc from 0x03894e to 0x000072 (0x388db bytes) 000266 1734112436.481265 read 0x000052 thru 0x000072 (0x21 bytes) 000267 1734112436.483538 write 0x01ed53 thru 0x02e533 (0xf7e1 bytes) 000268 1734112436.486292 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000272 1734112436.530323 trunc from 0x02e534 to 0x01d954 (0x10bdf bytes) 000273 1734112436.535919 mapwrite 0x019637 thru 0x01ed7b (0x5745 bytes) 000274 1734112436.578953 mapread 0x0191a7 thru 0x01a529 (0x1383 bytes) 000276 1734112436.579646 write 0x017a17 thru 0x01e734 (0x6d1e bytes) 000277 1734112436.580164 write 0x01f4e7 thru 0x027068 (0x7b82 bytes) 000278 1734112436.580713 write 0x038fee thru 0x0391b2 (0x1c5 bytes) 000279 1734112436.581125 write 0x027761 thru 0x02d20b (0x5aab bytes) 000280 1734112436.583432 mapread 0x028aa7 thru 0x034dfb (0xc355 bytes) 000281 1734112436.586893 trunc from 0x0391b3 to 0x034122 (0x5090 bytes) 000282 1734112436.592500 write 0x02ef74 thru 0x0365b8 (0x7645 bytes) 000287 1734112436.593983 mapread 0x00e56c thru 0x011615 (0x30aa bytes) 000289 1734112436.597997 read 0x005081 thru 0x00fb0a (0xaa8a bytes) 000290 1734112436.601272 mapwrite 0x0167cb thru 0x01e3b5 (0x7beb bytes) 000292 1734112436.628493 mapwrite 0x02b7da thru 0x033df4 (0x861b bytes) 000297 1734112436.649837 mapread 0x0346e1 thru 0x0348b1 (0x1d1 bytes) 000298 1734112436.650365 trunc from 0x0365b9 to 0x000eba (0x356fe bytes) 000299 1734112436.657337 read 0x00053f thru 0x000eba (0x97c bytes) 000300 1734112436.659675 write 0x001bf8 thru 0x0065de (0x49e7 bytes) 000301 1734112436.660204 mapread 0x001257 thru 0x0065de (0x5388 bytes) 000303 1734112436.661078 write 0x038ae0 thru 0x03ffff (0x7520 bytes) 000304 1734112436.661652 trunc from 0x040000 to 0x0173d7 (0x28c28 bytes) 000305 1734112436.667182 mapread 0x0129c6 thru 0x0173d7 (0x4a12 bytes) 000306 1734112436.671290 mapread 0x00e727 thru 0x0173d7 (0x8cb1 bytes) 000307 1734112436.671907 write 0x0215d1 thru 0x02ac8a (0x96ba bytes) 000309 1734112436.672722 read 0x0150d2 thru 0x0164c0 (0x13ef bytes) 000313 1734112436.673683 mapread 0x017757 thru 0x024ced (0xd597 bytes) 000315 1734112436.677631 read 0x00c0d9 thru 0x01788b (0xb7b3 bytes) 000316 1734112436.677935 mapwrite 0x00684f thru 0x00ef76 (0x8728 bytes) 000318 1734112436.703039 trunc from 0x02ac8b to 0x019573 (0x11717 bytes) 000319 1734112436.711158 MIRROR_EXTEND to 2 mirrors 000321 1734112436.849523 mapread 0x0071ca thru 0x00a686 (0x34bd bytes) 000324 1734112436.862414 trunc from 0x019574 to 0x00447b (0x150f8 bytes) 000326 1734112436.892269 write 0x02442e thru 0x0331b4 (0xed87 bytes) 000327 1734112436.895068 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000332 1734112436.939776 mapwrite 0x00d19a thru 0x013251 (0x60b8 bytes) 000333 1734112436.974294 trunc from 0x0331b5 to 0x01b8f3 (0x178c1 bytes) 000335 1734112436.980453 mapwrite 0x011a4d thru 0x0159f1 (0x3fa5 bytes) 000338 1734112437.005431 mapread 0x00989d thru 0x01106d (0x77d1 bytes) 000339 1734112437.005929 read 0x015b38 thru 0x0163a3 (0x86c bytes) 000340 1734112437.006181 mapwrite 0x03bde9 thru 0x03f9fc (0x3c14 bytes) 000342 1734112437.038870 mapread 0x032d15 thru 0x033941 (0xc2d bytes) 000343 1734112437.042433 mapread 0x011de9 thru 0x018425 (0x663d bytes) 000345 1734112437.046210 trunc from 0x03f9fd to 0x008764 (0x37298 bytes) 000346 1734112437.053630 MIRROR_EXTEND to 2 mirrors 000347 1734112437.201829 mapread 0x003db2 thru 0x008764 (0x49b3 bytes) 000348 1734112437.214326 trunc from 0x008765 to 0x0344df (0x2bd7b bytes) 000351 1734112437.241934 trunc from 0x0344e0 to 0x014be6 (0x1f8f9 bytes) 000354 1734112437.249558 read 0x01045f thru 0x014be6 (0x4788 bytes) 000355 1734112437.252971 read 0x001037 thru 0x002f27 (0x1ef1 bytes) 000356 1734112437.256679 mapread 0x00bbb6 thru 0x0110a9 (0x54f4 bytes) 000360 1734112437.257749 write 0x03190f thru 0x03a167 (0x8859 bytes) 000362 1734112437.260403 MIRROR_RESYNC 2 mirrors 000364 1734112437.406125 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000367 1734112437.447602 mapread 0x02cb6a thru 0x0310b7 (0x454e bytes) 000368 1734112437.455506 mapread 0x027bdc thru 0x02bd57 (0x417c bytes) 000369 1734112437.458871 mapwrite 0x02eb68 thru 0x032053 (0x34ec bytes) 000376 1734112437.504207 write 0x03eced thru 0x03ffff (0x1313 bytes) 000377 1734112437.504663 mapread 0x035ba7 thru 0x03ab2b (0x4f85 bytes) 000380 1734112437.508447 trunc from 0x040000 to 0x014da2 (0x2b25d bytes) 000381 1734112437.519974 read 0x0043a9 thru 0x007900 (0x3558 bytes) 000384 1734112437.522830 read 0x004fd7 thru 0x00e2d1 (0x92fb bytes) 000385 1734112437.526700 write 0x029e78 thru 0x02cbdf (0x2d68 bytes) 000390 1734112437.530286 MIRROR_EXTEND to 2 mirrors 000392 1734112437.676781 MIRROR_EXTEND to 3 mirrors 000394 1734112437.825398 trunc from 0x02cbe0 to 0x03adbe (0xe1df bytes) 000396 1734112437.857961 mapread 0x018927 thru 0x019cd2 (0x13ac bytes) 000398 1734112437.862402 write 0x02485a thru 0x02c679 (0x7e20 bytes) 000399 1734112437.868234 MIRROR_EXTEND to 4 mirrors 000400 1734112438.028422 write 0x019a94 thru 0x023f4d (0xa4ba bytes) 000403 1734112438.057049 MIRROR_RESYNC 4 mirrors 000404 1734112438.277772 trunc from 0x03adbf to 0x02701b (0x13da3 bytes) 000405 1734112438.297832 mapwrite 0x000d3d thru 0x002a15 (0x1cd9 bytes) 000407 1734112438.325499 mapread 0x02239f thru 0x02701b (0x4c7d bytes) 000408 1734112438.328822 mapwrite 0x00779b thru 0x012fbe (0xb824 bytes) 000410 1734112438.354663 read 0x01e34c thru 0x02701b (0x8cd0 bytes) 000411 1734112438.356699 mapread 0x023594 thru 0x02701b (0x3a88 bytes) 000413 1734112438.357284 write 0x0160d6 thru 0x021e6b (0xbd96 bytes) 000415 1734112438.358106 trunc from 0x02701c to 0x034bf9 (0xdbde bytes) 000417 1734112438.364572 read 0x006ff6 thru 0x00cf28 (0x5f33 bytes) 000418 1734112438.366502 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000419 1734112438.427158 mapwrite 0x02752c thru 0x0365e5 (0xf0ba bytes) 000420 1734112438.463424 read 0x022d5e thru 0x027f50 (0x51f3 bytes) 000422 1734112438.464012 read 0x033305 thru 0x0365e5 (0x32e1 bytes) 000423 1734112438.464298 trunc from 0x0365e6 to 0x0245e8 (0x11ffd bytes) 000427 1734112438.469840 mapread 0x00da17 thru 0x01157f (0x3b69 bytes) 000428 1734112438.473700 read 0x0244fc thru 0x0245e8 (0xed bytes) 000434 1734112438.475020 write 0x006f13 thru 0x0092a4 (0x2392 bytes) 000435 1734112438.475493 read 0x022205 thru 0x0245e8 (0x23e4 bytes) 000437 1734112438.476000 write 0x0180dd thru 0x027073 (0xef97 bytes) 000440 1734112438.478591 MIRROR_RESYNC 3 mirrors 000441 1734112438.624931 mapread 0x00a8a0 thru 0x00a8cc (0x2d bytes) 000442 1734112438.626757 trunc from 0x027074 to 0x019992 (0xd6e1 bytes) 000445 1734112438.654010 write 0x025fde thru 0x0295ba (0x35dd bytes) 000446 1734112438.654579 write 0x024077 thru 0x02e770 (0xa6fa bytes) 000449 1734112438.657190 mapwrite 0x012220 thru 0x0166d0 (0x44b1 bytes) 000450 1734112438.682839 mapwrite 0x019367 thru 0x01f2e8 (0x5f82 bytes) 000453 1734112438.705882 mapread 0x025f4d thru 0x027ec3 (0x1f77 bytes) 000455 1734112438.706479 mapread 0x027815 thru 0x02e770 (0x6f5c bytes) 000456 1734112438.706880 write 0x0121df thru 0x017d8d (0x5baf bytes) 000458 1734112438.707579 write 0x00f346 thru 0x01896b (0x9626 bytes) 000461 1734112438.708519 mapwrite 0x01e4d7 thru 0x027ddc (0x9906 bytes) 000462 1734112438.729974 mapread 0x025a2d thru 0x02e770 (0x8d44 bytes) 000463 1734112438.732187 MIRROR_EXTEND to 4 mirrors 000467 1734112438.847426 mapread 0x01d548 thru 0x01f067 (0x1b20 bytes) 000470 1734112438.858398 read 0x01fef0 thru 0x02002a (0x13b bytes) 000475 1734112438.865251 write 0x006b4a thru 0x006c9f (0x156 bytes) 000476 1734112438.887533 read 0x01e0fa thru 0x0238fc (0x5803 bytes) 000477 1734112438.890944 mapread 0x01c6a0 thru 0x023981 (0x72e2 bytes) 000478 1734112438.891568 read 0x00fc94 thru 0x017b94 (0x7f01 bytes) 000480 1734112438.892304 mapread 0x01cc71 thru 0x024af2 (0x7e82 bytes) 000485 1734112438.895867 MIRROR_RESYNC 4 mirrors 000486 1734112439.086146 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000488 1734112439.159065 mapread 0x002cc8 thru 0x0079b7 (0x4cf0 bytes) 000492 1734112439.167263 MIRROR_RESYNC 3 mirrors 000494 1734112439.209739 read 0x00e140 thru 0x017182 (0x9043 bytes) 000495 1734112439.212713 trunc from 0x02e771 to 0x025a8a (0x8ce6 bytes) 000496 1734112439.237560 read 0x01d380 thru 0x0203ba (0x303b bytes) 000497 1734112439.241409 mapwrite 0x003f15 thru 0x00d419 (0x9505 bytes) 000498 1734112439.260516 read 0x001ff3 thru 0x011260 (0xf26e bytes) 000499 1734112439.260904 mapread 0x005a6c thru 0x00e3d5 (0x896a bytes) 000500 1734112439.261293 mapwrite 0x023c11 thru 0x02713e (0x352e bytes) 000501 1734112439.287016 write 0x0241c7 thru 0x034156 (0xff90 bytes) 000502 1734112439.288056 mapread 0x01e25a thru 0x021828 (0x35cf bytes) 000504 1734112439.289071 write 0x01b190 thru 0x020628 (0x5499 bytes) 000505 1734112439.289737 mapwrite 0x03febd thru 0x03ffff (0x143 bytes) 000508 1734112439.317471 mapread 0x01b20b thru 0x029d18 (0xeb0e bytes) 000509 1734112439.317950 write 0x02cfb8 thru 0x036dd9 (0x9e22 bytes) 000511 1734112439.318727 write 0x03ecbb thru 0x03ffff (0x1345 bytes) 000513 1734112439.321117 MIRROR_EXTEND to 4 mirrors 000514 1734112439.454403 mapread 0x0189aa thru 0x01908a (0x6e1 bytes) 000515 1734112439.460143 mapread 0x013cb1 thru 0x01e7ea (0xab3a bytes) 000516 1734112439.467424 mapread 0x0091eb thru 0x00f0dc (0x5ef2 bytes) 000517 1734112439.472785 read 0x010771 thru 0x01b792 (0xb022 bytes) 000518 1734112439.474349 mapread 0x00056f thru 0x004882 (0x4314 bytes) 000519 1734112439.480864 mapread 0x013031 thru 0x0202d0 (0xd2a0 bytes) 000520 1734112439.483727 write 0x03252e thru 0x03d8fc (0xb3cf bytes) 000521 1734112439.509690 write 0x008074 thru 0x00a423 (0x23b0 bytes) 000522 1734112439.512804 write 0x01f91a thru 0x027eba (0x85a1 bytes) 000524 1734112439.515913 mapwrite 0x02d054 thru 0x03691b (0x98c8 bytes) 000526 1734112439.539111 mapwrite 0x006787 thru 0x011203 (0xaa7d bytes) 000527 1734112439.563054 mapread 0x029969 thru 0x031d67 (0x83ff bytes) 000528 1734112439.565518 mapwrite 0x0096c7 thru 0x00b3e9 (0x1d23 bytes) 000529 1734112439.580969 mapread 0x032069 thru 0x0371b0 (0x5148 bytes) 000532 1734112439.581833 write 0x0003a6 thru 0x010293 (0xfeee bytes) 000534 1734112439.584235 mapread 0x00efcd thru 0x01dd5e (0xed92 bytes) 000535 1734112439.584693 write 0x024018 thru 0x0298ae (0x5897 bytes) 000539 1734112439.585800 write 0x03a946 thru 0x03ca48 (0x2103 bytes) 000541 1734112439.586490 write 0x031ed8 thru 0x03fd87 (0xdeb0 bytes) 000542 1734112439.588529 read 0x03b76c thru 0x03ffff (0x4894 bytes) 000543 1734112439.588824 write 0x02a681 thru 0x02db04 (0x3484 bytes) 000544 1734112439.589246 trunc from 0x040000 to 0x019389 (0x26c76 bytes) 000545 1734112439.599349 mapread 0x008f13 thru 0x00a93a (0x1a28 bytes) 000546 1734112439.599820 write 0x039668 thru 0x03ffff (0x6998 bytes) 000547 1734112439.600371 mapwrite 0x01ec3a thru 0x021937 (0x2cfe bytes) 000549 1734112439.623182 write 0x01ff6d thru 0x029a28 (0x9abc bytes) 000552 1734112439.624177 read 0x00b3d2 thru 0x01215c (0x6d8b bytes) 000553 1734112439.624466 write 0x01e6ec thru 0x02ce17 (0xe72c bytes) 000554 1734112439.625100 trunc from 0x040000 to 0x01231d (0x2dce2 bytes) 000555 1734112439.632030 MIRROR_RESYNC 4 mirrors 000559 1734112439.799835 read 0x011298 thru 0x01231d (0x1086 bytes) 000561 1734112439.801792 read 0x0112a2 thru 0x01231d (0x107c bytes) 000562 1734112439.801949 trunc from 0x01231e to 0x032a8f (0x20772 bytes) 000564 1734112439.828545 write 0x0366cd thru 0x03e79b (0x80cf bytes) 000565 1734112439.829621 trunc from 0x03e79c to 0x013020 (0x2b77b bytes) 000572 1734112439.837159 mapread 0x0050ee thru 0x013020 (0xdf33 bytes) 000573 1734112439.842128 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000576 1734112439.897612 mapwrite 0x0307bb thru 0x03381f (0x3065 bytes) 000577 1734112439.929362 mapwrite 0x00f628 thru 0x01e4a7 (0xee80 bytes) 000578 1734112439.958036 mapwrite 0x006d9f thru 0x014b28 (0xdd8a bytes) 000579 1734112439.983130 mapread 0x0034a2 thru 0x00abd9 (0x7738 bytes) 000580 1734112439.983652 read 0x0276e5 thru 0x0336fb (0xc017 bytes) 000581 1734112439.986078 trunc from 0x033820 to 0x010e80 (0x2299f bytes) 000582 1734112439.991975 read 0x006c63 thru 0x009c42 (0x2fe0 bytes) 000583 1734112439.992287 write 0x001c25 thru 0x0020db (0x4b7 bytes) 000585 1734112439.992985 mapread 0x00e91d thru 0x010d44 (0x2428 bytes) 000586 1734112439.993377 read 0x00be14 thru 0x010e80 (0x506d bytes) 000587 1734112439.995634 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000589 1734112440.059479 write 0x018380 thru 0x02675f (0xe3e0 bytes) 000591 1734112440.060514 trunc from 0x026760 to 0x02884a (0x20eb bytes) 000592 1734112440.065208 trunc from 0x02884b to 0x022bee (0x5c5c bytes) 000593 1734112440.070976 MIRROR_RESYNC 2 mirrors 000594 1734112440.188956 read 0x01d50d thru 0x022bee (0x56e2 bytes) 000596 1734112440.191278 read 0x01006a thru 0x010943 (0x8da bytes) 000597 1734112440.195447 MIRROR_RESYNC 2 mirrors 000598 1734112440.232598 trunc from 0x022bef to 0x001488 (0x21766 bytes) 000599 1734112440.251788 trunc from 0x001489 to 0x021884 (0x203fc bytes) 000600 1734112440.258007 trunc from 0x021885 to 0x01f036 (0x284e bytes) 000603 1734112440.262640 trunc from 0x01f037 to 0x030a35 (0x119ff bytes) 000604 1734112440.266873 trunc from 0x030a36 to 0x03bcf0 (0xb2bb bytes) 000605 1734112440.272525 MIRROR_EXTEND to 3 mirrors 000608 1734112440.391241 write 0x015a85 thru 0x01ccde (0x725a bytes) 000609 1734112440.410919 MIRROR_EXTEND to 4 mirrors 000610 1734112440.527229 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000611 1734112440.597629 mapread 0x002b2b thru 0x009fac (0x7482 bytes) 000612 1734112440.608617 trunc from 0x03bcf1 to 0x0307a3 (0xb54d bytes) 000613 1734112440.626017 mapread 0x00aef8 thru 0x00d7df (0x28e8 bytes) 000614 1734112440.629734 write 0x0219ce thru 0x0230ea (0x171d bytes) 000615 1734112440.632900 read 0x009962 thru 0x013028 (0x96c7 bytes) 000616 1734112440.634609 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000622 1734112440.684075 read 0x02af24 thru 0x0307a3 (0x5880 bytes) 000623 1734112440.687707 trunc from 0x0307a4 to 0x0256c5 (0xb0de bytes) 000624 1734112440.692551 read 0x02194d thru 0x0256c5 (0x3d79 bytes) 000625 1734112440.697909 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000627 1734112440.732543 trunc from 0x0256c6 to 0x031937 (0xc272 bytes) 000628 1734112440.736891 write 0x01db98 thru 0x02178f (0x3bf8 bytes) 000631 1734112440.740313 read 0x029131 thru 0x031937 (0x8807 bytes) 000634 1734112440.743396 mapwrite 0x00e19e thru 0x0152fd (0x7160 bytes) 000635 1734112440.762147 trunc from 0x031938 to 0x000a5c (0x30edb bytes) 000637 1734112440.766523 mapread 0x00073e thru 0x000a5c (0x31f bytes) 000638 1734112440.768298 mapwrite 0x00400d thru 0x007879 (0x386d bytes) 000639 1734112440.790017 read 0x0039fd thru 0x007879 (0x3e7d bytes) 000641 1734112440.790762 mapread 0x000454 thru 0x007879 (0x7426 bytes) 000645 1734112440.791733 read 0x004fde thru 0x007879 (0x289c bytes) 000646 1734112440.791988 mapwrite 0x023680 thru 0x02b7a1 (0x8122 bytes) 000648 1734112440.816657 write 0x027a0c thru 0x02ee57 (0x744c bytes) 000653 1734112440.819666 MIRROR_EXTEND to 2 mirrors 000654 1734112440.937113 trunc from 0x02ee58 to 0x02713f (0x7d18 bytes) 000655 1734112440.958423 read 0x01ee7e thru 0x02713f (0x82c2 bytes) 000656 1734112440.961265 read 0x0057bd thru 0x00ae46 (0x568a bytes) 000660 1734112440.965466 write 0x02094e thru 0x02d32b (0xc9de bytes) 000661 1734112440.966114 read 0x02687e thru 0x02d32b (0x6aae bytes) 000663 1734112440.967905 MIRROR_EXTEND to 3 mirrors 000664 1734112441.091373 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000665 1734112441.145361 read 0x01cd1a thru 0x028740 (0xba27 bytes) 000666 1734112441.148543 mapwrite 0x03d068 thru 0x03ffff (0x2f98 bytes) 000667 1734112441.179846 write 0x01727a thru 0x025da4 (0xeb2b bytes) 000668 1734112441.186289 MIRROR_RESYNC 2 mirrors 000669 1734112441.300161 mapread 0x021b1d thru 0x0260a5 (0x4589 bytes) 000670 1734112441.305345 mapread 0x00b4dd thru 0x01232a (0x6e4e bytes) 000674 1734112441.308119 read 0x012425 thru 0x017f41 (0x5b1d bytes) 000676 1734112441.308395 write 0x039f58 thru 0x03ffff (0x60a8 bytes) 000677 1734112441.330229 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000680 1734112441.368522 trunc from 0x040000 to 0x02bb4e (0x144b1 bytes) 000683 1734112441.377264 read 0x0155f1 thru 0x018ffc (0x3a0c bytes) 000684 1734112441.379234 write 0x011cf4 thru 0x01752f (0x583c bytes) 000688 1734112441.382077 write 0x039a31 thru 0x03ffff (0x65cf bytes) 000689 1734112441.382608 write 0x031652 thru 0x034779 (0x3128 bytes) 000691 1734112441.388436 MIRROR_EXTEND to 2 mirrors 000692 1734112441.531905 write 0x00d544 thru 0x011151 (0x3c0e bytes) 000693 1734112441.551178 read 0x01c8e3 thru 0x01e192 (0x18b0 bytes) 000695 1734112441.553223 mapwrite 0x004a05 thru 0x0068f8 (0x1ef4 bytes) 000696 1734112441.574701 mapread 0x02f729 thru 0x038367 (0x8c3f bytes) 000697 1734112441.579908 mapread 0x0087cb thru 0x016909 (0xe13f bytes) 000698 1734112441.582488 mapwrite 0x02dddd thru 0x033406 (0x562a bytes) 000699 1734112441.600956 trunc from 0x040000 to 0x00bf9e (0x34061 bytes) 000701 1734112441.612885 mapwrite 0x029a71 thru 0x0357e6 (0xbd76 bytes) 000704 1734112441.644475 mapwrite 0x0311c4 thru 0x03ffff (0xee3c bytes) 000706 1734112441.672509 trunc from 0x040000 to 0x03e3be (0x1c41 bytes) 000707 1734112441.676825 write 0x034fca thru 0x03e24a (0x9281 bytes) 000708 1734112441.677780 mapread 0x00844b thru 0x00cab1 (0x4667 bytes) 000710 1734112441.680978 trunc from 0x03e3bf to 0x0348dc (0x9ae2 bytes) 000713 1734112441.685845 write 0x03d321 thru 0x03ffff (0x2cdf bytes) 000714 1734112441.686306 read 0x00465c thru 0x009a89 (0x542e bytes) 000715 1734112441.686594 mapread 0x00e0a9 thru 0x018121 (0xa079 bytes) 000716 1734112441.691043 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000719 1734112441.725463 MIRROR_EXTEND to 2 mirrors 000722 1734112441.832736 trunc from 0x040000 to 0x0362bb (0x9d44 bytes) 000726 1734112441.851990 write 0x016bc0 thru 0x01fafe (0x8f3f bytes) 000727 1734112441.855672 read 0x03138a thru 0x0362bb (0x4f32 bytes) 000729 1734112441.859963 MIRROR_EXTEND to 3 mirrors 000730 1734112441.987077 mapwrite 0x00efc4 thru 0x0128f6 (0x3933 bytes) 000732 1734112442.036585 mapwrite 0x036fbb thru 0x03983f (0x2885 bytes) 000733 1734112442.064952 MIRROR_RESYNC 3 mirrors 000734 1734112442.179916 write 0x00b0c0 thru 0x00fd80 (0x4cc1 bytes) 000735 1734112442.195718 write 0x0030ac thru 0x00a179 (0x70ce bytes) 000736 1734112442.199708 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000737 1734112442.244920 MIRROR_RESYNC 2 mirrors 000740 1734112442.331072 mapwrite 0x028d28 thru 0x034e16 (0xc0ef bytes) 000741 1734112442.376478 read 0x01c973 thru 0x02a051 (0xd6df bytes) 000742 1734112442.378907 trunc from 0x039840 to 0x01fb63 (0x19cdc bytes) 000744 1734112442.387857 write 0x018c2c thru 0x01bf1a (0x32ef bytes) 000749 1734112442.392342 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000750 1734112442.422408 write 0x03c40d thru 0x03ffff (0x3bf3 bytes) 000751 1734112442.422989 mapread 0x03dc22 thru 0x03ffff (0x23de bytes) 000752 1734112442.423370 read 0x02b579 thru 0x0368b7 (0xb33f bytes) 000759 1734112442.426965 trunc from 0x040000 to 0x023343 (0x1ccbc bytes) 000760 1734112442.430773 write 0x008f7b thru 0x017100 (0xe186 bytes) 000761 1734112442.433878 write 0x02cdf9 thru 0x03bbdb (0xede3 bytes) 000762 1734112442.435011 trunc from 0x03bbdc to 0x0202d9 (0x1b902 bytes) 000763 1734112442.439009 mapwrite 0x03c2e1 thru 0x03ffff (0x3d1f bytes) 000764 1734112442.459153 mapread 0x03fddc thru 0x03ffff (0x224 bytes) 000765 1734112442.459577 write 0x033ba9 thru 0x03f629 (0xba81 bytes) 000766 1734112442.462175 mapwrite 0x01143c thru 0x01be5d (0xaa22 bytes) 000768 1734112442.478590 trunc from 0x040000 to 0x03246d (0xdb92 bytes) 000770 1734112442.483211 write 0x03c3c8 thru 0x03ffff (0x3c38 bytes) 000771 1734112442.483700 mapread 0x03a717 thru 0x03ffff (0x58e9 bytes) 000772 1734112442.486030 mapread 0x0139fe thru 0x020c00 (0xd203 bytes) 000773 1734112442.487596 MIRROR_EXTEND to 2 mirrors 000776 1734112442.588567 read 0x0367db thru 0x03d665 (0x6e8b bytes) 000777 1734112442.591695 read 0x0085d6 thru 0x017d5e (0xf789 bytes) 000778 1734112442.595418 trunc from 0x040000 to 0x02996d (0x16692 bytes) 000779 1734112442.615160 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000781 1734112442.643964 trunc from 0x02996e to 0x038510 (0xeba3 bytes) 000783 1734112442.648678 mapread 0x0221ba thru 0x0306dc (0xe523 bytes) 000785 1734112442.657624 write 0x03bfd1 thru 0x03ffff (0x402f bytes) 000787 1734112442.658871 trunc from 0x040000 to 0x0072e6 (0x38d19 bytes) 000789 1734112442.664677 read 0x004bb9 thru 0x0072e6 (0x272e bytes) 000791 1734112442.667328 trunc from 0x0072e7 to 0x01221c (0xaf36 bytes) 000792 1734112442.671309 mapread 0x0021fb thru 0x006b2d (0x4933 bytes) 000794 1734112442.676120 MIRROR_EXTEND to 2 mirrors 000795 1734112442.774627 mapread 0x010499 thru 0x010707 (0x26f bytes) 000796 1734112442.778218 trunc from 0x01221d to 0x005c34 (0xc5e8 bytes) 000797 1734112442.797848 mapread 0x001a17 thru 0x003b91 (0x217b bytes) 000798 1734112442.801121 mapread 0x001d0e thru 0x005c34 (0x3f27 bytes) 000799 1734112442.801557 read 0x00541a thru 0x005c34 (0x81b bytes) 000801 1734112442.802060 trunc from 0x005c35 to 0x01de7c (0x18248 bytes) 000804 1734112442.807552 write 0x000006 thru 0x00bbf9 (0xbbf4 bytes) 000806 1734112442.810863 mapread 0x0044d9 thru 0x01416d (0xfc95 bytes) 000807 1734112442.813456 read 0x002acf thru 0x00a094 (0x75c6 bytes) 000808 1734112442.813757 mapread 0x005385 thru 0x011842 (0xc4be bytes) 000809 1734112442.814144 mapwrite 0x010321 thru 0x01a627 (0xa307 bytes) 000810 1734112442.827683 trunc from 0x01de7d to 0x012958 (0xb524 bytes) 000812 1734112442.832560 mapwrite 0x01457b thru 0x01cada (0x8560 bytes) 000814 1734112442.853493 write 0x038c70 thru 0x03ffff (0x7390 bytes) 000815 1734112442.854503 trunc from 0x040000 to 0x02e31b (0x11ce4 bytes) 000816 1734112442.858526 trunc from 0x02e31c to 0x008d1d (0x255fe bytes) 000820 1734112442.863427 mapwrite 0x02715c thru 0x028c7c (0x1b21 bytes) 000821 1734112442.880667 trunc from 0x028c7d to 0x03b018 (0x1239c bytes) 000822 1734112442.884403 read 0x02fa25 thru 0x03b018 (0xb5f4 bytes) 000823 1734112442.888150 mapwrite 0x00bd87 thru 0x0162d0 (0xa54a bytes) 000827 1734112442.903288 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000828 1734112442.931652 mapread 0x02b5a7 thru 0x02fd02 (0x475c bytes) 000829 1734112442.936466 mapread 0x00047c thru 0x00b5d2 (0xb157 bytes) 000831 1734112442.939075 write 0x02f9f6 thru 0x0322f3 (0x28fe bytes) 000832 1734112442.939534 mapwrite 0x01d5e3 thru 0x029db0 (0xc7ce bytes) 000833 1734112442.954971 mapread 0x0215d6 thru 0x028a58 (0x7483 bytes) 000834 1734112442.955459 mapread 0x0233d5 thru 0x0268a2 (0x34ce bytes) 000835 1734112442.956002 mapwrite 0x03654b thru 0x03ecd0 (0x8786 bytes) 000836 1734112442.976290 mapread 0x00c5d9 thru 0x01117a (0x4ba2 bytes) 000837 1734112442.978933 mapread 0x039ba6 thru 0x03c940 (0x2d9b bytes) 000838 1734112442.979289 mapread 0x0168aa thru 0x016fa2 (0x6f9 bytes) 000839 1734112442.979688 mapwrite 0x01b48a thru 0x0214b4 (0x602b bytes) 000842 1734112442.993268 mapwrite 0x02a353 thru 0x031236 (0x6ee4 bytes) 000843 1734112443.005627 read 0x020b1f thru 0x02544d (0x492f bytes) 000844 1734112443.007600 MIRROR_EXTEND to 2 mirrors 000847 1734112443.111236 mapread 0x0065df thru 0x00e511 (0x7f33 bytes) 000848 1734112443.119451 write 0x03f529 thru 0x03ffff (0xad7 bytes) 000849 1734112443.131758 mapwrite 0x032adb thru 0x0391ae (0x66d4 bytes) 000852 1734112443.147895 mapread 0x0195e4 thru 0x01a3ab (0xdc8 bytes) 000853 1734112443.151890 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000855 1734112443.179190 trunc from 0x040000 to 0x012ec2 (0x2d13d bytes) 000859 1734112443.183858 mapwrite 0x015dd7 thru 0x0255e9 (0xf813 bytes) 000861 1734112443.208237 read 0x01610e thru 0x0212fd (0xb1f0 bytes) 000862 1734112443.208584 write 0x003e86 thru 0x0073bb (0x3536 bytes) 000863 1734112443.211198 mapread 0x013b7b thru 0x01e8ca (0xad50 bytes) 000866 1734112443.212194 trunc from 0x0255ea to 0x015a05 (0xfbe4 bytes) 000867 1734112443.216277 read 0x00a093 thru 0x01027c (0x61ea bytes) 000869 1734112443.217116 mapwrite 0x0104a1 thru 0x015971 (0x54d1 bytes) 000870 1734112443.232454 write 0x017764 thru 0x024750 (0xcfed bytes) 000874 1734112443.234005 write 0x0066f7 thru 0x00a527 (0x3e31 bytes) 000876 1734112443.234692 mapread 0x020ee1 thru 0x024750 (0x3870 bytes) 000877 1734112443.235068 read 0x011a38 thru 0x012b75 (0x113e bytes) 000879 1734112443.235519 mapread 0x00eee4 thru 0x01ca75 (0xdb92 bytes) 000880 1734112443.237993 write 0x018567 thru 0x0269e9 (0xe483 bytes) 000881 1734112443.238665 mapwrite 0x029393 thru 0x033f1f (0xab8d bytes) 000882 1734112443.260685 mapwrite 0x00afda thru 0x017d62 (0xcd89 bytes) 000887 1734112443.276059 mapwrite 0x0133b6 thru 0x01e021 (0xac6c bytes) 000888 1734112443.288985 mapwrite 0x03461f thru 0x03ffff (0xb9e1 bytes) 000889 1734112443.307935 trunc from 0x040000 to 0x01547e (0x2ab81 bytes) 000891 1734112443.311966 trunc from 0x01547f to 0x01c3d1 (0x6f53 bytes) 000894 1734112443.316936 MIRROR_EXTEND to 2 mirrors 000895 1734112443.405195 write 0x024585 thru 0x02a9d0 (0x644c bytes) 000896 1734112443.416969 mapwrite 0x0154fa thru 0x01b629 (0x6130 bytes) 000897 1734112443.434953 mapread 0x002743 thru 0x007ff3 (0x58b1 bytes) 000898 1734112443.437409 read 0x007128 thru 0x00fd97 (0x8c70 bytes) 000899 1734112443.437723 read 0x027f5c thru 0x02a9d0 (0x2a75 bytes) 000900 1734112443.437978 write 0x027800 thru 0x031d43 (0xa544 bytes) 000901 1734112443.438576 write 0x01ff84 thru 0x02bea7 (0xbf24 bytes) 000903 1734112443.439354 read 0x00fbb1 thru 0x017952 (0x7da2 bytes) 000904 1734112443.439637 trunc from 0x031d44 to 0x005b4c (0x2c1f7 bytes) 000905 1734112443.448227 write 0x02527c thru 0x034f16 (0xfc9b bytes) 000906 1734112443.449336 read 0x010fce thru 0x019bbd (0x8bf0 bytes) 000907 1734112443.453328 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000908 1734112443.480950 write 0x033316 thru 0x0364de (0x31c9 bytes) 000909 1734112443.482372 read 0x02aa54 thru 0x0342c5 (0x9872 bytes) 000910 1734112443.484061 trunc from 0x0364df to 0x009555 (0x2cf89 bytes) 000912 1734112443.487150 write 0x015faf thru 0x01767f (0x16d1 bytes) 000913 1734112443.488429 MIRROR_EXTEND to 2 mirrors 000914 1734112443.583193 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb0d:0x0]: Invalid argument (22) 000917 1734112443.614537 write 0x02b38d thru 0x038ac4 (0xd738 bytes) 000919 1734112443.617975 trunc from 0x038ac5 to 0x0325f2 (0x64d2 bytes) 000920 1734112443.625463 mapwrite 0x025998 thru 0x030b6b (0xb1d4 bytes) 000921 1734112443.647783 mapread 0x01cb78 thru 0x028c2b (0xc0b4 bytes) 000923 1734112443.650961 mapwrite 0x00f220 thru 0x013ddb (0x4bbc bytes) 000926 1734112443.668680 read 0x020a6a thru 0x02c24c (0xb7e3 bytes) 000927 1734112443.668987 write 0x024cd0 thru 0x02868c (0x39bd bytes) 000929 1734112443.669821 mapread 0x02a3f0 thru 0x02e1c4 (0x3dd5 bytes) 000932 1734112443.670566 write 0x00d808 thru 0x01b816 (0xe00f bytes) 000933 1734112443.671139 mapread 0x030159 thru 0x031b45 (0x19ed bytes) 000936 1734112443.671888 trunc from 0x0325f3 to 0x03193c (0xcb6 bytes) 000937 1734112443.676308 mapwrite 0x02d006 thru 0x032ddc (0x5dd7 bytes) 000938 1734112443.697481 trunc from 0x032ddd to 0x021e5e (0x10f7e bytes) 000939 1734112443.701443 write 0x00ff4c thru 0x01d10f (0xd1c4 bytes) 000940 1734112443.702068 write 0x00ba79 thru 0x0188d4 (0xce5c bytes) 000941 1734112443.702666 trunc from 0x021e5f to 0x01ab75 (0x72e9 bytes) 000942 1734112443.706565 write 0x034b7b thru 0x03ee6d (0xa2f3 bytes) 000943 1734112443.707226 mapwrite 0x0080da thru 0x009d81 (0x1ca8 bytes) 000946 1734112443.726800 mapwrite 0x00923c thru 0x015b3e (0xc903 bytes) 000947 1734112443.743456 write 0x0161ce thru 0x024948 (0xe77b bytes) 000948 1734112443.745755 write 0x0106d3 thru 0x01fb81 (0xf4af bytes) 000949 1734112443.746359 mapread 0x035bfa thru 0x03e33e (0x8745 bytes) 000952 1734112443.747144 trunc from 0x03ee6e to 0x01a6d6 (0x24797 bytes) 000959 1734112443.753290 read 0x00098d thru 0x0080af (0x7723 bytes) 000960 1734112443.755856 trunc from 0x01a6d7 to 0x01f7a4 (0x50ce bytes) 000963 1734112443.761416 MIRROR_EXTEND to 2 mirrors 000967 1734112443.870494 trunc from 0x01f7a5 to 0x035385 (0x15be1 bytes) 000969 1734112443.890071 mapwrite 0x00444d thru 0x0131f8 (0xedac bytes) 000971 1734112443.913474 mapwrite 0x024f4b thru 0x02946a (0x4520 bytes) 000972 1734112443.927918 MIRROR_EXTEND to 3 mirrors 000973 1734112444.024989 write 0x0114f7 thru 0x0126a4 (0x11ae bytes) 000974 1734112444.041834 write 0x008e96 thru 0x00df72 (0x50dd bytes) 000975 1734112444.045977 MIRROR_RESYNC 3 mirrors 000978 1734112444.175351 mapwrite 0x0161a7 thru 0x01a600 (0x445a bytes) 000979 1734112444.223774 read 0x0322a3 thru 0x035385 (0x30e3 bytes) 000981 1734112444.225927 read 0x018589 thru 0x027dcb (0xf843 bytes) 000982 1734112444.229235 trunc from 0x035386 to 0x017975 (0x1da10 bytes) 000983 1734112444.234990 write 0x014206 thru 0x0239aa (0xf7a5 bytes) 000988 1734112444.236593 write 0x02820d thru 0x030943 (0x8737 bytes) 000989 1734112444.237158 write 0x0086ac thru 0x0108ff (0x8254 bytes) 000993 1734112444.238419 read 0x01afd9 thru 0x020b21 (0x5b49 bytes) 000994 1734112444.238753 trunc from 0x030944 to 0x02e6ef (0x2254 bytes) 000995 1734112444.243407 mapread 0x019d9c thru 0x01c3dd (0x2642 bytes) 000996 1734112444.243848 read 0x021c52 thru 0x0224a3 (0x852 bytes) 000997 1734112444.244116 mapwrite 0x01527a thru 0x022ca7 (0xda2e bytes) 000998 1734112444.265236 mapwrite 0x035876 thru 0x039907 (0x4092 bytes) 000999 1734112444.293210 MIRROR_RESYNC 3 mirrors 001000 1734112444.424906 write 0x01e6c5 thru 0x026dd6 (0x8712 bytes) All operations completed A-OK! PASS 70a (14s) == sanity-flr test 200a: stress test ===================== 12:54:05 (1734112445) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre2 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5097568 .. Extending file size to 4558944 .. failed Extending file size to 6972960 .. Extending file size to 7435104 .. Extending file size to 7061920 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8067840 .. Extending file size to 8658080 .. Extending file size to 8088480 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8731040 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9123168 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8860672 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8867360 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8938752 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 2700 2701 2703 2705 2707 failed 192.168.201.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg131-client.virtnet /mnt/lustre2 (opts:) 192.168.201.131@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg131-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:55:09 (1734112509) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre2 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 993440 .. Extending file size to 1346080 .. done Extending file size to 7379648 .. Extending file size to 7769312 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7951424 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7892384 .. failed Extending file size to 8929632 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8319296 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9146976 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8412768 .. Extending file size to 8792480 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9110912 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 5941 5942 5944 5945 5948 192.168.201.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg131-client.virtnet /mnt/lustre2 (opts:) 192.168.201.131@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg131-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:56:12 (1734112572) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 5120 2203264 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 5120 2203264 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0291121 s, 72.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0724153 s, 29.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg131-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:56:16 (1734112576) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0327823 s, 64.0 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 12:56:18 (1734112578) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb57:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc4:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xdbe:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:56:20 (1734112580) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0253639 s, 82.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdbf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:56:21 (1734112581) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0229498 s, 91.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc2:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:56:23 (1734112583) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0218842 s, 95.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc4:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:56:46 (1734112606) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg131-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0168906 s, 62.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdca:0x0] } PASS 204d (42s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:57:28 (1734112648) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg131-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0230699 s, 90.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdcb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdcb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:58:12 (1734112692) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg131-server mds-ost sync done. Creating new pool test_204f oleg131-server: Pool lustre.test_204f created Adding targets to pool oleg131-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg131-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.043306 s, 48.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdcc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdcc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg131-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg131-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg131-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (52s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:59:04 (1734112744) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:59:06 (1734112746) PASS 205b (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:59:07 (1734112747) oleg131-server: Pool lustre.test_206 created Adding targets to pool oleg131-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg131-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg131-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg131-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg131-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:59:20 (1734112760) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0254441 s, 41.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00806052 s, 130 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x9cb40309 Mirror 3: 0x9cb40309 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:59:22 (1734112762) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:59:23 (1734112763) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:59:24 (1734112764) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0161918 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2150000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2010000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2010000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2010000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1dc0000 Read this (4 bytes): BBBB 192.168.201.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg131-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:59:38 (1734112778) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00342121 s, 2.4 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:59:39 (1734112779) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121673 s, 86.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xb7c lmm_fid: [0x200000404:0xb7c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xb7c lmm_fid: [0x200000404:0xb7c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd7:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836860 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-13 12:59:40.000000000 -0500 Modify: 2024-12-13 12:59:40.000000000 -0500 Change: 2024-12-13 12:59:40.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:59:41 (1734112781) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:59:42 (1734112782) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.106737 s, 384 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0141891 s, 289 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 13 12:59 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xb7e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 13 12:59 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212: I/O should skip the foreign FLR layout component ========================================================== 12:59:44 (1734112784) lfs mirror create: cannot create composite file '/mnt/lustre/f212.sanity-flr': Invalid argument sanity-flr test_212: @@@@@@ FAIL: /mnt/lustre/f212.sanity-flr: create failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7229:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:4568:test_212() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7602:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7665:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7483:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:4587:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_212.*.1734112785.log rsync: chown "/tmp/testlogs/.sanity-flr.test_212.debug_log.oleg131-server.1734112785.log.xrYXJo" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_212.dmesg.oleg131-server.1734112785.log.2hMSco" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 212 (2s) == sanity-flr test complete, duration 679 sec ============ 12:59:47 (1734112787) sanity-flr: FAIL: test_212 /mnt/lustre/f212.sanity-flr: create failed === sanity-flr: start cleanup 12:59:48 (1734112788) === === sanity-flr: finish cleanup 12:59:49 (1734112789) ===