-----============= acceptance-small: sanity-flr ============----- Wed Apr 23 21:02:25 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg306-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 21:02:30 (1745456550) === oleg306-client.virtnet: executing check_config_client /mnt/lustre oleg306-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg306-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.idle_timeout=debug osc.lustre-OST0001-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.idle_timeout=debug disable quota as required oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 21:02:37 (1745456557) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f6999] == sanity-flr test 0a: lfs mirror create with -N option == 21:02:37 (1745456557) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:02:40 (1745456560) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:02:41 (1745456561) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:02:42 (1745456562) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:02:46 (1745456566) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:02:47 (1745456567) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) == sanity-flr test 0g: lfs mirror create flags support === 21:02:47 (1745456567) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.stats=clear osc.lustre-OST0001-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:02:49 (1745456569) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:02:51 (1745456571) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:02:53 (1745456573) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:02:54 (1745456574) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 21:02:56 (1745456576) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:02:57 (1745456577) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:02:58 (1745456578) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:02:59 (1745456579) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:03:02 (1745456582) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:03:03 (1745456583) 18+0 records in 18+0 records out 18874368 bytes (19 MB) copied, 0.485986 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0527191 s, 19.9 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:03:06 (1745456586) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0328337 s, 192 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:03:08 (1745456588) Stopping /mnt/lustre-ost1 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid 50 osc.lustre-OST0000-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg306-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid 50 osc.lustre-OST0001-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid in DISCONN state after 2 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg306-server: *.lustre-OST0001.recovery_status status: RECOVERING oleg306-server: Waiting 1470 secs for *.lustre-OST0001.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (28s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:03:36 (1745456616) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.131097 s, 64.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid 50 osc.lustre-OST0000-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg306-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg306-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:03:52 (1745456632) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid 50 osc.lustre-OST0000-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg306-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid 50 osc.lustre-OST0001-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg306-server: *.lustre-OST0001.recovery_status status: RECOVERING oleg306-server: Waiting 1470 secs for *.lustre-OST0001.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (24s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:04:16 (1745456656) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.746873 s, 70.2 MB/s -rw-r--r-- 1 root root 52428800 Apr 23 21:04 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.203s user 0m0.001s sys 0m0.102s Stopping /mnt/lustre-ost1 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid 50 osc.lustre-OST0000-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.217s user 0m0.001s sys 0m0.114s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg306-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid 50 osc.lustre-OST0001-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.375s user 0m0.000s sys 0m0.124s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg306-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg306-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid 50 osc.lustre-OST0001-osc-880fc814-334c-4ee0-b96e-f4977ea299b2.ost_server_uuid in FULL state after 0 sec PASS 33b (29s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:04:45 (1745456685) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:04:46 (1745456686) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:04:48 (1745456688) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 21:04:50 (1745456690) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 21:04:52 (1745456692) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.123832 s, 102 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:04:56 (1745456696) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00843648 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00825254 s, 127 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:04:57 (1745456697) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:05:00 (1745456700) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00172329 s, 594 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000733888 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00084662 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00061372 s, 1.7 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 21:05:03 (1745456703) create test files with size 15760550 .. db20c8d6ac92917fd4e7ae1f07770e07 - 032e62527e750fe1df4bc7df3b9f6206 - 12838c39ecdd85a63eb24a670c8069ae - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg306-client.virtnet /mnt/lustre (opts:) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 21:05:09 (1745456709) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:05:13 (1745456713) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:05:15 (1745456715) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0384408 s, 27.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0149473 s, 70.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 21:05:18 (1745456718) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0383536 s, 54.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.131521 s, 31.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.120992 s, 17.3 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 21:05:22 (1745456722) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 21:05:23 (1745456723) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:05:24 (1745456724) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108041 s, 97.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 21:05:26 (1745456726) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:05:27 (1745456727) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:05:30 (1745456730) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0932456 s, 112 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg306-client.virtnet /mnt/lustre (opts:) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (20s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:05:51 (1745456751) llite.lustre-6b32e031-39eb-4826-9313-b485886046eb.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.073163 s, 143 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg306-client.virtnet /mnt/lustre (opts:) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-bc66073b-196d-43dc-a36d-6b3520645181.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:05:54 (1745456754) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:06:01 (1745456761) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (8s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:06:09 (1745456769) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 21:06:10 (1745456770) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0621718 s, 50.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.056034 s, 56.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 489a71f53673bcc9e00707f497b08c97 - mirror 1 checksum: 361fae08368fae2575dc59998e6beed9 - mirror 2 checksum: 489a71f53673bcc9e00707f497b08c97 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Apr 23 21:06:11 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Apr 23 21:06:11 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 489a71f53673bcc9e00707f497b08c97 - mirror 1 checksum: 361fae08368fae2575dc59998e6beed9 - mirror 2 checksum: 361fae08368fae2575dc59998e6beed9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Apr 23 21:06:11 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Apr 23 21:06:11 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (2s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:06:13 (1745456773) Failing mds1 on oleg306-server Stopping /mnt/lustre-mds1 (opts:) on oleg306-server 21:06:15 (1745456775) shut down facet: mds1 facet_host: oleg306-server facet_failover_host: oleg306-server Failover mds1 to oleg306-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 21:06:28 (1745456788) targets are mounted 21:06:28 (1745456788) facet_failover done oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0086686 s, 121 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (20s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:06:33 (1745456793) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:06:34 (1745456794) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:06:35 (1745456795) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00328555 s, 5.0 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:06:38 (1745456798) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:06:39 (1745456799) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:06:40 (1745456800) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:06:46 (1745456806) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:07:05 (1745456825) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:07:23 (1745456843) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:07:41 (1745456861) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 21:07:43 (1745456863) mirror create pid 22493 mirror split pid 22494 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7567: 22494 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 22493 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 21:08:45 (1745456925) Seed set to 6926 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x3aaf6 000007 1745456926.223270 trunc from 00000000 to 0x03aaf5 (0x3aaf6 bytes) 000011 1745456926.230995 write 0x01abcc thru 0x026bee (0xc023 bytes) 000015 1745456926.237361 read 0x02fb3d thru 0x037173 (0x7637 bytes) 000016 1745456926.241272 trunc from 0x03aaf6 to 0x016fe4 (0x23b11 bytes) 000018 1745456926.249662 mapread 0x001ef8 thru 0x00f6b0 (0xd7b9 bytes) 000019 1745456926.259435 read 0x00f67f thru 0x0119ac (0x232e bytes) 000021 1745456926.260223 mapread 0x000824 thru 0x006866 (0x6043 bytes) 000022 1745456926.262939 mapwrite 0x03a5bc thru 0x03ffff (0x5a44 bytes) 000024 1745456926.313721 mapwrite 0x01e0b2 thru 0x02776b (0x96ba bytes) 000026 1745456926.346195 trunc from 0x040000 to 0x00bb02 (0x344fd bytes) 000027 1745456926.352660 mapread 0x007771 thru 0x00bb02 (0x4392 bytes) 000029 1745456926.353971 mapwrite 0x019b4f thru 0x01e2bc (0x476e bytes) 000030 1745456926.401969 trunc from 0x01e2bd to 0x021a47 (0x378b bytes) 000035 1745456926.411948 write 0x039829 thru 0x03ffff (0x67d7 bytes) 000037 1745456926.413035 mapwrite 0x005e4b thru 0x006119 (0x2cf bytes) 000039 1745456926.437503 read 0x031e7a thru 0x0353be (0x3545 bytes) 000041 1745456926.445534 mapwrite 0x006460 thru 0x00b2a0 (0x4e41 bytes) 000042 1745456926.463905 read 0x0176ec thru 0x01ca9f (0x53b4 bytes) 000043 1745456926.464497 write 0x0025dc thru 0x002d25 (0x74a bytes) 000046 1745456926.465774 write 0x01a9c7 thru 0x01f127 (0x4761 bytes) 000048 1745456926.469792 MIRROR_EXTEND to 2 mirrors 000050 1745456926.667972 trunc from 0x040000 to 0x00563b (0x3a9c4 bytes) 000051 1745456926.717415 mapread 0x00188d thru 0x00563b (0x3daf bytes) 000052 1745456926.728416 read 0x00379f thru 0x00563b (0x1e9d bytes) 000054 1745456926.732023 mapread 0x00494f thru 0x00563b (0xced bytes) 000058 1745456926.734279 write 0x01bfbc thru 0x024a1a (0x8a5f bytes) 000059 1745456926.735327 mapread 0x002696 thru 0x004e87 (0x27f2 bytes) 000060 1745456926.735857 trunc from 0x024a1b to 0x01348c (0x1158e bytes) 000061 1745456926.744690 mapwrite 0x0387e1 thru 0x03bbae (0x33ce bytes) 000063 1745456926.799356 trunc from 0x03bbaf to 0x02415e (0x17a50 bytes) 000065 1745456926.811024 mapread 0x00d27c thru 0x01639d (0x9122 bytes) 000067 1745456926.832400 trunc from 0x02415f to 0x00b2f6 (0x18e68 bytes) 000068 1745456926.851054 write 0x02126e thru 0x02371c (0x24af bytes) 000070 1745456926.856427 MIRROR_RESYNC 2 mirrors 000073 1745456927.036200 mapread 0x00096b thru 0x00cf9b (0xc631 bytes) 000074 1745456927.051446 write 0x014985 thru 0x02015d (0xb7d9 bytes) 000076 1745456927.091418 mapread 0x00f54a thru 0x018aae (0x9565 bytes) 000077 1745456927.092228 mapread 0x021881 thru 0x02371c (0x1e9c bytes) 000078 1745456927.096935 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000082 1745456927.156854 mapwrite 0x01589f thru 0x0240bd (0xe81f bytes) 000083 1745456927.223830 read 0x01d6d8 thru 0x0240bd (0x69e6 bytes) 000085 1745456927.224598 trunc from 0x0240be to 0x01b7f1 (0x88cc bytes) 000086 1745456927.230419 mapwrite 0x026eee thru 0x034894 (0xd9a7 bytes) 000088 1745456927.267388 mapread 0x02a067 thru 0x02de8e (0x3e28 bytes) 000089 1745456927.267899 mapread 0x02da5f thru 0x034894 (0x6e36 bytes) 000090 1745456927.268469 trunc from 0x034895 to 0x00c5fe (0x28296 bytes) 000091 1745456927.274685 mapwrite 0x012715 thru 0x020bb3 (0xe49f bytes) 000093 1745456927.332987 mapwrite 0x03340f thru 0x03c921 (0x9513 bytes) 000095 1745456927.386682 trunc from 0x03c922 to 0x01fe8b (0x1ca96 bytes) 000096 1745456927.397922 write 0x017017 thru 0x019682 (0x266c bytes) 000097 1745456927.398624 read 0x00031f thru 0x00fda6 (0xfa88 bytes) 000098 1745456927.403610 mapwrite 0x0383bd thru 0x03f59e (0x71e2 bytes) 000100 1745456927.455378 mapread 0x028de4 thru 0x0339d4 (0xabf1 bytes) 000101 1745456927.456385 write 0x02d5fb thru 0x037847 (0xa24d bytes) 000106 1745456927.460086 write 0x03c3ef thru 0x03ffff (0x3c11 bytes) 000108 1745456927.462397 write 0x01456e thru 0x01a23f (0x5cd2 bytes) 000111 1745456927.463652 mapwrite 0x038582 thru 0x03ffff (0x7a7e bytes) 000118 1745456927.486440 mapread 0x0238ac thru 0x0318cf (0xe024 bytes) 000120 1745456927.487598 mapread 0x008fdb thru 0x010d02 (0x7d28 bytes) 000124 1745456927.489303 read 0x029212 thru 0x02d53c (0x432b bytes) 000125 1745456927.489680 mapread 0x00ce89 thru 0x0159e3 (0x8b5b bytes) 000126 1745456927.492986 MIRROR_EXTEND to 2 mirrors 000127 1745456927.701277 mapwrite 0x00504f thru 0x0120fd (0xd0af bytes) 000129 1745456927.798518 mapread 0x0066d8 thru 0x015da4 (0xf6cd bytes) 000130 1745456927.799240 mapread 0x03819a thru 0x03ffff (0x7e66 bytes) 000131 1745456927.806904 write 0x026631 thru 0x02f25a (0x8c2a bytes) truncating to largest ever: 0x3ea67 000133 1745456927.822339 trunc from 0x040000 to 0x03ea66 (0x1599 bytes) 000135 1745456927.841260 mapwrite 0x0313f7 thru 0x03ccc1 (0xb8cb bytes) 000136 1745456927.873437 MIRROR_EXTEND to 3 mirrors 000137 1745456928.086844 write 0x01b32d thru 0x01f36f (0x4043 bytes) 000138 1745456928.111242 trunc from 0x03ea67 to 0x017fb8 (0x26aae bytes) 000139 1745456928.117690 write 0x0275f3 thru 0x031cf2 (0xa700 bytes) 000140 1745456928.118784 trunc from 0x031cf3 to 0x031cb6 (0x3c bytes) 000143 1745456928.125117 trunc from 0x031cb7 to 0x0011a7 (0x30b0f bytes) 000144 1745456928.131874 mapread 0x00011c thru 0x0011a7 (0x108c bytes) 000147 1745456928.138371 read 0x001127 thru 0x0011a7 (0x81 bytes) 000149 1745456928.138919 trunc from 0x0011a8 to 0x003ffa (0x2e53 bytes) 000150 1745456928.144706 write 0x025c8c thru 0x028c6e (0x2fe3 bytes) 000151 1745456928.147925 MIRROR_EXTEND to 4 mirrors 000152 1745456928.355394 mapwrite 0x0331bf thru 0x03eabe (0xb900 bytes) 000153 1745456928.436683 write 0x004b90 thru 0x011565 (0xc9d6 bytes) 000156 1745456928.445479 write 0x032c6c thru 0x03bb68 (0x8efd bytes) 000158 1745456928.447983 mapread 0x01539c thru 0x022a2e (0xd693 bytes) 000161 1745456928.462099 read 0x030bae thru 0x034886 (0x3cd9 bytes) 000163 1745456928.463141 read 0x019a9d thru 0x01c0fa (0x265e bytes) 000165 1745456928.465616 MIRROR_EXTEND to 5 mirrors 000167 1745456928.657354 trunc from 0x03eabf to 0x031989 (0xd135 bytes) 000169 1745456928.702315 trunc from 0x03198a to 0x01bf4c (0x15a3d bytes) 000171 1745456928.710063 mapread 0x01109f thru 0x012981 (0x18e3 bytes) 000173 1745456928.717500 MIRROR_RESYNC 5 mirrors 000174 1745456929.038140 mapread 0x0091ce thru 0x00995f (0x792 bytes) 000176 1745456929.042707 trunc from 0x01bf4d to 0x035e5e (0x19f12 bytes) 000178 1745456929.077881 write 0x017c34 thru 0x0203e1 (0x87ae bytes) 000179 1745456929.081731 mapwrite 0x019e8e thru 0x023397 (0x950a bytes) 000181 1745456929.109934 mapread 0x01f230 thru 0x029cf8 (0xaac9 bytes) 000182 1745456929.110689 read 0x030d0f thru 0x035e5e (0x5150 bytes) 000186 1745456929.119791 mapread 0x01b887 thru 0x0297d8 (0xdf52 bytes) 000191 1745456929.121276 read 0x0071b3 thru 0x0089c2 (0x1810 bytes) 000193 1745456929.130704 MIRROR_RESYNC 5 mirrors 000196 1745456929.412673 mapread 0x00e1f4 thru 0x01c53a (0xe347 bytes) 000197 1745456929.422630 read 0x010a9d thru 0x01954c (0x8ab0 bytes) 000198 1745456929.422908 write 0x01adb5 thru 0x021c58 (0x6ea4 bytes) 000199 1745456929.450311 write 0x012821 thru 0x0151f3 (0x29d3 bytes) 000200 1745456929.453987 mapwrite 0x00a116 thru 0x010626 (0x6511 bytes) 000201 1745456929.493526 write 0x03ebfc thru 0x03ffff (0x1404 bytes) 000203 1745456929.494745 mapread 0x022303 thru 0x031252 (0xef50 bytes) 000204 1745456929.503107 mapwrite 0x014461 thru 0x019593 (0x5133 bytes) 000205 1745456929.530971 write 0x02aa1f thru 0x02b590 (0xb72 bytes) 000207 1745456929.532193 write 0x02828c thru 0x0284bc (0x231 bytes) 000209 1745456929.533537 mapread 0x00ca4c thru 0x014f57 (0x850c bytes) 000210 1745456929.534248 read 0x035fec thru 0x03a796 (0x47ab bytes) 000211 1745456929.538966 trunc from 0x040000 to 0x02e557 (0x11aa8 bytes) 000213 1745456929.555094 mapwrite 0x034dea thru 0x03ad8e (0x5fa5 bytes) 000214 1745456929.625973 mapwrite 0x01c083 thru 0x021243 (0x51c1 bytes) 000216 1745456929.667649 write 0x009e44 thru 0x00ca7a (0x2c37 bytes) 000218 1745456929.668810 write 0x024f7d thru 0x02df2a (0x8fae bytes) 000219 1745456929.669622 mapwrite 0x033e88 thru 0x03511f (0x1298 bytes) 000221 1745456929.697054 trunc from 0x03ad8f to 0x031d81 (0x900d bytes) 000222 1745456929.705937 mapread 0x02f2d0 thru 0x031d81 (0x2ab2 bytes) 000223 1745456929.712608 write 0x0006a8 thru 0x00ed91 (0xe6ea bytes) 000225 1745456929.713855 trunc from 0x031d82 to 0x0169cc (0x1b3b5 bytes) 000226 1745456929.720280 read 0x012dab thru 0x012e7b (0xd1 bytes) 000227 1745456929.723681 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000228 1745456929.844696 write 0x036e37 thru 0x03a0f5 (0x32bf bytes) 000229 1745456929.847505 MIRROR_RESYNC 4 mirrors 000231 1745456930.185661 write 0x02f961 thru 0x03d1cf (0xd86f bytes) 000232 1745456930.213143 mapwrite 0x02cc97 thru 0x03acb6 (0xe020 bytes) 000233 1745456930.255845 trunc from 0x03d1d0 to 0x03a99c (0x2833 bytes) 000234 1745456930.264257 mapwrite 0x037390 thru 0x03ffff (0x8c70 bytes) 000235 1745456930.301778 mapwrite 0x0263e7 thru 0x030076 (0x9c90 bytes) 000236 1745456930.337475 mapwrite 0x03ee21 thru 0x03ffff (0x11df bytes) 000237 1745456930.366054 mapwrite 0x0181f2 thru 0x018adf (0x8ee bytes) 000238 1745456930.391185 trunc from 0x040000 to 0x002d6e (0x3d291 bytes) 000240 1745456930.399460 write 0x00e277 thru 0x0163c4 (0x814e bytes) 000242 1745456930.400794 mapwrite 0x01cf81 thru 0x027719 (0xa799 bytes) 000244 1745456930.444939 write 0x035403 thru 0x03840f (0x300d bytes) 000246 1745456930.446093 write 0x02cc28 thru 0x03c375 (0xf74e bytes) 000248 1745456930.450241 mapwrite 0x01dbb6 thru 0x027e49 (0xa294 bytes) 000249 1745456930.472690 mapread 0x029f86 thru 0x02bb40 (0x1bbb bytes) 000250 1745456930.473530 mapread 0x02291c thru 0x02d1ed (0xa8d2 bytes) 000251 1745456930.474390 trunc from 0x03c376 to 0x022c50 (0x19725 bytes) 000252 1745456930.485857 MIRROR_RESYNC 4 mirrors 000254 1745456930.737885 mapread 0x00b58b thru 0x01a6ff (0xf175 bytes) 000255 1745456930.749833 mapread 0x021953 thru 0x022c50 (0x12fe bytes) 000256 1745456930.752269 mapwrite 0x00fbb2 thru 0x014ff9 (0x5448 bytes) 000257 1745456930.816020 trunc from 0x022c51 to 0x0274b2 (0x4862 bytes) 000258 1745456930.828298 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000263 1745456930.923809 MIRROR_RESYNC 3 mirrors 000266 1745456931.159583 mapread 0x001c57 thru 0x0072ee (0x5698 bytes) 000268 1745456931.166948 write 0x02dbe1 thru 0x030702 (0x2b22 bytes) 000269 1745456931.189232 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000271 1745456931.275690 mapwrite 0x02b83a thru 0x03b571 (0xfd38 bytes) 000272 1745456931.332865 read 0x024fd6 thru 0x025134 (0x15f bytes) 000274 1745456931.336731 trunc from 0x03b572 to 0x038273 (0x32fe bytes) 000275 1745456931.343245 read 0x00a0f1 thru 0x0178f1 (0xd801 bytes) 000277 1745456931.351559 trunc from 0x038274 to 0x01892b (0x1f948 bytes) 000279 1745456931.360868 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000280 1745456931.423163 mapwrite 0x00189f thru 0x003c93 (0x23f5 bytes) 000282 1745456931.454311 mapwrite 0x00629b thru 0x012a46 (0xc7ac bytes) 000283 1745456931.473014 write 0x031a35 thru 0x035878 (0x3e44 bytes) 000285 1745456931.475272 trunc from 0x035879 to 0x018e5a (0x1ca1e bytes) 000286 1745456931.481644 trunc from 0x018e5b to 0x025de7 (0xcf8d bytes) 000287 1745456931.486628 trunc from 0x025de8 to 0x026bfa (0xe13 bytes) 000288 1745456931.491024 mapread 0x01bda6 thru 0x026bfa (0xae55 bytes) 000289 1745456931.497466 mapread 0x0095ce thru 0x00fb4b (0x657e bytes) 000291 1745456931.498398 mapwrite 0x031a80 thru 0x037a1f (0x5fa0 bytes) 000292 1745456931.534709 trunc from 0x037a20 to 0x01f265 (0x187ba bytes) 000293 1745456931.541048 mapread 0x0082f3 thru 0x009543 (0x1251 bytes) 000294 1745456931.541595 write 0x032cfe thru 0x039515 (0x6818 bytes) 000295 1745456931.542413 trunc from 0x039516 to 0x006b7a (0x3299b bytes) 000298 1745456931.553902 write 0x02ef2f thru 0x03c832 (0xd904 bytes) 000301 1745456931.555658 write 0x03dc2a thru 0x03ffff (0x23d6 bytes) 000304 1745456931.558198 mapwrite 0x014f7a thru 0x018ae3 (0x3b6a bytes) 000307 1745456931.595602 mapwrite 0x03ce04 thru 0x03ffff (0x31fc bytes) 000308 1745456931.616777 read 0x023ee7 thru 0x02f8f1 (0xba0b bytes) 000309 1745456931.621073 mapwrite 0x019dfb thru 0x022c46 (0x8e4c bytes) 000311 1745456931.653226 write 0x02c084 thru 0x03228f (0x620c bytes) 000313 1745456931.659785 write 0x024089 thru 0x027d7d (0x3cf5 bytes) 000315 1745456931.660889 mapread 0x022665 thru 0x02a4b4 (0x7e50 bytes) 000317 1745456931.664238 mapwrite 0x007d19 thru 0x00c2ac (0x4594 bytes) 000320 1745456931.692381 mapwrite 0x01aa00 thru 0x02a4a9 (0xfaaa bytes) 000321 1745456931.724681 trunc from 0x040000 to 0x011acd (0x2e532 bytes) 000322 1745456931.732820 read 0x004e52 thru 0x011acd (0xcc7c bytes) 000323 1745456931.733310 write 0x03eb2e thru 0x03ffff (0x14d2 bytes) 000325 1745456931.734613 write 0x020aa3 thru 0x02e697 (0xdbf5 bytes) 000327 1745456931.744165 mapread 0x0366f0 thru 0x03878c (0x209d bytes) 000328 1745456931.744904 trunc from 0x040000 to 0x00f94b (0x306b4 bytes) 000331 1745456931.758828 mapread 0x003455 thru 0x003f4e (0xafa bytes) 000333 1745456931.760085 mapwrite 0x015aad thru 0x01fa70 (0x9fc4 bytes) 000337 1745456931.827355 mapread 0x00cd96 thru 0x016095 (0x9300 bytes) 000339 1745456931.837273 mapread 0x003183 thru 0x00636a (0x31e8 bytes) 000340 1745456931.837845 mapwrite 0x03507b thru 0x03ad0b (0x5c91 bytes) 000341 1745456931.878690 read 0x034b5a thru 0x03ad0b (0x61b2 bytes) 000342 1745456931.882534 read 0x0315d0 thru 0x0387fb (0x722c bytes) 000344 1745456931.883691 write 0x03fb43 thru 0x03ffff (0x4bd bytes) 000347 1745456931.886093 mapwrite 0x02ec0b thru 0x030ce0 (0x20d6 bytes) 000350 1745456931.911267 read 0x00ca26 thru 0x00f5a3 (0x2b7e bytes) 000351 1745456931.911698 mapread 0x01782a thru 0x023ada (0xc2b1 bytes) 000352 1745456931.912330 read 0x02c24d thru 0x031fcd (0x5d81 bytes) 000353 1745456931.912770 mapread 0x02e2f3 thru 0x031bf7 (0x3905 bytes) 000354 1745456931.913370 write 0x0123b1 thru 0x021c29 (0xf879 bytes) 000355 1745456931.914358 trunc from 0x040000 to 0x01c093 (0x23f6c bytes) 000356 1745456931.920830 write 0x03c9dc thru 0x03ffff (0x3624 bytes) 000358 1745456931.921903 write 0x02f5b9 thru 0x031919 (0x2361 bytes) 000359 1745456931.927404 read 0x0284a6 thru 0x0340c9 (0xbc24 bytes) 000360 1745456931.927861 write 0x03f7cd thru 0x03ffff (0x833 bytes) 000361 1745456931.928246 trunc from 0x040000 to 0x02ecdd (0x11322 bytes) 000362 1745456931.935675 write 0x007b12 thru 0x00a008 (0x24f7 bytes) 000365 1745456931.937304 read 0x02b84e thru 0x02dc86 (0x2439 bytes) 000367 1745456931.938028 write 0x022e8b thru 0x02bdaa (0x8f20 bytes) 000368 1745456931.938786 trunc from 0x02ecde to 0x008b08 (0x261d5 bytes) 000374 1745456931.953530 mapwrite 0x01c063 thru 0x0223e0 (0x637e bytes) 000375 1745456932.002074 MIRROR_EXTEND to 2 mirrors 000376 1745456932.167582 trunc from 0x0223e1 to 0x011ee8 (0x104f8 bytes) 000377 1745456932.191283 trunc from 0x011ee9 to 0x02e81d (0x1c935 bytes) 000378 1745456932.198382 mapread 0x02c3b8 thru 0x02e81d (0x2466 bytes) 000379 1745456932.206638 MIRROR_RESYNC 2 mirrors 000381 1745456932.351403 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000382 1745456932.414018 read 0x01e4c9 thru 0x02db9f (0xf6d7 bytes) 000383 1745456932.424791 MIRROR_EXTEND to 2 mirrors 000387 1745456932.634812 write 0x01bc9e thru 0x01e1fd (0x2560 bytes) 000388 1745456932.671945 mapread 0x01a61e thru 0x027ecd (0xd8b0 bytes) 000389 1745456932.691052 read 0x0263ef thru 0x02d4c5 (0x70d7 bytes) 000393 1745456932.692554 mapwrite 0x03c059 thru 0x03ffff (0x3fa7 bytes) 000394 1745456932.764668 mapwrite 0x02b7c3 thru 0x02d336 (0x1b74 bytes) 000397 1745456932.788623 mapread 0x02f22c thru 0x032bc8 (0x399d bytes) 000398 1745456932.793299 write 0x01e177 thru 0x02cc9c (0xeb26 bytes) 000399 1745456932.796818 MIRROR_EXTEND to 3 mirrors 000401 1745456933.064455 read 0x02540d thru 0x029241 (0x3e35 bytes) 000405 1745456933.068975 mapread 0x01e64b thru 0x01ec38 (0x5ee bytes) 000406 1745456933.073252 write 0x017087 thru 0x022ce5 (0xbc5f bytes) 000407 1745456933.116390 write 0x02bd42 thru 0x032417 (0x66d6 bytes) 000408 1745456933.123725 write 0x01904f thru 0x023b16 (0xaac8 bytes) 000411 1745456933.127265 mapwrite 0x025a42 thru 0x030616 (0xabd5 bytes) 000413 1745456933.166363 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000414 1745456933.269436 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000418 1745456933.345687 read 0x035d0d thru 0x039d87 (0x407b bytes) 000419 1745456933.352721 mapwrite 0x0014e7 thru 0x007db7 (0x68d1 bytes) 000421 1745456933.391393 mapwrite 0x0154b1 thru 0x022f14 (0xda64 bytes) 000422 1745456933.435611 mapwrite 0x0172f2 thru 0x0266af (0xf3be bytes) 000424 1745456933.481504 mapread 0x010150 thru 0x01dc6c (0xdb1d bytes) 000427 1745456933.482899 mapwrite 0x02a6a6 thru 0x02a787 (0xe2 bytes) 000432 1745456933.509136 mapread 0x00fce0 thru 0x01ca46 (0xcd67 bytes) 000434 1745456933.510400 trunc from 0x040000 to 0x002f1f (0x3d0e0 bytes) 000437 1745456933.523797 write 0x01b3a2 thru 0x023537 (0x8196 bytes) 000438 1745456933.524710 write 0x019ed6 thru 0x01b114 (0x123f bytes) 000442 1745456933.530288 write 0x03dc44 thru 0x03ffff (0x23bc bytes) 000443 1745456933.530978 mapwrite 0x026352 thru 0x02cc18 (0x68c7 bytes) 000449 1745456933.567953 mapread 0x004bb7 thru 0x00c5aa (0x79f4 bytes) 000450 1745456933.575158 MIRROR_EXTEND to 2 mirrors 000451 1745456933.780241 mapread 0x03d853 thru 0x03ffff (0x27ad bytes) 000452 1745456933.792031 MIRROR_RESYNC 2 mirrors 000453 1745456933.861121 mapwrite 0x01ce04 thru 0x022ef6 (0x60f3 bytes) 000454 1745456933.956057 mapread 0x036bdf thru 0x03b27f (0x46a1 bytes) 000455 1745456933.961068 mapwrite 0x02fd4b thru 0x0307e8 (0xa9e bytes) 000456 1745456933.993770 write 0x0324a8 thru 0x033ca3 (0x17fc bytes) 000457 1745456933.994642 write 0x03dde9 thru 0x03ffff (0x2217 bytes) 000460 1745456934.001554 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000461 1745456934.097493 write 0x00907b thru 0x0095b1 (0x537 bytes) 000465 1745456934.101013 mapread 0x01541b thru 0x01ddb6 (0x899c bytes) 000466 1745456934.110608 mapwrite 0x006205 thru 0x0087f6 (0x25f2 bytes) 000467 1745456934.150301 mapread 0x00523b thru 0x00b30d (0x60d3 bytes) 000469 1745456934.151512 read 0x002ba2 thru 0x00c4ad (0x990c bytes) 000470 1745456934.157921 MIRROR_EXTEND to 2 mirrors 000475 1745456934.411737 mapwrite 0x01b942 thru 0x026513 (0xabd2 bytes) 000476 1745456934.511811 trunc from 0x040000 to 0x00da72 (0x3258d bytes) 000477 1745456934.519426 write 0x037e09 thru 0x03ffff (0x81f7 bytes) 000478 1745456934.520479 mapwrite 0x019b7d thru 0x029a88 (0xff0c bytes) 000479 1745456934.574365 write 0x02467b thru 0x02f532 (0xaeb8 bytes) 000481 1745456934.575432 trunc from 0x040000 to 0x00dc22 (0x323dd bytes) 000482 1745456934.586320 read 0x00a440 thru 0x00dc22 (0x37e3 bytes) 000483 1745456934.593893 read 0x009051 thru 0x00dc22 (0x4bd2 bytes) 000484 1745456934.599649 trunc from 0x00dc23 to 0x02047d (0x1285b bytes) 000485 1745456934.609252 mapread 0x0072ae thru 0x013d23 (0xca76 bytes) 000486 1745456934.615499 trunc from 0x02047e to 0x01b899 (0x4be4 bytes) 000487 1745456934.624874 mapread 0x0192c8 thru 0x01b899 (0x25d2 bytes) 000488 1745456934.627658 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000489 1745456934.686846 mapwrite 0x030bba thru 0x03de4a (0xd291 bytes) 000492 1745456934.741909 trunc from 0x03de4b to 0x0113ba (0x2ca90 bytes) 000493 1745456934.749080 trunc from 0x0113bb to 0x023b23 (0x12769 bytes) 000495 1745456934.756883 mapwrite 0x037779 thru 0x03ffff (0x8887 bytes) 000499 1745456934.818986 MIRROR_EXTEND to 2 mirrors 000500 1745456935.015090 read 0x03a001 thru 0x03ffff (0x5fff bytes) 000501 1745456935.018713 trunc from 0x040000 to 0x012110 (0x2deef bytes) 000502 1745456935.062201 mapread 0x0044df thru 0x00d9ec (0x950e bytes) 000503 1745456935.077302 mapwrite 0x03b10f thru 0x03ffff (0x4ef1 bytes) 000504 1745456935.124170 mapwrite 0x022be6 thru 0x027e35 (0x5250 bytes) 000506 1745456935.147396 MIRROR_RESYNC 2 mirrors 000508 1745456935.331339 mapwrite 0x005a1f thru 0x015501 (0xfae3 bytes) 000509 1745456935.435546 trunc from 0x040000 to 0x01b12b (0x24ed4 bytes) 000513 1745456935.453332 MIRROR_RESYNC 2 mirrors 000514 1745456935.611467 MIRROR_RESYNC 2 mirrors 000516 1745456935.680748 mapread 0x000f46 thru 0x00ecd6 (0xdd91 bytes) 000517 1745456935.707859 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000518 1745456935.784025 mapwrite 0x01fcbf thru 0x029bc3 (0x9f05 bytes) 000519 1745456935.849172 mapwrite 0x02298a thru 0x02f4c8 (0xcb3f bytes) 000522 1745456935.895033 read 0x02be15 thru 0x02f4c8 (0x36b4 bytes) 000523 1745456935.899607 mapread 0x02bc1a thru 0x02f4c8 (0x38af bytes) 000527 1745456935.901314 read 0x019b19 thru 0x029038 (0xf520 bytes) 000530 1745456935.909624 write 0x012577 thru 0x01b956 (0x93e0 bytes) 000531 1745456935.916051 MIRROR_EXTEND to 2 mirrors 000532 1745456936.193293 mapwrite 0x00e010 thru 0x0194d1 (0xb4c2 bytes) 000534 1745456936.286259 mapwrite 0x01db8e thru 0x02637d (0x87f0 bytes) 000537 1745456936.322435 MIRROR_EXTEND to 3 mirrors 000540 1745456936.511571 MIRROR_RESYNC 3 mirrors 000541 1745456936.745231 write 0x0150da thru 0x01b380 (0x62a7 bytes) 000542 1745456936.795516 mapwrite 0x02bc06 thru 0x037bbc (0xbfb7 bytes) 000543 1745456936.853126 read 0x01aa24 thru 0x02287d (0x7e5a bytes) 000544 1745456936.856571 mapwrite 0x001f94 thru 0x00abb4 (0x8c21 bytes) 000547 1745456936.891805 mapread 0x021133 thru 0x022e72 (0x1d40 bytes) 000548 1745456936.892283 mapread 0x01f4ea thru 0x023ff5 (0x4b0c bytes) 000550 1745456936.896220 trunc from 0x037bbd to 0x0327d1 (0x53eb bytes) 000553 1745456936.906973 mapread 0x008f97 thru 0x015410 (0xc47a bytes) 000554 1745456936.915593 write 0x02dab8 thru 0x0339b4 (0x5efd bytes) 000556 1745456936.921139 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000562 1745456937.019950 read 0x016cad thru 0x01b295 (0x45e9 bytes) 000563 1745456937.026058 read 0x014001 thru 0x01a7c9 (0x67c9 bytes) 000564 1745456937.034695 write 0x030304 thru 0x03e231 (0xdf2e bytes) 000566 1745456937.036148 trunc from 0x03e232 to 0x01748f (0x26da2 bytes) 000567 1745456937.051813 MIRROR_EXTEND to 3 mirrors 000569 1745456937.308211 write 0x015f41 thru 0x016087 (0x147 bytes) 000570 1745456937.344165 write 0x01f2e9 thru 0x01f473 (0x18b bytes) 000572 1745456937.345019 mapread 0x008926 thru 0x00c535 (0x3c10 bytes) 000573 1745456937.349442 write 0x02ddf2 thru 0x03adf1 (0xd000 bytes) 000574 1745456937.350517 mapwrite 0x016aae thru 0x01a659 (0x3bac bytes) 000575 1745456937.383701 write 0x001a35 thru 0x00b248 (0x9814 bytes) 000577 1745456937.387492 trunc from 0x03adf2 to 0x02ca8a (0xe367 bytes) 000578 1745456937.412154 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000579 1745456937.519737 mapwrite 0x03701e thru 0x03ffff (0x8fe2 bytes) 000580 1745456937.577507 mapread 0x010ab3 thru 0x01e69a (0xdbe8 bytes) 000582 1745456937.586507 read 0x012224 thru 0x01efb2 (0xcd8f bytes) 000586 1745456937.588374 mapread 0x018266 thru 0x024ca9 (0xca44 bytes) 000587 1745456937.591155 read 0x012b36 thru 0x021c71 (0xf13c bytes) 000588 1745456937.591599 mapwrite 0x036cc4 thru 0x0385d9 (0x1916 bytes) 000589 1745456937.616914 trunc from 0x040000 to 0x028933 (0x176cc bytes) 000590 1745456937.627048 write 0x03e11c thru 0x03ffff (0x1ee4 bytes) 000591 1745456937.632370 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000592 1745456937.705735 read 0x002a34 thru 0x00e9f0 (0xbfbd bytes) 000595 1745456937.720047 MIRROR_EXTEND to 2 mirrors 000596 1745456938.056626 MIRROR_EXTEND to 3 mirrors 000600 1745456938.379679 mapread 0x00efa0 thru 0x01dec8 (0xef29 bytes) 000601 1745456938.407131 mapread 0x023003 thru 0x029415 (0x6413 bytes) 000602 1745456938.419082 write 0x001877 thru 0x00efb2 (0xd73c bytes) 000603 1745456938.487734 MIRROR_EXTEND to 4 mirrors 000606 1745456938.790018 mapread 0x02e9b8 thru 0x03297d (0x3fc6 bytes) 000607 1745456938.816732 mapread 0x029882 thru 0x0354e7 (0xbc66 bytes) 000608 1745456938.829614 trunc from 0x040000 to 0x032531 (0xdace bytes) 000611 1745456938.880613 trunc from 0x032532 to 0x02e0b0 (0x4481 bytes) 000615 1745456938.897331 mapwrite 0x00b43b thru 0x013076 (0x7c3c bytes) 000617 1745456938.960882 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000620 1745456939.066174 mapwrite 0x003193 thru 0x00439e (0x120c bytes) 000621 1745456939.105053 read 0x000d21 thru 0x002b73 (0x1e53 bytes) 000625 1745456939.109385 mapread 0x020949 thru 0x028aff (0x81b7 bytes) 000626 1745456939.128649 MIRROR_RESYNC 3 mirrors 000627 1745456939.458714 mapread 0x00d08d thru 0x00df1d (0xe91 bytes) 000628 1745456939.464909 write 0x01f3f7 thru 0x02a023 (0xac2d bytes) 000629 1745456939.499919 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000630 1745456939.635157 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000631 1745456939.703390 write 0x025845 thru 0x030ed9 (0xb695 bytes) 000632 1745456939.707931 read 0x006cb4 thru 0x0082d2 (0x161f bytes) 000633 1745456939.711337 mapread 0x02199f thru 0x029ab9 (0x811b bytes) 000634 1745456939.721740 read 0x0297cd thru 0x030385 (0x6bb9 bytes) 000638 1745456939.723148 mapread 0x0027a5 thru 0x005b96 (0x33f2 bytes) 000639 1745456939.731719 trunc from 0x030eda to 0x021c0e (0xf2cb bytes) 000647 1745456939.759735 trunc from 0x021c0f to 0x035439 (0x1382b bytes) 000648 1745456939.769366 write 0x01ba41 thru 0x0221c5 (0x6785 bytes) 000649 1745456939.775879 write 0x00b69c thru 0x010c01 (0x5566 bytes) 000650 1745456939.777037 mapwrite 0x014086 thru 0x01c90d (0x8888 bytes) 000652 1745456939.812494 mapwrite 0x03ef2c thru 0x03ffff (0x10d4 bytes) 000655 1745456939.854218 read 0x000bd3 thru 0x0057f0 (0x4c1e bytes) 000658 1745456939.855232 trunc from 0x040000 to 0x03cc11 (0x33ee bytes) 000659 1745456939.863034 write 0x01cb9c thru 0x020fe9 (0x444e bytes) 000661 1745456939.864686 write 0x02f99d thru 0x037c9d (0x8301 bytes) 000662 1745456939.871630 trunc from 0x03cc12 to 0x015707 (0x2750a bytes) 000663 1745456939.879497 mapwrite 0x01d4a9 thru 0x028124 (0xac7c bytes) 000664 1745456939.949798 trunc from 0x028125 to 0x0017b3 (0x26971 bytes) 000666 1745456939.957362 mapread 0x001395 thru 0x0017b3 (0x41f bytes) 000669 1745456939.958341 mapwrite 0x024cf4 thru 0x02ded9 (0x91e6 bytes) 000670 1745456940.026075 mapread 0x00ed06 thru 0x01bbf3 (0xceee bytes) 000671 1745456940.054980 read 0x011cf0 thru 0x01c0cb (0xa3dc bytes) 000672 1745456940.058499 trunc from 0x02deda to 0x01aef0 (0x12fe9 bytes) 000675 1745456940.066196 read 0x0015ab thru 0x00f1cb (0xdc21 bytes) 000676 1745456940.066623 trunc from 0x01aef1 to 0x036d1c (0x1be2c bytes) 000679 1745456940.075656 mapwrite 0x0257ea thru 0x027592 (0x1da9 bytes) 000680 1745456940.120189 trunc from 0x036d1d to 0x025239 (0x11ae3 bytes) 000681 1745456940.126783 read 0x01ee5e thru 0x025239 (0x63dc bytes) 000685 1745456940.128722 mapread 0x018a74 thru 0x01bff5 (0x3582 bytes) 000686 1745456940.131371 trunc from 0x02523a to 0x025406 (0x1cd bytes) 000687 1745456940.136511 mapread 0x011c63 thru 0x013a75 (0x1e13 bytes) 000691 1745456940.137855 read 0x006b88 thru 0x01398c (0xce05 bytes) 000692 1745456940.138222 mapread 0x021204 thru 0x022af8 (0x18f5 bytes) 000694 1745456940.138891 mapread 0x017bfc thru 0x01d23d (0x5642 bytes) 000696 1745456940.139562 trunc from 0x025407 to 0x001dfa (0x2360c bytes) 000697 1745456940.144048 mapwrite 0x001b38 thru 0x00f779 (0xdc42 bytes) 000702 1745456940.170000 read 0x00adf1 thru 0x00f779 (0x4989 bytes) 000704 1745456940.170897 write 0x021b45 thru 0x02828d (0x6749 bytes) 000705 1745456940.172346 write 0x0129e7 thru 0x015fab (0x35c5 bytes) 000706 1745456940.175613 write 0x02e65e thru 0x036ffc (0x899f bytes) 000707 1745456940.176441 mapwrite 0x0160b1 thru 0x018481 (0x23d1 bytes) 000710 1745456940.199384 mapread 0x011912 thru 0x01316c (0x185b bytes) 000711 1745456940.199918 mapread 0x0096a2 thru 0x00b6cd (0x202c bytes) 000713 1745456940.201027 mapwrite 0x00d080 thru 0x014b4f (0x7ad0 bytes) 000714 1745456940.230242 read 0x022b7d thru 0x029494 (0x6918 bytes) 000715 1745456940.235269 trunc from 0x036ffd to 0x028e02 (0xe1fa bytes) 000717 1745456940.243772 mapread 0x0095a9 thru 0x0165c6 (0xd01e bytes) 000721 1745456940.246550 read 0x00b030 thru 0x011400 (0x63d1 bytes) 000722 1745456940.247076 write 0x004772 thru 0x007526 (0x2db5 bytes) 000724 1745456940.248069 mapwrite 0x03136b thru 0x039513 (0x81a9 bytes) 000725 1745456940.297283 trunc from 0x039514 to 0x02a42a (0xf0e9 bytes) 000726 1745456940.303282 write 0x022aa1 thru 0x0314dc (0xea3c bytes) 000728 1745456940.304746 read 0x02b7db thru 0x0314dc (0x5d02 bytes) 000730 1745456940.307858 MIRROR_EXTEND to 2 mirrors 000731 1745456940.518282 mapread 0x014f4f thru 0x01e9aa (0x9a5c bytes) 000732 1745456940.537561 read 0x01f0d7 thru 0x028fc0 (0x9eea bytes) 000733 1745456940.540098 mapwrite 0x032540 thru 0x03695e (0x441f bytes) 000734 1745456940.629353 read 0x033045 thru 0x03695e (0x391a bytes) 000735 1745456940.629836 write 0x03ab80 thru 0x03c0b4 (0x1535 bytes) 000736 1745456940.630590 trunc from 0x03c0b5 to 0x03d1cc (0x1118 bytes) 000737 1745456940.638949 mapread 0x016010 thru 0x01ed57 (0x8d48 bytes) 000738 1745456940.648962 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000740 1745456940.757916 read 0x0048da thru 0x00bb06 (0x722d bytes) 000741 1745456940.764609 read 0x023efe thru 0x033001 (0xf104 bytes) 000742 1745456940.772828 mapread 0x03bda5 thru 0x03d1cc (0x1428 bytes) 000743 1745456940.773977 read 0x01e551 thru 0x02b490 (0xcf40 bytes) 000744 1745456940.775158 trunc from 0x03d1cd to 0x0108d4 (0x2c8f8 bytes) 000745 1745456940.785548 write 0x00be00 thru 0x01605e (0xa25f bytes) 000746 1745456940.786638 write 0x022a66 thru 0x024795 (0x1d30 bytes) 000748 1745456940.788288 read 0x00e263 thru 0x016bb4 (0x8952 bytes) 000749 1745456940.788691 mapread 0x0189a1 thru 0x024795 (0xbdf5 bytes) 000750 1745456940.793434 trunc from 0x024796 to 0x01db20 (0x6c75 bytes) 000753 1745456940.805024 read 0x01066c thru 0x01db20 (0xd4b5 bytes) 000754 1745456940.805473 write 0x03713c thru 0x03ffff (0x8ec4 bytes) 000756 1745456940.807571 mapwrite 0x03bb9e thru 0x03ffff (0x4462 bytes) 000757 1745456940.851690 write 0x024117 thru 0x02e09d (0x9f87 bytes) 000758 1745456940.860791 mapwrite 0x00a9a1 thru 0x019d26 (0xf386 bytes) 000760 1745456940.896736 mapread 0x01dc6e thru 0x02cb02 (0xee95 bytes) 000762 1745456940.897738 mapwrite 0x013c65 thru 0x01b66d (0x7a09 bytes) 000772 1745456940.932575 write 0x013006 thru 0x019813 (0x680e bytes) 000774 1745456940.933566 trunc from 0x040000 to 0x001b5e (0x3e4a1 bytes) 000776 1745456940.944455 mapwrite 0x038997 thru 0x03f7ea (0x6e54 bytes) 000779 1745456941.026425 mapread 0x030fa8 thru 0x03f7ea (0xe843 bytes) 000780 1745456941.060682 mapread 0x032432 thru 0x03b7c4 (0x9393 bytes) 000784 1745456941.063599 mapwrite 0x019eca thru 0x01fbff (0x5d36 bytes) 000787 1745456941.117962 read 0x02c590 thru 0x037d0b (0xb77c bytes) 000793 1745456941.126390 write 0x019459 thru 0x027071 (0xdc19 bytes) 000797 1745456941.127715 read 0x01116d thru 0x01489d (0x3731 bytes) 000798 1745456941.128209 trunc from 0x03f7eb to 0x024212 (0x1b5d8 bytes) 000799 1745456941.137256 read 0x01ac72 thru 0x024212 (0x95a1 bytes) 000801 1745456941.138163 mapwrite 0x00cbb0 thru 0x018d9e (0xc1ef bytes) 000805 1745456941.172419 write 0x01dd44 thru 0x0250d1 (0x738e bytes) 000806 1745456941.174075 trunc from 0x0250d2 to 0x011606 (0x13acb bytes) 000808 1745456941.186394 mapwrite 0x010877 thru 0x0165e9 (0x5d73 bytes) 000809 1745456941.238345 read 0x002041 thru 0x00cbab (0xab6b bytes) 000811 1745456941.239265 mapread 0x016207 thru 0x0165e9 (0x3e3 bytes) 000812 1745456941.239910 mapread 0x00e2a2 thru 0x00ed5d (0xabc bytes) 000814 1745456941.241031 mapwrite 0x032001 thru 0x0351e6 (0x31e6 bytes) 000816 1745456941.276241 trunc from 0x0351e7 to 0x0260e7 (0xf0ff bytes) 000817 1745456941.284544 read 0x0003be thru 0x003ec0 (0x3b03 bytes) 000821 1745456941.286174 mapread 0x01e2aa thru 0x0260e7 (0x7e3e bytes) 000822 1745456941.286831 read 0x017107 thru 0x0214ff (0xa3f9 bytes) 000823 1745456941.287766 mapread 0x005cdc thru 0x0126d0 (0xc9f5 bytes) 000824 1745456941.288713 write 0x00c716 thru 0x016ed8 (0xa7c3 bytes) 000825 1745456941.289564 mapread 0x01d825 thru 0x0260e7 (0x88c3 bytes) 000829 1745456941.291503 mapwrite 0x036bed thru 0x039ab3 (0x2ec7 bytes) 000831 1745456941.332436 write 0x0298ed thru 0x0345cc (0xace0 bytes) 000833 1745456941.333545 mapread 0x008e79 thru 0x0181ba (0xf342 bytes) 000837 1745456941.335477 read 0x026cda thru 0x02ae02 (0x4129 bytes) 000838 1745456941.335881 write 0x035e7a thru 0x03feb1 (0xa038 bytes) 000839 1745456941.336711 trunc from 0x03feb2 to 0x00558c (0x3a925 bytes) 000840 1745456941.347428 mapwrite 0x004221 thru 0x007b28 (0x3908 bytes) 000841 1745456941.378678 write 0x018695 thru 0x01886a (0x1d6 bytes) 000842 1745456941.379430 mapread 0x00b597 thru 0x01886a (0xd2d4 bytes) 000844 1745456941.383155 read 0x016f7a thru 0x01886a (0x18f1 bytes) 000845 1745456941.383565 mapread 0x009e39 thru 0x00ca88 (0x2c50 bytes) 000847 1745456941.384489 write 0x03eec6 thru 0x03ffff (0x113a bytes) 000849 1745456941.386293 mapwrite 0x013f7f thru 0x020d2d (0xcdaf bytes) 000850 1745456941.417197 mapwrite 0x02f85e thru 0x034d1a (0x54bd bytes) 000852 1745456941.436518 trunc from 0x040000 to 0x004113 (0x3beec bytes) 000853 1745456941.443395 write 0x037a05 thru 0x037fab (0x5a7 bytes) 000854 1745456941.443992 trunc from 0x037fac to 0x028ec3 (0xf0e8 bytes) 000855 1745456941.451003 mapwrite 0x01048b thru 0x01894b (0x84c1 bytes) 000856 1745456941.511876 trunc from 0x028ec4 to 0x029e74 (0xfb1 bytes) 000857 1745456941.521595 write 0x0261a0 thru 0x030666 (0xa4c7 bytes) 000858 1745456941.524305 mapwrite 0x027336 thru 0x0372f1 (0xffbc bytes) 000859 1745456941.566047 trunc from 0x0372f2 to 0x0329fd (0x48f4 bytes) 000861 1745456941.576560 MIRROR_EXTEND to 2 mirrors 000865 1745456941.742384 trunc from 0x0329fe to 0x01f225 (0x137d8 bytes) 000868 1745456941.789725 mapread 0x0197a5 thru 0x01f225 (0x5a81 bytes) 000869 1745456941.804287 MIRROR_EXTEND to 3 mirrors 000870 1745456942.043309 mapread 0x000e8f thru 0x00161a (0x78c bytes) 000872 1745456942.051059 trunc from 0x01f226 to 0x02f060 (0xfe3b bytes) 000873 1745456942.106154 write 0x00b50f thru 0x00c0bb (0xbad bytes) 000874 1745456942.116573 MIRROR_EXTEND to 4 mirrors 000875 1745456942.365426 read 0x00f1c8 thru 0x012df5 (0x3c2e bytes) 000877 1745456942.378213 write 0x00800f thru 0x0148e8 (0xc8da bytes) 000878 1745456942.434507 mapwrite 0x02376f thru 0x028951 (0x51e3 bytes) 000880 1745456942.488261 MIRROR_RESYNC 4 mirrors 000881 1745456942.846034 mapwrite 0x000739 thru 0x0079f6 (0x72be bytes) 000882 1745456942.976800 mapwrite 0x022ebb thru 0x02f390 (0xc4d6 bytes) 000883 1745456943.052909 mapread 0x017f1e thru 0x01b308 (0x33eb bytes) 000884 1745456943.059915 trunc from 0x02f391 to 0x022df7 (0xc599 bytes) 000885 1745456943.068809 read 0x01f739 thru 0x022df7 (0x36bf bytes) 000886 1745456943.069335 mapwrite 0x01aa37 thru 0x02999d (0xef67 bytes) 000887 1745456943.129230 trunc from 0x02999e to 0x0271e4 (0x27b9 bytes) 000888 1745456943.140426 mapread 0x01b7c0 thru 0x01d41e (0x1c5f bytes) 000889 1745456943.142583 write 0x00f7f1 thru 0x014c48 (0x5458 bytes) 000890 1745456943.143433 trunc from 0x0271e5 to 0x030955 (0x9771 bytes) 000891 1745456943.155275 read 0x0195fe thru 0x023818 (0xa21b bytes) 000892 1745456943.156524 mapwrite 0x03a8d0 thru 0x03dacf (0x3200 bytes) 000895 1745456943.217334 read 0x01f0c1 thru 0x024e6f (0x5daf bytes) 000896 1745456943.217804 mapwrite 0x01f7eb thru 0x02d96f (0xe185 bytes) 000897 1745456943.254030 write 0x0050e4 thru 0x01245d (0xd37a bytes) 000899 1745456943.258111 MIRROR_EXTEND to 5 mirrors 000901 1745456943.477738 trunc from 0x03dad0 to 0x028528 (0x155a7 bytes) 000903 1745456943.535116 read 0x01dbab thru 0x02712f (0x9585 bytes) 000905 1745456943.540359 mapread 0x004ab9 thru 0x013341 (0xe889 bytes) 000909 1745456943.556612 mapwrite 0x0307cb thru 0x035a46 (0x527c bytes) 000910 1745456943.599247 mapread 0x02da8f thru 0x035a46 (0x7fb8 bytes) 000913 1745456943.600208 read 0x02d6a6 thru 0x035a46 (0x83a1 bytes) 000915 1745456943.600955 read 0x00700b thru 0x012308 (0xb2fe bytes) 000918 1745456943.606349 MIRROR_RESYNC 5 mirrors 000920 1745456943.948515 read 0x004c20 thru 0x014280 (0xf661 bytes) 000921 1745456943.953062 read 0x00ea5c thru 0x013a59 (0x4ffe bytes) 000922 1745456943.953288 mapread 0x02c78a thru 0x035a46 (0x92bd bytes) 000923 1745456943.958141 mapread 0x02cdc0 thru 0x02fd1d (0x2f5e bytes) 000924 1745456943.961242 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000925 1745456944.074062 write 0x0229df thru 0x02901c (0x663e bytes) 000926 1745456944.119456 read 0x008b83 thru 0x00c96d (0x3deb bytes) 000927 1745456944.122903 read 0x012a50 thru 0x02104b (0xe5fc bytes) 000929 1745456944.127248 write 0x01d070 thru 0x01d751 (0x6e2 bytes) 000931 1745456944.129374 write 0x020b3d thru 0x02b3d6 (0xa89a bytes) 000932 1745456944.130391 mapread 0x003a80 thru 0x00e110 (0xa691 bytes) 000933 1745456944.131059 read 0x0007a8 thru 0x002cbc (0x2515 bytes) 000934 1745456944.133453 MIRROR_EXTEND to 5 mirrors 000935 1745456944.348877 trunc from 0x035a47 to 0x017932 (0x1e114 bytes) 000937 1745456944.390282 mapread 0x00126a thru 0x009180 (0x7f17 bytes) 000938 1745456944.400716 trunc from 0x017933 to 0x00412f (0x13803 bytes) 000940 1745456944.410044 trunc from 0x004130 to 0x00ae9a (0x6d6b bytes) 000941 1745456944.419114 mapwrite 0x01072c thru 0x0132a8 (0x2b7d bytes) 000944 1745456944.466393 write 0x010eb4 thru 0x0167ca (0x5917 bytes) 000945 1745456944.467174 read 0x00deb2 thru 0x0133ba (0x5509 bytes) 000946 1745456944.469193 MIRROR_RESYNC 5 mirrors 000949 1745456944.825827 write 0x020142 thru 0x0289dc (0x889b bytes) 000951 1745456944.874215 read 0x001696 thru 0x00e69e (0xd009 bytes) 000952 1745456944.882440 trunc from 0x0289dd to 0x00a290 (0x1e74c bytes) 000953 1745456944.899240 read 0x007285 thru 0x00a290 (0x300c bytes) 000954 1745456944.899700 mapwrite 0x018ade thru 0x01adb0 (0x22d3 bytes) 000955 1745456944.952541 trunc from 0x01adb1 to 0x032729 (0x17979 bytes) 000956 1745456944.964945 write 0x01cf73 thru 0x024aef (0x7b7d bytes) 000957 1745456944.972283 read 0x01a14c thru 0x01f2ee (0x51a3 bytes) 000959 1745456944.972997 read 0x02d626 thru 0x032729 (0x5104 bytes) 000960 1745456944.973291 write 0x0099d0 thru 0x00cb42 (0x3173 bytes) 000961 1745456944.978149 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 000964 1745456945.114629 MIRROR_EXTEND to 5 mirrors 000965 1745456945.479448 mapread 0x013a7d thru 0x020e6d (0xd3f1 bytes) 000966 1745456945.508133 mapread 0x000b9c thru 0x0091a3 (0x8608 bytes) 000968 1745456945.556382 write 0x005b0d thru 0x007fcd (0x24c1 bytes) 000970 1745456945.607673 write 0x00dbe7 thru 0x01848c (0xa8a6 bytes) 000971 1745456945.614740 read 0x032158 thru 0x032729 (0x5d2 bytes) 000972 1745456945.619420 mapwrite 0x018533 thru 0x01bcba (0x3788 bytes) 000974 1745456945.676436 read 0x0023cc thru 0x010f11 (0xeb46 bytes) 000976 1745456945.677393 write 0x02fe96 thru 0x0363be (0x6529 bytes) 000977 1745456945.687715 read 0x0160c1 thru 0x01e50d (0x844d bytes) 000978 1745456945.688515 mapwrite 0x02db27 thru 0x037630 (0x9b0a bytes) 000979 1745456945.757000 mapwrite 0x018171 thru 0x020f9a (0x8e2a bytes) 000981 1745456945.814503 read 0x03230d thru 0x037630 (0x5324 bytes) 000982 1745456945.815026 read 0x02ff2d thru 0x030f5f (0x1033 bytes) 000985 1745456945.816124 trunc from 0x037631 to 0x03a24f (0x2c1f bytes) 000986 1745456945.827387 mapwrite 0x0295c1 thru 0x038bec (0xf62c bytes) 000990 1745456945.865264 trunc from 0x03a250 to 0x035ff1 (0x425e bytes) 000992 1745456945.875022 mapwrite 0x02495a thru 0x02ef1f (0xa5c6 bytes) 000994 1745456945.927374 trunc from 0x035ff2 to 0x02fbf7 (0x63fa bytes) 000997 1745456945.948469 trunc from 0x02fbf8 to 0x014637 (0x1b5c0 bytes) 000998 1745456945.967493 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ef:0x0]: Invalid argument (22) 001000 1745456946.090062 read 0x000e5e thru 0x002bf6 (0x1d99 bytes) All operations completed A-OK! PASS 70a (22s) == sanity-flr test 71: check mirror extend parent fid ==== 21:09:07 (1745456947) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 21:09:08 (1745456948) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre2 Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6240992 .. Extending file size to 5130816 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8013472 .. Extending file size to 6574944 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6994368 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8832192 .. Extending file size to 8925376 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8392896 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8854240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8950688 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9037632 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9068544 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9317216 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 26537 26538 26540 26541 26542 192.168.203.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre2 (opts:) 192.168.203.106@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:10:14 (1745457014) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre2 Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5452896 .. Extending file size to 6664576 .. done Extending file size to 5494656 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7897024 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7279104 .. Extending file size to 8961248 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7770432 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8664896 .. Extending file size to 9171744 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9271168 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9206208 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9234400 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9288512 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 30271 30272 30274 30277 30278 192.168.203.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre2 (opts:) 192.168.203.106@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:11:18 (1745457078) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4608 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4608 2203776 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0345369 s, 60.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.11899 s, 17.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:11:23 (1745457083) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0309153 s, 67.8 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 21:11:25 (1745457085) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x52a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f1:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x5eb:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:11:27 (1745457087) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0405414 s, 51.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ed:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:11:29 (1745457089) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0511202 s, 41.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ef:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:11:31 (1745457091) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0297202 s, 70.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f1:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:11:54 (1745457114) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg306-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0263768 s, 39.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f7:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:12:38 (1745457158) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg306-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0324155 s, 64.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:13:23 (1745457203) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg306-server mds-ost sync done. Creating new pool test_204f oleg306-server: Pool lustre.test_204f created Adding targets to pool oleg306-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg306-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.02999 s, 69.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg306-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg306-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg306-server: Pool lustre.test_204f destroyed PASS 204f (50s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:14:13 (1745457253) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:14:15 (1745457255) PASS 205b (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:14:16 (1745457256) oleg306-server: Pool lustre.test_206 created Adding targets to pool oleg306-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg306-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg306-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg306-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg306-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:14:29 (1745457269) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0181009 s, 57.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103346 s, 101 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xf31c4e8 Mirror 3: 0xf31c4e8 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:14:31 (1745457271) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:14:32 (1745457272) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (0s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:14:32 (1745457272) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0242622 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1b00000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1380000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1380000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1380000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xb00000 Read this (4 bytes): BBBB 192.168.203.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:14:46 (1745457286) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0024216 s, 3.4 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:14:47 (1745457287) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00814311 s, 129 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x54f lmm_fid: [0x200000404:0x54f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x54f lmm_fid: [0x200000404:0x54f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x603:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835279 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-23 21:14:48.000000000 -0400 Modify: 2025-04-23 21:14:48.000000000 -0400 Change: 2025-04-23 21:14:48.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:14:49 (1745457289) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:14:50 (1745457290) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0948157 s, 432 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0131687 s, 311 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 23 21:14 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x551:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 23 21:14 /mnt/lustre/f211.sanity-flr PASS 211 (1s) == sanity-flr test complete, duration 746 sec ============ 21:14:51 (1745457291) === sanity-flr: start cleanup 21:14:52 (1745457292) === === sanity-flr: finish cleanup 21:14:53 (1745457293) ===