-----============= acceptance-small: sanity-flr ============----- Thu Mar 13 16:30:46 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg110-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 16:30:59 (1741897859) === oleg110-client.virtnet: executing check_config_client /mnt/lustre oleg110-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg110-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6d0e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6d0e800.idle_timeout=debug disable quota as required oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 16:31:16 (1741897876) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7035] == sanity-flr test 0a: lfs mirror create with -N option == 16:31:17 (1741897877) PASS 0a (11s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:31:28 (1741897888) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:31:30 (1741897890) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:31:32 (1741897892) PASS 0d (15s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:31:47 (1741897907) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:31:49 (1741897909) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 16:31:51 (1741897911) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6d0e800.stats=clear osc.lustre-OST0001-osc-ffff8800b6d0e800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:31:55 (1741897915) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:32:01 (1741897921) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:32:06 (1741897926) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:32:08 (1741897928) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 16:32:14 (1741897934) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:32:18 (1741897938) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:32:20 (1741897940) lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:32:25 (1741897945) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:32:30 (1741897950) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:32:35 (1741897955) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 1.06257 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.103988 s, 10.1 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:32:40 (1741897960) 19+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.195874 s, 102 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:32:44 (1741897964) Stopping /mnt/lustre-ost1 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6d0e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6d0e800.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg110-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6d0e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6d0e800.ost_server_uuid in DISCONN state after 2 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg110-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (40s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:33:24 (1741898004) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.300646 s, 20.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6d0e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6d0e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg110-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (21s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:33:46 (1741898026) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6d0e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6d0e800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg110-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6d0e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6d0e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg110-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (34s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:34:19 (1741898059) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.14154 s, 45.9 MB/s -rw-r--r-- 1 root root 52428800 Mar 13 16:34 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.461s user 0m0.000s sys 0m0.164s Stopping /mnt/lustre-ost1 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6d0e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6d0e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.503s user 0m0.000s sys 0m0.167s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg110-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg110-server oleg110-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6d0e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6d0e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.614s user 0m0.000s sys 0m0.174s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg110-server: oleg110-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg110-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg110-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6d0e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6d0e800.ost_server_uuid in FULL state after 0 sec PASS 33b (35s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:34:55 (1741898095) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:34:56 (1741898096) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:34:58 (1741898098) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 16:35:00 (1741898100) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 16:35:03 (1741898103) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.356905 s, 35.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:35:10 (1741898110) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0343376 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0321138 s, 32.7 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:35:14 (1741898114) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:35:19 (1741898119) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0017214 s, 595 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00105612 s, 970 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00247595 s, 414 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000889477 s, 1.2 MB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 16:35:24 (1741898124) create test files with size 15728713 .. 983876cd9edbecc6fd9af490859a016e - 5913eff43d651c2123adb0c9c02dfe0e - 079b009d37f40b06b9b59c3a742af13a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre PASS 37 (12s) == sanity-flr test 38: resync ============================ 16:35:36 (1741898136) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:35:42 (1741898142) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:35:45 (1741898145) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0652949 s, 16.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0243738 s, 43.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 16:35:49 (1741898149) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0638921 s, 32.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.20509 s, 20.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.135352 s, 15.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 16:35:55 (1741898155) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 16:35:57 (1741898157) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:35:59 (1741898159) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0218446 s, 48.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 16:36:03 (1741898163) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:36:04 (1741898164) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:36:09 (1741898169) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.135036 s, 77.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:36:24 (1741898184) llite.lustre-ffff8800aa197800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0965214 s, 109 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800b61bf800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:36:29 (1741898189) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:36:44 (1741898204) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (14s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:36:58 (1741898218) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 16:37:00 (1741898220) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.126611 s, 24.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0999421 s, 31.5 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 712be564cbe5c18225112e6a1ed9ff9e - mirror 1 checksum: 0f82273499068854529dd563d33c2d3b - mirror 2 checksum: 712be564cbe5c18225112e6a1ed9ff9e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Mar 13 16:37:01 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Mar 13 16:37:01 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 712be564cbe5c18225112e6a1ed9ff9e - mirror 1 checksum: 0f82273499068854529dd563d33c2d3b - mirror 2 checksum: 0f82273499068854529dd563d33c2d3b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Mar 13 16:37:01 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Mar 13 16:37:01 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (5s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:37:05 (1741898225) Failing mds1 on oleg110-server Stopping /mnt/lustre-mds1 (opts:) on oleg110-server 16:37:09 (1741898229) shut down facet: mds1 facet_host: oleg110-server facet_failover_host: oleg110-server Failover mds1 to oleg110-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-MDT0000 16:37:24 (1741898244) targets are mounted 16:37:24 (1741898244) facet_failover done oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0204973 s, 51.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:37:32 (1741898252) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:37:34 (1741898254) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:37:35 (1741898255) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00479865 s, 3.4 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:37:39 (1741898259) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:37:41 (1741898261) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:37:43 (1741898263) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (10s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:37:53 (1741898273) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:38:13 (1741898293) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:38:31 (1741898311) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:38:50 (1741898330) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 16:38:53 (1741898333) mirror create pid 22613 mirror split pid 22615 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 22613 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 22615 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 16:39:56 (1741898396) Seed set to 8397 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1741898397.216679 write 0x026635 thru 0x02ffdf (0x99ab bytes) 000003 1741898397.224996 read 0x022c90 thru 0x02d355 (0xa6c6 bytes) 000004 1741898397.228808 mapwrite 0x03805f thru 0x03f9e2 (0x7984 bytes) 000005 1741898397.327757 read 0x016d12 thru 0x01fc17 (0x8f06 bytes) 000006 1741898397.342458 mapread 0x00439e thru 0x008cf4 (0x4957 bytes) 000008 1741898397.344615 mapread 0x001621 thru 0x007691 (0x6071 bytes) 000011 1741898397.346205 mapread 0x02b2b4 thru 0x032cea (0x7a37 bytes) truncating to largest ever: 0x3317f 000015 1741898397.349574 trunc from 0x03f9e3 to 0x03317e (0xc864 bytes) 000016 1741898397.362013 write 0x0315e5 thru 0x03c1ad (0xabc9 bytes) 000017 1741898397.366509 mapwrite 0x02289f thru 0x02ed68 (0xc4ca bytes) 000019 1741898397.434476 mapwrite 0x006e3e thru 0x00df96 (0x7159 bytes) 000020 1741898397.496006 write 0x027419 thru 0x029cef (0x28d7 bytes) 000022 1741898397.497333 read 0x03bbe7 thru 0x03c1ad (0x5c7 bytes) 000023 1741898397.497781 read 0x0048a3 thru 0x00f252 (0xa9b0 bytes) 000024 1741898397.498177 write 0x02581c thru 0x032fe7 (0xd7cc bytes) 000025 1741898397.499525 read 0x01d0c7 thru 0x02a79e (0xd6d8 bytes) 000026 1741898397.499959 trunc from 0x03c1ae to 0x029c47 (0x12566 bytes) 000032 1741898397.518215 read 0x0239e8 thru 0x029c47 (0x6260 bytes) 000036 1741898397.522408 trunc from 0x029c48 to 0x00d379 (0x1c8ce bytes) 000041 1741898397.540406 mapwrite 0x01deac thru 0x01ef91 (0x10e6 bytes) 000042 1741898397.612984 mapwrite 0x00cde0 thru 0x00d6fe (0x91f bytes) 000043 1741898397.660980 mapread 0x00b93f thru 0x018422 (0xcae4 bytes) 000044 1741898397.661748 write 0x02e037 thru 0x03039d (0x2367 bytes) 000046 1741898397.663817 read 0x02866d thru 0x02bdf3 (0x3787 bytes) 000048 1741898397.671139 mapread 0x004381 thru 0x0119df (0xd65f bytes) 000049 1741898397.672679 trunc from 0x03039e to 0x02c7c6 (0x3bd7 bytes) 000051 1741898397.684724 write 0x00c907 thru 0x01b445 (0xeb3f bytes) 000054 1741898397.688631 read 0x01fc55 thru 0x02018b (0x537 bytes) 000056 1741898397.689198 mapread 0x015f00 thru 0x021743 (0xb844 bytes) 000062 1741898397.692986 mapread 0x006291 thru 0x00f7d1 (0x9541 bytes) 000063 1741898397.693915 mapwrite 0x00c2b5 thru 0x0145d4 (0x8320 bytes) 000069 1741898397.748367 write 0x003fcd thru 0x00ee5b (0xae8f bytes) truncating to largest ever: 0x39e29 000070 1741898397.749398 trunc from 0x02c7c7 to 0x039e28 (0xd662 bytes) 000073 1741898397.762298 trunc from 0x039e29 to 0x018d85 (0x210a3 bytes) 000074 1741898397.770657 mapread 0x0178b9 thru 0x018d85 (0x14cd bytes) 000075 1741898397.772125 mapread 0x0114f6 thru 0x01341f (0x1f2a bytes) 000077 1741898397.773504 write 0x00ba98 thru 0x0126c4 (0x6c2d bytes) 000079 1741898397.776167 mapwrite 0x00ae6f thru 0x01a373 (0xf505 bytes) 000080 1741898397.848863 mapread 0x00df45 thru 0x0177ed (0x98a9 bytes) 000082 1741898397.850170 mapread 0x011f22 thru 0x017daa (0x5e89 bytes) 000084 1741898397.851370 mapwrite 0x03e6ca thru 0x03ffff (0x1936 bytes) 000087 1741898397.929515 write 0x03f3b3 thru 0x03ffff (0xc4d bytes) 000088 1741898397.932986 MIRROR_EXTEND to 2 mirrors 000090 1741898398.302673 mapread 0x01af30 thru 0x01f9c7 (0x4a98 bytes) 000091 1741898398.346783 MIRROR_RESYNC 2 mirrors 000092 1741898398.489426 mapwrite 0x011293 thru 0x019d16 (0x8a84 bytes) 000094 1741898398.634764 trunc from 0x040000 to 0x00dab8 (0x32547 bytes) 000096 1741898398.654167 MIRROR_EXTEND to 3 mirrors 000098 1741898398.940928 write 0x01fc9e thru 0x026274 (0x65d7 bytes) 000099 1741898399.006860 mapwrite 0x02b503 thru 0x032d30 (0x782e bytes) 000101 1741898399.111086 mapwrite 0x029671 thru 0x0308ff (0x728f bytes) 000102 1741898399.167307 write 0x0333df thru 0x03838e (0x4fb0 bytes) 000104 1741898399.169443 trunc from 0x03838f to 0x0164c9 (0x21ec5 bytes) 000105 1741898399.181466 trunc from 0x0164ca to 0x020b6f (0xa6a6 bytes) 000109 1741898399.192706 mapread 0x0020ca thru 0x00652e (0x4465 bytes) 000110 1741898399.201288 trunc from 0x020b70 to 0x01e620 (0x254f bytes) 000111 1741898399.214004 mapwrite 0x01a3de thru 0x0205bd (0x61e0 bytes) 000112 1741898399.295823 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000113 1741898399.418567 mapwrite 0x004696 thru 0x011d50 (0xd6bb bytes) 000116 1741898399.487197 mapread 0x006f28 thru 0x00a253 (0x332c bytes) 000117 1741898399.488106 trunc from 0x0205be to 0x032f8a (0x129cd bytes) 000119 1741898399.502750 read 0x00b742 thru 0x0196ad (0xdf6c bytes) 000120 1741898399.503593 write 0x01edc5 thru 0x01f6a6 (0x8e2 bytes) 000121 1741898399.511624 trunc from 0x032f8b to 0x01bc21 (0x17369 bytes) 000124 1741898399.525508 write 0x01571a thru 0x01ddb2 (0x8699 bytes) 000128 1741898399.528675 write 0x021efd thru 0x02882a (0x692e bytes) 000129 1741898399.532827 mapread 0x0197a5 thru 0x022c84 (0x94e0 bytes) 000132 1741898399.538634 read 0x022013 thru 0x02882a (0x6818 bytes) 000133 1741898399.542025 write 0x00aa8c thru 0x0134e1 (0x8a56 bytes) 000134 1741898399.544594 trunc from 0x02882b to 0x013289 (0x155a1 bytes) 000135 1741898399.555441 mapread 0x001a22 thru 0x00e7e2 (0xcdc1 bytes) 000137 1741898399.557034 read 0x004a8f thru 0x013289 (0xe7fb bytes) 000139 1741898399.558520 mapwrite 0x032126 thru 0x03edfc (0xccd7 bytes) 000140 1741898399.636915 write 0x01764f thru 0x01af0f (0x38c1 bytes) 000141 1741898399.638058 write 0x0226e5 thru 0x026158 (0x3a74 bytes) 000143 1741898399.644522 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000145 1741898399.753694 trunc from 0x03edfd to 0x00006e (0x3ed8e bytes) 000150 1741898399.763654 write 0x01e4be thru 0x02de12 (0xf955 bytes) 000151 1741898399.764702 mapread 0x00ed3f thru 0x015504 (0x67c6 bytes) 000152 1741898399.773309 write 0x03980f thru 0x03c4ad (0x2c9f bytes) 000154 1741898399.774087 write 0x0342ba thru 0x03ffff (0xbd46 bytes) 000158 1741898399.782023 trunc from 0x040000 to 0x025182 (0x1ae7d bytes) 000160 1741898399.793605 read 0x01e13f thru 0x020dc1 (0x2c83 bytes) 000161 1741898399.794342 mapwrite 0x00bc52 thru 0x0166d1 (0xaa80 bytes) 000162 1741898399.845977 read 0x01dcec thru 0x01eb57 (0xe6c bytes) 000163 1741898399.850724 write 0x024341 thru 0x02dd3d (0x99fd bytes) 000166 1741898399.853008 mapread 0x0189f9 thru 0x01b416 (0x2a1e bytes) 000167 1741898399.855938 MIRROR_EXTEND to 2 mirrors 000171 1741898400.079399 trunc from 0x02dd3e to 0x017507 (0x16836 bytes) 000174 1741898400.136546 trunc from 0x017508 to 0x027749 (0x10242 bytes) 000176 1741898400.150372 write 0x024aff thru 0x02f367 (0xa869 bytes) 000178 1741898400.155631 write 0x00d124 thru 0x01247d (0x535a bytes) 000179 1741898400.166206 mapread 0x0284e7 thru 0x02f367 (0x6e81 bytes) 000180 1741898400.167269 mapwrite 0x0354ee thru 0x039178 (0x3c8b bytes) truncating to largest ever: 0x3ac04 000181 1741898400.228678 trunc from 0x039179 to 0x03ac03 (0x1a8b bytes) 000183 1741898400.242586 read 0x00242d thru 0x0091f4 (0x6dc8 bytes) 000184 1741898400.253536 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000186 1741898400.321305 mapwrite 0x00b7c7 thru 0x00e9af (0x31e9 bytes) 000189 1741898400.368373 trunc from 0x03ac04 to 0x039e45 (0xdbe bytes) 000191 1741898400.380139 read 0x000444 thru 0x0061c9 (0x5d86 bytes) 000194 1741898400.387078 read 0x015958 thru 0x023930 (0xdfd9 bytes) 000196 1741898400.399011 read 0x0075de thru 0x00d165 (0x5b88 bytes) 000198 1741898400.400322 trunc from 0x039e46 to 0x00211c (0x37d29 bytes) 000201 1741898400.409129 write 0x039071 thru 0x03eb53 (0x5ae3 bytes) 000202 1741898400.410624 mapread 0x039310 thru 0x03d111 (0x3e02 bytes) 000203 1741898400.413070 MIRROR_EXTEND to 2 mirrors 000204 1741898400.773469 read 0x00efd6 thru 0x016676 (0x76a1 bytes) 000205 1741898400.786625 mapread 0x006c27 thru 0x012577 (0xb951 bytes) 000207 1741898400.820054 trunc from 0x03eb54 to 0x02333a (0x1b819 bytes) 000208 1741898400.881632 mapwrite 0x03b335 thru 0x03ffff (0x4ccb bytes) 000210 1741898400.945913 MIRROR_EXTEND to 3 mirrors 000212 1741898401.252991 MIRROR_RESYNC 3 mirrors 000214 1741898401.585160 mapread 0x0203c6 thru 0x02f55d (0xf198 bytes) 000216 1741898401.604940 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000218 1741898401.718891 mapread 0x039664 thru 0x03e9ea (0x5387 bytes) 000219 1741898401.733463 mapwrite 0x0331b0 thru 0x03ffff (0xce50 bytes) 000220 1741898401.861539 write 0x016baa thru 0x025005 (0xe45c bytes) 000221 1741898401.870606 mapread 0x019c3a thru 0x028e1d (0xf1e4 bytes) 000223 1741898401.880974 mapread 0x012a9c thru 0x020081 (0xd5e6 bytes) 000229 1741898401.883723 mapread 0x028313 thru 0x035f4e (0xdc3c bytes) 000230 1741898401.884261 read 0x006575 thru 0x016055 (0xfae1 bytes) 000231 1741898401.896402 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000232 1741898402.017108 trunc from 0x040000 to 0x01c652 (0x239ad bytes) 000235 1741898402.033405 read 0x00d80a thru 0x00fb8c (0x2383 bytes) 000240 1741898402.037557 write 0x03abef thru 0x03ffff (0x5411 bytes) 000241 1741898402.038914 write 0x02c289 thru 0x037222 (0xaf9a bytes) 000243 1741898402.046782 write 0x038a09 thru 0x03ffff (0x75f7 bytes) truncating to largest ever: 0x3df7b 000244 1741898402.052806 trunc from 0x040000 to 0x03df7a (0x2085 bytes) 000247 1741898402.064041 trunc from 0x03df7b to 0x01bfa8 (0x21fd2 bytes) 000248 1741898402.076077 mapwrite 0x007859 thru 0x015a9a (0xe242 bytes) 000250 1741898402.161696 mapwrite 0x03325a thru 0x03d38a (0xa131 bytes) 000252 1741898402.258418 read 0x005e38 thru 0x00dafc (0x7cc5 bytes) 000253 1741898402.259627 mapread 0x018ba0 thru 0x020df9 (0x825a bytes) 000254 1741898402.268058 read 0x01f0ba thru 0x02966f (0xa5b6 bytes) 000258 1741898402.270784 mapread 0x026703 thru 0x033589 (0xce87 bytes) 000261 1741898402.273690 mapwrite 0x004fc2 thru 0x010ea6 (0xbee5 bytes) 000262 1741898402.322611 write 0x01a94c thru 0x029e6c (0xf521 bytes) 000263 1741898402.323486 read 0x022d79 thru 0x028c0a (0x5e92 bytes) 000265 1741898402.324380 trunc from 0x03d38b to 0x008a90 (0x348fa bytes) 000268 1741898402.333359 trunc from 0x008a91 to 0x0288d3 (0x1fe43 bytes) 000269 1741898402.342511 read 0x0224d0 thru 0x0288d3 (0x6404 bytes) 000270 1741898402.351744 mapwrite 0x02439d thru 0x02b656 (0x72ba bytes) 000273 1741898402.428261 write 0x03bedd thru 0x03df3b (0x205f bytes) 000275 1741898402.430383 write 0x00dbbd thru 0x01233d (0x4781 bytes) 000280 1741898402.432717 mapread 0x02673b thru 0x0321ad (0xba73 bytes) 000281 1741898402.442834 read 0x00e0de thru 0x01d2ba (0xf1dd bytes) 000284 1741898402.444160 read 0x006c01 thru 0x00ade6 (0x41e6 bytes) 000289 1741898402.450278 mapwrite 0x00ffb5 thru 0x01a0bc (0xa108 bytes) 000291 1741898402.495621 mapwrite 0x038d52 thru 0x03ffff (0x72ae bytes) 000293 1741898402.563600 read 0x015c63 thru 0x0251f7 (0xf595 bytes) 000294 1741898402.564614 trunc from 0x040000 to 0x025cda (0x1a325 bytes) 000295 1741898402.588067 read 0x01ae32 thru 0x0202a7 (0x5476 bytes) 000296 1741898402.588542 trunc from 0x025cdb to 0x026bbb (0xee1 bytes) 000297 1741898402.599585 write 0x02ac85 thru 0x02e0f3 (0x346f bytes) 000298 1741898402.600881 mapread 0x02d9a8 thru 0x02e0f3 (0x74c bytes) 000299 1741898402.602630 mapread 0x014937 thru 0x01930d (0x49d7 bytes) 000301 1741898402.604144 read 0x02442f thru 0x02b725 (0x72f7 bytes) 000302 1741898402.612333 mapread 0x0128b8 thru 0x01ab50 (0x8299 bytes) 000303 1741898402.613138 write 0x015f78 thru 0x022735 (0xc7be bytes) 000305 1741898402.615412 mapwrite 0x02d2c9 thru 0x035fff (0x8d37 bytes) 000307 1741898402.700269 trunc from 0x036000 to 0x00e575 (0x27a8a bytes) 000308 1741898402.721638 MIRROR_EXTEND to 2 mirrors 000309 1741898402.990318 write 0x037923 thru 0x03ffff (0x86dd bytes) 000310 1741898403.041558 mapread 0x03c645 thru 0x03f56c (0x2f28 bytes) 000311 1741898403.043369 trunc from 0x040000 to 0x023c35 (0x1c3ca bytes) 000313 1741898403.068320 write 0x002cc5 thru 0x00fe17 (0xd153 bytes) 000315 1741898403.080865 read 0x009e51 thru 0x00cb6e (0x2d1e bytes) 000316 1741898403.081449 mapread 0x011a18 thru 0x013aa0 (0x2089 bytes) 000317 1741898403.099068 MIRROR_EXTEND to 3 mirrors 000318 1741898403.394883 trunc from 0x023c36 to 0x012609 (0x1162c bytes) 000319 1741898403.509375 mapread 0x00ac19 thru 0x012609 (0x79f1 bytes) 000322 1741898403.536460 write 0x021e96 thru 0x0259c0 (0x3b2b bytes) 000323 1741898403.538548 trunc from 0x0259c1 to 0x03aa2b (0x1506b bytes) 000324 1741898403.567184 read 0x038b21 thru 0x03aa2b (0x1f0b bytes) 000325 1741898403.572949 mapread 0x02a063 thru 0x032d44 (0x8ce2 bytes) 000326 1741898403.596442 trunc from 0x03aa2c to 0x0022bb (0x38770 bytes) 000327 1741898403.615397 write 0x00a498 thru 0x013c13 (0x977c bytes) 000328 1741898403.632078 MIRROR_RESYNC 3 mirrors 000331 1741898404.061001 write 0x011619 thru 0x011c9d (0x685 bytes) 000333 1741898404.127134 mapread 0x011e2a thru 0x013c13 (0x1dea bytes) 000334 1741898404.146753 trunc from 0x013c14 to 0x00bcaf (0x7f64 bytes) 000335 1741898404.174671 trunc from 0x00bcb0 to 0x0000c0 (0xbbef bytes) 000336 1741898404.185666 write 0x02474f thru 0x032ce4 (0xe596 bytes) 000337 1741898404.191228 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000339 1741898404.344035 mapread 0x01756e thru 0x024ec7 (0xd95a bytes) 000340 1741898404.377729 write 0x028d15 thru 0x031332 (0x861e bytes) 000341 1741898404.383573 read 0x01cadd thru 0x02394c (0x6e70 bytes) 000343 1741898404.384383 read 0x00fcf1 thru 0x01e612 (0xe922 bytes) 000345 1741898404.403164 read 0x00d58d thru 0x010a4d (0x34c1 bytes) 000347 1741898404.404465 trunc from 0x032ce5 to 0x02e633 (0x46b1 bytes) 000350 1741898404.428019 read 0x017dd3 thru 0x026673 (0xe8a1 bytes) 000353 1741898404.439757 MIRROR_RESYNC 2 mirrors 000359 1741898404.822340 write 0x00cbb7 thru 0x017e8d (0xb2d7 bytes) 000361 1741898404.868780 write 0x03bcce thru 0x03ffff (0x4332 bytes) 000363 1741898404.878897 MIRROR_EXTEND to 3 mirrors 000364 1741898405.339732 read 0x0210b8 thru 0x0254fe (0x4447 bytes) 000367 1741898405.348829 trunc from 0x040000 to 0x00ca7a (0x33585 bytes) 000368 1741898405.415337 write 0x0089e8 thru 0x012fba (0xa5d3 bytes) 000369 1741898405.423464 write 0x00441b thru 0x013689 (0xf26f bytes) 000373 1741898405.434404 read 0x0072d7 thru 0x00ae8b (0x3bb5 bytes) 000374 1741898405.435055 write 0x0357f3 thru 0x03bc4e (0x645c bytes) 000376 1741898405.436698 trunc from 0x03bc4f to 0x01c8c4 (0x1f38a bytes) 000379 1741898405.465070 mapread 0x0119bc thru 0x012d81 (0x13c6 bytes) 000380 1741898405.466480 mapwrite 0x01d963 thru 0x021a5e (0x40fc bytes) 000381 1741898405.560362 mapwrite 0x0114d7 thru 0x01b77c (0xa2a6 bytes) 000383 1741898405.627494 write 0x01c982 thru 0x0231e0 (0x685f bytes) 000384 1741898405.630709 MIRROR_EXTEND to 4 mirrors 000388 1741898406.011582 mapread 0x01c9ff thru 0x01cf54 (0x556 bytes) 000389 1741898406.029393 read 0x0137bf thru 0x01ac4b (0x748d bytes) 000392 1741898406.051807 MIRROR_EXTEND to 5 mirrors 000393 1741898406.421559 trunc from 0x0231e1 to 0x03d617 (0x1a437 bytes) 000394 1741898406.490099 read 0x0161b4 thru 0x02132c (0xb179 bytes) 000395 1741898406.496730 write 0x026ce5 thru 0x0290df (0x23fb bytes) 000397 1741898406.503602 mapwrite 0x02705e thru 0x0301a0 (0x9143 bytes) 000398 1741898406.570841 MIRROR_RESYNC 5 mirrors 000399 1741898407.079894 mapread 0x01475b thru 0x018c9a (0x4540 bytes) 000400 1741898407.094616 read 0x017a6a thru 0x0208ba (0x8e51 bytes) 000402 1741898407.102446 MIRROR_RESYNC 5 mirrors 000403 1741898407.239349 mapread 0x0070f9 thru 0x00e28b (0x7193 bytes) 000404 1741898407.268629 mapwrite 0x03509e thru 0x03aaf0 (0x5a53 bytes) 000405 1741898407.439710 trunc from 0x03d618 to 0x03502a (0x85ed bytes) 000407 1741898407.474655 mapwrite 0x0099a1 thru 0x00a60b (0xc6b bytes) 000409 1741898407.548942 mapwrite 0x00a512 thru 0x00df52 (0x3a41 bytes) 000413 1741898407.592328 mapread 0x006322 thru 0x0118c6 (0xb5a5 bytes) 000414 1741898407.593021 read 0x0209d9 thru 0x02d6ea (0xcd12 bytes) 000416 1741898407.602824 mapwrite 0x01c8d3 thru 0x026fe3 (0xa711 bytes) 000418 1741898407.678060 trunc from 0x03502b to 0x023cd2 (0x11358 bytes) 000419 1741898407.696512 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000420 1741898407.861932 write 0x01d0a4 thru 0x02ad1e (0xdc7b bytes) 000421 1741898407.868027 write 0x01552b thru 0x0164f0 (0xfc6 bytes) 000422 1741898407.877973 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000423 1741898408.074659 read 0x01797d thru 0x01ac4c (0x32d0 bytes) 000424 1741898408.079881 trunc from 0x02ad1f to 0x01049b (0x1a883 bytes) 000426 1741898408.091729 read 0x00f5e6 thru 0x01049b (0xeb6 bytes) 000427 1741898408.109493 MIRROR_RESYNC 3 mirrors 000429 1741898408.483554 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000430 1741898408.661352 trunc from 0x01049c to 0x032486 (0x21feb bytes) 000431 1741898408.718933 mapread 0x0192ca thru 0x0229dd (0x9714 bytes) 000432 1741898408.739628 mapread 0x00cf7c thru 0x010a2f (0x3ab4 bytes) 000434 1741898408.749653 trunc from 0x032487 to 0x011432 (0x21054 bytes) 000435 1741898408.765544 read 0x001a82 thru 0x006a8d (0x500c bytes) 000437 1741898408.772121 write 0x00bd41 thru 0x0128e0 (0x6ba0 bytes) 000438 1741898408.774288 mapwrite 0x02420c thru 0x02dc8d (0x9a82 bytes) 000439 1741898408.859028 read 0x0144bc thru 0x021c3f (0xd784 bytes) 000441 1741898408.870483 write 0x010cb7 thru 0x01a5c8 (0x9912 bytes) 000442 1741898408.872001 mapwrite 0x006fb8 thru 0x015eb3 (0xeefc bytes) 000443 1741898408.938580 read 0x01b704 thru 0x01ff05 (0x4802 bytes) 000444 1741898408.940215 write 0x005933 thru 0x0134cb (0xdb99 bytes) 000446 1741898408.941690 read 0x0084fd thru 0x00baa1 (0x35a5 bytes) 000448 1741898408.942607 write 0x01b5a5 thru 0x02874f (0xd1ab bytes) 000450 1741898408.951511 MIRROR_EXTEND to 3 mirrors 000451 1741898409.286456 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000452 1741898409.480998 trunc from 0x02dc8e to 0x03c5c9 (0xe93c bytes) 000453 1741898409.534280 write 0x030fff thru 0x03577f (0x4781 bytes) 000454 1741898409.539688 read 0x01bbdb thru 0x028482 (0xc8a8 bytes) 000456 1741898409.550657 read 0x024510 thru 0x02db31 (0x9622 bytes) 000457 1741898409.563501 mapread 0x00d87f thru 0x014d06 (0x7488 bytes) 000458 1741898409.564433 write 0x0311ae thru 0x03dba0 (0xc9f3 bytes) 000460 1741898409.567762 write 0x0239dc thru 0x032e93 (0xf4b8 bytes) 000462 1741898409.570724 write 0x0309d3 thru 0x03235e (0x198c bytes) 000463 1741898409.572481 read 0x027cf6 thru 0x0325a3 (0xa8ae bytes) 000464 1741898409.573697 mapread 0x003a33 thru 0x00d97d (0x9f4b bytes) 000466 1741898409.576875 write 0x01e57d thru 0x026937 (0x83bb bytes) 000467 1741898409.587383 MIRROR_RESYNC 2 mirrors 000468 1741898409.923020 MIRROR_EXTEND to 3 mirrors 000469 1741898410.301550 write 0x02f18d thru 0x0385d4 (0x9448 bytes) 000472 1741898410.358194 MIRROR_RESYNC 3 mirrors 000474 1741898410.731604 write 0x03cf43 thru 0x03ffff (0x30bd bytes) 000476 1741898410.814196 MIRROR_EXTEND to 4 mirrors 000478 1741898411.258301 mapwrite 0x00dc35 thru 0x012c2d (0x4ff9 bytes) 000480 1741898411.401484 mapread 0x008d0f thru 0x01037a (0x766c bytes) 000482 1741898411.402487 mapwrite 0x006071 thru 0x013760 (0xd6f0 bytes) 000483 1741898411.476558 trunc from 0x040000 to 0x02c579 (0x13a86 bytes) 000484 1741898411.496408 trunc from 0x02c57a to 0x0048e3 (0x27c96 bytes) 000485 1741898411.509732 write 0x009d47 thru 0x00ee7f (0x5139 bytes) 000486 1741898411.510506 write 0x034390 thru 0x03ffff (0xbc70 bytes) 000487 1741898411.513511 read 0x0394cd thru 0x03ffff (0x6b33 bytes) 000488 1741898411.514018 mapread 0x01dee7 thru 0x01f9cd (0x1ae7 bytes) 000490 1741898411.527267 read 0x02b4bb thru 0x032bfc (0x7742 bytes) 000491 1741898411.538732 MIRROR_RESYNC 4 mirrors 000493 1741898412.010557 read 0x01c9a7 thru 0x01de54 (0x14ae bytes) 000494 1741898412.013663 trunc from 0x040000 to 0x019ac9 (0x26536 bytes) 000495 1741898412.068586 MIRROR_EXTEND to 5 mirrors 000496 1741898412.399038 read 0x017db1 thru 0x019ac9 (0x1d19 bytes) 000497 1741898412.406750 read 0x01508b thru 0x019ac9 (0x4a3f bytes) 000498 1741898412.427002 MIRROR_RESYNC 5 mirrors 000499 1741898412.866016 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000501 1741898413.039648 MIRROR_EXTEND to 5 mirrors 000502 1741898413.369476 MIRROR_RESYNC 5 mirrors 000505 1741898413.493294 mapwrite 0x01f3cf thru 0x023fe1 (0x4c13 bytes) 000507 1741898413.644254 mapwrite 0x00febe thru 0x01d4c8 (0xd60b bytes) 000508 1741898413.733934 write 0x000a19 thru 0x007406 (0x69ee bytes) 000510 1741898413.746165 trunc from 0x023fe2 to 0x02aaf7 (0x6b16 bytes) 000511 1741898413.761665 mapread 0x00babc thru 0x01020a (0x474f bytes) 000512 1741898413.763079 mapwrite 0x00092d thru 0x00e695 (0xdd69 bytes) 000514 1741898413.818998 read 0x00faa9 thru 0x01f689 (0xfbe1 bytes) 000515 1741898413.819538 write 0x02f2aa thru 0x03415b (0x4eb2 bytes) 000517 1741898413.820724 mapread 0x01c508 thru 0x0241ce (0x7cc7 bytes) 000523 1741898413.842841 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000524 1741898414.039673 mapread 0x00e8b5 thru 0x017666 (0x8db2 bytes) 000525 1741898414.053114 trunc from 0x03415c to 0x02b398 (0x8dc3 bytes) 000527 1741898414.066702 read 0x0200c2 thru 0x02545c (0x539b bytes) 000528 1741898414.078476 trunc from 0x02b399 to 0x004ee9 (0x264af bytes) 000529 1741898414.097229 write 0x03ce8e thru 0x03e165 (0x12d8 bytes) 000530 1741898414.098592 trunc from 0x03e166 to 0x03dd2a (0x43b bytes) 000533 1741898414.109327 write 0x034480 thru 0x035005 (0xb86 bytes) 000534 1741898414.119926 trunc from 0x03dd2b to 0x02381a (0x1a510 bytes) 000535 1741898414.132060 trunc from 0x02381b to 0x030768 (0xcf4e bytes) 000536 1741898414.139356 mapwrite 0x01230f thru 0x01aa02 (0x86f4 bytes) 000538 1741898414.183401 write 0x01152a thru 0x013b6e (0x2645 bytes) 000539 1741898414.184099 trunc from 0x030769 to 0x020957 (0xfe11 bytes) 000540 1741898414.192574 mapwrite 0x00726a thru 0x014b1c (0xd8b3 bytes) 000541 1741898414.234601 trunc from 0x020958 to 0x012097 (0xe8c0 bytes) 000543 1741898414.241766 write 0x015e4f thru 0x023d0c (0xdebe bytes) 000544 1741898414.245229 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000546 1741898414.398051 read 0x012b75 thru 0x01cb89 (0xa015 bytes) 000547 1741898414.406468 mapwrite 0x03d585 thru 0x03fbbd (0x2639 bytes) 000548 1741898414.499503 write 0x033f0e thru 0x035f8f (0x2082 bytes) 000549 1741898414.506464 read 0x02e3a7 thru 0x0324c8 (0x4122 bytes) 000551 1741898414.507491 read 0x0058d8 thru 0x00faba (0xa1e3 bytes) 000552 1741898414.519866 MIRROR_RESYNC 3 mirrors 000556 1741898414.876957 trunc from 0x03fbbe to 0x00d7bc (0x32401 bytes) 000558 1741898414.956507 trunc from 0x00d7bd to 0x02165b (0x13e9f bytes) 000559 1741898414.976009 MIRROR_EXTEND to 4 mirrors 000560 1741898415.390051 trunc from 0x02165c to 0x0131b6 (0xe4a5 bytes) 000562 1741898415.484715 mapread 0x00f4f3 thru 0x0131b6 (0x3cc4 bytes) 000563 1741898415.505524 read 0x00e72b thru 0x012347 (0x3c1d bytes) 000564 1741898415.505892 mapread 0x00c77c thru 0x0131b6 (0x6a3b bytes) 000565 1741898415.506896 write 0x01b767 thru 0x01c52e (0xdc8 bytes) 000566 1741898415.507549 mapread 0x0144a2 thru 0x01c52e (0x808d bytes) 000568 1741898415.516619 MIRROR_EXTEND to 5 mirrors 000569 1741898415.865286 write 0x012592 thru 0x015d9e (0x380d bytes) 000570 1741898415.916714 read 0x00265e thru 0x009735 (0x70d8 bytes) 000572 1741898415.925464 MIRROR_RESYNC 5 mirrors 000573 1741898416.411021 trunc from 0x01c52f to 0x000c4c (0x1b8e2 bytes) 000574 1741898416.453007 write 0x000cf0 thru 0x0073a7 (0x66b8 bytes) 000575 1741898416.459012 read 0x000632 thru 0x004943 (0x4312 bytes) 000577 1741898416.459981 mapread 0x006cde thru 0x0073a7 (0x6ca bytes) 000578 1741898416.460887 trunc from 0x0073a8 to 0x0322d8 (0x2af31 bytes) 000579 1741898416.476528 trunc from 0x0322d9 to 0x02e218 (0x40c0 bytes) 000581 1741898416.489271 read 0x02d11f thru 0x02e218 (0x10fa bytes) 000582 1741898416.498516 mapwrite 0x02f8db thru 0x036a85 (0x71ab bytes) 000583 1741898416.562547 mapwrite 0x013676 thru 0x014657 (0xfe2 bytes) 000584 1741898416.618025 mapwrite 0x033ac6 thru 0x03f492 (0xb9cd bytes) 000585 1741898416.691141 read 0x00da04 thru 0x010361 (0x295e bytes) 000586 1741898416.692288 mapread 0x002e13 thru 0x003f26 (0x1114 bytes) 000587 1741898416.693998 mapread 0x0095ef thru 0x00c8a6 (0x32b8 bytes) 000589 1741898416.696971 trunc from 0x03f493 to 0x00caf5 (0x3299d bytes) 000590 1741898416.715591 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000591 1741898416.851920 trunc from 0x00caf6 to 0x02def3 (0x213fe bytes) 000592 1741898416.861168 mapwrite 0x02a159 thru 0x0336cc (0x9574 bytes) 000593 1741898416.952112 write 0x0235ad thru 0x029f5c (0x69b0 bytes) 000594 1741898416.966656 trunc from 0x0336cd to 0x0358d9 (0x220d bytes) 000595 1741898416.985644 write 0x0137fe thru 0x01ce23 (0x9626 bytes) 000598 1741898416.994636 mapwrite 0x0167ef thru 0x01d94f (0x7161 bytes) 000599 1741898417.055899 trunc from 0x0358da to 0x00b1cf (0x2a70a bytes) 000600 1741898417.065910 mapread 0x009598 thru 0x00b1cf (0x1c38 bytes) 000601 1741898417.077852 read 0x00273e thru 0x009ad9 (0x739c bytes) 000602 1741898417.087389 mapwrite 0x02f67b thru 0x03b225 (0xbbab bytes) 000604 1741898417.172158 trunc from 0x03b226 to 0x008e57 (0x323ce bytes) 000606 1741898417.181162 write 0x00ed3b thru 0x0110fb (0x23c1 bytes) 000608 1741898417.184695 MIRROR_RESYNC 4 mirrors 000609 1741898417.601720 MIRROR_EXTEND to 5 mirrors 000610 1741898417.902029 MIRROR_RESYNC 5 mirrors 000611 1741898418.022701 mapwrite 0x02304c thru 0x02c91a (0x98cf bytes) 000612 1741898418.180737 read 0x00e813 thru 0x018167 (0x9955 bytes) 000613 1741898418.185866 trunc from 0x02c91b to 0x003420 (0x294fa bytes) 000614 1741898418.202980 read 0x00063d thru 0x003420 (0x2de4 bytes) 000615 1741898418.207809 mapwrite 0x01582e thru 0x01e388 (0x8b5b bytes) 000616 1741898418.304779 trunc from 0x01e389 to 0x00b3a7 (0x12fe1 bytes) 000620 1741898418.322545 mapread 0x008108 thru 0x00b3a7 (0x32a0 bytes) 000621 1741898418.323205 write 0x01a7ee thru 0x022094 (0x78a7 bytes) 000622 1741898418.325079 mapread 0x01c1e8 thru 0x022094 (0x5ead bytes) 000623 1741898418.326579 mapread 0x01dd96 thru 0x022094 (0x42ff bytes) 000624 1741898418.327864 write 0x029f57 thru 0x0387be (0xe868 bytes) 000625 1741898418.330786 mapwrite 0x002ff0 thru 0x00e882 (0xb893 bytes) 000626 1741898418.405684 mapwrite 0x001833 thru 0x00c588 (0xad56 bytes) 000627 1741898418.468500 MIRROR_RESYNC 5 mirrors 000628 1741898419.047226 mapread 0x017270 thru 0x01bbb5 (0x4946 bytes) 000630 1741898419.056810 read 0x027807 thru 0x02d686 (0x5e80 bytes) 000631 1741898419.061664 MIRROR_EXTEND to 6 mirrors 000632 1741898419.429682 MIRROR_EXTEND to 7 mirrors 000633 1741898419.903956 mapread 0x022d3a thru 0x03012a (0xd3f1 bytes) 000635 1741898419.923133 mapread 0x01f07f thru 0x01fd19 (0xc9b bytes) 000638 1741898419.924164 mapread 0x02c8c0 thru 0x0387be (0xbeff bytes) 000639 1741898419.933402 read 0x031457 thru 0x035f44 (0x4aee bytes) 000640 1741898419.933829 write 0x032ac6 thru 0x03dbfe (0xb139 bytes) 000641 1741898419.986660 write 0x007595 thru 0x016da4 (0xf810 bytes) 000642 1741898420.000732 trunc from 0x03dbff to 0x01a558 (0x236a6 bytes) 000645 1741898420.041720 trunc from 0x01a559 to 0x02a925 (0x103cd bytes) 000646 1741898420.058359 mapwrite 0x036c66 thru 0x03ffff (0x939a bytes) 000648 1741898420.168480 mapwrite 0x003821 thru 0x010ff8 (0xd7d8 bytes) 000650 1741898420.225575 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000651 1741898420.335288 trunc from 0x040000 to 0x03d17b (0x2e84 bytes) 000652 1741898420.344931 trunc from 0x03d17c to 0x021544 (0x1bc37 bytes) 000656 1741898420.353218 read 0x00afc6 thru 0x00e1fb (0x3236 bytes) 000657 1741898420.358017 MIRROR_EXTEND to 7 mirrors 000658 1741898420.734331 write 0x03029a thru 0x0364a6 (0x620d bytes) 000659 1741898420.797717 write 0x022589 thru 0x02bca0 (0x9718 bytes) 000662 1741898420.805424 mapwrite 0x00a680 thru 0x00aa40 (0x3c1 bytes) 000663 1741898420.857188 trunc from 0x0364a7 to 0x0261f9 (0x102ad bytes) 000664 1741898420.907963 mapread 0x0238f5 thru 0x0261f9 (0x2905 bytes) 000665 1741898420.912060 trunc from 0x0261fa to 0x026e88 (0xc8f bytes) 000666 1741898420.933311 read 0x00dae3 thru 0x01780a (0x9d28 bytes) 000667 1741898420.938710 mapread 0x01902f thru 0x01f7d6 (0x67a8 bytes) 000673 1741898420.945464 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000674 1741898421.122531 mapread 0x0048fa thru 0x01197b (0xd082 bytes) 000677 1741898421.147527 read 0x003a98 thru 0x0110d0 (0xd639 bytes) 000679 1741898421.148457 trunc from 0x026e89 to 0x003c15 (0x23273 bytes) 000680 1741898421.162001 trunc from 0x003c16 to 0x0256c7 (0x21ab2 bytes) 000681 1741898421.176418 write 0x01b6d8 thru 0x02305f (0x7988 bytes) 000682 1741898421.203402 write 0x024d32 thru 0x029cd7 (0x4fa6 bytes) 000684 1741898421.209428 write 0x00086f thru 0x007dd3 (0x7565 bytes) 000686 1741898421.214472 trunc from 0x029cd8 to 0x00e9cc (0x1b30b bytes) 000687 1741898421.227534 mapread 0x00626a thru 0x00e9cc (0x8763 bytes) 000688 1741898421.230566 read 0x00635d thru 0x00a2da (0x3f7e bytes) 000689 1741898421.235689 MIRROR_EXTEND to 7 mirrors 000691 1741898421.619000 trunc from 0x00e9cd to 0x02b51d (0x1cb51 bytes) 000692 1741898421.713057 trunc from 0x02b51e to 0x00685d (0x24cc0 bytes) 000694 1741898421.729514 mapwrite 0x0365fb thru 0x03ffff (0x9a05 bytes) 000695 1741898421.815794 write 0x027fc7 thru 0x031112 (0x914c bytes) 000697 1741898421.826018 trunc from 0x040000 to 0x027498 (0x18b67 bytes) 000698 1741898421.840363 read 0x0077e2 thru 0x00b62f (0x3e4e bytes) 000699 1741898421.848691 mapread 0x010416 thru 0x01bfcd (0xbbb8 bytes) 000701 1741898421.863969 MIRROR_RESYNC 7 mirrors 000702 1741898422.667486 mapread 0x00d57a thru 0x018a0b (0xb492 bytes) 000704 1741898422.696503 mapwrite 0x023360 thru 0x026bee (0x388f bytes) 000706 1741898422.851890 read 0x006bd2 thru 0x0074e7 (0x916 bytes) 000710 1741898422.858700 mapread 0x00d8e9 thru 0x01afee (0xd706 bytes) 000713 1741898422.875547 read 0x00aab3 thru 0x00f78e (0x4cdc bytes) 000715 1741898422.882206 read 0x0155b2 thru 0x01d026 (0x7a75 bytes) 000716 1741898422.884976 MIRROR_RESYNC 7 mirrors 000718 1741898423.756531 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000720 1741898423.993423 MIRROR_RESYNC 6 mirrors 000721 1741898424.159371 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000722 1741898424.373992 mapwrite 0x01f612 thru 0x0227f8 (0x31e7 bytes) 000723 1741898424.546796 mapread 0x0041c3 thru 0x006879 (0x26b7 bytes) 000724 1741898424.558381 write 0x01c9f7 thru 0x01e863 (0x1e6d bytes) 000725 1741898424.561093 read 0x0236e3 thru 0x025982 (0x22a0 bytes) 000728 1741898424.562227 trunc from 0x027499 to 0x01685c (0x10c3c bytes) 000729 1741898424.583299 mapwrite 0x02fa1b thru 0x0360fc (0x66e2 bytes) 000730 1741898424.657488 read 0x0016b6 thru 0x00499b (0x32e6 bytes) 000731 1741898424.658354 mapread 0x019390 thru 0x020e41 (0x7ab2 bytes) 000734 1741898424.671173 trunc from 0x0360fd to 0x026154 (0xffa8 bytes) 000735 1741898424.684684 trunc from 0x026155 to 0x033509 (0xd3b5 bytes) 000737 1741898424.703576 read 0x008532 thru 0x00d2e6 (0x4db5 bytes) 000738 1741898424.704065 trunc from 0x03350a to 0x02e9e2 (0x4b27 bytes) 000740 1741898424.717312 mapwrite 0x000363 thru 0x00dad8 (0xd776 bytes) 000741 1741898424.752315 mapwrite 0x03ac7e thru 0x03ffff (0x5382 bytes) 000743 1741898424.816686 mapwrite 0x01d2b0 thru 0x01dd47 (0xa98 bytes) 000744 1741898424.854962 read 0x0053c6 thru 0x00a329 (0x4f64 bytes) 000745 1741898424.856340 mapread 0x019596 thru 0x0256eb (0xc156 bytes) 000746 1741898424.860641 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000748 1741898425.004882 mapwrite 0x038710 thru 0x03c72c (0x401d bytes) 000749 1741898425.089463 mapwrite 0x015e83 thru 0x0173c0 (0x153e bytes) 000751 1741898425.145477 trunc from 0x040000 to 0x00a43c (0x35bc3 bytes) 000753 1741898425.160718 MIRROR_RESYNC 4 mirrors 000754 1741898425.667945 read 0x008d3b thru 0x00a43c (0x1702 bytes) 000755 1741898425.672623 read 0x000030 thru 0x0030fa (0x30cb bytes) 000758 1741898425.679484 trunc from 0x00a43d to 0x0328fb (0x284bf bytes) 000760 1741898425.737147 write 0x03e2ae thru 0x03ffff (0x1d52 bytes) 000761 1741898425.737970 read 0x0339f9 thru 0x039756 (0x5d5e bytes) 000765 1741898425.747560 write 0x027da6 thru 0x034b31 (0xcd8c bytes) 000766 1741898425.748322 trunc from 0x040000 to 0x003876 (0x3c789 bytes) 000769 1741898425.758544 mapread 0x001fdc thru 0x003876 (0x189b bytes) 000770 1741898425.759754 read 0x001925 thru 0x003876 (0x1f52 bytes) 000771 1741898425.760084 mapwrite 0x016152 thru 0x023868 (0xd717 bytes) 000772 1741898425.867598 read 0x01f7f0 thru 0x023868 (0x4079 bytes) 000775 1741898425.869700 read 0x009283 thru 0x0124c7 (0x9245 bytes) 000777 1741898425.871486 read 0x01d5be thru 0x022038 (0x4a7b bytes) 000778 1741898425.872071 mapread 0x000804 thru 0x003bd5 (0x33d2 bytes) 000779 1741898425.873377 read 0x01ee11 thru 0x023868 (0x4a58 bytes) 000780 1741898425.874535 mapwrite 0x032402 thru 0x03c883 (0xa482 bytes) 000781 1741898425.969741 write 0x033a28 thru 0x03d2ae (0x9887 bytes) 000784 1741898425.973383 mapwrite 0x017a11 thru 0x0277ab (0xfd9b bytes) 000785 1741898426.034456 mapwrite 0x0350c5 thru 0x03ad80 (0x5cbc bytes) 000787 1741898426.093877 read 0x02cff6 thru 0x0308e4 (0x38ef bytes) 000789 1741898426.094675 mapread 0x02e7e9 thru 0x03c938 (0xe150 bytes) 000790 1741898426.096000 mapread 0x03b4e1 thru 0x03ce61 (0x1981 bytes) 000791 1741898426.097589 trunc from 0x03d2af to 0x016957 (0x26957 bytes) 000792 1741898426.107352 mapwrite 0x00d03b thru 0x017c85 (0xac4b bytes) 000795 1741898426.179131 mapwrite 0x0116f3 thru 0x014791 (0x309f bytes) 000797 1741898426.230094 read 0x00770b thru 0x0092d7 (0x1bcd bytes) 000798 1741898426.230548 read 0x00e277 thru 0x0170d2 (0x8e5c bytes) 000799 1741898426.230956 mapwrite 0x00ac02 thru 0x00bf10 (0x130f bytes) 000803 1741898426.286006 write 0x03e80b thru 0x03ffff (0x17f5 bytes) 000804 1741898426.287172 trunc from 0x040000 to 0x0334b0 (0xcb4f bytes) 000806 1741898426.297128 trunc from 0x0334b1 to 0x025271 (0xe23f bytes) 000807 1741898426.308312 trunc from 0x025272 to 0x027356 (0x20e5 bytes) 000808 1741898426.321686 trunc from 0x027357 to 0x038c3a (0x118e4 bytes) 000811 1741898426.342024 MIRROR_RESYNC 4 mirrors 000813 1741898426.759661 read 0x00fb77 thru 0x0169a6 (0x6e30 bytes) 000814 1741898426.764528 read 0x00fb12 thru 0x01cb30 (0xd01f bytes) 000815 1741898426.770558 mapwrite 0x01b314 thru 0x027669 (0xc356 bytes) 000816 1741898426.935406 MIRROR_EXTEND to 5 mirrors 000818 1741898427.314632 mapread 0x01869c thru 0x021e02 (0x9767 bytes) 000819 1741898427.344183 mapread 0x0002c0 thru 0x00f810 (0xf551 bytes) 000820 1741898427.370657 write 0x006318 thru 0x00f969 (0x9652 bytes) 000821 1741898427.429915 write 0x032921 thru 0x03ffff (0xd6df bytes) 000826 1741898427.443914 write 0x0024a9 thru 0x00f78b (0xd2e3 bytes) 000827 1741898427.445535 mapwrite 0x033ed0 thru 0x03c33e (0x846f bytes) 000828 1741898427.505816 mapread 0x005048 thru 0x0124c6 (0xd47f bytes) 000832 1741898427.519171 trunc from 0x040000 to 0x033fe0 (0xc01f bytes) 000833 1741898427.538045 trunc from 0x033fe1 to 0x0264b2 (0xdb2e bytes) 000834 1741898427.562374 mapwrite 0x019f34 thru 0x01ffc4 (0x6091 bytes) 000835 1741898427.615348 write 0x004f27 thru 0x008cae (0x3d88 bytes) 000836 1741898427.617348 read 0x00e263 thru 0x016842 (0x85e0 bytes) 000837 1741898427.617797 trunc from 0x0264b3 to 0x039f37 (0x13a85 bytes) 000839 1741898427.631930 mapwrite 0x02026e thru 0x02e552 (0xe2e5 bytes) 000840 1741898427.701723 mapread 0x028cf6 thru 0x031739 (0x8a44 bytes) 000843 1741898427.706614 mapwrite 0x03d573 thru 0x03ffff (0x2a8d bytes) 000844 1741898427.783266 mapread 0x02b08d thru 0x02f813 (0x4787 bytes) truncating to largest ever: 0x3eead 000845 1741898427.784139 trunc from 0x040000 to 0x03eeac (0x1153 bytes) 000846 1741898427.792992 mapread 0x024216 thru 0x026c30 (0x2a1b bytes) 000847 1741898427.794380 mapwrite 0x03377d thru 0x03dc6b (0xa4ef bytes) 000849 1741898427.842141 read 0x0212f0 thru 0x02296d (0x167e bytes) 000852 1741898427.851029 MIRROR_EXTEND to 6 mirrors 000853 1741898428.141539 read 0x035807 thru 0x03796f (0x2169 bytes) 000854 1741898428.146639 mapread 0x014be9 thru 0x01676f (0x1b87 bytes) 000855 1741898428.169827 write 0x034b92 thru 0x03ffff (0xb46e bytes) 000857 1741898428.215627 mapwrite 0x0337c9 thru 0x039938 (0x6170 bytes) 000859 1741898428.259755 read 0x03f332 thru 0x03ffff (0xcce bytes) 000861 1741898428.262528 mapwrite 0x010981 thru 0x01e8c8 (0xdf48 bytes) 000864 1741898428.332043 MIRROR_EXTEND to 7 mirrors 000867 1741898428.688610 MIRROR_EXTEND to 8 mirrors 000868 1741898429.049143 mapwrite 0x03ce93 thru 0x03ffff (0x316d bytes) 000869 1741898429.207497 mapread 0x038df6 thru 0x03ffff (0x720a bytes) 000870 1741898429.208376 read 0x01e132 thru 0x022e9b (0x4d6a bytes) 000872 1741898429.213324 read 0x010642 thru 0x010c10 (0x5cf bytes) 000873 1741898429.219591 mapread 0x016d6b thru 0x01ad4f (0x3fe5 bytes) 000874 1741898429.221925 write 0x00da2a thru 0x0120f1 (0x46c8 bytes) 000876 1741898429.222828 mapwrite 0x00e78c thru 0x01d3d1 (0xec46 bytes) 000877 1741898429.268831 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000879 1741898429.389654 trunc from 0x040000 to 0x023e17 (0x1c1e8 bytes) 000881 1741898429.413423 mapread 0x00883a thru 0x0166f7 (0xdebe bytes) 000882 1741898429.439915 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000884 1741898429.570751 read 0x0196ec thru 0x022c8d (0x95a2 bytes) 000886 1741898429.578585 read 0x00bb01 thru 0x011679 (0x5b79 bytes) 000887 1741898429.587708 read 0x010d49 thru 0x01744f (0x6707 bytes) 000888 1741898429.588280 read 0x017b76 thru 0x01bf8e (0x4419 bytes) 000889 1741898429.592648 MIRROR_RESYNC 6 mirrors 000890 1741898430.125932 mapwrite 0x03bee1 thru 0x03ffff (0x411f bytes) 000891 1741898430.235044 read 0x01f188 thru 0x01fb88 (0xa01 bytes) 000893 1741898430.245857 write 0x02ab69 thru 0x02c95a (0x1df2 bytes) 000897 1741898430.261671 read 0x01e5fb thru 0x02b925 (0xd32b bytes) 000898 1741898430.277004 MIRROR_EXTEND to 7 mirrors 000899 1741898430.557304 mapwrite 0x03a28b thru 0x03ffff (0x5d75 bytes) 000901 1741898430.690921 read 0x00408e thru 0x010afd (0xca70 bytes) 000902 1741898430.705473 MIRROR_EXTEND to 8 mirrors 000903 1741898431.000408 mapwrite 0x00dea4 thru 0x01d5c6 (0xf723 bytes) 000905 1741898431.127602 trunc from 0x040000 to 0x02e86d (0x11792 bytes) 000906 1741898431.155603 read 0x0214ee thru 0x0219d0 (0x4e3 bytes) 000907 1741898431.159914 write 0x00770e thru 0x009c67 (0x255a bytes) 000909 1741898431.161461 trunc from 0x02e86e to 0x009c67 (0x24c06 bytes) 000910 1741898431.178959 write 0x0035f6 thru 0x00e705 (0xb110 bytes) 000911 1741898431.190400 MIRROR_RESYNC 8 mirrors 000913 1741898431.837531 mapwrite 0x034a37 thru 0x03ffff (0xb5c9 bytes) 000914 1741898431.957608 mapwrite 0x03b1b2 thru 0x03d7f5 (0x2644 bytes) 000915 1741898431.982740 mapwrite 0x002e87 thru 0x00c5ff (0x9779 bytes) 000916 1741898432.020160 trunc from 0x040000 to 0x01c732 (0x238cd bytes) 000917 1741898432.034787 mapwrite 0x012d23 thru 0x0216ad (0xe98b bytes) 000919 1741898432.097484 read 0x005a0d thru 0x00caaa (0x709e bytes) 000920 1741898432.097998 read 0x0213a0 thru 0x0216ad (0x30e bytes) 000922 1741898432.098974 read 0x00a389 thru 0x015bd0 (0xb848 bytes) 000923 1741898432.099620 trunc from 0x0216ae to 0x02e7c1 (0xd114 bytes) 000926 1741898432.111905 mapwrite 0x00067f thru 0x00a944 (0xa2c6 bytes) truncating to largest ever: 0x3f68f 000928 1741898432.160160 trunc from 0x02e7c2 to 0x03f68e (0x10ecd bytes) 000929 1741898432.167465 trunc from 0x03f68f to 0x0078ee (0x37da0 bytes) 000930 1741898432.174825 mapread 0x0015aa thru 0x0078ee (0x6345 bytes) 000931 1741898432.176005 trunc from 0x0078ef to 0x02524d (0x1d95f bytes) 000933 1741898432.188767 MIRROR_RESYNC 8 mirrors 000935 1741898432.741743 mapwrite 0x00218c thru 0x00a767 (0x85dc bytes) 000936 1741898432.871596 mapwrite 0x038266 thru 0x03e73a (0x64d5 bytes) 000937 1741898432.956431 mapwrite 0x0199a7 thru 0x025b25 (0xc17f bytes) 000938 1741898433.025571 write 0x001373 thru 0x0064ea (0x5178 bytes) 000939 1741898433.026504 write 0x021cc9 thru 0x026247 (0x457f bytes) 000941 1741898433.027907 mapread 0x02c374 thru 0x030baf (0x483c bytes) 000946 1741898433.039758 read 0x02e3e8 thru 0x030a26 (0x263f bytes) 000948 1741898433.046708 MIRROR_RESYNC 8 mirrors 000950 1741898433.784520 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000952 1741898433.998672 read 0x00b90e thru 0x011bb6 (0x62a9 bytes) 000953 1741898434.006692 mapwrite 0x01ce08 thru 0x0265ff (0x97f8 bytes) 000954 1741898434.144643 read 0x024abe thru 0x02cc84 (0x81c7 bytes) 000955 1741898434.145128 mapwrite 0x00e4f1 thru 0x017c00 (0x9710 bytes) 000956 1741898434.199122 read 0x004e1f thru 0x00a59a (0x577c bytes) 000959 1741898434.200769 mapwrite 0x03651b thru 0x03ffff (0x9ae5 bytes) 000960 1741898434.287586 read 0x01cbce thru 0x02a9cd (0xde00 bytes) 000961 1741898434.288077 trunc from 0x040000 to 0x0398ce (0x6731 bytes) 000962 1741898434.299713 write 0x02de5c thru 0x031286 (0x342b bytes) 000963 1741898434.300944 read 0x026fbe thru 0x02e812 (0x7855 bytes) 000964 1741898434.301422 trunc from 0x0398cf to 0x016993 (0x22f3b bytes) 000965 1741898434.314832 trunc from 0x016994 to 0x02503e (0xe6ab bytes) 000968 1741898434.332127 trunc from 0x02503f to 0x0031ff (0x21e3f bytes) 000970 1741898434.346584 trunc from 0x003200 to 0x02f6ab (0x2c4ac bytes) 000971 1741898434.357296 trunc from 0x02f6ac to 0x005510 (0x2a19b bytes) 000972 1741898434.367511 trunc from 0x005511 to 0x028e69 (0x23959 bytes) 000973 1741898434.382162 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000974 1741898434.612062 write 0x0151d6 thru 0x015209 (0x34 bytes) 000976 1741898434.616539 read 0x02132d thru 0x028e69 (0x7b3d bytes) 000979 1741898434.635516 MIRROR_EXTEND to 7 mirrors 000981 1741898435.032660 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000982 1741898435.254173 read 0x024ef5 thru 0x028e69 (0x3f75 bytes) 000984 1741898435.289405 MIRROR_EXTEND to 7 mirrors 000985 1741898435.715939 MIRROR_RESYNC 7 mirrors 000986 1741898436.222662 write 0x018aad thru 0x022cfb (0xa24f bytes) 000988 1741898436.280731 trunc from 0x028e6a to 0x026c39 (0x2230 bytes) 000989 1741898436.303150 read 0x00048d thru 0x00cf3b (0xcaaf bytes) 000990 1741898436.308352 read 0x01c61f thru 0x023106 (0x6ae8 bytes) 000991 1741898436.312756 write 0x01d7ac thru 0x02b2a1 (0xdaf6 bytes) 000994 1741898436.314644 read 0x000d6d thru 0x001ceb (0xf7f bytes) 000995 1741898436.316976 trunc from 0x02b2a2 to 0x00f7a5 (0x1bafc bytes) 000997 1741898436.332228 mapwrite 0x019d46 thru 0x01e82f (0x4aea bytes) 000998 1741898436.401560 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) 000999 1741898436.548745 read 0x012b2d thru 0x015045 (0x2519 bytes) 001000 1741898436.563451 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x415:0x0]: Invalid argument (22) All operations completed A-OK! PASS 70a (42s) == sanity-flr test 71: check mirror extend parent fid ==== 16:40:38 (1741898438) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 200a: stress test ===================== 16:40:40 (1741898440) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3473504 .. Extending file size to 9045568 .. Extending file size to 7442496 .. Extending file size to 8703712 .. Extending file size to 8827616 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9077728 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8854368 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9226016 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9214752 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9217600 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 26471 26472 26473 26475 26478 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) 192.168.201.110@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg110-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:41:47 (1741898507) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4085376 .. Extending file size to 4898016 .. failed Extending file size to 8686016 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 5779200 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7789440 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7839360 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8476960 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8696896 .. Extending file size to 8737888 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9080800 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9211296 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 29526 29528 29529 29530 29531 done 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) 192.168.201.110@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg110-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:42:54 (1741898574) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4480 2203904 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4480 2203904 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0495578 s, 42.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.145016 s, 14.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:43:01 (1741898581) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0287529 s, 72.9 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 16:43:03 (1741898583) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x45b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x522:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x522:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x523:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x51d:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:43:06 (1741898586) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0548035 s, 38.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x524:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:43:08 (1741898588) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0663777 s, 31.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x525:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x526:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x520:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x525:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x526:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x520:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x521:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:43:11 (1741898591) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0497958 s, 42.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x527:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x522:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x527:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x522:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x523:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:43:35 (1741898615) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg110-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.044303 s, 23.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x528:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x525:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x524:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x528:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x525:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x524:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x529:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:44:21 (1741898661) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg110-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x526:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x527:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0662488 s, 31.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x526:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x527:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x52a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x526:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x527:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x52a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:45:06 (1741898706) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg110-server mds-ost sync done. Creating new pool test_204f oleg110-server: Pool lustre.test_204f created Adding targets to pool oleg110-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg110-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0419984 s, 49.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x528:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x52b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x528:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x52b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg110-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg110-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg110-server: Pool lustre.test_204f destroyed PASS 204f (55s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:46:01 (1741898761) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:46:04 (1741898764) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:46:08 (1741898768) oleg110-server: Pool lustre.test_206 created Adding targets to pool oleg110-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg110-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg110-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg110-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg110-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:46:21 (1741898781) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0272476 s, 38.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102387 s, 102 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x7d2f621 Mirror 3: 0x7d2f621 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:46:24 (1741898784) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:46:25 (1741898785) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:46:26 (1741898786) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0300601 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1700000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1280000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1280000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1280000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x24a0000 Read this (4 bytes): BBBB 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:46:42 (1741898802) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0111438 s, 735 kB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:46:45 (1741898805) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173783 s, 60.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x480 lmm_fid: [0x200000404:0x480:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x480 lmm_fid: [0x200000404:0x480:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x536:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835072 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-03-13 16:46:45.000000000 -0400 Modify: 2025-03-13 16:46:45.000000000 -0400 Change: 2025-03-13 16:46:46.000000000 -0400 Birth: - PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:46:48 (1741898808) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:46:49 (1741898809) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.241344 s, 170 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0289159 s, 142 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 13 16:46 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x482:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 13 16:46 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 965 sec ============ 16:46:52 (1741898812) === sanity-flr: start cleanup 16:46:53 (1741898813) === === sanity-flr: finish cleanup 16:46:55 (1741898815) ===