-----============= acceptance-small: sanity-flr ============----- Sun Aug 31 17:14:51 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg423-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 17:14:57 (1756674897) === oleg423-client.virtnet: executing check_config_client /mnt/lustre oleg423-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg423-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99bf05e64800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99bf05e64800.idle_timeout=debug disable quota as required oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 17:15:07 (1756674907) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7554] == sanity-flr test 0a: lfs mirror create with -N option == 17:15:08 (1756674908) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 17:15:12 (1756674912) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 17:15:13 (1756674913) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 17:15:14 (1756674914) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 17:15:20 (1756674920) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 17:15:21 (1756674921) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 17:15:22 (1756674922) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff99bf05e64800.stats=clear osc.lustre-OST0001-osc-ffff99bf05e64800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:15:25 (1756674925) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:15:28 (1756674928) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:15:30 (1756674930) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:15:31 (1756674931) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 17:15:34 (1756674934) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:15:36 (1756674936) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:15:37 (1756674937) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:15:39 (1756674939) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:15:42 (1756674942) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:15:44 (1756674944) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.130045 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369595 s, 28.4 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:15:47 (1756674947) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.108528 s, 116 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:15:50 (1756674950) Stopping /mnt/lustre-ost1 (opts:) on oleg423-server oleg423-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99bf05e64800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99bf05e64800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg423-server: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg423-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg423-server oleg423-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99bf05e64800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99bf05e64800.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg423-server: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg423-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (25s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:16:16 (1756674976) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.209503 s, 55.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg423-server oleg423-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99bf05e64800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99bf05e64800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg423-server: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg423-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (11s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:16:27 (1756674987) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg423-server oleg423-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99bf05e64800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99bf05e64800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg423-server: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg423-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg423-server oleg423-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99bf05e64800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99bf05e64800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg423-server: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg423-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (25s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:16:52 (1756675012) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.780917 s, 67.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 31 17:16 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.222s user 0m0.000s sys 0m0.135s Stopping /mnt/lustre-ost1 (opts:) on oleg423-server oleg423-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99bf05e64800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99bf05e64800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... real 0m0.280s user 0m0.000s sys 0m0.169s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg423-server: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg423-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg423-server oleg423-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99bf05e64800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99bf05e64800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.279s user 0m0.000s sys 0m0.158s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg423-server: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg423-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg423-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99bf05e64800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99bf05e64800.ost_server_uuid in FULL state after 0 sec PASS 33b (27s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:17:19 (1756675039) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:17:21 (1756675041) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:17:22 (1756675042) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 17:17:23 (1756675043) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 17:17:25 (1756675045) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.141262 s, 89.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:17:29 (1756675049) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00975463 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00942368 s, 111 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:17:31 (1756675051) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:17:34 (1756675054) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000574686 s, 1.8 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000489281 s, 2.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000674996 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000599094 s, 1.7 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 17:17:37 (1756675057) create test files with size 15747758 .. c1002926623922cff79b05b65f5cb801 - 832436284393235c16d2549433717806 - a08d1d255ed1c9edfa47b2595f3b6bbd - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg423-client.virtnet /mnt/lustre (opts:) Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 17:17:43 (1756675063) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:17:46 (1756675066) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:17:49 (1756675069) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258117 s, 40.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x43:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156471 s, 67.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 17:17:51 (1756675071) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0331587 s, 63.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.103828 s, 40.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0501322 s, 41.8 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 17:17:55 (1756675075) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 17:17:56 (1756675076) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:17:57 (1756675077) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00886596 s, 118 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 17:17:59 (1756675079) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:18:00 (1756675080) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:18:04 (1756675084) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0946413 s, 111 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg423-client.virtnet /mnt/lustre (opts:) Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (21s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:18:25 (1756675105) llite.lustre-ffff99bf05e62000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0795651 s, 132 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg423-client.virtnet /mnt/lustre (opts:) Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff99bf087b6800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:18:29 (1756675109) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:18:40 (1756675120) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (10s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:18:50 (1756675130) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 17:18:51 (1756675131) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0675371 s, 46.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0480579 s, 65.5 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: ac483632ec578b784d8818a990ef854f - mirror 1 checksum: 564a1c380aa9e2e7993f49ad9462a39d - mirror 2 checksum: ac483632ec578b784d8818a990ef854f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Aug 31 17:18:52 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Aug 31 17:18:52 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: ac483632ec578b784d8818a990ef854f - mirror 1 checksum: 564a1c380aa9e2e7993f49ad9462a39d - mirror 2 checksum: 564a1c380aa9e2e7993f49ad9462a39d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Aug 31 17:18:52 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Aug 31 17:18:52 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:18:55 (1756675135) Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server 17:18:57 (1756675137) shut down facet: mds1 facet_host: oleg423-server facet_failover_host: oleg423-server Failover mds1 to oleg423-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 17:19:11 (1756675151) targets are mounted 17:19:11 (1756675151) facet_failover done oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00870865 s, 120 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (22s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:19:17 (1756675157) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:19:18 (1756675158) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:19:19 (1756675159) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00286513 s, 5.7 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:19:21 (1756675161) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:19:22 (1756675162) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:19:23 (1756675163) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (7s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:19:30 (1756675170) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 17:19:49 (1756675189) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 17:20:06 (1756675206) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 17:20:24 (1756675224) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 17:20:26 (1756675226) mirror create pid 52250 mirror split pid 52251 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 52250 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 52251 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 17:21:28 (1756675288) Seed set to 5289 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1756675289.320657 mapwrite 0x024909 thru 0x032724 (0xde1c bytes) 000005 1756675289.349574 read 0x002825 thru 0x003e79 (0x1655 bytes) 000006 1756675289.351975 mapwrite 0x039294 thru 0x03efa0 (0x5d0d bytes) 000008 1756675289.372318 mapwrite 0x0327f0 thru 0x03bd54 (0x9565 bytes) 000009 1756675289.386243 read 0x00bed5 thru 0x014658 (0x8784 bytes) truncating to largest ever: 0x19a9f 000011 1756675289.389711 trunc from 0x03efa1 to 0x019a9e (0x25502 bytes) 000015 1756675289.395980 mapwrite 0x003ce0 thru 0x007a09 (0x3d2a bytes) truncating to largest ever: 0x23e4d 000017 1756675289.409306 trunc from 0x019a9f to 0x023e4c (0xa3ae bytes) 000022 1756675289.414927 trunc from 0x023e4d to 0x018d1a (0xb132 bytes) 000028 1756675289.423509 MIRROR_EXTEND to 2 mirrors 000030 1756675289.553678 read 0x00997f thru 0x014d44 (0xb3c6 bytes) 000031 1756675289.557134 mapwrite 0x00e10c thru 0x016ddd (0x8cd2 bytes) 000032 1756675289.604818 mapwrite 0x015beb thru 0x01a9a7 (0x4dbd bytes) 000033 1756675289.632482 trunc from 0x01a9a8 to 0x005d93 (0x14c14 bytes) 000034 1756675289.638388 mapwrite 0x024350 thru 0x028f58 (0x4c09 bytes) 000035 1756675289.661552 mapwrite 0x024f7d thru 0x031467 (0xc4eb bytes) 000036 1756675289.693649 mapwrite 0x02fb53 thru 0x031669 (0x1b17 bytes) 000037 1756675289.723593 write 0x01942d thru 0x01bf64 (0x2b38 bytes) 000039 1756675289.729234 write 0x01ef4a thru 0x02c225 (0xd2dc bytes) 000042 1756675289.731086 mapread 0x027634 thru 0x029c4a (0x2617 bytes) 000043 1756675289.731812 write 0x026224 thru 0x031aa6 (0xb883 bytes) 000044 1756675289.735411 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000047 1756675289.798448 read 0x00e1a8 thru 0x01b663 (0xd4bc bytes) 000052 1756675289.804538 mapwrite 0x02d198 thru 0x038f1f (0xbd88 bytes) 000053 1756675289.853953 read 0x01bd58 thru 0x022195 (0x643e bytes) 000054 1756675289.858897 write 0x01bc72 thru 0x0211b2 (0x5541 bytes) 000055 1756675289.859588 read 0x029793 thru 0x03423e (0xaaac bytes) 000060 1756675289.863556 MIRROR_EXTEND to 2 mirrors 000061 1756675290.040635 MIRROR_EXTEND to 3 mirrors truncating to largest ever: 0x35343 000062 1756675290.185438 trunc from 0x038f20 to 0x035342 (0x3bdd bytes) 000063 1756675290.209019 trunc from 0x035343 to 0x01f535 (0x15e0d bytes) 000064 1756675290.214045 mapwrite 0x023d8f thru 0x02c210 (0x8482 bytes) 000065 1756675290.239375 MIRROR_RESYNC 3 mirrors 000066 1756675290.396710 mapread 0x01058a thru 0x011d16 (0x178d bytes) 000068 1756675290.401591 MIRROR_RESYNC 3 mirrors 000069 1756675290.450804 trunc from 0x02c211 to 0x030ffc (0x4dec bytes) 000070 1756675290.477562 read 0x01b276 thru 0x01fabd (0x4848 bytes) truncating to largest ever: 0x3ef23 000071 1756675290.480709 trunc from 0x030ffd to 0x03ef22 (0xdf26 bytes) 000072 1756675290.486561 read 0x0355aa thru 0x03ef22 (0x9979 bytes) 000073 1756675290.490759 read 0x032f6c thru 0x03dc14 (0xaca9 bytes) 000074 1756675290.491381 mapwrite 0x02b29d thru 0x031843 (0x65a7 bytes) 000076 1756675290.510693 MIRROR_EXTEND to 4 mirrors 000077 1756675290.664185 mapread 0x02db79 thru 0x039d4d (0xc1d5 bytes) 000080 1756675290.676075 read 0x01414a thru 0x022359 (0xe210 bytes) 000081 1756675290.680743 read 0x025e17 thru 0x026d5e (0xf48 bytes) 000084 1756675290.688806 write 0x01941e thru 0x0200f2 (0x6cd5 bytes) 000085 1756675290.717635 mapread 0x0290ed thru 0x0336fe (0xa612 bytes) 000086 1756675290.735073 mapwrite 0x005650 thru 0x010a03 (0xb3b4 bytes) 000087 1756675290.768633 trunc from 0x03ef23 to 0x034096 (0xae8c bytes) 000091 1756675290.780295 write 0x035ada thru 0x03f072 (0x9599 bytes) 000092 1756675290.781662 mapwrite 0x0385d9 thru 0x03eaf9 (0x6521 bytes) 000093 1756675290.794700 trunc from 0x03f073 to 0x035e3e (0x9234 bytes) 000094 1756675290.800720 read 0x00f46b thru 0x0147c7 (0x535d bytes) 000097 1756675290.805914 mapwrite 0x03d834 thru 0x03d9fa (0x1c7 bytes) 000098 1756675290.832738 mapread 0x0032a7 thru 0x007f3e (0x4c98 bytes) 000099 1756675290.833456 mapread 0x03c79b thru 0x03d9fa (0x1260 bytes) 000100 1756675290.836135 MIRROR_RESYNC 4 mirrors 000103 1756675291.070528 mapread 0x012bf3 thru 0x01fdcf (0xd1dd bytes) 000104 1756675291.077880 mapwrite 0x00a355 thru 0x00b26d (0xf19 bytes) 000105 1756675291.129577 MIRROR_EXTEND to 5 mirrors 000107 1756675291.302317 trunc from 0x03d9fb to 0x035815 (0x81e5 bytes) 000108 1756675291.320869 write 0x020f61 thru 0x02b30e (0xa3ae bytes) 000110 1756675291.324748 mapread 0x00ab23 thru 0x00b6a8 (0xb86 bytes) 000112 1756675291.328018 write 0x027eea thru 0x02bf51 (0x4068 bytes) 000113 1756675291.328606 trunc from 0x035816 to 0x027614 (0xe201 bytes) 000114 1756675291.335450 write 0x02f218 thru 0x035d83 (0x6b6c bytes) 000116 1756675291.338980 MIRROR_EXTEND to 6 mirrors 000120 1756675291.538063 read 0x03473d thru 0x035d83 (0x1647 bytes) 000121 1756675291.543259 mapwrite 0x025214 thru 0x02c5d4 (0x73c1 bytes) 000125 1756675291.606134 write 0x0392c0 thru 0x03ffff (0x6d40 bytes) 000126 1756675291.606999 read 0x0323e5 thru 0x03ffff (0xdc1b bytes) 000128 1756675291.610788 read 0x0062a2 thru 0x007736 (0x1495 bytes) 000130 1756675291.611466 trunc from 0x040000 to 0x020823 (0x1f7dc bytes) 000131 1756675291.617360 trunc from 0x020824 to 0x00061e (0x20205 bytes) 000132 1756675291.623616 MIRROR_RESYNC 6 mirrors 000136 1756675291.900403 MIRROR_RESYNC 6 mirrors 000138 1756675291.964411 mapwrite 0x02033f thru 0x022794 (0x2456 bytes) 000139 1756675292.038610 mapwrite 0x02bd43 thru 0x02fe43 (0x4101 bytes) 000140 1756675292.060575 write 0x01b452 thru 0x01d563 (0x2112 bytes) 000142 1756675292.061599 mapwrite 0x010c75 thru 0x011448 (0x7d4 bytes) 000145 1756675292.076733 mapwrite 0x03b94b thru 0x03ffff (0x46b5 bytes) 000146 1756675292.098884 MIRROR_EXTEND to 7 mirrors 000148 1756675292.254580 read 0x03d456 thru 0x03ffff (0x2baa bytes) 000150 1756675292.259679 mapwrite 0x001d95 thru 0x0099a0 (0x7c0c bytes) 000157 1756675292.315422 read 0x018690 thru 0x021092 (0x8a03 bytes) 000158 1756675292.318678 trunc from 0x040000 to 0x02d95d (0x126a2 bytes) 000159 1756675292.329004 trunc from 0x02d95e to 0x026e82 (0x6adb bytes) 000163 1756675292.334450 write 0x015168 thru 0x021e17 (0xccb0 bytes) 000164 1756675292.335367 mapread 0x0106d7 thru 0x01a65f (0x9f89 bytes) 000165 1756675292.335956 trunc from 0x026e83 to 0x03779f (0x1091d bytes) 000168 1756675292.341076 mapread 0x0112e6 thru 0x015db8 (0x4ad3 bytes) 000170 1756675292.341916 mapwrite 0x0282ce thru 0x035f41 (0xdc74 bytes) 000171 1756675292.378509 mapwrite 0x039974 thru 0x03ffff (0x668c bytes) 000172 1756675292.406878 mapwrite 0x017066 thru 0x019d76 (0x2d11 bytes) 000174 1756675292.419676 trunc from 0x040000 to 0x00aa6d (0x35592 bytes) 000179 1756675292.426047 read 0x005761 thru 0x00aa6d (0x530d bytes) 000181 1756675292.426699 read 0x00099b thru 0x003840 (0x2ea6 bytes) 000182 1756675292.427057 write 0x034271 thru 0x0367d9 (0x2569 bytes) 000186 1756675292.428502 trunc from 0x0367da to 0x02e304 (0x84d5 bytes) 000189 1756675292.433457 mapwrite 0x032f2c thru 0x03e8ae (0xb983 bytes) 000190 1756675292.456737 write 0x01d62a thru 0x01fcc5 (0x269c bytes) 000191 1756675292.457352 mapwrite 0x00f981 thru 0x015e01 (0x6481 bytes) 000193 1756675292.473086 mapread 0x019a04 thru 0x01d72a (0x3d27 bytes) 000195 1756675292.475649 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000201 1756675292.534362 read 0x0384db thru 0x03e8ae (0x63d4 bytes) 000205 1756675292.537777 mapread 0x00e6aa thru 0x014fd8 (0x692f bytes) 000206 1756675292.543201 trunc from 0x03e8af to 0x01eefe (0x1f9b0 bytes) 000208 1756675292.548488 mapread 0x016185 thru 0x01eefe (0x8d7a bytes) 000209 1756675292.550957 write 0x0390dc thru 0x03cc16 (0x3b3b bytes) 000213 1756675292.552514 trunc from 0x03cc17 to 0x020bf3 (0x1c023 bytes) 000216 1756675292.557036 mapwrite 0x033c33 thru 0x03ffff (0xc3cd bytes) 000220 1756675292.582577 read 0x0095de thru 0x009ae4 (0x507 bytes) 000222 1756675292.584814 MIRROR_RESYNC 6 mirrors 000223 1756675292.839082 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000225 1756675292.938288 MIRROR_EXTEND to 6 mirrors 000226 1756675293.117702 trunc from 0x040000 to 0x026730 (0x198cf bytes) 000227 1756675293.149952 read 0x00090e thru 0x00400b (0x36fe bytes) 000228 1756675293.152772 mapwrite 0x023877 thru 0x02e1ef (0xa979 bytes) 000229 1756675293.180684 read 0x026105 thru 0x02e1ef (0x80eb bytes) 000231 1756675293.182717 MIRROR_EXTEND to 7 mirrors 000232 1756675293.318063 write 0x0235b9 thru 0x025b11 (0x2559 bytes) 000233 1756675293.339672 mapwrite 0x02e60a thru 0x03c302 (0xdcf9 bytes) 000235 1756675293.369323 write 0x00c024 thru 0x019e3e (0xde1b bytes) 000239 1756675293.375020 write 0x035247 thru 0x03a525 (0x52df bytes) 000240 1756675293.375638 mapwrite 0x00a8d8 thru 0x014c52 (0xa37b bytes) 000243 1756675293.393265 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000246 1756675293.445628 read 0x0285b1 thru 0x034162 (0xbbb2 bytes) 000247 1756675293.448998 read 0x034e25 thru 0x03c302 (0x74de bytes) 000248 1756675293.452496 write 0x01a677 thru 0x01badc (0x1466 bytes) 000250 1756675293.453507 read 0x03aeb1 thru 0x03c302 (0x1452 bytes) 000251 1756675293.453901 trunc from 0x03c303 to 0x002dc0 (0x39542 bytes) 000253 1756675293.459731 trunc from 0x002dc1 to 0x005190 (0x23d0 bytes) 000254 1756675293.463629 read 0x00032a thru 0x001085 (0xd5c bytes) 000255 1756675293.464026 mapread 0x00254b thru 0x005190 (0x2c46 bytes) 000256 1756675293.466169 mapwrite 0x001b88 thru 0x00ae18 (0x9291 bytes) 000257 1756675293.490539 trunc from 0x00ae19 to 0x0263f7 (0x1b5df bytes) 000259 1756675293.496793 read 0x00d29f thru 0x018876 (0xb5d8 bytes) 000260 1756675293.500832 mapread 0x022a10 thru 0x0263f7 (0x39e8 bytes) 000261 1756675293.501607 mapread 0x010756 thru 0x0141ea (0x3a95 bytes) 000262 1756675293.502359 trunc from 0x0263f8 to 0x00c4ae (0x19f49 bytes) 000266 1756675293.509199 mapread 0x00a311 thru 0x00c4ae (0x219e bytes) 000267 1756675293.509951 read 0x00900d thru 0x00c4ae (0x34a2 bytes) 000268 1756675293.510477 trunc from 0x00c4af to 0x015ab7 (0x9609 bytes) 000270 1756675293.515878 mapwrite 0x023b03 thru 0x02c81a (0x8d18 bytes) 000273 1756675293.542073 MIRROR_EXTEND to 7 mirrors 000274 1756675293.686966 mapread 0x020fed thru 0x0285c0 (0x75d4 bytes) 000275 1756675293.694423 read 0x025e3c thru 0x025e9b (0x60 bytes) 000277 1756675293.697049 mapread 0x001a16 thru 0x010048 (0xe633 bytes) 000280 1756675293.706209 mapwrite 0x01b2ca thru 0x02248f (0x71c6 bytes) 000281 1756675293.760334 mapwrite 0x026f89 thru 0x036f75 (0xffed bytes) 000282 1756675293.794627 read 0x002644 thru 0x0118a5 (0xf262 bytes) 000285 1756675293.799631 mapread 0x013d37 thru 0x020905 (0xcbcf bytes) 000288 1756675293.801160 write 0x0178fa thru 0x019471 (0x1b78 bytes) 000289 1756675293.803386 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000290 1756675293.872514 mapread 0x01aebd thru 0x023834 (0x8978 bytes) 000291 1756675293.879432 mapwrite 0x035fe7 thru 0x0389eb (0x2a05 bytes) 000293 1756675293.904858 write 0x034c6c thru 0x03de75 (0x920a bytes) 000294 1756675293.905830 mapread 0x00c4c9 thru 0x00cf6d (0xaa5 bytes) 000296 1756675293.909850 mapread 0x022b4a thru 0x026e99 (0x4350 bytes) 000297 1756675293.912930 write 0x0187f9 thru 0x019bc0 (0x13c8 bytes) 000298 1756675293.913472 mapwrite 0x03481c thru 0x037f4f (0x3734 bytes) 000299 1756675293.934554 trunc from 0x03de76 to 0x0208fc (0x1d579 bytes) 000300 1756675293.940244 write 0x015881 thru 0x020382 (0xab02 bytes) 000301 1756675293.940966 mapwrite 0x009a25 thru 0x010291 (0x686d bytes) 000303 1756675293.958665 mapwrite 0x008823 thru 0x00bc4d (0x342b bytes) 000304 1756675293.971363 mapread 0x01f8d9 thru 0x0208fc (0x1024 bytes) 000305 1756675293.971905 write 0x02c290 thru 0x02f5ef (0x3360 bytes) 000307 1756675293.972898 read 0x00c552 thru 0x00e370 (0x1e1f bytes) 000308 1756675293.973446 trunc from 0x02f5f0 to 0x03821e (0x8c2f bytes) 000309 1756675293.978829 trunc from 0x03821f to 0x01bcc4 (0x1c55a bytes) 000312 1756675293.984944 mapwrite 0x0288e2 thru 0x03568d (0xcdac bytes) 000315 1756675294.011348 mapread 0x0340dd thru 0x03568d (0x15b1 bytes) 000317 1756675294.012747 trunc from 0x03568e to 0x00a03c (0x2b651 bytes) 000318 1756675294.017639 trunc from 0x00a03d to 0x00263d (0x79ff bytes) 000321 1756675294.022523 read 0x001761 thru 0x00263d (0xedd bytes) 000323 1756675294.026169 MIRROR_EXTEND to 7 mirrors 000326 1756675294.167965 write 0x001679 thru 0x003c47 (0x25cf bytes) 000327 1756675294.188383 trunc from 0x003c48 to 0x009ee8 (0x62a1 bytes) 000329 1756675294.200985 MIRROR_RESYNC 7 mirrors 000330 1756675294.468920 trunc from 0x009ee9 to 0x02a5c3 (0x206db bytes) 000331 1756675294.497172 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000338 1756675294.555611 trunc from 0x02a5c4 to 0x0057ec (0x24dd7 bytes) 000340 1756675294.560603 trunc from 0x0057ed to 0x03e757 (0x38f6b bytes) 000345 1756675294.568448 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000348 1756675294.623776 mapwrite 0x02b013 thru 0x02cd8d (0x1d7b bytes) 000349 1756675294.639306 trunc from 0x03e758 to 0x00a25a (0x344fd bytes) 000351 1756675294.645543 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000353 1756675294.702331 mapread 0x000302 thru 0x003a9c (0x379b bytes) 000354 1756675294.709528 mapread 0x002176 thru 0x003bb6 (0x1a41 bytes) 000355 1756675294.710115 read 0x008e78 thru 0x00a25a (0x13e3 bytes) 000359 1756675294.712057 write 0x035ddd thru 0x03ffff (0xa223 bytes) 000362 1756675294.715049 MIRROR_EXTEND to 5 mirrors 000363 1756675294.891869 write 0x02cca6 thru 0x030e54 (0x41af bytes) 000364 1756675294.920057 read 0x01fab5 thru 0x02ad0e (0xb25a bytes) 000365 1756675294.923445 mapwrite 0x0124a7 thru 0x01996d (0x74c7 bytes) 000366 1756675294.961293 mapwrite 0x022df2 thru 0x027c83 (0x4e92 bytes) 000367 1756675294.977801 trunc from 0x040000 to 0x038759 (0x78a6 bytes) 000368 1756675294.989490 read 0x00b642 thru 0x00fb99 (0x4558 bytes) 000370 1756675294.990258 trunc from 0x03875a to 0x0100b7 (0x286a2 bytes) 000371 1756675294.995589 mapread 0x001d41 thru 0x009f3e (0x81fe bytes) 000372 1756675294.998893 read 0x00f319 thru 0x0100b7 (0xd9f bytes) 000373 1756675294.999341 trunc from 0x0100b8 to 0x02bdc2 (0x1bd0b bytes) 000375 1756675295.004253 mapread 0x02540c thru 0x02bdc2 (0x69b7 bytes) 000376 1756675295.008979 MIRROR_EXTEND to 6 mirrors 000378 1756675295.178049 read 0x00b70f thru 0x01168d (0x5f7f bytes) 000379 1756675295.182655 mapread 0x0016d7 thru 0x00cb8d (0xb4b7 bytes) 000381 1756675295.194189 read 0x010cd6 thru 0x0127b3 (0x1ade bytes) 000383 1756675295.201251 mapwrite 0x00edb0 thru 0x0140b3 (0x5304 bytes) 000384 1756675295.261845 trunc from 0x02bdc3 to 0x02c91f (0xb5d bytes) 000387 1756675295.269132 trunc from 0x02c920 to 0x03965f (0xcd40 bytes) 000389 1756675295.273979 read 0x01dac4 thru 0x02421d (0x675a bytes) 000390 1756675295.278004 write 0x035a6b thru 0x03ffff (0xa595 bytes) 000391 1756675295.278787 trunc from 0x040000 to 0x031c52 (0xe3ad bytes) 000393 1756675295.284187 read 0x01f055 thru 0x021b92 (0x2b3e bytes) 000395 1756675295.285284 trunc from 0x031c53 to 0x013e24 (0x1de2e bytes) 000396 1756675295.290058 write 0x03edeb thru 0x03ffff (0x1215 bytes) 000400 1756675295.291623 write 0x0181fc thru 0x01d748 (0x554d bytes) 000401 1756675295.296539 MIRROR_EXTEND to 7 mirrors 000403 1756675295.459379 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000404 1756675295.519557 trunc from 0x040000 to 0x033782 (0xc87d bytes) 000405 1756675295.531441 write 0x00c6e1 thru 0x015031 (0x8951 bytes) 000406 1756675295.534653 trunc from 0x033783 to 0x00c9e2 (0x26da0 bytes) 000408 1756675295.539843 trunc from 0x00c9e3 to 0x025c15 (0x19233 bytes) 000409 1756675295.544038 mapread 0x00186e thru 0x0090c9 (0x785c bytes) 000410 1756675295.548231 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000411 1756675295.603903 mapread 0x010b6e thru 0x0197e2 (0x8c75 bytes) 000412 1756675295.609955 mapread 0x0038cd thru 0x00655a (0x2c8e bytes) 000413 1756675295.612906 mapread 0x00009d thru 0x005d2f (0x5c93 bytes) 000414 1756675295.615426 mapwrite 0x01777c thru 0x01d339 (0x5bbe bytes) 000416 1756675295.633805 mapwrite 0x032798 thru 0x03c5a3 (0x9e0c bytes) 000417 1756675295.659953 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000418 1756675295.713810 read 0x004182 thru 0x00475d (0x5dc bytes) 000419 1756675295.715616 read 0x030216 thru 0x035aee (0x58d9 bytes) 000421 1756675295.720010 read 0x02dcde thru 0x03837a (0xa69d bytes) 000422 1756675295.720412 read 0x0393bc thru 0x03c5a3 (0x31e8 bytes) 000423 1756675295.720778 mapwrite 0x00ce0b thru 0x016541 (0x9737 bytes) 000424 1756675295.738524 trunc from 0x03c5a4 to 0x01e27d (0x1e326 bytes) 000425 1756675295.743535 mapwrite 0x019ca8 thru 0x019d7c (0xd5 bytes) 000427 1756675295.757450 read 0x012efc thru 0x01ab91 (0x7c96 bytes) 000430 1756675295.760144 MIRROR_RESYNC 4 mirrors 000432 1756675295.974209 mapwrite 0x032459 thru 0x039c56 (0x77fe bytes) 000433 1756675296.024293 MIRROR_RESYNC 4 mirrors 000434 1756675296.226816 mapwrite 0x023662 thru 0x029079 (0x5a18 bytes) 000436 1756675296.279173 mapwrite 0x0309f2 thru 0x03a607 (0x9c16 bytes) 000437 1756675296.308090 MIRROR_EXTEND to 5 mirrors 000443 1756675296.461780 mapread 0x001432 thru 0x00285c (0x142b bytes) 000445 1756675296.467785 read 0x02709b thru 0x02bde3 (0x4d49 bytes) 000446 1756675296.471036 write 0x037f60 thru 0x03bdee (0x3e8f bytes) 000448 1756675296.494298 read 0x014abe thru 0x0197d7 (0x4d1a bytes) 000449 1756675296.497509 mapwrite 0x0016f8 thru 0x004c30 (0x3539 bytes) 000452 1756675296.514925 mapread 0x02d51c thru 0x030209 (0x2cee bytes) 000453 1756675296.515459 write 0x0010a6 thru 0x00563a (0x4595 bytes) 000456 1756675296.516611 read 0x014a52 thru 0x017c1a (0x31c9 bytes) 000457 1756675296.516983 mapwrite 0x028d5c thru 0x0316fe (0x89a3 bytes) 000458 1756675296.534420 mapread 0x00197f thru 0x00bd40 (0xa3c2 bytes) 000459 1756675296.535013 mapread 0x031143 thru 0x03bdee (0xacac bytes) 000460 1756675296.535559 trunc from 0x03bdef to 0x018aae (0x23340 bytes) 000461 1756675296.546537 trunc from 0x018aaf to 0x010a53 (0x805b bytes) 000462 1756675296.552253 MIRROR_EXTEND to 6 mirrors 000463 1756675296.711022 mapread 0x00e19e thru 0x010a53 (0x28b6 bytes) 000465 1756675296.720622 read 0x008229 thru 0x00a0de (0x1eb6 bytes) 000466 1756675296.723517 mapwrite 0x035c55 thru 0x036048 (0x3f4 bytes) 000467 1756675296.767471 trunc from 0x036049 to 0x03694f (0x907 bytes) 000468 1756675296.772699 mapwrite 0x008107 thru 0x00c6af (0x45a9 bytes) 000469 1756675296.789012 write 0x0302bc thru 0x03f46f (0xf1b4 bytes) 000471 1756675296.792359 MIRROR_RESYNC 6 mirrors 000472 1756675297.023550 mapread 0x01184c thru 0x01872e (0x6ee3 bytes) 000475 1756675297.029588 mapread 0x003837 thru 0x01154c (0xdd16 bytes) 000476 1756675297.032478 mapread 0x03965d thru 0x03ba26 (0x23ca bytes) 000478 1756675297.035312 mapwrite 0x02804e thru 0x03568d (0xd640 bytes) 000482 1756675297.093323 mapwrite 0x00d03d thru 0x010c12 (0x3bd6 bytes) 000483 1756675297.107984 mapread 0x00eb1c thru 0x012da1 (0x4286 bytes) 000484 1756675297.108517 trunc from 0x03f470 to 0x0239e4 (0x1ba8b bytes) 000485 1756675297.119458 mapread 0x012de6 thru 0x013a8f (0xcaa bytes) 000486 1756675297.120322 mapread 0x02374d thru 0x0239e4 (0x298 bytes) 000487 1756675297.121176 read 0x001c17 thru 0x00a6c7 (0x8ab1 bytes) 000488 1756675297.123717 read 0x01dcec thru 0x0239e4 (0x5cf9 bytes) 000490 1756675297.127322 trunc from 0x0239e5 to 0x030409 (0xca25 bytes) 000492 1756675297.131892 mapwrite 0x014b21 thru 0x021a9d (0xcf7d bytes) 000493 1756675297.147033 trunc from 0x03040a to 0x021a3b (0xe9ce bytes) 000494 1756675297.151408 read 0x021157 thru 0x021a3b (0x8e5 bytes) 000495 1756675297.151773 write 0x01c66c thru 0x028213 (0xbba8 bytes) 000496 1756675297.152620 mapwrite 0x03e6a8 thru 0x03ffff (0x1958 bytes) 000497 1756675297.174188 MIRROR_EXTEND to 7 mirrors 000498 1756675297.322929 mapwrite 0x030a9a thru 0x03df5a (0xd4c1 bytes) 000499 1756675297.383624 mapwrite 0x010956 thru 0x015c38 (0x52e3 bytes) 000500 1756675297.399512 trunc from 0x040000 to 0x0122d6 (0x2dd29 bytes) 000501 1756675297.411318 mapwrite 0x013146 thru 0x01e4c7 (0xb382 bytes) 000502 1756675297.437022 mapread 0x0191b0 thru 0x01e4c7 (0x5318 bytes) 000503 1756675297.437765 read 0x0193e2 thru 0x01e4c7 (0x50e6 bytes) 000505 1756675297.439808 MIRROR_EXTEND to 8 mirrors 000507 1756675297.590493 mapwrite 0x01130b thru 0x013347 (0x203d bytes) 000508 1756675297.640872 write 0x03a9cd thru 0x03ffff (0x5633 bytes) 000511 1756675297.642814 mapwrite 0x017a15 thru 0x01bcaf (0x429b bytes) 000512 1756675297.658602 trunc from 0x040000 to 0x026590 (0x19a6f bytes) 000514 1756675297.669057 read 0x0195bc thru 0x0201a0 (0x6be5 bytes) 000518 1756675297.671086 read 0x01c869 thru 0x024f84 (0x871c bytes) 000519 1756675297.674336 mapwrite 0x01078d thru 0x016034 (0x58a8 bytes) 000520 1756675297.688523 read 0x018e08 thru 0x01f1da (0x63d3 bytes) 000521 1756675297.689009 trunc from 0x026591 to 0x00eda9 (0x177e7 bytes) 000523 1756675297.695442 MIRROR_EXTEND to 9 mirrors 000524 1756675297.841640 mapwrite 0x03a089 thru 0x03ffff (0x5f77 bytes) 000527 1756675297.887909 mapread 0x010ca5 thru 0x01ec53 (0xdfaf bytes) 000528 1756675297.893893 read 0x02a135 thru 0x033a88 (0x9954 bytes) 000529 1756675297.897046 write 0x01df7f thru 0x026f65 (0x8fe7 bytes) 000531 1756675297.899947 mapwrite 0x020a65 thru 0x022286 (0x1822 bytes) 000534 1756675297.913636 mapwrite 0x011f35 thru 0x013788 (0x1854 bytes) 000535 1756675297.928337 mapwrite 0x03e4bb thru 0x03ffff (0x1b45 bytes) 000537 1756675297.941583 read 0x020f0d thru 0x02b5cf (0xa6c3 bytes) 000538 1756675297.945000 trunc from 0x040000 to 0x015238 (0x2adc7 bytes) 000542 1756675297.952291 mapread 0x00820a thru 0x012293 (0xa08a bytes) 000543 1756675297.952953 trunc from 0x015239 to 0x03360d (0x1e3d5 bytes) 000547 1756675297.958848 read 0x00e674 thru 0x0158af (0x723c bytes) 000548 1756675297.959342 mapread 0x010286 thru 0x0166c3 (0x643e bytes) 000551 1756675297.964594 write 0x01e421 thru 0x029caf (0xb88f bytes) 000552 1756675297.965920 mapwrite 0x01dd1f thru 0x02ac11 (0xcef3 bytes) 000555 1756675297.983191 mapwrite 0x005023 thru 0x01140d (0xc3eb bytes) 000556 1756675297.999769 mapwrite 0x031d08 thru 0x03e70a (0xca03 bytes) 000557 1756675298.023412 trunc from 0x03e70b to 0x006d14 (0x379f6 bytes) 000559 1756675298.028860 mapread 0x003ea5 thru 0x005d11 (0x1e6d bytes) 000560 1756675298.029395 trunc from 0x006d15 to 0x028c32 (0x21f1e bytes) 000561 1756675298.033797 mapwrite 0x01eddb thru 0x02dcec (0xef12 bytes) 000562 1756675298.076596 mapread 0x01d630 thru 0x020742 (0x3113 bytes) 000564 1756675298.079005 read 0x024610 thru 0x02dcec (0x96dd bytes) 000567 1756675298.079947 trunc from 0x02dced to 0x016c93 (0x17059 bytes) 000570 1756675298.085361 mapread 0x002bdf thru 0x0082bc (0x56de bytes) 000572 1756675298.088921 read 0x00c7ff thru 0x01189d (0x509f bytes) 000573 1756675298.089316 mapwrite 0x032fa5 thru 0x03a7c7 (0x7823 bytes) 000574 1756675298.116561 read 0x008058 thru 0x0104dc (0x8485 bytes) 000575 1756675298.118443 MIRROR_RESYNC 9 mirrors 000576 1756675298.437927 mapread 0x034aad thru 0x03a7c7 (0x5d1b bytes) 000580 1756675298.443880 mapwrite 0x03c07d thru 0x03ffff (0x3f83 bytes) 000581 1756675298.491754 read 0x032fd8 thru 0x0343ed (0x1416 bytes) 000582 1756675298.495845 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000583 1756675298.559616 mapread 0x00eb64 thru 0x00f2b9 (0x756 bytes) 000585 1756675298.563538 mapwrite 0x036ddc thru 0x037af1 (0xd16 bytes) 000591 1756675298.579920 read 0x015654 thru 0x020c52 (0xb5ff bytes) 000592 1756675298.585124 MIRROR_RESYNC 8 mirrors 000593 1756675298.902387 read 0x006a05 thru 0x014cc9 (0xe2c5 bytes) 000594 1756675298.905213 write 0x001083 thru 0x009929 (0x88a7 bytes) 000595 1756675298.926336 trunc from 0x040000 to 0x00301e (0x3cfe1 bytes) 000596 1756675298.937878 MIRROR_EXTEND to 9 mirrors 000597 1756675299.097124 mapwrite 0x0387ca thru 0x03b2bb (0x2af2 bytes) 000598 1756675299.139184 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000601 1756675299.197746 MIRROR_EXTEND to 9 mirrors 000603 1756675299.364063 mapread 0x018c8c thru 0x027ff6 (0xf36b bytes) 000604 1756675299.374074 mapwrite 0x03dc1d thru 0x03ec3c (0x1020 bytes) 000605 1756675299.412702 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000610 1756675299.480843 read 0x006689 thru 0x0149d0 (0xe348 bytes) 000617 1756675299.486075 read 0x002761 thru 0x003420 (0xcc0 bytes) 000618 1756675299.492070 MIRROR_RESYNC 8 mirrors 000619 1756675299.804347 write 0x034bf1 thru 0x038e96 (0x42a6 bytes) 000621 1756675299.822475 write 0x021465 thru 0x030cf0 (0xf88c bytes) 000622 1756675299.826647 write 0x03adb0 thru 0x03ffff (0x5250 bytes) 000625 1756675299.831320 MIRROR_RESYNC 8 mirrors 000630 1756675300.103036 mapread 0x013941 thru 0x014eb6 (0x1576 bytes) 000634 1756675300.106328 trunc from 0x040000 to 0x002b2d (0x3d4d2 bytes) 000635 1756675300.125717 read 0x000cb6 thru 0x002b2d (0x1e78 bytes) 000636 1756675300.127596 trunc from 0x002b2e to 0x027f4e (0x25421 bytes) 000637 1756675300.132634 write 0x030cbd thru 0x034fed (0x4331 bytes) 000638 1756675300.133375 trunc from 0x034fee to 0x013cf7 (0x212f6 bytes) 000639 1756675300.137236 read 0x00de2e thru 0x012748 (0x491b bytes) 000640 1756675300.139803 mapwrite 0x001fde thru 0x00e0b2 (0xc0d5 bytes) 000641 1756675300.154512 write 0x013fb7 thru 0x01728e (0x32d8 bytes) 000643 1756675300.157429 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000644 1756675300.212864 write 0x02c84c thru 0x02ed80 (0x2535 bytes) 000646 1756675300.214320 write 0x011a4b thru 0x017ab3 (0x6069 bytes) 000650 1756675300.218384 mapread 0x02a8a0 thru 0x02ed80 (0x44e1 bytes) 000651 1756675300.220888 mapread 0x007543 thru 0x012eb3 (0xb971 bytes) 000652 1756675300.223720 write 0x01409d thru 0x01a01d (0x5f81 bytes) 000653 1756675300.226396 mapread 0x0023ee thru 0x00306b (0xc7e bytes) 000655 1756675300.227274 write 0x02eb2a thru 0x03643e (0x7915 bytes) 000656 1756675300.229342 MIRROR_EXTEND to 8 mirrors 000659 1756675300.378918 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000660 1756675300.434571 MIRROR_RESYNC 7 mirrors 000661 1756675300.707316 MIRROR_EXTEND to 8 mirrors 000662 1756675300.863115 write 0x013443 thru 0x01da15 (0xa5d3 bytes) 000663 1756675300.884553 trunc from 0x03643f to 0x00197c (0x34ac2 bytes) 000664 1756675300.893877 write 0x02d45a thru 0x03b132 (0xdcd9 bytes) 000669 1756675300.896015 read 0x02491f thru 0x02cae1 (0x81c3 bytes) 000670 1756675300.898779 mapread 0x003c4f thru 0x005c13 (0x1fc5 bytes) 000672 1756675300.902035 trunc from 0x03b133 to 0x02e34c (0xcde6 bytes) 000674 1756675300.906611 trunc from 0x02e34d to 0x017aae (0x1689e bytes) 000675 1756675300.910747 write 0x00457d thru 0x00f7cf (0xb253 bytes) 000679 1756675300.912351 trunc from 0x017aaf to 0x018eb7 (0x1409 bytes) 000680 1756675300.916763 write 0x01c478 thru 0x01cf94 (0xb1d bytes) 000681 1756675300.917319 mapread 0x011738 thru 0x01a75f (0x9028 bytes) 000683 1756675300.921631 trunc from 0x01cf95 to 0x011ec1 (0xb0d3 bytes) 000684 1756675300.925918 mapread 0x009e0a thru 0x00ecf2 (0x4ee9 bytes) 000685 1756675300.926492 read 0x007508 thru 0x011ec1 (0xa9ba bytes) 000686 1756675300.926903 write 0x02fce8 thru 0x038495 (0x87ae bytes) 000687 1756675300.927708 write 0x032f79 thru 0x0396f7 (0x677f bytes) 000688 1756675300.929880 MIRROR_RESYNC 8 mirrors 000689 1756675301.201512 write 0x00a5e7 thru 0x01587a (0xb294 bytes) 000691 1756675301.220418 mapread 0x0369ce thru 0x0396f7 (0x2d2a bytes) 000692 1756675301.224510 MIRROR_EXTEND to 9 mirrors 000697 1756675301.372983 MIRROR_RESYNC 9 mirrors 000698 1756675301.641914 write 0x02d32f thru 0x039ba0 (0xc872 bytes) 000699 1756675301.668188 trunc from 0x039ba1 to 0x007a28 (0x32178 bytes) 000701 1756675301.678691 read 0x001eca thru 0x007a28 (0x5b5f bytes) 000704 1756675301.681840 mapread 0x000716 thru 0x007a28 (0x7313 bytes) 000705 1756675301.684138 mapwrite 0x008b5a thru 0x00b04a (0x24f1 bytes) 000706 1756675301.703642 mapwrite 0x035046 thru 0x03ffff (0xafba bytes) 000710 1756675301.725988 mapwrite 0x003a74 thru 0x00f019 (0xb5a6 bytes) 000712 1756675301.743567 mapwrite 0x0256d4 thru 0x0290f0 (0x3a1d bytes) 000713 1756675301.760003 read 0x023ed0 thru 0x032e7c (0xefad bytes) 000714 1756675301.760529 read 0x0281cc thru 0x028493 (0x2c8 bytes) 000715 1756675301.761022 write 0x02e592 thru 0x02e670 (0xdf bytes) 000716 1756675301.761700 read 0x01ed98 thru 0x01fe4b (0x10b4 bytes) 000717 1756675301.762146 mapread 0x008f1b thru 0x00a164 (0x124a bytes) 000718 1756675301.762742 trunc from 0x040000 to 0x020699 (0x1f966 bytes) 000719 1756675301.767961 mapwrite 0x02465a thru 0x03305f (0xea06 bytes) 000720 1756675301.790038 mapwrite 0x037722 thru 0x03934e (0x1c2d bytes) 000722 1756675301.809936 MIRROR_RESYNC 9 mirrors 000726 1756675302.109361 mapwrite 0x016021 thru 0x01dc8f (0x7c6f bytes) 000727 1756675302.169525 MIRROR_RESYNC 9 mirrors 000729 1756675302.512584 trunc from 0x03934f to 0x020b8e (0x187c0 bytes) 000731 1756675302.536786 read 0x0085c5 thru 0x00f052 (0x6a8e bytes) 000732 1756675302.539703 read 0x01a132 thru 0x020789 (0x6658 bytes) 000733 1756675302.542580 mapwrite 0x012216 thru 0x012f20 (0xd0b bytes) 000734 1756675302.558407 mapwrite 0x02222b thru 0x023c65 (0x1a3b bytes) 000737 1756675302.580416 trunc from 0x023c66 to 0x003f0e (0x1fd57 bytes) 000738 1756675302.585740 read 0x00076f thru 0x003f0e (0x37a0 bytes) 000740 1756675302.586458 write 0x03c5e7 thru 0x03ffff (0x3a19 bytes) 000742 1756675302.587498 mapwrite 0x0032bb thru 0x00d4c0 (0xa206 bytes) 000743 1756675302.609335 mapread 0x025fe4 thru 0x02caf2 (0x6b0f bytes) 000744 1756675302.610017 trunc from 0x040000 to 0x011e93 (0x2e16c bytes) 000745 1756675302.616192 mapread 0x00a886 thru 0x00ac1e (0x399 bytes) 000747 1756675302.617030 mapwrite 0x02f6ed thru 0x034e41 (0x5755 bytes) 000748 1756675302.646380 mapread 0x01b8a1 thru 0x023068 (0x77c8 bytes) 000749 1756675302.658717 mapread 0x0243e1 thru 0x02872f (0x434f bytes) 000750 1756675302.665872 trunc from 0x034e42 to 0x031777 (0x36ca bytes) 000751 1756675302.671054 mapread 0x00fe18 thru 0x01e636 (0xe81f bytes) 000753 1756675302.683645 trunc from 0x031778 to 0x01535b (0x1c41c bytes) 000754 1756675302.688564 mapwrite 0x03f90e thru 0x03ffff (0x6f2 bytes) 000755 1756675302.708658 read 0x0218be thru 0x02a560 (0x8ca3 bytes) 000756 1756675302.712491 trunc from 0x040000 to 0x02f8a9 (0x10756 bytes) 000763 1756675302.718938 write 0x023a14 thru 0x02ce36 (0x9423 bytes) 000764 1756675302.719683 read 0x0133d9 thru 0x017514 (0x413c bytes) 000767 1756675302.722224 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000768 1756675302.784357 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000769 1756675302.836494 read 0x027226 thru 0x02ba84 (0x485f bytes) 000773 1756675302.839780 mapwrite 0x0171fe thru 0x025dd2 (0xebd5 bytes) 000774 1756675302.863647 read 0x0175fc thru 0x022879 (0xb27e bytes) 000775 1756675302.864131 read 0x02903a thru 0x02f8a9 (0x6870 bytes) 000776 1756675302.867436 read 0x026e66 thru 0x02f8a9 (0x8a44 bytes) 000779 1756675302.868402 trunc from 0x02f8aa to 0x011d39 (0x1db70 bytes) 000780 1756675302.873316 mapread 0x00e8af thru 0x011d39 (0x348b bytes) 000781 1756675302.873849 mapread 0x004d3d thru 0x011d39 (0xcffd bytes) 000782 1756675302.874407 mapread 0x0048d3 thru 0x011d39 (0xd467 bytes) 000783 1756675302.874928 mapread 0x001f94 thru 0x00b46b (0x94d8 bytes) 000788 1756675302.876621 write 0x0298d9 thru 0x03684b (0xcf73 bytes) 000789 1756675302.877544 mapwrite 0x00a5ca thru 0x0172ca (0xcd01 bytes) 000790 1756675302.897477 trunc from 0x03684c to 0x0236fd (0x1314e bytes) 000793 1756675302.903843 trunc from 0x0236fe to 0x030845 (0xd148 bytes) 000794 1756675302.909409 MIRROR_RESYNC 7 mirrors 000795 1756675303.189860 read 0x00cf8d thru 0x01a9e4 (0xda58 bytes) 000799 1756675303.194129 MIRROR_RESYNC 7 mirrors 000800 1756675303.251547 read 0x02e36b thru 0x030845 (0x24db bytes) 000802 1756675303.253655 write 0x036e37 thru 0x03aa82 (0x3c4c bytes) 000804 1756675303.270691 mapread 0x000d7e thru 0x001329 (0x5ac bytes) 000805 1756675303.274335 write 0x02e1ca thru 0x038fa3 (0xadda bytes) 000807 1756675303.276845 mapwrite 0x0316b1 thru 0x037eb8 (0x6808 bytes) 000809 1756675303.294337 trunc from 0x03aa83 to 0x01812b (0x22957 bytes) 000810 1756675303.300964 read 0x0000bb thru 0x001ec8 (0x1e0e bytes) 000812 1756675303.302084 mapread 0x012e79 thru 0x01812b (0x52b3 bytes) 000813 1756675303.305641 mapread 0x012721 thru 0x01812b (0x5a0b bytes) 000814 1756675303.307649 MIRROR_EXTEND to 8 mirrors 000816 1756675303.445291 mapread 0x006a7a thru 0x007773 (0xcfa bytes) 000819 1756675303.452758 mapwrite 0x003b8d thru 0x00f11f (0xb593 bytes) 000821 1756675303.505384 trunc from 0x01812c to 0x008e31 (0xf2fa bytes) 000823 1756675303.515965 trunc from 0x008e32 to 0x03bab3 (0x32c82 bytes) 000825 1756675303.520342 trunc from 0x03bab4 to 0x01cadd (0x1efd6 bytes) 000831 1756675303.526113 trunc from 0x01cade to 0x016ccf (0x5e0e bytes) 000832 1756675303.530196 write 0x02c6f5 thru 0x03b648 (0xef54 bytes) 000833 1756675303.531246 trunc from 0x03b649 to 0x01d188 (0x1e4c0 bytes) 000834 1756675303.535795 read 0x00e1b4 thru 0x00e438 (0x285 bytes) 000835 1756675303.537857 write 0x035e43 thru 0x037b11 (0x1ccf bytes) 000837 1756675303.538888 trunc from 0x037b12 to 0x017e65 (0x1fcac bytes) 000838 1756675303.543263 write 0x0331a6 thru 0x036307 (0x3162 bytes) 000839 1756675303.543901 read 0x019940 thru 0x025545 (0xbc06 bytes) 000840 1756675303.547258 mapread 0x00356b thru 0x00cb7e (0x9614 bytes) 000846 1756675303.549254 mapread 0x036066 thru 0x036307 (0x2a2 bytes) 000847 1756675303.549724 trunc from 0x036308 to 0x02a6aa (0xbc5d bytes) 000848 1756675303.554186 mapwrite 0x01b639 thru 0x01e844 (0x320c bytes) 000849 1756675303.566696 trunc from 0x02a6ab to 0x03647a (0xbdd0 bytes) 000850 1756675303.571463 trunc from 0x03647b to 0x036519 (0x9f bytes) 000854 1756675303.576745 trunc from 0x03651a to 0x0089b8 (0x2db61 bytes) 000856 1756675303.582205 read 0x000f1f thru 0x0089b8 (0x7a9a bytes) 000857 1756675303.582860 read 0x006fd6 thru 0x0089b8 (0x19e3 bytes) 000858 1756675303.584858 MIRROR_RESYNC 8 mirrors 000859 1756675303.871487 mapwrite 0x01ad4b thru 0x01b0df (0x395 bytes) 000864 1756675303.917317 mapread 0x001014 thru 0x004518 (0x3505 bytes) 000865 1756675303.920338 read 0x00f708 thru 0x01944e (0x9d47 bytes) 000870 1756675303.924484 write 0x01bbf9 thru 0x01bca0 (0xa8 bytes) 000872 1756675303.925578 write 0x038b4e thru 0x03ffff (0x74b2 bytes) 000874 1756675303.928584 MIRROR_EXTEND to 9 mirrors 000875 1756675304.098616 read 0x00128f thru 0x0084ff (0x7271 bytes) 000877 1756675304.104761 mapwrite 0x01dc6d thru 0x02a785 (0xcb19 bytes) 000878 1756675304.164825 trunc from 0x040000 to 0x02a0b6 (0x15f49 bytes) 000880 1756675304.172324 MIRROR_RESYNC 9 mirrors 000882 1756675304.452936 mapwrite 0x030be6 thru 0x0370ba (0x64d5 bytes) 000883 1756675304.488247 mapread 0x0096b8 thru 0x014b5b (0xb4a4 bytes) 000885 1756675304.493923 mapwrite 0x014bed thru 0x02430b (0xf71f bytes) 000886 1756675304.512648 trunc from 0x0370bb to 0x018a80 (0x1e63a bytes) 000887 1756675304.518298 read 0x014d71 thru 0x018a80 (0x3d10 bytes) 000890 1756675304.519223 write 0x006897 thru 0x00eb5d (0x82c7 bytes) 000891 1756675304.519878 trunc from 0x018a81 to 0x01bb5e (0x30de bytes) 000894 1756675304.525831 MIRROR_RESYNC 9 mirrors 000895 1756675304.792540 trunc from 0x01bb5f to 0x03b9d0 (0x1fe72 bytes) 000897 1756675304.810685 trunc from 0x03b9d1 to 0x038115 (0x38bb bytes) 000899 1756675304.816459 write 0x002a7a thru 0x00e64e (0xbbd5 bytes) 000900 1756675304.821307 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000901 1756675304.875686 mapwrite 0x02ae10 thru 0x03215d (0x734e bytes) 000902 1756675304.897701 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000903 1756675304.975414 read 0x01cfa7 thru 0x0220db (0x5135 bytes) 000904 1756675304.980541 trunc from 0x038116 to 0x0015ef (0x36b26 bytes) 000905 1756675304.989150 trunc from 0x0015f0 to 0x004182 (0x2b93 bytes) 000906 1756675304.996572 read 0x000e0e thru 0x004182 (0x3375 bytes) 000909 1756675305.002227 trunc from 0x004183 to 0x022b96 (0x1ea14 bytes) 000910 1756675305.010205 mapwrite 0x01554d thru 0x018f00 (0x39b4 bytes) 000911 1756675305.041042 mapread 0x0115a0 thru 0x0209bf (0xf420 bytes) 000913 1756675305.042864 write 0x010ad0 thru 0x0165e9 (0x5b1a bytes) 000914 1756675305.043896 trunc from 0x022b97 to 0x036fa7 (0x14411 bytes) 000916 1756675305.053340 MIRROR_EXTEND to 8 mirrors 000918 1756675305.197888 trunc from 0x036fa8 to 0x038e96 (0x1eef bytes) 000920 1756675305.227871 mapread 0x0217e7 thru 0x0307e5 (0xefff bytes) 000921 1756675305.236414 mapwrite 0x024037 thru 0x02ccd6 (0x8ca0 bytes) 000923 1756675305.256731 trunc from 0x038e97 to 0x003098 (0x35dfe bytes) 000924 1756675305.263517 write 0x0140fb thru 0x01d16c (0x9072 bytes) 000925 1756675305.264562 trunc from 0x01d16d to 0x032590 (0x15424 bytes) 000926 1756675305.269532 mapwrite 0x03562e thru 0x03ffff (0xa9d2 bytes) 000927 1756675305.297051 MIRROR_RESYNC 8 mirrors 000929 1756675305.557038 mapread 0x00a616 thru 0x00d882 (0x326d bytes) 000931 1756675305.562244 mapwrite 0x014529 thru 0x018175 (0x3c4d bytes) 000932 1756675305.607051 read 0x02af74 thru 0x02dbbf (0x2c4c bytes) 000933 1756675305.609431 trunc from 0x040000 to 0x0084a4 (0x37b5b bytes) 000934 1756675305.621172 MIRROR_EXTEND to 9 mirrors 000937 1756675305.763269 read 0x00158d thru 0x001b5b (0x5cf bytes) 000938 1756675305.767161 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000939 1756675305.840084 mapread 0x00114a thru 0x0084a4 (0x735b bytes) 000940 1756675305.847281 mapwrite 0x0110a8 thru 0x01bad5 (0xaa2e bytes) 000941 1756675305.893973 trunc from 0x01bad6 to 0x00f053 (0xca82 bytes) 000943 1756675305.900337 mapread 0x0094ed thru 0x00f053 (0x5b67 bytes) 000946 1756675305.903904 mapwrite 0x02144a thru 0x0242ed (0x2ea4 bytes) 000948 1756675305.922858 mapwrite 0x03703b thru 0x03ffff (0x8fc5 bytes) 000949 1756675305.942353 read 0x000d43 thru 0x010809 (0xfac7 bytes) 000952 1756675305.945474 trunc from 0x040000 to 0x01c656 (0x239a9 bytes) 000955 1756675305.950448 mapwrite 0x031427 thru 0x03a075 (0x8c4f bytes) 000956 1756675305.972621 mapwrite 0x03a077 thru 0x03d984 (0x390e bytes) 000958 1756675305.991654 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000959 1756675306.047523 mapread 0x03649b thru 0x03d984 (0x74ea bytes) 000960 1756675306.052667 mapwrite 0x003976 thru 0x010ac7 (0xd152 bytes) 000961 1756675306.070261 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x67d:0x0]: Invalid argument (22) 000964 1756675306.120897 mapread 0x0113aa thru 0x01e974 (0xd5cb bytes) 000965 1756675306.128458 trunc from 0x03d985 to 0x039f09 (0x3a7b bytes) 000966 1756675306.133081 read 0x02c6e9 thru 0x033f2f (0x7847 bytes) 000967 1756675306.135820 trunc from 0x039f0a to 0x02d2fa (0xcc0f bytes) 000968 1756675306.140243 write 0x001072 thru 0x0059a4 (0x4933 bytes) 000970 1756675306.143293 mapwrite 0x028bd4 thru 0x02dbdc (0x5009 bytes) 000973 1756675306.163401 read 0x01ae88 thru 0x01cb5b (0x1cd4 bytes) 000974 1756675306.163989 mapwrite 0x0186f8 thru 0x028325 (0xfc2e bytes) 000975 1756675306.178390 write 0x00199a thru 0x0085da (0x6c41 bytes) 000976 1756675306.179083 read 0x0057f5 thru 0x0132e9 (0xdaf5 bytes) 000979 1756675306.180064 read 0x013160 thru 0x01f0cf (0xbf70 bytes) 000981 1756675306.180959 mapread 0x01cc14 thru 0x0289fb (0xbde8 bytes) 000982 1756675306.181507 write 0x022924 thru 0x027a3e (0x511b bytes) 000984 1756675306.184282 MIRROR_EXTEND to 7 mirrors 000985 1756675306.325954 mapread 0x017492 thru 0x01f1bb (0x7d2a bytes) 000989 1756675306.336814 read 0x00a392 thru 0x01932e (0xef9d bytes) 000990 1756675306.340228 mapread 0x00ea79 thru 0x01878a (0x9d12 bytes) 000992 1756675306.344292 write 0x03b2c4 thru 0x03ffff (0x4d3c bytes) 000993 1756675306.364428 MIRROR_EXTEND to 8 mirrors 000996 1756675306.510844 trunc from 0x040000 to 0x0089f3 (0x3760c bytes) 000998 1756675306.539082 mapread 0x0065d8 thru 0x0089f3 (0x241c bytes) All operations completed A-OK! PASS 70a (20s) == sanity-flr test 71: check mirror extend parent fid ==== 17:21:48 (1756675308) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 17:21:49 (1756675309) Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre2 Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7181344 .. Extending file size to 5771200 .. done Extending file size to 7953760 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7442400 .. Extending file size to 7822912 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8709696 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8444512 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8664192 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9225504 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9160384 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9284032 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 58356 58357 58359 58360 58363 192.168.204.123@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg423-client.virtnet /mnt/lustre2 (opts:) 192.168.204.123@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg423-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:22:54 (1756675374) Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre2 Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5183808 .. Extending file size to 5210752 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6971328 .. Extending file size to 7137376 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8021888 .. Extending file size to 8525216 .. failed Extending file size to 8214016 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8963872 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9275744 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 61916 61917 61919 61921 61923 192.168.204.123@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg423-client.virtnet /mnt/lustre2 (opts:) 192.168.204.123@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg423-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:23:59 (1756675439) Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5120 2203008 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3755008 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5120 2203008 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 14336 3755008 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0321259 s, 65.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102734 s, 20.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.123@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg423-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:24:05 (1756675445) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0227898 s, 92.0 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 17:24:07 (1756675447) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6c0:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x998:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x998:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x999:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x99b:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:24:09 (1756675449) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0298024 s, 70.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x99a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x99c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x99d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:24:12 (1756675452) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0386794 s, 54.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x99b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x99c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x99e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x99b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x99c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x99e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x99f:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:24:14 (1756675454) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0283742 s, 73.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x99d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9a0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x99d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9a0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9a1:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:24:38 (1756675478) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg423-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202591 s, 51.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x99e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9a2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x99e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9a2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x99f:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:25:24 (1756675524) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg423-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9a1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0286487 s, 73.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9a1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9a4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9a1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9a4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:26:10 (1756675570) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg423-server mds-ost sync done. Creating new pool test_204f oleg423-server: Pool lustre.test_204f created Adding targets to pool oleg423-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg423-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0279734 s, 75.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9a5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9a5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg423-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg423-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg423-server: Pool lustre.test_204f destroyed PASS 204f (60s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:27:10 (1756675630) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:27:12 (1756675632) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:27:15 (1756675635) oleg423-server: Pool lustre.test_206 created Adding targets to pool oleg423-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg423-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg423-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg423-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg423-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:27:28 (1756675648) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252855 s, 41.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142367 s, 73.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x9d83ca8a Mirror 3: 0x9d83ca8a PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:27:31 (1756675651) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:27:32 (1756675652) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:27:34 (1756675654) Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0255978 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x6e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1b10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1b10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1b10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x15c0000 Read this (4 bytes): BBBB 192.168.204.123@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg423-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:27:49 (1756675669) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00308049 s, 2.7 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:27:51 (1756675671) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153988 s, 68.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x6e5 lmm_fid: [0x200000404:0x6e5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x6e5 lmm_fid: [0x200000404:0x6e5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9b0:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835685 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-08-31 17:27:52.000000000 -0400 Modify: 2025-08-31 17:27:52.000000000 -0400 Change: 2025-08-31 17:27:52.000000000 -0400 Birth: 2025-08-31 17:27:51.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:27:54 (1756675674) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (0s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:27:55 (1756675675) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.129568 s, 316 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0140886 s, 291 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 31 17:27 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e7:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 31 17:27 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 786 sec ============ 17:27:57 (1756675677) === sanity-flr: start cleanup 17:27:58 (1756675678) === === sanity-flr: finish cleanup 17:27:59 (1756675679) ===