-----============= acceptance-small: sanity-flr ============----- Wed Jul 16 13:43:54 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg142-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 13:44:00 (1752687840) === oleg142-client.virtnet: executing check_config_client /mnt/lustre oleg142-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg142-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b176059a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b176059a000.idle_timeout=debug disable quota as required oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 13:44:11 (1752687851) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7538] == sanity-flr test 0a: lfs mirror create with -N option == 13:44:11 (1752687851) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:44:16 (1752687856) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:44:17 (1752687857) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:44:18 (1752687858) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:44:24 (1752687864) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:44:25 (1752687865) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 13:44:26 (1752687866) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b176059a000.stats=clear osc.lustre-OST0001-osc-ffff9b176059a000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:44:29 (1752687869) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:44:32 (1752687872) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:44:35 (1752687875) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:44:36 (1752687876) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 13:44:39 (1752687879) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:44:41 (1752687881) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:44:43 (1752687883) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:44:45 (1752687885) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:44:49 (1752687889) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:44:52 (1752687892) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.537466 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553523 s, 18.9 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:44:56 (1752687896) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.121944 s, 120 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:44:59 (1752687899) Stopping /mnt/lustre-ost1 (opts:) on oleg142-server oleg142-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b176059a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b176059a000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg142-server: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg142-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg142-server oleg142-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b176059a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b176059a000.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg142-server: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg142-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (28s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:45:27 (1752687927) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0595099 s, 52.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg142-server oleg142-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b176059a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b176059a000.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg142-server: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg142-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:45:42 (1752687942) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg142-server oleg142-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b176059a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b176059a000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg142-server: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg142-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg142-server oleg142-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b176059a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b176059a000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg142-server: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg142-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (27s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:46:09 (1752687969) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.963817 s, 54.4 MB/s -rw-r--r-- 1 root root 52428800 Jul 16 13:46 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.268s user 0m0.000s sys 0m0.153s Stopping /mnt/lustre-ost1 (opts:) on oleg142-server oleg142-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b176059a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b176059a000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.275s user 0m0.001s sys 0m0.154s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg142-server: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg142-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg142-server oleg142-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b176059a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b176059a000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.291s user 0m0.001s sys 0m0.152s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg142-server: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg142-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg142-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b176059a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b176059a000.ost_server_uuid in FULL state after 0 sec PASS 33b (27s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:46:36 (1752687996) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:46:37 (1752687997) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:46:38 (1752687998) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 13:46:39 (1752687999) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 13:46:42 (1752688002) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.15116 s, 83.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:46:46 (1752688006) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109238 s, 96.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00915047 s, 115 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:46:48 (1752688008) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:46:51 (1752688011) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000886538 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000655958 s, 1.6 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000805422 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000681026 s, 1.5 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 13:46:54 (1752688014) create test files with size 15745724 .. 51638e4c501daf6d5d1ee3d47cf9dc71 - 31816c0ce7881a1fd84ac7b3ecc93392 - 2e93d5654f7a62b3d41948db31008e9e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg142-client.virtnet /mnt/lustre (opts:) Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 13:47:01 (1752688021) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:47:05 (1752688025) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:47:08 (1752688028) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480305 s, 21.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01278 s, 82.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 13:47:11 (1752688031) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0476446 s, 44.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.169502 s, 24.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128017 s, 16.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (7s) == sanity-flr test 42: lfs mirror verify ================= 13:47:18 (1752688038) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 13:47:20 (1752688040) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:47:23 (1752688043) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398712 s, 26.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 44a: lfs mirror split check =========== 13:47:28 (1752688048) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:47:31 (1752688051) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:47:38 (1752688058) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.308746 s, 34.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg142-client.virtnet /mnt/lustre (opts:) Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:47:48 (1752688068) llite.lustre-ffff9b1745570800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.174967 s, 59.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg142-client.virtnet /mnt/lustre (opts:) Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9b1747ca8800.xattr_cache=1 PASS 44e (10s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:47:58 (1752688078) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:48:38 (1752688118) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:49:09 (1752688149) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 13:49:11 (1752688151) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.147187 s, 21.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.090373 s, 34.8 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: ce381550dfa8cd8d8555e89476714016 - mirror 1 checksum: 0a68bc3433158e7931344fdb70726c25 - mirror 2 checksum: ce381550dfa8cd8d8555e89476714016 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Jul 16 13:49:13 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Jul 16 13:49:13 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: ce381550dfa8cd8d8555e89476714016 - mirror 1 checksum: 0a68bc3433158e7931344fdb70726c25 - mirror 2 checksum: 0a68bc3433158e7931344fdb70726c25 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Jul 16 13:49:13 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Jul 16 13:49:13 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (7s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:49:19 (1752688159) Failing mds1 on oleg142-server Stopping /mnt/lustre-mds1 (opts:) on oleg142-server 13:49:23 (1752688163) shut down facet: mds1 facet_host: oleg142-server facet_failover_host: oleg142-server Failover mds1 to oleg142-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-MDT0000 13:49:41 (1752688181) targets are mounted 13:49:41 (1752688181) facet_failover done oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154316 s, 67.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (32s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:49:51 (1752688191) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:49:53 (1752688193) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:49:55 (1752688195) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00468565 s, 3.5 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:50:00 (1752688200) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:50:02 (1752688202) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:50:04 (1752688204) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (15s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:50:19 (1752688219) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (23s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:50:42 (1752688242) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:51:03 (1752688263) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (21s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:51:25 (1752688285) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 13:51:29 (1752688289) mirror create pid 52252 mirror split pid 52254 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7670: 52252 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7670: 52254 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 13:52:34 (1752688354) Seed set to 8355 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1752688356.055496 mapwrite 0x025432 thru 0x031354 (0xbf23 bytes) 000004 1752688356.196199 read 0x019121 thru 0x02879b (0xf67b bytes) 000007 1752688356.209155 read 0x02d4bb thru 0x030e53 (0x3999 bytes) 000008 1752688356.210529 mapread 0x00b2a0 thru 0x015a8b (0xa7ec bytes) 000009 1752688356.227199 mapwrite 0x01d4a5 thru 0x021faf (0x4b0b bytes) 000012 1752688356.274580 mapread 0x017f7c thru 0x01fabc (0x7b41 bytes) 000015 1752688356.276574 read 0x021aea thru 0x02fcce (0xe1e5 bytes) 000018 1752688356.278469 mapread 0x027ce7 thru 0x031354 (0x966e bytes) 000019 1752688356.279485 mapread 0x0277ba thru 0x02bd08 (0x454f bytes) truncating to largest ever: 0x167a8 000020 1752688356.280238 trunc from 0x031355 to 0x0167a7 (0x1abad bytes) truncating to largest ever: 0x234d7 000021 1752688356.302693 trunc from 0x0167a8 to 0x0234d6 (0xcd2f bytes) 000022 1752688356.319567 mapwrite 0x01c434 thru 0x01e600 (0x21cd bytes) 000024 1752688356.384804 write 0x01d878 thru 0x025004 (0x778d bytes) 000025 1752688356.386146 write 0x01ed1e thru 0x0293be (0xa6a1 bytes) truncating to largest ever: 0x2a7ef 000026 1752688356.387565 trunc from 0x0293bf to 0x02a7ee (0x1430 bytes) 000032 1752688356.403869 mapwrite 0x0022aa thru 0x0051b2 (0x2f09 bytes) 000033 1752688356.453006 MIRROR_EXTEND to 2 mirrors 000034 1752688356.856539 trunc from 0x02a7ef to 0x00b477 (0x1f377 bytes) 000035 1752688356.939516 read 0x0072c3 thru 0x00b477 (0x41b5 bytes) 000036 1752688356.948814 write 0x038ebc thru 0x039c53 (0xd98 bytes) 000038 1752688356.950587 mapread 0x02c23c thru 0x035347 (0x910c bytes) 000039 1752688356.975568 mapread 0x009e32 thru 0x019201 (0xf3d0 bytes) 000041 1752688356.996055 read 0x027bb4 thru 0x02ba09 (0x3e56 bytes) 000043 1752688356.997306 mapread 0x009c8e thru 0x00cd44 (0x30b7 bytes) 000044 1752688356.998342 read 0x02eb4e thru 0x02fa04 (0xeb7 bytes) 000045 1752688357.002590 MIRROR_EXTEND to 3 mirrors 000049 1752688357.523667 mapwrite 0x0161e3 thru 0x01778a (0x15a8 bytes) 000054 1752688357.668638 read 0x01daa8 thru 0x0221c8 (0x4721 bytes) 000055 1752688357.680335 write 0x00d33d thru 0x01ca4b (0xf70f bytes) 000056 1752688357.691290 read 0x0380f6 thru 0x039c53 (0x1b5e bytes) 000057 1752688357.699995 mapwrite 0x000b9a thru 0x00f1c9 (0xe630 bytes) 000058 1752688357.752744 mapwrite 0x00aa8f thru 0x01a8e9 (0xfe5b bytes) 000059 1752688357.805809 mapread 0x001ccd thru 0x0081a9 (0x64dd bytes) 000060 1752688357.807597 mapread 0x008788 thru 0x00c148 (0x39c1 bytes) 000062 1752688357.810902 trunc from 0x039c54 to 0x02a39a (0xf8b9 bytes) 000064 1752688357.842254 write 0x03cfe7 thru 0x03ffff (0x3019 bytes) 000065 1752688357.846896 trunc from 0x040000 to 0x01c33c (0x23cc3 bytes) 000066 1752688357.867060 read 0x00b931 thru 0x00d273 (0x1943 bytes) 000068 1752688357.868286 mapwrite 0x011df2 thru 0x021b41 (0xfd50 bytes) 000069 1752688357.945754 mapwrite 0x02b6df thru 0x038d4d (0xd66f bytes) 000071 1752688358.032845 trunc from 0x038d4e to 0x025040 (0x13d0d bytes) 000072 1752688358.057733 MIRROR_EXTEND to 4 mirrors 000073 1752688358.573628 read 0x023814 thru 0x025040 (0x182d bytes) 000074 1752688358.592118 read 0x0173b2 thru 0x019856 (0x24a5 bytes) 000076 1752688358.628210 read 0x018107 thru 0x01a7f1 (0x26eb bytes) 000077 1752688358.642712 trunc from 0x025041 to 0x0274ee (0x24ae bytes) 000080 1752688358.725851 MIRROR_EXTEND to 5 mirrors 000081 1752688359.261204 write 0x032a60 thru 0x03cbbd (0xa15e bytes) 000082 1752688359.333765 read 0x02f820 thru 0x035a02 (0x61e3 bytes) 000083 1752688359.343960 read 0x0357e4 thru 0x03cbbd (0x73da bytes) 000086 1752688359.345564 write 0x02d1b5 thru 0x0384fa (0xb346 bytes) 000088 1752688359.371230 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000090 1752688359.575387 read 0x0394e9 thru 0x03cbbd (0x36d5 bytes) 000091 1752688359.588544 MIRROR_RESYNC 4 mirrors 000092 1752688360.169490 write 0x0010f3 thru 0x009317 (0x8225 bytes) 000093 1752688360.246032 read 0x008bb2 thru 0x009cb2 (0x1101 bytes) 000094 1752688360.246654 mapread 0x031b24 thru 0x03998b (0x7e68 bytes) 000095 1752688360.255054 mapwrite 0x010fa5 thru 0x01b14d (0xa1a9 bytes) 000096 1752688360.332042 mapread 0x032859 thru 0x03bc1d (0x93c5 bytes) 000098 1752688360.344853 MIRROR_EXTEND to 5 mirrors 000099 1752688360.775540 read 0x031735 thru 0x03aabd (0x9389 bytes) 000101 1752688360.793274 trunc from 0x03cbbe to 0x023f35 (0x18c88 bytes) 000102 1752688360.887924 mapread 0x0232f9 thru 0x023f35 (0xc3d bytes) 000104 1752688360.896731 read 0x003ad7 thru 0x004302 (0x82c bytes) 000105 1752688360.907222 trunc from 0x023f36 to 0x013936 (0x105ff bytes) 000109 1752688360.935282 write 0x01900a thru 0x019d01 (0xcf8 bytes) 000111 1752688360.940728 read 0x01558b thru 0x015c30 (0x6a6 bytes) 000114 1752688360.958707 MIRROR_EXTEND to 6 mirrors 000116 1752688361.467977 trunc from 0x019d02 to 0x00039a (0x19967 bytes) 000118 1752688361.535759 mapwrite 0x011419 thru 0x01edd2 (0xd9ba bytes) 000120 1752688361.632732 read 0x003b96 thru 0x00aeef (0x735a bytes) 000121 1752688361.645933 MIRROR_RESYNC 6 mirrors 000124 1752688362.447898 MIRROR_EXTEND to 7 mirrors 000126 1752688363.056554 write 0x003604 thru 0x01248b (0xee88 bytes) 000127 1752688363.148575 MIRROR_RESYNC 7 mirrors 000129 1752688363.920717 mapread 0x009ddd thru 0x019d1f (0xff43 bytes) 000130 1752688363.947214 MIRROR_EXTEND to 8 mirrors 000131 1752688364.397626 write 0x022a74 thru 0x0285c6 (0x5b53 bytes) truncating to largest ever: 0x3b416 000134 1752688364.449902 trunc from 0x0285c7 to 0x03b415 (0x12e4f bytes) 000135 1752688364.468262 read 0x016460 thru 0x02303c (0xcbdd bytes) 000136 1752688364.479163 write 0x036e21 thru 0x03ffff (0x91df bytes) 000138 1752688364.489789 write 0x00c7cd thru 0x013685 (0x6eb9 bytes) 000140 1752688364.501577 mapread 0x00ba39 thru 0x01024d (0x4815 bytes) 000141 1752688364.514182 mapread 0x034295 thru 0x03ffff (0xbd6b bytes) 000144 1752688364.528203 mapread 0x03bd96 thru 0x03fdde (0x4049 bytes) 000145 1752688364.529089 read 0x01c334 thru 0x026640 (0xa30d bytes) 000147 1752688364.530860 mapread 0x013e09 thru 0x0190db (0x52d3 bytes) 000149 1752688364.543585 trunc from 0x040000 to 0x039862 (0x679d bytes) 000151 1752688364.558550 write 0x0092b0 thru 0x00f422 (0x6173 bytes) 000152 1752688364.559621 mapwrite 0x03392e thru 0x03ffff (0xc6d2 bytes) 000154 1752688364.710515 write 0x0376c7 thru 0x03885f (0x1199 bytes) 000155 1752688364.711572 write 0x0284c3 thru 0x028f76 (0xab4 bytes) 000156 1752688364.715766 write 0x00dd8d thru 0x01765c (0x98d0 bytes) 000159 1752688364.726288 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000162 1752688364.901491 MIRROR_RESYNC 7 mirrors 000163 1752688365.803509 mapread 0x0339d7 thru 0x03ffff (0xc629 bytes) 000165 1752688365.828378 write 0x01fff7 thru 0x02c5f8 (0xc602 bytes) 000166 1752688365.910978 read 0x012897 thru 0x0161b0 (0x391a bytes) 000168 1752688365.919259 mapwrite 0x00747e thru 0x012f4c (0xbacf bytes) 000169 1752688366.028374 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000171 1752688366.292567 MIRROR_EXTEND to 7 mirrors 000173 1752688366.742794 write 0x018fe5 thru 0x023645 (0xa661 bytes) 000174 1752688366.805660 mapread 0x0072a3 thru 0x00f775 (0x84d3 bytes) 000175 1752688366.821181 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000177 1752688367.042624 mapread 0x03d801 thru 0x03ffff (0x27ff bytes) 000178 1752688367.063106 mapread 0x005c64 thru 0x012e3f (0xd1dc bytes) 000179 1752688367.083341 mapread 0x038f26 thru 0x03ffff (0x70da bytes) 000180 1752688367.084272 trunc from 0x040000 to 0x0133cd (0x2cc32 bytes) 000181 1752688367.103835 mapwrite 0x03dc06 thru 0x03e517 (0x912 bytes) 000185 1752688367.186999 mapread 0x0258dd thru 0x02c021 (0x6745 bytes) 000186 1752688367.201896 write 0x024d4d thru 0x028d26 (0x3fda bytes) 000188 1752688367.203321 mapwrite 0x036b88 thru 0x03d7bd (0x6c36 bytes) 000190 1752688367.248249 trunc from 0x03e518 to 0x025403 (0x19114 bytes) 000191 1752688367.266211 read 0x014b0d thru 0x020180 (0xb674 bytes) 000192 1752688367.282163 mapread 0x0238ff thru 0x025403 (0x1b05 bytes) 000194 1752688367.283635 mapwrite 0x019e0b thru 0x01ecda (0x4ed0 bytes) 000196 1752688367.369174 write 0x01d9e8 thru 0x020395 (0x29ae bytes) 000197 1752688367.374145 mapread 0x00b8f2 thru 0x01ab9a (0xf2a9 bytes) 000202 1752688367.388951 MIRROR_EXTEND to 7 mirrors 000206 1752688367.835178 write 0x009f17 thru 0x0110a8 (0x7192 bytes) 000207 1752688367.920072 mapwrite 0x002ced thru 0x00a349 (0x765d bytes) 000209 1752688368.005274 mapwrite 0x00c4a7 thru 0x016f11 (0xaa6b bytes) 000210 1752688368.072085 mapwrite 0x0349c1 thru 0x03ffff (0xb63f bytes) 000212 1752688368.213407 write 0x01f1f8 thru 0x021973 (0x277c bytes) 000213 1752688368.223672 read 0x008590 thru 0x00f32b (0x6d9c bytes) 000216 1752688368.225197 read 0x03b663 thru 0x03c220 (0xbbe bytes) 000217 1752688368.225745 trunc from 0x040000 to 0x022068 (0x1df97 bytes) 000221 1752688368.249229 MIRROR_EXTEND to 8 mirrors 000222 1752688368.710969 mapread 0x003342 thru 0x0052a0 (0x1f5f bytes) 000224 1752688368.763246 mapread 0x00d343 thru 0x015774 (0x8432 bytes) 000225 1752688368.790998 read 0x00ae4e thru 0x00db83 (0x2d36 bytes) 000226 1752688368.799650 mapread 0x0012c6 thru 0x01122e (0xff69 bytes) 000227 1752688368.814063 write 0x02582f thru 0x028568 (0x2d3a bytes) 000229 1752688368.892003 write 0x02135e thru 0x02b97c (0xa61f bytes) 000230 1752688368.900056 mapwrite 0x022d2c thru 0x028540 (0x5815 bytes) 000231 1752688368.964435 trunc from 0x02b97d to 0x03725e (0xb8e2 bytes) 000233 1752688368.983198 trunc from 0x03725f to 0x01c60c (0x1ac52 bytes) 000235 1752688369.013873 MIRROR_EXTEND to 9 mirrors 000236 1752688369.549733 trunc from 0x01c60d to 0x01c555 (0xb7 bytes) 000240 1752688369.625331 mapread 0x019f27 thru 0x01c555 (0x262f bytes) 000243 1752688369.642645 mapread 0x000396 thru 0x0009f4 (0x65f bytes) 000244 1752688369.649112 trunc from 0x01c556 to 0x025e29 (0x98d4 bytes) 000247 1752688369.668375 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000248 1752688369.909742 read 0x0018e6 thru 0x006dc2 (0x54dd bytes) 000249 1752688369.916131 mapread 0x001354 thru 0x00c02c (0xacd9 bytes) 000250 1752688369.934906 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000251 1752688370.133213 MIRROR_RESYNC 7 mirrors 000260 1752688370.885496 read 0x00218c thru 0x0066e6 (0x455b bytes) 000261 1752688370.893066 mapread 0x0240c2 thru 0x025e29 (0x1d68 bytes) 000262 1752688370.915876 mapread 0x01343d thru 0x019098 (0x5c5c bytes) 000263 1752688370.926328 read 0x02020d thru 0x0205dc (0x3d0 bytes) 000267 1752688370.954352 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000268 1752688371.252095 mapwrite 0x01c6ff thru 0x01e24f (0x1b51 bytes) 000269 1752688371.385262 mapwrite 0x025b0d thru 0x02b05c (0x5550 bytes) 000270 1752688371.456297 mapwrite 0x027d17 thru 0x030ee5 (0x91cf bytes) 000273 1752688371.530392 mapwrite 0x0207e1 thru 0x02846a (0x7c8a bytes) 000274 1752688371.582258 write 0x03e73c thru 0x03ffff (0x18c4 bytes) 000275 1752688371.584133 mapwrite 0x032a62 thru 0x03fbeb (0xd18a bytes) 000278 1752688371.670954 write 0x00498c thru 0x0125d2 (0xdc47 bytes) 000280 1752688371.691744 mapread 0x012c82 thru 0x0141b1 (0x1530 bytes) 000282 1752688371.701315 trunc from 0x040000 to 0x01997a (0x26685 bytes) 000285 1752688371.722637 write 0x01e0f3 thru 0x02543e (0x734c bytes) 000287 1752688371.724643 trunc from 0x02543f to 0x02371e (0x1d20 bytes) 000288 1752688371.740930 mapwrite 0x01185b thru 0x018942 (0x70e8 bytes) 000289 1752688371.795695 mapread 0x016eb9 thru 0x01b825 (0x496d bytes) 000292 1752688371.806882 mapread 0x014849 thru 0x01b055 (0x680d bytes) 000293 1752688371.808304 mapread 0x00562f thru 0x00a660 (0x5032 bytes) 000294 1752688371.809208 read 0x0045d8 thru 0x00c9ea (0x8413 bytes) 000295 1752688371.815986 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000296 1752688372.045227 mapread 0x003ad0 thru 0x00a4a3 (0x69d4 bytes) 000298 1752688372.081903 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000299 1752688372.361573 read 0x0088dc thru 0x014290 (0xb9b5 bytes) 000302 1752688372.375176 MIRROR_EXTEND to 5 mirrors 000303 1752688373.014052 mapwrite 0x03318b thru 0x03ffff (0xce75 bytes) 000304 1752688373.193261 write 0x01a972 thru 0x01b716 (0xda5 bytes) 000307 1752688373.210045 read 0x01818b thru 0x019e46 (0x1cbc bytes) 000310 1752688373.211857 read 0x027bda thru 0x027ca6 (0xcd bytes) 000311 1752688373.233168 trunc from 0x040000 to 0x01129c (0x2ed63 bytes) 000312 1752688373.260605 MIRROR_RESYNC 5 mirrors 000314 1752688374.059705 read 0x00e7ee thru 0x01129c (0x2aaf bytes) 000316 1752688374.069822 trunc from 0x01129d to 0x039a14 (0x28778 bytes) 000317 1752688374.157670 read 0x022f0f thru 0x030124 (0xd216 bytes) 000321 1752688374.179856 MIRROR_RESYNC 5 mirrors 000323 1752688374.842552 trunc from 0x039a15 to 0x024509 (0x1550b bytes) 000330 1752688374.991181 read 0x01a740 thru 0x024509 (0x9dca bytes) 000331 1752688374.997852 write 0x027c28 thru 0x02d1af (0x5588 bytes) 000334 1752688375.000301 trunc from 0x02d1b0 to 0x017643 (0x15b6c bytes) 000335 1752688375.023768 mapread 0x000cd4 thru 0x006675 (0x59a2 bytes) 000337 1752688375.042851 mapwrite 0x00986f thru 0x012f32 (0x96c4 bytes) 000338 1752688375.106227 read 0x000bc0 thru 0x009472 (0x88b3 bytes) 000340 1752688375.107388 mapread 0x001d6d thru 0x005b27 (0x3dbb bytes) 000343 1752688375.110379 read 0x0059d8 thru 0x00978e (0x3db7 bytes) 000344 1752688375.111889 read 0x006147 thru 0x011142 (0xaffc bytes) 000347 1752688375.113736 read 0x010e74 thru 0x017643 (0x67d0 bytes) 000348 1752688375.119950 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000349 1752688375.374963 MIRROR_RESYNC 4 mirrors 000351 1752688376.023220 mapread 0x008eac thru 0x010ce2 (0x7e37 bytes) 000354 1752688376.055733 trunc from 0x017644 to 0x009883 (0xddc0 bytes) 000355 1752688376.125779 read 0x009247 thru 0x009883 (0x63d bytes) 000356 1752688376.133627 mapwrite 0x015622 thru 0x017008 (0x19e7 bytes) 000358 1752688376.214114 write 0x03a0d2 thru 0x03ffff (0x5f2e bytes) 000359 1752688376.216349 read 0x026801 thru 0x0328a7 (0xc0a7 bytes) 000360 1752688376.229285 mapwrite 0x01bce8 thru 0x0211ea (0x5503 bytes) 000361 1752688376.306751 MIRROR_EXTEND to 5 mirrors 000363 1752688376.759960 MIRROR_RESYNC 5 mirrors 000365 1752688377.406325 read 0x03cf6e thru 0x03ffff (0x3092 bytes) 000366 1752688377.416203 MIRROR_EXTEND to 6 mirrors 000367 1752688378.045786 mapwrite 0x00533d thru 0x014f74 (0xfc38 bytes) 000368 1752688378.218112 mapwrite 0x00c35d thru 0x01b5d9 (0xf27d bytes) 000371 1752688378.286480 read 0x03390b thru 0x03610c (0x2802 bytes) 000372 1752688378.305701 write 0x0047ae thru 0x00b177 (0x69ca bytes) 000374 1752688378.307539 mapread 0x00b348 thru 0x00b3a3 (0x5c bytes) 000375 1752688378.309055 mapwrite 0x0310c8 thru 0x035da7 (0x4ce0 bytes) 000376 1752688378.391564 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000377 1752688378.642676 read 0x0015df thru 0x00b5dc (0x9ffe bytes) 000379 1752688378.650766 mapread 0x01babc thru 0x01f9d0 (0x3f15 bytes) 000382 1752688378.674202 trunc from 0x040000 to 0x00d23f (0x32dc0 bytes) 000385 1752688378.717980 write 0x01444c thru 0x01a34c (0x5f01 bytes) 000388 1752688378.720715 mapwrite 0x020f50 thru 0x02ce5a (0xbf0b bytes) 000390 1752688378.836866 trunc from 0x02ce5b to 0x0171ef (0x15c6b bytes) 000391 1752688378.857185 MIRROR_EXTEND to 6 mirrors 000396 1752688379.451524 write 0x035488 thru 0x03f5d4 (0xa14d bytes) 000398 1752688379.552541 mapwrite 0x02bc7e thru 0x03b64a (0xf9cd bytes) 000399 1752688379.663031 write 0x030e25 thru 0x0342ef (0x34cb bytes) 000400 1752688379.676036 MIRROR_RESYNC 6 mirrors 000401 1752688380.501882 mapread 0x01f4b2 thru 0x029f18 (0xaa67 bytes) 000403 1752688380.527655 mapwrite 0x0172a0 thru 0x01f070 (0x7dd1 bytes) 000404 1752688380.703975 read 0x01230f thru 0x01bf52 (0x9c44 bytes) 000405 1752688380.710286 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000406 1752688380.947011 read 0x015080 thru 0x0165ef (0x1570 bytes) 000407 1752688380.955907 mapwrite 0x0387e2 thru 0x03ffff (0x781e bytes) 000410 1752688381.129706 mapread 0x019c68 thru 0x01adc4 (0x115d bytes) 000411 1752688381.162866 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000413 1752688381.360848 trunc from 0x040000 to 0x017d5e (0x282a1 bytes) 000414 1752688381.373910 read 0x00713c thru 0x010034 (0x8ef9 bytes) 000415 1752688381.386487 mapwrite 0x028b17 thru 0x02bac0 (0x2faa bytes) 000416 1752688381.515321 mapwrite 0x039733 thru 0x03ffff (0x68cd bytes) 000417 1752688381.591894 mapread 0x0365b0 thru 0x0379f1 (0x1442 bytes) 000419 1752688381.593403 mapwrite 0x0183b9 thru 0x027352 (0xef9a bytes) 000420 1752688381.665840 read 0x02ce5c thru 0x03597e (0x8b23 bytes) 000422 1752688381.677208 mapread 0x03edf1 thru 0x03ffff (0x120f bytes) 000423 1752688381.682286 MIRROR_RESYNC 4 mirrors 000424 1752688382.393398 write 0x01e8cf thru 0x023a78 (0x51aa bytes) 000428 1752688382.536343 mapread 0x011370 thru 0x012c74 (0x1905 bytes) 000431 1752688382.556285 MIRROR_RESYNC 4 mirrors 000433 1752688383.349835 mapwrite 0x01f026 thru 0x0235c0 (0x459b bytes) 000435 1752688383.506825 read 0x00f9ef thru 0x01134a (0x195c bytes) 000436 1752688383.520948 MIRROR_EXTEND to 5 mirrors 000437 1752688384.094768 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000439 1752688384.454846 read 0x01eb56 thru 0x026d0f (0x81ba bytes) 000440 1752688384.477825 read 0x00fdd2 thru 0x01d37f (0xd5ae bytes) 000442 1752688384.505286 trunc from 0x040000 to 0x02c7d6 (0x13829 bytes) 000444 1752688384.630878 write 0x02a57c thru 0x02e41e (0x3ea3 bytes) 000447 1752688384.637101 mapread 0x00237b thru 0x0060a7 (0x3d2d bytes) 000448 1752688384.656984 mapread 0x017f52 thru 0x01f29a (0x7349 bytes) 000449 1752688384.709060 read 0x01352b thru 0x01d720 (0xa1f6 bytes) 000451 1752688384.722242 trunc from 0x02e41f to 0x00d3a8 (0x21076 bytes) 000452 1752688384.755688 read 0x003de5 thru 0x00773c (0x3958 bytes) 000453 1752688384.757148 write 0x011130 thru 0x017e8d (0x6d5e bytes) 000457 1752688384.760941 mapread 0x0051c9 thru 0x00f36d (0xa1a5 bytes) 000458 1752688384.769667 mapread 0x002248 thru 0x008308 (0x60c1 bytes) 000462 1752688384.772013 read 0x014338 thru 0x014916 (0x5df bytes) 000465 1752688384.774788 trunc from 0x017e8e to 0x00ea5e (0x942f bytes) 000466 1752688384.792090 mapwrite 0x0103a2 thru 0x01e962 (0xe5c1 bytes) 000467 1752688384.862245 read 0x007267 thru 0x01514e (0xdee8 bytes) 000469 1752688384.864137 read 0x00269a thru 0x00a702 (0x8069 bytes) 000470 1752688384.864839 write 0x00bf1e thru 0x00ff5f (0x4042 bytes) 000472 1752688384.866889 mapread 0x0046e9 thru 0x012b11 (0xe429 bytes) 000473 1752688384.867924 mapwrite 0x0247d9 thru 0x033d9a (0xf5c2 bytes) 000474 1752688384.995565 write 0x02509c thru 0x026b65 (0x1aca bytes) 000477 1752688384.997292 mapread 0x017f1b thru 0x023e01 (0xbee7 bytes) 000480 1752688385.002384 mapwrite 0x01bfa2 thru 0x022675 (0x66d4 bytes) 000481 1752688385.050774 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000482 1752688385.324580 mapread 0x00e040 thru 0x018753 (0xa714 bytes) 000487 1752688385.341959 mapwrite 0x0170c9 thru 0x01c652 (0x558a bytes) 000488 1752688385.426150 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000493 1752688385.645884 read 0x02bb6f thru 0x033d9a (0x822c bytes) 000495 1752688385.654939 write 0x03bce1 thru 0x03ffff (0x431f bytes) 000496 1752688385.656140 mapwrite 0x03c4c8 thru 0x03ffff (0x3b38 bytes) 000497 1752688385.700271 read 0x031334 thru 0x0391a3 (0x7e70 bytes) 000498 1752688385.720086 trunc from 0x040000 to 0x0376f5 (0x890a bytes) 000499 1752688385.743398 read 0x00be97 thru 0x00e107 (0x2271 bytes) 000500 1752688385.744145 read 0x01d33d thru 0x01f1c4 (0x1e88 bytes) 000501 1752688385.744736 trunc from 0x0376f6 to 0x00b582 (0x2c173 bytes) 000503 1752688385.773164 mapwrite 0x024d97 thru 0x02dc30 (0x8e9a bytes) 000506 1752688385.872950 mapread 0x01d548 thru 0x0241b3 (0x6c6c bytes) 000508 1752688385.877119 mapwrite 0x009393 thru 0x0102a4 (0x6f12 bytes) 000510 1752688385.951989 mapread 0x01ad4a thru 0x01e5cd (0x3884 bytes) 000511 1752688385.953070 read 0x008026 thru 0x014d18 (0xccf3 bytes) 000512 1752688385.953789 trunc from 0x02dc31 to 0x00cd8a (0x20ea6 bytes) 000513 1752688385.977150 read 0x0080b1 thru 0x008e7a (0xdca bytes) 000514 1752688385.978300 mapwrite 0x00dd80 thru 0x0114ba (0x373b bytes) 000518 1752688386.055033 mapread 0x00e65a thru 0x0114ba (0x2e61 bytes) 000519 1752688386.055891 write 0x02342b thru 0x02bfd4 (0x8baa bytes) 000521 1752688386.057777 write 0x02708b thru 0x0354d7 (0xe44d bytes) 000522 1752688386.059133 read 0x001dad thru 0x006341 (0x4595 bytes) 000524 1752688386.060211 read 0x013d25 thru 0x01bd58 (0x8034 bytes) 000525 1752688386.069388 write 0x034b55 thru 0x038ad4 (0x3f80 bytes) 000526 1752688386.070654 mapread 0x037091 thru 0x038ad4 (0x1a44 bytes) 000527 1752688386.071678 mapwrite 0x025103 thru 0x02c85f (0x775d bytes) 000529 1752688386.115175 write 0x014505 thru 0x01eac7 (0xa5c3 bytes) 000532 1752688386.118400 trunc from 0x038ad5 to 0x035ee6 (0x2bee bytes) 000533 1752688386.140828 MIRROR_EXTEND to 3 mirrors 000534 1752688386.687141 write 0x02caf0 thru 0x034da1 (0x82b2 bytes) 000535 1752688386.749364 write 0x01cc77 thru 0x01d6e5 (0xa6f bytes) 000536 1752688386.771297 MIRROR_RESYNC 3 mirrors 000538 1752688387.414778 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000539 1752688387.697043 MIRROR_RESYNC 2 mirrors 000540 1752688387.861788 trunc from 0x035ee7 to 0x01274a (0x2379c bytes) 000541 1752688387.959213 read 0x00f8d5 thru 0x01274a (0x2e76 bytes) 000542 1752688387.966479 mapwrite 0x017459 thru 0x01b0db (0x3c83 bytes) 000544 1752688388.110138 read 0x005298 thru 0x00ca36 (0x779f bytes) 000545 1752688388.127893 read 0x00c275 thru 0x01b0db (0xee67 bytes) 000551 1752688388.130802 read 0x002c42 thru 0x009002 (0x63c1 bytes) 000552 1752688388.136469 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000555 1752688388.279950 read 0x004766 thru 0x00c5a2 (0x7e3d bytes) 000557 1752688388.294555 mapread 0x0172af thru 0x01a4bb (0x320d bytes) 000560 1752688388.322708 read 0x017bb5 thru 0x01b0db (0x3527 bytes) 000561 1752688388.327090 trunc from 0x01b0dc to 0x0202b7 (0x51dc bytes) 000562 1752688388.351795 write 0x000b15 thru 0x007c69 (0x7155 bytes) 000565 1752688388.365542 write 0x023590 thru 0x02a6ca (0x713b bytes) 000568 1752688388.368968 read 0x013a4e thru 0x021894 (0xde47 bytes) 000569 1752688388.376349 mapwrite 0x00f4c2 thru 0x01359f (0x40de bytes) 000573 1752688388.451543 write 0x0097bd thru 0x016ed6 (0xd71a bytes) 000574 1752688388.453031 mapread 0x0123d1 thru 0x01ec24 (0xc854 bytes) 000575 1752688388.455319 read 0x008a43 thru 0x00e07d (0x563b bytes) 000577 1752688388.456347 mapread 0x016c04 thru 0x01f4ab (0x88a8 bytes) 000579 1752688388.457746 read 0x0036cb thru 0x005606 (0x1f3c bytes) 000581 1752688388.458790 mapwrite 0x027dea thru 0x02be22 (0x4039 bytes) 000582 1752688388.548588 mapread 0x0185ac thru 0x0229e3 (0xa438 bytes) 000584 1752688388.550468 read 0x024515 thru 0x02be22 (0x790e bytes) 000585 1752688388.551193 trunc from 0x02be23 to 0x01afae (0x10e74 bytes) 000589 1752688388.574536 write 0x032f98 thru 0x03ffff (0xd068 bytes) 000591 1752688388.577461 mapread 0x00dd06 thru 0x01192c (0x3c27 bytes) 000595 1752688388.580897 trunc from 0x040000 to 0x0012f5 (0x3ed0a bytes) 000596 1752688388.605890 write 0x0276ba thru 0x036016 (0xe95d bytes) 000597 1752688388.611499 mapread 0x002854 thru 0x004645 (0x1df2 bytes) 000599 1752688388.624940 mapwrite 0x017e42 thru 0x023157 (0xb316 bytes) 000600 1752688388.689727 read 0x01db35 thru 0x02bab2 (0xdf7e bytes) 000601 1752688388.690735 read 0x01aac3 thru 0x01e4ff (0x3a3d bytes) 000602 1752688388.703624 MIRROR_EXTEND to 2 mirrors 000603 1752688389.202992 trunc from 0x036017 to 0x0172ce (0x1ed48 bytes) 000606 1752688389.353024 read 0x0047ec thru 0x00f3d0 (0xabe5 bytes) 000607 1752688389.370306 mapwrite 0x02ee3c thru 0x03ca11 (0xdbd6 bytes) 000612 1752688389.489171 trunc from 0x03ca12 to 0x017b4d (0x24ec4 bytes) 000616 1752688389.511756 write 0x009141 thru 0x014eb1 (0xbd71 bytes) 000618 1752688389.521114 trunc from 0x017b4e to 0x01b0f5 (0x35a8 bytes) 000620 1752688389.545134 read 0x013e7c thru 0x01b0f5 (0x727a bytes) 000621 1752688389.560819 MIRROR_EXTEND to 3 mirrors 000622 1752688390.070957 mapread 0x002210 thru 0x00c0d4 (0x9ec5 bytes) truncating to largest ever: 0x3c42e 000623 1752688390.117719 trunc from 0x01b0f6 to 0x03c42d (0x21338 bytes) 000624 1752688390.273880 MIRROR_EXTEND to 4 mirrors 000625 1752688390.865543 trunc from 0x03c42e to 0x029754 (0x12cd9 bytes) 000626 1752688390.951229 mapread 0x024e17 thru 0x029754 (0x493e bytes) 000627 1752688390.978636 mapwrite 0x03dde8 thru 0x03ffff (0x2218 bytes) 000628 1752688391.097793 read 0x026a30 thru 0x033e67 (0xd438 bytes) 000629 1752688391.108359 mapread 0x00cf4d thru 0x013df3 (0x6ea7 bytes) 000633 1752688391.119768 trunc from 0x040000 to 0x039499 (0x6b66 bytes) 000636 1752688391.142274 mapread 0x00a064 thru 0x00de51 (0x3dee bytes) 000637 1752688391.143146 read 0x0023ec thru 0x005fdf (0x3bf4 bytes) 000638 1752688391.156657 mapwrite 0x0038f1 thru 0x005c3c (0x234c bytes) 000640 1752688391.197227 trunc from 0x03949a to 0x03434f (0x514a bytes) 000642 1752688391.220852 read 0x01f332 thru 0x026837 (0x7506 bytes) 000643 1752688391.221405 mapwrite 0x039135 thru 0x039b3e (0xa0a bytes) 000646 1752688391.315465 read 0x026074 thru 0x02d876 (0x7803 bytes) 000648 1752688391.316540 trunc from 0x039b3f to 0x01788c (0x222b2 bytes) 000649 1752688391.343626 write 0x034e66 thru 0x03ffff (0xb19a bytes) 000650 1752688391.345039 trunc from 0x040000 to 0x0208b8 (0x1f747 bytes) 000653 1752688391.373359 write 0x038535 thru 0x038b75 (0x641 bytes) 000658 1752688391.386293 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000659 1752688391.656678 mapread 0x02e175 thru 0x038b75 (0xaa01 bytes) 000660 1752688391.692734 MIRROR_RESYNC 3 mirrors 000661 1752688392.378998 mapwrite 0x03e2f9 thru 0x03ffff (0x1d07 bytes) 000662 1752688392.519592 trunc from 0x040000 to 0x03be39 (0x41c6 bytes) 000665 1752688392.542824 mapread 0x0390a0 thru 0x03be39 (0x2d9a bytes) 000668 1752688392.564989 MIRROR_EXTEND to 4 mirrors 000671 1752688393.064591 mapwrite 0x00ad5a thru 0x01444c (0x96f3 bytes) 000672 1752688393.307469 read 0x00f00e thru 0x011dfa (0x2ded bytes) 000674 1752688393.308576 read 0x038e22 thru 0x03be39 (0x3018 bytes) 000675 1752688393.321580 read 0x0185ac thru 0x023af3 (0xb548 bytes) 000676 1752688393.327071 MIRROR_RESYNC 4 mirrors 000678 1752688393.974311 read 0x01aea5 thru 0x021bde (0x6d3a bytes) 000681 1752688393.983017 mapread 0x014b36 thru 0x01dfb5 (0x9480 bytes) 000683 1752688394.013584 mapread 0x014df4 thru 0x021579 (0xc786 bytes) 000684 1752688394.014228 read 0x026fce thru 0x02e659 (0x768c bytes) 000685 1752688394.024075 write 0x0061cf thru 0x008127 (0x1f59 bytes) 000686 1752688394.133481 mapwrite 0x03ecb3 thru 0x03ffff (0x134d bytes) 000687 1752688394.271022 mapread 0x03a500 thru 0x03ae4f (0x950 bytes) 000688 1752688394.282901 read 0x002f7d thru 0x006a54 (0x3ad8 bytes) 000692 1752688394.307620 mapread 0x0345a1 thru 0x03e70f (0xa16f bytes) 000693 1752688394.309443 trunc from 0x040000 to 0x026177 (0x19e88 bytes) 000694 1752688394.335944 read 0x0122a4 thru 0x012634 (0x391 bytes) 000696 1752688394.341631 MIRROR_RESYNC 4 mirrors 000697 1752688395.060607 mapwrite 0x030d3b thru 0x03d55d (0xc823 bytes) 000699 1752688395.317998 write 0x01f730 thru 0x029fdb (0xa8ac bytes) 000700 1752688395.348694 mapwrite 0x033d61 thru 0x03fa26 (0xbcc6 bytes) 000702 1752688395.429841 trunc from 0x03fa27 to 0x01f9d2 (0x20054 bytes) 000703 1752688395.450238 mapwrite 0x000e76 thru 0x00da11 (0xcb9c bytes) 000704 1752688395.515192 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000705 1752688395.776250 trunc from 0x01f9d3 to 0x00faa4 (0xff2e bytes) 000706 1752688395.791976 trunc from 0x00faa5 to 0x0352c3 (0x2581f bytes) 000707 1752688395.818668 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000711 1752688396.059260 MIRROR_RESYNC 2 mirrors 000712 1752688396.570401 read 0x0004c2 thru 0x0021c4 (0x1d03 bytes) 000713 1752688396.581747 mapwrite 0x01d4b8 thru 0x0206ab (0x31f4 bytes) 000715 1752688396.756800 MIRROR_EXTEND to 3 mirrors 000716 1752688397.306202 mapwrite 0x037892 thru 0x03cb17 (0x5286 bytes) 000717 1752688397.477339 read 0x034de6 thru 0x035fd5 (0x11f0 bytes) 000720 1752688397.478921 read 0x00e4ea thru 0x015851 (0x7368 bytes) 000721 1752688397.492659 write 0x02e927 thru 0x0369f5 (0x80cf bytes) 000722 1752688397.493590 trunc from 0x03cb18 to 0x0003e0 (0x3c737 bytes) 000723 1752688397.517567 mapwrite 0x015f6d thru 0x018636 (0x26ca bytes) 000725 1752688397.627776 mapread 0x009e37 thru 0x00e1af (0x4379 bytes) 000728 1752688397.655973 mapwrite 0x006b1a thru 0x00fd36 (0x921d bytes) 000729 1752688397.722197 read 0x010859 thru 0x01309a (0x2842 bytes) 000730 1752688397.739867 MIRROR_RESYNC 3 mirrors 000731 1752688398.258067 trunc from 0x018637 to 0x030c91 (0x1865b bytes) 000732 1752688398.323524 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000735 1752688398.558141 trunc from 0x030c92 to 0x015a79 (0x1b218 bytes) 000736 1752688398.582245 read 0x008e84 thru 0x0097aa (0x927 bytes) 000737 1752688398.590323 mapread 0x003152 thru 0x011a86 (0xe935 bytes) 000740 1752688398.626792 mapwrite 0x039e8a thru 0x03fc1a (0x5d91 bytes) 000741 1752688398.733618 MIRROR_EXTEND to 3 mirrors 000745 1752688399.189265 read 0x008e23 thru 0x014607 (0xb7e5 bytes) 000749 1752688399.208782 mapwrite 0x01a0ea thru 0x01fa1b (0x5932 bytes) 000751 1752688399.403499 write 0x029441 thru 0x033240 (0x9e00 bytes) 000753 1752688399.421383 write 0x039b3b thru 0x03ffff (0x64c5 bytes) 000754 1752688399.431472 write 0x012da4 thru 0x016c65 (0x3ec2 bytes) 000755 1752688399.439778 mapread 0x0230b2 thru 0x026d15 (0x3c64 bytes) 000756 1752688399.446523 MIRROR_EXTEND to 4 mirrors 000757 1752688399.943405 mapread 0x032710 thru 0x03b333 (0x8c24 bytes) 000762 1752688399.993816 read 0x030441 thru 0x038cf0 (0x88b0 bytes) 000763 1752688399.998198 write 0x01f4ab thru 0x027b50 (0x86a6 bytes) 000764 1752688400.064324 MIRROR_EXTEND to 5 mirrors 000766 1752688400.638997 mapread 0x002475 thru 0x00f1a2 (0xcd2e bytes) 000769 1752688400.696308 read 0x036ac5 thru 0x03ffff (0x953b bytes) 000770 1752688400.714860 MIRROR_EXTEND to 6 mirrors 000771 1752688401.288133 trunc from 0x040000 to 0x03b6ca (0x4935 bytes) 000773 1752688401.374631 mapwrite 0x02177f thru 0x02b500 (0x9d82 bytes) 000774 1752688401.443081 write 0x02ce84 thru 0x038415 (0xb592 bytes) 000776 1752688401.463469 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000777 1752688401.688404 read 0x0278db thru 0x02f5af (0x7cd5 bytes) 000781 1752688401.710814 trunc from 0x03b6cb to 0x011acb (0x29bff bytes) 000782 1752688401.740798 trunc from 0x011acc to 0x002dea (0xece1 bytes) 000784 1752688401.759205 write 0x01742b thru 0x0249ca (0xd5a0 bytes) 000785 1752688401.761143 trunc from 0x0249cb to 0x00491b (0x200af bytes) 000791 1752688401.785733 mapread 0x00209f thru 0x00491b (0x287d bytes) 000793 1752688401.809834 mapwrite 0x03d152 thru 0x03ffff (0x2eae bytes) 000795 1752688401.895268 MIRROR_EXTEND to 6 mirrors 000796 1752688402.501166 mapwrite 0x01872c thru 0x01bbd8 (0x34ad bytes) 000797 1752688402.676231 mapread 0x002c3b thru 0x006f0f (0x42d5 bytes) 000798 1752688402.693793 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000800 1752688402.897157 MIRROR_RESYNC 5 mirrors 000801 1752688403.659789 trunc from 0x040000 to 0x033baa (0xc455 bytes) 000805 1752688403.748086 write 0x02dd62 thru 0x03d506 (0xf7a5 bytes) 000807 1752688403.757892 trunc from 0x03d507 to 0x03842f (0x50d7 bytes) 000808 1752688403.775265 mapread 0x0042b5 thru 0x010cb7 (0xca03 bytes) 000809 1752688403.794221 mapwrite 0x014fb8 thru 0x01905f (0x40a8 bytes) 000810 1752688403.860232 mapread 0x01a1e9 thru 0x024229 (0xa041 bytes) 000815 1752688403.876620 MIRROR_EXTEND to 6 mirrors 000817 1752688404.385910 trunc from 0x038430 to 0x01cda6 (0x1b689 bytes) 000819 1752688404.473581 mapread 0x01a996 thru 0x01cda6 (0x2411 bytes) 000822 1752688404.500344 mapread 0x006037 thru 0x006724 (0x6ee bytes) 000824 1752688404.515947 mapread 0x00423b thru 0x004711 (0x4d7 bytes) 000827 1752688404.518040 read 0x0125ff thru 0x0129be (0x3c0 bytes) 000829 1752688404.526400 mapread 0x01889c thru 0x01cda6 (0x450b bytes) 000831 1752688404.528116 trunc from 0x01cda7 to 0x016430 (0x6976 bytes) 000832 1752688404.553832 write 0x028c64 thru 0x02f71e (0x6abb bytes) 000833 1752688404.557383 trunc from 0x02f71f to 0x007c39 (0x27ae5 bytes) 000835 1752688404.583136 mapread 0x004334 thru 0x007c39 (0x3906 bytes) 000836 1752688404.584186 mapread 0x0018dc thru 0x00692b (0x5050 bytes) 000838 1752688404.587648 mapread 0x00768a thru 0x007c39 (0x5b0 bytes) 000840 1752688404.589106 trunc from 0x007c3a to 0x01ca14 (0x14ddb bytes) 000843 1752688404.605972 mapwrite 0x01576b thru 0x01c650 (0x6ee6 bytes) 000844 1752688404.665328 mapread 0x0192cc thru 0x01ab6c (0x18a1 bytes) 000845 1752688404.672303 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000846 1752688404.882814 trunc from 0x01ca15 to 0x00fce2 (0xcd32 bytes) 000848 1752688404.919474 MIRROR_RESYNC 5 mirrors 000849 1752688405.606104 trunc from 0x00fce3 to 0x017dc9 (0x80e7 bytes) 000851 1752688405.677895 write 0x006cab thru 0x00d721 (0x6a77 bytes) 000852 1752688405.686953 trunc from 0x017dca to 0x0372c2 (0x1f4f9 bytes) 000854 1752688405.707531 mapwrite 0x0216c8 thru 0x02ca58 (0xb391 bytes) 000855 1752688405.782864 MIRROR_RESYNC 5 mirrors 000856 1752688406.395241 read 0x02e434 thru 0x0372c2 (0x8e8f bytes) 000857 1752688406.402353 mapwrite 0x015ab1 thru 0x020123 (0xa673 bytes) 000858 1752688406.571808 trunc from 0x0372c3 to 0x01f0ce (0x181f4 bytes) 000860 1752688406.591111 mapwrite 0x028ed1 thru 0x0356e8 (0xc818 bytes) 000862 1752688406.687341 mapwrite 0x0371ab thru 0x03ffff (0x8e55 bytes) 000863 1752688406.791606 mapwrite 0x026972 thru 0x0294a5 (0x2b34 bytes) 000864 1752688406.834864 mapwrite 0x01f0ba thru 0x024527 (0x546e bytes) 000865 1752688406.875131 mapwrite 0x03cbe8 thru 0x03ffff (0x3418 bytes) 000866 1752688406.909703 read 0x0246c1 thru 0x024b8b (0x4cb bytes) 000868 1752688406.910854 read 0x009007 thru 0x00e5dc (0x55d6 bytes) 000869 1752688406.918150 write 0x03d8a0 thru 0x03dfa0 (0x701 bytes) 000870 1752688406.919143 mapwrite 0x004188 thru 0x007842 (0x36bb bytes) 000871 1752688406.955260 write 0x03a92a thru 0x03f6b7 (0x4d8e bytes) 000872 1752688406.956351 read 0x039a65 thru 0x03ffff (0x659b bytes) 000874 1752688406.969872 MIRROR_RESYNC 5 mirrors 000875 1752688407.562506 MIRROR_EXTEND to 6 mirrors 000876 1752688407.994368 mapwrite 0x03912c thru 0x03ffff (0x6ed4 bytes) 000877 1752688408.135868 write 0x013cc2 thru 0x016c30 (0x2f6f bytes) 000879 1752688408.150739 trunc from 0x040000 to 0x01b5ee (0x24a11 bytes) 000880 1752688408.177894 read 0x00ec9d thru 0x01b206 (0xc56a bytes) 000881 1752688408.178747 mapwrite 0x01e750 thru 0x02843d (0x9cee bytes) 000882 1752688408.281545 trunc from 0x02843e to 0x019d14 (0xe729 bytes) 000883 1752688408.300464 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000885 1752688408.474874 trunc from 0x019d15 to 0x03a44f (0x2073b bytes) 000886 1752688408.493326 write 0x0381d8 thru 0x03ffff (0x7e28 bytes) 000887 1752688408.498465 read 0x00cd96 thru 0x013b06 (0x6d71 bytes) 000888 1752688408.512672 MIRROR_RESYNC 5 mirrors 000889 1752688409.032217 trunc from 0x040000 to 0x001a5b (0x3e5a4 bytes) 000891 1752688409.111392 write 0x010bf6 thru 0x010ee5 (0x2f0 bytes) 000892 1752688409.112589 trunc from 0x010ee6 to 0x013405 (0x2520 bytes) 000894 1752688409.129572 mapread 0x005cac thru 0x00a541 (0x4896 bytes) 000895 1752688409.153185 trunc from 0x013406 to 0x01db77 (0xa772 bytes) 000896 1752688409.166819 trunc from 0x01db78 to 0x029baf (0xc038 bytes) 000898 1752688409.186742 mapread 0x02168c thru 0x02446f (0x2de4 bytes) 000900 1752688409.200769 mapwrite 0x0394f0 thru 0x03ee48 (0x5959 bytes) 000901 1752688409.301359 mapwrite 0x012b71 thru 0x01a21f (0x76af bytes) 000903 1752688409.351752 read 0x00a72c thru 0x00d1b0 (0x2a85 bytes) 000904 1752688409.358245 MIRROR_RESYNC 5 mirrors 000907 1752688409.952593 read 0x00f09a thru 0x01106a (0x1fd1 bytes) 000909 1752688409.962905 MIRROR_RESYNC 5 mirrors 000910 1752688410.126141 read 0x00ee63 thru 0x0163d8 (0x7576 bytes) 000912 1752688410.137718 mapread 0x02400d thru 0x029356 (0x534a bytes) 000914 1752688410.159509 mapwrite 0x03c418 thru 0x03ffff (0x3be8 bytes) 000915 1752688410.294011 mapwrite 0x0148b6 thru 0x01fdfd (0xb548 bytes) 000917 1752688410.344845 trunc from 0x040000 to 0x039130 (0x6ecf bytes) 000918 1752688410.366205 mapread 0x012b56 thru 0x01937e (0x6829 bytes) 000919 1752688410.368379 mapwrite 0x01a233 thru 0x01efc9 (0x4d97 bytes) 000920 1752688410.442644 write 0x02e0be thru 0x0325c1 (0x4504 bytes) 000922 1752688410.456181 read 0x02ed5b thru 0x036b80 (0x7e26 bytes) 000924 1752688410.458226 mapwrite 0x030a56 thru 0x0359c6 (0x4f71 bytes) 000925 1752688410.539400 mapread 0x036e26 thru 0x0374dd (0x6b8 bytes) 000927 1752688410.540889 mapwrite 0x01bb7d thru 0x0283b9 (0xc83d bytes) 000928 1752688410.609057 trunc from 0x039131 to 0x029d40 (0xf3f0 bytes) 000929 1752688410.621803 read 0x01096a thru 0x01a885 (0x9f1c bytes) 000930 1752688410.622577 read 0x000fe2 thru 0x0082f1 (0x7310 bytes) 000931 1752688410.628584 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000932 1752688410.802945 mapread 0x00c9bc thru 0x01b57a (0xebbf bytes) 000933 1752688410.827890 read 0x00455a thru 0x0059a5 (0x144c bytes) 000934 1752688410.835033 mapwrite 0x003935 thru 0x0101fc (0xc8c8 bytes) 000935 1752688410.895599 write 0x034ebf thru 0x03dd51 (0x8e93 bytes) 000937 1752688410.898231 mapwrite 0x02425c thru 0x02a5f8 (0x639d bytes) 000938 1752688410.963208 trunc from 0x03dd52 to 0x00f2fe (0x2ea53 bytes) 000939 1752688410.989098 mapwrite 0x03f20b thru 0x03ffff (0xdf5 bytes) 000940 1752688411.071288 MIRROR_EXTEND to 5 mirrors 000944 1752688411.516576 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000946 1752688411.855942 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21d:0x0]: Invalid argument (22) 000947 1752688412.178492 mapread 0x0108d7 thru 0x01af22 (0xa64c bytes) 000950 1752688412.227083 mapwrite 0x028bd1 thru 0x033bf9 (0xb029 bytes) 000951 1752688412.419908 mapread 0x025418 thru 0x028a38 (0x3621 bytes) 000952 1752688412.420836 read 0x0198b4 thru 0x02274f (0x8e9c bytes) 000953 1752688412.430096 mapwrite 0x03f591 thru 0x03ffff (0xa6f bytes) 000954 1752688412.486840 trunc from 0x040000 to 0x02e48e (0x11b71 bytes) 000955 1752688412.522826 write 0x03c331 thru 0x03ffff (0x3ccf bytes) 000956 1752688412.528306 MIRROR_RESYNC 3 mirrors 000957 1752688413.006048 write 0x0226de thru 0x028c1e (0x6541 bytes) 000958 1752688413.080300 mapwrite 0x03ebf6 thru 0x03ffff (0x140a bytes) 000959 1752688413.126658 mapread 0x02a65c thru 0x02e896 (0x423b bytes) 000960 1752688413.140902 write 0x010160 thru 0x018b3a (0x89db bytes) 000963 1752688413.152736 mapwrite 0x03913a thru 0x03ffff (0x6ec6 bytes) 000967 1752688413.197327 mapread 0x01ef6c thru 0x02a504 (0xb599 bytes) 000969 1752688413.210923 trunc from 0x040000 to 0x0169ea (0x29615 bytes) 000971 1752688413.245052 trunc from 0x0169eb to 0x0037d8 (0x13212 bytes) 000972 1752688413.262400 mapread 0x001aee thru 0x002ce0 (0x11f3 bytes) 000973 1752688413.271452 trunc from 0x0037d9 to 0x027519 (0x23d41 bytes) 000974 1752688413.291763 write 0x037644 thru 0x03ad2e (0x36eb bytes) 000975 1752688413.293264 read 0x039325 thru 0x03ad2e (0x1a0a bytes) 000976 1752688413.294042 write 0x0143ca thru 0x023c7c (0xf8b3 bytes) 000977 1752688413.310974 trunc from 0x03ad2f to 0x01e3ca (0x1c964 bytes) 000978 1752688413.325105 mapwrite 0x03da4e thru 0x03ffff (0x25b2 bytes) 000979 1752688413.397108 read 0x00864f thru 0x018301 (0xfcb3 bytes) 000980 1752688413.405772 write 0x022066 thru 0x02b8cc (0x9867 bytes) 000983 1752688413.408106 read 0x011ab8 thru 0x018daa (0x72f3 bytes) 000985 1752688413.414711 MIRROR_RESYNC 3 mirrors 000987 1752688413.826242 read 0x033610 thru 0x03fc1b (0xc60c bytes) 000988 1752688413.835341 mapwrite 0x003185 thru 0x005023 (0x1e9f bytes) 000992 1752688414.009336 trunc from 0x040000 to 0x028909 (0x176f6 bytes) 000993 1752688414.043841 mapread 0x013f40 thru 0x0167de (0x289f bytes) 000995 1752688414.053371 read 0x014fb1 thru 0x024edd (0xff2d bytes) 000996 1752688414.061320 mapread 0x00eef8 thru 0x019e16 (0xaf1f bytes) 000999 1752688414.064096 read 0x00de41 thru 0x01222e (0x43ee bytes) 001000 1752688414.064911 read 0x004513 thru 0x008b98 (0x4686 bytes) All operations completed A-OK! PASS 70a (62s) == sanity-flr test 71: check mirror extend parent fid ==== 13:53:36 (1752688416) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 200a: stress test ===================== 13:53:38 (1752688418) Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre2 Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5199872 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2127968 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8604128 .. Extending file size to 7540928 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7668192 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8203104 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8715840 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8812288 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9078720 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8818208 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9147712 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 55015 55016 55019 55020 55021 192.168.201.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg142-client.virtnet /mnt/lustre2 (opts:) 192.168.201.142@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg142-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (70s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:54:48 (1752688488) Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre2 Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5703936 .. Extending file size to 8591360 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 6383648 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7166784 .. Extending file size to 8932256 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8637024 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9017152 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8671840 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8680832 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9214560 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8975648 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 57791 57792 57793 57796 57798 192.168.201.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg142-client.virtnet /mnt/lustre2 (opts:) 192.168.201.142@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg142-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (68s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:55:56 (1752688556) Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4224 2204160 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4224 2204160 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0504756 s, 41.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.183075 s, 11.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg142-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:56:05 (1752688565) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0591434 s, 35.5 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 13:56:09 (1752688569) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x267:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x319:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x319:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31a:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:56:13 (1752688573) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.063887 s, 32.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:56:17 (1752688577) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116151 s, 18.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:56:21 (1752688581) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0554823 s, 37.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } PASS 204c (26s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:56:47 (1752688607) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg142-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457736 s, 22.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x320:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:57:32 (1752688652) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg142-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0615093 s, 34.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x321:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x321:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:58:19 (1752688699) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg142-server mds-ost sync done. Creating new pool test_204f oleg142-server: Pool lustre.test_204f created Adding targets to pool oleg142-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg142-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0541045 s, 38.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x322:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x322:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg142-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg142-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg142-server: Pool lustre.test_204f destroyed PASS 204f (58s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:59:17 (1752688757) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:59:21 (1752688761) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:59:24 (1752688764) oleg142-server: Pool lustre.test_206 created Adding targets to pool oleg142-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg142-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg142-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg142-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg142-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:59:40 (1752688780) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03174 s, 33.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193219 s, 54.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x8e9f7380 Mirror 3: 0x8e9f7380 PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:59:45 (1752688785) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:59:47 (1752688787) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:59:48 (1752688788) Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0492167 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa80000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x920000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x920000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x920000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1650000 Read this (4 bytes): BBBB 192.168.201.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg142-client.virtnet /mnt/lustre2 (opts:) PASS 209a (20s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:00:08 (1752688808) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00693006 s, 1.2 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:00:11 (1752688811) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267974 s, 39.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x28c lmm_fid: [0x200000404:0x28c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x28c lmm_fid: [0x200000404:0x28c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834572 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-16 14:00:12.000000000 -0400 Modify: 2025-07-16 14:00:12.000000000 -0400 Change: 2025-07-16 14:00:12.000000000 -0400 Birth: 2025-07-16 14:00:11.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:00:15 (1752688815) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:00:16 (1752688816) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.285353 s, 144 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0441085 s, 92.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 16 14:00 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x28e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 16 14:00 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test complete, duration 986 sec ============ 14:00:21 (1752688821) === sanity-flr: start cleanup 14:00:21 (1752688821) === === sanity-flr: finish cleanup 14:00:24 (1752688824) ===