-----============= acceptance-small: sanity-flr ============----- Tue Aug 25 07:50:03 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg207-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 07:50:26 (1787658626) === oleg207-client.virtnet: executing check_config_client /mnt/lustre oleg207-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg207-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d91d08fb800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d91d08fb800.idle_timeout=debug disable quota as required oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 07:51:09 (1787658669) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7626] == sanity-flr test 0a: lfs mirror create with -N option == 07:51:11 (1787658671) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:51:32 (1787658692) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:51:36 (1787658696) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:51:40 (1787658700) PASS 0d (34s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:52:13 (1787658733) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:52:17 (1787658737) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 07:52:21 (1787658741) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d91d08fb800.stats=clear osc.lustre-OST0001-osc-ffff8d91d08fb800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:52:30 (1787658750) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:52:43 (1787658763) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:52:52 (1787658772) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 07:52:56 (1787658776) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:53:06 (1787658786) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 07:53:15 (1787658795) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:53:24 (1787658804) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:53:29 (1787658809) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:53:37 (1787658817) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:53:51 (1787658831) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:53:58 (1787658838) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 1.56157 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178566 s, 5.9 MB/s PASS 21 (14s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:54:12 (1787658852) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.57511 s, 23.7 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 07:54:22 (1787658862) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (14s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:54:36 (1787658876) Stopping /mnt/lustre-ost1 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d91d08fb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d91d08fb800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg207-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d91d08fb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d91d08fb800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg207-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (96s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:56:12 (1787658972) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.227604 s, 13.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d91d08fb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d91d08fb800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg207-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (64s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:57:17 (1787659037) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d91d08fb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d91d08fb800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg207-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d91d08fb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d91d08fb800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg207-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (92s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:58:49 (1787659129) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.58937 s, 9.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 25 07:58 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.087s user 0m0.001s sys 0m0.176s Stopping /mnt/lustre-ost1 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d91d08fb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d91d08fb800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.147s user 0m0.001s sys 0m0.171s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg207-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d91d08fb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d91d08fb800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.568s user 0m0.000s sys 0m0.157s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg207-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg207-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d91d08fb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d91d08fb800.ost_server_uuid in FULL state after 0 sec PASS 33b (104s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:00:34 (1787659234) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:00:37 (1787659237) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:00:41 (1787659241) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 08:00:45 (1787659245) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 08:00:52 (1787659252) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.721259 s, 17.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:01:07 (1787659267) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469817 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256698 s, 40.8 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:01:15 (1787659275) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:01:23 (1787659283) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000748987 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00099163 s, 1.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00110397 s, 928 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00630466 s, 162 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 08:01:36 (1787659296) create test files with size 15750396 .. 78e26c88ec07b68d0c1936237a139008 - f96af1c0402ec86b593d1330d81558c3 - b0c5d99f0afdd15711d93810f3ef60ae - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg207-client.virtnet /mnt/lustre (opts:) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre PASS 37 (30s) == sanity-flr test 38: resync ============================ 08:02:07 (1787659327) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:02:22 (1787659342) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:02:29 (1787659349) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104554 s, 10.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284941 s, 36.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 08:02:38 (1787659358) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141308 s, 14.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.356606 s, 11.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.233723 s, 9.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 08:02:58 (1787659378) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 08:03:01 (1787659381) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:03:06 (1787659386) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200032 s, 52.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 08:03:14 (1787659394) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:03:18 (1787659398) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:03:28 (1787659408) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.289834 s, 36.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg207-client.virtnet /mnt/lustre (opts:) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:03:41 (1787659421) llite.lustre-ffff8d91d108f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.311669 s, 33.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg207-client.virtnet /mnt/lustre (opts:) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8d91c4a9b000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:03:54 (1787659434) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (34s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:04:29 (1787659469) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:05:10 (1787659510) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 08:05:14 (1787659514) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.244016 s, 12.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0692949 s, 45.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x846884af (nosync,stale) Mirror 2: 0x846884af (nosync,stale) ** verify mirror 2 doesn't change original checksum: fc624592a093ffd4591a3854910fc17d - mirror 1 checksum: 242dd073895f926f344814ad2254aa78 - mirror 2 checksum: fc624592a093ffd4591a3854910fc17d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-25T08:05:17-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-25T08:05:17-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: fc624592a093ffd4591a3854910fc17d - mirror 1 checksum: 242dd073895f926f344814ad2254aa78 - mirror 2 checksum: 242dd073895f926f344814ad2254aa78 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-25T08:05:17-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-25T08:05:17-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:05:28 (1787659528) Failing mds1 on oleg207-server Stopping /mnt/lustre-mds1 (opts:) on oleg207-server 08:05:35 (1787659535) shut down facet: mds1 facet_host: oleg207-server facet_failover_host: oleg207-server Failover mds1 to oleg207-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 08:05:58 (1787659558) targets are mounted 08:05:58 (1787659558) facet_failover done oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266237 s, 39.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:06:16 (1787659576) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:06:19 (1787659579) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:06:22 (1787659582) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.010393 s, 1.6 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:06:29 (1787659589) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:06:32 (1787659592) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0077204 s, 531 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0528548 s, 77.5 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:06:39 (1787659599) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:07:03 (1787659623) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:07:33 (1787659653) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:07:56 (1787659676) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:08:20 (1787659700) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 08:08:28 (1787659708) mirror create pid 54342 mirror split pid 54343 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54342 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54343 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 08:09:34 (1787659774) Seed set to 9776 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x23035 000001 1787659776.831995 trunc from 00000000 to 0x023034 (0x23035 bytes) 000004 1787659776.855133 mapread 0x01e1f0 thru 0x01f6e5 (0x14f6 bytes) 000005 1787659776.868332 trunc from 0x023035 to 0x011ca4 (0x11390 bytes) 000007 1787659776.887390 mapread 0x008df1 thru 0x011ca4 (0x8eb4 bytes) 000009 1787659776.920250 MIRROR_EXTEND to 2 mirrors 000010 1787659778.069027 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000012 1787659778.212468 trunc from 0x011ca5 to 0x014ed1 (0x322d bytes) 000013 1787659778.249935 write 0x0329fa thru 0x035e15 (0x341c bytes) 000015 1787659778.258869 write 0x02c1af thru 0x039006 (0xce58 bytes) 000016 1787659778.268733 mapread 0x007478 thru 0x01585f (0xe3e8 bytes) 000021 1787659778.311225 trunc from 0x039007 to 0x014fb8 (0x2404e bytes) 000022 1787659778.336887 MIRROR_EXTEND to 2 mirrors 000024 1787659779.241371 mapwrite 0x03f2f8 thru 0x03ffff (0xd08 bytes) 000025 1787659779.437154 trunc from 0x040000 to 0x00a036 (0x35fc9 bytes) 000027 1787659779.483296 mapwrite 0x02d0f8 thru 0x031747 (0x4650 bytes) 000028 1787659779.648409 mapread 0x026b05 thru 0x03173a (0xac36 bytes) 000029 1787659779.695489 mapread 0x01db35 thru 0x020afb (0x2fc7 bytes) 000030 1787659779.704507 trunc from 0x031748 to 0x012c87 (0x1eac0 bytes) 000031 1787659779.722863 trunc from 0x012c88 to 0x005721 (0xd566 bytes) 000034 1787659779.749402 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x3ab81 000035 1787659780.294631 trunc from 0x005722 to 0x03ab80 (0x3545f bytes) 000038 1787659780.376994 mapwrite 0x006157 thru 0x01316e (0xd018 bytes) 000039 1787659780.503116 write 0x019e79 thru 0x01bae2 (0x1c6a bytes) 000040 1787659780.536256 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) truncating to largest ever: 0x3cc50 000041 1787659780.781879 trunc from 0x03ab81 to 0x03cc4f (0x20cf bytes) 000042 1787659780.811620 write 0x034e2a thru 0x037520 (0x26f7 bytes) 000043 1787659780.825485 mapread 0x028de5 thru 0x02cc3a (0x3e56 bytes) 000044 1787659780.850901 trunc from 0x03cc50 to 0x01d3b7 (0x1f898 bytes) 000046 1787659780.875814 trunc from 0x01d3b8 to 0x0099ff (0x139b8 bytes) 000048 1787659780.916048 write 0x0189fc thru 0x01e998 (0x5f9d bytes) 000049 1787659780.918279 trunc from 0x01e999 to 0x037f2a (0x19592 bytes) 000050 1787659780.951363 mapwrite 0x035769 thru 0x03ffff (0xa897 bytes) 000053 1787659781.072292 trunc from 0x040000 to 0x000dbd (0x3f242 bytes) 000054 1787659781.105778 write 0x0032f9 thru 0x005b0b (0x2813 bytes) 000055 1787659781.107194 write 0x00d3a9 thru 0x012f28 (0x5b80 bytes) 000056 1787659781.121426 read 0x010429 thru 0x012f28 (0x2b00 bytes) 000057 1787659781.122871 write 0x026f7a thru 0x0341fd (0xd284 bytes) 000058 1787659781.133003 trunc from 0x0341fe to 0x0007e4 (0x33a19 bytes) 000066 1787659781.170672 mapwrite 0x0055f7 thru 0x00dcf4 (0x86fe bytes) 000067 1787659781.278704 write 0x03e9d4 thru 0x03ffff (0x162c bytes) 000068 1787659781.279941 mapread 0x023870 thru 0x031159 (0xd8ea bytes) 000070 1787659781.306586 MIRROR_EXTEND to 2 mirrors 000072 1787659782.424870 MIRROR_EXTEND to 3 mirrors 000076 1787659783.443307 mapwrite 0x0091c8 thru 0x010c68 (0x7aa1 bytes) 000077 1787659783.692724 mapread 0x020d16 thru 0x03017a (0xf465 bytes) 000078 1787659783.724253 read 0x03ee0c thru 0x03ffff (0x11f4 bytes) 000080 1787659783.739258 trunc from 0x040000 to 0x0374f6 (0x8b09 bytes) 000081 1787659783.791375 MIRROR_RESYNC 3 mirrors 000082 1787659784.983333 MIRROR_RESYNC 3 mirrors 000083 1787659785.341173 trunc from 0x0374f7 to 0x014bf9 (0x228fd bytes) 000084 1787659785.425562 read 0x011beb thru 0x014bf9 (0x300f bytes) 000085 1787659785.433292 write 0x016ba2 thru 0x016e23 (0x282 bytes) 000086 1787659785.434644 mapread 0x00acdb thru 0x016e23 (0xc149 bytes) 000087 1787659785.466431 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x16e24): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16e24): Mirror 2: 0xd2ecd083 (stale) Mirror 3: 0xd2ecd083 (stale) CRC-32 checksum value for chunk [0, 0x16e24): Mirror 1: 0x8a004ce8 Mirror 4: 0x8a004ce8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16e24: skipped 000092 1787659786.538569 mapread 0x014e92 thru 0x016e23 (0x1f92 bytes) 000093 1787659786.570697 trunc from 0x016e24 to 0x024067 (0xd244 bytes) 000096 1787659786.675934 read 0x007844 thru 0x010c23 (0x93e0 bytes) 000099 1787659786.689805 mapread 0x01c510 thru 0x024067 (0x7b58 bytes) 000100 1787659786.702197 read 0x0112e1 thru 0x0179d0 (0x66f0 bytes) 000101 1787659786.715691 read 0x014a69 thru 0x019cd1 (0x5269 bytes) 000102 1787659786.728888 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24068): Mirror 2: 0xcf753e28 (stale) Mirror 3: 0xcf753e28 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24068): Mirror 2: 0xcf753e28 (stale) Mirror 3: 0xcf753e28 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24068: skipped 000104 1787659787.013644 mapread 0x015cbe thru 0x01e554 (0x8897 bytes) 000105 1787659787.038555 mapwrite 0x03693d thru 0x03ffff (0x96c3 bytes) 000106 1787659787.140577 read 0x01b445 thru 0x01c7cb (0x1387 bytes) 000107 1787659787.148226 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc7daa546 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc7daa546 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000108 1787659787.451551 mapread 0x0281ec thru 0x02f8c8 (0x76dd bytes) 000110 1787659787.466245 read 0x03489c thru 0x03ffff (0xb764 bytes) 000112 1787659787.472931 mapread 0x0046ff thru 0x00f01a (0xa91c bytes) 000115 1787659787.485697 read 0x0133e0 thru 0x01ed4e (0xb96f bytes) 000116 1787659787.495244 write 0x000851 thru 0x006c4a (0x63fa bytes) 000117 1787659787.496282 read 0x0067d9 thru 0x014ec6 (0xe6ee bytes) 000118 1787659787.497924 mapwrite 0x013f03 thru 0x01989a (0x5998 bytes) 000119 1787659787.635494 write 0x03ce7f thru 0x03ffff (0x3181 bytes) 000120 1787659787.637815 write 0x0090d6 thru 0x013cea (0xac15 bytes) 000121 1787659787.642165 trunc from 0x040000 to 0x007ec4 (0x3813b bytes) 000123 1787659787.675856 trunc from 0x007ec5 to 0x01a5a2 (0x126de bytes) 000124 1787659787.696170 mapread 0x0047b9 thru 0x0120f2 (0xd93a bytes) 000125 1787659787.726423 mapread 0x00e215 thru 0x01559d (0x7389 bytes) 000126 1787659787.737694 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000127 1787659787.882781 mapread 0x00890b thru 0x00c8b5 (0x3fab bytes) 000131 1787659787.901023 read 0x0176f7 thru 0x0182f5 (0xbff bytes) 000132 1787659787.905116 trunc from 0x01a5a3 to 0x001ade (0x18ac4 bytes) 000133 1787659787.919751 trunc from 0x001adf to 0x0033fb (0x191d bytes) 000134 1787659787.937822 trunc from 0x0033fc to 0x006dab (0x39b0 bytes) 000137 1787659787.948748 trunc from 0x006dac to 0x03ac09 (0x33e5e bytes) 000138 1787659787.964583 write 0x0269b6 thru 0x03365c (0xcca7 bytes) 000139 1787659787.973197 read 0x01b17e thru 0x01d39e (0x2221 bytes) 000141 1787659787.980862 write 0x0198d6 thru 0x026806 (0xcf31 bytes) 000145 1787659787.983420 read 0x0268e0 thru 0x031c95 (0xb3b6 bytes) 000147 1787659787.985184 read 0x01b830 thru 0x02135b (0x5b2c bytes) 000148 1787659787.985764 trunc from 0x03ac0a to 0x029028 (0x11be1 bytes) 000149 1787659787.995480 mapread 0x019346 thru 0x022806 (0x94c1 bytes) 000150 1787659787.997018 mapwrite 0x009b93 thru 0x00f2e5 (0x5753 bytes) 000151 1787659788.031997 mapread 0x01538f thru 0x01c3d8 (0x704a bytes) 000153 1787659788.033435 trunc from 0x029029 to 0x00e288 (0x1ada0 bytes) 000154 1787659788.048637 trunc from 0x00e289 to 0x008d07 (0x5581 bytes) 000155 1787659788.060503 mapwrite 0x004a29 thru 0x0085d0 (0x3ba8 bytes) 000157 1787659788.108251 mapwrite 0x03b375 thru 0x03d11c (0x1da8 bytes) 000158 1787659788.187370 write 0x037141 thru 0x03ffff (0x8ebf bytes) 000160 1787659788.191565 mapwrite 0x0359ca thru 0x037e5b (0x2492 bytes) 000161 1787659788.249839 write 0x038ca8 thru 0x03fd78 (0x70d1 bytes) 000163 1787659788.253874 mapwrite 0x036f01 thru 0x03ffff (0x90ff bytes) 000165 1787659788.302453 mapwrite 0x024ba1 thru 0x028a61 (0x3ec1 bytes) 000166 1787659788.340854 read 0x0053b4 thru 0x0132cb (0xdf18 bytes) 000167 1787659788.341984 trunc from 0x040000 to 0x008a03 (0x375fc bytes) 000170 1787659788.365229 read 0x005c54 thru 0x008a03 (0x2db0 bytes) 000171 1787659788.366524 mapwrite 0x019d7f thru 0x01dc3a (0x3ebc bytes) 000172 1787659788.432926 write 0x0056bc thru 0x00d084 (0x79c9 bytes) 000175 1787659788.436845 trunc from 0x01dc3b to 0x006939 (0x17301 bytes) 000179 1787659788.463348 trunc from 0x00693a to 0x01e737 (0x17dfe bytes) 000180 1787659788.505942 write 0x02009a thru 0x026a62 (0x69c9 bytes) 000182 1787659788.509876 mapread 0x013c4a thru 0x01b4ab (0x7862 bytes) 000183 1787659788.605465 mapwrite 0x0315dc thru 0x0399b4 (0x83d9 bytes) 000185 1787659788.672929 mapread 0x0084ac thru 0x0106d8 (0x822d bytes) 000189 1787659788.717197 MIRROR_EXTEND to 2 mirrors 000191 1787659789.688720 read 0x02a075 thru 0x02dd3c (0x3cc8 bytes) 000192 1787659789.698401 trunc from 0x0399b5 to 0x001398 (0x3861c bytes) 000196 1787659789.794395 mapwrite 0x018f5d thru 0x023b93 (0xac37 bytes) 000197 1787659789.893924 read 0x003786 thru 0x00984a (0x60c5 bytes) 000198 1787659789.909510 write 0x0181a6 thru 0x027f4d (0xfda8 bytes) 000199 1787659789.911324 mapwrite 0x001a1e thru 0x0098ab (0x7e8e bytes) 000200 1787659790.014849 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000201 1787659790.190185 mapwrite 0x028957 thru 0x02de8d (0x5537 bytes) 000202 1787659790.302754 read 0x01d117 thru 0x01e067 (0xf51 bytes) 000207 1787659790.312472 write 0x01ad0c thru 0x01fdbf (0x50b4 bytes) 000208 1787659790.322222 mapread 0x011d09 thru 0x01cbcb (0xaec3 bytes) 000209 1787659790.331342 read 0x007be9 thru 0x00dcf8 (0x6110 bytes) 000210 1787659790.338547 write 0x02d0f0 thru 0x03cf4f (0xfe60 bytes) 000211 1787659790.340231 mapread 0x029365 thru 0x02bc4c (0x28e8 bytes) 000213 1787659790.341709 mapwrite 0x035872 thru 0x03ef87 (0x9716 bytes) 000214 1787659790.430616 write 0x026dbb thru 0x035eb0 (0xf0f6 bytes) 000217 1787659790.436234 trunc from 0x03ef88 to 0x02c101 (0x12e86 bytes) 000218 1787659790.458485 trunc from 0x02c102 to 0x01edcb (0xd336 bytes) 000219 1787659790.475002 read 0x01ec49 thru 0x01edcb (0x183 bytes) 000220 1787659790.475534 read 0x00d4af thru 0x01b515 (0xe067 bytes) 000222 1787659790.476781 read 0x0153b1 thru 0x01edcb (0x9a1b bytes) 000223 1787659790.478539 read 0x001478 thru 0x002929 (0x14b2 bytes) 000226 1787659790.481489 trunc from 0x01edcc to 0x030a7f (0x11cb4 bytes) 000229 1787659790.511220 read 0x00b75f thru 0x018fcb (0xd86d bytes) 000230 1787659790.512140 read 0x000464 thru 0x00babb (0xb658 bytes) 000231 1787659790.514408 trunc from 0x030a80 to 0x01aaf3 (0x15f8c bytes) 000232 1787659790.545281 write 0x032e3e thru 0x03764f (0x4812 bytes) 000237 1787659790.553513 write 0x01bcff thru 0x027420 (0xb722 bytes) 000238 1787659790.573660 read 0x02c618 thru 0x02cd9a (0x783 bytes) 000240 1787659790.580670 mapread 0x0040c9 thru 0x00cb92 (0x8aca bytes) 000241 1787659790.583677 mapwrite 0x0211de thru 0x02a818 (0x963b bytes) 000243 1787659790.662548 read 0x027b1b thru 0x02c524 (0x4a0a bytes) 000245 1787659790.665448 trunc from 0x037650 to 0x03b17f (0x3b30 bytes) 000246 1787659790.685135 read 0x001007 thru 0x006cb3 (0x5cad bytes) 000247 1787659790.686377 read 0x00b690 thru 0x00be10 (0x781 bytes) 000249 1787659790.689240 read 0x03498e thru 0x03b17f (0x67f2 bytes) 000250 1787659790.700008 mapwrite 0x02c17d thru 0x038507 (0xc38b bytes) 000253 1787659790.760228 mapwrite 0x01474b thru 0x01f2f0 (0xaba6 bytes) 000256 1787659790.818728 write 0x035bee thru 0x03c74e (0x6b61 bytes) 000257 1787659790.820830 mapwrite 0x005dc2 thru 0x0112e9 (0xb528 bytes) 000259 1787659790.946512 mapread 0x029398 thru 0x038fd0 (0xfc39 bytes) 000262 1787659790.949718 read 0x036ece thru 0x03c74e (0x5881 bytes) 000263 1787659790.950745 mapwrite 0x01fc47 thru 0x02c24c (0xc606 bytes) 000264 1787659791.036496 trunc from 0x03c74f to 0x0358ee (0x6e60 bytes) 000265 1787659791.061238 write 0x00523f thru 0x00e806 (0x95c8 bytes) 000266 1787659791.065156 write 0x014985 thru 0x021c6c (0xd2e8 bytes) 000267 1787659791.068244 mapwrite 0x030a1b thru 0x03c58e (0xbb74 bytes) 000268 1787659791.167416 read 0x012826 thru 0x01d075 (0xa850 bytes) 000269 1787659791.169122 mapread 0x00fed2 thru 0x018626 (0x8755 bytes) 000272 1787659791.174312 mapwrite 0x008379 thru 0x015572 (0xd1fa bytes) 000278 1787659791.236208 read 0x01d082 thru 0x023d5d (0x6cdc bytes) 000279 1787659791.237236 mapread 0x02ad35 thru 0x02e20e (0x34da bytes) 000280 1787659791.238900 write 0x03417a thru 0x03ffff (0xbe86 bytes) 000281 1787659791.241305 mapwrite 0x0242a0 thru 0x02527d (0xfde bytes) 000284 1787659791.284275 write 0x018ea7 thru 0x021fa3 (0x90fd bytes) 000290 1787659791.296931 read 0x03c60d thru 0x03ffff (0x39f3 bytes) 000291 1787659791.297798 write 0x038786 thru 0x03b606 (0x2e81 bytes) 000292 1787659791.298933 mapread 0x00a4a3 thru 0x00bb69 (0x16c7 bytes) 000294 1787659791.304813 write 0x02dd3e thru 0x0320c9 (0x438c bytes) 000296 1787659791.308286 mapwrite 0x004a76 thru 0x00c0dc (0x7667 bytes) 000299 1787659791.370260 mapread 0x020438 thru 0x0212e3 (0xeac bytes) 000300 1787659791.377212 MIRROR_EXTEND to 2 mirrors 000301 1787659791.960574 write 0x021ce0 thru 0x022b63 (0xe84 bytes) 000303 1787659792.106638 read 0x02614f thru 0x02d4b0 (0x7362 bytes) 000304 1787659792.118843 trunc from 0x040000 to 0x039542 (0x6abd bytes) 000308 1787659792.185858 mapread 0x02bb9f thru 0x0360c6 (0xa528 bytes) 000309 1787659792.211234 trunc from 0x039543 to 0x01c9c8 (0x1cb7a bytes) 000311 1787659792.238669 read 0x01c028 thru 0x01c9c8 (0x9a1 bytes) 000313 1787659792.254163 mapwrite 0x036335 thru 0x03d82a (0x74f6 bytes) 000315 1787659792.359483 mapread 0x01d502 thru 0x020975 (0x3474 bytes) 000319 1787659792.379513 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000323 1787659792.533549 read 0x026555 thru 0x02cc12 (0x66be bytes) 000324 1787659792.544847 write 0x01d51b thru 0x02817c (0xac62 bytes) 000325 1787659792.555448 trunc from 0x03d82b to 0x03882a (0x5000 bytes) 000327 1787659792.580731 mapread 0x01ee7b thru 0x020c42 (0x1dc8 bytes) 000328 1787659792.582408 mapwrite 0x027a07 thru 0x02d62c (0x5c26 bytes) 000329 1787659792.663027 read 0x01e397 thru 0x01e90b (0x575 bytes) 000330 1787659792.665687 trunc from 0x03882b to 0x01f582 (0x192a8 bytes) 000331 1787659792.683508 mapwrite 0x00892f thru 0x00c769 (0x3e3b bytes) 000333 1787659792.754640 write 0x008e56 thru 0x009295 (0x440 bytes) 000334 1787659792.755844 mapread 0x01a046 thru 0x01f582 (0x553d bytes) 000336 1787659792.760401 read 0x002966 thru 0x00a168 (0x7803 bytes) 000337 1787659792.762964 read 0x010cd3 thru 0x012a8a (0x1db8 bytes) 000340 1787659792.765918 mapread 0x01bd52 thru 0x01f582 (0x3831 bytes) 000344 1787659792.770309 write 0x01e0d0 thru 0x0288e3 (0xa814 bytes) 000349 1787659792.776226 write 0x0228bd thru 0x02cfc0 (0xa704 bytes) 000352 1787659792.780837 read 0x01c4cf thru 0x0244b9 (0x7feb bytes) 000353 1787659792.781887 write 0x028559 thru 0x02df22 (0x59ca bytes) 000358 1787659792.788806 mapread 0x019a45 thru 0x01c468 (0x2a24 bytes) 000359 1787659792.790468 write 0x03fa7e thru 0x03ffff (0x582 bytes) 000360 1787659792.792228 mapwrite 0x036b36 thru 0x03731e (0x7e9 bytes) 000362 1787659792.863543 write 0x03a41b thru 0x03cee8 (0x2ace bytes) 000364 1787659792.866799 mapwrite 0x00f507 thru 0x012d65 (0x385f bytes) 000365 1787659792.945508 mapread 0x03437f thru 0x03ffff (0xbc81 bytes) 000367 1787659792.949019 trunc from 0x040000 to 0x02bdaf (0x14250 bytes) 000368 1787659792.967837 trunc from 0x02bdb0 to 0x028552 (0x385d bytes) 000369 1787659792.999454 MIRROR_EXTEND to 2 mirrors 000371 1787659793.734411 read 0x006d0e thru 0x00bbac (0x4e9f bytes) 000372 1787659793.743983 write 0x01d758 thru 0x01e2c9 (0xb72 bytes) 000373 1787659793.856640 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x28553): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28553): Mirror 2: 0x2c3fd6f (stale) CRC-32 checksum value for chunk [0, 0x28553): Mirror 1: 0x5a1edf1b Mirror 3: 0x5a1edf1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28553: skipped 000374 1787659794.990595 read 0x024710 thru 0x028552 (0x3e43 bytes) 000377 1787659795.040821 mapread 0x0007b3 thru 0x009a25 (0x9273 bytes) 000380 1787659795.109900 mapread 0x000584 thru 0x00933b (0x8db8 bytes) 000381 1787659795.131306 read 0x00c3c7 thru 0x00ccf5 (0x92f bytes) 000384 1787659795.160211 mapread 0x016ede thru 0x01b48c (0x45af bytes) 000386 1787659795.225557 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28553): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28553): Mirror 2: 0x2c3fd6f (stale) CRC-32 checksum value for chunk [0, 0x28553): Mirror 1: 0x5a1edf1b Mirror 3: 0x5a1edf1b Mirror 4: 0x5a1edf1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28553: skipped 000387 1787659796.441285 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000389 1787659796.848492 write 0x01c339 thru 0x02653a (0xa202 bytes) 000391 1787659797.000270 write 0x01c6b5 thru 0x028eb9 (0xc805 bytes) 000392 1787659797.010876 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28eba): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28eba): Mirror 3: 0xd04aa47d (stale) Mirror 4: 0xd04aa47d (stale) CRC-32 checksum value for chunk [0, 0x28eba): Mirror 1: 0x5771c7da Mirror 5: 0x5771c7da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28eba: skipped 000396 1787659798.173527 mapwrite 0x0339a8 thru 0x03ba46 (0x809f bytes) 000399 1787659798.346360 mapread 0x037624 thru 0x03ba46 (0x4423 bytes) 000400 1787659798.348041 mapread 0x02ebf7 thru 0x038763 (0x9b6d bytes) 000403 1787659798.353587 write 0x02c43e thru 0x02dbda (0x179d bytes) 000404 1787659798.366682 read 0x02957b thru 0x02e958 (0x53de bytes) 000405 1787659798.373106 read 0x032d59 thru 0x039e58 (0x7100 bytes) 000407 1787659798.374231 trunc from 0x03ba47 to 0x0364d5 (0x5571 bytes) 000409 1787659798.401834 mapread 0x014b8c thru 0x0188f1 (0x3d66 bytes) 000411 1787659798.414645 mapread 0x014782 thru 0x020b2b (0xc3aa bytes) 000416 1787659798.423670 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x364d6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364d6): Mirror 3: 0x240ccdb8 (stale) Mirror 4: 0x240ccdb8 (stale) Mirror 5: 0xf1e9d627 (stale) CRC-32 checksum value for chunk [0, 0x364d6): Mirror 1: 0xa2b356f6 Mirror 6: 0xa2b356f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364d6: skipped 000418 1787659799.368668 write 0x0398bd thru 0x03ffff (0x6743 bytes) 000420 1787659799.455821 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x35d44ce6 (stale) Mirror 4: 0x35d44ce6 (stale) Mirror 5: 0xc91cc243 (stale) Mirror 6: 0x7fd8e643 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x93d4bcee Mirror 7: 0x93d4bcee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000421 1787659800.834605 mapread 0x019d5a thru 0x0241b8 (0xa45f bytes) 000423 1787659800.888776 trunc from 0x040000 to 0x0098b7 (0x36748 bytes) 000424 1787659801.015901 mapread 0x00350f thru 0x008281 (0x4d73 bytes) 000426 1787659801.028393 read 0x004a26 thru 0x007791 (0x2d6c bytes) 000427 1787659801.029103 mapread 0x005d0b thru 0x006ca7 (0xf9d bytes) 000428 1787659801.036168 mapread 0x001c7a thru 0x0098b7 (0x7c3e bytes) 000433 1787659801.041047 read 0x0094eb thru 0x0098b7 (0x3cd bytes) 000437 1787659801.050701 mapwrite 0x03caaf thru 0x03ffff (0x3551 bytes) 000439 1787659801.142309 write 0x03865a thru 0x03ffff (0x79a6 bytes) 000440 1787659801.143506 mapread 0x027975 thru 0x032c8f (0xb31b bytes) 000441 1787659801.146382 mapwrite 0x031068 thru 0x0318e3 (0x87c bytes) 000442 1787659801.198107 trunc from 0x040000 to 0x0012ad (0x3ed52 bytes) 000444 1787659801.223012 trunc from 0x0012ae to 0x02a2d4 (0x29027 bytes) 000445 1787659801.242678 mapread 0x0148fd thru 0x019a4a (0x514e bytes) 000447 1787659801.272691 mapread 0x016a81 thru 0x016eed (0x46d bytes) 000448 1787659801.273806 mapwrite 0x00b073 thru 0x011cc8 (0x6c56 bytes) 000451 1787659801.359182 mapread 0x004d4a thru 0x006e31 (0x20e8 bytes) 000452 1787659801.375117 write 0x009633 thru 0x00dfdc (0x49aa bytes) 000453 1787659801.385211 MIRROR_RESYNC 6 mirrors 000454 1787659803.223385 mapread 0x024b55 thru 0x02a2d4 (0x5780 bytes) 000455 1787659803.244377 mapwrite 0x0275fd thru 0x02d1ae (0x5bb2 bytes) 000457 1787659803.393420 write 0x03799c thru 0x03c026 (0x468b bytes) 000458 1787659803.394594 mapwrite 0x02c2fd thru 0x02e280 (0x1f84 bytes) 000460 1787659803.456653 read 0x01160f thru 0x021464 (0xfe56 bytes) 000461 1787659803.466217 trunc from 0x03c027 to 0x0173ef (0x24c37 bytes) 000462 1787659803.494227 mapwrite 0x0172fd thru 0x01e652 (0x7356 bytes) 000464 1787659803.605342 mapwrite 0x022e0c thru 0x027963 (0x4b58 bytes) 000466 1787659803.709734 trunc from 0x027964 to 0x023b54 (0x3e0f bytes) 000469 1787659803.732879 read 0x00b15c thru 0x0190d4 (0xdf79 bytes) 000471 1787659803.743364 mapwrite 0x002437 thru 0x00736d (0x4f37 bytes) 000475 1787659803.802494 mapread 0x01d4b3 thru 0x023b54 (0x66a2 bytes) 000476 1787659803.804748 mapread 0x0234bf thru 0x023b54 (0x696 bytes) 000479 1787659803.807614 mapwrite 0x006632 thru 0x013ad2 (0xd4a1 bytes) 000481 1787659803.900175 trunc from 0x023b55 to 0x01b27d (0x88d7 bytes) 000483 1787659803.934007 read 0x01841d thru 0x01b27d (0x2e61 bytes) 000485 1787659803.936013 read 0x00134a thru 0x008135 (0x6dec bytes) 000487 1787659803.938152 write 0x021fa2 thru 0x02c151 (0xa1b0 bytes) 000488 1787659803.941171 mapread 0x0238d1 thru 0x02c151 (0x8881 bytes) 000490 1787659803.944522 mapread 0x015e68 thru 0x024ce9 (0xee82 bytes) 000492 1787659803.960025 MIRROR_RESYNC 6 mirrors 000494 1787659806.003397 mapwrite 0x026c35 thru 0x02c3c7 (0x5793 bytes) 000495 1787659806.228177 read 0x026dd7 thru 0x02c3c7 (0x55f1 bytes) 000496 1787659806.229167 mapwrite 0x0106e9 thru 0x0132e8 (0x2c00 bytes) 000497 1787659806.317113 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c3c8): Mirror 4: 0x97e856e7 (stale) Mirror 5: 0x97e856e7 (stale) Mirror 6: 0x97e856e7 (stale) Mirror 7: 0x97e856e7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c3c8): Mirror 4: 0x97e856e7 (stale) Mirror 5: 0x97e856e7 (stale) Mirror 6: 0x97e856e7 (stale) Mirror 7: 0x97e856e7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c3c8: skipped 000498 1787659806.799427 mapwrite 0x036dec thru 0x03cfd7 (0x61ec bytes) 000499 1787659806.931330 write 0x0026dd thru 0x012481 (0xfda5 bytes) 000501 1787659806.953094 read 0x01c7cc thru 0x02579f (0x8fd4 bytes) 000504 1787659806.963865 write 0x022c47 thru 0x02fc7a (0xd034 bytes) 000505 1787659806.973499 mapread 0x00128a thru 0x010adf (0xf856 bytes) 000506 1787659806.983283 mapread 0x010e1e thru 0x01922e (0x8411 bytes) 000508 1787659807.000917 mapread 0x0311ce thru 0x03cfd7 (0xbe0a bytes) 000512 1787659807.006493 trunc from 0x03cfd8 to 0x0006e6 (0x3c8f1 bytes) 000514 1787659807.031358 write 0x01dd1e thru 0x021c88 (0x3f6b bytes) 000516 1787659807.035976 read 0x004535 thru 0x00b27a (0x6d46 bytes) 000517 1787659807.051941 write 0x00de10 thru 0x016c62 (0x8e53 bytes) 000521 1787659807.056809 write 0x007b41 thru 0x00d13e (0x55fe bytes) 000522 1787659807.068382 MIRROR_RESYNC 5 mirrors 000524 1787659809.354575 trunc from 0x021c89 to 0x01adcd (0x6ebb bytes) 000525 1787659809.527284 mapread 0x006e05 thru 0x00ac2c (0x3e28 bytes) 000526 1787659809.553127 mapwrite 0x029f5b thru 0x037564 (0xd60a bytes) 000528 1787659809.732674 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37565): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37565): Mirror 4: 0xc9ff621e (stale) Mirror 5: 0xc9ff621e (stale) Mirror 6: 0xc9ff621e (stale) Mirror 7: 0xc9ff621e (stale) CRC-32 checksum value for chunk [0, 0x37565): Mirror 1: 0x5f7dc603 Mirror 8: 0x5f7dc603 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37565: skipped 000529 1787659811.226652 mapread 0x001ecc thru 0x00b933 (0x9a68 bytes) 000530 1787659811.276967 trunc from 0x037565 to 0x02199f (0x15bc5 bytes) 000531 1787659811.402382 MIRROR_RESYNC 6 mirrors 000534 1787659813.027653 mapwrite 0x03de78 thru 0x03ffff (0x2188 bytes) 000535 1787659813.250090 write 0x003cdd thru 0x00b247 (0x756b bytes) 000538 1787659813.262682 trunc from 0x040000 to 0x0247af (0x1b850 bytes) 000540 1787659813.287700 trunc from 0x0247b0 to 0x015fce (0xe7e1 bytes) 000541 1787659813.308613 read 0x00071b thru 0x007b51 (0x7437 bytes) 000544 1787659813.319272 trunc from 0x015fcf to 0x014d4b (0x1283 bytes) 000546 1787659813.340992 trunc from 0x014d4c to 0x02513c (0x103f1 bytes) 000547 1787659813.364896 write 0x003c48 thru 0x00ad54 (0x710d bytes) 000548 1787659813.367579 mapread 0x015fcd thru 0x016cdc (0xd10 bytes) 000550 1787659813.391038 trunc from 0x02513d to 0x037ef1 (0x12db5 bytes) 000551 1787659813.414741 mapwrite 0x014267 thru 0x01e3df (0xa179 bytes) 000552 1787659813.606575 read 0x02d378 thru 0x037ef1 (0xab7a bytes) 000560 1787659813.627984 mapwrite 0x032a53 thru 0x035a44 (0x2ff2 bytes) 000561 1787659813.693805 read 0x02ac53 thru 0x036bf5 (0xbfa3 bytes) 000562 1787659813.696854 read 0x01bd2b thru 0x025c48 (0x9f1e bytes) 000563 1787659813.698702 mapwrite 0x011278 thru 0x01a872 (0x95fb bytes) 000566 1787659813.843114 MIRROR_RESYNC 6 mirrors truncating to largest ever: 0x3d1b9 000567 1787659815.534632 trunc from 0x037ef2 to 0x03d1b8 (0x52c7 bytes) 000568 1787659815.609158 MIRROR_RESYNC 6 mirrors 000569 1787659817.761213 mapwrite 0x015446 thru 0x01713f (0x1cfa bytes) 000571 1787659818.014963 MIRROR_RESYNC 6 mirrors 000572 1787659820.235929 mapwrite 0x020125 thru 0x023cb0 (0x3b8c bytes) 000573 1787659820.410547 trunc from 0x03d1b9 to 0x019941 (0x23877 bytes) 000580 1787659820.448473 trunc from 0x019942 to 0x01a00d (0x6cc bytes) 000581 1787659820.469925 mapread 0x00d969 thru 0x0114a4 (0x3b3c bytes) 000583 1787659820.480271 write 0x0227c7 thru 0x027e1f (0x5659 bytes) 000584 1787659820.481515 trunc from 0x027e20 to 0x008808 (0x1f617 bytes) 000585 1787659820.498622 write 0x012bd5 thru 0x0142af (0x16db bytes) 000588 1787659820.502677 mapread 0x01233c thru 0x0142af (0x1f74 bytes) 000589 1787659820.504193 mapwrite 0x005d00 thru 0x00a3c3 (0x46c4 bytes) 000590 1787659820.571457 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x142b0): Mirror 4: 0xb339adc1 (stale) Mirror 5: 0xb339adc1 (stale) Mirror 7: 0xb339adc1 (stale) Mirror 8: 0xb339adc1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x142b0): Mirror 4: 0xb339adc1 (stale) Mirror 5: 0xb339adc1 (stale) Mirror 7: 0xb339adc1 (stale) Mirror 8: 0xb339adc1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x142b0: skipped 000591 1787659820.953117 trunc from 0x0142b0 to 0x0349b6 (0x20707 bytes) 000592 1787659820.973560 write 0x02716b thru 0x031efb (0xad91 bytes) 000593 1787659820.984179 write 0x037a0f thru 0x03ffff (0x85f1 bytes) 000596 1787659820.987244 read 0x02ac00 thru 0x0305fe (0x59ff bytes) 000599 1787659820.989491 mapwrite 0x01dff7 thru 0x0215c9 (0x35d3 bytes) 000601 1787659821.062829 read 0x02e8bf thru 0x039fdd (0xb71f bytes) 000602 1787659821.072985 write 0x011fac thru 0x0161cc (0x4221 bytes) 000603 1787659821.074181 trunc from 0x040000 to 0x00a511 (0x35aee bytes) 000604 1787659821.098223 trunc from 0x00a512 to 0x020c51 (0x16740 bytes) 000607 1787659821.122396 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x20c52): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20c52): Mirror 4: 0x812bc0ef (stale) Mirror 5: 0x812bc0ef (stale) Mirror 7: 0x812bc0ef (stale) Mirror 8: 0x812bc0ef (stale) CRC-32 checksum value for chunk [0, 0x20c52): Mirror 1: 0xa49d0fd2 Mirror 9: 0xa49d0fd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20c52: skipped 000608 1787659822.625291 mapread 0x018d06 thru 0x01ef9f (0x629a bytes) 000609 1787659822.697712 trunc from 0x020c52 to 0x0160d9 (0xab78 bytes) 000610 1787659822.870747 mapwrite 0x01e8ed thru 0x025689 (0x6d9d bytes) 000611 1787659822.988449 mapwrite 0x0325b4 thru 0x03a01c (0x7a69 bytes) 000612 1787659823.069662 mapwrite 0x009075 thru 0x00b9b2 (0x293e bytes) 000613 1787659823.116143 trunc from 0x03a01d to 0x012efc (0x27120 bytes) 000616 1787659823.132469 mapread 0x004aaa thru 0x004ae3 (0x3a bytes) 000618 1787659823.134476 write 0x020dca thru 0x02abe0 (0x9e17 bytes) 000619 1787659823.135946 mapwrite 0x0038a6 thru 0x01088d (0xcfe8 bytes) 000620 1787659823.205970 trunc from 0x02abe1 to 0x0186e7 (0x124f9 bytes) 000622 1787659823.229582 write 0x02630d thru 0x02d662 (0x7356 bytes) truncating to largest ever: 0x3fb09 000625 1787659823.233898 trunc from 0x02d663 to 0x03fb08 (0x124a6 bytes) 000626 1787659823.252900 write 0x0255a4 thru 0x0271e4 (0x1c41 bytes) 000627 1787659823.261041 trunc from 0x03fb09 to 0x021af9 (0x1e00f bytes) 000628 1787659823.281701 write 0x0021a5 thru 0x00cb22 (0xa97e bytes) 000629 1787659823.283638 trunc from 0x021afa to 0x01b513 (0x65e6 bytes) 000630 1787659823.306545 mapread 0x01324f thru 0x017eec (0x4c9e bytes) 000631 1787659823.320900 read 0x00dbf7 thru 0x01079d (0x2ba7 bytes) 000632 1787659823.327574 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1b514): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b514): Mirror 4: 0xaea7200b (stale) Mirror 5: 0xaea7200b (stale) Mirror 7: 0xaea7200b (stale) Mirror 8: 0xaea7200b (stale) Mirror 9: 0x7e78f600 (stale) CRC-32 checksum value for chunk [0, 0x1b514): Mirror 1: 0x6641561a Mirror 10: 0x6641561a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b514: skipped 000633 1787659824.449981 mapread 0x010b39 thru 0x01b123 (0xa5eb bytes) 000634 1787659824.499046 mapread 0x01a9bb thru 0x01b513 (0xb59 bytes) 000635 1787659824.513513 trunc from 0x01b514 to 0x02b768 (0x10255 bytes) 000637 1787659824.602503 write 0x03fd0a thru 0x03ffff (0x2f6 bytes) 000638 1787659824.610183 MIRROR_RESYNC 7 mirrors 000643 1787659826.622962 MIRROR_EXTEND to 8 mirrors 000646 1787659827.310161 mapread 0x019b9b thru 0x01ec42 (0x50a8 bytes) 000647 1787659827.335722 mapread 0x00acd3 thru 0x01768a (0xc9b8 bytes) 000648 1787659827.356802 read 0x001d49 thru 0x00ceb3 (0xb16b bytes) 000649 1787659827.362024 read 0x017bbc thru 0x021066 (0x94ab bytes) 000650 1787659827.362510 trunc from 0x040000 to 0x02a824 (0x157db bytes) 000651 1787659827.480334 trunc from 0x02a825 to 0x002bf2 (0x27c32 bytes) 000653 1787659827.508297 trunc from 0x002bf3 to 0x0179e2 (0x14df0 bytes) 000655 1787659827.528729 mapread 0x011685 thru 0x01268d (0x1009 bytes) 000657 1787659827.540470 mapwrite 0x01aac3 thru 0x0278c0 (0xcdfe bytes) 000658 1787659827.627351 mapread 0x011a6c thru 0x0192f0 (0x7885 bytes) 000659 1787659827.628563 mapwrite 0x02fef9 thru 0x03167e (0x1786 bytes) 000662 1787659827.710384 read 0x004f92 thru 0x014a94 (0xfb03 bytes) 000664 1787659827.711677 write 0x0086b4 thru 0x00b7ba (0x3107 bytes) 000665 1787659827.714460 read 0x01f09c thru 0x026088 (0x6fed bytes) 000666 1787659827.715345 read 0x023d41 thru 0x031526 (0xd7e6 bytes) 000668 1787659827.717559 mapwrite 0x014632 thru 0x021461 (0xce30 bytes) 000671 1787659827.772748 mapwrite 0x03c2a7 thru 0x03ffff (0x3d59 bytes) 000674 1787659827.875763 write 0x0045db thru 0x00e688 (0xa0ae bytes) 000675 1787659827.878803 mapread 0x014620 thru 0x0149be (0x39f bytes) 000679 1787659827.883183 read 0x010427 thru 0x0192cd (0x8ea7 bytes) 000682 1787659827.885463 trunc from 0x040000 to 0x00700d (0x38ff2 bytes) 000684 1787659827.909663 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x700e): Mirror 4: 0x9510ace6 (stale) Mirror 5: 0x9510ace6 (stale) Mirror 7: 0x9510ace6 (stale) Mirror 8: 0x9510ace6 (stale) Mirror 9: 0x9510ace6 (stale) Mirror 11: 0x9510ace6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x700e): Mirror 4: 0x9510ace6 (stale) Mirror 5: 0x9510ace6 (stale) Mirror 7: 0x9510ace6 (stale) Mirror 8: 0x9510ace6 (stale) Mirror 9: 0x9510ace6 (stale) Mirror 11: 0x9510ace6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x700e: skipped 000685 1787659828.320231 read 0x00200d thru 0x00700d (0x5001 bytes) 000686 1787659828.331479 read 0x004d1e thru 0x00700d (0x22f0 bytes) 000687 1787659828.338599 MIRROR_RESYNC 7 mirrors 000689 1787659830.341831 MIRROR_EXTEND to 8 mirrors 000690 1787659831.036288 mapread 0x003aa5 thru 0x00700d (0x3569 bytes) 000691 1787659831.054507 trunc from 0x00700e to 0x0270fc (0x200ef bytes) 000695 1787659831.164857 mapread 0x00dff9 thru 0x01d1de (0xf1e6 bytes) 000697 1787659831.184223 mapwrite 0x012436 thru 0x01cfb2 (0xab7d bytes) 000699 1787659831.240887 write 0x01fde6 thru 0x02b6e1 (0xb8fc bytes) 000701 1787659831.247713 read 0x027603 thru 0x0277ec (0x1ea bytes) 000702 1787659831.248626 mapread 0x00ef73 thru 0x013093 (0x4121 bytes) 000703 1787659831.249998 trunc from 0x02b6e2 to 0x0046d2 (0x2700f bytes) 000704 1787659831.276572 trunc from 0x0046d3 to 0x01b226 (0x16b54 bytes) 000705 1787659831.294194 write 0x022cc1 thru 0x0281e4 (0x5524 bytes) 000707 1787659831.298159 read 0x01d3e1 thru 0x023a47 (0x6667 bytes) 000709 1787659831.312693 read 0x016d1f thru 0x022c99 (0xbf7b bytes) 000711 1787659831.314844 write 0x036abf thru 0x03cb80 (0x60c2 bytes) 000712 1787659831.316781 read 0x03b527 thru 0x03cb80 (0x165a bytes) 000713 1787659831.317785 read 0x02997f thru 0x02e87d (0x4eff bytes) 000714 1787659831.325715 mapwrite 0x01b519 thru 0x01e307 (0x2def bytes) 000716 1787659831.390470 write 0x014491 thru 0x020d35 (0xc8a5 bytes) 000718 1787659831.392373 mapwrite 0x0065de thru 0x00bcfc (0x571f bytes) 000721 1787659831.456322 mapread 0x0160e3 thru 0x02425f (0xe17d bytes) 000725 1787659831.458715 trunc from 0x03cb81 to 0x01b433 (0x2174d bytes) 000726 1787659831.480646 write 0x007de9 thru 0x0139b2 (0xbbca bytes) 000728 1787659831.484603 trunc from 0x01b434 to 0x03cf5b (0x21b28 bytes) 000732 1787659831.511330 trunc from 0x03cf5c to 0x006c37 (0x36324 bytes) 000733 1787659831.529100 mapread 0x00422e thru 0x005a65 (0x1838 bytes) 000734 1787659831.530492 mapwrite 0x003d23 thru 0x00ea94 (0xad72 bytes) 000735 1787659831.781901 mapwrite 0x002500 thru 0x004bf6 (0x26f7 bytes) 000737 1787659831.893031 mapread 0x005a6e thru 0x00e1a3 (0x8736 bytes) 000739 1787659831.912532 write 0x00c556 thru 0x01acb2 (0xe75d bytes) 000740 1787659831.914191 write 0x00a973 thru 0x015262 (0xa8f0 bytes) 000741 1787659831.930855 read 0x00beca thru 0x01561b (0x9752 bytes) 000742 1787659831.935941 mapread 0x006a9e thru 0x00bb53 (0x50b6 bytes) 000743 1787659831.948700 mapwrite 0x016753 thru 0x023c49 (0xd4f7 bytes) 000744 1787659832.148997 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23c4a): Mirror 5: 0xea708ea1 (stale) Mirror 7: 0xea708ea1 (stale) Mirror 8: 0xea708ea1 (stale) Mirror 9: 0xea708ea1 (stale) Mirror 11: 0xea708ea1 (stale) Mirror 12: 0xea708ea1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23c4a): Mirror 5: 0xea708ea1 (stale) Mirror 7: 0xea708ea1 (stale) Mirror 8: 0xea708ea1 (stale) Mirror 9: 0xea708ea1 (stale) Mirror 11: 0xea708ea1 (stale) Mirror 12: 0xea708ea1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23c4a: skipped 000745 1787659832.682917 MIRROR_RESYNC 7 mirrors 000746 1787659834.215614 read 0x011a0d thru 0x014bb9 (0x31ad bytes) 000749 1787659834.222235 read 0x0055f6 thru 0x0130f4 (0xdaff bytes) 000750 1787659834.232448 mapwrite 0x036e60 thru 0x03c6b3 (0x5854 bytes) 000755 1787659834.376130 trunc from 0x03c6b4 to 0x026736 (0x15f7d bytes) 000756 1787659834.404318 write 0x00741a thru 0x00dc7d (0x6864 bytes) 000757 1787659834.417511 trunc from 0x026737 to 0x00a9cf (0x1bd67 bytes) 000758 1787659834.439209 mapread 0x0063a9 thru 0x00a9cf (0x4627 bytes) 000762 1787659834.448231 write 0x011ff4 thru 0x014774 (0x2781 bytes) 000763 1787659834.449446 mapwrite 0x021282 thru 0x030554 (0xf2d3 bytes) 000765 1787659834.717917 MIRROR_RESYNC 7 mirrors 000766 1787659837.760456 mapread 0x01535d thru 0x01fa01 (0xa6a5 bytes) 000771 1787659837.782790 mapread 0x00d726 thru 0x014013 (0x68ee bytes) 000772 1787659837.800824 trunc from 0x030555 to 0x02c4a6 (0x40ae bytes) 000774 1787659837.875428 write 0x028514 thru 0x030109 (0x7bf6 bytes) 000777 1787659837.906436 mapread 0x02b35f thru 0x030109 (0x4dab bytes) 000778 1787659837.913640 MIRROR_RESYNC 7 mirrors 000780 1787659840.613211 trunc from 0x03010a to 0x01bb71 (0x14598 bytes) 000781 1787659840.829032 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bb72): Mirror 5: 0x3ee35c50 (stale) Mirror 7: 0x3ee35c50 (stale) Mirror 8: 0x3ee35c50 (stale) Mirror 9: 0x3ee35c50 (stale) Mirror 12: 0x3ee35c50 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bb72): Mirror 5: 0x3ee35c50 (stale) Mirror 7: 0x3ee35c50 (stale) Mirror 8: 0x3ee35c50 (stale) Mirror 9: 0x3ee35c50 (stale) Mirror 12: 0x3ee35c50 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bb72: skipped 000782 1787659841.371579 trunc from 0x01bb72 to 0x038556 (0x1c9e5 bytes) 000783 1787659841.401452 read 0x008faf thru 0x01269a (0x96ec bytes) 000784 1787659841.413913 write 0x0025f5 thru 0x0031b4 (0xbc0 bytes) 000785 1787659841.435865 trunc from 0x038557 to 0x00914e (0x2f408 bytes) 000786 1787659841.467231 mapread 0x005485 thru 0x00914e (0x3cca bytes) 000788 1787659841.495391 write 0x0229a2 thru 0x02b9b9 (0x9018 bytes) 000789 1787659841.498341 read 0x00f8d1 thru 0x01dfc8 (0xe6f8 bytes) 000790 1787659841.509669 write 0x004b70 thru 0x00ceeb (0x837c bytes) 000791 1787659841.512022 write 0x01348e thru 0x01e04f (0xabc2 bytes) 000793 1787659841.524544 mapwrite 0x02637a thru 0x032b9f (0xc826 bytes) 000794 1787659841.701315 mapread 0x00a87c thru 0x00b905 (0x108a bytes) 000795 1787659841.702511 mapread 0x0072de thru 0x00e5fd (0x7320 bytes) 000798 1787659841.708547 trunc from 0x032ba0 to 0x033bcf (0x1030 bytes) 000800 1787659841.739682 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33bd0): Mirror 5: 0x5fd0179f (stale) Mirror 7: 0x5fd0179f (stale) Mirror 9: 0x5fd0179f (stale) Mirror 12: 0x5fd0179f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33bd0): Mirror 5: 0x5fd0179f (stale) Mirror 7: 0x5fd0179f (stale) Mirror 9: 0x5fd0179f (stale) Mirror 12: 0x5fd0179f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33bd0: skipped 000801 1787659842.123435 mapwrite 0x01f94f thru 0x02c68d (0xcd3f bytes) 000804 1787659842.270493 trunc from 0x033bd0 to 0x02ceac (0x6d23 bytes) 000805 1787659842.301950 mapread 0x00e8b6 thru 0x016217 (0x7962 bytes) 000806 1787659842.321574 mapread 0x0140e8 thru 0x014925 (0x83e bytes) 000807 1787659842.323949 mapread 0x01777e thru 0x022c08 (0xb48b bytes) 000810 1787659842.343565 write 0x0101aa thru 0x0132a7 (0x30fe bytes) 000811 1787659842.357244 write 0x00c470 thru 0x011886 (0x5417 bytes) 000812 1787659842.377258 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2cead): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cead): Mirror 5: 0xdb8a39f8 (stale) Mirror 7: 0xdb8a39f8 (stale) Mirror 9: 0xdb8a39f8 (stale) Mirror 12: 0xdb8a39f8 (stale) CRC-32 checksum value for chunk [0, 0x2cead): Mirror 1: 0x716a0ac3 Mirror 13: 0x716a0ac3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cead: skipped 000813 1787659843.391534 mapread 0x01b8e5 thru 0x01ffef (0x470b bytes) 000816 1787659843.449822 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cead): Mirror 5: 0xdb8a39f8 (stale) Mirror 7: 0xdb8a39f8 (stale) Mirror 12: 0xdb8a39f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cead): Mirror 5: 0xdb8a39f8 (stale) Mirror 7: 0xdb8a39f8 (stale) Mirror 12: 0xdb8a39f8 (stale) CRC-32 checksum value for chunk [0, 0x2cead): Mirror 1: 0x716a0ac3 Mirror 13: 0x716a0ac3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cead: skipped 000817 1787659844.153016 mapread 0x025ff5 thru 0x02ceac (0x6eb8 bytes) 000818 1787659844.199049 write 0x01798e thru 0x020f6a (0x95dd bytes) 000820 1787659844.283324 write 0x0079d6 thru 0x00a5a4 (0x2bcf bytes) 000822 1787659844.298253 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cead): Mirror 5: 0xdb8a39f8 (stale) Mirror 12: 0xdb8a39f8 (stale) Mirror 13: 0x716a0ac3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cead): Mirror 5: 0xdb8a39f8 (stale) Mirror 12: 0xdb8a39f8 (stale) Mirror 13: 0x716a0ac3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cead: skipped 000824 1787659844.652318 trunc from 0x02cead to 0x0170fc (0x15db0 bytes) 000825 1787659844.686401 mapread 0x0108ec thru 0x0170fc (0x6811 bytes) 000829 1787659844.710186 mapwrite 0x0333b3 thru 0x039dc5 (0x6a13 bytes) 000831 1787659844.810324 write 0x009d43 thru 0x011be7 (0x7ea5 bytes) 000832 1787659844.818577 trunc from 0x039dc6 to 0x0170de (0x22ce7 bytes) 000834 1787659844.842589 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x170df): Mirror 5: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x170df): Mirror 5: 0x6f7813ca (stale) Mirror 12: 0x6f7813ca (stale) Mirror 13: 0xa5ddb28a (stale) CRC-32 checksum value for chunk [0, 0x170df): Mirror 1: 0xbbf26a36 Mirror 14: 0xbbf26a36 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x170df: skipped 000835 1787659846.664280 mapread 0x01002f thru 0x0170de (0x70b0 bytes) 000836 1787659846.741667 read 0x002151 thru 0x00756d (0x541d bytes) 000838 1787659846.768657 mapwrite 0x00e482 thru 0x01bcb3 (0xd832 bytes) 000839 1787659846.949798 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bcb4): Mirror 5: 0xc4032540 (stale) Mirror 12: 0xc4032540 (stale) Mirror 14: 0xcd29cd9a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bcb4): Mirror 5: 0xc4032540 (stale) Mirror 12: 0xc4032540 (stale) Mirror 14: 0xcd29cd9a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bcb4: skipped 000840 1787659847.335980 mapwrite 0x011c0d thru 0x0142d5 (0x26c9 bytes) 000841 1787659847.404731 trunc from 0x01bcb4 to 0x0098df (0x123d4 bytes) 000844 1787659847.421989 write 0x02a423 thru 0x030fff (0x6bdd bytes) 000846 1787659847.431764 trunc from 0x031000 to 0x013e55 (0x1d1aa bytes) 000847 1787659847.450449 read 0x001940 thru 0x00acb3 (0x9374 bytes) 000849 1787659847.460816 mapwrite 0x03d442 thru 0x03ffff (0x2bbe bytes) 000850 1787659847.549587 mapread 0x01f3b1 thru 0x0241a8 (0x4df8 bytes) 000851 1787659847.561352 write 0x028394 thru 0x02ee20 (0x6a8d bytes) 000853 1787659847.571694 mapwrite 0x02d838 thru 0x033861 (0x602a bytes) 000854 1787659847.625715 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x45286b14 (stale) Mirror 12: 0x45286b14 (stale) Mirror 14: 0x7a95d2e7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc152df07 Mirror 15: 0xc152df07 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000855 1787659849.149864 trunc from 0x040000 to 0x0276b7 (0x18948 bytes) 000857 1787659849.284422 trunc from 0x0276b8 to 0x031d8c (0xa6d5 bytes) 000858 1787659849.320716 MIRROR_RESYNC 5 mirrors 000859 1787659851.568731 write 0x00d5d6 thru 0x00f54c (0x1f77 bytes) 000862 1787659851.635283 mapread 0x006ad9 thru 0x008bec (0x2114 bytes) 000863 1787659851.653606 read 0x001637 thru 0x003052 (0x1a1c bytes) 000864 1787659851.654528 mapread 0x006b9c thru 0x0080d8 (0x153d bytes) 000867 1787659851.658003 trunc from 0x031d8d to 0x01a327 (0x17a65 bytes) 000868 1787659851.679701 mapwrite 0x0101f6 thru 0x01c22e (0xc039 bytes) 000869 1787659851.811139 write 0x02ed35 thru 0x03c91b (0xdbe7 bytes) 000870 1787659851.814316 write 0x009ab4 thru 0x016139 (0xc686 bytes) 000871 1787659851.817935 mapread 0x01557f thru 0x01e55f (0x8fe1 bytes) 000873 1787659851.830929 mapwrite 0x024624 thru 0x029842 (0x521f bytes) 000875 1787659851.884941 mapwrite 0x038270 thru 0x03ffff (0x7d90 bytes) 000877 1787659851.997026 read 0x025af6 thru 0x02fae7 (0x9ff2 bytes) 000879 1787659852.005461 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x1871e5d3 (stale) Mirror 14: 0x1871e5d3 (stale) Mirror 15: 0x1871e5d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x1871e5d3 (stale) Mirror 14: 0x1871e5d3 (stale) Mirror 15: 0x1871e5d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000880 1787659852.526994 mapread 0x018513 thru 0x025e13 (0xd901 bytes) 000884 1787659852.584377 mapwrite 0x03dd1d thru 0x03ffff (0x22e3 bytes) 000888 1787659852.691028 read 0x001933 thru 0x011483 (0xfb51 bytes) 000889 1787659852.706471 trunc from 0x040000 to 0x035645 (0xa9ba bytes) 000890 1787659852.724914 read 0x004b50 thru 0x005e42 (0x12f3 bytes) 000891 1787659852.725735 mapwrite 0x00caa1 thru 0x014ad9 (0x8039 bytes) 000895 1787659852.837306 read 0x0122d8 thru 0x021d6b (0xfa94 bytes) 000896 1787659852.841470 trunc from 0x035646 to 0x03b30c (0x5cc7 bytes) 000898 1787659852.863812 write 0x01db18 thru 0x028cce (0xb1b7 bytes) 000899 1787659852.866769 write 0x01589b thru 0x021f9f (0xc705 bytes) 000901 1787659852.871304 mapwrite 0x003692 thru 0x00e7e5 (0xb154 bytes) 000902 1787659853.028260 write 0x038b86 thru 0x03d6b5 (0x4b30 bytes) 000905 1787659853.055180 mapwrite 0x000975 thru 0x00e164 (0xd7f0 bytes) 000906 1787659853.119516 trunc from 0x03d6b6 to 0x01fc90 (0x1da25 bytes) 000908 1787659853.141887 mapwrite 0x03bdc8 thru 0x03ffff (0x4238 bytes) 000909 1787659853.302444 trunc from 0x040000 to 0x001e23 (0x3e1dc bytes) 000910 1787659853.331963 trunc from 0x001e24 to 0x0192e2 (0x174bf bytes) 000911 1787659853.381985 read 0x0004fc thru 0x0045b7 (0x40bc bytes) 000913 1787659853.396964 mapwrite 0x006aa4 thru 0x012714 (0xbc71 bytes) 000914 1787659853.608309 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x192e3): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x192e3): Mirror 12: 0x4812dd82 (stale) Mirror 14: 0x4812dd82 (stale) Mirror 15: 0x4812dd82 (stale) CRC-32 checksum value for chunk [0, 0x192e3): Mirror 1: 0xf9418d41 Mirror 16: 0xf9418d41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x192e3: skipped 000916 1787659855.152504 MIRROR_RESYNC 5 mirrors 000917 1787659856.477891 trunc from 0x0192e3 to 0x02fc5c (0x1697a bytes) 000919 1787659856.628671 write 0x009c13 thru 0x0118f3 (0x7ce1 bytes) 000920 1787659856.641704 write 0x02e18f thru 0x036dff (0x8c71 bytes) 000921 1787659856.649735 read 0x026ef4 thru 0x02a3ca (0x34d7 bytes) 000922 1787659856.658616 mapread 0x02e2be thru 0x032b9b (0x48de bytes) 000924 1787659856.661975 mapwrite 0x010876 thru 0x01f0e3 (0xe86e bytes) 000925 1787659856.764280 trunc from 0x036e00 to 0x03f13e (0x833f bytes) 000928 1787659856.793716 mapread 0x0393e1 thru 0x03f13e (0x5d5e bytes) 000929 1787659856.811117 MIRROR_RESYNC 5 mirrors 000931 1787659858.739800 mapread 0x01847a thru 0x01b6e0 (0x3267 bytes) 000934 1787659858.764480 read 0x03151e thru 0x03da1e (0xc501 bytes) 000937 1787659858.774788 trunc from 0x03f13f to 0x02c9de (0x12760 bytes) 000938 1787659858.842830 mapread 0x02282c thru 0x024b0a (0x22df bytes) 000939 1787659858.853234 mapread 0x00ba54 thru 0x01a37b (0xe928 bytes) 000941 1787659858.871784 read 0x01b835 thru 0x01ec90 (0x345c bytes) 000942 1787659858.872587 trunc from 0x02c9df to 0x00c751 (0x2028d bytes) 000946 1787659858.903475 mapwrite 0x004f0b thru 0x013c29 (0xed1f bytes) 000949 1787659859.023629 write 0x01860f thru 0x01e0db (0x5acd bytes) 000950 1787659859.026398 write 0x00270f thru 0x005ec6 (0x37b8 bytes) 000952 1787659859.030055 trunc from 0x01e0dc to 0x03bbaf (0x1dad4 bytes) 000955 1787659859.063869 mapwrite 0x02ea04 thru 0x033895 (0x4e92 bytes) 000957 1787659859.144806 read 0x035742 thru 0x03bbaf (0x646e bytes) 000961 1787659859.150090 mapwrite 0x03be89 thru 0x03cc21 (0xd99 bytes) 000962 1787659859.237644 write 0x00c68d thru 0x0185f0 (0xbf64 bytes) 000963 1787659859.241245 mapread 0x015da0 thru 0x01db56 (0x7db7 bytes) 000964 1787659859.251665 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3cc22): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc22): Mirror 12: 0xaf1f7f96 (stale) Mirror 14: 0xaf1f7f96 (stale) Mirror 15: 0xaf1f7f96 (stale) Mirror 16: 0xaf1f7f96 (stale) CRC-32 checksum value for chunk [0, 0x3cc22): Mirror 1: 0x3a41994e Mirror 17: 0x3a41994e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc22: skipped 000967 1787659860.760559 trunc from 0x03cc22 to 0x02f398 (0xd889 bytes) 000969 1787659860.943007 MIRROR_RESYNC 6 mirrors 000970 1787659862.759174 mapwrite 0x015f83 thru 0x02568c (0xf70a bytes) 000973 1787659862.999313 write 0x014cbe thru 0x01b129 (0x646c bytes) 000975 1787659863.005902 mapread 0x006033 thru 0x00af0a (0x4ed8 bytes) 000976 1787659863.017821 trunc from 0x02f399 to 0x02a521 (0x4e77 bytes) 000977 1787659863.048111 mapwrite 0x02c249 thru 0x0302aa (0x4062 bytes) 000978 1787659863.186599 read 0x005fc2 thru 0x01453d (0xe57c bytes) 000979 1787659863.192274 trunc from 0x0302ab to 0x00651f (0x29d8b bytes) 000980 1787659863.228892 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6520): Mirror 12: 0xe4dd37ee (stale) Mirror 14: 0xe4dd37ee (stale) Mirror 15: 0xe4dd37ee (stale) Mirror 17: 0xe4dd37ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6520): Mirror 12: 0xe4dd37ee (stale) Mirror 14: 0xe4dd37ee (stale) Mirror 15: 0xe4dd37ee (stale) Mirror 17: 0xe4dd37ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6520: skipped 000981 1787659863.678310 mapread 0x006299 thru 0x00651f (0x287 bytes) 000983 1787659863.689232 write 0x035894 thru 0x03f186 (0x98f3 bytes) 000984 1787659863.699443 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f187): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f187): Mirror 12: 0x668a730b (stale) Mirror 14: 0x668a730b (stale) Mirror 15: 0x668a730b (stale) Mirror 17: 0x668a730b (stale) CRC-32 checksum value for chunk [0, 0x3f187): Mirror 1: 0x2966ee64 Mirror 18: 0x2966ee64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f187: skipped 000985 1787659864.954519 mapwrite 0x027a9a thru 0x02cd5f (0x52c6 bytes) 000989 1787659865.181884 mapread 0x00befa thru 0x019498 (0xd59f bytes) 000990 1787659865.203369 trunc from 0x03f187 to 0x007823 (0x37963 bytes) 000991 1787659865.235425 trunc from 0x007824 to 0x034f04 (0x2d6e1 bytes) 000992 1787659865.258205 write 0x022544 thru 0x022f5c (0xa19 bytes) 000993 1787659865.267024 write 0x032d1c thru 0x037301 (0x45e6 bytes) 000994 1787659865.290197 read 0x02eeba thru 0x0316bd (0x2804 bytes) 000995 1787659865.299558 trunc from 0x037302 to 0x027923 (0xf9de bytes) 000998 1787659865.319385 write 0x003dba thru 0x00db57 (0x9d9e bytes) 000999 1787659865.334272 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x27924): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27924): Mirror 12: 0xe9400d26 (stale) Mirror 14: 0xe9400d26 (stale) Mirror 15: 0xe9400d26 (stale) Mirror 17: 0xe9400d26 (stale) Mirror 18: 0x8320d04d (stale) CRC-32 checksum value for chunk [0, 0x27924): Mirror 1: 0xc84e72ab Mirror 19: 0xc84e72ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27924: skipped All operations completed A-OK! PASS 70a (98s) == sanity-flr test 71: check mirror extend parent fid ==== 08:11:12 (1787659872) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 08:11:17 (1787659877) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224813 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189541 s, 5.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 08:11:28 (1787659888) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre2 Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5527840 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4334304 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 4443392 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6903232 .. Extending file size to 8842144 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8755168 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9017408 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57370 57371 57373 57374 57377 done 192.168.202.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre2 (opts:) 192.168.202.107@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:12:56 (1787659976) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre2 Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8924416 .. Extending file size to 4743840 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6629248 .. Extending file size to 6943104 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9104800 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9291296 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58840 58841 58842 58844 58848 done 192.168.202.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre2 (opts:) 192.168.202.107@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:14:18 (1787660058) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149289 s, 14.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.629342 s, 3.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xad57e5a1 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:14:43 (1787660083) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134122 s, 15.6 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 08:14:53 (1787660093) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f4:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:15:03 (1787660103) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.254432 s, 8.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:15:13 (1787660113) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.218179 s, 9.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:15:24 (1787660124) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.221268 s, 9.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:15:59 (1787660159) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg207-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115009 s, 9.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:17:02 (1787660222) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg207-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.389265 s, 5.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (62s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:18:04 (1787660284) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg207-server mds-ost sync done. Creating new pool test_204f oleg207-server: Pool lustre.test_204f created Adding targets to pool oleg207-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg207-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.500892 s, 4.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg207-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg207-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg207-server: Pool lustre.test_204f destroyed PASS 204f (87s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:19:31 (1787660371) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:19:40 (1787660380) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:19:47 (1787660387) oleg207-server: Pool lustre.test_206 created Adding targets to pool oleg207-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg207-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg207-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg207-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg207-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:20:15 (1787660415) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10835 s, 9.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839602 s, 12.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x9e1e766b Mirror 3: 0x9e1e766b PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:20:28 (1787660428) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:20:32 (1787660432) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:20:37 (1787660437) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.136014 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x11e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x11e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x11e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x23e0000 Read this (4 bytes): BBBB 192.168.202.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:21:15 (1787660475) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.021362 s, 383 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:21:23 (1787660483) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412224 s, 25.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x219 lmm_fid: [0x200000404:0x219:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x219 lmm_fid: [0x200000404:0x219:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834457 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-25 08:21:25.000000000 -0400 Modify: 2026-08-25 08:21:25.000000000 -0400 Change: 2026-08-25 08:21:26.000000000 -0400 Birth: 2026-08-25 08:21:25.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:21:31 (1787660491) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:21:35 (1787660495) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.607564 s, 67.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0487282 s, 84.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 25 08:21 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 25 08:21 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:21:43 (1787660503) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:21:50 (1787660510) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1913 sec =========== 08:21:58 (1787660518) === sanity-flr: start cleanup 08:22:00 (1787660520) === === sanity-flr: finish cleanup 08:22:07 (1787660527) ===