-----============= acceptance-small: sanity-flr ============----- Tue Sep 1 19:29:50 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg427-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:30:06 (1788305406) === oleg427-client.virtnet: executing check_config_client /mnt/lustre oleg427-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg427-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97b546470800.idle_timeout=debug osc.lustre-OST0001-osc-ffff97b546470800.idle_timeout=debug disable quota as required oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:30:33 (1788305433) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7606] == sanity-flr test 0a: lfs mirror create with -N option == 19:30:35 (1788305435) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:30:55 (1788305455) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:30:58 (1788305458) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:31:01 (1788305461) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:31:23 (1788305483) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:31:26 (1788305486) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 19:31:30 (1788305490) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff97b546470800.stats=clear osc.lustre-OST0001-osc-ffff97b546470800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:31:37 (1788305497) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:31:48 (1788305508) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:31:55 (1788305515) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:31:58 (1788305518) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:32:04 (1788305524) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 19:32:13 (1788305533) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:32:20 (1788305540) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:32:24 (1788305544) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:32:30 (1788305550) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:32:40 (1788305560) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:32:47 (1788305567) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.35402 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899588 s, 11.7 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:32:58 (1788305578) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.20193 s, 26.0 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:33:07 (1788305587) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:33:16 (1788305596) Stopping /mnt/lustre-ost1 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97b546470800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97b546470800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg427-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg427-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg427-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg427-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg427-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg427-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg427-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg427-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg427-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg427-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg427-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg427-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg427-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg427-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg427-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg427-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg427-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg427-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg427-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg427-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg427-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg427-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg427-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg427-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg427-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97b546470800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97b546470800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg427-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (126s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:35:22 (1788305722) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.118237 s, 26.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97b546470800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97b546470800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg427-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (34s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:35:56 (1788305756) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97b546470800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97b546470800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg427-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97b546470800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97b546470800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg427-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (59s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:36:55 (1788305815) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.23882 s, 12.4 MB/s -rw-r--r-- 1 root root 52428800 Sep 1 19:37 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.903s user 0m0.001s sys 0m0.162s Stopping /mnt/lustre-ost1 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97b546470800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97b546470800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.106s user 0m0.002s sys 0m0.176s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg427-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97b546470800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97b546470800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.201s user 0m0.000s sys 0m0.207s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg427-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg427-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97b546470800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97b546470800.ost_server_uuid in FULL state after 0 sec PASS 33b (84s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:38:19 (1788305899) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:38:23 (1788305903) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:38:26 (1788305906) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 19:38:29 (1788305909) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 19:38:36 (1788305916) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.484853 s, 26.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:38:48 (1788305928) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329969 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218107 s, 48.1 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:38:54 (1788305934) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:39:03 (1788305943) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00118255 s, 866 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00103594 s, 988 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00119565 s, 856 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00258502 s, 396 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 19:39:12 (1788305952) create test files with size 15743957 .. 68287432bb1129fbace848f52f3f2f29 - c70d5e522068a2ba2424bd043411ed0e - 7bd4b3079b5cc79ab44e7bc0b0857e52 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg427-client.virtnet /mnt/lustre (opts:) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 19:39:31 (1788305971) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (10s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:39:41 (1788305981) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:39:47 (1788305987) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875696 s, 12.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256693 s, 40.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 19:39:54 (1788305994) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121421 s, 17.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.257233 s, 16.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162454 s, 12.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 19:40:08 (1788306008) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 19:40:11 (1788306011) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:40:14 (1788306014) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023259 s, 45.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 19:40:22 (1788306022) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:40:26 (1788306026) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdd:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:40:35 (1788306035) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.298329 s, 35.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg427-client.virtnet /mnt/lustre (opts:) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:40:46 (1788306046) llite.lustre-ffff97b547f3f000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.293216 s, 35.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 28686 flags: 1 Stopping client oleg427-client.virtnet /mnt/lustre (opts:) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 28686 flags: 1 llite.lustre-ffff97b552086800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:41:00 (1788306060) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (32s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:41:33 (1788306093) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:42:04 (1788306124) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 19:42:07 (1788306127) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.231659 s, 13.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.136954 s, 23.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xed653b1 (nosync,stale) Mirror 2: 0xed653b1 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5cf56b4c060e879801adc47e3ea5ea0a - mirror 1 checksum: 7f168de227ccfb753f9c6eceb1ea981e - mirror 2 checksum: 5cf56b4c060e879801adc47e3ea5ea0a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-01T19:42:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-01T19:42:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5cf56b4c060e879801adc47e3ea5ea0a - mirror 1 checksum: 7f168de227ccfb753f9c6eceb1ea981e - mirror 2 checksum: 7f168de227ccfb753f9c6eceb1ea981e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-01T19:42:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-01T19:42:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:42:18 (1788306138) Failing mds1 on oleg427-server Stopping /mnt/lustre-mds1 (opts:) on oleg427-server 19:42:23 (1788306143) shut down facet: mds1 facet_host: oleg427-server facet_failover_host: oleg427-server Failover mds1 to oleg427-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 19:42:58 (1788306178) targets are mounted 19:42:58 (1788306178) facet_failover done oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370377 s, 28.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } PASS 50A (59s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:43:17 (1788306197) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:43:20 (1788306200) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:43:23 (1788306203) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0173355 s, 945 kB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:43:31 (1788306211) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:43:34 (1788306214) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00708918 s, 578 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0487281 s, 84.1 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:43:40 (1788306220) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:44:01 (1788306241) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:44:28 (1788306268) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:44:51 (1788306291) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:45:15 (1788306315) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 19:45:21 (1788306321) mirror create pid 54284 mirror split pid 54285 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54284 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54285 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 19:46:29 (1788306389) Seed set to 6391 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003 1788306391.309046 MIRROR_EXTEND to 2 mirrors 000004 1788306391.809331 MIRROR_RESYNC 2 mirrors 000006 1788306391.987460 write 0x03a1bc thru 0x03ffff (0x5e44 bytes) 000007 1788306392.065492 mapread 0x03f64d thru 0x03ffff (0x9b3 bytes) 000008 1788306392.067615 mapread 0x01befa thru 0x02038b (0x4492 bytes) 000010 1788306392.100118 read 0x010ddc thru 0x0203b6 (0xf5db bytes) 000014 1788306392.119127 mapwrite 0x006673 thru 0x00c51f (0x5ead bytes) 000018 1788306392.224752 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x9532 000019 1788306393.235193 trunc from 0x040000 to 0x009531 (0x36ace bytes) 000020 1788306393.388516 write 0x00ef9a thru 0x0112c9 (0x2330 bytes) 000021 1788306393.389984 mapwrite 0x0296e7 thru 0x0368be (0xd1d8 bytes) 000022 1788306393.505624 mapread 0x0314d9 thru 0x0368be (0x53e6 bytes) truncating to largest ever: 0xde02 000023 1788306393.509955 trunc from 0x0368bf to 0x00de01 (0x28abd bytes) 000024 1788306393.551763 write 0x000bb3 thru 0x00ee8e (0xe2dc bytes) 000025 1788306393.559411 mapwrite 0x03a160 thru 0x03ffff (0x5ea0 bytes) 000026 1788306393.639134 write 0x021bb1 thru 0x0298e5 (0x7d35 bytes) 000027 1788306393.658781 read 0x01a318 thru 0x020941 (0x662a bytes) 000030 1788306393.671356 mapwrite 0x0274ff thru 0x030252 (0x8d54 bytes) 000034 1788306393.815979 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7524cf7e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7346dbd0 Mirror 3: 0x7346dbd0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x25615 000036 1788306394.731602 trunc from 0x040000 to 0x025614 (0x1a9eb bytes) 000040 1788306394.813100 write 0x03bfad thru 0x03ffff (0x4053 bytes) truncating to largest ever: 0x2c198 000043 1788306394.816627 trunc from 0x040000 to 0x02c197 (0x13e68 bytes) 000044 1788306394.843131 write 0x009b1a thru 0x015292 (0xb779 bytes) 000045 1788306394.852969 trunc from 0x02c198 to 0x008404 (0x23d93 bytes) 000047 1788306394.869030 mapwrite 0x0188de thru 0x018b31 (0x254 bytes) 000048 1788306394.929796 write 0x02eae1 thru 0x03db2f (0xf04f bytes) 000049 1788306394.931374 mapread 0x032b1d thru 0x0394a8 (0x698c bytes) 000052 1788306394.934583 read 0x02a249 thru 0x034981 (0xa739 bytes) 000053 1788306394.940284 mapread 0x01df91 thru 0x02174b (0x37bb bytes) 000054 1788306394.953581 trunc from 0x03db30 to 0x015320 (0x2880f bytes) 000055 1788306394.969851 mapwrite 0x011130 thru 0x012c1a (0x1aeb bytes) 000057 1788306395.012699 write 0x033aac thru 0x03f9db (0xbf30 bytes) 000058 1788306395.014194 trunc from 0x03f9dc to 0x01b577 (0x24464 bytes) 000059 1788306395.030515 read 0x006392 thru 0x01614a (0xfdb9 bytes) truncating to largest ever: 0x33c2b 000063 1788306395.046732 trunc from 0x01b578 to 0x033c2a (0x186b3 bytes) 000066 1788306395.060738 read 0x031a22 thru 0x03218d (0x76c bytes) 000068 1788306395.074604 mapwrite 0x01bf28 thru 0x02a424 (0xe4fd bytes) 000069 1788306395.138935 write 0x02964f thru 0x029ebe (0x870 bytes) 000070 1788306395.146488 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33c2b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c2b): Mirror 2: 0x9f6b4112 (stale) Mirror 3: 0xa699e07e (stale) CRC-32 checksum value for chunk [0, 0x33c2b): Mirror 1: 0xb71bd42a Mirror 4: 0xb71bd42a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c2b: skipped 000071 1788306396.040031 trunc from 0x033c2b to 0x004bbc (0x2f06e bytes) 000072 1788306396.171665 MIRROR_RESYNC 4 mirrors 000073 1788306397.180317 write 0x00136f thru 0x001784 (0x416 bytes) 000074 1788306397.240839 read 0x001ed4 thru 0x004bbc (0x2ce9 bytes) 000075 1788306397.251118 mapread 0x00028b thru 0x004bac (0x4922 bytes) 000076 1788306397.260093 mapread 0x001aa0 thru 0x004bbc (0x311d bytes) 000077 1788306397.261337 write 0x026a51 thru 0x035840 (0xedf0 bytes) 000078 1788306397.268454 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35841): Mirror 3: 0x6949bc1b (stale) Mirror 4: 0x6949bc1b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35841): Mirror 3: 0x6949bc1b (stale) Mirror 4: 0x6949bc1b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35841: skipped 000079 1788306397.605432 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35841): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35841): Mirror 3: 0x6949bc1b (stale) Mirror 4: 0x6949bc1b (stale) CRC-32 checksum value for chunk [0, 0x35841): Mirror 1: 0xc554fa58 Mirror 5: 0xc554fa58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35841: skipped 000080 1788306398.735141 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35841): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35841): Mirror 3: 0x6949bc1b (stale) Mirror 4: 0x6949bc1b (stale) CRC-32 checksum value for chunk [0, 0x35841): Mirror 1: 0xc554fa58 Mirror 5: 0xc554fa58 Mirror 6: 0xc554fa58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35841: skipped 000082 1788306399.833258 mapread 0x01a308 thru 0x024b6d (0xa866 bytes) 000083 1788306399.859148 mapwrite 0x02a095 thru 0x032441 (0x83ad bytes) 000088 1788306400.066725 write 0x00f11f thru 0x011e3d (0x2d1f bytes) 000089 1788306400.079296 write 0x024e90 thru 0x0331f0 (0xe361 bytes) 000091 1788306400.087367 write 0x02313c thru 0x0295fd (0x64c2 bytes) 000092 1788306400.089725 write 0x031bbe thru 0x03ffff (0xe442 bytes) 000093 1788306400.098326 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1eef4f2b (stale) Mirror 5: 0x8e8567da (stale) Mirror 6: 0x8e8567da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1eef4f2b (stale) Mirror 5: 0x8e8567da (stale) Mirror 6: 0x8e8567da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000094 1788306400.509339 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1eef4f2b (stale) Mirror 5: 0x8e8567da (stale) Mirror 6: 0x8e8567da (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf00d12b7 Mirror 7: 0xf00d12b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000095 1788306401.620790 write 0x03dda2 thru 0x03ffff (0x225e bytes) 000098 1788306401.716011 mapwrite 0x005fa0 thru 0x00df0c (0x7f6d bytes) 000099 1788306401.789259 read 0x00ab9d thru 0x00b548 (0x9ac bytes) 000100 1788306401.790146 write 0x00490d thru 0x00fe3e (0xb532 bytes) 000101 1788306401.793078 write 0x03db97 thru 0x03ffff (0x2469 bytes) 000104 1788306401.796667 trunc from 0x040000 to 0x02881c (0x177e3 bytes) 000106 1788306401.828039 write 0x02e4bb thru 0x03111a (0x2c60 bytes) 000107 1788306401.829959 mapread 0x00ac9a thru 0x012e3c (0x81a3 bytes) 000109 1788306401.861369 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3111b): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3111b): Mirror 3: 0x3158af6a (stale) Mirror 5: 0x816f6ae (stale) Mirror 6: 0x816f6ae (stale) Mirror 7: 0xf8d8aef0 (stale) CRC-32 checksum value for chunk [0, 0x3111b): Mirror 1: 0x62c9b4ae Mirror 8: 0x62c9b4ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3111b: skipped 000110 1788306403.295529 trunc from 0x03111b to 0x002499 (0x2ec81 bytes) 000113 1788306403.421788 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x249a): Mirror 5: 0x41e31202 (stale) Mirror 6: 0x41e31202 (stale) Mirror 7: 0x41e31202 (stale) Mirror 8: 0x41e31202 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x249a): Mirror 5: 0x41e31202 (stale) Mirror 6: 0x41e31202 (stale) Mirror 7: 0x41e31202 (stale) Mirror 8: 0x41e31202 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x249a: skipped 000116 1788306403.880882 mapwrite 0x034870 thru 0x03ffff (0xb790 bytes) 000117 1788306404.025440 mapwrite 0x038d7e thru 0x03fc59 (0x6edc bytes) 000121 1788306404.096649 trunc from 0x040000 to 0x03277d (0xd882 bytes) 000122 1788306404.122443 MIRROR_RESYNC 5 mirrors 000124 1788306405.535788 mapread 0x0034f2 thru 0x004b22 (0x1631 bytes) 000126 1788306405.546350 mapwrite 0x037538 thru 0x03cd2f (0x57f8 bytes) 000128 1788306405.688996 trunc from 0x03cd30 to 0x020224 (0x1cb0b bytes) 000129 1788306405.708941 write 0x0203d6 thru 0x02a54a (0xa175 bytes) 000130 1788306405.714279 mapwrite 0x038d15 thru 0x03d2f4 (0x45e0 bytes) 000132 1788306405.786744 trunc from 0x03d2f5 to 0x031fbc (0xb338 bytes) 000133 1788306405.801052 read 0x021e6f thru 0x02e2d5 (0xc467 bytes) 000134 1788306405.807544 mapread 0x002526 thru 0x00585e (0x3339 bytes) 000137 1788306405.818770 trunc from 0x031fbd to 0x02f38b (0x2c31 bytes) 000138 1788306405.832529 read 0x015fbf thru 0x01b6f6 (0x5738 bytes) 000140 1788306405.846423 mapread 0x029b9a thru 0x02f38b (0x57f2 bytes) 000143 1788306405.849836 write 0x00d521 thru 0x01ca52 (0xf532 bytes) 000144 1788306405.865642 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f38c): Mirror 5: 0x7c51c43c (stale) Mirror 7: 0x7c51c43c (stale) Mirror 8: 0x7c51c43c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f38c): Mirror 5: 0x7c51c43c (stale) Mirror 7: 0x7c51c43c (stale) Mirror 8: 0x7c51c43c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f38c: skipped 000146 1788306406.366661 mapwrite 0x03cee6 thru 0x03ffff (0x311a bytes) 000148 1788306406.454372 write 0x00d642 thru 0x0105b1 (0x2f70 bytes) 000149 1788306406.466390 mapread 0x011302 thru 0x018b91 (0x7890 bytes) 000155 1788306406.481293 read 0x00b7e1 thru 0x01406b (0x888b bytes) 000156 1788306406.482223 write 0x02f839 thru 0x03d015 (0xd7dd bytes) 000158 1788306406.490539 trunc from 0x040000 to 0x00bcdf (0x34320 bytes) 000159 1788306406.516351 MIRROR_RESYNC 4 mirrors truncating to largest ever: 0x38e61 000160 1788306407.584246 trunc from 0x00bce0 to 0x038e60 (0x2d181 bytes) 000161 1788306407.770080 trunc from 0x038e61 to 0x00ab4a (0x2e316 bytes) 000163 1788306407.798822 read 0x006133 thru 0x007ad9 (0x19a7 bytes) 000164 1788306407.808902 read 0x00016d thru 0x00ab4a (0xa9de bytes) 000166 1788306407.825882 read 0x00026a thru 0x00ab4a (0xa8e1 bytes) 000167 1788306407.826990 mapwrite 0x02a333 thru 0x02cc31 (0x28ff bytes) 000168 1788306407.901901 read 0x026c3c thru 0x02cc31 (0x5ff6 bytes) 000169 1788306407.907154 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cc32): Mirror 5: 0x2e2f4916 (stale) Mirror 7: 0x2e2f4916 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cc32): Mirror 5: 0x2e2f4916 (stale) Mirror 7: 0x2e2f4916 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cc32: skipped 000170 1788306408.415936 write 0x026eba thru 0x02e351 (0x7498 bytes) 000172 1788306408.455814 read 0x0287cc thru 0x02b2dd (0x2b12 bytes) 000173 1788306408.457280 trunc from 0x02e352 to 0x00361b (0x2ad36 bytes) 000174 1788306408.501871 write 0x02ca6d thru 0x0391ec (0xc780 bytes) 000175 1788306408.505452 trunc from 0x0391ed to 0x036a61 (0x278b bytes) 000176 1788306408.548751 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36a62): Mirror 7: 0x9a9595bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a62): Mirror 7: 0x9a9595bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a62: skipped 000177 1788306409.072052 read 0x006c65 thru 0x007254 (0x5f0 bytes) 000178 1788306409.079521 mapread 0x012c44 thru 0x012c4f (0xc bytes) 000179 1788306409.092788 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36a62): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a62): Mirror 7: 0x9a9595bf (stale) CRC-32 checksum value for chunk [0, 0x36a62): Mirror 1: 0x9166b8de Mirror 8: 0x9166b8de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a62: skipped 000180 1788306410.378649 mapread 0x022f6a thru 0x03117a (0xe211 bytes) 000182 1788306410.427023 mapread 0x025074 thru 0x034f2c (0xfeb9 bytes) 000183 1788306410.431854 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36a62): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a62): Mirror 7: 0x9a9595bf (stale) CRC-32 checksum value for chunk [0, 0x36a62): Mirror 1: 0x9166b8de Mirror 8: 0x9166b8de Mirror 9: 0x9166b8de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a62: skipped 000185 1788306411.498926 write 0x0010a6 thru 0x0092cc (0x8227 bytes) 000186 1788306411.563026 mapwrite 0x0278bb thru 0x0346e8 (0xce2e bytes) 000189 1788306411.649660 trunc from 0x036a62 to 0x01602b (0x20a36 bytes) 000192 1788306411.684050 trunc from 0x01602c to 0x023cd4 (0xdca9 bytes) 000193 1788306411.711643 mapread 0x0032a3 thru 0x011e5a (0xebb8 bytes) 000194 1788306411.726279 mapread 0x018813 thru 0x022605 (0x9df3 bytes) 000196 1788306411.753760 trunc from 0x023cd5 to 0x02a8a5 (0x6bd1 bytes) 000199 1788306411.776625 mapread 0x01592c thru 0x018157 (0x282c bytes) 000205 1788306411.793490 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a8a6): Mirror 8: 0xa33ae7ca (stale) Mirror 9: 0xa33ae7ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a8a6): Mirror 8: 0xa33ae7ca (stale) Mirror 9: 0xa33ae7ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a8a6: skipped 000206 1788306412.305789 trunc from 0x02a8a6 to 0x006bbd (0x23ce8 bytes) 000208 1788306412.333069 mapwrite 0x02d5e0 thru 0x030696 (0x30b7 bytes) 000209 1788306412.398070 mapwrite 0x01515e thru 0x0222bf (0xd162 bytes) 000212 1788306412.477550 mapread 0x00bdcd thru 0x01a5d1 (0xe805 bytes) 000214 1788306412.486056 mapwrite 0x00fcbb thru 0x0127d1 (0x2b17 bytes) 000215 1788306412.532390 read 0x00c712 thru 0x00fdff (0x36ee bytes) 000217 1788306412.534014 mapwrite 0x015b36 thru 0x02522d (0xf6f8 bytes) 000218 1788306412.604425 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30697): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30697): Mirror 8: 0xbd94872f (stale) Mirror 9: 0xbd94872f (stale) CRC-32 checksum value for chunk [0, 0x30697): Mirror 1: 0x50b0474 Mirror 10: 0x50b0474 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30697: skipped 000219 1788306413.459079 mapwrite 0x0111bd thru 0x01766c (0x64b0 bytes) 000220 1788306413.655852 write 0x018abe thru 0x021fc3 (0x9506 bytes) 000221 1788306413.666593 read 0x0115ea thru 0x01daca (0xc4e1 bytes) 000222 1788306413.672980 MIRROR_RESYNC 4 mirrors 000223 1788306414.698378 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) 000224 1788306415.149958 mapread 0x00fd59 thru 0x019d40 (0x9fe8 bytes) 000225 1788306415.177870 write 0x029e12 thru 0x032812 (0x8a01 bytes) 000228 1788306415.273970 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32813): Mirror 10: 0xa3bf5800 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32813): Mirror 10: 0xa3bf5800 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32813: skipped 000229 1788306415.684077 mapwrite 0x00ab32 thru 0x0177bf (0xcc8e bytes) 000231 1788306415.816683 write 0x0215a4 thru 0x02e463 (0xcec0 bytes) 000232 1788306415.841273 read 0x03272d thru 0x032812 (0xe6 bytes) 000233 1788306415.848101 write 0x036ffc thru 0x03ffff (0x9004 bytes) 000234 1788306415.856681 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xe71144ed (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcec4156e Mirror 11: 0xcec4156e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000236 1788306416.934380 mapwrite 0x026edb thru 0x02ca85 (0x5bab bytes) 000238 1788306417.147404 read 0x02c0db thru 0x0311e6 (0x510c bytes) 000239 1788306417.165213 trunc from 0x040000 to 0x02b170 (0x14e8f bytes) 000240 1788306417.205993 mapread 0x00f97c thru 0x013da6 (0x442b bytes) 000241 1788306417.229775 mapread 0x01dfc8 thru 0x027f1b (0x9f54 bytes) 000242 1788306417.239439 mapwrite 0x0072db thru 0x00edd5 (0x7afb bytes) 000244 1788306417.293236 read 0x000171 thru 0x00699c (0x682c bytes) 000245 1788306417.301973 MIRROR_RESYNC 3 mirrors 000246 1788306418.053936 MIRROR_EXTEND to 4 mirrors 000248 1788306418.752783 mapread 0x025df5 thru 0x02b170 (0x537c bytes) 000251 1788306418.778799 read 0x0006fa thru 0x000a46 (0x34d bytes) 000252 1788306418.784940 mapread 0x008938 thru 0x01427f (0xb948 bytes) 000253 1788306418.804965 write 0x0368cb thru 0x03ffff (0x9735 bytes) 000254 1788306418.863730 write 0x031de3 thru 0x03cf21 (0xb13f bytes) 000258 1788306418.875084 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xb656a623 (stale) Mirror 12: 0xb656a623 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xb656a623 (stale) Mirror 12: 0xb656a623 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000261 1788306419.207862 read 0x01d256 thru 0x02921c (0xbfc7 bytes) 000263 1788306419.218588 mapread 0x028dfb thru 0x037f75 (0xf17b bytes) 000264 1788306419.250657 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xb656a623 (stale) Mirror 12: 0xb656a623 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf433c10e Mirror 13: 0xf433c10e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000267 1788306420.142564 read 0x026623 thru 0x02ceed (0x68cb bytes) 000270 1788306420.168604 read 0x01812c thru 0x01af72 (0x2e47 bytes) 000272 1788306420.192891 write 0x03b7e5 thru 0x03ffff (0x481b bytes) 000274 1788306420.267403 trunc from 0x040000 to 0x006a9a (0x39565 bytes) 000275 1788306420.289971 write 0x01a5a5 thru 0x01d236 (0x2c92 bytes) 000276 1788306420.291708 mapread 0x0171b6 thru 0x01a605 (0x3450 bytes) 000278 1788306420.311677 write 0x033fb9 thru 0x03ffff (0xc047 bytes) 000279 1788306420.314861 read 0x005edf thru 0x006ad5 (0xbf7 bytes) 000280 1788306420.337631 read 0x00b50f thru 0x00c78c (0x127e bytes) 000281 1788306420.346586 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xb656a623 (stale) Mirror 13: 0xf433c10e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xb656a623 (stale) Mirror 13: 0xf433c10e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000284 1788306420.809820 write 0x00ca16 thru 0x019ed8 (0xd4c3 bytes) 000288 1788306420.828649 write 0x00452d thru 0x00e109 (0x9bdd bytes) 000289 1788306420.835028 write 0x02e2e1 thru 0x03dec5 (0xfbe5 bytes) truncating to largest ever: 0x3a730 000290 1788306420.846133 trunc from 0x040000 to 0x03a72f (0x58d0 bytes) 000292 1788306420.866920 mapwrite 0x0190e5 thru 0x028850 (0xf76c bytes) 000293 1788306420.946294 write 0x0013f0 thru 0x00f753 (0xe364 bytes) 000296 1788306420.953676 mapread 0x00b31a thru 0x01468a (0x9371 bytes) 000297 1788306420.955098 mapread 0x02d1da thru 0x0354bb (0x82e2 bytes) 000299 1788306420.957128 read 0x037e0b thru 0x03a72f (0x2925 bytes) 000302 1788306420.959312 trunc from 0x03a730 to 0x00ff4e (0x2a7e1 bytes) 000303 1788306420.975417 trunc from 0x00ff4f to 0x013bc5 (0x3c77 bytes) 000305 1788306420.993930 mapwrite 0x02026d thru 0x028886 (0x861a bytes) 000306 1788306421.089629 mapwrite 0x03373e thru 0x03ffff (0xc8c2 bytes) 000308 1788306421.172888 mapread 0x01bd14 thru 0x024166 (0x8453 bytes) 000309 1788306421.183056 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xb656a623 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xb656a623 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000310 1788306421.449885 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x3f654 000313 1788306422.341962 trunc from 0x040000 to 0x03f653 (0x9ac bytes) 000314 1788306422.454359 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f654): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f654): Mirror 11: 0x357e653b (stale) CRC-32 checksum value for chunk [0, 0x3f654): Mirror 1: 0x357e653b Mirror 12: 0x357e653b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f654: skipped 000315 1788306423.775366 read 0x00066b thru 0x00efc6 (0xe95c bytes) 000316 1788306423.800262 write 0x005ee6 thru 0x007658 (0x1773 bytes) 000317 1788306423.973465 trunc from 0x03f654 to 0x02b052 (0x14601 bytes) 000318 1788306423.998357 trunc from 0x02b053 to 0x002669 (0x289e9 bytes) 000319 1788306424.024595 trunc from 0x00266a to 0x015c0b (0x135a2 bytes) 000320 1788306424.043023 write 0x01931f thru 0x01c252 (0x2f34 bytes) 000323 1788306424.051790 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1c253): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c253): Mirror 11: 0x53468918 (stale) Mirror 12: 0x53468918 (stale) CRC-32 checksum value for chunk [0, 0x1c253): Mirror 1: 0x36cc60c7 Mirror 13: 0x36cc60c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c253: skipped 000324 1788306425.182922 trunc from 0x01c253 to 0x01eafe (0x28ac bytes) 000325 1788306425.299949 mapwrite 0x02cb8a thru 0x03a257 (0xd6ce bytes) 000326 1788306425.455808 MIRROR_RESYNC 4 mirrors 000328 1788306427.326276 trunc from 0x03a258 to 0x03a30d (0xb6 bytes) 000330 1788306427.425909 mapread 0x01b872 thru 0x023d0b (0x849a bytes) 000332 1788306427.447472 write 0x02afa8 thru 0x03af4b (0xffa4 bytes) 000333 1788306427.456890 mapread 0x036a0b thru 0x038836 (0x1e2c bytes) 000339 1788306427.462139 read 0x00dfa1 thru 0x015d8a (0x7dea bytes) 000340 1788306427.482785 trunc from 0x03af4c to 0x0183c8 (0x22b83 bytes) 000341 1788306427.501855 mapread 0x012685 thru 0x0183c8 (0x5d44 bytes) 000342 1788306427.502876 read 0x014584 thru 0x0183c8 (0x3e45 bytes) 000344 1788306427.505080 trunc from 0x0183c9 to 0x03957f (0x211b7 bytes) 000345 1788306427.519079 mapread 0x01effb thru 0x027eaa (0x8eb0 bytes) 000348 1788306427.539129 read 0x02364b thru 0x02a3e1 (0x6d97 bytes) 000349 1788306427.540418 mapwrite 0x00d4e3 thru 0x00d806 (0x324 bytes) 000351 1788306427.584909 trunc from 0x039580 to 0x03a67b (0x10fc bytes) 000352 1788306427.617914 mapwrite 0x030399 thru 0x032e8d (0x2af5 bytes) 000353 1788306427.701892 mapread 0x002e2d thru 0x0106e6 (0xd8ba bytes) 000355 1788306427.703390 mapwrite 0x011b35 thru 0x014404 (0x28d0 bytes) 000357 1788306427.753485 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a67c): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a67c): Mirror 11: 0x26226e7e (stale) Mirror 12: 0x26226e7e (stale) Mirror 13: 0x26226e7e (stale) CRC-32 checksum value for chunk [0, 0x3a67c): Mirror 1: 0xc6717ec9 Mirror 14: 0xc6717ec9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a67c: skipped 000358 1788306428.710437 trunc from 0x03a67c to 0x01349f (0x271dc bytes) 000359 1788306428.790212 mapread 0x000a80 thru 0x00d15c (0xc6dd bytes) 000360 1788306428.817247 trunc from 0x0134a0 to 0x00b2b5 (0x81ea bytes) 000361 1788306428.839356 mapwrite 0x01f764 thru 0x02e8d3 (0xf170 bytes) 000362 1788306429.038651 trunc from 0x02e8d4 to 0x00922a (0x256a9 bytes) 000366 1788306429.070626 mapread 0x000fbb thru 0x00922a (0x8270 bytes) 000367 1788306429.072524 mapread 0x006588 thru 0x00922a (0x2ca3 bytes) 000369 1788306429.086435 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x922b): Mirror 11: 0xfc00c5a3 (stale) Mirror 13: 0xfc00c5a3 (stale) Mirror 14: 0xfc00c5a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x922b): Mirror 11: 0xfc00c5a3 (stale) Mirror 13: 0xfc00c5a3 (stale) Mirror 14: 0xfc00c5a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x922b: skipped 000370 1788306429.498832 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x922b): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x922b): Mirror 11: 0xfc00c5a3 (stale) Mirror 13: 0xfc00c5a3 (stale) Mirror 14: 0xfc00c5a3 (stale) CRC-32 checksum value for chunk [0, 0x922b): Mirror 1: 0xfc00c5a3 Mirror 15: 0xfc00c5a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x922b: skipped 000372 1788306430.702987 read 0x000d87 thru 0x00922a (0x84a4 bytes) 000373 1788306430.719361 write 0x00af2f thru 0x015355 (0xa427 bytes) 000374 1788306430.808715 mapread 0x005160 thru 0x00b738 (0x65d9 bytes) 000379 1788306430.837119 write 0x01ed53 thru 0x02870d (0x99bb bytes) 000380 1788306430.839324 trunc from 0x02870e to 0x014bfc (0x13b11 bytes) 000381 1788306430.865322 MIRROR_RESYNC 5 mirrors 000382 1788306431.922796 MIRROR_EXTEND to 6 mirrors 000383 1788306432.693710 write 0x00cd2e thru 0x019f93 (0xd266 bytes) 000385 1788306432.798726 MIRROR_RESYNC 6 mirrors 000389 1788306434.134565 mapwrite 0x0222f2 thru 0x0264af (0x41be bytes) 000391 1788306434.388955 trunc from 0x0264b0 to 0x02b9e6 (0x5537 bytes) 000394 1788306434.421750 mapread 0x00be47 thru 0x00d767 (0x1921 bytes) 000395 1788306434.439663 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2b9e7): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b9e7): Mirror 11: 0x9640e25c (stale) Mirror 13: 0x9640e25c (stale) Mirror 14: 0x9640e25c (stale) Mirror 15: 0x9640e25c (stale) Mirror 16: 0x9640e25c (stale) CRC-32 checksum value for chunk [0, 0x2b9e7): Mirror 1: 0x4f3fb5c3 Mirror 17: 0x4f3fb5c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b9e7: skipped 000397 1788306435.669990 mapwrite 0x02c647 thru 0x039970 (0xd32a bytes) 000398 1788306435.817140 trunc from 0x039971 to 0x015ae8 (0x23e88 bytes) 000402 1788306435.844825 trunc from 0x015ae9 to 0x007eb7 (0xdc31 bytes) 000403 1788306435.864583 mapwrite 0x02d140 thru 0x03ada3 (0xdc64 bytes) 000404 1788306435.968501 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ada4): Mirror 11: 0x5e48334d (stale) Mirror 13: 0x5e48334d (stale) Mirror 14: 0x5e48334d (stale) Mirror 15: 0x5e48334d (stale) Mirror 16: 0x5e48334d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ada4): Mirror 11: 0x5e48334d (stale) Mirror 13: 0x5e48334d (stale) Mirror 14: 0x5e48334d (stale) Mirror 15: 0x5e48334d (stale) Mirror 16: 0x5e48334d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ada4: skipped 000405 1788306436.489571 read 0x033b17 thru 0x03ada3 (0x728d bytes) 000407 1788306436.498241 mapread 0x01d84a thru 0x024151 (0x6908 bytes) 000409 1788306436.539728 mapread 0x000457 thru 0x00b1b8 (0xad62 bytes) 000410 1788306436.571054 trunc from 0x03ada4 to 0x026e1e (0x13f85 bytes) 000415 1788306436.611988 read 0x000a9e thru 0x008c8b (0x81ee bytes) 000418 1788306436.617925 mapwrite 0x028a0a thru 0x02fca4 (0x729b bytes) 000419 1788306436.757732 read 0x01a977 thru 0x0277f4 (0xce7e bytes) 000420 1788306436.759082 mapwrite 0x0168f5 thru 0x018e99 (0x25a5 bytes) 000422 1788306436.833632 read 0x016d67 thru 0x020f56 (0xa1f0 bytes) 000423 1788306436.844397 MIRROR_RESYNC 6 mirrors 000425 1788306439.950568 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) 000426 1788306440.402424 mapread 0x017541 thru 0x01b33b (0x3dfb bytes) 000429 1788306440.458570 trunc from 0x02fca5 to 0x0362ff (0x665b bytes) 000430 1788306440.567490 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36300): Mirror 14: 0xb0372759 (stale) Mirror 15: 0xb0372759 (stale) Mirror 16: 0xb0372759 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36300): Mirror 14: 0xb0372759 (stale) Mirror 15: 0xb0372759 (stale) Mirror 16: 0xb0372759 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36300: skipped 000432 1788306440.935744 read 0x01cff3 thru 0x028a22 (0xba30 bytes) 000433 1788306440.946714 mapwrite 0x03d8ed thru 0x03ffff (0x2713 bytes) 000436 1788306441.027661 mapwrite 0x02a64c thru 0x02fafb (0x54b0 bytes) 000437 1788306441.093951 trunc from 0x040000 to 0x010f13 (0x2f0ec bytes) 000439 1788306441.122880 mapwrite 0x00d784 thru 0x00f7d6 (0x2053 bytes) 000440 1788306441.225910 trunc from 0x010f14 to 0x03248d (0x2157a bytes) 000444 1788306441.262220 mapwrite 0x01273e thru 0x01a225 (0x7ae8 bytes) 000445 1788306441.316744 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3248e): Mirror 15: 0xa56997a9 (stale) Mirror 16: 0xa56997a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3248e): Mirror 15: 0xa56997a9 (stale) Mirror 16: 0xa56997a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3248e: skipped 000448 1788306441.742123 read 0x028957 thru 0x03248d (0x9b37 bytes) 000449 1788306441.755272 mapwrite 0x039385 thru 0x03aa77 (0x16f3 bytes) 000450 1788306441.833658 MIRROR_RESYNC 3 mirrors 000453 1788306442.967109 MIRROR_EXTEND to 4 mirrors 000454 1788306443.952105 write 0x0085e3 thru 0x0162d2 (0xdcf0 bytes) 000456 1788306444.026440 write 0x0075cb thru 0x008bb8 (0x15ee bytes) 000461 1788306444.036477 mapwrite 0x008f35 thru 0x010c56 (0x7d22 bytes) 000463 1788306444.088356 mapwrite 0x032fdb thru 0x038d9e (0x5dc4 bytes) 000465 1788306444.160999 read 0x02a033 thru 0x0323be (0x838c bytes) 000466 1788306444.167501 mapread 0x02da41 thru 0x038c56 (0xb216 bytes) 000468 1788306444.169566 write 0x032898 thru 0x03729c (0x4a05 bytes) 000469 1788306444.176846 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3aa78): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa78): Mirror 15: 0xb10da90f (stale) Mirror 16: 0xb10da90f (stale) Mirror 17: 0xb10da90f (stale) CRC-32 checksum value for chunk [0, 0x3aa78): Mirror 1: 0xc1fbf257 Mirror 18: 0xc1fbf257 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa78: skipped 000470 1788306445.228546 mapwrite 0x02bc9c thru 0x02dd83 (0x20e8 bytes) 000471 1788306445.474351 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3aa78): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa78): Mirror 15: 0xb10da90f (stale) Mirror 16: 0xb10da90f (stale) Mirror 17: 0xb10da90f (stale) Mirror 18: 0xc1fbf257 (stale) CRC-32 checksum value for chunk [0, 0x3aa78): Mirror 1: 0x6c0066e0 Mirror 19: 0x6c0066e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa78: skipped 000472 1788306446.671546 mapread 0x004172 thru 0x01085f (0xc6ee bytes) 000473 1788306446.689432 read 0x017ad4 thru 0x01c665 (0x4b92 bytes) 000476 1788306446.695745 read 0x037c89 thru 0x03aa77 (0x2def bytes) 000480 1788306446.705583 mapread 0x0140cf thru 0x0211e0 (0xd112 bytes) 000481 1788306446.706522 mapwrite 0x01d7f7 thru 0x0234a8 (0x5cb2 bytes) 000483 1788306446.866798 trunc from 0x03aa78 to 0x027bae (0x12ec9 bytes) 000486 1788306446.897480 write 0x0298c8 thru 0x02ec4f (0x5388 bytes) 000487 1788306446.898981 mapwrite 0x03dd01 thru 0x03ee62 (0x1162 bytes) 000488 1788306446.955483 mapread 0x03af4e thru 0x03d837 (0x28ea bytes) 000489 1788306446.961370 read 0x0121c8 thru 0x0190ad (0x6ee6 bytes) 000491 1788306446.970523 mapwrite 0x0103aa thru 0x02029d (0xfef4 bytes) 000493 1788306447.013062 mapwrite 0x010270 thru 0x015642 (0x53d3 bytes) 000494 1788306447.056635 mapread 0x02f9c2 thru 0x039ecd (0xa50c bytes) 000496 1788306447.058057 trunc from 0x03ee63 to 0x003884 (0x3b5de bytes) 000498 1788306447.074267 trunc from 0x003885 to 0x02cc03 (0x2937f bytes) 000500 1788306447.087643 read 0x027061 thru 0x02cc03 (0x5ba3 bytes) 000501 1788306447.100690 mapread 0x0285e7 thru 0x02cc03 (0x461d bytes) 000502 1788306447.103492 trunc from 0x02cc04 to 0x0209e4 (0xc21f bytes) 000503 1788306447.123151 mapwrite 0x0300bf thru 0x03610b (0x604d bytes) 000505 1788306447.204669 mapwrite 0x00c264 thru 0x01339b (0x7138 bytes) 000507 1788306447.275793 write 0x00f08a thru 0x01e7da (0xf751 bytes) 000509 1788306447.280829 mapwrite 0x03fb17 thru 0x03ffff (0x4e9 bytes) 000514 1788306447.379753 trunc from 0x040000 to 0x01592f (0x2a6d0 bytes) 000516 1788306447.401145 write 0x01def3 thru 0x01f56c (0x167a bytes) 000517 1788306447.407860 MIRROR_RESYNC 6 mirrors 000519 1788306448.647409 read 0x00cd9d thru 0x011b17 (0x4d7b bytes) 000520 1788306448.658396 trunc from 0x01f56d to 0x010fcc (0xe5a0 bytes) 000521 1788306448.727633 trunc from 0x010fcd to 0x00b707 (0x58c5 bytes) 000525 1788306448.754375 write 0x01a9ec thru 0x01b344 (0x959 bytes) 000527 1788306448.757502 write 0x009298 thru 0x01074b (0x74b4 bytes) 000528 1788306448.768118 mapread 0x0049a6 thru 0x00e048 (0x96a3 bytes) 000529 1788306448.790964 write 0x002767 thru 0x0080d3 (0x596d bytes) 000530 1788306448.792781 write 0x03750d thru 0x03ffff (0x8af3 bytes) 000532 1788306448.796036 mapwrite 0x032ef6 thru 0x03a6e7 (0x77f2 bytes) 000533 1788306448.854282 read 0x0351c2 thru 0x0372a6 (0x20e5 bytes) 000534 1788306448.854887 mapwrite 0x00ee93 thru 0x00f90f (0xa7d bytes) 000536 1788306448.888147 read 0x01ff65 thru 0x02e7ea (0xe886 bytes) 000540 1788306448.899899 mapread 0x01db16 thru 0x028035 (0xa520 bytes) 000541 1788306448.901039 mapread 0x03918d thru 0x03ffff (0x6e73 bytes) 000542 1788306448.902558 read 0x03fd5b thru 0x03ffff (0x2a5 bytes) 000543 1788306448.903439 read 0x02db63 thru 0x03abc3 (0xd061 bytes) 000544 1788306448.904394 write 0x036ad7 thru 0x036e9b (0x3c5 bytes) 000546 1788306448.912767 MIRROR_RESYNC 6 mirrors 000547 1788306450.638327 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) 000549 1788306451.037675 mapread 0x01b72b thru 0x029069 (0xd93f bytes) 000550 1788306451.095461 MIRROR_EXTEND to 6 mirrors 000551 1788306452.084015 mapread 0x03c5a0 thru 0x03ffff (0x3a60 bytes) 000552 1788306452.109922 trunc from 0x040000 to 0x0202e9 (0x1fd16 bytes) 000553 1788306452.311244 mapread 0x0018f7 thru 0x005773 (0x3e7d bytes) 000554 1788306452.354563 read 0x001588 thru 0x006acc (0x5545 bytes) 000555 1788306452.356399 write 0x03c7e3 thru 0x03ffff (0x381d bytes) 000558 1788306452.363482 mapwrite 0x019844 thru 0x01d746 (0x3f03 bytes) 000560 1788306452.463330 trunc from 0x040000 to 0x00b0b7 (0x34f48 bytes) 000561 1788306452.496027 mapwrite 0x014de6 thru 0x022dd6 (0xdff1 bytes) 000562 1788306452.633002 mapread 0x01fad3 thru 0x020d83 (0x12b1 bytes) 000563 1788306452.640962 mapwrite 0x02d480 thru 0x030845 (0x33c6 bytes) 000566 1788306452.749416 write 0x0027ca thru 0x00aa2d (0x8264 bytes) 000570 1788306452.753264 mapwrite 0x0172bc thru 0x01918c (0x1ed1 bytes) 000571 1788306452.790142 trunc from 0x030846 to 0x0177db (0x1906a bytes) 000574 1788306452.820396 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x177dc): Mirror 15: 0x893829f1 (stale) Mirror 16: 0x893829f1 (stale) Mirror 18: 0x893829f1 (stale) Mirror 19: 0x893829f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x177dc): Mirror 15: 0x893829f1 (stale) Mirror 16: 0x893829f1 (stale) Mirror 18: 0x893829f1 (stale) Mirror 19: 0x893829f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x177dc: skipped 000575 1788306453.428642 mapread 0x008ce6 thru 0x01145c (0x8777 bytes) 000576 1788306453.449971 mapread 0x00cab0 thru 0x0177db (0xad2c bytes) 000578 1788306453.468457 write 0x0390c0 thru 0x03ffff (0x6f40 bytes) 000579 1788306453.469719 trunc from 0x040000 to 0x01a5e7 (0x25a18 bytes) 000580 1788306453.495395 mapread 0x019f97 thru 0x01a5e7 (0x651 bytes) 000581 1788306453.511485 mapread 0x000c9a thru 0x003a6e (0x2dd5 bytes) 000582 1788306453.521762 read 0x019a1a thru 0x01a5e7 (0xbce bytes) 000583 1788306453.522363 read 0x00ee72 thru 0x0154de (0x666d bytes) 000584 1788306453.530210 MIRROR_RESYNC 5 mirrors 000587 1788306454.795350 mapwrite 0x0217c8 thru 0x029315 (0x7b4e bytes) 000588 1788306454.992488 read 0x02665e thru 0x029315 (0x2cb8 bytes) 000590 1788306454.995684 read 0x021212 thru 0x029315 (0x8104 bytes) 000592 1788306454.997153 mapread 0x019f16 thru 0x01ec24 (0x4d0f bytes) 000594 1788306455.013262 read 0x01b876 thru 0x01ccf0 (0x147b bytes) 000596 1788306455.014353 write 0x032e9f thru 0x03ffff (0xd161 bytes) 000598 1788306455.018330 write 0x026b99 thru 0x032608 (0xba70 bytes) 000599 1788306455.020854 mapwrite 0x02223f thru 0x023bf3 (0x19b5 bytes) 000600 1788306455.092668 trunc from 0x040000 to 0x01ba09 (0x245f6 bytes) 000601 1788306455.137652 trunc from 0x01ba0a to 0x013db3 (0x7c56 bytes) 000602 1788306455.160313 trunc from 0x013db4 to 0x011358 (0x2a5b bytes) 000603 1788306455.193931 mapwrite 0x00c388 thru 0x011f0d (0x5b86 bytes) 000604 1788306455.307159 trunc from 0x011f0e to 0x015aba (0x3bad bytes) 000605 1788306455.330332 read 0x013795 thru 0x015811 (0x207d bytes) 000606 1788306455.338463 trunc from 0x015abb to 0x014456 (0x1664 bytes) 000607 1788306455.361388 trunc from 0x014457 to 0x00913c (0xb31a bytes) 000608 1788306455.401873 write 0x035dcc thru 0x03ffff (0xa234 bytes) 000609 1788306455.405775 trunc from 0x040000 to 0x01cee1 (0x2311e bytes) 000611 1788306455.442129 write 0x006ef6 thru 0x00c0d3 (0x51de bytes) 000615 1788306455.463893 mapread 0x016d1a thru 0x01cee1 (0x61c8 bytes) 000616 1788306455.471059 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cee2): Mirror 16: 0x60bad0bd (stale) Mirror 18: 0x60bad0bd (stale) Mirror 19: 0x60bad0bd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cee2): Mirror 16: 0x60bad0bd (stale) Mirror 18: 0x60bad0bd (stale) Mirror 19: 0x60bad0bd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cee2: skipped 000617 1788306455.885369 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1cee2): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cee2): Mirror 16: 0x60bad0bd (stale) Mirror 18: 0x60bad0bd (stale) Mirror 19: 0x60bad0bd (stale) CRC-32 checksum value for chunk [0, 0x1cee2): Mirror 1: 0x47b73c63 Mirror 20: 0x47b73c63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cee2: skipped 000620 1788306457.206417 mapwrite 0x024636 thru 0x031b36 (0xd501 bytes) 000621 1788306457.414889 mapread 0x013575 thru 0x016790 (0x321c bytes) 000622 1788306457.426402 mapwrite 0x008119 thru 0x00ab43 (0x2a2b bytes) 000623 1788306457.471988 write 0x0378f4 thru 0x03deda (0x65e7 bytes) 000625 1788306457.478105 mapwrite 0x03d654 thru 0x03ffff (0x29ac bytes) 000626 1788306457.635330 mapread 0x0363e8 thru 0x03dc9c (0x78b5 bytes) 000627 1788306457.644975 mapwrite 0x03a000 thru 0x03ffff (0x6000 bytes) 000628 1788306457.690017 read 0x032c22 thru 0x036c4a (0x4029 bytes) 000632 1788306457.697597 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xd2323cab (stale) Mirror 18: 0xd2323cab (stale) Mirror 19: 0xd2323cab (stale) Mirror 20: 0x93776ac3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8451b04f Mirror 21: 0x8451b04f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000633 1788306458.694530 mapwrite 0x00d958 thru 0x012c3c (0x52e5 bytes) 000634 1788306458.853769 trunc from 0x040000 to 0x02f951 (0x106ae bytes) 000635 1788306458.876664 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2f952): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f952): Mirror 16: 0xe9c1b476 (stale) Mirror 18: 0xe9c1b476 (stale) Mirror 19: 0xe9c1b476 (stale) Mirror 20: 0xa0266bfd (stale) Mirror 21: 0xfa05d4fb (stale) CRC-32 checksum value for chunk [0, 0x2f952): Mirror 1: 0xa3c11237 Mirror 22: 0xa3c11237 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f952: skipped 000639 1788306460.240584 write 0x016487 thru 0x024dab (0xe925 bytes) 000640 1788306460.342145 mapread 0x029152 thru 0x02f951 (0x6800 bytes) 000641 1788306460.390803 mapwrite 0x011d81 thru 0x012734 (0x9b4 bytes) 000643 1788306460.480100 mapread 0x005d32 thru 0x014a19 (0xece8 bytes) 000644 1788306460.491727 mapwrite 0x00bbde thru 0x0179b0 (0xbdd3 bytes) 000645 1788306460.548745 mapread 0x0028b8 thru 0x00eeae (0xc5f7 bytes) 000648 1788306460.552474 mapwrite 0x036cbf thru 0x03ffff (0x9341 bytes) 000649 1788306460.648212 mapwrite 0x03a461 thru 0x03ffff (0x5b9f bytes) 000652 1788306460.686783 trunc from 0x040000 to 0x03c2a4 (0x3d5b bytes) 000653 1788306460.703934 mapwrite 0x01677d thru 0x0235ce (0xce52 bytes) 000654 1788306460.756627 read 0x0266dd thru 0x029b79 (0x349d bytes) 000655 1788306460.757268 read 0x009dc6 thru 0x016d34 (0xcf6f bytes) 000657 1788306460.758257 trunc from 0x03c2a5 to 0x02ca53 (0xf851 bytes) 000659 1788306460.781964 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2ca54): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ca54): Mirror 16: 0xab89f5a1 (stale) Mirror 18: 0xab89f5a1 (stale) Mirror 19: 0xab89f5a1 (stale) Mirror 20: 0xcc926349 (stale) Mirror 21: 0xe3e54c21 (stale) Mirror 22: 0xf7607699 (stale) CRC-32 checksum value for chunk [0, 0x2ca54): Mirror 1: 0x78fa09c1 Mirror 23: 0x78fa09c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ca54: skipped 000660 1788306461.743914 mapwrite 0x039662 thru 0x03ffff (0x699e bytes) 000661 1788306461.957773 write 0x03feae thru 0x03ffff (0x152 bytes) 000662 1788306461.963249 MIRROR_RESYNC 8 mirrors 000664 1788306464.031572 trunc from 0x040000 to 0x032ba8 (0xd457 bytes) 000665 1788306464.135196 mapwrite 0x0170be thru 0x024614 (0xd557 bytes) 000667 1788306464.208503 mapwrite 0x03f9d0 thru 0x03ffff (0x630 bytes) 000668 1788306464.286966 write 0x00ad09 thru 0x00d25a (0x2552 bytes) 000671 1788306464.300243 mapread 0x02328b thru 0x02b77c (0x84f2 bytes) 000676 1788306464.312370 read 0x004ee5 thru 0x00eeb0 (0x9fcc bytes) 000680 1788306464.323236 read 0x03dd5f thru 0x03ffff (0x22a1 bytes) 000681 1788306464.324098 write 0x016f20 thru 0x01cfbc (0x609d bytes) 000682 1788306464.325988 mapread 0x02ff39 thru 0x030cbb (0xd83 bytes) 000683 1788306464.327314 mapwrite 0x01dbfd thru 0x02bd4b (0xe14f bytes) 000684 1788306464.382043 read 0x02bc64 thru 0x02f8a1 (0x3c3e bytes) 000685 1788306464.382734 write 0x00f1b0 thru 0x00fabd (0x90e bytes) 000686 1788306464.384737 mapwrite 0x03b8af thru 0x03ffff (0x4751 bytes) 000687 1788306464.430962 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x848759da (stale) Mirror 19: 0x848759da (stale) Mirror 20: 0x848759da (stale) Mirror 21: 0x848759da (stale) Mirror 22: 0x848759da (stale) Mirror 23: 0x848759da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x848759da (stale) Mirror 19: 0x848759da (stale) Mirror 20: 0x848759da (stale) Mirror 21: 0x848759da (stale) Mirror 22: 0x848759da (stale) Mirror 23: 0x848759da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000692 1788306464.887224 mapread 0x03b780 thru 0x03ffff (0x4880 bytes) 000693 1788306464.906814 write 0x027dee thru 0x0339e6 (0xbbf9 bytes) 000695 1788306464.921120 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x848759da (stale) Mirror 19: 0x848759da (stale) Mirror 21: 0x848759da (stale) Mirror 22: 0x848759da (stale) Mirror 23: 0x848759da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x848759da (stale) Mirror 19: 0x848759da (stale) Mirror 21: 0x848759da (stale) Mirror 22: 0x848759da (stale) Mirror 23: 0x848759da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000697 1788306465.385600 mapread 0x011750 thru 0x014ec5 (0x3776 bytes) 000698 1788306465.431436 read 0x00de6b thru 0x018558 (0xa6ee bytes) 000699 1788306465.432202 trunc from 0x040000 to 0x007f08 (0x380f7 bytes) 000700 1788306465.462051 trunc from 0x007f09 to 0x015daa (0xdea2 bytes) 000704 1788306465.485948 mapwrite 0x0119db thru 0x0141ae (0x27d4 bytes) 000707 1788306465.538952 trunc from 0x015dab to 0x02d550 (0x177a6 bytes) 000709 1788306465.558003 mapread 0x02d450 thru 0x02d550 (0x101 bytes) 000710 1788306465.569987 write 0x01d985 thru 0x029759 (0xbdd5 bytes) 000711 1788306465.578199 trunc from 0x02d551 to 0x02e9b4 (0x1464 bytes) 000712 1788306465.599074 mapwrite 0x032f0d thru 0x033e9e (0xf92 bytes) 000713 1788306465.688920 read 0x0008d9 thru 0x000fe0 (0x708 bytes) 000715 1788306465.701782 read 0x00ec87 thru 0x014593 (0x590d bytes) 000716 1788306465.702554 mapwrite 0x03926f thru 0x03ffff (0x6d91 bytes) 000717 1788306465.768403 mapread 0x003dbb thru 0x004e76 (0x10bc bytes) 000718 1788306465.769479 mapread 0x023fee thru 0x02c6b6 (0x86c9 bytes) 000720 1788306465.772101 write 0x00cd90 thru 0x015175 (0x83e6 bytes) 000722 1788306465.774355 write 0x02a999 thru 0x03650a (0xbb72 bytes) 000725 1788306465.776798 mapread 0x028315 thru 0x03395a (0xb646 bytes) 000727 1788306465.779743 read 0x030a32 thru 0x037071 (0x6640 bytes) 000729 1788306465.781574 read 0x033663 thru 0x037d8f (0x472d bytes) 000732 1788306465.783732 mapwrite 0x00f01c thru 0x015556 (0x653b bytes) 000733 1788306465.840353 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x848759da (stale) Mirror 19: 0x848759da (stale) Mirror 21: 0x848759da (stale) Mirror 22: 0x848759da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x848759da (stale) Mirror 19: 0x848759da (stale) Mirror 21: 0x848759da (stale) Mirror 22: 0x848759da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000734 1788306466.205833 trunc from 0x040000 to 0x009471 (0x36b8e bytes) 000736 1788306466.229388 mapread 0x007d3d thru 0x009471 (0x1735 bytes) 000737 1788306466.265688 mapwrite 0x00629c thru 0x00e388 (0x80ed bytes) 000739 1788306466.365508 write 0x0298d0 thru 0x032a80 (0x91b1 bytes) 000740 1788306466.367668 trunc from 0x032a81 to 0x02e5c3 (0x44bd bytes) 000741 1788306466.389536 write 0x01ec2d thru 0x02d790 (0xeb64 bytes) 000742 1788306466.399868 mapwrite 0x03c4f7 thru 0x03ffff (0x3b09 bytes) 000744 1788306466.477025 mapwrite 0x00ef9c thru 0x01baa3 (0xcb08 bytes) 000745 1788306466.534242 trunc from 0x040000 to 0x0150d0 (0x2af2f bytes) 000746 1788306466.555092 read 0x009871 thru 0x012e8e (0x961e bytes) 000748 1788306466.557239 mapwrite 0x00dbe2 thru 0x016e93 (0x92b2 bytes) 000749 1788306466.686684 trunc from 0x016e94 to 0x002ce3 (0x141b0 bytes) 000750 1788306466.714675 mapread 0x001b88 thru 0x002ce3 (0x115c bytes) 000753 1788306466.721754 trunc from 0x002ce4 to 0x02340f (0x2072c bytes) 000754 1788306466.738946 write 0x0344cf thru 0x03ffff (0xbb31 bytes) 000755 1788306466.740958 read 0x0138e1 thru 0x013e82 (0x5a2 bytes) 000756 1788306466.756045 read 0x02cd39 thru 0x0383ec (0xb6b4 bytes) 000757 1788306466.756740 mapwrite 0x00f4cd thru 0x013f3b (0x4a6f bytes) 000758 1788306466.824498 trunc from 0x040000 to 0x0187cb (0x27834 bytes) 000759 1788306466.861704 MIRROR_RESYNC 5 mirrors 000760 1788306468.297709 read 0x008a87 thru 0x00c341 (0x38bb bytes) 000761 1788306468.323600 read 0x008952 thru 0x00a864 (0x1f13 bytes) 000762 1788306468.332318 MIRROR_EXTEND to 6 mirrors 000765 1788306469.021948 mapread 0x00656c thru 0x00bc00 (0x5695 bytes) 000768 1788306469.044565 MIRROR_RESYNC 6 mirrors 000769 1788306469.388882 mapwrite 0x02cf1b thru 0x030bb7 (0x3c9d bytes) 000770 1788306469.576705 mapwrite 0x027da4 thru 0x02ee4d (0x70aa bytes) 000771 1788306469.637006 write 0x01fda1 thru 0x02f0de (0xf33e bytes) 000772 1788306469.651519 trunc from 0x030bb8 to 0x030631 (0x586 bytes) 000774 1788306469.686865 trunc from 0x030632 to 0x02bfe7 (0x464a bytes) 000775 1788306469.708897 write 0x02e6c6 thru 0x03b765 (0xd0a0 bytes) 000776 1788306469.710540 mapread 0x01f953 thru 0x026314 (0x69c2 bytes) 000777 1788306469.715877 mapread 0x0044d3 thru 0x009edf (0x5a0d bytes) 000778 1788306469.734554 mapread 0x03aa5d thru 0x03b765 (0xd09 bytes) 000780 1788306469.737272 write 0x0397be thru 0x03d869 (0x40ac bytes) 000781 1788306469.744880 mapread 0x005cd1 thru 0x009e87 (0x41b7 bytes) 000782 1788306469.748968 write 0x02bd2c thru 0x03a636 (0xe90b bytes) 000784 1788306469.755727 mapwrite 0x0083a9 thru 0x00d611 (0x5269 bytes) 000785 1788306469.836441 trunc from 0x03d86a to 0x0304a6 (0xd3c3 bytes) 000786 1788306469.865672 MIRROR_RESYNC 6 mirrors 000788 1788306471.208493 read 0x01299a thru 0x01a0a8 (0x770f bytes) 000789 1788306471.219739 mapwrite 0x014ec0 thru 0x0193eb (0x452c bytes) 000790 1788306471.424814 MIRROR_RESYNC 6 mirrors 000793 1788306472.425037 mapread 0x009270 thru 0x01641a (0xd1ab bytes) 000796 1788306472.457387 read 0x01b5ac thru 0x024adc (0x9531 bytes) 000797 1788306472.467609 trunc from 0x0304a7 to 0x01da28 (0x12a7e bytes) 000801 1788306472.587043 read 0x012ea6 thru 0x01da28 (0xab83 bytes) 000802 1788306472.617931 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1da29): Mirror 16: 0x47d1eb82 (stale) Mirror 19: 0x47d1eb82 (stale) Mirror 21: 0x47d1eb82 (stale) Mirror 22: 0x47d1eb82 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1da29): Mirror 16: 0x47d1eb82 (stale) Mirror 19: 0x47d1eb82 (stale) Mirror 21: 0x47d1eb82 (stale) Mirror 22: 0x47d1eb82 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1da29: skipped 000805 1788306473.136988 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1da29): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1da29): Mirror 16: 0x47d1eb82 (stale) Mirror 19: 0x47d1eb82 (stale) Mirror 21: 0x47d1eb82 (stale) Mirror 22: 0x47d1eb82 (stale) CRC-32 checksum value for chunk [0, 0x1da29): Mirror 1: 0x47d1eb82 Mirror 23: 0x47d1eb82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1da29: skipped 000807 1788306474.071301 mapwrite 0x039c11 thru 0x03e3a0 (0x4790 bytes) 000809 1788306474.324661 mapwrite 0x023f32 thru 0x030a51 (0xcb20 bytes) 000811 1788306474.436964 read 0x02c43d thru 0x037b1b (0xb6df bytes) 000812 1788306474.441771 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e3a1): Mirror 19: 0x81850b58 (stale) Mirror 21: 0x81850b58 (stale) Mirror 22: 0x81850b58 (stale) Mirror 23: 0xfca0e3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e3a1): Mirror 19: 0x81850b58 (stale) Mirror 21: 0x81850b58 (stale) Mirror 22: 0x81850b58 (stale) Mirror 23: 0xfca0e3e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e3a1: skipped 000814 1788306474.778789 mapread 0x0282a1 thru 0x02a115 (0x1e75 bytes) 000815 1788306474.828409 mapwrite 0x0119a4 thru 0x0204d3 (0xeb30 bytes) 000816 1788306474.942913 mapwrite 0x016ebb thru 0x02601c (0xf162 bytes) 000817 1788306475.060864 read 0x01fe59 thru 0x02e37a (0xe522 bytes) 000818 1788306475.061545 read 0x034c61 thru 0x03d723 (0x8ac3 bytes) 000821 1788306475.106953 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e3a1): Mirror 21: 0x81850b58 (stale) Mirror 22: 0x81850b58 (stale) Mirror 23: 0xfca0e3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e3a1): Mirror 21: 0x81850b58 (stale) Mirror 22: 0x81850b58 (stale) Mirror 23: 0xfca0e3e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e3a1: skipped 000822 1788306475.611060 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e3a1): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e3a1): Mirror 21: 0x81850b58 (stale) Mirror 22: 0x81850b58 (stale) Mirror 23: 0xfca0e3e (stale) CRC-32 checksum value for chunk [0, 0x3e3a1): Mirror 1: 0xc3d24cb5 Mirror 24: 0xc3d24cb5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e3a1: skipped 000824 1788306476.584430 write 0x001b83 thru 0x00675d (0x4bdb bytes) 000827 1788306476.672553 read 0x03e348 thru 0x03e3a0 (0x59 bytes) 000828 1788306476.678362 trunc from 0x03e3a1 to 0x0348a4 (0x9afc bytes) 000829 1788306476.699406 write 0x03e4cc thru 0x03ffff (0x1b34 bytes) 000831 1788306476.701986 write 0x00e242 thru 0x01a618 (0xc3d7 bytes) 000834 1788306476.716327 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xafe3c0a3 (stale) Mirror 23: 0x63142f5a (stale) Mirror 24: 0x54f3dbd7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xafe3c0a3 (stale) Mirror 23: 0x63142f5a (stale) Mirror 24: 0x54f3dbd7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000835 1788306477.094468 trunc from 0x040000 to 0x00c163 (0x33e9c bytes) 000836 1788306477.111839 mapwrite 0x008a84 thru 0x013944 (0xaec1 bytes) 000837 1788306477.249016 mapread 0x00bf42 thru 0x00f21b (0x32da bytes) 000838 1788306477.250578 mapwrite 0x026965 thru 0x02b349 (0x49e5 bytes) 000840 1788306477.323916 trunc from 0x02b34a to 0x01087c (0x1aacd bytes) 000844 1788306477.340975 write 0x02c9be thru 0x02e271 (0x18b4 bytes) 000846 1788306477.345028 trunc from 0x02e272 to 0x03f0c2 (0x10e51 bytes) 000848 1788306477.374947 MIRROR_RESYNC 4 mirrors 000849 1788306479.108528 write 0x024953 thru 0x0324ac (0xdb5a bytes) 000850 1788306479.182531 mapread 0x000803 thru 0x00572c (0x4f2a bytes) 000852 1788306479.206392 read 0x024871 thru 0x025751 (0xee1 bytes) 000853 1788306479.211393 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f0c3): Mirror 22: 0xcbc87193 (stale) Mirror 23: 0xcbc87193 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f0c3): Mirror 22: 0xcbc87193 (stale) Mirror 23: 0xcbc87193 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f0c3: skipped 000856 1788306479.629916 trunc from 0x03f0c3 to 0x0109d8 (0x2e6ea bytes) 000857 1788306479.650837 trunc from 0x0109d9 to 0x02f5f0 (0x1ec18 bytes) 000858 1788306479.693802 mapread 0x0083ab thru 0x00cde7 (0x4a3d bytes) 000860 1788306479.720365 mapwrite 0x0018a6 thru 0x002a58 (0x11b3 bytes) 000862 1788306479.792114 read 0x00e1de thru 0x01738d (0x91b0 bytes) 000864 1788306479.803453 trunc from 0x02f5f1 to 0x00fdea (0x1f806 bytes) 000865 1788306479.828419 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xfdeb): Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfdeb): Mirror 22: 0xc98be500 (stale) Mirror 23: 0xc98be500 (stale) CRC-32 checksum value for chunk [0, 0xfdeb): Mirror 1: 0xab75d3a4 Mirror 24: 0xab75d3a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfdeb: skipped 000867 1788306480.666054 mapwrite 0x03342d thru 0x03ffff (0xcbd3 bytes) 000868 1788306480.891216 mapwrite 0x035318 thru 0x039781 (0x446a bytes) 000870 1788306480.977831 trunc from 0x040000 to 0x025a43 (0x1a5bc bytes) 000871 1788306481.022407 write 0x005ef1 thru 0x0069f5 (0xb05 bytes) 000872 1788306481.035891 mapwrite 0x00c39a thru 0x01298b (0x65f2 bytes) 000873 1788306481.136768 mapwrite 0x03f351 thru 0x03ffff (0xcaf bytes) 000874 1788306481.231787 trunc from 0x040000 to 0x004bac (0x3b453 bytes) 000875 1788306481.270098 write 0x030b1e thru 0x033e5f (0x3342 bytes) 000878 1788306481.279049 MIRROR_RESYNC 4 mirrors 000879 1788306482.633507 trunc from 0x033e60 to 0x02ea33 (0x542c bytes) 000881 1788306482.721581 mapwrite 0x0073e6 thru 0x007f6e (0xb89 bytes) 000882 1788306482.781909 trunc from 0x02ea34 to 0x0369bc (0x7f89 bytes) 000888 1788306482.813669 mapwrite 0x03be74 thru 0x03ffff (0x418c bytes) 000889 1788306482.944433 write 0x016089 thru 0x01d821 (0x7799 bytes) 000890 1788306482.979069 trunc from 0x040000 to 0x03d526 (0x2ad9 bytes) 000891 1788306483.010156 mapread 0x0294b4 thru 0x02a63c (0x1189 bytes) 000892 1788306483.034615 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d527): Mirror 22: 0x5c828f8e (stale) Mirror 23: 0x5c828f8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d527): Mirror 22: 0x5c828f8e (stale) Mirror 23: 0x5c828f8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d527: skipped 000893 1788306483.421096 write 0x01c2ee thru 0x025c73 (0x9986 bytes) 000895 1788306483.432039 read 0x005719 thru 0x013a9f (0xe387 bytes) 000896 1788306483.451075 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d527): Mirror 23: 0x5c828f8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d527): Mirror 23: 0x5c828f8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d527: skipped 000897 1788306483.744241 mapread 0x010ef6 thru 0x01bc71 (0xad7c bytes) 000899 1788306483.773003 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) 000900 1788306483.880648 write 0x01dbc7 thru 0x0218db (0x3d15 bytes) 000902 1788306483.900941 MIRROR_EXTEND to 2 mirrors 000903 1788306485.286270 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) 000906 1788306485.402400 MIRROR_EXTEND to 2 mirrors 000907 1788306486.349743 trunc from 0x03d527 to 0x0354da (0x804c bytes) 000908 1788306486.468991 mapread 0x023356 thru 0x0283d1 (0x507c bytes) 000910 1788306486.496980 trunc from 0x0354db to 0x02a9fd (0xaadd bytes) 000911 1788306486.520305 trunc from 0x02a9fe to 0x021ddd (0x8c20 bytes) 000912 1788306486.533123 trunc from 0x021dde to 0x02b790 (0x99b3 bytes) 000913 1788306486.550422 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) 000915 1788306486.668555 mapwrite 0x02b447 thru 0x03a378 (0xef32 bytes) 000916 1788306486.781619 trunc from 0x03a379 to 0x005da0 (0x345d8 bytes) 000917 1788306486.803842 mapread 0x0036a1 thru 0x005da0 (0x2700 bytes) 000919 1788306486.846718 MIRROR_EXTEND to 2 mirrors 000920 1788306487.629072 write 0x026acf thru 0x02d2b1 (0x67e3 bytes) 000921 1788306487.735563 mapwrite 0x00c0dc thru 0x013987 (0x78ac bytes) 000924 1788306487.798064 mapwrite 0x00fb6c thru 0x010b80 (0x1015 bytes) 000928 1788306487.835999 read 0x008e52 thru 0x010389 (0x7538 bytes) 000930 1788306487.838391 mapread 0x029356 thru 0x02d2b1 (0x3f5c bytes) 000932 1788306487.840020 read 0x003f35 thru 0x00477b (0x847 bytes) 000933 1788306487.857752 trunc from 0x02d2b2 to 0x03e0ef (0x10e3e bytes) 000934 1788306487.885110 trunc from 0x03e0f0 to 0x023ecd (0x1a222 bytes) 000940 1788306487.912396 mapwrite 0x01d092 thru 0x02b3a7 (0xe316 bytes) 000941 1788306488.069780 read 0x016bc5 thru 0x025269 (0xe6a5 bytes) 000948 1788306488.079814 MIRROR_RESYNC 2 mirrors 000950 1788306488.738011 MIRROR_EXTEND to 3 mirrors 000952 1788306489.377648 trunc from 0x02b3a8 to 0x000d13 (0x2a694 bytes) 000953 1788306489.489715 read 0x000380 thru 0x000d13 (0x994 bytes) 000954 1788306489.495469 trunc from 0x000d14 to 0x02140b (0x206f8 bytes) 000955 1788306489.518507 trunc from 0x02140c to 0x03013e (0xed33 bytes) 000956 1788306489.540041 mapwrite 0x031dfe thru 0x035028 (0x322b bytes) 000960 1788306489.638540 mapread 0x0249fe thru 0x0286b3 (0x3cb6 bytes) 000962 1788306489.649549 mapwrite 0x000c95 thru 0x00b7e1 (0xab4d bytes) 000963 1788306489.713319 mapread 0x0308a0 thru 0x035028 (0x4789 bytes) 000964 1788306489.714729 trunc from 0x035029 to 0x03c2be (0x7296 bytes) 000966 1788306489.742571 trunc from 0x03c2bf to 0x007855 (0x34a69 bytes) 000967 1788306489.761129 trunc from 0x007856 to 0x020b63 (0x1930e bytes) 000968 1788306489.779782 mapwrite 0x004914 thru 0x00cb00 (0x81ed bytes) 000972 1788306489.842637 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20b64): Mirror 3: 0x381f736f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20b64): Mirror 3: 0x381f736f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20b64: skipped 000975 1788306490.127204 MIRROR_RESYNC 2 mirrors 000976 1788306490.967871 mapread 0x0180f4 thru 0x020b63 (0x8a70 bytes) 000979 1788306490.988961 mapread 0x000d54 thru 0x007b5a (0x6e07 bytes) 000982 1788306490.998105 trunc from 0x020b64 to 0x00a611 (0x16552 bytes) 000983 1788306491.067470 trunc from 0x00a612 to 0x01c4a2 (0x11e91 bytes) 000984 1788306491.091374 read 0x015a24 thru 0x01be5b (0x6438 bytes) 000985 1788306491.106535 mapwrite 0x033b4e thru 0x03e833 (0xace6 bytes) 000989 1788306491.212945 read 0x021c35 thru 0x028d8b (0x7157 bytes) 000991 1788306491.223966 write 0x02c5c8 thru 0x031472 (0x4eab bytes) 000993 1788306491.226072 mapwrite 0x02a142 thru 0x030aeb (0x69aa bytes) 000994 1788306491.276638 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bd:0x0]: Invalid argument (22) 000996 1788306491.423921 mapread 0x0185e0 thru 0x01a73a (0x215b bytes) 000998 1788306491.455638 trunc from 0x03e834 to 0x002eab (0x3b988 bytes) 000999 1788306491.479544 MIRROR_EXTEND to 2 mirrors 001000 1788306491.980663 mapread 0x000ecd thru 0x002eab (0x1fdf bytes) All operations completed A-OK! PASS 70a (107s) == sanity-flr test 71: check mirror extend parent fid ==== 19:48:16 (1788306496) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 19:48:19 (1788306499) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.215389 s, 9.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275068 s, 3.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 19:48:28 (1788306508) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre2 Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6342560 .. Extending file size to 1709920 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 2430848 .. Extending file size to 4548704 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7748672 .. Extending file size to 8418592 .. Extending file size to 8556416 .. Extending file size to 8690240 .. Extending file size to 8874944 .. Extending file size to 8927584 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9082880 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9221216 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57454 57455 57457 57458 57461 done 192.168.204.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg427-client.virtnet /mnt/lustre2 (opts:) 192.168.204.127@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg427-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:49:50 (1788306590) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre2 Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6637568 .. Extending file size to 4935776 .. Extending file size to 9173088 .. Extending file size to 8413184 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8772224 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 58961 58962 58965 58966 58967 192.168.204.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg427-client.virtnet /mnt/lustre2 (opts:) 192.168.204.127@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg427-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:51:11 (1788306671) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0997843 s, 21.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.423009 s, 5.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x469aec94 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg427-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:51:30 (1788306690) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0870053 s, 24.1 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 19:51:37 (1788306697) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x215:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:51:43 (1788306703) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149192 s, 14.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:51:49 (1788306709) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.230175 s, 9.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:51:57 (1788306717) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206347 s, 10.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:52:29 (1788306749) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg427-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20056 s, 5.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:53:32 (1788306812) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg427-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22003 s, 9.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:54:28 (1788306868) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg427-server mds-ost sync done. Creating new pool test_204f oleg427-server: Pool lustre.test_204f created Adding targets to pool oleg427-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg427-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.256317 s, 8.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg427-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg427-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg427-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:55:44 (1788306944) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:55:50 (1788306950) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:55:57 (1788306957) oleg427-server: Pool lustre.test_206 created Adding targets to pool oleg427-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg427-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg427-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg427-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg427-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:56:21 (1788306981) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666346 s, 15.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465995 s, 22.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x55c1eb02 Mirror 3: 0x55c1eb02 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:56:31 (1788306991) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:56:35 (1788306995) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:56:37 (1788306997) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0932353 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2430000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2430000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2430000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2520000 Read this (4 bytes): BBBB 192.168.204.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg427-client.virtnet /mnt/lustre2 (opts:) PASS 209a (32s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:57:09 (1788307029) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0181412 s, 452 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:57:15 (1788307035) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562521 s, 18.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x23a lmm_fid: [0x200000404:0x23a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x23a lmm_fid: [0x200000404:0x23a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x318:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834490 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 19:57:16.000000000 -0400 Modify: 2026-09-01 19:57:16.000000000 -0400 Change: 2026-09-01 19:57:18.000000000 -0400 Birth: 2026-09-01 19:57:18.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:57:23 (1788307043) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:57:26 (1788307046) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.475008 s, 86.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0416383 s, 98.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 1 19:57 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 1 19:57 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:57:33 (1788307053) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:57:40 (1788307060) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1678 sec =========== 19:57:49 (1788307069) === sanity-flr: start cleanup 19:57:51 (1788307071) === === sanity-flr: finish cleanup 19:57:57 (1788307077) ===