-----============= acceptance-small: sanity-flr ============----- Fri Aug 14 18:36:45 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg308-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 18:37:03 (1786747023) === oleg308-client.virtnet: executing check_config_client /mnt/lustre oleg308-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg308-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa0f584c13800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0f584c13800.idle_timeout=debug disable quota as required oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 18:37:41 (1786747061) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8243] == sanity-flr test 0a: lfs mirror create with -N option == 18:37:43 (1786747063) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:38:00 (1786747080) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:38:02 (1786747082) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:38:05 (1786747085) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:38:29 (1786747109) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:38:33 (1786747113) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 18:38:37 (1786747117) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa0f584c13800.stats=clear osc.lustre-OST0001-osc-ffffa0f584c13800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:38:45 (1786747125) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:38:56 (1786747136) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:39:04 (1786747144) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 18:39:07 (1786747147) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:39:14 (1786747154) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 18:39:23 (1786747163) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:39:30 (1786747170) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:39:39 (1786747179) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:39:49 (1786747189) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:40:00 (1786747200) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:40:07 (1786747207) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.05089 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827351 s, 12.7 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:40:16 (1786747216) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.264296 s, 23.8 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 18:40:24 (1786747224) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:40:34 (1786747234) Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0f584c13800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0f584c13800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0f584c13800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0f584c13800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (128s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:42:42 (1786747362) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0979368 s, 21.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0f584c13800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0f584c13800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:43:23 (1786747403) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0f584c13800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0f584c13800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0f584c13800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0f584c13800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (73s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:44:36 (1786747476) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.66264 s, 19.7 MB/s -rw-r--r-- 1 root root 52428800 Aug 14 18:44 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.979s user 0m0.001s sys 0m0.197s Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0f584c13800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0f584c13800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.865s user 0m0.000s sys 0m0.183s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0f584c13800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0f584c13800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.180s user 0m0.001s sys 0m0.154s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg308-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0f584c13800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0f584c13800.ost_server_uuid in FULL state after 0 sec PASS 33b (88s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:46:05 (1786747565) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:46:08 (1786747568) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:46:12 (1786747572) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 18:46:15 (1786747575) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 18:46:22 (1786747582) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.726118 s, 17.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:46:35 (1786747595) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567101 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489047 s, 21.4 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:46:44 (1786747604) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:46:53 (1786747613) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000816799 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00160546 s, 638 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000852315 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00138723 s, 738 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 18:47:02 (1786747622) create test files with size 15752687 .. a69d2dac4852dbf4fc9b624ceb15af7f - 00760ee2c41f01b1ca86142527885367 - a3e1f09c0cd5b9892d4a0e19ad6bf296 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 18:47:23 (1786747643) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:47:35 (1786747655) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:47:42 (1786747662) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044473 s, 23.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637886 s, 16.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 18:47:49 (1786747669) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105309 s, 19.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.302262 s, 13.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162124 s, 12.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 18:48:03 (1786747683) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 18:48:06 (1786747686) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:48:09 (1786747689) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483262 s, 21.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 18:48:16 (1786747696) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.285823 s, 11.0 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:48:25 (1786747705) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:48:35 (1786747715) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.348328 s, 30.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:48:45 (1786747725) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffffa0f5a608f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.244546 s, 42.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffffa0f584c3b800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:48:58 (1786747738) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:49:36 (1786747776) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:50:15 (1786747815) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 18:50:19 (1786747819) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.26274 s, 12.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.194721 s, 16.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xfc22feba (nosync,stale) Mirror 2: 0xfc22feba (nosync,stale) ** verify mirror 2 doesn't change original checksum: 10cf430aa4043144bdb8b8be75a5b149 - mirror 1 checksum: 106b99fe6c5ac9980509d9ebda165d54 - mirror 2 checksum: 10cf430aa4043144bdb8b8be75a5b149 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T18:50:23-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T18:50:23-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 10cf430aa4043144bdb8b8be75a5b149 - mirror 1 checksum: 106b99fe6c5ac9980509d9ebda165d54 - mirror 2 checksum: 106b99fe6c5ac9980509d9ebda165d54 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T18:50:23-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T18:50:23-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 18:50:30 (1786747830) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381552 s, 27.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:50:38 (1786747838) Failing mds1 on oleg308-server Stopping /mnt/lustre-mds1 (opts:) on oleg308-server 18:50:49 (1786747849) shut down facet: mds1 facet_host: oleg308-server facet_failover_host: oleg308-server Failover mds1 to oleg308-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 18:51:19 (1786747879) targets are mounted 18:51:19 (1786747879) facet_failover done oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290957 s, 36.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:51:38 (1786747898) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0641437 s, 319 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0257846 s, 794 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:51:47 (1786747907) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.06546 s, 313 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0263167 s, 778 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:51:57 (1786747917) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0173343 s, 945 kB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:52:04 (1786747924) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0386829 s, 529 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:52:11 (1786747931) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00686962 s, 596 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0305762 s, 134 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:52:18 (1786747938) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:52:38 (1786747958) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (24s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:53:02 (1786747982) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:53:25 (1786748005) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:53:49 (1786748029) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 18:53:54 (1786748034) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57575 mirror split pid 57576 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7817: 57576 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57575 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 18:55:02 (1786748102) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 8104 skipping zero size read skipping zero length zero range truncating to largest ever: 0x16c22 000003 1786748104.497651 trunc from 00000000 to 0x016c21 (0x16c22 bytes) 000004 1786748104.519447 mapwrite 0x02c5c9 thru 0x038c5a (0xc692 bytes) 000005 1786748104.606469 fallocate 0x00517a thru 0x0099ee (0x4875 bytes) 000006 1786748104.623582 mapread 0x000fd6 thru 0x00ab76 (0x9ba1 bytes) 000008 1786748104.654758 mapread 0x004664 thru 0x00e0d5 (0x9a72 bytes) 000009 1786748104.656286 punch from 0x02e5d5 to 0x0327b5 (0x41e1 bytes) 000010 1786748104.664589 mapwrite 0x026271 thru 0x02906d (0x2dfd bytes) 000011 1786748104.723912 trunc from 0x038c5b to 0x0112cd (0x2798d bytes) fallocating to largest ever: 0x40000 000013 1786748104.758252 fallocate 0x03f427 thru 0x03ffff (0xbd9 bytes) 000014 1786748104.766449 read 0x003ad8 thru 0x008faa (0x54d3 bytes) 000015 1786748104.767390 trunc from 0x016565 to 0x02565d (0xf0f9 bytes) 000016 1786748104.773579 fallocate 0x030a0c thru 0x03a0be (0x96b3 bytes) 000017 1786748104.782467 write 0x0146a9 thru 0x014b9a (0x4f2 bytes) 000018 1786748104.790646 read 0x03bbb2 thru 0x03ffff (0x444e bytes) 000019 1786748104.823617 fallocate 0x0363df thru 0x039246 (0x2e68 bytes) 000020 1786748104.831695 fallocate 0x011b76 thru 0x01d38d (0xb818 bytes) 000021 1786748104.839801 write 0x013175 thru 0x01f42e (0xc2ba bytes) 000022 1786748104.842630 trunc from 0x0379b9 to 0x03b4b0 (0x3af8 bytes) 000024 1786748104.848705 mapwrite 0x007c9a thru 0x015653 (0xd9ba bytes) 000025 1786748104.878939 mapwrite 0x030154 thru 0x0363f1 (0x629e bytes) 000026 1786748104.906922 read 0x00129c thru 0x01066a (0xf3cf bytes) 000027 1786748104.908717 trunc from 0x0173da to 0x020fda (0x9c01 bytes) 000028 1786748104.920531 write 0x0344e2 thru 0x03ca7c (0x859b bytes) 000030 1786748104.928306 write 0x01d0a9 thru 0x025663 (0x85bb bytes) 000032 1786748104.945716 trunc from 0x030177 to 0x03507f (0x4f09 bytes) 000033 1786748104.973427 mapwrite 0x03cbd9 thru 0x03f46d (0x2895 bytes) 000034 1786748104.998476 punch from 0x024ca5 to 0x027dee (0x314a bytes) 000035 1786748105.005851 mapread 0x0032c5 thru 0x009974 (0x66b0 bytes) 000036 1786748105.008457 mapread 0x010305 thru 0x01de40 (0xdb3c bytes) 000037 1786748105.010994 mapwrite 0x01ecf8 thru 0x02e5b4 (0xf8bd bytes) 000038 1786748105.066931 trunc from 0x040000 to 0x01bef1 (0x2410e bytes) 000039 1786748105.087018 fallocate 0x03b89a thru 0x03ffff (0x4766 bytes) 000041 1786748105.093672 trunc from 0x040000 to 0x0374da (0x8b25 bytes) 000044 1786748105.114627 mapwrite 0x015337 thru 0x024855 (0xf51f bytes) 000045 1786748105.176865 read 0x0203b5 thru 0x024cf1 (0x493d bytes) 000046 1786748105.177861 mapread 0x0331be thru 0x0374da (0x431d bytes) 000047 1786748105.202711 mapread 0x027780 thru 0x02db34 (0x63b5 bytes) 000048 1786748105.205730 mapwrite 0x03e2bf thru 0x03ffff (0x1d41 bytes) 000049 1786748105.255849 fallocate 0x02c7d8 thru 0x02db43 (0x136c bytes) 000050 1786748105.272268 MIRROR_EXTEND to 2 mirrors 000051 1786748106.062735 fallocate 0x000c8e thru 0x00f634 (0xe9a7 bytes) 000052 1786748106.116955 mapwrite 0x00dfad thru 0x0105d7 (0x262b bytes) 000053 1786748106.184063 trunc from 0x040000 to 0x02032d (0x1fcd2 bytes) 000054 1786748106.293099 mapread 0x00fbc7 thru 0x0160be (0x64f8 bytes) 000055 1786748106.296237 read 0x0173d4 thru 0x0185a8 (0x11d5 bytes) 000056 1786748106.302557 trunc from 0x02032e to 0x024933 (0x4606 bytes) 000057 1786748106.346953 mapwrite 0x009ef1 thru 0x00ac17 (0xd27 bytes) 000058 1786748106.380829 mapwrite 0x028775 thru 0x0298df (0x116b bytes) 000059 1786748106.454245 mapread 0x0240b9 thru 0x0298df (0x5827 bytes) 000061 1786748106.459035 trunc from 0x011b8f to 0x01613b (0x45ad bytes) 000062 1786748106.469198 punch from 0x011b0c to 0x018394 (0x6889 bytes) 000063 1786748106.491321 MIRROR_RESYNC 2 mirrors 000064 1786748107.184926 mapread 0x01b874 thru 0x0256b4 (0x9e41 bytes) 000066 1786748107.216693 read 0x023f97 thru 0x0298df (0x5949 bytes) 000067 1786748107.225207 trunc from 0x01ff6d to 0x022318 (0x23ac bytes) 000068 1786748107.277381 write 0x000ff3 thru 0x001d2b (0xd39 bytes) 000070 1786748107.288882 trunc from 0x00e078 to 0x01c65c (0xe5e5 bytes) 000071 1786748107.296588 mapread 0x007e2d thru 0x00f7db (0x79af bytes) 000073 1786748107.308660 mapread 0x006ef4 thru 0x0150a7 (0xe1b4 bytes) 000074 1786748107.325805 punch from 0x008c12 to 0x01588c (0xcc7b bytes) 000075 1786748107.338195 write 0x028f6d thru 0x036770 (0xd804 bytes) 000076 1786748107.348452 mapread 0x026ee5 thru 0x02d685 (0x67a1 bytes) 000077 1786748107.365152 write 0x02aad6 thru 0x037b9c (0xd0c7 bytes) 000078 1786748107.368518 trunc from 0x030818 to 0x037b9c (0x7385 bytes) 000079 1786748107.399751 read 0x033046 thru 0x037b9c (0x4b57 bytes) 000080 1786748107.413834 punch from 0x021d11 to 0x02b9cc (0x9cbc bytes) 000082 1786748107.429803 fallocate 0x0166b7 thru 0x01a042 (0x398c bytes) 000083 1786748107.437946 mapread 0x010b66 thru 0x011980 (0xe1b bytes) 000084 1786748107.440870 mapread 0x012cc2 thru 0x02231e (0xf65d bytes) 000085 1786748107.460675 fallocate 0x00fff9 thru 0x01f4c2 (0xf4ca bytes) 000086 1786748107.469863 MIRROR_RESYNC 2 mirrors 000087 1786748108.172212 fallocate 0x0365c8 thru 0x03d307 (0x6d40 bytes) 000088 1786748108.223274 write 0x025b50 thru 0x0344ab (0xe95c bytes) 000090 1786748108.256246 trunc from 0x00f10b to 0x019297 (0xa18d bytes) 000091 1786748108.272080 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000092 1786748108.574539 write 0x038534 thru 0x03b6a9 (0x3176 bytes) 000093 1786748108.597313 fallocate 0x03454c thru 0x03cff3 (0x8aa8 bytes) 000094 1786748108.606751 fallocate 0x02c6eb thru 0x02f297 (0x2bad bytes) 000095 1786748108.617598 fallocate 0x011041 thru 0x013ff9 (0x2fb9 bytes) 000096 1786748108.630324 trunc from 0x03d308 to 0x0140d3 (0x29234 bytes) 000097 1786748108.691387 write 0x01f800 thru 0x029167 (0x9968 bytes) 000098 1786748108.697543 write 0x038f06 thru 0x03c198 (0x3293 bytes) 000099 1786748108.703336 punch from 0x02704f to 0x02ec12 (0x7bc4 bytes) 000100 1786748108.726088 punch from 0x011d32 to 0x01cc57 (0xaf26 bytes) 000103 1786748108.736225 write 0x01dd0c thru 0x01e6fc (0x9f1 bytes) 000104 1786748108.756866 punch from 0x00af3e to 0x00e9b6 (0x3a79 bytes) 000105 1786748108.774027 trunc from 0x035297 to 0x03c198 (0x6f02 bytes) 000106 1786748108.786923 trunc from 0x03c199 to 0x00c4f0 (0x2fca8 bytes) 000107 1786748108.815337 fallocate 0x008a0d thru 0x00d952 (0x4f46 bytes) 000109 1786748108.820865 fallocate 0x036d06 thru 0x03931c (0x2617 bytes) 000111 1786748108.827339 trunc from 0x0022b9 to 0x00ea54 (0xc79c bytes) 000112 1786748108.834778 punch from 0x02e633 to 0x036c69 (0x8637 bytes) 000113 1786748108.844296 trunc from 0x03931d to 0x0135a5 (0x25d77 bytes) 000114 1786748108.864936 trunc from 0x011e12 to 0x0135a5 (0x1794 bytes) 000115 1786748108.872825 trunc from 0x010c48 to 0x011b10 (0xec9 bytes) 000116 1786748108.879317 mapread 0x00965e thru 0x0135a5 (0x9f48 bytes) 000117 1786748108.916817 trunc from 0x00e327 to 0x0135a5 (0x527f bytes) 000118 1786748108.927760 write 0x01b4c4 thru 0x02021c (0x4d59 bytes) 000119 1786748108.931624 read 0x018e5c thru 0x02021c (0x73c1 bytes) 000120 1786748108.940619 read 0x01fc14 thru 0x02021c (0x609 bytes) 000121 1786748108.941466 punch from 0x0044c9 to 0x00700a (0x2b42 bytes) 000122 1786748108.951922 punch from 0x01d35b to 0x02021c (0x2ec2 bytes) 000124 1786748108.982055 trunc from 0x02021d to 0x0248a9 (0x468d bytes) 000125 1786748109.004460 fallocate 0x007d3f thru 0x0107e2 (0x8aa4 bytes) 000126 1786748109.021330 MIRROR_EXTEND to 2 mirrors 000127 1786748109.731330 mapread 0x00724f thru 0x01342c (0xc1de bytes) 000128 1786748109.772209 mapread 0x010464 thru 0x018d85 (0x8922 bytes) 000130 1786748109.780486 punch from 0x0051d8 to 0x00db86 (0x89af bytes) 000131 1786748109.832394 punch from 0x015594 to 0x016a48 (0x14b5 bytes) 000132 1786748109.838356 read 0x01cd7e thru 0x0248a9 (0x7b2c bytes) 000133 1786748109.848027 trunc from 0x0244d0 to 0x0248a9 (0x3da bytes) 000134 1786748109.869551 mapread 0x004c43 thru 0x011805 (0xcbc3 bytes) 000135 1786748109.889980 trunc from 0x0222f1 to 0x0248a9 (0x25b9 bytes) 000136 1786748109.896770 write 0x01bec2 thru 0x02b00f (0xf14e bytes) 000137 1786748109.911567 fallocate 0x00b86d thru 0x00c7a9 (0xf3d bytes) 000138 1786748109.921778 fallocate 0x02b325 thru 0x035cbe (0xa99a bytes) 000139 1786748109.927942 mapread 0x01e9ef thru 0x02b9c7 (0xcfd9 bytes) 000140 1786748109.929953 fallocate 0x016135 thru 0x022324 (0xc1f0 bytes) 000141 1786748109.941530 mapwrite 0x008a8c thru 0x008cfa (0x26f bytes) 000142 1786748109.969778 mapread 0x01a186 thru 0x02244c (0x82c7 bytes) 000143 1786748109.974533 trunc from 0x035cbf to 0x02f943 (0x637b bytes) 000144 1786748110.000046 trunc from 0x02f944 to 0x01ad7c (0x14bc7 bytes) 000145 1786748110.017715 mapread 0x00119e thru 0x008671 (0x74d4 bytes) 000146 1786748110.019881 write 0x012251 thru 0x021425 (0xf1d5 bytes) 000147 1786748110.022207 write 0x001ec1 thru 0x00a9ea (0x8b2a bytes) 000148 1786748110.024401 trunc from 0x00ba73 to 0x00f36e (0x38fc bytes) 000150 1786748110.031315 fallocate 0x01d37b thru 0x02829a (0xaf20 bytes) 000151 1786748110.043266 read 0x0057cf thru 0x009c25 (0x4457 bytes) 000152 1786748110.044683 trunc from 0x021426 to 0x030706 (0xf2e1 bytes) 000153 1786748110.074206 trunc from 0x030707 to 0x013101 (0x1d605 bytes) 000154 1786748110.098186 mapread 0x00e602 thru 0x013101 (0x4b00 bytes) 000155 1786748110.116595 write 0x025b18 thru 0x02bbda (0x60c3 bytes) 000156 1786748110.118674 mapread 0x00e439 thru 0x01d869 (0xf431 bytes) 000157 1786748110.128798 punch from 0x00e705 to 0x01b4f6 (0xcdf2 bytes) 000158 1786748110.149396 trunc from 0x02bbdb to 0x002bc0 (0x2901a bytes) 000159 1786748110.190540 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000160 1786748110.400096 trunc from 0x002bc1 to 0x01b571 (0x189b1 bytes) 000161 1786748110.458912 trunc from 0x01b572 to 0x037d0f (0x1c79e bytes) 000162 1786748110.506968 fallocate 0x038e3f thru 0x03ffff (0x71c1 bytes) 000163 1786748110.525616 read 0x0282f2 thru 0x033785 (0xb494 bytes) 000164 1786748110.553083 mapread 0x019e01 thru 0x01d95b (0x3b5b bytes) 000167 1786748110.611879 trunc from 0x01a4b8 to 0x0240ce (0x9c17 bytes) 000168 1786748110.621692 mapread 0x032d0d thru 0x037d0f (0x5003 bytes) 000169 1786748110.641081 read 0x02f084 thru 0x037d0f (0x8c8c bytes) 000170 1786748110.642490 mapread 0x030b27 thru 0x037d0f (0x71e9 bytes) 000172 1786748110.655308 mapwrite 0x00d979 thru 0x012e32 (0x54ba bytes) 000173 1786748110.697538 read 0x01d49d thru 0x02748a (0x9fee bytes) 000174 1786748110.698856 read 0x027dee thru 0x034c8a (0xce9d bytes) 000175 1786748110.701982 punch from 0x000ee8 to 0x00f404 (0xe51d bytes) 000176 1786748110.716415 trunc from 0x018329 to 0x0244ca (0xc1a2 bytes) 000177 1786748110.732886 MIRROR_EXTEND to 2 mirrors 000178 1786748111.921612 punch from 0x005845 to 0x013fe8 (0xe7a4 bytes) 000179 1786748111.959180 read 0x01f797 thru 0x02addc (0xb646 bytes) 000180 1786748111.970961 fallocate 0x0244e8 thru 0x02aa7e (0x6597 bytes) 000181 1786748111.977765 punch from 0x025189 to 0x02dcdd (0x8b55 bytes) 000182 1786748111.984818 mapread 0x0129c1 thru 0x0214ab (0xeaeb bytes) 000184 1786748112.012619 punch from 0x03303a to 0x037d0f (0x4cd6 bytes) 000185 1786748112.020704 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000186 1786748112.250387 MIRROR_EXTEND to 2 mirrors 000187 1786748113.089509 mapread 0x015196 thru 0x021beb (0xca56 bytes) 000188 1786748113.140690 trunc from 0x037d10 to 0x00d5a6 (0x2a769 bytes) 000189 1786748113.220416 mapwrite 0x02a918 thru 0x02cded (0x24d6 bytes) 000190 1786748113.286605 fallocate 0x02c4a9 thru 0x03505c (0x8bb4 bytes) 000191 1786748113.297110 write 0x012bf9 thru 0x017ebe (0x52c6 bytes) 000192 1786748113.322782 trunc from 0x029033 to 0x02cded (0x3dbb bytes) 000193 1786748113.353587 fallocate 0x01af7b thru 0x02714c (0xc1d2 bytes) 000194 1786748113.369571 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2cdee): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cdee): Mirror 2: 0x62cdad45 (stale) CRC-32 checksum value for chunk [0, 0x2cdee): Mirror 1: 0x170d6ab Mirror 3: 0x170d6ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cdee: skipped 000195 1786748114.321427 mapread 0x00de82 thru 0x0146ca (0x6849 bytes) 000196 1786748114.371077 read 0x011336 thru 0x0132eb (0x1fb6 bytes) 000197 1786748114.371697 read 0x009609 thru 0x0162a2 (0xcc9a bytes) 000198 1786748114.373855 trunc from 0x02cdee to 0x02cfa3 (0x1b6 bytes) 000200 1786748114.483185 mapread 0x00b047 thru 0x011d08 (0x6cc2 bytes) 000201 1786748114.496064 mapread 0x00bc80 thru 0x0126d8 (0x6a59 bytes) 000202 1786748114.497718 read 0x00c05c thru 0x00c247 (0x1ec bytes) 000203 1786748114.506475 MIRROR_RESYNC 3 mirrors 000204 1786748115.596726 write 0x0080d8 thru 0x00d350 (0x5279 bytes) 000205 1786748115.659605 mapwrite 0x012fa0 thru 0x016643 (0x36a4 bytes) 000206 1786748115.708224 mapread 0x00b30c thru 0x0143c5 (0x90ba bytes) 000207 1786748115.711855 write 0x01480b thru 0x01eb30 (0xa326 bytes) 000208 1786748115.720715 read 0x014e1c thru 0x0200d9 (0xb2be bytes) 000209 1786748115.726975 punch from 0x025163 to 0x029d12 (0x4bb0 bytes) 000210 1786748115.733167 trunc from 0x02bdf7 to 0x02cbba (0xdc4 bytes) 000211 1786748115.738235 trunc from 0x02cfa4 to 0x00f65b (0x1d948 bytes) 000212 1786748115.776377 fallocate 0x00d83b thru 0x016706 (0x8ecc bytes) 000213 1786748115.793514 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xf65c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf65c): Mirror 2: 0x649355d4 (stale) Mirror 3: 0x649355d4 (stale) CRC-32 checksum value for chunk [0, 0xf65c): Mirror 1: 0x376cecc7 Mirror 4: 0x376cecc7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf65c: skipped 000214 1786748116.865771 mapwrite 0x036eb1 thru 0x03e373 (0x74c3 bytes) 000215 1786748117.018471 write 0x02c875 thru 0x03bb85 (0xf311 bytes) 000216 1786748117.025819 punch from 0x01ae41 to 0x01e820 (0x39e0 bytes) 000217 1786748117.031801 mapread 0x02e324 thru 0x03d550 (0xf22d bytes) 000218 1786748117.034728 punch from 0x01d44a to 0x028ac8 (0xb67f bytes) 000219 1786748117.046744 mapread 0x00f55a thru 0x01392b (0x43d2 bytes) 000220 1786748117.066613 write 0x000aad thru 0x00fa50 (0xefa4 bytes) 000221 1786748117.094576 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e374): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e374): Mirror 2: 0x8b399d80 (stale) Mirror 3: 0x8b399d80 (stale) Mirror 4: 0x6f0ebb3 (stale) CRC-32 checksum value for chunk [0, 0x3e374): Mirror 1: 0xc0660708 Mirror 5: 0xc0660708 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e374: skipped 000222 1786748118.126354 write 0x01bc3f thru 0x0212e0 (0x56a2 bytes) 000223 1786748118.219557 punch from 0x00c1ed to 0x00d120 (0xf34 bytes) 000224 1786748118.228542 trunc from 0x03e374 to 0x03540a (0x8f69 bytes) 000225 1786748118.308487 trunc from 0x02f2b8 to 0x03540a (0x6153 bytes) 000226 1786748118.323471 MIRROR_RESYNC 5 mirrors 000227 1786748119.994371 MIRROR_RESYNC 5 mirrors 000229 1786748120.359795 mapwrite 0x02e81f thru 0x03046d (0x1c4f bytes) 000230 1786748120.492935 fallocate 0x00195a thru 0x001a27 (0xce bytes) 000231 1786748120.498438 read 0x0251f8 thru 0x02b42f (0x6238 bytes) 000232 1786748120.504796 trunc from 0x03540b to 0x01c2dd (0x1912d bytes) 000233 1786748120.524487 mapwrite 0x021712 thru 0x0265fa (0x4ee9 bytes) 000234 1786748120.566272 fallocate 0x037547 thru 0x03e18c (0x6c46 bytes) 000235 1786748120.571891 mapread 0x005e3e thru 0x00b5b8 (0x577b bytes) 000236 1786748120.582369 read 0x011079 thru 0x0125dc (0x1564 bytes) 000237 1786748120.608527 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e18d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e18d): Mirror 2: 0x24301516 (stale) Mirror 3: 0x24301516 (stale) Mirror 4: 0x24301516 (stale) Mirror 5: 0x24301516 (stale) CRC-32 checksum value for chunk [0, 0x3e18d): Mirror 1: 0x17b92041 Mirror 6: 0x17b92041 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e18d: skipped 000238 1786748121.562249 write 0x01b880 thru 0x01cf85 (0x1706 bytes) 000239 1786748121.612812 mapwrite 0x01c646 thru 0x027bea (0xb5a5 bytes) 000240 1786748121.657466 fallocate 0x01af85 thru 0x01f6dd (0x4759 bytes) 000241 1786748121.662617 punch from 0x016bd4 to 0x01efae (0x83db bytes) 000242 1786748121.668888 punch from 0x02d1ae to 0x03aff5 (0xde48 bytes) 000243 1786748121.674398 mapwrite 0x02af37 thru 0x036fbf (0xc089 bytes) 000244 1786748121.719361 write 0x024051 thru 0x02a38b (0x633b bytes) 000245 1786748121.721788 mapread 0x014908 thru 0x01da24 (0x911d bytes) 000246 1786748121.747324 mapread 0x00fcb5 thru 0x01ec28 (0xef74 bytes) 000247 1786748121.752660 write 0x0011ce thru 0x006958 (0x578b bytes) 000248 1786748121.763541 trunc from 0x004afe to 0x00e8ba (0x9dbd bytes) 000249 1786748121.779514 fallocate 0x036c8b thru 0x03b6f8 (0x4a6e bytes) 000250 1786748121.784716 write 0x00356a thru 0x00d5bc (0xa053 bytes) 000251 1786748121.794771 write 0x0353b1 thru 0x03c281 (0x6ed1 bytes) 000252 1786748121.803483 trunc from 0x01e374 to 0x027727 (0x93b4 bytes) 000253 1786748121.814457 fallocate 0x030b9a thru 0x039212 (0x8679 bytes) 000254 1786748121.821452 punch from 0x031527 to 0x039caa (0x8784 bytes) 000255 1786748121.857950 mapwrite 0x03084f thru 0x0385aa (0x7d5c bytes) 000256 1786748121.906769 trunc from 0x03e18d to 0x02046a (0x1dd22 bytes) 000258 1786748121.947321 punch from 0x01fcac to 0x0202e1 (0x636 bytes) 000259 1786748121.956448 write 0x019320 thru 0x027fab (0xec8c bytes) 000260 1786748121.958724 punch from 0x00d618 to 0x011a3d (0x4426 bytes) 000261 1786748121.966282 mapread 0x00a3b8 thru 0x00a913 (0x55c bytes) 000262 1786748121.967369 trunc from 0x027fac to 0x01828f (0xfd1c bytes) 000263 1786748122.009369 MIRROR_RESYNC 6 mirrors 000264 1786748123.048447 trunc from 0x018290 to 0x029af1 (0x11862 bytes) 000265 1786748123.125387 mapwrite 0x01df2e thru 0x027ccc (0x9d9f bytes) 000266 1786748123.186552 mapread 0x00045b thru 0x003b03 (0x36a9 bytes) 000268 1786748123.196910 mapwrite 0x0116fc thru 0x018ac5 (0x73ca bytes) 000269 1786748123.225255 mapread 0x017cdc thru 0x01c3db (0x4700 bytes) 000270 1786748123.232621 mapread 0x00c8ca thru 0x01657b (0x9cb2 bytes) 000271 1786748123.234539 punch from 0x024ba7 to 0x027d54 (0x31ae bytes) 000274 1786748123.255970 read 0x00a4ef thru 0x00baec (0x15fe bytes) 000275 1786748123.256729 trunc from 0x022afd to 0x029af1 (0x6ff5 bytes) 000276 1786748123.281052 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29af2): Mirror 2: 0xc2ee5523 (stale) Mirror 3: 0xc2ee5523 (stale) Mirror 5: 0xc2ee5523 (stale) Mirror 6: 0xc2ee5523 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29af2): Mirror 2: 0xc2ee5523 (stale) Mirror 3: 0xc2ee5523 (stale) Mirror 5: 0xc2ee5523 (stale) Mirror 6: 0xc2ee5523 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29af2: skipped 000277 1786748123.760394 fallocate 0x03bcfb thru 0x03ffff (0x4305 bytes) 000278 1786748123.766878 trunc from 0x040000 to 0x01e15b (0x21ea4 bytes) 000279 1786748123.791954 trunc from 0x0114e3 to 0x019998 (0x84b6 bytes) 000280 1786748123.807912 MIRROR_RESYNC 5 mirrors 000281 1786748124.861490 punch from 0x01c308 to 0x01df7e (0x1c77 bytes) 000282 1786748124.904721 mapwrite 0x01a3a4 thru 0x01aed6 (0xb33 bytes) 000284 1786748124.921489 mapread 0x00b1d8 thru 0x016d15 (0xbb3e bytes) 000285 1786748124.946808 punch from 0x0122ce to 0x01e15b (0xbe8e bytes) 000286 1786748124.952202 trunc from 0x007f1e to 0x015f2c (0xe00f bytes) 000287 1786748124.958522 trunc from 0x01e15c to 0x012cb3 (0xb4a8 bytes) 000288 1786748124.979540 fallocate 0x00d1a3 thru 0x01721e (0xa07c bytes) 000289 1786748124.986110 mapread 0x00203c thru 0x00628d (0x4252 bytes) 000290 1786748124.997207 trunc from 0x005e12 to 0x00a6bd (0x48ac bytes) 000291 1786748125.001620 mapwrite 0x037647 thru 0x03ffff (0x89b9 bytes) 000292 1786748125.044047 trunc from 0x030da7 to 0x030edc (0x136 bytes) 000293 1786748125.050306 mapwrite 0x00dbae thru 0x01aa7c (0xcecf bytes) 000294 1786748125.119707 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x57d512e0 (stale) Mirror 3: 0x57d512e0 (stale) Mirror 6: 0x57d512e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x57d512e0 (stale) Mirror 3: 0x57d512e0 (stale) Mirror 6: 0x57d512e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000295 1786748125.487026 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x57d512e0 (stale) Mirror 6: 0x57d512e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x57d512e0 (stale) Mirror 6: 0x57d512e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000296 1786748125.839788 mapwrite 0x0174fa thru 0x019967 (0x246e bytes) 000298 1786748125.871235 punch from 0x03aee3 to 0x03ffff (0x511d bytes) 000299 1786748125.875611 fallocate 0x023a02 thru 0x0326a5 (0xeca4 bytes) 000300 1786748125.880766 mapread 0x01155b thru 0x013196 (0x1c3c bytes) 000301 1786748125.890311 read 0x009657 thru 0x013a4f (0xa3f9 bytes) 000302 1786748125.895325 read 0x0023f6 thru 0x003726 (0x1331 bytes) 000303 1786748125.904806 trunc from 0x040000 to 0x02b9f3 (0x1460c bytes) 000304 1786748125.928212 mapwrite 0x022c0b thru 0x02fe26 (0xd21c bytes) 000305 1786748125.987708 trunc from 0x02fe27 to 0x02f557 (0x8cf bytes) 000306 1786748126.007192 write 0x00ab33 thru 0x013233 (0x8701 bytes) 000307 1786748126.014663 mapread 0x00bc4d thru 0x013001 (0x73b5 bytes) 000308 1786748126.016561 trunc from 0x00bf3e to 0x0165f0 (0xa6b3 bytes) 000309 1786748126.032357 fallocate 0x02e58c thru 0x02fd98 (0x180d bytes) 000311 1786748126.039062 punch from 0x00aeed to 0x0106da (0x57ee bytes) 000312 1786748126.045114 mapwrite 0x01a5c5 thru 0x020fa8 (0x69e4 bytes) 000313 1786748126.063613 read 0x02411a thru 0x02f557 (0xb43e bytes) 000315 1786748126.064838 mapwrite 0x036b3d thru 0x03ffff (0x94c3 bytes) 000316 1786748126.112371 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x57d512e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x57d512e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000317 1786748126.343628 write 0x027037 thru 0x036639 (0xf603 bytes) 000318 1786748126.357567 punch from 0x02a9cb to 0x0383dd (0xda13 bytes) 000319 1786748126.378533 punch from 0x00138c to 0x001843 (0x4b8 bytes) 000320 1786748126.389515 MIRROR_RESYNC 2 mirrors 000321 1786748127.389735 write 0x03b521 thru 0x03ffff (0x4adf bytes) 000322 1786748127.451094 punch from 0x03dfba to 0x03ffff (0x2046 bytes) 000323 1786748127.479303 mapread 0x02ad3b thru 0x0368b2 (0xbb78 bytes) 000324 1786748127.558937 read 0x00723d thru 0x016f02 (0xfcc6 bytes) 000325 1786748127.573511 mapwrite 0x00ce97 thru 0x010077 (0x31e1 bytes) 000326 1786748127.600268 mapread 0x02b66e thru 0x03240c (0x6d9f bytes) 000327 1786748127.602075 mapread 0x028899 thru 0x031722 (0x8e8a bytes) 000328 1786748127.605218 punch from 0x031aa4 to 0x035216 (0x3773 bytes) 000329 1786748127.621800 read 0x03ada1 thru 0x03ffff (0x525f bytes) 000330 1786748127.640318 trunc from 0x040000 to 0x02ac73 (0x1538c bytes) 000331 1786748127.667232 write 0x013244 thru 0x01b91a (0x86d7 bytes) 000332 1786748127.669744 read 0x011300 thru 0x01afdf (0x9ce0 bytes) 000333 1786748127.676758 MIRROR_RESYNC 2 mirrors 000334 1786748128.513314 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000335 1786748128.721586 trunc from 0x02ac74 to 0x00e9a2 (0x1c2d1 bytes) 000336 1786748128.775709 MIRROR_EXTEND to 2 mirrors 000338 1786748129.490615 trunc from 0x008446 to 0x00e9a2 (0x655d bytes) 000339 1786748129.529575 fallocate 0x0145aa thru 0x01657f (0x1fd6 bytes) 000342 1786748129.536684 write 0x039cf6 thru 0x03ffff (0x630a bytes) 000344 1786748129.545852 write 0x02e662 thru 0x031454 (0x2df3 bytes) 000345 1786748129.555284 trunc from 0x040000 to 0x0393ec (0x6c13 bytes) 000346 1786748129.582310 fallocate 0x01a4d8 thru 0x028be9 (0xe712 bytes) 000347 1786748129.593883 trunc from 0x01079b to 0x0112eb (0xb51 bytes) 000349 1786748129.604883 punch from 0x0139a8 to 0x0166b7 (0x2d10 bytes) 000350 1786748129.617050 MIRROR_RESYNC 2 mirrors 000352 1786748130.368554 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000353 1786748130.525803 trunc from 0x0393ed to 0x010b11 (0x288db bytes) 000354 1786748130.564163 trunc from 0x010b12 to 0x02351b (0x12a0a bytes) 000355 1786748130.581272 read 0x0150fd thru 0x015a56 (0x95a bytes) 000356 1786748130.587594 mapwrite 0x028fff thru 0x02f1d8 (0x61da bytes) 000357 1786748130.657946 fallocate 0x00fdd9 thru 0x0175e6 (0x780e bytes) 000358 1786748130.665695 trunc from 0x021cef to 0x02dd94 (0xc0a6 bytes) 000359 1786748130.674701 punch from 0x016342 to 0x0187c3 (0x2482 bytes) 000360 1786748130.680658 punch from 0x01259b to 0x01f7fb (0xd261 bytes) 000361 1786748130.688845 read 0x019615 thru 0x0225f3 (0x8fdf bytes) 000362 1786748130.705325 trunc from 0x02a23c to 0x02ed55 (0x4b1a bytes) 000363 1786748130.718741 write 0x000930 thru 0x002bb4 (0x2285 bytes) 000364 1786748130.719875 write 0x0066fd thru 0x0164d3 (0xfdd7 bytes) 000365 1786748130.723815 fallocate 0x0037af thru 0x00caee (0x9340 bytes) 000366 1786748130.732509 trunc from 0x02f1d9 to 0x024b1f (0xa6b9 bytes) 000368 1786748130.759750 trunc from 0x01bda3 to 0x0249e7 (0x8c45 bytes) 000369 1786748130.765316 trunc from 0x024b20 to 0x004f19 (0x1fc06 bytes) 000370 1786748130.784452 fallocate 0x00a84a thru 0x016d66 (0xc51d bytes) 000371 1786748130.789246 punch from 0x006fe2 to 0x0117bf (0xa7de bytes) 000372 1786748130.794308 write 0x00eaa9 thru 0x01a8fd (0xbe55 bytes) 000373 1786748130.800691 write 0x018c29 thru 0x022241 (0x9619 bytes) 000375 1786748130.803771 write 0x0162a6 thru 0x021515 (0xb270 bytes) 000377 1786748130.811400 read 0x00e3de thru 0x016792 (0x83b5 bytes) 000378 1786748130.816319 fallocate 0x01047b thru 0x019b46 (0x96cc bytes) 000379 1786748130.822651 mapread 0x021ee1 thru 0x022241 (0x361 bytes) 000380 1786748130.824085 mapwrite 0x006d56 thru 0x011464 (0xa70f bytes) 000381 1786748130.856418 fallocate 0x004613 thru 0x00f5a0 (0xaf8e bytes) 000382 1786748130.865816 write 0x039025 thru 0x03ffff (0x6fdb bytes) 000384 1786748130.867662 trunc from 0x040000 to 0x02d1bb (0x12e44 bytes) 000385 1786748130.889654 punch from 0x0002b2 to 0x0038c7 (0x3616 bytes) 000386 1786748130.894463 trunc from 0x02d1bc to 0x0074af (0x25d0c bytes) 000387 1786748130.913470 trunc from 0x0074b0 to 0x025838 (0x1e389 bytes) 000388 1786748130.944218 fallocate 0x021a09 thru 0x02471f (0x2d17 bytes) 000389 1786748130.954480 trunc from 0x008c60 to 0x015a3a (0xcddb bytes) 000390 1786748130.961677 fallocate 0x024358 thru 0x02457e (0x227 bytes) 000391 1786748130.970721 trunc from 0x025839 to 0x02d1dd (0x79a5 bytes) 000392 1786748131.008627 punch from 0x01cefa to 0x02a1fc (0xd303 bytes) 000393 1786748131.015794 trunc from 0x02d1de to 0x02a2b1 (0x2f2c bytes) 000394 1786748131.032074 write 0x002b6b thru 0x005e20 (0x32b6 bytes) 000395 1786748131.041588 write 0x032eb7 thru 0x03bcfb (0x8e45 bytes) 000396 1786748131.043859 read 0x001374 thru 0x00e5e9 (0xd276 bytes) 000397 1786748131.064511 mapread 0x01539f thru 0x01da28 (0x868a bytes) 000398 1786748131.066700 mapwrite 0x021132 thru 0x02ac40 (0x9b0f bytes) 000399 1786748131.101838 fallocate 0x017efe thru 0x0275e8 (0xf6eb bytes) 000401 1786748131.121697 trunc from 0x00ac55 to 0x0109c2 (0x5d6e bytes) 000402 1786748131.127106 punch from 0x02c383 to 0x02f19c (0x2e1a bytes) 000403 1786748131.140658 mapread 0x01649c thru 0x0218dd (0xb442 bytes) 000404 1786748131.148460 mapread 0x024df8 thru 0x033aa7 (0xecb0 bytes) 000405 1786748131.189414 read 0x0025bc thru 0x00d93f (0xb384 bytes) 000406 1786748131.201679 mapread 0x0098e5 thru 0x012365 (0x8a81 bytes) 000407 1786748131.211770 read 0x0010f6 thru 0x001109 (0x14 bytes) 000408 1786748131.216724 read 0x03172e thru 0x0374c7 (0x5d9a bytes) 000409 1786748131.217604 read 0x002fc3 thru 0x0064ec (0x352a bytes) 000410 1786748131.225804 trunc from 0x03bcfc to 0x02f0bd (0xcc3e bytes) 000411 1786748131.277812 fallocate 0x01b53d thru 0x025706 (0xa1ca bytes) 000412 1786748131.292016 write 0x02c81f thru 0x032fbb (0x679d bytes) 000413 1786748131.297759 trunc from 0x032fbc to 0x009970 (0x2964b bytes) 000414 1786748131.337861 fallocate 0x039812 thru 0x03ef8d (0x577c bytes) 000415 1786748131.346753 punch from 0x038d46 to 0x03ef8d (0x6248 bytes) 000416 1786748131.356746 fallocate 0x01557e thru 0x017b0f (0x2592 bytes) 000417 1786748131.370226 mapread 0x019656 thru 0x0254a6 (0xbe51 bytes) 000418 1786748131.390218 trunc from 0x03ef8e to 0x03fbd0 (0xc43 bytes) 000419 1786748131.412664 read 0x0237f8 thru 0x02a40a (0x6c13 bytes) 000421 1786748131.413973 write 0x00ce00 thru 0x00ef98 (0x2199 bytes) 000422 1786748131.415532 punch from 0x008ab6 to 0x013fbd (0xb508 bytes) 000423 1786748131.432806 write 0x005777 thru 0x013139 (0xd9c3 bytes) 000426 1786748131.443421 mapread 0x0330ac thru 0x03fbd0 (0xcb25 bytes) 000427 1786748131.450767 punch from 0x0139da to 0x01400c (0x633 bytes) 000428 1786748131.469499 mapread 0x02aa57 thru 0x0345a6 (0x9b50 bytes) 000429 1786748131.474856 MIRROR_EXTEND to 2 mirrors 000430 1786748132.272694 fallocate 0x02cd24 thru 0x037112 (0xa3ef bytes) 000431 1786748132.343075 read 0x00e61a thru 0x019c7f (0xb666 bytes) 000432 1786748132.353454 mapread 0x000237 thru 0x005c79 (0x5a43 bytes) 000433 1786748132.379503 trunc from 0x03fbd1 to 0x03e0f3 (0x1add bytes) 000434 1786748132.432312 mapread 0x0263fb thru 0x026c46 (0x84c bytes) 000435 1786748132.441832 punch from 0x013263 to 0x01c571 (0x930f bytes) 000436 1786748132.447090 read 0x02ad4a thru 0x039e03 (0xf0ba bytes) 000437 1786748132.457518 mapread 0x02d74f thru 0x03bd20 (0xe5d2 bytes) 000438 1786748132.459065 mapread 0x02efc9 thru 0x03d63b (0xe673 bytes) 000439 1786748132.467261 mapread 0x032789 thru 0x037a66 (0x52de bytes) 000440 1786748132.469126 mapwrite 0x01d744 thru 0x0298f2 (0xc1af bytes) 000441 1786748132.496683 punch from 0x0067cc to 0x007472 (0xca7 bytes) 000442 1786748132.502747 write 0x020512 thru 0x02ca21 (0xc510 bytes) 000443 1786748132.504681 write 0x00c4e1 thru 0x01b61f (0xf13f bytes) 000444 1786748132.507644 punch from 0x017b67 to 0x01ca2f (0x4ec9 bytes) 000445 1786748132.522678 punch from 0x02b0dd to 0x02c59d (0x14c1 bytes) 000446 1786748132.533387 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000447 1786748132.674051 mapwrite 0x030f6b thru 0x0330dc (0x2172 bytes) 000448 1786748132.713170 mapread 0x001229 thru 0x006bf5 (0x59cd bytes) 000449 1786748132.724447 trunc from 0x001beb to 0x002404 (0x81a bytes) 000451 1786748132.731657 punch from 0x02ce92 to 0x03ac74 (0xdde3 bytes) 000452 1786748132.738282 trunc from 0x03e0f4 to 0x02b9a4 (0x1274f bytes) 000453 1786748132.763650 fallocate 0x03b4c3 thru 0x03ffff (0x4b3d bytes) 000454 1786748132.770670 read 0x026e88 thru 0x03302f (0xc1a8 bytes) 000456 1786748132.787037 punch from 0x018430 to 0x025ee3 (0xdab4 bytes) 000457 1786748132.794358 trunc from 0x001b1d to 0x0074da (0x59be bytes) 000458 1786748132.799755 mapread 0x013d3d thru 0x01fdef (0xc0b3 bytes) 000459 1786748132.824709 read 0x032094 thru 0x03ffff (0xdf6c bytes) 000460 1786748132.834972 trunc from 0x01f1c0 to 0x02dd35 (0xeb76 bytes) 000461 1786748132.842008 mapread 0x0376c9 thru 0x03f0ff (0x7a37 bytes) 000462 1786748132.843827 trunc from 0x00c3c2 to 0x016836 (0xa475 bytes) 000463 1786748132.850428 write 0x032e1b thru 0x03f44f (0xc635 bytes) 000465 1786748132.854500 mapwrite 0x007a90 thru 0x01074a (0x8cbb bytes) 000466 1786748132.906385 read 0x018731 thru 0x024932 (0xc202 bytes) 000467 1786748132.942733 MIRROR_EXTEND to 2 mirrors 000468 1786748133.890571 mapread 0x0338c0 thru 0x03ffff (0xc740 bytes) 000469 1786748133.923931 MIRROR_EXTEND to 3 mirrors 000471 1786748134.818660 mapread 0x015dd5 thru 0x0236cb (0xd8f7 bytes) 000472 1786748134.854857 write 0x03a00c thru 0x03ffff (0x5ff4 bytes) 000474 1786748134.919702 write 0x02c802 thru 0x039c9b (0xd49a bytes) 000475 1786748134.951683 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8c33a7cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8c33a7cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000476 1786748135.306454 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8c33a7cc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x68f9ce0b Mirror 4: 0x68f9ce0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000477 1786748136.494549 trunc from 0x040000 to 0x031237 (0xedc8 bytes) 000478 1786748136.555340 trunc from 0x00b327 to 0x016a6a (0xb744 bytes) 000479 1786748136.562252 read 0x01a42b thru 0x029c25 (0xf7fb bytes) 000480 1786748136.572663 mapread 0x015292 thru 0x01d966 (0x86d5 bytes) 000481 1786748136.607856 mapread 0x006c70 thru 0x00ec35 (0x7fc6 bytes) 000482 1786748136.633120 read 0x004d42 thru 0x011f75 (0xd234 bytes) 000483 1786748136.634082 mapread 0x014403 thru 0x0169d3 (0x25d1 bytes) 000484 1786748136.636366 trunc from 0x02a8c8 to 0x031237 (0x6970 bytes) 000486 1786748136.644639 read 0x01acde thru 0x01eb41 (0x3e64 bytes) 000487 1786748136.645505 mapwrite 0x021f1c thru 0x0312b9 (0xf39e bytes) 000488 1786748136.723535 mapwrite 0x02d114 thru 0x033f72 (0x6e5f bytes) 000489 1786748136.770780 mapwrite 0x017782 thru 0x01a99d (0x321c bytes) 000490 1786748136.794520 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33f73): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33f73): Mirror 3: 0xa08e303b (stale) Mirror 4: 0x2219d528 (stale) CRC-32 checksum value for chunk [0, 0x33f73): Mirror 1: 0xb8332c60 Mirror 5: 0xb8332c60 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33f73: skipped 000491 1786748137.991111 mapwrite 0x0298b7 thru 0x02a9a6 (0x10f0 bytes) 000492 1786748138.102826 trunc from 0x033f73 to 0x010de5 (0x2318d bytes) 000493 1786748138.149486 write 0x0156ec thru 0x015c65 (0x57a bytes) 000494 1786748138.150865 trunc from 0x00ce15 to 0x00f92d (0x2b19 bytes) 000495 1786748138.164419 mapwrite 0x002ed1 thru 0x00a6b8 (0x77e8 bytes) 000496 1786748138.229521 mapread 0x013597 thru 0x015c65 (0x26cf bytes) 000498 1786748138.242920 punch from 0x002b10 to 0x006863 (0x3d54 bytes) 000499 1786748138.265809 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x15c66): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15c66): Mirror 3: 0x23fc3051 (stale) Mirror 4: 0x23fc3051 (stale) Mirror 5: 0x3d6d5d5 (stale) CRC-32 checksum value for chunk [0, 0x15c66): Mirror 1: 0x7c5f19e2 Mirror 6: 0x7c5f19e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15c66: skipped 000500 1786748139.210984 mapread 0x00fe52 thru 0x015c65 (0x5e14 bytes) 000501 1786748139.229315 trunc from 0x0110d7 to 0x012caf (0x1bd9 bytes) 000503 1786748139.288874 punch from 0x011948 to 0x015c65 (0x431e bytes) 000504 1786748139.305105 mapwrite 0x00fc73 thru 0x013542 (0x38d0 bytes) 000505 1786748139.364018 trunc from 0x015c66 to 0x03098a (0x1ad25 bytes) 000506 1786748139.411271 punch from 0x0163fa to 0x01a945 (0x454c bytes) 000507 1786748139.416795 read 0x00c37b thru 0x0158ea (0x9570 bytes) 000508 1786748139.423009 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3098b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3098b): Mirror 3: 0xc21a337e (stale) Mirror 4: 0xa87d1d17 (stale) Mirror 5: 0x6e7fa533 (stale) Mirror 6: 0xc6d3ab54 (stale) CRC-32 checksum value for chunk [0, 0x3098b): Mirror 1: 0x5694625 Mirror 7: 0x5694625 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3098b: skipped 000509 1786748140.394059 mapread 0x0224cf thru 0x02ebbd (0xc6ef bytes) 000510 1786748140.420420 read 0x005e87 thru 0x0106e2 (0xa85c bytes) 000511 1786748140.431670 punch from 0x0058d6 to 0x01322d (0xd958 bytes) 000513 1786748140.483489 read 0x0040ba thru 0x00b77e (0x76c5 bytes) 000514 1786748140.513548 trunc from 0x03098b to 0x02e5e4 (0x23a6 bytes) 000515 1786748140.558523 fallocate 0x001c93 thru 0x00204f (0x3bd bytes) 000516 1786748140.566409 mapwrite 0x021141 thru 0x027163 (0x6023 bytes) 000517 1786748140.615305 punch from 0x01f20c to 0x02ac97 (0xba8c bytes) 000519 1786748140.627663 trunc from 0x02e5e5 to 0x039f54 (0xb970 bytes) 000522 1786748140.667163 trunc from 0x039f55 to 0x01276f (0x277e5 bytes) 000523 1786748140.690681 trunc from 0x008906 to 0x0110d7 (0x87d2 bytes) 000524 1786748140.697276 fallocate 0x02f576 thru 0x038da9 (0x9834 bytes) 000525 1786748140.713565 trunc from 0x038daa to 0x022a7e (0x1632b bytes) 000526 1786748140.739666 mapwrite 0x0038ac thru 0x010949 (0xd09e bytes) 000527 1786748140.849684 read 0x01442d thru 0x020fa1 (0xcb75 bytes) 000528 1786748140.898966 punch from 0x004f3b to 0x014868 (0xf92e bytes) 000529 1786748140.909664 read 0x014e38 thru 0x0223be (0xd587 bytes) 000530 1786748140.932108 write 0x00129a thru 0x00bb69 (0xa8d0 bytes) 000531 1786748140.938304 mapwrite 0x016342 thru 0x023534 (0xd1f3 bytes) 000532 1786748140.994609 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x23535): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23535): Mirror 3: 0x8bcef12a (stale) Mirror 4: 0x8bcef12a (stale) Mirror 5: 0xf9129b73 (stale) Mirror 6: 0x42054079 (stale) Mirror 7: 0x53770490 (stale) CRC-32 checksum value for chunk [0, 0x23535): Mirror 1: 0x8f7bb0e2 Mirror 8: 0x8f7bb0e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23535: skipped 000533 1786748142.079243 write 0x012e3f thru 0x01e816 (0xb9d8 bytes) 000535 1786748142.135294 mapwrite 0x03206e thru 0x03ba94 (0x9a27 bytes) 000536 1786748142.212021 mapwrite 0x03a696 thru 0x03ffff (0x596a bytes) 000537 1786748142.280076 mapwrite 0x01bf19 thru 0x01d90d (0x19f5 bytes) 000538 1786748142.321366 trunc from 0x0351f5 to 0x03ffff (0xae0b bytes) 000539 1786748142.342262 mapwrite 0x03e529 thru 0x03ffff (0x1ad7 bytes) 000540 1786748142.410211 punch from 0x00d299 to 0x017ea8 (0xac10 bytes) 000541 1786748142.418548 punch from 0x03bf6c to 0x03ffff (0x4094 bytes) 000542 1786748142.429957 read 0x00e513 thru 0x01bf72 (0xda60 bytes) 000543 1786748142.457386 write 0x02a886 thru 0x02b35f (0xada bytes) 000544 1786748142.486080 trunc from 0x0015b0 to 0x00f861 (0xe2b2 bytes) 000545 1786748142.502670 fallocate 0x03e9ab thru 0x03ffff (0x1655 bytes) 000546 1786748142.521248 trunc from 0x040000 to 0x015ae4 (0x2a51b bytes) 000548 1786748142.556220 fallocate 0x01d09b thru 0x02c1b9 (0xf11f bytes) 000549 1786748142.572253 read 0x00ef1d thru 0x015ae4 (0x6bc8 bytes) 000550 1786748142.592751 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x15ae5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15ae5): Mirror 3: 0x1143d8b3 (stale) Mirror 4: 0x1143d8b3 (stale) Mirror 5: 0x9002a017 (stale) Mirror 6: 0x55c94acb (stale) Mirror 7: 0x22e59179 (stale) Mirror 8: 0x370d2e2 (stale) CRC-32 checksum value for chunk [0, 0x15ae5): Mirror 1: 0x9a553b4d Mirror 9: 0x9a553b4d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15ae5: skipped 000551 1786748143.781740 mapread 0x002d0f thru 0x00e14c (0xb43e bytes) 000552 1786748143.802991 trunc from 0x015ae5 to 0x03f050 (0x2956c bytes) 000553 1786748143.847165 punch from 0x00d1ff to 0x0122af (0x50b1 bytes) 000554 1786748143.857373 MIRROR_RESYNC 8 mirrors 000555 1786748145.179295 trunc from 0x03f051 to 0x030d9c (0xe2b4 bytes) 000556 1786748145.272461 punch from 0x0183e6 to 0x022128 (0x9d43 bytes) 000557 1786748145.283330 mapwrite 0x0246b4 thru 0x03425c (0xfba9 bytes) 000558 1786748145.428258 MIRROR_RESYNC 8 mirrors 000559 1786748147.058899 mapread 0x029622 thru 0x02f1c5 (0x5ba4 bytes) 000560 1786748147.097176 read 0x008030 thru 0x008338 (0x309 bytes) 000561 1786748147.117559 read 0x00b156 thru 0x0185e4 (0xd48f bytes) 000562 1786748147.144878 punch from 0x004354 to 0x0128a9 (0xe556 bytes) 000564 1786748147.207207 fallocate 0x0392eb thru 0x03ffff (0x6d15 bytes) 000565 1786748147.226773 trunc from 0x03425d to 0x02b9cc (0x8890 bytes) 000566 1786748147.275749 mapwrite 0x00de18 thru 0x0119d4 (0x3bbd bytes) 000567 1786748147.329218 mapwrite 0x031d12 thru 0x036e68 (0x5157 bytes) 000568 1786748147.381275 trunc from 0x036e69 to 0x027913 (0xf555 bytes) 000569 1786748147.399929 read 0x0011dc thru 0x00ecb1 (0xdad6 bytes) 000570 1786748147.413088 fallocate 0x0106a7 thru 0x01ba22 (0xb37c bytes) 000571 1786748147.419444 trunc from 0x027914 to 0x000864 (0x270af bytes) 000572 1786748147.433927 mapwrite 0x02911c thru 0x02af87 (0x1e6c bytes) 000573 1786748147.462020 trunc from 0x02af88 to 0x020765 (0xa822 bytes) 000574 1786748147.493068 write 0x03e5a6 thru 0x03ffff (0x1a5a bytes) 000575 1786748147.494458 write 0x014c7b thru 0x01b7af (0x6b35 bytes) 000576 1786748147.515401 read 0x000c62 thru 0x001324 (0x6c3 bytes) 000577 1786748147.547525 MIRROR_RESYNC 8 mirrors 000578 1786748149.396635 mapwrite 0x0337fc thru 0x03ba1b (0x8220 bytes) 000579 1786748149.536457 punch from 0x011b3f to 0x012e49 (0x130b bytes) 000580 1786748149.546836 punch from 0x01d42b to 0x02a4f8 (0xd0ce bytes) 000581 1786748149.551494 mapwrite 0x02cb44 thru 0x031d4d (0x520a bytes) 000582 1786748149.598627 punch from 0x00d013 to 0x01bea5 (0xee93 bytes) 000584 1786748149.605602 trunc from 0x040000 to 0x009414 (0x36beb bytes) 000585 1786748149.631014 trunc from 0x009415 to 0x028537 (0x1f123 bytes) 000586 1786748149.651726 mapwrite 0x002600 thru 0x00df1c (0xb91d bytes) 000587 1786748149.724810 write 0x029b70 thru 0x02ad32 (0x11c3 bytes) 000588 1786748149.729519 mapread 0x024e61 thru 0x02ad32 (0x5ed2 bytes) 000589 1786748149.760612 mapwrite 0x034e1d thru 0x03ffff (0xb1e3 bytes) 000590 1786748149.887216 fallocate 0x030747 thru 0x03ffd5 (0xf88f bytes) 000591 1786748149.898871 fallocate 0x017672 thru 0x0262e5 (0xec74 bytes) 000592 1786748149.904008 mapwrite 0x01f601 thru 0x029353 (0x9d53 bytes) 000593 1786748149.941894 trunc from 0x01cb4c to 0x0269bd (0x9e72 bytes) 000594 1786748149.952996 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x790d1335 (stale) Mirror 4: 0x790d1335 (stale) Mirror 5: 0x790d1335 (stale) Mirror 6: 0x790d1335 (stale) Mirror 7: 0x790d1335 (stale) Mirror 8: 0x790d1335 (stale) Mirror 9: 0x790d1335 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6b6fa6c Mirror 10: 0x6b6fa6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000595 1786748151.156865 fallocate 0x00ce97 thru 0x0159d6 (0x8b40 bytes) 000596 1786748151.216721 fallocate 0x02f9b4 thru 0x035b21 (0x616e bytes) 000597 1786748151.224575 punch from 0x03e3a4 to 0x03ffff (0x1c5c bytes) 000598 1786748151.231544 mapread 0x02248e thru 0x02e6a4 (0xc217 bytes) 000599 1786748151.269288 read 0x039eb3 thru 0x03bdcb (0x1f19 bytes) 000600 1786748151.279047 trunc from 0x040000 to 0x02a074 (0x15f8b bytes) 000601 1786748151.327004 fallocate 0x015cf2 thru 0x022c43 (0xcf52 bytes) 000602 1786748151.339645 mapwrite 0x00ba6d thru 0x01a40a (0xe99e bytes) 000604 1786748151.447861 punch from 0x01587f to 0x01f426 (0x9ba8 bytes) 000605 1786748151.462931 trunc from 0x025547 to 0x026cc8 (0x1782 bytes) 000606 1786748151.476968 mapwrite 0x029f9c thru 0x032993 (0x89f8 bytes) 000607 1786748151.554850 punch from 0x029bd5 to 0x02b9fa (0x1e26 bytes) 000608 1786748151.561849 mapread 0x0283c7 thru 0x02ec1a (0x6854 bytes) 000609 1786748151.569860 trunc from 0x032994 to 0x038c0f (0x627c bytes) 000610 1786748151.583730 trunc from 0x038c10 to 0x030918 (0x82f7 bytes) 000611 1786748151.604577 mapread 0x00bf19 thru 0x00f779 (0x3861 bytes) 000612 1786748151.605781 mapread 0x008b52 thru 0x00a98a (0x1e39 bytes) 000613 1786748151.607721 fallocate 0x0147dd thru 0x01a8f4 (0x6118 bytes) 000614 1786748151.613280 punch from 0x01d64f to 0x01d82b (0x1dd bytes) 000615 1786748151.620811 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30919): Mirror 3: 0xa1d40b2d (stale) Mirror 4: 0xa1d40b2d (stale) Mirror 5: 0xa1d40b2d (stale) Mirror 6: 0xa1d40b2d (stale) Mirror 7: 0xa1d40b2d (stale) Mirror 9: 0xa1d40b2d (stale) Mirror 10: 0x5605845c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30919): Mirror 3: 0xa1d40b2d (stale) Mirror 4: 0xa1d40b2d (stale) Mirror 5: 0xa1d40b2d (stale) Mirror 6: 0xa1d40b2d (stale) Mirror 7: 0xa1d40b2d (stale) Mirror 9: 0xa1d40b2d (stale) Mirror 10: 0x5605845c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30919: skipped 000616 1786748152.231511 trunc from 0x005ad1 to 0x007df9 (0x2329 bytes) 000618 1786748152.237760 write 0x02d246 thru 0x038d73 (0xbb2e bytes) 000620 1786748152.248793 write 0x013b9f thru 0x014e32 (0x1294 bytes) 000621 1786748152.266013 mapread 0x011af4 thru 0x014662 (0x2b6f bytes) 000622 1786748152.289492 MIRROR_RESYNC 8 mirrors 000623 1786748155.539589 write 0x018910 thru 0x0229cb (0xa0bc bytes) 000624 1786748155.584493 read 0x0199e1 thru 0x01a3ba (0x9da bytes) 000625 1786748155.585887 trunc from 0x028301 to 0x02bd76 (0x3a76 bytes) 000626 1786748155.593600 punch from 0x008492 to 0x012661 (0xa1d0 bytes) 000627 1786748155.599044 mapread 0x00f29a thru 0x019d13 (0xaa7a bytes) 000628 1786748155.629314 punch from 0x008203 to 0x00df6b (0x5d69 bytes) 000629 1786748155.641005 punch from 0x00d0eb to 0x017453 (0xa369 bytes) 000630 1786748155.652088 MIRROR_RESYNC 8 mirrors 000631 1786748158.596782 fallocate 0x02f568 thru 0x03901c (0x9ab5 bytes) 000632 1786748158.657895 trunc from 0x01a8dd to 0x01baf4 (0x1218 bytes) 000633 1786748158.663323 punch from 0x02b3b9 to 0x02fcf6 (0x493e bytes) 000634 1786748158.668987 mapwrite 0x039166 thru 0x03ffff (0x6e9a bytes) 000635 1786748158.722255 punch from 0x00adae to 0x011420 (0x6673 bytes) 000636 1786748158.727041 write 0x035037 thru 0x037280 (0x224a bytes) 000637 1786748158.728790 mapread 0x01ebff thru 0x029020 (0xa422 bytes) 000638 1786748158.790826 mapwrite 0x037411 thru 0x03e2d2 (0x6ec2 bytes) 000639 1786748158.828625 trunc from 0x040000 to 0x0073ba (0x38c45 bytes) 000640 1786748158.872825 fallocate 0x036ea1 thru 0x03ffff (0x915f bytes) 000641 1786748158.883765 fallocate 0x008959 thru 0x00ac7d (0x2325 bytes) 000642 1786748158.893077 trunc from 0x040000 to 0x02bedb (0x14124 bytes) 000643 1786748158.941997 write 0x03bb66 thru 0x03ffff (0x449a bytes) 000644 1786748158.947951 trunc from 0x0128bb to 0x015b95 (0x32db bytes) 000645 1786748158.961896 punch from 0x0337da to 0x03ffff (0xc826 bytes) 000646 1786748158.989849 fallocate 0x0046f3 thru 0x006b1d (0x242b bytes) 000647 1786748159.005534 mapread 0x03d427 thru 0x03ffff (0x2bd9 bytes) 000648 1786748159.025435 trunc from 0x040000 to 0x02875d (0x178a2 bytes) 000649 1786748159.082331 trunc from 0x002a31 to 0x00dc9e (0xb26e bytes) 000650 1786748159.098101 trunc from 0x02875e to 0x01ba27 (0xcd36 bytes) 000651 1786748159.165895 trunc from 0x01ba28 to 0x0035a4 (0x18483 bytes) 000652 1786748159.202884 write 0x01e045 thru 0x022dbc (0x4d78 bytes) 000653 1786748159.238530 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x22dbd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22dbd): Mirror 3: 0xc931da50 (stale) Mirror 4: 0xc931da50 (stale) Mirror 5: 0xc931da50 (stale) Mirror 6: 0xc931da50 (stale) Mirror 7: 0xc931da50 (stale) Mirror 9: 0xc931da50 (stale) Mirror 10: 0xc931da50 (stale) CRC-32 checksum value for chunk [0, 0x22dbd): Mirror 1: 0x4edd509 Mirror 11: 0x4edd509 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22dbd: skipped 000654 1786748160.421091 MIRROR_RESYNC 9 mirrors 000656 1786748162.206340 trunc from 0x00d3f9 to 0x010263 (0x2e6b bytes) 000657 1786748162.282118 trunc from 0x003451 to 0x009564 (0x6114 bytes) 000658 1786748162.290466 trunc from 0x022dbd to 0x02aa28 (0x7c6c bytes) 000659 1786748162.323169 fallocate 0x0277f5 thru 0x02f84b (0x8057 bytes) 000660 1786748162.332361 mapwrite 0x001899 thru 0x00e43f (0xcba7 bytes) 000661 1786748162.445462 write 0x00fe39 thru 0x01de68 (0xe030 bytes) 000662 1786748162.461696 punch from 0x00a520 to 0x0181bf (0xdca0 bytes) 000663 1786748162.489999 mapread 0x002947 thru 0x0085c8 (0x5c82 bytes) 000664 1786748162.491316 mapwrite 0x028f23 thru 0x03725d (0xe33b bytes) 000665 1786748162.584490 mapread 0x01f9fd thru 0x02ac06 (0xb20a bytes) 000666 1786748162.603063 mapread 0x01698b thru 0x022efe (0xc574 bytes) 000667 1786748162.622262 fallocate 0x00e37a thru 0x015f35 (0x7bbc bytes) 000668 1786748162.628280 punch from 0x00380e to 0x00a731 (0x6f24 bytes) 000669 1786748162.634889 mapwrite 0x02536a thru 0x02550f (0x1a6 bytes) 000670 1786748162.651014 read 0x025a0f thru 0x03443e (0xea30 bytes) 000671 1786748162.659813 MIRROR_RESYNC 9 mirrors 000672 1786748165.581049 fallocate 0x0285ba thru 0x035489 (0xced0 bytes) 000673 1786748165.629222 read 0x019cbb thru 0x01e229 (0x456f bytes) 000674 1786748165.644748 punch from 0x01b2cd to 0x021f23 (0x6c57 bytes) 000675 1786748165.657223 trunc from 0x01f1d2 to 0x01fbfc (0xa2b bytes) 000676 1786748165.673544 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3725e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3725e): Mirror 3: 0x49e2df28 (stale) Mirror 4: 0x49e2df28 (stale) Mirror 5: 0x49e2df28 (stale) Mirror 6: 0x49e2df28 (stale) Mirror 7: 0x49e2df28 (stale) Mirror 9: 0x49e2df28 (stale) Mirror 10: 0x49e2df28 (stale) Mirror 11: 0x49e2df28 (stale) CRC-32 checksum value for chunk [0, 0x3725e): Mirror 1: 0x6dabee69 Mirror 12: 0x6dabee69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3725e: skipped 000677 1786748167.225231 read 0x02fdb1 thru 0x030782 (0x9d2 bytes) 000678 1786748167.232978 mapwrite 0x01822b thru 0x02677f (0xe555 bytes) 000679 1786748167.359476 write 0x000042 thru 0x00be45 (0xbe04 bytes) 000680 1786748167.384542 mapwrite 0x023585 thru 0x02f260 (0xbcdc bytes) 000681 1786748167.433539 read 0x014223 thru 0x01f009 (0xade7 bytes) 000682 1786748167.434544 read 0x02763a thru 0x029fe0 (0x29a7 bytes) 000683 1786748167.436114 write 0x036e7c thru 0x03f791 (0x8916 bytes) 000684 1786748167.448081 read 0x00505e thru 0x00a1ae (0x5151 bytes) 000685 1786748167.448854 trunc from 0x03f792 to 0x0312b0 (0xe4e1 bytes) 000686 1786748167.474822 trunc from 0x0312b1 to 0x037126 (0x5e76 bytes) 000687 1786748167.508678 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37127): Mirror 3: 0x3129c0ae (stale) Mirror 4: 0x3129c0ae (stale) Mirror 5: 0x3129c0ae (stale) Mirror 6: 0x3129c0ae (stale) Mirror 7: 0x3129c0ae (stale) Mirror 9: 0x3129c0ae (stale) Mirror 10: 0x3129c0ae (stale) Mirror 11: 0x3129c0ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37127): Mirror 3: 0x3129c0ae (stale) Mirror 4: 0x3129c0ae (stale) Mirror 5: 0x3129c0ae (stale) Mirror 6: 0x3129c0ae (stale) Mirror 7: 0x3129c0ae (stale) Mirror 9: 0x3129c0ae (stale) Mirror 10: 0x3129c0ae (stale) Mirror 11: 0x3129c0ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37127: skipped 000688 1786748168.114672 write 0x01780b thru 0x025e84 (0xe67a bytes) 000689 1786748168.136990 punch from 0x011d1a to 0x0148b6 (0x2b9d bytes) 000690 1786748168.142410 trunc from 0x02cafd to 0x0356cd (0x8bd1 bytes) 000691 1786748168.153062 trunc from 0x02760b to 0x02fd34 (0x872a bytes) 000693 1786748168.159776 read 0x0255a5 thru 0x0255d1 (0x2d bytes) 000694 1786748168.164506 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37127): Mirror 3: 0x3129c0ae (stale) Mirror 4: 0x3129c0ae (stale) Mirror 5: 0x3129c0ae (stale) Mirror 6: 0x3129c0ae (stale) Mirror 7: 0x3129c0ae (stale) Mirror 9: 0x3129c0ae (stale) Mirror 11: 0x3129c0ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37127): Mirror 3: 0x3129c0ae (stale) Mirror 4: 0x3129c0ae (stale) Mirror 5: 0x3129c0ae (stale) Mirror 6: 0x3129c0ae (stale) Mirror 7: 0x3129c0ae (stale) Mirror 9: 0x3129c0ae (stale) Mirror 11: 0x3129c0ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37127: skipped 000696 1786748168.860167 mapwrite 0x039d50 thru 0x03b28d (0x153e bytes) 000697 1786748168.914900 trunc from 0x03b28e to 0x000ed5 (0x3a3b8 bytes) 000699 1786748168.941945 fallocate 0x033cf8 thru 0x03d130 (0x9439 bytes) 000700 1786748168.956877 read 0x031ebc thru 0x033d90 (0x1ed5 bytes) 000701 1786748168.965571 punch from 0x0089e9 to 0x00a4f7 (0x1b0f bytes) 000702 1786748168.975755 read 0x000e69 thru 0x002160 (0x12f8 bytes) 000703 1786748168.997519 fallocate 0x028084 thru 0x02c6bf (0x463c bytes) 000704 1786748169.007010 mapread 0x022baf thru 0x028c77 (0x60c9 bytes) 000705 1786748169.012461 mapread 0x020804 thru 0x02c61e (0xbe1b bytes) 000706 1786748169.015232 mapwrite 0x03ae8c thru 0x03ffff (0x5174 bytes) 000707 1786748169.067790 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x41fff878 (stale) Mirror 5: 0x41fff878 (stale) Mirror 6: 0x41fff878 (stale) Mirror 7: 0x41fff878 (stale) Mirror 9: 0x41fff878 (stale) Mirror 11: 0x41fff878 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x41fff878 (stale) Mirror 5: 0x41fff878 (stale) Mirror 6: 0x41fff878 (stale) Mirror 7: 0x41fff878 (stale) Mirror 9: 0x41fff878 (stale) Mirror 11: 0x41fff878 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000708 1786748169.590767 read 0x0333d8 thru 0x03a7be (0x73e7 bytes) 000709 1786748169.603476 fallocate 0x00553e thru 0x011343 (0xbe06 bytes) 000710 1786748169.611338 trunc from 0x0259d3 to 0x02ce0a (0x7438 bytes) 000711 1786748169.616751 trunc from 0x040000 to 0x003754 (0x3c8ab bytes) 000712 1786748169.637024 mapwrite 0x00e865 thru 0x011647 (0x2de3 bytes) 000713 1786748169.732746 trunc from 0x011648 to 0x0129e2 (0x139b bytes) 000714 1786748169.754737 trunc from 0x0129e3 to 0x02cc73 (0x1a291 bytes) 000715 1786748169.788024 mapread 0x00bb35 thru 0x012392 (0x685e bytes) 000716 1786748169.820056 read 0x010159 thru 0x0195f5 (0x949d bytes) 000717 1786748169.829692 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cc74): Mirror 4: 0xf30317d0 (stale) Mirror 5: 0xf30317d0 (stale) Mirror 6: 0xf30317d0 (stale) Mirror 7: 0xf30317d0 (stale) Mirror 11: 0xf30317d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cc74): Mirror 4: 0xf30317d0 (stale) Mirror 5: 0xf30317d0 (stale) Mirror 6: 0xf30317d0 (stale) Mirror 7: 0xf30317d0 (stale) Mirror 11: 0xf30317d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cc74: skipped 000718 1786748170.419544 trunc from 0x006610 to 0x016458 (0xfe49 bytes) 000720 1786748170.431720 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2cc74): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cc74): Mirror 4: 0xf30317d0 (stale) Mirror 5: 0xf30317d0 (stale) Mirror 6: 0xf30317d0 (stale) Mirror 7: 0xf30317d0 (stale) Mirror 11: 0xf30317d0 (stale) CRC-32 checksum value for chunk [0, 0x2cc74): Mirror 1: 0xc6a15b0f Mirror 12: 0xc6a15b0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cc74: skipped 000721 1786748171.535676 mapread 0x014bce thru 0x02059e (0xb9d1 bytes) 000723 1786748171.634862 mapwrite 0x004fcc thru 0x00dc30 (0x8c65 bytes) 000724 1786748171.845081 trunc from 0x02cc74 to 0x020699 (0xc5da bytes) 000725 1786748171.887933 read 0x019047 thru 0x01dbb0 (0x4b6a bytes) 000726 1786748171.905084 punch from 0x00b46a to 0x010151 (0x4ce8 bytes) 000727 1786748171.913689 mapread 0x000ddf thru 0x0083e8 (0x760a bytes) 000728 1786748171.936180 trunc from 0x01d824 to 0x01e776 (0xf53 bytes) 000729 1786748171.968451 MIRROR_RESYNC 7 mirrors 000730 1786748173.679419 write 0x037c86 thru 0x03ee9b (0x7216 bytes) 000731 1786748173.724908 read 0x02c3da thru 0x037979 (0xb5a0 bytes) 000732 1786748173.754217 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ee9c): Mirror 4: 0xf2abf562 (stale) Mirror 5: 0xf2abf562 (stale) Mirror 6: 0xf2abf562 (stale) Mirror 11: 0xf2abf562 (stale) Mirror 12: 0xf2abf562 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee9c): Mirror 4: 0xf2abf562 (stale) Mirror 5: 0xf2abf562 (stale) Mirror 6: 0xf2abf562 (stale) Mirror 11: 0xf2abf562 (stale) Mirror 12: 0xf2abf562 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee9c: skipped 000733 1786748174.147327 punch from 0x01b910 to 0x022b4b (0x723c bytes) 000734 1786748174.152193 punch from 0x027c0b to 0x02b37a (0x3770 bytes) 000735 1786748174.158307 trunc from 0x03ee9c to 0x03f469 (0x5ce bytes) 000736 1786748174.179332 write 0x016596 thru 0x01ade8 (0x4853 bytes) 000737 1786748174.190935 fallocate 0x01e581 thru 0x02213a (0x3bba bytes) 000738 1786748174.195979 read 0x03bff4 thru 0x03dd29 (0x1d36 bytes) 000739 1786748174.205962 MIRROR_RESYNC 6 mirrors 000740 1786748176.193586 mapwrite 0x03b096 thru 0x03d4d8 (0x2443 bytes) 000741 1786748176.284061 trunc from 0x03f46a to 0x017d6c (0x276fd bytes) 000742 1786748176.316372 mapwrite 0x02255c thru 0x027eda (0x597f bytes) 000743 1786748176.379461 trunc from 0x02627a to 0x027eda (0x1c61 bytes) 000744 1786748176.388026 fallocate 0x033e5a thru 0x03e6a0 (0xa847 bytes) 000745 1786748176.393179 fallocate 0x03c6b5 thru 0x03ffff (0x394b bytes) 000746 1786748176.407835 trunc from 0x0273e1 to 0x0329c4 (0xb5e4 bytes) 000747 1786748176.414986 mapread 0x02520b thru 0x02d25a (0x8050 bytes) 000748 1786748176.430589 fallocate 0x032d97 thru 0x033df2 (0x105c bytes) 000749 1786748176.437456 trunc from 0x016fe8 to 0x019318 (0x2331 bytes) 000750 1786748176.451865 mapread 0x01176a thru 0x0209cf (0xf266 bytes) 000751 1786748176.506909 MIRROR_RESYNC 6 mirrors 000752 1786748178.682302 write 0x00a093 thru 0x00fe51 (0x5dbf bytes) 000753 1786748178.774080 trunc from 0x040000 to 0x034daf (0xb250 bytes) 000755 1786748178.814838 write 0x00d91f thru 0x01ac8f (0xd371 bytes) 000756 1786748178.827493 mapwrite 0x02cd58 thru 0x03325c (0x6505 bytes) 000757 1786748178.917118 mapwrite 0x037de5 thru 0x03fddb (0x7ff7 bytes) 000758 1786748178.992110 mapwrite 0x00680f thru 0x00d194 (0x6986 bytes) 000761 1786748179.104848 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3fddc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fddc): Mirror 4: 0x8963e144 (stale) Mirror 5: 0x8963e144 (stale) Mirror 6: 0x8963e144 (stale) Mirror 11: 0x8963e144 (stale) Mirror 12: 0x8963e144 (stale) CRC-32 checksum value for chunk [0, 0x3fddc): Mirror 1: 0xc1e43cc Mirror 13: 0xc1e43cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fddc: skipped 000762 1786748181.053117 read 0x031659 thru 0x039fef (0x8997 bytes) 000763 1786748181.116071 MIRROR_RESYNC 7 mirrors 000765 1786748183.297855 fallocate 0x028fee thru 0x031fa9 (0x8fbc bytes) 000766 1786748183.341175 mapwrite 0x026231 thru 0x02adce (0x4b9e bytes) 000767 1786748183.389112 mapread 0x03534a thru 0x0397a5 (0x445c bytes) 000768 1786748183.401394 write 0x00f596 thru 0x01184f (0x22ba bytes) 000769 1786748183.411192 read 0x033c6a thru 0x03b625 (0x79bc bytes) 000770 1786748183.413263 trunc from 0x033612 to 0x03fddb (0xc7ca bytes) 000771 1786748183.424945 punch from 0x008253 to 0x009040 (0xdee bytes) 000772 1786748183.433448 trunc from 0x019bf7 to 0x019d5d (0x167 bytes) 000773 1786748183.440311 read 0x013447 thru 0x0208cd (0xd487 bytes) 000774 1786748183.453076 write 0x004674 thru 0x004ced (0x67a bytes) 000775 1786748183.454345 fallocate 0x0260ac thru 0x0280ea (0x203f bytes) 000776 1786748183.460804 trunc from 0x032fb3 to 0x03cde5 (0x9e33 bytes) 000777 1786748183.468969 trunc from 0x03fddc to 0x036ea2 (0x8f39 bytes) 000778 1786748183.503437 punch from 0x013f16 to 0x019fd7 (0x60c2 bytes) 000780 1786748183.520881 punch from 0x03665a to 0x036ea2 (0x849 bytes) 000781 1786748183.527364 trunc from 0x015a64 to 0x0223cf (0xc96c bytes) 000782 1786748183.538035 trunc from 0x0046cf to 0x009b02 (0x5434 bytes) 000783 1786748183.570443 mapwrite 0x013b57 thru 0x01b61c (0x7ac6 bytes) 000784 1786748183.666953 read 0x032a0b thru 0x036ea2 (0x4498 bytes) 000785 1786748183.677844 punch from 0x029833 to 0x035d8d (0xc55b bytes) 000786 1786748183.687023 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36ea3): Mirror 5: 0xde3b7b06 (stale) Mirror 6: 0xde3b7b06 (stale) Mirror 11: 0xde3b7b06 (stale) Mirror 12: 0xde3b7b06 (stale) Mirror 13: 0xde3b7b06 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ea3): Mirror 5: 0xde3b7b06 (stale) Mirror 6: 0xde3b7b06 (stale) Mirror 11: 0xde3b7b06 (stale) Mirror 12: 0xde3b7b06 (stale) Mirror 13: 0xde3b7b06 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ea3: skipped 000787 1786748184.261643 trunc from 0x022df6 to 0x0274a5 (0x46b0 bytes) 000788 1786748184.266701 write 0x016b55 thru 0x021826 (0xacd2 bytes) 000789 1786748184.285360 punch from 0x02532c to 0x034ee9 (0xfbbe bytes) 000790 1786748184.291410 write 0x0134a1 thru 0x014993 (0x14f3 bytes) 000791 1786748184.301399 mapread 0x0177a3 thru 0x026b5d (0xf3bb bytes) 000792 1786748184.336506 mapread 0x021276 thru 0x0237df (0x256a bytes) 000793 1786748184.346311 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x36ea3): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ea3): Mirror 5: 0xde3b7b06 (stale) Mirror 6: 0xde3b7b06 (stale) Mirror 11: 0xde3b7b06 (stale) Mirror 12: 0xde3b7b06 (stale) Mirror 13: 0xde3b7b06 (stale) CRC-32 checksum value for chunk [0, 0x36ea3): Mirror 1: 0x963d4f6c Mirror 14: 0x963d4f6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ea3: skipped 000794 1786748185.509469 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x36ea3): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ea3): Mirror 5: 0xde3b7b06 (stale) Mirror 6: 0xde3b7b06 (stale) Mirror 11: 0xde3b7b06 (stale) Mirror 12: 0xde3b7b06 (stale) Mirror 13: 0xde3b7b06 (stale) CRC-32 checksum value for chunk [0, 0x36ea3): Mirror 1: 0x963d4f6c Mirror 14: 0x963d4f6c Mirror 15: 0x963d4f6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ea3: skipped 000795 1786748186.642211 MIRROR_RESYNC 8 mirrors 000796 1786748188.396068 mapwrite 0x03b8db thru 0x03ffff (0x4725 bytes) 000797 1786748188.505081 fallocate 0x00a99c thru 0x00c600 (0x1c65 bytes) 000798 1786748188.513225 write 0x02bd52 thru 0x0331bc (0x746b bytes) 000799 1786748188.524323 write 0x024ac9 thru 0x032e23 (0xe35b bytes) 000800 1786748188.530805 mapwrite 0x01f285 thru 0x02a5de (0xb35a bytes) 000801 1786748188.566988 fallocate 0x02b4fd thru 0x037a6d (0xc571 bytes) 000802 1786748188.572322 trunc from 0x040000 to 0x0334bd (0xcb42 bytes) 000803 1786748188.601934 mapread 0x0096b7 thru 0x0128c1 (0x920b bytes) 000804 1786748188.625860 trunc from 0x00c3f3 to 0x01923f (0xce4d bytes) 000806 1786748188.645305 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x334be): Mirror 5: 0x8b5e7568 (stale) Mirror 11: 0x8b5e7568 (stale) Mirror 12: 0x8b5e7568 (stale) Mirror 13: 0x8b5e7568 (stale) Mirror 14: 0x8b5e7568 (stale) Mirror 15: 0x8b5e7568 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x334be): Mirror 5: 0x8b5e7568 (stale) Mirror 11: 0x8b5e7568 (stale) Mirror 12: 0x8b5e7568 (stale) Mirror 13: 0x8b5e7568 (stale) Mirror 14: 0x8b5e7568 (stale) Mirror 15: 0x8b5e7568 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x334be: skipped 000807 1786748189.191465 punch from 0x01817a to 0x026d04 (0xeb8b bytes) 000809 1786748189.197740 read 0x00d6b5 thru 0x019073 (0xb9bf bytes) 000810 1786748189.215096 trunc from 0x0334be to 0x00757f (0x2bf3e bytes) 000811 1786748189.241851 mapread 0x006f84 thru 0x00757f (0x5fc bytes) 000812 1786748189.265771 trunc from 0x007580 to 0x0219a1 (0x1a422 bytes) 000813 1786748189.294194 read 0x0042ac thru 0x008c9a (0x49ef bytes) 000814 1786748189.312122 punch from 0x010543 to 0x01186a (0x1328 bytes) 000815 1786748189.317984 read 0x018192 thru 0x0219a1 (0x9810 bytes) 000816 1786748189.321683 punch from 0x0051ac to 0x00c3a6 (0x71fb bytes) 000817 1786748189.332269 punch from 0x008b12 to 0x018885 (0xfd74 bytes) 000818 1786748189.342090 trunc from 0x0219a2 to 0x02ddbf (0xc41e bytes) 000819 1786748189.364836 trunc from 0x00c913 to 0x01514d (0x883b bytes) 000821 1786748189.375661 mapread 0x011ede thru 0x0141e5 (0x2308 bytes) 000822 1786748189.395466 trunc from 0x02ddc0 to 0x022f80 (0xae3f bytes) 000823 1786748189.429550 trunc from 0x00c0b3 to 0x01170c (0x565a bytes) 000824 1786748189.434839 write 0x029b77 thru 0x02e59f (0x4a29 bytes) 000825 1786748189.436971 trunc from 0x010b8c to 0x014535 (0x39aa bytes) 000826 1786748189.446074 mapwrite 0x010fe8 thru 0x01f9db (0xe9f4 bytes) 000827 1786748189.518164 trunc from 0x010333 to 0x016c15 (0x68e3 bytes) 000828 1786748189.526844 fallocate 0x000218 thru 0x0008f8 (0x6e1 bytes) 000830 1786748189.532730 trunc from 0x02e5a0 to 0x0283bf (0x61e0 bytes) 000831 1786748189.550320 fallocate 0x020dfd thru 0x02bbfe (0xae02 bytes) 000832 1786748189.559567 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x283c0): Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x283c0): Mirror 5: 0x4e75c353 (stale) Mirror 11: 0x4e75c353 (stale) Mirror 12: 0x4e75c353 (stale) Mirror 13: 0x4e75c353 (stale) Mirror 14: 0x4e75c353 (stale) Mirror 15: 0x4e75c353 (stale) CRC-32 checksum value for chunk [0, 0x283c0): Mirror 1: 0xe2a7f0f5 Mirror 16: 0xe2a7f0f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x283c0: skipped 000833 1786748190.945365 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x283c0): Mirror 5: 0x4e75c353 (stale) Mirror 11: 0x4e75c353 (stale) Mirror 12: 0x4e75c353 (stale) Mirror 13: 0x4e75c353 (stale) Mirror 15: 0x4e75c353 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x283c0): Mirror 5: 0x4e75c353 (stale) Mirror 11: 0x4e75c353 (stale) Mirror 12: 0x4e75c353 (stale) Mirror 13: 0x4e75c353 (stale) Mirror 15: 0x4e75c353 (stale) CRC-32 checksum value for chunk [0, 0x283c0): Mirror 1: 0xe2a7f0f5 Mirror 16: 0xe2a7f0f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x283c0: skipped 000834 1786748191.951397 write 0x02c789 thru 0x0373be (0xac36 bytes) 000835 1786748192.044425 write 0x01afd1 thru 0x01e29e (0x32ce bytes) 000836 1786748192.059399 mapwrite 0x021c14 thru 0x0315ca (0xf9b7 bytes) 000837 1786748192.123965 write 0x012338 thru 0x017333 (0x4ffc bytes) 000838 1786748192.160506 read 0x011e86 thru 0x0209a9 (0xeb24 bytes) 000839 1786748192.177105 fallocate 0x002192 thru 0x0038ef (0x175e bytes) 000841 1786748192.189017 fallocate 0x02efb0 thru 0x033ccf (0x4d20 bytes) 000842 1786748192.194669 trunc from 0x0373bf to 0x00be1a (0x2b5a4 bytes) 000844 1786748192.231631 fallocate 0x039251 thru 0x03bae4 (0x2894 bytes) 000845 1786748192.242191 read 0x028257 thru 0x02f73c (0x74e6 bytes) 000846 1786748192.262364 read 0x01bb2b thru 0x026a1e (0xaef4 bytes) 000847 1786748192.263855 mapwrite 0x011e33 thru 0x01fdbc (0xdf8a bytes) 000848 1786748192.326195 mapwrite 0x001af4 thru 0x0037c6 (0x1cd3 bytes) 000849 1786748192.353784 trunc from 0x03bae5 to 0x0048c6 (0x3721e bytes) 000850 1786748192.381994 MIRROR_RESYNC 7 mirrors 000851 1786748193.417472 trunc from 0x001efa to 0x0048c6 (0x29cd bytes) 000852 1786748193.480014 punch from 0x0017b4 to 0x00197d (0x1ca bytes) 000854 1786748193.489061 fallocate 0x015147 thru 0x01b31c (0x61d6 bytes) 000855 1786748193.495409 read 0x0036b9 thru 0x0048c6 (0x120e bytes) 000856 1786748193.502777 punch from 0x0002bd to 0x0048c6 (0x460a bytes) 000857 1786748193.513795 MIRROR_RESYNC 7 mirrors 000858 1786748195.120211 mapread 0x004438 thru 0x0048c6 (0x48f bytes) 000859 1786748195.126059 read 0x0004b3 thru 0x0048c6 (0x4414 bytes) 000860 1786748195.132940 mapwrite 0x0177e8 thru 0x01ce7e (0x5697 bytes) 000862 1786748195.201092 fallocate 0x039761 thru 0x03f95e (0x61fe bytes) 000863 1786748195.214004 fallocate 0x00f578 thru 0x01bad4 (0xc55d bytes) 000864 1786748195.219555 mapwrite 0x00710f thru 0x00f478 (0x836a bytes) 000865 1786748195.260284 trunc from 0x03f95f to 0x0123ba (0x2d5a4 bytes) 000866 1786748195.283273 mapwrite 0x01645e thru 0x01ccfa (0x689d bytes) 000867 1786748195.340312 mapread 0x01752e thru 0x01ccfa (0x57cd bytes) 000868 1786748195.342363 fallocate 0x01d636 thru 0x021f9d (0x4968 bytes) 000869 1786748195.349094 trunc from 0x017e21 to 0x01874e (0x92e bytes) 000870 1786748195.355155 mapread 0x01e7da thru 0x021f9d (0x37c4 bytes) 000871 1786748195.366989 fallocate 0x0272f2 thru 0x02e636 (0x7345 bytes) 000872 1786748195.372933 punch from 0x00c766 to 0x0127eb (0x6086 bytes) 000873 1786748195.397464 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2e637): Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e637): Mirror 5: 0xf47bc012 (stale) Mirror 11: 0xf47bc012 (stale) Mirror 12: 0xf47bc012 (stale) Mirror 13: 0xf47bc012 (stale) Mirror 15: 0xf47bc012 (stale) Mirror 16: 0xf47bc012 (stale) CRC-32 checksum value for chunk [0, 0x2e637): Mirror 1: 0x6e15d2b9 Mirror 17: 0x6e15d2b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e637: skipped 000874 1786748196.623337 punch from 0x023fc0 to 0x02899b (0x49dc bytes) 000875 1786748196.681988 fallocate 0x02935c thru 0x02b7ae (0x2453 bytes) 000876 1786748196.688604 write 0x015caf thru 0x01db45 (0x7e97 bytes) 000877 1786748196.724939 trunc from 0x02e637 to 0x0119fa (0x1cc3c bytes) 000878 1786748196.776699 trunc from 0x0119fb to 0x03e5b2 (0x2cbb8 bytes) 000879 1786748196.797041 trunc from 0x03e5b3 to 0x00a2a2 (0x34310 bytes) 000880 1786748196.846541 MIRROR_RESYNC 8 mirrors 000881 1786748198.766304 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000882 1786748199.263420 trunc from 0x008085 to 0x00a2a2 (0x221e bytes) 000883 1786748199.335555 read 0x002b6c thru 0x00563e (0x2ad3 bytes) 000884 1786748199.344343 mapwrite 0x01a3b4 thru 0x0201aa (0x5df7 bytes) 000885 1786748199.455585 read 0x00eb1d thru 0x015929 (0x6e0d bytes) 000886 1786748199.485419 trunc from 0x0201ab to 0x032f23 (0x12d79 bytes) 000887 1786748199.526126 mapread 0x00492c thru 0x005d60 (0x1435 bytes) 000888 1786748199.527227 trunc from 0x032f24 to 0x031cac (0x1277 bytes) 000889 1786748199.565549 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31cad): Mirror 11: 0xeade2ee3 (stale) Mirror 12: 0xeade2ee3 (stale) Mirror 13: 0xeade2ee3 (stale) Mirror 15: 0xeade2ee3 (stale) Mirror 16: 0xeade2ee3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31cad): Mirror 11: 0xeade2ee3 (stale) Mirror 12: 0xeade2ee3 (stale) Mirror 13: 0xeade2ee3 (stale) Mirror 15: 0xeade2ee3 (stale) Mirror 16: 0xeade2ee3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31cad: skipped 000890 1786748199.965030 mapwrite 0x02c47c thru 0x033d79 (0x78fe bytes) 000891 1786748200.037956 write 0x0232a9 thru 0x02b28b (0x7fe3 bytes) 000892 1786748200.045809 punch from 0x02ce04 to 0x02f0a6 (0x22a3 bytes) 000893 1786748200.055375 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x33d7a): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d7a): Mirror 11: 0x1b24a842 (stale) Mirror 12: 0x1b24a842 (stale) Mirror 13: 0x1b24a842 (stale) Mirror 15: 0x1b24a842 (stale) Mirror 16: 0x1b24a842 (stale) CRC-32 checksum value for chunk [0, 0x33d7a): Mirror 1: 0x31aade1c Mirror 17: 0x31aade1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d7a: skipped 000894 1786748201.334022 write 0x03a0a4 thru 0x03dc76 (0x3bd3 bytes) 000895 1786748201.383283 mapread 0x0193a2 thru 0x019e6b (0xaca bytes) 000896 1786748201.389238 trunc from 0x00400b to 0x007d46 (0x3d3c bytes) 000897 1786748201.396055 trunc from 0x03dc77 to 0x02f461 (0xe815 bytes) 000898 1786748201.431199 MIRROR_RESYNC 7 mirrors 000899 1786748204.535774 trunc from 0x02f462 to 0x02eb5f (0x902 bytes) 000900 1786748204.594110 read 0x01802c thru 0x01903d (0x1012 bytes) 000901 1786748204.604336 punch from 0x008437 to 0x00aaee (0x26b8 bytes) 000902 1786748204.617607 write 0x01195e thru 0x01890d (0x6fb0 bytes) 000903 1786748204.627158 write 0x0224e9 thru 0x02c3e0 (0x9ef8 bytes) 000904 1786748204.644016 trunc from 0x025368 to 0x02b8ff (0x6598 bytes) 000905 1786748204.670311 write 0x036687 thru 0x03d1f7 (0x6b71 bytes) 000906 1786748204.677350 fallocate 0x039cd4 thru 0x03ffff (0x632c bytes) 000907 1786748204.688043 mapread 0x01d8bb thru 0x02b595 (0xdcdb bytes) 000908 1786748204.740650 read 0x0255d1 thru 0x02c69b (0x70cb bytes) 000909 1786748204.742202 mapwrite 0x00284d thru 0x00d26a (0xaa1e bytes) 000911 1786748204.802395 mapwrite 0x0081ec thru 0x00aab7 (0x28cc bytes) 000912 1786748204.822288 mapread 0x016ab3 thru 0x024f59 (0xe4a7 bytes) 000913 1786748204.825181 write 0x024089 thru 0x029392 (0x530a bytes) 000914 1786748204.829032 punch from 0x01402c to 0x01dce6 (0x9cbb bytes) 000915 1786748204.839030 fallocate 0x00361d thru 0x00f771 (0xc155 bytes) 000916 1786748204.851860 mapread 0x002f8f thru 0x00ca12 (0x9a84 bytes) 000917 1786748204.853973 mapread 0x01988f thru 0x01af01 (0x1673 bytes) 000918 1786748204.868249 trunc from 0x03d1f8 to 0x01c69c (0x20b5b bytes) 000919 1786748204.894762 punch from 0x00b458 to 0x00b6ef (0x298 bytes) 000920 1786748204.902009 mapwrite 0x0055c4 thru 0x00f462 (0x9e9f bytes) 000921 1786748204.929634 mapwrite 0x008a1b thru 0x012250 (0x9836 bytes) 000922 1786748204.980542 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1c69d): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c69d): Mirror 11: 0xca826984 (stale) Mirror 12: 0xca826984 (stale) Mirror 13: 0xca826984 (stale) Mirror 15: 0xca826984 (stale) Mirror 16: 0xca826984 (stale) Mirror 17: 0xca826984 (stale) CRC-32 checksum value for chunk [0, 0x1c69d): Mirror 1: 0x79cefcd4 Mirror 18: 0x79cefcd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c69d: skipped 000923 1786748206.210005 read 0x005ad9 thru 0x0157ce (0xfcf6 bytes) 000924 1786748206.225611 punch from 0x013817 to 0x016d9f (0x3589 bytes) 000925 1786748206.286415 mapwrite 0x034b9c thru 0x03b64c (0x6ab1 bytes) 000926 1786748206.374868 punch from 0x014658 to 0x01f3c0 (0xad69 bytes) 000927 1786748206.386202 punch from 0x0328b9 to 0x03b64c (0x8d94 bytes) 000928 1786748206.400762 trunc from 0x02dab7 to 0x02fc25 (0x216f bytes) 000929 1786748206.411537 read 0x03376b thru 0x03b64c (0x7ee2 bytes) 000930 1786748206.437609 mapwrite 0x01b7ac thru 0x025f83 (0xa7d8 bytes) 000931 1786748206.486385 mapwrite 0x03b1fa thru 0x03be6c (0xc73 bytes) 000932 1786748206.529804 read 0x023ce0 thru 0x02b949 (0x7c6a bytes) 000933 1786748206.531452 read 0x012a42 thru 0x014f61 (0x2520 bytes) 000934 1786748206.532458 trunc from 0x03be6d to 0x0004b2 (0x3b9ba bytes) 000935 1786748206.569920 fallocate 0x03026e thru 0x0360ab (0x5e3e bytes) 000936 1786748206.577949 trunc from 0x0360ac to 0x022899 (0x13812 bytes) 000938 1786748206.599369 fallocate 0x00c636 thru 0x0172c2 (0xac8d bytes) 000939 1786748206.605191 trunc from 0x02289a to 0x027d51 (0x54b8 bytes) 000940 1786748206.656741 mapwrite 0x030732 thru 0x03aca3 (0xa572 bytes) 000941 1786748206.751516 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3aca4): Mirror 11: 0xd3e98d44 (stale) Mirror 12: 0xd3e98d44 (stale) Mirror 13: 0xd3e98d44 (stale) Mirror 15: 0xd3e98d44 (stale) Mirror 16: 0xd3e98d44 (stale) Mirror 18: 0xf8613bf1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aca4): Mirror 11: 0xd3e98d44 (stale) Mirror 12: 0xd3e98d44 (stale) Mirror 13: 0xd3e98d44 (stale) Mirror 15: 0xd3e98d44 (stale) Mirror 16: 0xd3e98d44 (stale) Mirror 18: 0xf8613bf1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aca4: skipped 000942 1786748207.442744 trunc from 0x03aca4 to 0x001828 (0x3947b bytes) 000944 1786748207.486705 punch from 0x000013 to 0x001828 (0x1816 bytes) 000945 1786748207.497984 write 0x03cac5 thru 0x03ffff (0x353b bytes) 000946 1786748207.499532 trunc from 0x040000 to 0x034635 (0xb9ca bytes) 000947 1786748207.524662 mapwrite 0x023c6e thru 0x02b877 (0x7c0a bytes) 000948 1786748207.632950 mapread 0x023446 thru 0x02ae51 (0x7a0c bytes) 000949 1786748207.643642 punch from 0x02cc2b to 0x02e04a (0x1420 bytes) 000950 1786748207.667005 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x34636): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34636): Mirror 11: 0x20f3edde (stale) Mirror 12: 0x20f3edde (stale) Mirror 13: 0x20f3edde (stale) Mirror 15: 0x20f3edde (stale) Mirror 16: 0x20f3edde (stale) Mirror 18: 0x561c3868 (stale) CRC-32 checksum value for chunk [0, 0x34636): Mirror 1: 0xef6d6076 Mirror 19: 0xef6d6076 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34636: skipped 000951 1786748209.040656 mapread 0x005bbe thru 0x011797 (0xbbda bytes) 000952 1786748209.087434 mapwrite 0x006899 thru 0x008ec0 (0x2628 bytes) 000953 1786748209.169414 trunc from 0x0187a5 to 0x02056d (0x7dc9 bytes) 000954 1786748209.174025 mapread 0x002e2e thru 0x00b644 (0x8817 bytes) 000955 1786748209.176301 write 0x035681 thru 0x03ffff (0xa97f bytes) 000956 1786748209.182734 trunc from 0x0052ec to 0x0100a5 (0xadba bytes) 000957 1786748209.187177 trunc from 0x040000 to 0x013fe9 (0x2c016 bytes) 000958 1786748209.209118 mapread 0x004332 thru 0x00c204 (0x7ed3 bytes) 000959 1786748209.228963 trunc from 0x013fea to 0x01ff64 (0xbf7b bytes) 000960 1786748209.247002 read 0x0071c9 thru 0x00d4e7 (0x631f bytes) 000961 1786748209.248460 read 0x00a703 thru 0x00e887 (0x4185 bytes) 000962 1786748209.250711 read 0x0009f2 thru 0x00b712 (0xad21 bytes) 000963 1786748209.268897 mapread 0x01e684 thru 0x01ff64 (0x18e1 bytes) 000964 1786748209.271023 trunc from 0x011e74 to 0x01282f (0x9bc bytes) 000965 1786748209.276516 punch from 0x018b8b to 0x01ff64 (0x73da bytes) 000966 1786748209.283940 fallocate 0x006757 thru 0x011f53 (0xb7fd bytes) 000967 1786748209.291040 write 0x01186c thru 0x01db6e (0xc303 bytes) 000968 1786748209.303450 read 0x00c9c7 thru 0x01b4d0 (0xeb0a bytes) 000969 1786748209.304612 read 0x01eef9 thru 0x01ff64 (0x106c bytes) 000970 1786748209.312665 trunc from 0x01ff65 to 0x0383a2 (0x1843e bytes) 000971 1786748209.333999 trunc from 0x0383a3 to 0x03ed30 (0x698e bytes) 000972 1786748209.365904 fallocate 0x029467 thru 0x02ef8c (0x5b26 bytes) 000973 1786748209.375002 punch from 0x02f8b0 to 0x032b45 (0x3296 bytes) 000974 1786748209.392123 write 0x007a52 thru 0x00f72b (0x7cda bytes) 000975 1786748209.399639 write 0x01f19e thru 0x01ffdf (0xe42 bytes) 000977 1786748209.416939 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed31): Mirror 12: 0xb4f56156 (stale) Mirror 13: 0xb4f56156 (stale) Mirror 15: 0xb4f56156 (stale) Mirror 16: 0xb4f56156 (stale) Mirror 18: 0xa9fa30a9 (stale) Mirror 19: 0x33e77b96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed31): Mirror 12: 0xb4f56156 (stale) Mirror 13: 0xb4f56156 (stale) Mirror 15: 0xb4f56156 (stale) Mirror 16: 0xb4f56156 (stale) Mirror 18: 0xa9fa30a9 (stale) Mirror 19: 0x33e77b96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed31: skipped 000978 1786748209.856987 write 0x0292dc thru 0x02ce69 (0x3b8e bytes) 000979 1786748209.880202 fallocate 0x031ed0 thru 0x03e645 (0xc776 bytes) 000980 1786748209.887225 mapread 0x000daf thru 0x0095dc (0x882e bytes) 000981 1786748209.918412 read 0x0398f6 thru 0x03ed30 (0x543b bytes) 000982 1786748209.938009 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ed31): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed31): Mirror 12: 0xb4f56156 (stale) Mirror 13: 0xb4f56156 (stale) Mirror 15: 0xb4f56156 (stale) Mirror 16: 0xb4f56156 (stale) Mirror 18: 0xa9fa30a9 (stale) Mirror 19: 0x33e77b96 (stale) CRC-32 checksum value for chunk [0, 0x3ed31): Mirror 1: 0xa8b088e0 Mirror 20: 0xa8b088e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed31: skipped 000983 1786748211.363566 mapread 0x01a5e0 thru 0x01cfbd (0x29de bytes) 000984 1786748211.424755 read 0x019a59 thru 0x027cc4 (0xe26c bytes) 000985 1786748211.448482 trunc from 0x03ed31 to 0x0375b7 (0x7779 bytes) 000986 1786748211.565428 fallocate 0x034f51 thru 0x03ffff (0xb0af bytes) 000987 1786748211.576691 mapread 0x028b11 thru 0x0375b7 (0xeaa7 bytes) 000988 1786748211.614261 mapread 0x01113f thru 0x017be0 (0x6aa2 bytes) 000989 1786748211.635525 MIRROR_RESYNC 8 mirrors 000990 1786748214.446797 trunc from 0x0375b8 to 0x009fac (0x2d60b bytes) 000992 1786748214.505084 MIRROR_RESYNC 8 mirrors 000993 1786748216.040503 fallocate 0x01f4c4 thru 0x02b7a8 (0xc2e5 bytes) 000994 1786748216.108820 trunc from 0x00021f to 0x0032ae (0x3090 bytes) 000995 1786748216.124024 write 0x02c891 thru 0x02ee95 (0x2605 bytes) 000996 1786748216.136770 trunc from 0x02ee96 to 0x0266cf (0x87c6 bytes) 000998 1786748216.202551 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x266d0): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x266d0): Mirror 12: 0x8d8a24f5 (stale) Mirror 13: 0x8d8a24f5 (stale) Mirror 15: 0x8d8a24f5 (stale) Mirror 16: 0x8d8a24f5 (stale) Mirror 18: 0x8d8a24f5 (stale) Mirror 19: 0x8d8a24f5 (stale) Mirror 20: 0x8d8a24f5 (stale) CRC-32 checksum value for chunk [0, 0x266d0): Mirror 1: 0x8d8a24f5 Mirror 21: 0x8d8a24f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x266d0: skipped 000999 1786748217.271423 fallocate 0x0183bc thru 0x019a55 (0x169a bytes) 001000 1786748217.343058 trunc from 0x007a99 to 0x00c9ee (0x4f56 bytes) All operations completed A-OK! PASS 70a (119s) == sanity-flr test 71: check mirror extend parent fid ==== 18:57:02 (1786748222) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x8d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (7s) == sanity-flr test 72: check append on FLR file ========== 18:57:09 (1786748229) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.179968 s, 11.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128317 s, 8.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x139:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 18:57:17 (1786748237) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3323808 .. Extending file size to 5740256 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9139200 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6897248 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7151872 .. Extending file size to 8366144 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8787936 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9110112 .. failed Waiting 61391 61392 61393 61394 61395 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) 192.168.203.108@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:58:34 (1786748314) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2601088 .. Extending file size to 1835392 .. Extending file size to 4024288 .. done Extending file size to 7912256 .. Extending file size to 5574432 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5832160 .. Extending file size to 7548800 .. Extending file size to 8240928 .. Extending file size to 8642368 .. Extending file size to 9044768 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8589024 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62771 62773 62775 62776 62777 failed 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) 192.168.203.108@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:59:54 (1786748394) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1268856 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2084 1269220 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21872 3585148 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21432 3585588 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43304 7170736 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1268856 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2084 1269220 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 21872 3585148 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 21432 3585588 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43304 7170736 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105176 s, 19.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.167412 s, 12.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x3795a54a (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) PASS 200c (14s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:00:10 (1786748410) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123736 s, 16.9 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 19:00:19 (1786748419) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:00:28 (1786748428) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106589 s, 19.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:00:36 (1786748436) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148902 s, 14.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:00:45 (1786748445) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117716 s, 17.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:01:20 (1786748480) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780143 s, 13.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x220:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x220:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x220:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:02:14 (1786748534) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0914172 s, 22.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x153:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x153:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:02:59 (1786748579) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg308-server: Pool lustre.test_204f created Adding targets to pool oleg308-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg308-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0895946 s, 23.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x154:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x154:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg308-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg308-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg308-server: Pool lustre.test_204f destroyed PASS 204f (70s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:04:09 (1786748649) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:04:16 (1786748656) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:04:22 (1786748662) oleg308-server: Pool lustre.test_206 created Adding targets to pool oleg308-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg308-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg308-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg308-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg308-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:04:49 (1786748689) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692272 s, 15.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504111 s, 20.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x507cb0fb Mirror 3: 0x507cb0fb PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:04:59 (1786748699) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:05:03 (1786748703) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:05:06 (1786748706) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0543728 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1180000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1650000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1650000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1650000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1de0000 Read this (4 bytes): BBBB 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:05:45 (1786748745) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.034955 s, 234 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:05:56 (1786748756) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127634 s, 8.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbd lmm_fid: [0x200000405:0xbd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbd lmm_fid: [0x200000405:0xbd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611325 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-14 19:05:58.000000000 -0400 Modify: 2026-08-14 19:05:58.000000000 -0400 Change: 2026-08-14 19:06:00.000000000 -0400 Birth: 2026-08-14 19:05:58.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:06:06 (1786748766) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650701 s, 16.1 MB/s oleg308-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg308-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg308-server: O/280000401/d0/160: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:06:31 (1786748791) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.262934 s, 156 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0211882 s, 193 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 14 19:06 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xc2:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 14 19:06 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:06:40 (1786748800) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:06:48 (1786748808) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1811 sec =========== 19:06:58 (1786748818) === sanity-flr: start cleanup 19:07:00 (1786748820) === === sanity-flr: finish cleanup 19:07:09 (1786748829) ===