-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 13:52:44 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg339-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 13:53:05 (1787853185) === oleg339-client.virtnet: executing check_config_client /mnt/lustre oleg339-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg339-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9fc1c52e3000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fc1c52e3000.idle_timeout=debug disable quota as required oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 13:53:48 (1787853228) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8206] == sanity-flr test 0a: lfs mirror create with -N option == 13:53:51 (1787853231) PASS 0a (23s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:54:14 (1787853254) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:54:18 (1787853258) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:54:23 (1787853263) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:54:48 (1787853288) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:54:52 (1787853292) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 13:54:55 (1787853295) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9fc1c52e3000.stats=clear osc.lustre-OST0001-osc-ffff9fc1c52e3000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:55:03 (1787853303) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:55:14 (1787853314) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:55:22 (1787853322) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:55:26 (1787853326) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:55:35 (1787853335) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 13:55:44 (1787853344) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:55:50 (1787853350) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:55:57 (1787853357) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:56:05 (1787853365) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:56:15 (1787853375) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:56:21 (1787853381) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.28704 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886361 s, 11.8 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:56:29 (1787853389) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.255185 s, 41.1 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:56:38 (1787853398) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:56:46 (1787853406) Stopping /mnt/lustre-ost1 (opts:) on oleg339-server oleg339-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fc1c52e3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fc1c52e3000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg339-server: oleg339-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg339-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg339-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg339-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg339-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg339-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg339-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg339-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg339-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg339-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg339-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg339-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg339-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg339-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg339-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg339-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg339-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg339-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg339-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg339-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg339-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg339-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg339-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg339-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg339-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg339-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg339-server oleg339-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fc1c52e3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fc1c52e3000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg339-server: oleg339-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg339-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (126s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:58:53 (1787853533) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.505906 s, 18.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg339-server oleg339-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fc1c52e3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fc1c52e3000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg339-server: oleg339-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg339-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:59:33 (1787853573) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg339-server oleg339-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fc1c52e3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fc1c52e3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg339-server: oleg339-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg339-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg339-server oleg339-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fc1c52e3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fc1c52e3000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg339-server: oleg339-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg339-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (75s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:00:48 (1787853648) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.38108 s, 22.0 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 14:00 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.860s user 0m0.001s sys 0m0.201s Stopping /mnt/lustre-ost1 (opts:) on oleg339-server oleg339-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fc1c52e3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fc1c52e3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.889s user 0m0.002s sys 0m0.194s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg339-server: oleg339-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg339-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg339-server oleg339-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fc1c52e3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fc1c52e3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.007s user 0m0.000s sys 0m0.201s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg339-server: oleg339-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg339-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fc1c52e3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fc1c52e3000.ost_server_uuid in FULL state after 0 sec PASS 33b (86s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:02:14 (1787853734) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:02:18 (1787853738) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:02:21 (1787853741) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 14:02:24 (1787853744) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 14:02:31 (1787853751) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.765267 s, 16.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:02:45 (1787853765) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255558 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282348 s, 37.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:02:52 (1787853772) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:03:00 (1787853780) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00202032 s, 507 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000927433 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00408145 s, 251 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0023855 s, 429 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 14:03:10 (1787853790) create test files with size 15759027 .. 87ffaaa05461b970374b066c1b12deba - 372641ff8b2fd9e40cc143441df428c8 - ed3ce72c8459c725ef0a09c2723a7011 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg339-client.virtnet /mnt/lustre (opts:) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 14:03:33 (1787853813) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:03:46 (1787853826) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:03:53 (1787853833) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608325 s, 17.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255747 s, 41.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 14:04:02 (1787853842) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0913253 s, 23.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.23803 s, 17.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.189831 s, 11.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 14:04:16 (1787853856) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 14:04:20 (1787853860) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:04:24 (1787853864) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393493 s, 26.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 14:04:33 (1787853873) llite.lustre-ffff9fc1c9adf000.enable_compression=1 striped dir -i1 -c2 -H all_char /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 14:04:41 (1787853881) llite.lustre-ffff9fc1c9adf000.enable_compression=1 striped dir -i1 -c2 -H crush /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532345 s, 19.7 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214414 s, 48.9 MB/s **verify components PASS 43d (8s) == sanity-flr test 44a: lfs mirror split check =========== 14:04:49 (1787853889) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.359048 s, 8.8 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:05:00 (1787853900) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xda:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xda:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:05:09 (1787853909) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.353827 s, 29.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg339-client.virtnet /mnt/lustre (opts:) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:05:19 (1787853919) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9fc1c9039000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.304598 s, 34.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg339-client.virtnet /mnt/lustre (opts:) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9fc1c4e25000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:05:33 (1787853933) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (33s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:06:06 (1787853966) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:06:36 (1787853996) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 14:06:39 (1787853999) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.398157 s, 7.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.146248 s, 21.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa4a621e8 (nosync,stale) Mirror 2: 0xa4a621e8 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2ec4b8f503367ac1ba1a654e60808c11 - mirror 1 checksum: a8c11093032ed3913292fd9723768f1f - mirror 2 checksum: 2ec4b8f503367ac1ba1a654e60808c11 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T14:06:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T14:06:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2ec4b8f503367ac1ba1a654e60808c11 - mirror 1 checksum: a8c11093032ed3913292fd9723768f1f - mirror 2 checksum: a8c11093032ed3913292fd9723768f1f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T14:06:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T14:06:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 14:06:48 (1787854008) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387068 s, 27.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:06:55 (1787854015) Failing mds1 on oleg339-server Stopping /mnt/lustre-mds1 (opts:) on oleg339-server 14:07:06 (1787854026) shut down facet: mds1 facet_host: oleg339-server facet_failover_host: oleg339-server Failover mds1 to oleg339-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 14:07:39 (1787854059) targets are mounted 14:07:39 (1787854059) facet_failover done oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474696 s, 22.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } PASS 50A (58s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:07:54 (1787854074) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0699306 s, 293 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0196132 s, 1.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:08:03 (1787854083) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.05848 s, 350 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.01276 s, 1.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:08:12 (1787854092) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00804698 s, 2.0 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:08:19 (1787854099) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0346456 s, 591 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (6s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:08:26 (1787854106) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00996935 s, 411 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0421461 s, 97.2 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:08:33 (1787854113) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:08:56 (1787854136) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:09:23 (1787854163) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:09:46 (1787854186) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:10:09 (1787854209) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 14:10:17 (1787854217) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 58835 mirror split pid 58836 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58835 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58836 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 14:11:25 (1787854285) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 4286 000002 1787854287.059019 write 0x01bfa4 thru 0x020790 (0x47ed bytes) truncating to largest ever: 0x33f23 000004 1787854287.069127 trunc from 0x020791 to 0x033f22 (0x13792 bytes) 000007 1787854287.087739 trunc from 0x015235 to 0x0204f6 (0xb2c2 bytes) 000010 1787854287.115700 punch from 0x0234b3 to 0x02c7ef (0x933d bytes) 000012 1787854287.125794 write 0x002ae8 thru 0x00ddbf (0xb2d8 bytes) 000013 1787854287.144868 trunc from 0x00fe1c to 0x01dae0 (0xdcc5 bytes) 000014 1787854287.149106 trunc from 0x029057 to 0x033f22 (0xaecc bytes) 000016 1787854287.156875 read 0x005000 thru 0x009a02 (0x4a03 bytes) 000017 1787854287.157771 mapread 0x005cd7 thru 0x011303 (0xb62d bytes) 000018 1787854287.215165 MIRROR_EXTEND to 2 mirrors 000019 1787854287.970842 write 0x0349cf thru 0x03ffff (0xb631 bytes) zero_range to largest ever: 0x3f7d8 000020 1787854288.048437 trunc from 0x03618f to 0x03f7d7 (0x9649 bytes) 000021 1787854288.078364 mapread 0x01fe54 thru 0x027e15 (0x7fc2 bytes) 000023 1787854288.109760 trunc from 0x0266d0 to 0x02f4cf (0x8e00 bytes) 000025 1787854288.117913 mapwrite 0x0305e9 thru 0x03cf68 (0xc980 bytes) 000026 1787854288.164798 write 0x03cca3 thru 0x03f2b0 (0x260e bytes) 000028 1787854288.167000 write 0x00c8b2 thru 0x01b7c7 (0xef16 bytes) 000029 1787854288.173525 fallocate 0x00a08b thru 0x00fa7d (0x59f3 bytes) 000030 1787854288.178950 mapread 0x009069 thru 0x014d49 (0xbce1 bytes) 000031 1787854288.188890 trunc from 0x03e438 to 0x03ffff (0x1bc8 bytes) 000032 1787854288.207786 mapread 0x03e566 thru 0x03ffff (0x1a9a bytes) 000034 1787854288.225857 read 0x019f2e thru 0x01a2e0 (0x3b3 bytes) 000035 1787854288.227159 mapread 0x03e320 thru 0x03ffff (0x1ce0 bytes) 000036 1787854288.228250 mapread 0x00e7b2 thru 0x01dc16 (0xf465 bytes) 000038 1787854288.230202 punch from 0x03cd82 to 0x03ffff (0x327e bytes) 000039 1787854288.246177 mapwrite 0x012061 thru 0x01ea80 (0xca20 bytes) 000040 1787854288.283725 fallocate 0x032d73 thru 0x0388a7 (0x5b35 bytes) 000041 1787854288.298947 MIRROR_RESYNC 2 mirrors 000042 1787854289.097527 fallocate 0x01cb90 thru 0x02c489 (0xf8fa bytes) 000043 1787854289.137657 trunc from 0x040000 to 0x02df91 (0x1206e bytes) 000044 1787854289.173176 trunc from 0x00aa4e to 0x012992 (0x7f45 bytes) 000045 1787854289.178302 mapwrite 0x03b3c0 thru 0x03ffff (0x4c40 bytes) 000046 1787854289.276196 punch from 0x021271 to 0x026b50 (0x58e0 bytes) 000047 1787854289.281694 mapwrite 0x01b890 thru 0x02b160 (0xf8d1 bytes) 000048 1787854289.353472 mapread 0x019c57 thru 0x01ec4a (0x4ff4 bytes) 000049 1787854289.356860 mapread 0x0023bf thru 0x01118a (0xedcc bytes) 000050 1787854289.391082 trunc from 0x040000 to 0x00dda3 (0x3225c bytes) 000051 1787854289.422310 mapread 0x00dcbc thru 0x00dda3 (0xe8 bytes) 000052 1787854289.426089 mapread 0x008a1d thru 0x00dda3 (0x5387 bytes) 000053 1787854289.434444 trunc from 0x001372 to 0x00dda3 (0xca32 bytes) 000054 1787854289.441681 punch from 0x004e0e to 0x0068c4 (0x1ab7 bytes) 000055 1787854289.454644 mapwrite 0x0285b3 thru 0x02ee23 (0x6871 bytes) 000056 1787854289.508949 mapwrite 0x0223d5 thru 0x02c22a (0x9e56 bytes) 000057 1787854289.534934 mapwrite 0x019a41 thru 0x020ae4 (0x70a4 bytes) 000058 1787854289.591340 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ee24): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee24): Mirror 2: 0x439f8b3 (stale) CRC-32 checksum value for chunk [0, 0x2ee24): Mirror 1: 0x9ed6ec37 Mirror 3: 0x9ed6ec37 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee24: skipped 000059 1787854290.434729 mapwrite 0x03c066 thru 0x03ffff (0x3f9a bytes) 000061 1787854290.557084 trunc from 0x03e33d to 0x03ffff (0x1cc3 bytes) 000062 1787854290.562533 trunc from 0x040000 to 0x02e6c5 (0x1193a bytes) 000064 1787854290.579849 punch from 0x0145bd to 0x023bbf (0xf603 bytes) 000065 1787854290.585512 mapwrite 0x035ac5 thru 0x035edd (0x419 bytes) 000066 1787854290.637466 trunc from 0x03260a to 0x033072 (0xa69 bytes) 000068 1787854290.645751 read 0x02e3f2 thru 0x035edd (0x7aec bytes) 000069 1787854290.656333 read 0x0270c4 thru 0x02c662 (0x559f bytes) 000070 1787854290.669357 trunc from 0x035ede to 0x0203f4 (0x15ae9 bytes) 000071 1787854290.693832 punch from 0x0093da to 0x0131a7 (0x9dce bytes) 000072 1787854290.699133 mapread 0x004b20 thru 0x0129f5 (0xded6 bytes) 000073 1787854290.760572 fallocate 0x00818e thru 0x00d83b (0x56ae bytes) 000074 1787854290.774334 punch from 0x010cd3 to 0x016e05 (0x6133 bytes) 000075 1787854290.784572 read 0x017eb9 thru 0x0203f4 (0x853c bytes) 000076 1787854290.785279 punch from 0x01e049 to 0x0203f4 (0x23ac bytes) fallocating to largest ever: 0x40000 000077 1787854290.790819 fallocate 0x03a6e5 thru 0x03ffff (0x591b bytes) 000078 1787854290.797752 trunc from 0x0203f5 to 0x0342d7 (0x13ee3 bytes) 000079 1787854290.837876 trunc from 0x01f99d to 0x024da7 (0x540b bytes) 000080 1787854290.851740 read 0x03186b thru 0x0342d7 (0x2a6d bytes) 000081 1787854290.896966 trunc from 0x0342d8 to 0x03af9e (0x6cc7 bytes) 000082 1787854290.933838 fallocate 0x0202f4 thru 0x021320 (0x102d bytes) 000083 1787854290.941836 trunc from 0x03af9f to 0x01a770 (0x2082e bytes) 000084 1787854290.976832 trunc from 0x01a771 to 0x037f1f (0x1d7af bytes) 000085 1787854291.010830 punch from 0x032487 to 0x037f1f (0x5a99 bytes) 000086 1787854291.022329 trunc from 0x037f20 to 0x01038a (0x27b95 bytes) 000087 1787854291.048713 fallocate 0x0379d1 thru 0x03ffff (0x862f bytes) 000088 1787854291.057248 mapread 0x0083b8 thru 0x00fb45 (0x778e bytes) 000089 1787854291.059364 write 0x028143 thru 0x02a11b (0x1fd9 bytes) 000090 1787854291.062242 mapwrite 0x019def thru 0x025759 (0xb96b bytes) 000091 1787854291.156387 punch from 0x002123 to 0x00af7a (0x8e58 bytes) 000092 1787854291.163213 mapread 0x001cd7 thru 0x004004 (0x232e bytes) 000093 1787854291.180060 trunc from 0x00595d to 0x0097ec (0x3e90 bytes) 000094 1787854291.192156 mapread 0x015f57 thru 0x020cad (0xad57 bytes) 000095 1787854291.215503 trunc from 0x01bab8 to 0x02a11b (0xe664 bytes) 000097 1787854291.226047 mapwrite 0x038688 thru 0x03a40e (0x1d87 bytes) 000098 1787854291.304558 mapwrite 0x0235ff thru 0x02ad41 (0x7743 bytes) 000099 1787854291.431785 mapread 0x01369f thru 0x015f3c (0x289e bytes) 000100 1787854291.446196 trunc from 0x03a40f to 0x02682c (0x13be2 bytes) 000101 1787854291.463763 write 0x0099d6 thru 0x00da5b (0x4086 bytes) 000102 1787854291.470466 fallocate 0x01b362 thru 0x02a393 (0xf032 bytes) 000103 1787854291.476578 write 0x036aaf thru 0x03ffff (0x9551 bytes) 000104 1787854291.483688 MIRROR_RESYNC 3 mirrors 000106 1787854292.655869 write 0x038b08 thru 0x03bd54 (0x324d bytes) 000107 1787854292.713219 fallocate 0x003cac thru 0x008615 (0x496a bytes) 000108 1787854292.727793 mapwrite 0x03b90a thru 0x03ffff (0x46f6 bytes) 000109 1787854292.775876 trunc from 0x0342f0 to 0x03722b (0x2f3c bytes) 000110 1787854292.781775 read 0x01aa3e thru 0x02143f (0x6a02 bytes) 000111 1787854292.793046 fallocate 0x029935 thru 0x0369cc (0xd098 bytes) 000112 1787854292.800658 mapwrite 0x026ee0 thru 0x036824 (0xf945 bytes) 000113 1787854292.860508 fallocate 0x02e03a thru 0x031e14 (0x3ddb bytes) 000115 1787854292.866335 fallocate 0x011355 thru 0x020dea (0xfa96 bytes) 000118 1787854292.875646 write 0x025565 thru 0x02d776 (0x8212 bytes) 000119 1787854292.876941 punch from 0x028e6b to 0x02c44a (0x35e0 bytes) 000120 1787854292.911865 trunc from 0x040000 to 0x02b3bb (0x14c44 bytes) 000121 1787854292.942664 mapread 0x029d64 thru 0x02b3bb (0x1658 bytes) 000122 1787854292.957342 fallocate 0x02245e thru 0x02390e (0x14b1 bytes) 000123 1787854292.973893 trunc from 0x02b3bc to 0x02158e (0x9e2d bytes) 000124 1787854293.003672 trunc from 0x02158f to 0x018ecd (0x86c1 bytes) 000125 1787854293.040871 punch from 0x002342 to 0x011bba (0xf879 bytes) 000126 1787854293.047867 trunc from 0x018ece to 0x00e05a (0xae73 bytes) 000127 1787854293.066742 mapwrite 0x01c84d thru 0x02b4e8 (0xec9c bytes) 000128 1787854293.154899 punch from 0x009935 to 0x00ae8a (0x1556 bytes) 000129 1787854293.159044 trunc from 0x00c760 to 0x0141de (0x7a7f bytes) 000130 1787854293.173664 read 0x01da18 thru 0x02b4e8 (0xdad1 bytes) 000131 1787854293.174711 mapwrite 0x0246c3 thru 0x026086 (0x19c4 bytes) 000132 1787854293.200065 read 0x00fb52 thru 0x012254 (0x2703 bytes) 000133 1787854293.219979 punch from 0x002e86 to 0x007fc2 (0x513d bytes) 000134 1787854293.229223 fallocate 0x01ed97 thru 0x026e65 (0x80cf bytes) 000135 1787854293.248230 MIRROR_RESYNC 3 mirrors 000136 1787854293.868515 read 0x004748 thru 0x00eb09 (0xa3c2 bytes) 000137 1787854293.881270 fallocate 0x00dc49 thru 0x00eed9 (0x1291 bytes) 000138 1787854293.936060 trunc from 0x02b4e9 to 0x03e725 (0x1323d bytes) 000140 1787854294.015910 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e726): Mirror 2: 0xdb81dd3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e726): Mirror 2: 0xdb81dd3e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e726: skipped 000141 1787854294.451713 mapwrite 0x00e3d5 thru 0x018786 (0xa3b2 bytes) 000142 1787854294.533300 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) 000143 1787854294.722939 mapwrite 0x02ce95 thru 0x030499 (0x3605 bytes) 000144 1787854294.810515 mapread 0x02b50f thru 0x036de1 (0xb8d3 bytes) 000145 1787854294.813164 trunc from 0x03e726 to 0x033ae5 (0xac40 bytes) 000146 1787854294.853274 punch from 0x01ae19 to 0x01c3c9 (0x15b1 bytes) 000147 1787854294.861750 punch from 0x001dac to 0x002443 (0x698 bytes) 000148 1787854294.870752 trunc from 0x033ae6 to 0x00974c (0x2a399 bytes) 000149 1787854294.889660 trunc from 0x00974d to 0x0170ae (0xd962 bytes) 000150 1787854294.908285 trunc from 0x0170af to 0x00e6c9 (0x89e5 bytes) 000151 1787854294.949469 fallocate 0x01ac39 thru 0x01e09a (0x3462 bytes) 000154 1787854294.956079 punch from 0x01cced to 0x01e09a (0x13ae bytes) 000155 1787854294.962731 fallocate 0x032eac thru 0x03ffff (0xd154 bytes) 000156 1787854294.970305 punch from 0x016aa8 to 0x0190dd (0x2636 bytes) 000157 1787854294.975899 trunc from 0x02f6e7 to 0x0316da (0x1ff4 bytes) 000158 1787854294.982138 trunc from 0x00f46e to 0x01758f (0x8122 bytes) 000159 1787854294.988723 trunc from 0x01f12f to 0x022e3d (0x3d0f bytes) 000160 1787854294.999959 mapread 0x0257b9 thru 0x026f2d (0x1775 bytes) 000161 1787854295.021881 mapwrite 0x031ad5 thru 0x03ffff (0xe52b bytes) 000162 1787854295.072165 mapwrite 0x03a533 thru 0x03ffff (0x5acd bytes) 000163 1787854295.119036 write 0x007bb3 thru 0x009b6c (0x1fba bytes) 000164 1787854295.137081 mapwrite 0x0212d6 thru 0x029f9e (0x8cc9 bytes) 000165 1787854295.182781 mapread 0x038cfe thru 0x03cef9 (0x41fc bytes) 000166 1787854295.188800 fallocate 0x012850 thru 0x01b903 (0x90b4 bytes) 000167 1787854295.193973 punch from 0x00226e to 0x008959 (0x66ec bytes) 000168 1787854295.202118 fallocate 0x011652 thru 0x01d587 (0xbf36 bytes) 000169 1787854295.205637 mapwrite 0x028a3e thru 0x02dceb (0x52ae bytes) 000170 1787854295.274977 mapread 0x0065e7 thru 0x010264 (0x9c7e bytes) 000171 1787854295.291840 mapwrite 0x03af85 thru 0x03ffff (0x507b bytes) 000174 1787854295.311245 mapwrite 0x010a95 thru 0x0117a4 (0xd10 bytes) 000175 1787854295.358370 read 0x011a6e thru 0x01c2f4 (0xa887 bytes) 000176 1787854295.368202 mapwrite 0x032b39 thru 0x03b5d5 (0x8a9d bytes) 000177 1787854295.391866 mapwrite 0x039155 thru 0x03bf57 (0x2e03 bytes) 000179 1787854295.416748 write 0x010e54 thru 0x01cd67 (0xbf14 bytes) 000180 1787854295.420852 fallocate 0x005e47 thru 0x011d27 (0xbee1 bytes) 000181 1787854295.428361 mapwrite 0x027d81 thru 0x037a9b (0xfd1b bytes) 000182 1787854295.473203 trunc from 0x004d7e to 0x00e372 (0x95f5 bytes) 000183 1787854295.479379 punch from 0x01db70 to 0x021fbe (0x444f bytes) 000184 1787854295.484821 mapread 0x00577c thru 0x00f4f3 (0x9d78 bytes) 000185 1787854295.493582 mapread 0x02a3d5 thru 0x02f19a (0x4dc6 bytes) 000187 1787854295.494905 read 0x03cc37 thru 0x03ffff (0x33c9 bytes) 000188 1787854295.495494 mapwrite 0x023a86 thru 0x0314b2 (0xda2d bytes) 000189 1787854295.528766 write 0x026320 thru 0x0297bc (0x349d bytes) 000190 1787854295.530312 mapread 0x00d336 thru 0x014f31 (0x7bfc bytes) 000191 1787854295.531949 fallocate 0x01e801 thru 0x02e403 (0xfc03 bytes) 000192 1787854295.538141 write 0x039a6f thru 0x03c605 (0x2b97 bytes) 000193 1787854295.539216 mapread 0x02e542 thru 0x03bbc8 (0xd687 bytes) 000194 1787854295.541189 write 0x011074 thru 0x015ebb (0x4e48 bytes) 000195 1787854295.543080 mapread 0x02289a thru 0x02c124 (0x988b bytes) 000196 1787854295.546006 write 0x03b85f thru 0x03dc56 (0x23f8 bytes) 000197 1787854295.547382 trunc from 0x040000 to 0x01479b (0x2b864 bytes) 000198 1787854295.565286 write 0x00026a thru 0x009557 (0x92ee bytes) 000199 1787854295.571264 fallocate 0x03fa8f thru 0x03ffff (0x571 bytes) 000200 1787854295.576837 fallocate 0x02bc5c thru 0x03a25d (0xe602 bytes) 000202 1787854295.582176 trunc from 0x01479c to 0x013b79 (0xc22 bytes) 000204 1787854295.609037 mapread 0x0096bf thru 0x010bb4 (0x74f6 bytes) 000205 1787854295.616173 punch from 0x005d3e to 0x0125fe (0xc8c1 bytes) 000206 1787854295.659446 trunc from 0x00a490 to 0x013b79 (0x96ea bytes) 000207 1787854295.666969 fallocate 0x03794d thru 0x03ffff (0x86b3 bytes) 000208 1787854295.672604 fallocate 0x03292f thru 0x03ffff (0xd6d1 bytes) 000209 1787854295.677997 trunc from 0x040000 to 0x00f65a (0x309a5 bytes) 000211 1787854295.695427 read 0x00806d thru 0x008bfa (0xb8e bytes) 000212 1787854295.711172 mapwrite 0x014bcd thru 0x01bc7c (0x70b0 bytes) 000213 1787854295.789019 write 0x00dc90 thru 0x0121d9 (0x454a bytes) 000214 1787854295.792743 mapread 0x015a39 thru 0x0165ce (0xb96 bytes) 000215 1787854295.796298 mapread 0x015d26 thru 0x01bc7c (0x5f57 bytes) 000216 1787854295.797459 read 0x001d83 thru 0x0039b8 (0x1c36 bytes) 000217 1787854295.801543 read 0x01099b thru 0x01bc7c (0xb2e2 bytes) 000218 1787854295.811169 trunc from 0x019949 to 0x01bc7c (0x2334 bytes) 000219 1787854295.826172 mapwrite 0x03068e thru 0x039b9a (0x950d bytes) 000220 1787854295.933621 trunc from 0x039b9b to 0x02560a (0x14590 bytes) 000221 1787854295.973808 mapwrite 0x03c247 thru 0x03ffff (0x3db9 bytes) 000222 1787854296.088652 mapwrite 0x01612d thru 0x01851a (0x23ee bytes) 000223 1787854296.125240 punch from 0x033ab1 to 0x03e4e0 (0xaa30 bytes) 000224 1787854296.140387 fallocate 0x00a7d6 thru 0x00ee5f (0x468a bytes) 000225 1787854296.150385 punch from 0x033ac4 to 0x034839 (0xd76 bytes) 000226 1787854296.159647 trunc from 0x013b0e to 0x0171e7 (0x36da bytes) 000227 1787854296.184876 read 0x0020c9 thru 0x00cf0b (0xae43 bytes) 000229 1787854296.186157 trunc from 0x01db3a to 0x02ac5b (0xd122 bytes) 000231 1787854296.207383 mapwrite 0x00ffcd thru 0x012614 (0x2648 bytes) 000232 1787854296.255313 trunc from 0x040000 to 0x003b39 (0x3c4c6 bytes) 000233 1787854296.297827 mapwrite 0x01b2c3 thru 0x029fe2 (0xed20 bytes) 000236 1787854296.501226 mapread 0x006768 thru 0x00c6bf (0x5f58 bytes) 000237 1787854296.513707 MIRROR_EXTEND to 2 mirrors 000238 1787854297.586304 trunc from 0x01f058 to 0x029f7a (0xaf23 bytes) 000239 1787854297.653026 fallocate 0x01e668 thru 0x026ebd (0x8856 bytes) 000240 1787854297.663466 MIRROR_RESYNC 2 mirrors 000241 1787854298.698917 punch from 0x00a0d7 to 0x01694e (0xc878 bytes) 000242 1787854298.732937 write 0x0252b8 thru 0x027241 (0x1f8a bytes) 000243 1787854298.742081 read 0x00c04c thru 0x00d592 (0x1547 bytes) 000244 1787854298.755608 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) 000245 1787854298.946672 read 0x001d90 thru 0x00da62 (0xbcd3 bytes) 000246 1787854298.964703 write 0x03913d thru 0x03aad4 (0x1998 bytes) 000247 1787854298.971724 punch from 0x022a2f to 0x02687d (0x3e4f bytes) 000248 1787854298.979393 fallocate 0x03f1b5 thru 0x03ffff (0xe4b bytes) 000251 1787854298.989353 trunc from 0x040000 to 0x025210 (0x1adef bytes) 000252 1787854299.008916 punch from 0x012f2e to 0x013bbc (0xc8f bytes) 000253 1787854299.014244 fallocate 0x0364d9 thru 0x03b668 (0x5190 bytes) 000254 1787854299.023146 MIRROR_EXTEND to 2 mirrors 000255 1787854300.022043 trunc from 0x025211 to 0x035a93 (0x10883 bytes) 000256 1787854300.135315 trunc from 0x003ad2 to 0x00be07 (0x8336 bytes) 000257 1787854300.153235 write 0x01e5ae thru 0x025415 (0x6e68 bytes) 000258 1787854300.165115 mapread 0x0237f9 thru 0x02acc7 (0x74cf bytes) 000259 1787854300.205202 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) 000260 1787854300.367871 write 0x01da09 thru 0x023461 (0x5a59 bytes) 000261 1787854300.409873 fallocate 0x03aa9e thru 0x03ffff (0x5562 bytes) 000262 1787854300.417701 trunc from 0x03dcb2 to 0x03ffff (0x234e bytes) 000263 1787854300.430456 trunc from 0x027b36 to 0x02877a (0xc45 bytes) 000264 1787854300.442496 mapread 0x023d61 thru 0x02a21e (0x64be bytes) 000265 1787854300.501835 read 0x01278c thru 0x01609d (0x3912 bytes) 000267 1787854300.538691 mapwrite 0x023436 thru 0x02bd70 (0x893b bytes) 000268 1787854300.582290 punch from 0x01175c to 0x01bf5b (0xa800 bytes) 000269 1787854300.594679 punch from 0x00b903 to 0x019258 (0xd956 bytes) 000270 1787854300.601844 mapwrite 0x02fe7d thru 0x0345b9 (0x473d bytes) 000273 1787854300.656955 trunc from 0x040000 to 0x02b7f9 (0x14806 bytes) 000275 1787854300.686388 mapread 0x01473d thru 0x01a68f (0x5f53 bytes) 000277 1787854300.715943 trunc from 0x02b7fa to 0x00ebfd (0x1cbfc bytes) 000278 1787854300.736083 read 0x001c41 thru 0x0087e0 (0x6ba0 bytes) 000279 1787854300.746642 write 0x01ad40 thru 0x01b92f (0xbf0 bytes) 000280 1787854300.748358 punch from 0x013486 to 0x018426 (0x4fa1 bytes) 000282 1787854300.755107 mapwrite 0x024d18 thru 0x028859 (0x3b42 bytes) 000283 1787854300.812839 mapwrite 0x03c696 thru 0x03ffff (0x396a bytes) 000284 1787854300.873527 fallocate 0x015a6e thru 0x023344 (0xd8d7 bytes) 000285 1787854300.879199 fallocate 0x03aa97 thru 0x03ffff (0x5569 bytes) 000287 1787854300.885298 trunc from 0x001315 to 0x00f6de (0xe3ca bytes) 000288 1787854300.891246 punch from 0x03c6d7 to 0x03ffff (0x3929 bytes) 000289 1787854300.896842 fallocate 0x00c896 thru 0x01c388 (0xfaf3 bytes) 000291 1787854300.905180 punch from 0x01f4e0 to 0x02cd06 (0xd827 bytes) 000292 1787854300.911715 mapread 0x031e39 thru 0x037eea (0x60b2 bytes) 000293 1787854300.913280 write 0x00218c thru 0x00669c (0x4511 bytes) 000294 1787854300.923143 read 0x017db5 thru 0x026d90 (0xefdc bytes) 000295 1787854300.933160 trunc from 0x040000 to 0x023f11 (0x1c0ee bytes) 000296 1787854300.950316 mapread 0x02247c thru 0x023f11 (0x1a96 bytes) 000297 1787854300.956464 MIRROR_EXTEND to 2 mirrors 000298 1787854302.070461 mapwrite 0x00ffff thru 0x01d2f5 (0xd2f7 bytes) 000299 1787854302.163725 punch from 0x00a769 to 0x0127ae (0x8046 bytes) 000300 1787854302.169904 mapread 0x0144e1 thru 0x0221e0 (0xdd00 bytes) 000301 1787854302.176582 read 0x00de0c thru 0x019d92 (0xbf87 bytes) 000302 1787854302.187045 read 0x0012a7 thru 0x00ea69 (0xd7c3 bytes) 000303 1787854302.207389 punch from 0x013060 to 0x0222bb (0xf25c bytes) 000304 1787854302.214998 fallocate 0x03be20 thru 0x03ffff (0x41e0 bytes) 000305 1787854302.226459 punch from 0x032a32 to 0x03f9e0 (0xcfaf bytes) 000306 1787854302.236765 fallocate 0x029f46 thru 0x02d5bb (0x3676 bytes) 000307 1787854302.243680 read 0x032d13 thru 0x03f45e (0xc74c bytes) 000308 1787854302.261289 punch from 0x038fef to 0x03c1a3 (0x31b5 bytes) 000309 1787854302.267393 trunc from 0x040000 to 0x0243a6 (0x1bc59 bytes) 000310 1787854302.308456 trunc from 0x017c1d to 0x01f988 (0x7d6c bytes) 000311 1787854302.320165 read 0x0135d7 thru 0x016f7e (0x39a8 bytes) 000312 1787854302.323566 fallocate 0x0024f9 thru 0x006aae (0x45b6 bytes) 000313 1787854302.337340 mapwrite 0x019f6c thru 0x01de36 (0x3ecb bytes) 000314 1787854302.382919 trunc from 0x0111cb to 0x01e697 (0xd4cd bytes) 000315 1787854302.389565 mapwrite 0x00ba22 thru 0x01a4c2 (0xeaa1 bytes) 000316 1787854302.499985 read 0x016bf5 thru 0x018abd (0x1ec9 bytes) 000317 1787854302.501388 fallocate 0x0182ed thru 0x025122 (0xce36 bytes) 000318 1787854302.511446 mapwrite 0x03d8ae thru 0x03ffff (0x2752 bytes) 000319 1787854302.590882 mapread 0x031149 thru 0x0336f1 (0x25a9 bytes) 000320 1787854302.595772 trunc from 0x040000 to 0x02f548 (0x10ab7 bytes) 000321 1787854302.619101 punch from 0x026e3a to 0x02f548 (0x870f bytes) 000322 1787854302.627648 fallocate 0x03e668 thru 0x03ffff (0x1998 bytes) 000324 1787854302.634057 mapwrite 0x025c84 thru 0x034cad (0xf02a bytes) 000325 1787854302.752276 trunc from 0x040000 to 0x02d745 (0x128ba bytes) 000326 1787854302.771385 mapread 0x01a06f thru 0x01b489 (0x141b bytes) 000327 1787854302.772478 mapread 0x003918 thru 0x005067 (0x1750 bytes) 000328 1787854302.774267 trunc from 0x01a760 to 0x023422 (0x8cc3 bytes) 000329 1787854302.781920 trunc from 0x02d746 to 0x025b46 (0x7bff bytes) 000330 1787854302.811000 MIRROR_RESYNC 2 mirrors 000331 1787854303.636074 write 0x009dc8 thru 0x00cfc9 (0x3202 bytes) 000333 1787854303.674233 mapread 0x0190f3 thru 0x02266a (0x9578 bytes) 000334 1787854303.697382 mapread 0x018589 thru 0x022b7e (0xa5f6 bytes) 000335 1787854303.699338 trunc from 0x025b47 to 0x03a08a (0x14544 bytes) 000336 1787854303.737598 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) 000337 1787854303.915013 write 0x03095a thru 0x035965 (0x500c bytes) 000338 1787854303.937089 fallocate 0x03a699 thru 0x03ffff (0x5967 bytes) 000340 1787854303.946625 punch from 0x032e10 to 0x03bf19 (0x910a bytes) 000341 1787854303.968950 trunc from 0x030a5f to 0x035348 (0x48ea bytes) 000342 1787854303.977616 mapwrite 0x00dda5 thru 0x01b0f2 (0xd34e bytes) 000343 1787854304.045287 mapread 0x01128a thru 0x01b436 (0xa1ad bytes) 000344 1787854304.047353 mapwrite 0x015f34 thru 0x01f1f6 (0x92c3 bytes) 000345 1787854304.077966 write 0x038aad thru 0x03e9d2 (0x5f26 bytes) 000346 1787854304.090061 read 0x002d53 thru 0x00ae67 (0x8115 bytes) 000347 1787854304.106956 read 0x03c9c7 thru 0x03ffff (0x3639 bytes) 000348 1787854304.118684 read 0x0300dc thru 0x03581d (0x5742 bytes) 000349 1787854304.119646 read 0x037fce thru 0x03ffff (0x8032 bytes) 000350 1787854304.120788 read 0x0074c3 thru 0x015229 (0xdd67 bytes) 000351 1787854304.122192 write 0x00d985 thru 0x019a36 (0xc0b2 bytes) 000352 1787854304.124701 write 0x00a36e thru 0x00fa96 (0x5729 bytes) 000353 1787854304.126664 mapwrite 0x025b23 thru 0x026dbd (0x129b bytes) 000354 1787854304.172010 mapwrite 0x01858f thru 0x0279a4 (0xf416 bytes) 000355 1787854304.215104 mapwrite 0x00de6a thru 0x01bb55 (0xdcec bytes) 000356 1787854304.255245 read 0x03bba6 thru 0x03d7ad (0x1c08 bytes) 000358 1787854304.256686 mapread 0x011ad8 thru 0x01b698 (0x9bc1 bytes) 000359 1787854304.261067 trunc from 0x040000 to 0x013255 (0x2cdaa bytes) 000360 1787854304.290712 write 0x004b35 thru 0x00807a (0x3546 bytes) 000361 1787854304.294338 mapwrite 0x002bb6 thru 0x00f4b5 (0xc900 bytes) 000362 1787854304.314513 trunc from 0x013256 to 0x020b57 (0xd902 bytes) 000363 1787854304.367925 punch from 0x006fce to 0x00a1ff (0x3232 bytes) 000364 1787854304.377088 mapread 0x008119 thru 0x013d22 (0xbc0a bytes) 000365 1787854304.402166 mapread 0x01e5d2 thru 0x020b57 (0x2586 bytes) 000366 1787854304.404763 fallocate 0x03f88b thru 0x03ffff (0x775 bytes) 000367 1787854304.415234 mapwrite 0x003135 thru 0x00eeee (0xbdba bytes) 000368 1787854304.469780 MIRROR_EXTEND to 2 mirrors 000369 1787854305.595739 read 0x01110f thru 0x01e25a (0xd14c bytes) 000370 1787854305.610285 read 0x01095d thru 0x016224 (0x58c8 bytes) 000371 1787854305.632589 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) 000372 1787854305.783374 read 0x005623 thru 0x00994d (0x432b bytes) 000373 1787854305.798084 write 0x03f9d9 thru 0x03ffff (0x627 bytes) 000374 1787854305.818965 punch from 0x013663 to 0x015197 (0x1b35 bytes) 000375 1787854305.827035 mapwrite 0x03252f thru 0x033a56 (0x1528 bytes) 000376 1787854305.882575 read 0x013981 thru 0x01c85e (0x8ede bytes) 000377 1787854305.898857 trunc from 0x02aefd to 0x02d047 (0x214b bytes) 000378 1787854305.906060 read 0x0282a2 thru 0x02d0e2 (0x4e41 bytes) 000379 1787854305.923271 write 0x0103ba thru 0x016308 (0x5f4f bytes) 000380 1787854305.924747 mapwrite 0x02f493 thru 0x03c27e (0xcdec bytes) 000382 1787854305.978529 read 0x019d5e thru 0x022f57 (0x91fa bytes) 000383 1787854305.979823 fallocate 0x00c44b thru 0x011bb7 (0x576d bytes) 000384 1787854305.994315 trunc from 0x040000 to 0x02f608 (0x109f7 bytes) 000385 1787854306.031514 read 0x014574 thru 0x01d765 (0x91f2 bytes) 000386 1787854306.034347 trunc from 0x02f609 to 0x02183c (0xddcc bytes) 000387 1787854306.058778 punch from 0x0184ff to 0x02183c (0x933e bytes) 000388 1787854306.065542 mapwrite 0x02fd0c thru 0x03f608 (0xf8fd bytes) 000390 1787854306.143908 trunc from 0x03f609 to 0x0374ea (0x811e bytes) 000391 1787854306.164252 trunc from 0x0239e3 to 0x029501 (0x5b1f bytes) 000392 1787854306.170752 mapread 0x0336d0 thru 0x0374ea (0x3e1b bytes) 000393 1787854306.172587 trunc from 0x00f762 to 0x01ed08 (0xf5a7 bytes) 000394 1787854306.178017 read 0x03206b thru 0x03218e (0x124 bytes) 000395 1787854306.179695 trunc from 0x02ab6c to 0x0374ea (0xc97f bytes) 000396 1787854306.190682 mapwrite 0x02af65 thru 0x02c612 (0x16ae bytes) 000397 1787854306.237721 trunc from 0x0374eb to 0x03203e (0x54ac bytes) 000398 1787854306.275173 read 0x00105c thru 0x00df36 (0xcedb bytes) 000399 1787854306.279067 write 0x015adc thru 0x02284d (0xcd72 bytes) 000400 1787854306.283701 punch from 0x009d05 to 0x00b414 (0x1710 bytes) 000401 1787854306.296758 read 0x028dab thru 0x03203e (0x9294 bytes) 000402 1787854306.298807 read 0x02c291 thru 0x03203e (0x5dae bytes) 000403 1787854306.303479 fallocate 0x01c2d6 thru 0x0244db (0x8206 bytes) 000404 1787854306.321322 MIRROR_EXTEND to 2 mirrors 000405 1787854307.165786 fallocate 0x002a4d thru 0x0072a8 (0x485c bytes) 000406 1787854307.215783 fallocate 0x00ce3f thru 0x011c01 (0x4dc3 bytes) 000407 1787854307.221935 write 0x0200b7 thru 0x0248f7 (0x4841 bytes) 000408 1787854307.242052 read 0x0253d5 thru 0x03203e (0xcc6a bytes) 000409 1787854307.261042 MIRROR_RESYNC 2 mirrors 000411 1787854308.272270 mapread 0x0048e9 thru 0x0140cb (0xf7e3 bytes) 000412 1787854308.310780 fallocate 0x00467b thru 0x00a6b7 (0x603d bytes) 000413 1787854308.363595 MIRROR_RESYNC 2 mirrors 000415 1787854309.289395 write 0x03f8b5 thru 0x03ffff (0x74b bytes) 000416 1787854309.360215 write 0x035bba thru 0x038137 (0x257e bytes) 000417 1787854309.386732 MIRROR_RESYNC 2 mirrors 000419 1787854310.310686 trunc from 0x01e023 to 0x020142 (0x2120 bytes) 000420 1787854310.365721 trunc from 0x03d67d to 0x03ffff (0x2983 bytes) 000421 1787854310.381750 write 0x01172b thru 0x018121 (0x69f7 bytes) 000422 1787854310.408038 read 0x02eafa thru 0x03690b (0x7e12 bytes) 000424 1787854310.433985 trunc from 0x00f38f to 0x019e67 (0xaad9 bytes) 000425 1787854310.478327 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) 000426 1787854310.712629 punch from 0x0092b1 to 0x00bbd0 (0x2920 bytes) 000427 1787854310.720581 trunc from 0x00c10e to 0x00dfd1 (0x1ec4 bytes) 000428 1787854310.731185 trunc from 0x040000 to 0x0264fb (0x19b04 bytes) 000429 1787854310.775155 write 0x020dc0 thru 0x02ab46 (0x9d87 bytes) 000430 1787854310.783607 trunc from 0x02ab47 to 0x027b22 (0x3024 bytes) 000431 1787854310.817759 fallocate 0x03cc3d thru 0x03ffff (0x33c3 bytes) 000432 1787854310.826388 write 0x0341ab thru 0x038678 (0x44ce bytes) 000433 1787854310.853308 read 0x03e451 thru 0x03e72c (0x2dc bytes) 000435 1787854310.865578 trunc from 0x0013f7 to 0x0023f6 (0x1000 bytes) 000436 1787854310.875635 trunc from 0x028fed to 0x02b081 (0x2095 bytes) 000437 1787854310.881765 punch from 0x01d231 to 0x02778b (0xa55b bytes) 000438 1787854310.897728 trunc from 0x040000 to 0x02708a (0x18f75 bytes) 000439 1787854310.918812 trunc from 0x025ee2 to 0x02708a (0x11a9 bytes) 000440 1787854310.942475 mapread 0x00cc2d thru 0x01540b (0x87df bytes) 000441 1787854310.975488 mapwrite 0x008f42 thru 0x00c979 (0x3a38 bytes) 000442 1787854310.992186 trunc from 0x02708b to 0x01d177 (0x9f13 bytes) 000443 1787854311.006213 read 0x0037e4 thru 0x00a618 (0x6e35 bytes) 000446 1787854311.021024 mapread 0x001e6c thru 0x00e238 (0xc3cd bytes) 000447 1787854311.022946 mapwrite 0x00b383 thru 0x00f499 (0x4117 bytes) 000449 1787854311.049844 MIRROR_EXTEND to 2 mirrors 000450 1787854311.803664 mapread 0x00cb2a thru 0x01af98 (0xe46f bytes) 000451 1787854311.839069 trunc from 0x01d178 to 0x00e7b9 (0xe9be bytes) 000452 1787854311.966435 punch from 0x004a10 to 0x00e7b9 (0x9daa bytes) 000453 1787854311.977475 mapread 0x002521 thru 0x0064fe (0x3fde bytes) 000454 1787854312.001945 trunc from 0x009c12 to 0x00da49 (0x3e38 bytes) 000455 1787854312.009198 mapwrite 0x021292 thru 0x02fd50 (0xeabf bytes) 000456 1787854312.107953 read 0x008db2 thru 0x00a8ea (0x1b39 bytes) 000457 1787854312.119212 trunc from 0x00f963 to 0x01ba54 (0xc0f2 bytes) 000458 1787854312.126815 mapread 0x019f3d thru 0x0288ca (0xe98e bytes) 000459 1787854312.142866 read 0x02ba44 thru 0x02fd50 (0x430d bytes) 000460 1787854312.143882 read 0x02f296 thru 0x02fd50 (0xabb bytes) 000462 1787854312.145548 trunc from 0x01b11f to 0x029a0f (0xe8f1 bytes) 000463 1787854312.150858 read 0x003dd0 thru 0x0078ae (0x3adf bytes) 000464 1787854312.151799 mapread 0x02e4ae thru 0x02fd50 (0x18a3 bytes) 000465 1787854312.152658 write 0x01ca0b thru 0x023725 (0x6d1b bytes) 000466 1787854312.174953 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2fd51): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fd51): Mirror 2: 0x511097bf (stale) CRC-32 checksum value for chunk [0, 0x2fd51): Mirror 1: 0xb6154209 Mirror 3: 0xb6154209 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fd51: skipped 000467 1787854313.136302 punch from 0x001dc2 to 0x00d25b (0xb49a bytes) 000468 1787854313.186866 mapwrite 0x0283be thru 0x029906 (0x1549 bytes) 000471 1787854313.244612 trunc from 0x020ee0 to 0x0290a2 (0x81c3 bytes) 000472 1787854313.259752 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2fd51): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fd51): Mirror 2: 0x511097bf (stale) Mirror 3: 0xb6154209 (stale) CRC-32 checksum value for chunk [0, 0x2fd51): Mirror 1: 0x430c29b2 Mirror 4: 0x430c29b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fd51: skipped 000473 1787854314.403849 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fd51): Mirror 3: 0xb6154209 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fd51): Mirror 3: 0xb6154209 (stale) CRC-32 checksum value for chunk [0, 0x2fd51): Mirror 1: 0x430c29b2 Mirror 4: 0x430c29b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fd51: skipped 000474 1787854315.163146 read 0x006aa3 thru 0x0079ad (0xf0b bytes) 000475 1787854315.168961 write 0x03d3ac thru 0x03d71b (0x370 bytes) 000476 1787854315.243746 trunc from 0x036252 to 0x03d71b (0x74ca bytes) 000477 1787854315.260213 trunc from 0x03d71c to 0x013627 (0x2a0f4 bytes) 000478 1787854315.348908 mapwrite 0x01a773 thru 0x024525 (0x9db3 bytes) 000479 1787854315.458583 read 0x018922 thru 0x01b6a5 (0x2d84 bytes) 000480 1787854315.459719 trunc from 0x024526 to 0x00224d (0x222d8 bytes) 000481 1787854315.492446 trunc from 0x00224e to 0x021578 (0x1f32b bytes) 000482 1787854315.523656 trunc from 0x014232 to 0x01f983 (0xb752 bytes) 000483 1787854315.528144 trunc from 0x01721b to 0x01a983 (0x3769 bytes) 000485 1787854315.536958 trunc from 0x01fcfb to 0x021578 (0x187e bytes) 000486 1787854315.547118 trunc from 0x021579 to 0x036e4d (0x158d5 bytes) 000487 1787854315.566678 read 0x01e077 thru 0x02ca7f (0xea09 bytes) 000488 1787854315.584735 read 0x030a59 thru 0x036e4d (0x63f5 bytes) 000489 1787854315.585600 trunc from 0x036e4e to 0x00b1f6 (0x2bc57 bytes) 000490 1787854315.623761 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb1f7): Mirror 3: 0xdd0f3cfe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb1f7): Mirror 3: 0xdd0f3cfe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb1f7: skipped 000491 1787854315.996914 fallocate 0x01a268 thru 0x02279b (0x8534 bytes) 000493 1787854316.008705 write 0x03dcbf thru 0x03ffff (0x2341 bytes) 000494 1787854316.027876 fallocate 0x009169 thru 0x013d90 (0xac28 bytes) 000495 1787854316.042737 mapwrite 0x02b228 thru 0x02d9d2 (0x27ab bytes) 000496 1787854316.108253 read 0x02e912 thru 0x032fe5 (0x46d4 bytes) 000497 1787854316.109946 mapread 0x00e55d thru 0x013b59 (0x55fd bytes) 000498 1787854316.140164 trunc from 0x02d5a7 to 0x034181 (0x6bdb bytes) 000499 1787854316.154232 read 0x0283ad thru 0x0328f8 (0xa54c bytes) 000500 1787854316.176323 write 0x02eae3 thru 0x02ebd4 (0xf2 bytes) 000501 1787854316.177361 mapread 0x005427 thru 0x008e6a (0x3a44 bytes) 000502 1787854316.192862 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbaa23094 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb6ee725f Mirror 4: 0xb6ee725f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000503 1787854317.278026 mapread 0x010742 thru 0x011eb9 (0x1778 bytes) 000504 1787854317.297452 mapread 0x01e78a thru 0x02a191 (0xba08 bytes) 000505 1787854317.338503 read 0x00f26a thru 0x01cc8f (0xda26 bytes) 000506 1787854317.364872 read 0x03eed5 thru 0x03ffff (0x112b bytes) 000507 1787854317.392594 fallocate 0x01320c thru 0x0191a0 (0x5f95 bytes) 000508 1787854317.442551 write 0x02d570 thru 0x033db2 (0x6843 bytes) 000509 1787854317.454508 trunc from 0x035500 to 0x03ffff (0xab00 bytes) 000510 1787854317.458834 trunc from 0x01792f to 0x018ab8 (0x118a bytes) 000511 1787854317.463010 trunc from 0x01779d to 0x017892 (0xf6 bytes) 000512 1787854317.470368 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb6ee725f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb6ee725f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000513 1787854317.759765 trunc from 0x040000 to 0x027db1 (0x1824e bytes) 000514 1787854317.804224 mapread 0x0202fb thru 0x027db1 (0x7ab7 bytes) 000515 1787854317.825969 trunc from 0x027db2 to 0x039408 (0x11657 bytes) 000516 1787854317.849894 punch from 0x036255 to 0x039408 (0x31b4 bytes) 000517 1787854317.855281 fallocate 0x01c0fd thru 0x02242b (0x632f bytes) 000518 1787854317.864976 trunc from 0x00b21d to 0x00e9f6 (0x37da bytes) 000519 1787854317.871964 read 0x02fba6 thru 0x039408 (0x9863 bytes) 000520 1787854317.891773 trunc from 0x02a2da to 0x037493 (0xd1ba bytes) 000521 1787854317.902825 trunc from 0x00191a to 0x0091fa (0x78e1 bytes) 000522 1787854317.923268 mapwrite 0x01e779 thru 0x02c38d (0xdc15 bytes) 000523 1787854317.993067 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) 000525 1787854318.184227 write 0x02f332 thru 0x035610 (0x62df bytes) 000526 1787854318.208031 trunc from 0x02426c to 0x02c028 (0x7dbd bytes) 000527 1787854318.214213 write 0x0048c0 thru 0x00da33 (0x9174 bytes) 000529 1787854318.233540 read 0x038178 thru 0x039408 (0x1291 bytes) 000530 1787854318.244143 mapread 0x01f527 thru 0x023e61 (0x493b bytes) 000531 1787854318.280625 trunc from 0x039409 to 0x035c4f (0x37b9 bytes) 000532 1787854318.302888 mapwrite 0x0245f3 thru 0x02b84b (0x7259 bytes) 000533 1787854318.377863 mapwrite 0x02f56a thru 0x034782 (0x5219 bytes) 000534 1787854318.418170 trunc from 0x035c50 to 0x000c2f (0x35020 bytes) 000535 1787854318.446845 mapwrite 0x019803 thru 0x028d16 (0xf514 bytes) 000536 1787854318.538042 trunc from 0x01331e to 0x01b819 (0x84fc bytes) 000537 1787854318.544183 punch from 0x025e30 to 0x028d16 (0x2ee7 bytes) 000538 1787854318.561672 punch from 0x0109ac to 0x0149f2 (0x4047 bytes) 000539 1787854318.580734 write 0x00c230 thru 0x01015f (0x3f30 bytes) 000540 1787854318.602331 mapread 0x00cc0c thru 0x015cd1 (0x90c6 bytes) 000541 1787854318.612337 fallocate 0x039e1c thru 0x03d522 (0x3707 bytes) 000542 1787854318.619430 trunc from 0x028d17 to 0x02ce38 (0x4122 bytes) 000543 1787854318.635772 trunc from 0x029cd9 to 0x02b488 (0x17b0 bytes) 000544 1787854318.646348 write 0x012a52 thru 0x02112f (0xe6de bytes) 000545 1787854318.647918 punch from 0x01736f to 0x017ff6 (0xc88 bytes) 000546 1787854318.672438 trunc from 0x00b2ab to 0x018250 (0xcfa6 bytes) 000547 1787854318.678674 trunc from 0x02ce39 to 0x039f4e (0xd116 bytes) 000548 1787854318.704556 punch from 0x038a6d to 0x039f4e (0x14e2 bytes) 000549 1787854318.718304 fallocate 0x032ea9 thru 0x03ffff (0xd157 bytes) 000550 1787854318.726039 fallocate 0x014d4a thru 0x01d5d4 (0x888b bytes) 000551 1787854318.731323 mapwrite 0x005809 thru 0x00c3df (0x6bd7 bytes) 000554 1787854318.761294 write 0x0014d4 thru 0x00af80 (0x9aad bytes) 000555 1787854318.762604 trunc from 0x040000 to 0x03cd49 (0x32b6 bytes) 000557 1787854318.781563 trunc from 0x01cee6 to 0x01dc41 (0xd5c bytes) 000558 1787854318.786681 trunc from 0x037d8d to 0x03cd49 (0x4fbd bytes) 000561 1787854318.795501 read 0x02ee32 thru 0x031d27 (0x2ef6 bytes) 000562 1787854318.830569 trunc from 0x019b26 to 0x024ec2 (0xb39d bytes) 000563 1787854318.840011 punch from 0x00838c to 0x01418c (0xbe01 bytes) 000564 1787854318.867501 mapread 0x038701 thru 0x03cd49 (0x4649 bytes) 000565 1787854318.868731 read 0x022295 thru 0x023fdd (0x1d49 bytes) 000566 1787854318.880850 mapread 0x015ab5 thru 0x016c74 (0x11c0 bytes) 000567 1787854318.881745 punch from 0x034e56 to 0x03cd49 (0x7ef4 bytes) 000568 1787854318.889072 trunc from 0x03cd4a to 0x03eb45 (0x1dfc bytes) 000569 1787854318.908346 trunc from 0x00a0e5 to 0x01404a (0x9f66 bytes) 000570 1787854318.914099 trunc from 0x03eb46 to 0x025ee1 (0x18c64 bytes) 000571 1787854318.930698 fallocate 0x03b1a0 thru 0x03ffff (0x4e60 bytes) 000572 1787854318.935482 fallocate 0x006d64 thru 0x00957b (0x2818 bytes) 000573 1787854318.941104 write 0x009b3a thru 0x00fa91 (0x5f58 bytes) 000575 1787854318.948321 fallocate 0x01afa5 thru 0x026f0c (0xbf68 bytes) 000576 1787854318.954713 mapread 0x011583 thru 0x01dfbd (0xca3b bytes) 000577 1787854318.983046 fallocate 0x008804 thru 0x00ddb1 (0x55ae bytes) 000578 1787854318.990084 fallocate 0x035bbc thru 0x03ffff (0xa444 bytes) 000579 1787854318.997145 fallocate 0x01e9b7 thru 0x026b54 (0x819e bytes) 000580 1787854319.014188 mapread 0x00375d thru 0x005ea4 (0x2748 bytes) 000581 1787854319.016027 write 0x02bfee thru 0x030ce0 (0x4cf3 bytes) 000582 1787854319.042097 mapread 0x03c488 thru 0x03ffff (0x3b78 bytes) 000583 1787854319.053494 punch from 0x022984 to 0x02da3b (0xb0b8 bytes) 000584 1787854319.072573 trunc from 0x040000 to 0x021ae7 (0x1e518 bytes) 000585 1787854319.099340 trunc from 0x021ae8 to 0x010a6c (0x1107b bytes) 000586 1787854319.120657 mapwrite 0x022e44 thru 0x02fea0 (0xd05d bytes) 000587 1787854319.245338 mapread 0x0060ce thru 0x009baa (0x3add bytes) 000589 1787854319.247011 punch from 0x019e5e to 0x0255c2 (0xb765 bytes) 000590 1787854319.251930 mapread 0x00efb0 thru 0x01e542 (0xf593 bytes) 000591 1787854319.317639 trunc from 0x02fea1 to 0x026bce (0x92d2 bytes) 000592 1787854319.344498 read 0x016644 thru 0x01e81d (0x81da bytes) 000594 1787854319.345881 trunc from 0x026bcf to 0x03613d (0xf56f bytes) 000595 1787854319.367598 trunc from 0x019b2e to 0x028fe9 (0xf4bc bytes) 000596 1787854319.372582 fallocate 0x015d5a thru 0x020d37 (0xafde bytes) 000598 1787854319.378853 trunc from 0x03613e to 0x00bd6a (0x2a3d3 bytes) 000599 1787854319.401646 fallocate 0x030269 thru 0x03468e (0x4426 bytes) 000600 1787854319.407732 mapread 0x00aca1 thru 0x015ba0 (0xaf00 bytes) 000601 1787854319.421780 mapread 0x0016fd thru 0x0108a1 (0xf1a5 bytes) 000602 1787854319.423118 fallocate 0x02e767 thru 0x03dc0a (0xf4a4 bytes) 000603 1787854319.429701 mapread 0x01990d thru 0x0199c8 (0xbc bytes) 000604 1787854319.431102 punch from 0x01d0c0 to 0x025d94 (0x8cd5 bytes) 000605 1787854319.437035 write 0x033dc2 thru 0x03bda5 (0x7fe4 bytes) 000606 1787854319.449715 MIRROR_EXTEND to 2 mirrors 000607 1787854320.493581 trunc from 0x03dc0b to 0x00dcba (0x2ff50 bytes) 000608 1787854320.568114 read 0x00129a thru 0x0057b7 (0x451e bytes) 000609 1787854320.583317 punch from 0x002b99 to 0x00a107 (0x756f bytes) 000610 1787854320.591242 read 0x00d4db thru 0x00dcba (0x7e0 bytes) 000611 1787854320.602838 write 0x006dbd thru 0x00a7e4 (0x3a28 bytes) 000612 1787854320.603957 trunc from 0x00dcbb to 0x00832d (0x598d bytes) 000613 1787854320.642703 trunc from 0x006605 to 0x006e6d (0x869 bytes) 000614 1787854320.656238 trunc from 0x0057bd to 0x00832d (0x2b71 bytes) 000615 1787854320.673584 trunc from 0x00832e to 0x010b2e (0x8801 bytes) 000616 1787854320.714083 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) 000617 1787854320.911950 punch from 0x00895f to 0x010b2e (0x81d0 bytes) 000618 1787854320.916312 read 0x003c4d thru 0x0043a5 (0x759 bytes) 000619 1787854320.922329 fallocate 0x01f372 thru 0x02844b (0x90da bytes) 000620 1787854320.928427 mapwrite 0x00c28d thru 0x0168a3 (0xa617 bytes) 000621 1787854321.050552 trunc from 0x009cdd to 0x01455c (0xa880 bytes) 000622 1787854321.067465 mapread 0x00f350 thru 0x0168a3 (0x7554 bytes) 000623 1787854321.085837 fallocate 0x0143f8 thru 0x0220e5 (0xdcee bytes) 000624 1787854321.092132 punch from 0x009c9b to 0x00d107 (0x346d bytes) 000626 1787854321.099208 write 0x03f1a8 thru 0x03ffff (0xe58 bytes) 000627 1787854321.100317 trunc from 0x01f5d7 to 0x025443 (0x5e6d bytes) 000628 1787854321.108683 trunc from 0x040000 to 0x00482d (0x3b7d2 bytes) 000630 1787854321.129962 fallocate 0x00b4b1 thru 0x0122df (0x6e2f bytes) 000631 1787854321.134572 read 0x00c248 thru 0x010555 (0x430e bytes) 000632 1787854321.162163 punch from 0x00a3c5 to 0x00f5e9 (0x5225 bytes) 000634 1787854321.171752 mapwrite 0x026243 thru 0x02c9b3 (0x6771 bytes) 000635 1787854321.253009 mapwrite 0x007a91 thru 0x00fe0d (0x837d bytes) 000636 1787854321.321997 fallocate 0x038811 thru 0x03ffff (0x77ef bytes) 000638 1787854321.326875 mapwrite 0x00c4bc thru 0x00f1b3 (0x2cf8 bytes) 000639 1787854321.340771 trunc from 0x040000 to 0x005560 (0x3aa9f bytes) 000641 1787854321.355884 punch from 0x0009b1 to 0x005560 (0x4bb0 bytes) 000642 1787854321.362821 read 0x000ebe thru 0x005560 (0x46a3 bytes) 000643 1787854321.383105 trunc from 0x005561 to 0x0388d1 (0x33371 bytes) 000644 1787854321.411791 MIRROR_EXTEND to 2 mirrors 000645 1787854322.304985 write 0x01e604 thru 0x025bda (0x75d7 bytes) 000646 1787854322.384750 trunc from 0x0388d2 to 0x00689a (0x32037 bytes) 000647 1787854322.477459 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x689b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x689b): Mirror 2: 0xd3400cb1 (stale) CRC-32 checksum value for chunk [0, 0x689b): Mirror 1: 0xd3400cb1 Mirror 3: 0xd3400cb1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x689b: skipped 000648 1787854323.948140 trunc from 0x002ee6 to 0x00689a (0x39b5 bytes) 000649 1787854324.041995 MIRROR_RESYNC 3 mirrors 000650 1787854325.044051 mapwrite 0x01e4f2 thru 0x021846 (0x3355 bytes) 000652 1787854325.281855 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x21847): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21847): Mirror 2: 0x329004a5 (stale) Mirror 3: 0x329004a5 (stale) CRC-32 checksum value for chunk [0, 0x21847): Mirror 1: 0x93e3b28a Mirror 4: 0x93e3b28a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21847: skipped 000653 1787854326.545111 punch from 0x00b044 to 0x00e1db (0x3198 bytes) 000654 1787854326.613533 write 0x03f7df thru 0x03ffff (0x821 bytes) 000655 1787854326.621516 trunc from 0x040000 to 0x00a4cf (0x35b30 bytes) 000656 1787854326.713654 fallocate 0x026e0b thru 0x030657 (0x984d bytes) 000657 1787854326.722039 mapread 0x006091 thru 0x00fab3 (0x9a23 bytes) 000658 1787854326.779086 fallocate 0x01ad4b thru 0x0287e8 (0xda9e bytes) 000659 1787854326.792054 mapwrite 0x01f41b thru 0x029f75 (0xab5b bytes) 000660 1787854326.875511 mapread 0x006870 thru 0x00818d (0x191e bytes) 000661 1787854326.881929 mapread 0x024786 thru 0x02e56f (0x9dea bytes) 000662 1787854326.885074 write 0x02ff7f thru 0x030b53 (0xbd5 bytes) 000663 1787854326.887250 trunc from 0x026308 to 0x02e686 (0x837f bytes) 000665 1787854326.900169 write 0x00b583 thru 0x00c293 (0xd11 bytes) 000666 1787854326.902967 trunc from 0x027dd7 to 0x02d7fc (0x5a26 bytes) 000667 1787854326.916016 fallocate 0x0013ad thru 0x00a5d4 (0x9228 bytes) 000668 1787854326.937467 punch from 0x002c51 to 0x005f97 (0x3347 bytes) 000669 1787854326.958479 punch from 0x01d821 to 0x02b22b (0xda0b bytes) 000670 1787854327.005002 fallocate 0x01fe3f thru 0x025fa2 (0x6164 bytes) 000672 1787854327.022897 punch from 0x01d153 to 0x024861 (0x770f bytes) 000673 1787854327.037030 fallocate 0x027f32 thru 0x02c324 (0x43f3 bytes) 000674 1787854327.054629 mapwrite 0x0319ce thru 0x039e3e (0x8471 bytes) 000675 1787854327.189030 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39e3f): Mirror 3: 0xd2431f5b (stale) Mirror 4: 0x75e3b3c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e3f): Mirror 3: 0xd2431f5b (stale) Mirror 4: 0x75e3b3c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e3f: skipped 000676 1787854327.952172 mapwrite 0x01489a thru 0x020821 (0xbf88 bytes) 000677 1787854328.099306 read 0x0358bc thru 0x039e3e (0x4583 bytes) 000679 1787854328.132151 punch from 0x0225fc to 0x02c230 (0x9c35 bytes) 000680 1787854328.143167 punch from 0x001f06 to 0x005e12 (0x3f0d bytes) 000681 1787854328.157883 fallocate 0x00a4df thru 0x00c157 (0x1c79 bytes) 000683 1787854328.191011 mapread 0x038188 thru 0x039e3e (0x1cb7 bytes) 000684 1787854328.200471 trunc from 0x018663 to 0x019d54 (0x16f2 bytes) 000686 1787854328.220998 trunc from 0x019a6f to 0x02941e (0xf9b0 bytes) 000687 1787854328.240025 mapwrite 0x02d0c5 thru 0x02ea2e (0x196a bytes) 000688 1787854328.306327 trunc from 0x003dd0 to 0x00cd1e (0x8f4f bytes) 000689 1787854328.329558 punch from 0x01e0c6 to 0x021075 (0x2fb0 bytes) 000690 1787854328.348255 mapread 0x015314 thru 0x0202ea (0xafd7 bytes) 000692 1787854328.400677 trunc from 0x039e3f to 0x030194 (0x9caa bytes) 000693 1787854328.455216 write 0x02de81 thru 0x03c50d (0xe68d bytes) 000694 1787854328.460138 write 0x02eee1 thru 0x039747 (0xa867 bytes) 000696 1787854328.471001 mapread 0x013235 thru 0x014a4e (0x181a bytes) 000697 1787854328.480813 MIRROR_RESYNC 3 mirrors 000698 1787854330.150151 read 0x00f2a4 thru 0x01a92c (0xb689 bytes) 000699 1787854330.177512 write 0x03d214 thru 0x03ffff (0x2dec bytes) 000700 1787854330.215524 read 0x00380d thru 0x00ba63 (0x8257 bytes) 000701 1787854330.227209 fallocate 0x00af6b thru 0x014bb9 (0x9c4f bytes) 000702 1787854330.231890 trunc from 0x040000 to 0x0013ea (0x3ec15 bytes) 000703 1787854330.255567 punch from 0x0007ea to 0x0013ea (0xc01 bytes) 000705 1787854330.262903 write 0x0342f9 thru 0x036469 (0x2171 bytes) 000707 1787854330.269156 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3646a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3646a): Mirror 3: 0x5244c28f (stale) Mirror 4: 0x5244c28f (stale) CRC-32 checksum value for chunk [0, 0x3646a): Mirror 1: 0x5f38c2dd Mirror 5: 0x5f38c2dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3646a: skipped 000708 1787854331.451189 fallocate 0x015e77 thru 0x016df0 (0xf7a bytes) 000709 1787854331.524552 fallocate 0x00ac19 thru 0x0146c8 (0x9ab0 bytes) 000710 1787854331.538146 trunc from 0x03646a to 0x0110a5 (0x253c4 bytes) 000711 1787854331.634544 trunc from 0x0110a6 to 0x0198b7 (0x8812 bytes) 000712 1787854331.668347 trunc from 0x0198b8 to 0x02c220 (0x12969 bytes) 000713 1787854331.695897 trunc from 0x01dfa3 to 0x02510e (0x716c bytes) 000714 1787854331.705174 fallocate 0x0215bc thru 0x02df48 (0xc98d bytes) 000715 1787854331.712164 mapread 0x01f32a thru 0x02df48 (0xec1f bytes) 000716 1787854331.750019 write 0x018e36 thru 0x021de9 (0x8fb4 bytes) 000717 1787854331.765223 write 0x01b087 thru 0x01f052 (0x3fcc bytes) 000718 1787854331.772827 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2df49): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df49): Mirror 3: 0x6963fb44 (stale) Mirror 4: 0x6963fb44 (stale) Mirror 5: 0x15b98cdc (stale) CRC-32 checksum value for chunk [0, 0x2df49): Mirror 1: 0x734fb55e Mirror 6: 0x734fb55e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df49: skipped 000719 1787854333.031447 trunc from 0x0192b2 to 0x01cf3c (0x3c8b bytes) 000720 1787854333.092112 MIRROR_RESYNC 5 mirrors 000721 1787854334.747339 punch from 0x004522 to 0x012fd7 (0xeab6 bytes) 000722 1787854334.793944 fallocate 0x037a61 thru 0x03c5bc (0x4b5c bytes) 000723 1787854334.800515 fallocate 0x004386 thru 0x00cd53 (0x89ce bytes) 000724 1787854334.806079 fallocate 0x01baf5 thru 0x01f1e8 (0x36f4 bytes) 000726 1787854334.812071 mapwrite 0x03cb50 thru 0x03ffff (0x34b0 bytes) 000728 1787854334.869087 punch from 0x030c21 to 0x038b83 (0x7f63 bytes) 000729 1787854334.874676 trunc from 0x00026b to 0x00839c (0x8132 bytes) 000730 1787854334.881152 punch from 0x01ef7b to 0x02701e (0x80a4 bytes) 000731 1787854334.886958 trunc from 0x040000 to 0x01776b (0x28894 bytes) 000732 1787854334.923132 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1776c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1776c): Mirror 3: 0x43c0955c (stale) Mirror 4: 0x43c0955c (stale) Mirror 5: 0x43c0955c (stale) Mirror 6: 0x43c0955c (stale) CRC-32 checksum value for chunk [0, 0x1776c): Mirror 1: 0xa466d5e5 Mirror 7: 0xa466d5e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1776c: skipped 000733 1787854336.384925 mapwrite 0x02e009 thru 0x03ae11 (0xce09 bytes) 000734 1787854336.524944 trunc from 0x020ddf to 0x0246db (0x38fd bytes) 000735 1787854336.531671 fallocate 0x020f10 thru 0x026af7 (0x5be8 bytes) 000737 1787854336.537814 read 0x030ea4 thru 0x0368c6 (0x5a23 bytes) 000738 1787854336.542860 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ae12): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae12): Mirror 3: 0x8c2c0311 (stale) Mirror 4: 0x8c2c0311 (stale) Mirror 5: 0x8c2c0311 (stale) Mirror 6: 0x8c2c0311 (stale) Mirror 7: 0x5b25373d (stale) CRC-32 checksum value for chunk [0, 0x3ae12): Mirror 1: 0x65e9df7e Mirror 8: 0x65e9df7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae12: skipped 000739 1787854337.869658 write 0x017f13 thru 0x01d239 (0x5327 bytes) 000741 1787854337.914870 mapwrite 0x00e4bc thru 0x018d02 (0xa847 bytes) 000742 1787854337.955085 trunc from 0x03ae12 to 0x03ddb6 (0x2fa5 bytes) 000743 1787854337.985071 trunc from 0x00803f to 0x010820 (0x87e2 bytes) 000744 1787854338.002030 mapwrite 0x029b52 thru 0x02f812 (0x5cc1 bytes) 000745 1787854338.044961 trunc from 0x00ffed to 0x01838c (0x83a0 bytes) 000746 1787854338.054684 write 0x0017d4 thru 0x00ac22 (0x944f bytes) 000747 1787854338.072062 write 0x020bd2 thru 0x0238cb (0x2cfa bytes) 000748 1787854338.081307 mapwrite 0x030c8b thru 0x0395a9 (0x891f bytes) 000749 1787854338.146462 write 0x03b33b thru 0x03c5be (0x1284 bytes) 000750 1787854338.190040 punch from 0x037fa1 to 0x03ddb6 (0x5e16 bytes) 000751 1787854338.206461 fallocate 0x01efda thru 0x02a5f1 (0xb618 bytes) 000752 1787854338.212715 trunc from 0x03d09e to 0x03ddb6 (0xd19 bytes) 000753 1787854338.218484 fallocate 0x014bad thru 0x023a93 (0xeee7 bytes) 000754 1787854338.223683 mapread 0x00b58d thru 0x01714c (0xbbc0 bytes) 000755 1787854338.261380 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ddb7): Mirror 3: 0xc834a500 (stale) Mirror 4: 0xc834a500 (stale) Mirror 6: 0xc834a500 (stale) Mirror 7: 0x78604805 (stale) Mirror 8: 0x57e61299 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ddb7): Mirror 3: 0xc834a500 (stale) Mirror 4: 0xc834a500 (stale) Mirror 6: 0xc834a500 (stale) Mirror 7: 0x78604805 (stale) Mirror 8: 0x57e61299 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ddb7: skipped 000757 1787854338.892736 fallocate 0x03d562 thru 0x03ffff (0x2a9e bytes) 000758 1787854338.915113 punch from 0x008a3f to 0x013e83 (0xb445 bytes) 000759 1787854338.930036 fallocate 0x0027a6 thru 0x0105e2 (0xde3d bytes) 000760 1787854338.940793 mapread 0x03111c thru 0x03d00c (0xbef1 bytes) 000761 1787854339.003626 trunc from 0x040000 to 0x0124c9 (0x2db36 bytes) 000762 1787854339.045662 write 0x033c38 thru 0x03cd41 (0x910a bytes) 000763 1787854339.051113 trunc from 0x03cd42 to 0x01e5d7 (0x1e76a bytes) 000764 1787854339.092035 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e5d8): Mirror 4: 0xd4a357aa (stale) Mirror 6: 0xd4a357aa (stale) Mirror 7: 0x52a1c00d (stale) Mirror 8: 0x52a1c00d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e5d8): Mirror 4: 0xd4a357aa (stale) Mirror 6: 0xd4a357aa (stale) Mirror 7: 0x52a1c00d (stale) Mirror 8: 0x52a1c00d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e5d8: skipped 000765 1787854339.929255 fallocate 0x00713c thru 0x00adf4 (0x3cb9 bytes) 000767 1787854339.936894 fallocate 0x035e1c thru 0x0375f4 (0x17d9 bytes) 000768 1787854339.947399 MIRROR_RESYNC 5 mirrors 000769 1787854340.950728 mapwrite 0x007d71 thru 0x013074 (0xb304 bytes) 000770 1787854341.105548 mapread 0x019a1b thru 0x01e5d7 (0x4bbd bytes) 000771 1787854341.132734 fallocate 0x027bb1 thru 0x02fbf8 (0x8048 bytes) 000772 1787854341.146705 fallocate 0x029cd1 thru 0x034110 (0xa440 bytes) 000773 1787854341.157052 fallocate 0x0317e8 thru 0x034e79 (0x3692 bytes) 000774 1787854341.164066 punch from 0x004965 to 0x009911 (0x4fad bytes) 000775 1787854341.171559 mapread 0x008818 thru 0x0134fa (0xace3 bytes) 000776 1787854341.198809 MIRROR_RESYNC 5 mirrors 000777 1787854342.300526 fallocate 0x00f347 thru 0x01a243 (0xaefd bytes) 000778 1787854342.363398 mapread 0x030a5e thru 0x034110 (0x36b3 bytes) 000779 1787854342.382989 write 0x001f66 thru 0x00386c (0x1907 bytes) 000780 1787854342.406818 mapwrite 0x01918c thru 0x01f50b (0x6380 bytes) 000781 1787854342.450027 trunc from 0x034111 to 0x0388e9 (0x47d9 bytes) 000782 1787854342.472814 mapread 0x026894 thru 0x031cfc (0xb469 bytes) 000785 1787854342.484946 mapwrite 0x007115 thru 0x00fa5e (0x894a bytes) 000786 1787854342.518629 trunc from 0x002f80 to 0x00582e (0x28af bytes) 000787 1787854342.523919 fallocate 0x02683e thru 0x03636d (0xfb30 bytes) 000788 1787854342.529397 trunc from 0x00b49b to 0x00e80c (0x3372 bytes) 000789 1787854342.534616 punch from 0x005cb3 to 0x015b27 (0xfe75 bytes) 000790 1787854342.540287 read 0x00ff5f thru 0x014db6 (0x4e58 bytes) 000791 1787854342.548888 write 0x03ff48 thru 0x03ffff (0xb8 bytes) 000792 1787854342.553379 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1ffdb6fd (stale) Mirror 6: 0x1ffdb6fd (stale) Mirror 7: 0x1ffdb6fd (stale) Mirror 8: 0x1ffdb6fd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x19efa81b Mirror 9: 0x19efa81b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000793 1787854343.781992 trunc from 0x0023f9 to 0x00dc31 (0xb839 bytes) 000794 1787854343.869234 mapread 0x0368df thru 0x03965e (0x2d80 bytes) 000796 1787854343.889760 read 0x00b968 thru 0x015e7b (0xa514 bytes) 000797 1787854343.898429 write 0x011db8 thru 0x016cff (0x4f48 bytes) 000798 1787854343.907269 read 0x031556 thru 0x03ffff (0xeaaa bytes) 000800 1787854343.908692 trunc from 0x0309ab to 0x03360d (0x2c63 bytes) 000801 1787854343.914146 punch from 0x038d12 to 0x03ffff (0x72ee bytes) 000802 1787854343.922163 read 0x022195 thru 0x0296fe (0x756a bytes) 000804 1787854343.934661 fallocate 0x0019ec thru 0x00b321 (0x9936 bytes) 000805 1787854343.939728 punch from 0x014d28 to 0x01b7a1 (0x6a7a bytes) 000806 1787854343.955280 trunc from 0x040000 to 0x02fc43 (0x103bc bytes) 000807 1787854344.007103 trunc from 0x02fc44 to 0x03cad6 (0xce93 bytes) 000808 1787854344.022156 mapwrite 0x0068fd thru 0x00c76d (0x5e71 bytes) 000809 1787854344.050125 fallocate 0x01f986 thru 0x0213a5 (0x1a20 bytes) 000810 1787854344.067663 trunc from 0x0291c4 to 0x029847 (0x684 bytes) 000811 1787854344.095271 write 0x038e05 thru 0x03ffff (0x71fb bytes) 000812 1787854344.111832 mapwrite 0x02d3aa thru 0x03a0e5 (0xcd3c bytes) 000813 1787854344.192282 fallocate 0x03ff54 thru 0x03ffff (0xac bytes) 000814 1787854344.203210 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1ffdb6fd (stale) Mirror 6: 0x1ffdb6fd (stale) Mirror 7: 0x1ffdb6fd (stale) Mirror 8: 0x1ffdb6fd (stale) Mirror 9: 0x19efa81b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf25210ab Mirror 10: 0xf25210ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000815 1787854345.431466 MIRROR_RESYNC 7 mirrors 000819 1787854347.816224 MIRROR_EXTEND to 8 mirrors 000820 1787854348.587178 MIRROR_EXTEND to 9 mirrors 000822 1787854349.616368 write 0x00e9e6 thru 0x019c9e (0xb2b9 bytes) 000823 1787854349.761878 mapread 0x0341ff thru 0x03f510 (0xb312 bytes) 000824 1787854349.815395 fallocate 0x038cb1 thru 0x03ffff (0x734f bytes) 000825 1787854349.826673 mapwrite 0x037535 thru 0x03ffff (0x8acb bytes) 000826 1787854349.854957 write 0x01a080 thru 0x027e8f (0xde10 bytes) 000827 1787854349.876845 mapread 0x010e5b thru 0x01af8d (0xa133 bytes) 000828 1787854349.878060 trunc from 0x000181 to 0x009814 (0x9694 bytes) 000829 1787854349.886463 trunc from 0x040000 to 0x028156 (0x17ea9 bytes) 000830 1787854349.911044 punch from 0x01c6b2 to 0x01dfc0 (0x190f bytes) 000831 1787854349.934863 fallocate 0x021351 thru 0x02f100 (0xddb0 bytes) 000832 1787854349.941315 mapwrite 0x02cecf thru 0x03a582 (0xd6b4 bytes) 000833 1787854350.013864 trunc from 0x015017 to 0x01bb4e (0x6b38 bytes) 000834 1787854350.020471 punch from 0x0010a5 to 0x004237 (0x3193 bytes) 000835 1787854350.026005 trunc from 0x00b987 to 0x016958 (0xafd2 bytes) 000837 1787854350.032972 trunc from 0x03a583 to 0x02f01d (0xb565 bytes) 000838 1787854350.061931 punch from 0x02d5e3 to 0x02f01d (0x1a3b bytes) 000839 1787854350.077249 MIRROR_RESYNC 9 mirrors 000840 1787854353.436301 trunc from 0x02f01e to 0x017f38 (0x170e5 bytes) 000841 1787854353.539937 mapwrite 0x02b60e thru 0x03483b (0x922e bytes) 000842 1787854353.710631 write 0x01ddb1 thru 0x02751f (0x976f bytes) 000843 1787854353.722795 write 0x01fc57 thru 0x022cd1 (0x307b bytes) 000844 1787854353.726352 write 0x032f92 thru 0x03b7ac (0x881b bytes) 000846 1787854353.737015 mapread 0x0245a8 thru 0x026248 (0x1ca1 bytes) 000847 1787854353.743196 fallocate 0x0269f6 thru 0x029a4c (0x3057 bytes) 000848 1787854353.749326 write 0x004721 thru 0x00bfb1 (0x7891 bytes) 000849 1787854353.772839 punch from 0x01ed22 to 0x02d7ea (0xeac9 bytes) 000850 1787854353.789373 trunc from 0x00076d to 0x00b8c3 (0xb157 bytes) 000851 1787854353.794293 punch from 0x025398 to 0x02d23c (0x7ea5 bytes) 000852 1787854353.803644 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b7ad): Mirror 4: 0xb271eec0 (stale) Mirror 6: 0xb271eec0 (stale) Mirror 7: 0xb271eec0 (stale) Mirror 8: 0xb271eec0 (stale) Mirror 9: 0xb271eec0 (stale) Mirror 10: 0xb271eec0 (stale) Mirror 12: 0xb271eec0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7ad): Mirror 4: 0xb271eec0 (stale) Mirror 6: 0xb271eec0 (stale) Mirror 7: 0xb271eec0 (stale) Mirror 8: 0xb271eec0 (stale) Mirror 9: 0xb271eec0 (stale) Mirror 10: 0xb271eec0 (stale) Mirror 12: 0xb271eec0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7ad: skipped 000853 1787854354.311104 fallocate 0x00e76c thru 0x01d603 (0xee98 bytes) 000854 1787854354.316366 punch from 0x013237 to 0x01cda8 (0x9b72 bytes) 000855 1787854354.320944 mapread 0x015486 thru 0x01c606 (0x7181 bytes) 000856 1787854354.342322 mapwrite 0x03bb35 thru 0x03ffff (0x44cb bytes) 000857 1787854354.401696 fallocate 0x0275df thru 0x0356d4 (0xe0f6 bytes) 000858 1787854354.408567 mapwrite 0x02ace6 thru 0x034510 (0x982b bytes) 000859 1787854354.451789 mapwrite 0x01c1fa thru 0x0283ee (0xc1f5 bytes) 000860 1787854354.495529 trunc from 0x016b51 to 0x023512 (0xc9c2 bytes) 000861 1787854354.502281 trunc from 0x040000 to 0x026185 (0x19e7a bytes) 000862 1787854354.521679 read 0x00a384 thru 0x010d3a (0x69b7 bytes) 000863 1787854354.552187 write 0x01176c thru 0x019210 (0x7aa5 bytes) 000864 1787854354.560255 trunc from 0x003f33 to 0x009fc9 (0x6097 bytes) 000865 1787854354.565191 trunc from 0x026186 to 0x025236 (0xf4f bytes) 000866 1787854354.586860 mapread 0x00d74c thru 0x00e3f4 (0xca9 bytes) 000867 1787854354.588560 fallocate 0x02de75 thru 0x03c5d3 (0xe75f bytes) 000868 1787854354.595275 trunc from 0x0215d7 to 0x026310 (0x4d3a bytes) 000871 1787854354.603060 mapread 0x024319 thru 0x0317c1 (0xd4a9 bytes) 000872 1787854354.635330 read 0x01a9c7 thru 0x01fba8 (0x51e2 bytes) 000873 1787854354.647053 write 0x02bb4f thru 0x02d507 (0x19b9 bytes) 000874 1787854354.648522 fallocate 0x030ee2 thru 0x0377ab (0x68ca bytes) 000875 1787854354.655531 trunc from 0x03c5d4 to 0x024fe3 (0x175f0 bytes) 000876 1787854354.678955 mapread 0x022be6 thru 0x023bdb (0xff6 bytes) 000877 1787854354.689519 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x24fe4): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24fe4): Mirror 4: 0xe272262f (stale) Mirror 6: 0xe272262f (stale) Mirror 7: 0xe272262f (stale) Mirror 8: 0xe272262f (stale) Mirror 9: 0xe272262f (stale) Mirror 10: 0xe272262f (stale) Mirror 12: 0xe272262f (stale) CRC-32 checksum value for chunk [0, 0x24fe4): Mirror 1: 0x42b105a0 Mirror 13: 0x42b105a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24fe4: skipped 000878 1787854355.983921 fallocate 0x01c78d thru 0x023a75 (0x72e9 bytes) 000879 1787854356.034830 write 0x02b83f thru 0x03ab47 (0xf309 bytes) 000880 1787854356.043909 fallocate 0x015208 thru 0x01d81a (0x8613 bytes) 000881 1787854356.049202 punch from 0x0204be to 0x02b8f8 (0xb43b bytes) 000882 1787854356.075365 trunc from 0x03ab48 to 0x027283 (0x138c4 bytes) 000883 1787854356.107298 mapwrite 0x001084 thru 0x003fe7 (0x2f64 bytes) 000885 1787854356.144969 mapwrite 0x01befe thru 0x024494 (0x8597 bytes) 000886 1787854356.184234 fallocate 0x0091bb thru 0x0101cd (0x7013 bytes) 000887 1787854356.189284 trunc from 0x027284 to 0x03330b (0xc088 bytes) 000888 1787854356.206043 trunc from 0x001af9 to 0x00c753 (0xac5b bytes) 000889 1787854356.211100 read 0x02a57f thru 0x03330b (0x8d8d bytes) 000890 1787854356.232178 write 0x007454 thru 0x011a25 (0xa5d2 bytes) 000891 1787854356.254181 read 0x02348d thru 0x033015 (0xfb89 bytes) 000892 1787854356.263308 mapwrite 0x03a28e thru 0x03f0b5 (0x4e28 bytes) 000893 1787854356.319204 mapread 0x039dce thru 0x03e525 (0x4758 bytes) 000894 1787854356.320608 mapwrite 0x00a71f thru 0x017e22 (0xd704 bytes) 000895 1787854356.351599 mapwrite 0x037b8c thru 0x03ffff (0x8474 bytes) 000896 1787854356.435358 trunc from 0x00b4ce to 0x01914b (0xdc7e bytes) 000897 1787854356.442027 write 0x006e4c thru 0x010696 (0x984b bytes) 000898 1787854356.452240 write 0x024a42 thru 0x0259b5 (0xf74 bytes) 000899 1787854356.454620 trunc from 0x0038af to 0x00a910 (0x7062 bytes) 000900 1787854356.492341 write 0x005623 thru 0x0085b7 (0x2f95 bytes) 000901 1787854356.502614 mapwrite 0x0041d3 thru 0x00ded0 (0x9cfe bytes) 000902 1787854356.548761 trunc from 0x040000 to 0x010afa (0x2f505 bytes) 000903 1787854356.586625 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10afb): Mirror 6: 0x93594565 (stale) Mirror 7: 0x93594565 (stale) Mirror 8: 0x93594565 (stale) Mirror 9: 0x93594565 (stale) Mirror 10: 0x93594565 (stale) Mirror 12: 0x93594565 (stale) Mirror 13: 0x256a1e77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10afb): Mirror 6: 0x93594565 (stale) Mirror 7: 0x93594565 (stale) Mirror 8: 0x93594565 (stale) Mirror 9: 0x93594565 (stale) Mirror 10: 0x93594565 (stale) Mirror 12: 0x93594565 (stale) Mirror 13: 0x256a1e77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10afb: skipped 000904 1787854357.087158 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10afb): Mirror 6: 0x93594565 (stale) Mirror 7: 0x93594565 (stale) Mirror 8: 0x93594565 (stale) Mirror 10: 0x93594565 (stale) Mirror 12: 0x93594565 (stale) Mirror 13: 0x256a1e77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10afb): Mirror 6: 0x93594565 (stale) Mirror 7: 0x93594565 (stale) Mirror 8: 0x93594565 (stale) Mirror 10: 0x93594565 (stale) Mirror 12: 0x93594565 (stale) Mirror 13: 0x256a1e77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10afb: skipped 000905 1787854357.564871 read 0x0001ef thru 0x008765 (0x8577 bytes) 000906 1787854357.595943 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x10afb): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10afb): Mirror 6: 0x93594565 (stale) Mirror 7: 0x93594565 (stale) Mirror 8: 0x93594565 (stale) Mirror 10: 0x93594565 (stale) Mirror 12: 0x93594565 (stale) Mirror 13: 0x256a1e77 (stale) CRC-32 checksum value for chunk [0, 0x10afb): Mirror 1: 0x5aa11dac Mirror 14: 0x5aa11dac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10afb: skipped 000907 1787854358.954350 read 0x0075d3 thru 0x00bad5 (0x4503 bytes) 000908 1787854358.964647 write 0x03b02c thru 0x03ffff (0x4fd4 bytes) 000909 1787854359.024147 fallocate 0x00d8e1 thru 0x0110d9 (0x37f9 bytes) 000910 1787854359.030277 trunc from 0x02e03d to 0x03bef7 (0xdebb bytes) 000911 1787854359.043591 trunc from 0x040000 to 0x007142 (0x38ebd bytes) 000912 1787854359.059369 write 0x03b678 thru 0x03d504 (0x1e8d bytes) 000913 1787854359.061137 trunc from 0x017926 to 0x01f125 (0x7800 bytes) 000914 1787854359.066868 mapwrite 0x017474 thru 0x01f856 (0x83e3 bytes) 000915 1787854359.133920 read 0x02fb9d thru 0x035575 (0x59d9 bytes) 000916 1787854359.155485 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d505): Mirror 6: 0xb20e3fd1 (stale) Mirror 7: 0xb20e3fd1 (stale) Mirror 8: 0xb20e3fd1 (stale) Mirror 10: 0xb20e3fd1 (stale) Mirror 13: 0x742ab249 (stale) Mirror 14: 0x1bf69d83 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d505): Mirror 6: 0xb20e3fd1 (stale) Mirror 7: 0xb20e3fd1 (stale) Mirror 8: 0xb20e3fd1 (stale) Mirror 10: 0xb20e3fd1 (stale) Mirror 13: 0x742ab249 (stale) Mirror 14: 0x1bf69d83 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d505: skipped 000917 1787854359.933157 write 0x02041e thru 0x0258e7 (0x54ca bytes) 000918 1787854359.949575 punch from 0x007384 to 0x014548 (0xd1c5 bytes) 000919 1787854359.969891 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d505): Mirror 6: 0xb20e3fd1 (stale) Mirror 7: 0xb20e3fd1 (stale) Mirror 8: 0xb20e3fd1 (stale) Mirror 10: 0xb20e3fd1 (stale) Mirror 14: 0x1bf69d83 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d505): Mirror 6: 0xb20e3fd1 (stale) Mirror 7: 0xb20e3fd1 (stale) Mirror 8: 0xb20e3fd1 (stale) Mirror 10: 0xb20e3fd1 (stale) Mirror 14: 0x1bf69d83 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d505: skipped 000920 1787854360.603094 write 0x0139ce thru 0x0217d8 (0xde0b bytes) 000921 1787854360.620930 read 0x01d81b thru 0x028379 (0xab5f bytes) 000923 1787854360.635191 mapwrite 0x008dea thru 0x009c89 (0xea0 bytes) 000924 1787854360.665307 trunc from 0x029a69 to 0x02a3fd (0x995 bytes) 000925 1787854360.670698 read 0x039f33 thru 0x03d504 (0x35d2 bytes) 000926 1787854360.685807 punch from 0x01c6a6 to 0x0259b8 (0x9313 bytes) 000927 1787854360.691567 write 0x007248 thru 0x00f38c (0x8145 bytes) 000928 1787854360.693213 trunc from 0x023cf2 to 0x02ad7d (0x708c bytes) 000929 1787854360.698634 read 0x03c16f thru 0x03d504 (0x1396 bytes) 000930 1787854360.699961 read 0x00f993 thru 0x018c10 (0x927e bytes) 000931 1787854360.701631 mapwrite 0x029612 thru 0x02ac57 (0x1646 bytes) 000932 1787854360.731982 read 0x003c0a thru 0x00d369 (0x9760 bytes) 000933 1787854360.737152 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d505): Mirror 6: 0xb20e3fd1 (stale) Mirror 7: 0xb20e3fd1 (stale) Mirror 8: 0xb20e3fd1 (stale) Mirror 10: 0xb20e3fd1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d505): Mirror 6: 0xb20e3fd1 (stale) Mirror 7: 0xb20e3fd1 (stale) Mirror 8: 0xb20e3fd1 (stale) Mirror 10: 0xb20e3fd1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d505: skipped 000934 1787854361.242927 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d505): Mirror 7: 0xb20e3fd1 (stale) Mirror 8: 0xb20e3fd1 (stale) Mirror 10: 0xb20e3fd1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d505): Mirror 7: 0xb20e3fd1 (stale) Mirror 8: 0xb20e3fd1 (stale) Mirror 10: 0xb20e3fd1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d505: skipped 000936 1787854361.679126 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d505): Mirror 8: 0xb20e3fd1 (stale) Mirror 10: 0xb20e3fd1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d505): Mirror 8: 0xb20e3fd1 (stale) Mirror 10: 0xb20e3fd1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d505: skipped 000937 1787854362.149384 MIRROR_RESYNC 3 mirrors 000938 1787854363.419893 mapread 0x02cfba thru 0x03b055 (0xe09c bytes) 000939 1787854363.482026 write 0x03bb37 thru 0x03ffff (0x44c9 bytes) 000940 1787854363.525543 trunc from 0x040000 to 0x024a9e (0x1b561 bytes) 000941 1787854363.550140 trunc from 0x00651c to 0x00e73a (0x821f bytes) 000942 1787854363.556287 punch from 0x007df4 to 0x0117fe (0x9a0b bytes) 000943 1787854363.566788 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24a9f): Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24a9f): Mirror 8: 0xe9fa6f02 (stale) Mirror 10: 0xe9fa6f02 (stale) CRC-32 checksum value for chunk [0, 0x24a9f): Mirror 1: 0xb9c78e44 Mirror 11: 0xb9c78e44 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24a9f: skipped 000944 1787854364.798175 read 0x022d18 thru 0x024a9e (0x1d87 bytes) 000945 1787854364.805536 fallocate 0x0266b0 thru 0x026982 (0x2d3 bytes) 000946 1787854364.862546 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x24a9f): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24a9f): Mirror 8: 0xe9fa6f02 (stale) Mirror 10: 0xe9fa6f02 (stale) Mirror 11: 0xb9c78e44 (stale) CRC-32 checksum value for chunk [0, 0x24a9f): Mirror 1: 0xb9c78e44 Mirror 12: 0xb9c78e44 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24a9f: skipped 000947 1787854366.185829 read 0x011f9a thru 0x01d176 (0xb1dd bytes) 000949 1787854366.199506 mapread 0x0021fb thru 0x010c7e (0xea84 bytes) 000950 1787854366.220266 mapread 0x00f28f thru 0x01c23e (0xcfb0 bytes) 000951 1787854366.221988 fallocate 0x02161b thru 0x0295b8 (0x7f9e bytes) 000952 1787854366.260248 read 0x0006ac thru 0x009633 (0x8f88 bytes) 000953 1787854366.272637 read 0x010ebf thru 0x01925b (0x839d bytes) 000954 1787854366.273292 trunc from 0x0260cd to 0x0295b8 (0x34ec bytes) 000955 1787854366.283294 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x295b9): Mirror 8: 0x7b94a8ba (stale) Mirror 11: 0x9a4a0e47 (stale) Mirror 12: 0x9a4a0e47 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x295b9): Mirror 8: 0x7b94a8ba (stale) Mirror 11: 0x9a4a0e47 (stale) Mirror 12: 0x9a4a0e47 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x295b9: skipped 000956 1787854366.772172 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x295b9): Mirror 11: 0x9a4a0e47 (stale) Mirror 12: 0x9a4a0e47 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x295b9): Mirror 11: 0x9a4a0e47 (stale) Mirror 12: 0x9a4a0e47 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x295b9: skipped 000957 1787854367.185184 write 0x0007e2 thru 0x010540 (0xfd5f bytes) 000959 1787854367.221324 mapwrite 0x0316aa thru 0x03682a (0x5181 bytes) 000960 1787854367.346205 fallocate 0x00d353 thru 0x014180 (0x6e2e bytes) 000961 1787854367.360747 trunc from 0x03682b to 0x033188 (0x36a2 bytes) 000962 1787854367.385281 trunc from 0x0317cb to 0x033188 (0x19be bytes) 000963 1787854367.391714 mapwrite 0x03ea2f thru 0x03ffff (0x15d1 bytes) 000964 1787854367.447631 write 0x018218 thru 0x022d17 (0xab00 bytes) 000965 1787854367.463700 read 0x003f8e thru 0x0073c3 (0x3436 bytes) 000966 1787854367.465324 write 0x02ebdc thru 0x02f5b4 (0x9d9 bytes) 000967 1787854367.470157 mapread 0x00425f thru 0x009a1c (0x57be bytes) 000968 1787854367.475232 trunc from 0x040000 to 0x02520e (0x1adf1 bytes) 000970 1787854367.505035 mapread 0x023320 thru 0x02520e (0x1eef bytes) 000971 1787854367.517167 punch from 0x002a1f to 0x0054e6 (0x2ac8 bytes) 000972 1787854367.525891 fallocate 0x0081d5 thru 0x00bb91 (0x39bd bytes) 000973 1787854367.534724 write 0x033ad0 thru 0x03ffff (0xc530 bytes) 000974 1787854367.539013 punch from 0x01614d to 0x02106c (0xaf20 bytes) 000975 1787854367.570580 trunc from 0x018a28 to 0x01a22c (0x1805 bytes) 000976 1787854367.576362 write 0x01dbb7 thru 0x02407a (0x64c4 bytes) 000977 1787854367.585690 read 0x01268e thru 0x021209 (0xeb7c bytes) 000978 1787854367.600693 trunc from 0x040000 to 0x028774 (0x1788b bytes) 000979 1787854367.623639 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28775): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28775): Mirror 11: 0xfb43617b (stale) Mirror 12: 0xfb43617b (stale) CRC-32 checksum value for chunk [0, 0x28775): Mirror 1: 0xf6792915 Mirror 13: 0xf6792915 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28775: skipped 000980 1787854368.817290 write 0x02b170 thru 0x03479b (0x962c bytes) 000981 1787854368.886534 read 0x0128c1 thru 0x0162a4 (0x39e4 bytes) 000982 1787854368.894893 write 0x033228 thru 0x033a4f (0x828 bytes) 000983 1787854368.895745 trunc from 0x01acf2 to 0x0239fe (0x8d0d bytes) 000984 1787854368.900646 mapwrite 0x02b3e3 thru 0x038579 (0xd197 bytes) 000985 1787854368.979259 trunc from 0x016a05 to 0x019f57 (0x3553 bytes) 000986 1787854368.984680 trunc from 0x02c3ff to 0x02d48d (0x108f bytes) 000988 1787854368.993065 trunc from 0x01b3d1 to 0x01f02c (0x3c5c bytes) 000989 1787854368.997641 write 0x02386f thru 0x02ee47 (0xb5d9 bytes) 000990 1787854369.004362 fallocate 0x017aac thru 0x021a9b (0x9ff0 bytes) 000992 1787854369.011371 mapwrite 0x03f37f thru 0x03ffff (0xc81 bytes) 000994 1787854369.083688 punch from 0x01b081 to 0x020027 (0x4fa7 bytes) 000995 1787854369.089663 write 0x003e28 thru 0x00fd35 (0xbf0e bytes) 000996 1787854369.110545 punch from 0x028a5b to 0x036457 (0xd9fd bytes) 000997 1787854369.121750 trunc from 0x040000 to 0x011c27 (0x2e3d8 bytes) 000998 1787854369.150336 trunc from 0x00f9ca to 0x011c27 (0x225e bytes) 000999 1787854369.166629 trunc from 0x00d715 to 0x011c27 (0x4513 bytes) 001000 1787854369.172664 write 0x039256 thru 0x03ffff (0x6daa bytes) All operations completed A-OK! PASS 70a (90s) == sanity-flr test 71: check mirror extend parent fid ==== 14:12:54 (1787854374) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1b8:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 72: check append on FLR file ========== 14:13:03 (1787854383) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199302 s, 10.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109179 s, 9.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 14:13:10 (1787854390) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7295072 .. Extending file size to 8029824 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8099168 .. Extending file size to 9018944 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8109120 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8404288 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8885664 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9068128 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62543 62544 62545 62548 62550 done 192.168.203.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) 192.168.203.139@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:14:31 (1787854471) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 Extending file size to 7005632 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2676128 .. Extending file size to 7454688 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7275008 .. Extending file size to 7554464 .. Extending file size to 8783360 .. Extending file size to 7966272 .. Extending file size to 8552032 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9053344 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 64388 64389 64390 64394 64395 done 192.168.203.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) 192.168.203.139@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:15:50 (1787854550) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1268948 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2160 1269144 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25280 3581740 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17676 3589344 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42956 7171084 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2356 1268948 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2160 1269144 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25280 3581740 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17676 3589344 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42956 7171084 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129777 s, 16.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.279062 s, 7.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xa9810bc1 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.203.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:16:11 (1787854571) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0839392 s, 25.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 14:16:19 (1787854579) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x63:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:16:25 (1787854585) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0947118 s, 22.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x218:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x218:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x219:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:16:33 (1787854593) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0990799 s, 21.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:16:40 (1787854600) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0895776 s, 23.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x219:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x219:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21b:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:17:13 (1787854633) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg339-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816157 s, 12.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21b:0x0] } PASS 204d (51s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:18:04 (1787854684) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg339-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157285 s, 13.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:18:58 (1787854738) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg339-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg339-server: Pool lustre.test_204f created Adding targets to pool oleg339-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg339-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182654 s, 11.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg339-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg339-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg339-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:20:15 (1787854815) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:20:22 (1787854822) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:20:29 (1787854829) oleg339-server: Pool lustre.test_206 created Adding targets to pool oleg339-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg339-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg339-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg339-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg339-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:21:00 (1787854860) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945551 s, 11.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292638 s, 35.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe14487a4 Mirror 3: 0xe14487a4 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:21:11 (1787854871) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:21:15 (1787854875) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:21:19 (1787854879) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.113067 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2020000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1d90000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1d90000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1d90000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xd50000 Read this (4 bytes): BBBB 192.168.203.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:22:01 (1787854921) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.027281 s, 300 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:22:09 (1787854929) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677628 s, 15.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611266 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 14:22:10.000000000 -0400 Modify: 2026-08-27 14:22:11.000000000 -0400 Change: 2026-08-27 14:22:12.000000000 -0400 Birth: 2026-08-27 14:22:11.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:22:18 (1787854938) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415785 s, 25.2 MB/s oleg339-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg339-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg339-server: O/280000401/d0/140: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:22:39 (1787854959) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.200317 s, 204 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0261045 s, 157 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 14:22 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x87:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 14:22 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:22:47 (1787854967) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:22:55 (1787854975) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1820 sec =========== 14:23:06 (1787854986) === sanity-flr: start cleanup 14:23:08 (1787854988) === === sanity-flr: finish cleanup 14:23:18 (1787854998) ===