-----============= acceptance-small: sanity-flr ============----- Wed Aug 19 08:35:41 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg401-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 08:36:02 (1787142962) === oleg401-client.virtnet: executing check_config_client /mnt/lustre oleg401-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg401-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff973498ffd000.idle_timeout=debug osc.lustre-OST0001-osc-ffff973498ffd000.idle_timeout=debug disable quota as required oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 08:36:41 (1787143001) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8232] == sanity-flr test 0a: lfs mirror create with -N option == 08:36:43 (1787143003) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:37:03 (1787143023) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:37:07 (1787143027) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:37:11 (1787143031) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:37:34 (1787143054) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:37:37 (1787143057) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 08:37:40 (1787143060) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff973498ffd000.stats=clear osc.lustre-OST0001-osc-ffff973498ffd000.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:37:47 (1787143067) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:37:55 (1787143075) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:38:04 (1787143084) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 08:38:07 (1787143087) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:38:15 (1787143095) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 08:38:23 (1787143103) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:38:30 (1787143110) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:38:39 (1787143119) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:38:46 (1787143126) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:38:56 (1787143136) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:39:02 (1787143142) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 1.75108 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111765 s, 9.4 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:39:12 (1787143152) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0915485 s, 34.4 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:39:20 (1787143160) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:39:31 (1787143171) Stopping /mnt/lustre-ost1 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff973498ffd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff973498ffd000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg401-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg401-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg401-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg401-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg401-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg401-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg401-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg401-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg401-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg401-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg401-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg401-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg401-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg401-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg401-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg401-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg401-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg401-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg401-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg401-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg401-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg401-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg401-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg401-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg401-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg401-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg401-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff973498ffd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff973498ffd000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg401-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (139s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:41:51 (1787143311) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.617218 s, 17.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff973498ffd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff973498ffd000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg401-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:42:38 (1787143358) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff973498ffd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff973498ffd000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg401-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff973498ffd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff973498ffd000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg401-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:44:00 (1787143440) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.91203 s, 18.0 MB/s -rw-r--r-- 1 root root 52428800 Aug 19 08:44 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.996s user 0m0.001s sys 0m0.192s Stopping /mnt/lustre-ost1 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff973498ffd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff973498ffd000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.087s user 0m0.001s sys 0m0.216s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg401-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff973498ffd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff973498ffd000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.128s user 0m0.002s sys 0m0.189s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg401-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg401-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff973498ffd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff973498ffd000.ost_server_uuid in FULL state after 0 sec PASS 33b (81s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:45:21 (1787143521) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:45:25 (1787143525) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:45:28 (1787143528) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 08:45:32 (1787143532) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 08:45:39 (1787143539) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.654266 s, 19.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:45:52 (1787143552) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042943 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028256 s, 37.1 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:45:59 (1787143559) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:46:07 (1787143567) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00357309 s, 287 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00856982 s, 119 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00323988 s, 316 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00182215 s, 562 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 08:46:17 (1787143577) create test files with size 15759265 .. a4a68ede9cdb4db685b2679bf705993f - c8d520135987aac0d69fcf3a73066ce2 - dc07bdcddcc338f8d5d4d1c1ba770fb1 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg401-client.virtnet /mnt/lustre (opts:) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 08:46:39 (1787143599) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:46:53 (1787143613) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:46:59 (1787143619) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623795 s, 16.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268533 s, 39.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 08:47:06 (1787143626) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0972705 s, 21.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.268065 s, 15.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17583 s, 11.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 08:47:22 (1787143642) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 08:47:25 (1787143645) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:47:29 (1787143649) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317999 s, 33.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 08:47:37 (1787143657) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.213764 s, 14.7 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:47:47 (1787143667) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:47:57 (1787143677) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.333999 s, 31.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg401-client.virtnet /mnt/lustre (opts:) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:48:08 (1787143688) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff973488328000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.423012 s, 24.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg401-client.virtnet /mnt/lustre (opts:) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9734a8f4a000.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:48:27 (1787143707) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:49:10 (1787143750) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:49:50 (1787143790) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 08:49:53 (1787143793) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.215569 s, 14.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.148062 s, 21.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5de17ab4 (nosync,stale) Mirror 2: 0x5de17ab4 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 83bf895384d67949dda56fdc41e5474b - mirror 1 checksum: a85e4531d90def9d4f7c8d3b67de1335 - mirror 2 checksum: 83bf895384d67949dda56fdc41e5474b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T08:49:55-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T08:49:55-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 83bf895384d67949dda56fdc41e5474b - mirror 1 checksum: a85e4531d90def9d4f7c8d3b67de1335 - mirror 2 checksum: a85e4531d90def9d4f7c8d3b67de1335 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T08:49:55-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T08:49:55-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 08:50:04 (1787143804) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348927 s, 30.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:50:11 (1787143811) Failing mds1 on oleg401-server Stopping /mnt/lustre-mds1 (opts:) on oleg401-server 08:50:23 (1787143823) shut down facet: mds1 facet_host: oleg401-server facet_failover_host: oleg401-server Failover mds1 to oleg401-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-MDT0000 08:50:47 (1787143847) targets are mounted 08:50:47 (1787143847) facet_failover done oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234689 s, 44.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } PASS 50A (53s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:51:04 (1787143864) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0580021 s, 353 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0128604 s, 1.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:51:16 (1787143876) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.071064 s, 288 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0151098 s, 1.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:51:27 (1787143887) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0108913 s, 1.5 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:51:35 (1787143895) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0647421 s, 316 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:51:44 (1787143904) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00984312 s, 416 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0402457 s, 102 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:51:52 (1787143912) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:52:17 (1787143937) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:52:47 (1787143967) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:53:11 (1787143991) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:53:36 (1787144016) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 08:53:45 (1787144025) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57578 mirror split pid 57580 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57578 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57580 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 08:54:53 (1787144093) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 4094 fallocating to largest ever: 0x40000 000003 1787144095.173887 fallocate 0x03c7f2 thru 0x03ffff (0x380e bytes) skipping zero length zero range 000005 1787144095.180564 write 0x004f71 thru 0x00aa1b (0x5aab bytes) 000006 1787144095.182502 mapread 0x00346a thru 0x004a4e (0x15e5 bytes) 000007 1787144095.188459 trunc from 0x0048de to 0x00aa1b (0x613e bytes) 000008 1787144095.211508 trunc from 0x00516f to 0x00995d (0x47ef bytes) 000010 1787144095.217862 write 0x027062 thru 0x02ddb3 (0x6d52 bytes) 000011 1787144095.219765 write 0x03e131 thru 0x03ffff (0x1ecf bytes) 000013 1787144095.222884 mapwrite 0x038009 thru 0x03996d (0x1965 bytes) 000014 1787144095.271389 trunc from 0x040000 to 0x00bbfc (0x34403 bytes) 000015 1787144095.305711 punch from 0x006a43 to 0x00bbfc (0x51ba bytes) 000016 1787144095.312572 fallocate 0x0386ca thru 0x03b24b (0x2b82 bytes) 000017 1787144095.318723 trunc from 0x00bbfd to 0x031838 (0x25c3c bytes) 000018 1787144095.338209 trunc from 0x031839 to 0x02d1a6 (0x4692 bytes) 000019 1787144095.352889 fallocate 0x00504b thru 0x01062d (0xb5e3 bytes) 000020 1787144095.358027 read 0x014d39 thru 0x019af0 (0x4db8 bytes) 000021 1787144095.373263 MIRROR_EXTEND to 2 mirrors 000022 1787144096.146196 trunc from 0x000876 to 0x001079 (0x804 bytes) 000024 1787144096.206076 fallocate 0x02355c thru 0x023cda (0x77f bytes) 000025 1787144096.210830 read 0x014028 thru 0x018ef0 (0x4ec9 bytes) 000027 1787144096.232231 trunc from 0x02d1a7 to 0x01b82f (0x11977 bytes) 000028 1787144096.302385 trunc from 0x01b830 to 0x00f6dd (0xc152 bytes) 000029 1787144096.348363 mapwrite 0x02e20b thru 0x02fb85 (0x197b bytes) 000031 1787144096.400435 punch from 0x026ab0 to 0x02fb85 (0x90d6 bytes) 000032 1787144096.413485 fallocate 0x025894 thru 0x0356ae (0xfe1b bytes) 000033 1787144096.426862 mapwrite 0x0044a8 thru 0x00f666 (0xb1bf bytes) 000035 1787144096.461482 punch from 0x025107 to 0x02db9b (0x8a95 bytes) 000036 1787144096.470905 punch from 0x029fcc to 0x02fb85 (0x5bba bytes) 000037 1787144096.476730 punch from 0x000d59 to 0x00a6dc (0x9984 bytes) 000039 1787144096.486116 punch from 0x02f734 to 0x02fb85 (0x452 bytes) 000040 1787144096.498864 trunc from 0x00296f to 0x00d24c (0xa8de bytes) 000041 1787144096.516016 mapwrite 0x033a63 thru 0x03490e (0xeac bytes) 000042 1787144096.578256 write 0x03c20a thru 0x03fa24 (0x381b bytes) 000044 1787144096.589647 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fa25): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa25): Mirror 2: 0x250d7338 (stale) CRC-32 checksum value for chunk [0, 0x3fa25): Mirror 1: 0xfb683aca Mirror 3: 0xfb683aca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa25: skipped 000045 1787144097.726867 mapread 0x01ed59 thru 0x027b0a (0x8db2 bytes) 000046 1787144097.765684 MIRROR_RESYNC 3 mirrors 000047 1787144098.711232 fallocate 0x03cf01 thru 0x03ffff (0x30ff bytes) 000048 1787144098.745138 trunc from 0x00e583 to 0x01bbd7 (0xd655 bytes) 000049 1787144098.750648 mapread 0x020f77 thru 0x02df5a (0xcfe4 bytes) 000050 1787144098.782316 mapwrite 0x02cc8f thru 0x030e9e (0x4210 bytes) 000051 1787144098.803287 fallocate 0x006593 thru 0x006ce3 (0x751 bytes) 000052 1787144098.809225 mapwrite 0x030a29 thru 0x03f0a5 (0xe67d bytes) 000053 1787144098.852309 read 0x01b280 thru 0x02ad5a (0xfadb bytes) 000054 1787144098.857706 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fa25): Mirror 3: 0xfb683aca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa25): Mirror 3: 0xfb683aca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa25: skipped 000055 1787144099.270373 trunc from 0x02b7f0 to 0x033014 (0x7825 bytes) 000056 1787144099.275464 trunc from 0x01b2ba to 0x0281f9 (0xcf40 bytes) 000058 1787144099.281642 fallocate 0x01ec96 thru 0x02b5ce (0xc939 bytes) 000059 1787144099.292139 mapwrite 0x029d76 thru 0x02c2c7 (0x2552 bytes) 000060 1787144099.335329 punch from 0x026922 to 0x02dd4c (0x742b bytes) 000061 1787144099.342667 mapread 0x02738f thru 0x02bafe (0x4770 bytes) 000062 1787144099.361685 write 0x01fc84 thru 0x02481d (0x4b9a bytes) 000063 1787144099.375078 mapwrite 0x0056ee thru 0x010071 (0xa984 bytes) 000064 1787144099.420195 write 0x026ed6 thru 0x02b17a (0x42a5 bytes) 000065 1787144099.423324 punch from 0x02049b to 0x0285a1 (0x8107 bytes) 000066 1787144099.440805 punch from 0x032f23 to 0x03fa24 (0xcb02 bytes) 000067 1787144099.447326 trunc from 0x027988 to 0x030190 (0x8809 bytes) 000068 1787144099.453322 punch from 0x01c3ef to 0x01d994 (0x15a6 bytes) 000069 1787144099.459777 trunc from 0x00048a to 0x001d6d (0x18e4 bytes) 000070 1787144099.477315 write 0x02e715 thru 0x038aed (0xa3d9 bytes) 000071 1787144099.498988 trunc from 0x035c1e to 0x03683a (0xc1d bytes) 000072 1787144099.528804 mapread 0x02f140 thru 0x033adb (0x499c bytes) 000073 1787144099.531118 write 0x01d66e thru 0x0292fe (0xbc91 bytes) 000074 1787144099.548407 trunc from 0x025dcc to 0x02c0df (0x6314 bytes) 000075 1787144099.587528 punch from 0x02c441 to 0x039ff1 (0xdbb1 bytes) 000076 1787144099.612073 trunc from 0x028023 to 0x033c68 (0xbc46 bytes) 000077 1787144099.638155 trunc from 0x00df05 to 0x00fa41 (0x1b3d bytes) 000078 1787144099.673725 MIRROR_RESYNC 2 mirrors 000079 1787144100.749648 trunc from 0x03fa25 to 0x02e965 (0x110bf bytes) 000080 1787144100.810356 mapread 0x01dac7 thru 0x02c38d (0xe8c7 bytes) 000081 1787144100.876118 trunc from 0x0151d6 to 0x018a61 (0x388c bytes) 000082 1787144100.881810 mapwrite 0x00b43f thru 0x01a66a (0xf22c bytes) 000083 1787144100.943357 punch from 0x0057ad to 0x00eb25 (0x9379 bytes) 000085 1787144100.954331 MIRROR_RESYNC 2 mirrors 000086 1787144101.892761 MIRROR_RESYNC 2 mirrors 000087 1787144102.360745 write 0x018bab thru 0x01ad26 (0x217c bytes) 000089 1787144102.434327 mapwrite 0x00ad5d thru 0x0124e3 (0x7787 bytes) 000090 1787144102.504781 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000091 1787144102.701062 trunc from 0x029891 to 0x02e965 (0x50d5 bytes) 000093 1787144102.707075 read 0x01d18f thru 0x028cde (0xbb50 bytes) 000094 1787144102.720898 write 0x03803b thru 0x03ec94 (0x6c5a bytes) 000095 1787144102.726600 fallocate 0x00529e thru 0x00dbf8 (0x895b bytes) 000096 1787144102.732357 fallocate 0x00a6bd thru 0x01803f (0xd983 bytes) 000097 1787144102.737325 punch from 0x009f36 to 0x01385c (0x9927 bytes) 000098 1787144102.745736 mapread 0x03d060 thru 0x03ec94 (0x1c35 bytes) 000100 1787144102.752173 MIRROR_EXTEND to 2 mirrors 000101 1787144103.562269 mapread 0x030895 thru 0x037710 (0x6e7c bytes) 000102 1787144103.598620 fallocate 0x00ce6f thru 0x019008 (0xc19a bytes) 000103 1787144103.643157 trunc from 0x005e9c to 0x007ee6 (0x204b bytes) 000104 1787144103.656073 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000105 1787144103.817172 trunc from 0x03ec95 to 0x0042ef (0x3a9a5 bytes) 000106 1787144103.848283 fallocate 0x00f2cb thru 0x01475b (0x5491 bytes) 000107 1787144103.854226 trunc from 0x01475c to 0x0236a3 (0xef48 bytes) 000108 1787144103.875366 fallocate 0x03921c thru 0x03d6fe (0x44e3 bytes) 000110 1787144103.881789 fallocate 0x010d90 thru 0x01865e (0x78cf bytes) 000112 1787144103.890341 read 0x02edd5 thru 0x03b9ee (0xcc1a bytes) 000114 1787144103.903612 punch from 0x00efe5 to 0x01578f (0x67ab bytes) 000115 1787144103.908830 trunc from 0x00ce53 to 0x0166ad (0x985b bytes) 000116 1787144103.914291 trunc from 0x03d6ff to 0x02431b (0x193e3 bytes) 000117 1787144103.939240 mapread 0x01dc62 thru 0x021859 (0x3bf8 bytes) 000119 1787144103.976616 punch from 0x017285 to 0x017507 (0x283 bytes) 000120 1787144103.989616 fallocate 0x03f4d6 thru 0x03ffff (0xb2a bytes) 000121 1787144103.995533 mapwrite 0x01a67b thru 0x01c2bb (0x1c41 bytes) 000123 1787144104.025588 trunc from 0x02431c to 0x0292e7 (0x4fcc bytes) 000124 1787144104.058247 trunc from 0x015ef0 to 0x01703b (0x114c bytes) 000126 1787144104.063896 trunc from 0x02121b to 0x0215f7 (0x3dd bytes) 000127 1787144104.071876 mapread 0x00e6ed thru 0x0127b2 (0x40c6 bytes) 000129 1787144104.096839 fallocate 0x00ca2d thru 0x012232 (0x5806 bytes) 000130 1787144104.103607 write 0x00504c thru 0x01172e (0xc6e3 bytes) 000131 1787144104.115608 mapread 0x0156da thru 0x01822d (0x2b54 bytes) 000132 1787144104.125531 mapwrite 0x012f3b thru 0x02100a (0xe0d0 bytes) 000133 1787144104.164421 fallocate 0x02cd00 thru 0x03c193 (0xf494 bytes) 000134 1787144104.170805 write 0x0020c2 thru 0x011a7d (0xf9bc bytes) 000135 1787144104.199586 punch from 0x00ef64 to 0x015a3c (0x6ad9 bytes) 000137 1787144104.240009 trunc from 0x017ebf to 0x024c17 (0xcd59 bytes) 000138 1787144104.247998 mapread 0x00ae7b thru 0x014ccc (0x9e52 bytes) 000139 1787144104.273054 trunc from 0x0292e8 to 0x01df23 (0xb3c4 bytes) 000140 1787144104.308507 write 0x030a62 thru 0x03ecb0 (0xe24f bytes) 000141 1787144104.312019 trunc from 0x03ecb1 to 0x028e6d (0x15e43 bytes) 000143 1787144104.337578 punch from 0x0171c4 to 0x018dd8 (0x1c15 bytes) 000144 1787144104.348967 mapread 0x02585a thru 0x028e6d (0x3614 bytes) 000146 1787144104.362640 fallocate 0x02e75c thru 0x033a5b (0x5300 bytes) 000148 1787144104.368367 mapread 0x004f8f thru 0x0089d6 (0x3a48 bytes) 000149 1787144104.370486 mapread 0x012f32 thru 0x01caba (0x9b89 bytes) 000150 1787144104.385702 read 0x01aa4e thru 0x023462 (0x8a15 bytes) 000151 1787144104.386381 punch from 0x017ead to 0x0240b6 (0xc20a bytes) 000152 1787144104.393468 punch from 0x016777 to 0x0191e6 (0x2a70 bytes) 000153 1787144104.399876 trunc from 0x028e6e to 0x00320a (0x25c63 bytes) 000154 1787144104.417234 fallocate 0x035fe7 thru 0x03878f (0x27a9 bytes) 000155 1787144104.428275 MIRROR_EXTEND to 2 mirrors 000156 1787144105.440631 fallocate 0x02c7e4 thru 0x03a5d3 (0xddf0 bytes) 000158 1787144105.517992 trunc from 0x007fb4 to 0x0146f0 (0xc73d bytes) 000159 1787144105.532072 mapread 0x00d666 thru 0x00db8d (0x528 bytes) 000160 1787144105.552524 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000161 1787144105.746743 mapread 0x0062e5 thru 0x00e758 (0x8474 bytes) 000162 1787144105.789423 mapread 0x01bde9 thru 0x026cbb (0xaed3 bytes) 000163 1787144105.819821 punch from 0x01f6a3 to 0x02dbb9 (0xe517 bytes) 000164 1787144105.825361 trunc from 0x03a5d4 to 0x0138ce (0x26d05 bytes) 000166 1787144105.880021 MIRROR_EXTEND to 2 mirrors 000169 1787144106.833547 MIRROR_EXTEND to 3 mirrors 000170 1787144107.869728 mapread 0x003216 thru 0x0068e6 (0x36d1 bytes) 000171 1787144107.909693 mapread 0x002543 thru 0x003ffb (0x1ab9 bytes) 000172 1787144107.940061 write 0x0357b3 thru 0x03df13 (0x8761 bytes) 000173 1787144107.981129 trunc from 0x02058b to 0x028448 (0x7ebe bytes) 000174 1787144107.998458 trunc from 0x03df14 to 0x0219e2 (0x1c531 bytes) 000175 1787144108.065915 fallocate 0x021c99 thru 0x024bbe (0x2f26 bytes) 000176 1787144108.077014 mapread 0x001e6e thru 0x00778d (0x5920 bytes) 000177 1787144108.103453 mapread 0x01dc6b thru 0x024bbe (0x6f54 bytes) 000178 1787144108.122263 mapwrite 0x03fd35 thru 0x03ffff (0x2cb bytes) 000179 1787144108.244762 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbe7ebb1a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbe7ebb1a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000181 1787144108.691437 mapread 0x002b91 thru 0x00d9ee (0xae5e bytes) 000182 1787144108.724719 read 0x038893 thru 0x03a246 (0x19b4 bytes) 000183 1787144108.735263 read 0x01263d thru 0x0188eb (0x62af bytes) 000184 1787144108.758926 mapread 0x00f1c9 thru 0x0193fe (0xa236 bytes) 000185 1787144108.760052 read 0x038214 thru 0x039ff3 (0x1de0 bytes) 000186 1787144108.760994 mapread 0x01d32c thru 0x0276e0 (0xa3b5 bytes) 000187 1787144108.762580 mapread 0x002f69 thru 0x002f6f (0x7 bytes) 000190 1787144108.768463 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000191 1787144108.908708 trunc from 0x040000 to 0x036aaa (0x9555 bytes) 000192 1787144108.931591 write 0x028e54 thru 0x02aa73 (0x1c20 bytes) 000194 1787144108.947586 punch from 0x025e7b to 0x0285b5 (0x273b bytes) 000195 1787144108.960411 mapwrite 0x0134cf thru 0x01ba78 (0x85aa bytes) 000196 1787144109.036416 punch from 0x002294 to 0x00c7cb (0xa538 bytes) 000197 1787144109.054490 read 0x028be7 thru 0x036aaa (0xdec4 bytes) 000198 1787144109.064086 write 0x005476 thru 0x00f0ff (0x9c8a bytes) 000199 1787144109.078917 mapwrite 0x02db6a thru 0x034ce1 (0x7178 bytes) 000200 1787144109.122782 mapwrite 0x003abe thru 0x0119b5 (0xdef8 bytes) 000201 1787144109.184318 trunc from 0x036aab to 0x022ee1 (0x13bc9 bytes) 000202 1787144109.212603 punch from 0x009d9f to 0x00fbd9 (0x5e3b bytes) 000204 1787144109.219296 trunc from 0x022ee2 to 0x00c47c (0x16a65 bytes) 000206 1787144109.247163 trunc from 0x00c325 to 0x00c47c (0x158 bytes) 000207 1787144109.253220 trunc from 0x00c47d to 0x035893 (0x29417 bytes) 000208 1787144109.272463 trunc from 0x035894 to 0x0004f3 (0x353a0 bytes) 000209 1787144109.290581 write 0x015c71 thru 0x01835a (0x26ea bytes) 000210 1787144109.293508 punch from 0x006687 to 0x00df08 (0x7882 bytes) 000211 1787144109.297361 write 0x005cef thru 0x0131a6 (0xd4b8 bytes) 000212 1787144109.308616 read 0x003016 thru 0x00dddb (0xadc6 bytes) 000213 1787144109.316574 fallocate 0x01c402 thru 0x01f564 (0x3163 bytes) 000214 1787144109.322046 trunc from 0x00e3ac to 0x01a2d4 (0xbf29 bytes) 000215 1787144109.339817 fallocate 0x02d9ed thru 0x039d60 (0xc374 bytes) 000216 1787144109.345965 trunc from 0x01bf06 to 0x023efc (0x7ff7 bytes) 000218 1787144109.351059 punch from 0x00f97a to 0x01a724 (0xadab bytes) 000220 1787144109.357867 mapread 0x01d067 thru 0x026ec1 (0x9e5b bytes) 000221 1787144109.369192 mapread 0x03815f thru 0x039d60 (0x1c02 bytes) 000222 1787144109.370040 fallocate 0x0380ae thru 0x03bb24 (0x3a77 bytes) 000223 1787144109.375289 read 0x02be80 thru 0x02dca5 (0x1e26 bytes) 000224 1787144109.375769 read 0x039ea1 thru 0x03bb24 (0x1c84 bytes) 000225 1787144109.388468 MIRROR_EXTEND to 2 mirrors 000226 1787144110.106121 read 0x016dee thru 0x01af37 (0x414a bytes) 000227 1787144110.130378 mapread 0x00b52b thru 0x00e8f7 (0x33cd bytes) 000229 1787144110.175493 fallocate 0x00e952 thru 0x018f67 (0xa616 bytes) 000230 1787144110.258586 read 0x026265 thru 0x026e6f (0xc0b bytes) 000231 1787144110.270724 write 0x02e077 thru 0x0368d3 (0x885d bytes) 000232 1787144110.275628 trunc from 0x03bb25 to 0x034679 (0x74ab bytes) 000233 1787144110.346422 mapwrite 0x0205ac thru 0x026630 (0x6085 bytes) 000234 1787144110.401422 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000235 1787144110.653250 read 0x0074c9 thru 0x00f0d5 (0x7c0d bytes) 000237 1787144110.670898 read 0x002f5d thru 0x008aa8 (0x5b4c bytes) 000239 1787144110.679688 write 0x018c58 thru 0x01d59d (0x4946 bytes) 000240 1787144110.685255 punch from 0x0132e7 to 0x017f41 (0x4c5b bytes) 000241 1787144110.695488 fallocate 0x018b3e thru 0x023a88 (0xaf4b bytes) 000242 1787144110.703914 write 0x0096b1 thru 0x019514 (0xfe64 bytes) 000243 1787144110.707211 mapwrite 0x01c85b thru 0x024fee (0x8794 bytes) 000244 1787144110.733200 mapread 0x015d76 thru 0x01696b (0xbf6 bytes) 000245 1787144110.734129 mapwrite 0x007347 thru 0x0121b7 (0xae71 bytes) 000246 1787144110.760094 punch from 0x015618 to 0x01e90f (0x92f8 bytes) 000247 1787144110.765435 read 0x001311 thru 0x0102be (0xefae bytes) 000248 1787144110.766213 trunc from 0x01de87 to 0x022121 (0x429b bytes) 000249 1787144110.771848 trunc from 0x004631 to 0x00b3b4 (0x6d84 bytes) 000250 1787144110.776545 mapread 0x022185 thru 0x030928 (0xe7a4 bytes) 000252 1787144110.789136 write 0x00d77b thru 0x01ac1d (0xd4a3 bytes) 000253 1787144110.791134 write 0x03e4f5 thru 0x03f70a (0x1216 bytes) 000254 1787144110.792147 trunc from 0x03f70b to 0x0170fc (0x2860e bytes) 000256 1787144110.816250 read 0x004dfd thru 0x00d068 (0x826c bytes) 000257 1787144110.817780 mapwrite 0x02bf76 thru 0x035fb5 (0xa040 bytes) 000259 1787144110.887222 trunc from 0x035fb6 to 0x03669d (0x6e8 bytes) 000260 1787144110.906300 fallocate 0x01d0c8 thru 0x0273d2 (0xa30b bytes) 000261 1787144110.914776 write 0x01e75b thru 0x0247ad (0x6053 bytes) 000262 1787144110.917692 mapread 0x0148ac thru 0x01d046 (0x879b bytes) 000264 1787144110.921924 trunc from 0x03206b to 0x03669d (0x4633 bytes) 000265 1787144110.928272 mapread 0x01d583 thru 0x02d0c7 (0xfb45 bytes) 000266 1787144110.930555 trunc from 0x03669e to 0x019215 (0x1d488 bytes) 000267 1787144110.950008 trunc from 0x013ecf to 0x019215 (0x5347 bytes) 000268 1787144110.960899 MIRROR_EXTEND to 2 mirrors 000269 1787144111.907485 trunc from 0x019216 to 0x016864 (0x29b1 bytes) 000270 1787144111.996405 trunc from 0x016865 to 0x015c56 (0xc0e bytes) 000272 1787144112.047986 mapwrite 0x020ead thru 0x027336 (0x648a bytes) 000273 1787144112.149385 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000275 1787144112.333446 mapread 0x015904 thru 0x01e735 (0x8e32 bytes) 000277 1787144112.360970 punch from 0x00619b to 0x00f539 (0x939f bytes) 000278 1787144112.365768 mapread 0x009bc7 thru 0x00f579 (0x59b3 bytes) 000280 1787144112.377135 fallocate 0x022e80 thru 0x02bee8 (0x9069 bytes) 000281 1787144112.381690 trunc from 0x006b45 to 0x00cd8d (0x6249 bytes) 000282 1787144112.386708 trunc from 0x02bee9 to 0x015b29 (0x163bf bytes) 000283 1787144112.411426 fallocate 0x0211fc thru 0x02eae5 (0xd8ea bytes) 000285 1787144112.418823 trunc from 0x007064 to 0x015b29 (0xeac6 bytes) 000287 1787144112.425811 fallocate 0x023d31 thru 0x02883d (0x4b0d bytes) 000288 1787144112.437663 punch from 0x017973 to 0x027403 (0xfa91 bytes) 000289 1787144112.450670 write 0x03a10e thru 0x03d5db (0x34ce bytes) 000290 1787144112.453412 write 0x03182d thru 0x03bf10 (0xa6e4 bytes) 000291 1787144112.463610 mapwrite 0x007fae thru 0x00e593 (0x65e6 bytes) 000292 1787144112.506935 read 0x02c37a thru 0x030eed (0x4b74 bytes) 000293 1787144112.528501 read 0x03d17a thru 0x03d5db (0x462 bytes) 000294 1787144112.529105 read 0x024029 thru 0x02fc69 (0xbc41 bytes) 000295 1787144112.549874 trunc from 0x03d5dc to 0x03d968 (0x38d bytes) 000296 1787144112.582363 punch from 0x01c354 to 0x02b445 (0xf0f2 bytes) 000297 1787144112.592720 mapwrite 0x004da4 thru 0x00a992 (0x5bef bytes) 000298 1787144112.630652 mapread 0x0250a2 thru 0x02aa5b (0x59ba bytes) 000299 1787144112.706151 mapread 0x0336a6 thru 0x035386 (0x1ce1 bytes) 000300 1787144112.707725 mapwrite 0x02d103 thru 0x03b8f1 (0xe7ef bytes) 000302 1787144112.743681 mapwrite 0x003b87 thru 0x003c52 (0xcc bytes) 000304 1787144112.764348 read 0x00a455 thru 0x00a830 (0x3dc bytes) 000305 1787144112.767305 trunc from 0x03b359 to 0x03d968 (0x2610 bytes) 000306 1787144112.781122 read 0x00abdb thru 0x00c410 (0x1836 bytes) 000307 1787144112.789979 trunc from 0x03d969 to 0x004c48 (0x38d20 bytes) 000308 1787144112.851811 write 0x008cd4 thru 0x017d63 (0xf090 bytes) 000309 1787144112.857856 mapwrite 0x02ba0c thru 0x030d33 (0x5328 bytes) 000310 1787144112.952431 read 0x01fd11 thru 0x0274e2 (0x77d2 bytes) 000311 1787144112.953063 mapwrite 0x010407 thru 0x01cdf0 (0xc9ea bytes) 000312 1787144112.986918 write 0x00d2ce thru 0x00e26e (0xfa1 bytes) 000313 1787144112.987998 fallocate 0x02133c thru 0x02de3d (0xcb02 bytes) 000314 1787144112.993084 read 0x02b098 thru 0x030d33 (0x5c9c bytes) 000315 1787144112.993743 trunc from 0x02a4f7 to 0x02d277 (0x2d81 bytes) 000316 1787144113.000885 fallocate 0x01a7ab thru 0x01cfce (0x2824 bytes) 000317 1787144113.006120 mapwrite 0x03b2d7 thru 0x03ffff (0x4d29 bytes) 000318 1787144113.047684 mapread 0x027cde thru 0x030711 (0x8a34 bytes) 000319 1787144113.049667 write 0x0186f8 thru 0x01b4ca (0x2dd3 bytes) 000320 1787144113.053426 fallocate 0x00c7f1 thru 0x00d809 (0x1019 bytes) 000321 1787144113.058916 punch from 0x0043ea to 0x012b43 (0xe75a bytes) 000322 1787144113.066889 punch from 0x036d44 to 0x038591 (0x184e bytes) 000324 1787144113.074491 mapwrite 0x03a887 thru 0x03fbb3 (0x532d bytes) 000325 1787144113.093417 fallocate 0x018923 thru 0x021c78 (0x9356 bytes) 000326 1787144113.098743 write 0x02a27d thru 0x032f49 (0x8ccd bytes) 000327 1787144113.100430 fallocate 0x0207f8 thru 0x02544e (0x4c57 bytes) 000328 1787144113.105147 fallocate 0x01e95d thru 0x0268c3 (0x7f67 bytes) 000329 1787144113.109898 trunc from 0x040000 to 0x002821 (0x3d7de bytes) 000330 1787144113.126276 trunc from 0x002822 to 0x001a41 (0xde0 bytes) 000331 1787144113.141200 punch from 0x000c74 to 0x001a41 (0xdce bytes) 000333 1787144113.149160 read 0x000540 thru 0x001a41 (0x1502 bytes) 000334 1787144113.160168 MIRROR_EXTEND to 2 mirrors 000335 1787144113.863374 mapwrite 0x03bffe thru 0x03ddd3 (0x1dd6 bytes) 000336 1787144114.040726 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000337 1787144114.205175 mapwrite 0x005145 thru 0x00a332 (0x51ee bytes) 000338 1787144114.262140 trunc from 0x03ddd4 to 0x01918c (0x24c47 bytes) 000339 1787144114.297633 trunc from 0x007bcd to 0x01486e (0xcca2 bytes) 000340 1787144114.303457 trunc from 0x01918d to 0x030a8b (0x178ff bytes) 000341 1787144114.332912 fallocate 0x02e7cd thru 0x0313db (0x2c0f bytes) 000342 1787144114.339164 fallocate 0x0100f8 thru 0x01e942 (0xe84b bytes) 000344 1787144114.345602 write 0x0355e6 thru 0x03ffff (0xaa1a bytes) 000345 1787144114.348187 mapwrite 0x033144 thru 0x03a730 (0x75ed bytes) 000346 1787144114.378136 read 0x03271a thru 0x0338b4 (0x119b bytes) 000347 1787144114.379926 mapwrite 0x0085de thru 0x01122e (0x8c51 bytes) 000348 1787144114.429465 trunc from 0x040000 to 0x03c0fe (0x3f01 bytes) 000349 1787144114.460247 trunc from 0x003b03 to 0x013503 (0xfa01 bytes) 000350 1787144114.466924 read 0x021096 thru 0x030d3b (0xfca6 bytes) 000351 1787144114.483338 trunc from 0x03c0ff to 0x02d44f (0xecaf bytes) 000352 1787144114.506070 fallocate 0x030ea6 thru 0x035662 (0x47bd bytes) 000353 1787144114.513522 trunc from 0x0178de to 0x01f242 (0x7965 bytes) 000354 1787144114.522280 trunc from 0x01d74d to 0x01e6d8 (0xf8c bytes) 000355 1787144114.529258 read 0x00703a thru 0x0130ca (0xc091 bytes) 000356 1787144114.532035 trunc from 0x035663 to 0x004623 (0x3103f bytes) 000357 1787144114.556189 mapread 0x0038f9 thru 0x004623 (0xd2b bytes) 000358 1787144114.558269 write 0x033d5f thru 0x03ffff (0xc2a1 bytes) 000359 1787144114.562033 write 0x006efe thru 0x01179a (0xa89d bytes) 000360 1787144114.572451 trunc from 0x040000 to 0x032e89 (0xd176 bytes) 000361 1787144114.588767 mapwrite 0x013afe thru 0x018c4f (0x5152 bytes) 000362 1787144114.608838 trunc from 0x027c54 to 0x02af87 (0x3334 bytes) 000363 1787144114.613805 write 0x01ff09 thru 0x02d0f8 (0xd1f0 bytes) 000364 1787144114.616928 mapread 0x01a069 thru 0x0277e4 (0xd77c bytes) 000365 1787144114.618896 fallocate 0x00bda1 thru 0x00fd29 (0x3f89 bytes) 000366 1787144114.624145 punch from 0x00ae93 to 0x013b15 (0x8c83 bytes) 000367 1787144114.630493 mapwrite 0x01fee4 thru 0x02d232 (0xd34f bytes) 000368 1787144114.661437 read 0x00a344 thru 0x019c97 (0xf954 bytes) 000369 1787144114.678439 read 0x006b40 thru 0x00d4a2 (0x6963 bytes) 000371 1787144114.679795 punch from 0x003b68 to 0x00b1d1 (0x766a bytes) 000373 1787144114.686801 write 0x03f2bf thru 0x03ffff (0xd41 bytes) 000374 1787144114.688292 read 0x017fff thru 0x01a35c (0x235e bytes) 000375 1787144114.689437 write 0x021cf2 thru 0x0307bf (0xeace bytes) 000376 1787144114.692393 mapread 0x00adfc thru 0x0189c9 (0xdbce bytes) 000377 1787144114.703765 read 0x03465f thru 0x0391d7 (0x4b79 bytes) 000378 1787144114.704929 mapwrite 0x00e477 thru 0x01e036 (0xfbc0 bytes) 000379 1787144114.731735 mapwrite 0x02da68 thru 0x037cf9 (0xa292 bytes) 000380 1787144114.763121 trunc from 0x022946 to 0x022ed4 (0x58f bytes) 000381 1787144114.772867 read 0x00137a thru 0x0020b7 (0xd3e bytes) 000382 1787144114.773804 write 0x02ee68 thru 0x035225 (0x63be bytes) 000384 1787144114.778676 trunc from 0x040000 to 0x011617 (0x2e9e8 bytes) 000385 1787144114.808711 trunc from 0x00761d to 0x010c46 (0x962a bytes) 000386 1787144114.817925 punch from 0x00c625 to 0x011617 (0x4ff3 bytes) 000387 1787144114.829033 mapwrite 0x017501 thru 0x01f115 (0x7c15 bytes) 000388 1787144114.903135 read 0x01ad56 thru 0x01f115 (0x43c0 bytes) 000389 1787144114.905331 mapwrite 0x025337 thru 0x031da7 (0xca71 bytes) 000390 1787144114.975011 trunc from 0x031da8 to 0x01834d (0x19a5a bytes) 000391 1787144114.995117 mapread 0x00232a thru 0x0115a8 (0xf27f bytes) 000393 1787144114.998889 trunc from 0x013f72 to 0x016c51 (0x2ce0 bytes) 000394 1787144115.005355 read 0x00a024 thru 0x0117fc (0x77d9 bytes) 000395 1787144115.006356 write 0x03cac4 thru 0x03ffff (0x353c bytes) 000396 1787144115.009929 mapwrite 0x00b59e thru 0x0197a8 (0xe20b bytes) 000397 1787144115.049627 trunc from 0x0395cd to 0x03ffff (0x6a33 bytes) 000398 1787144115.055676 fallocate 0x02698d thru 0x02a51a (0x3b8e bytes) 000399 1787144115.061397 write 0x0013a5 thru 0x00699a (0x55f6 bytes) 000400 1787144115.063480 fallocate 0x030734 thru 0x0363aa (0x5c77 bytes) 000401 1787144115.069181 read 0x012d0f thru 0x022444 (0xf736 bytes) 000402 1787144115.072833 punch from 0x024162 to 0x02669d (0x253c bytes) 000403 1787144115.078370 punch from 0x030139 to 0x0302e1 (0x1a9 bytes) 000404 1787144115.082828 mapwrite 0x03a61a thru 0x03ffff (0x59e6 bytes) 000405 1787144115.111450 mapread 0x01d7a2 thru 0x023720 (0x5f7f bytes) 000406 1787144115.113934 mapwrite 0x003936 thru 0x00cc2c (0x92f7 bytes) 000407 1787144115.136321 punch from 0x00909e to 0x01806e (0xefd1 bytes) 000408 1787144115.143799 trunc from 0x040000 to 0x02aea0 (0x1515f bytes) 000409 1787144115.171028 write 0x02719b thru 0x027e6b (0xcd1 bytes) 000410 1787144115.173802 punch from 0x00022e to 0x00c2f8 (0xc0cb bytes) 000411 1787144115.179493 punch from 0x01d03b to 0x028d59 (0xbd1f bytes) 000414 1787144115.195849 mapwrite 0x005a89 thru 0x012002 (0xc57a bytes) 000415 1787144115.329701 trunc from 0x02aea1 to 0x02200c (0x8e94 bytes) 000416 1787144115.359535 mapread 0x01dfe1 thru 0x02200c (0x402c bytes) 000417 1787144115.399752 MIRROR_EXTEND to 2 mirrors 000418 1787144116.227051 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000419 1787144116.419571 MIRROR_EXTEND to 2 mirrors 000420 1787144117.309225 MIRROR_EXTEND to 3 mirrors 000421 1787144118.069468 trunc from 0x02200d to 0x010244 (0x11dc8 bytes) 000423 1787144118.123987 trunc from 0x0046f4 to 0x00ce84 (0x8791 bytes) 000424 1787144118.132774 read 0x004792 thru 0x007a18 (0x3287 bytes) 000425 1787144118.146210 read 0x00ebca thru 0x00f9e6 (0xe1d bytes) 000426 1787144118.160726 punch from 0x0051fe to 0x009d4d (0x4b50 bytes) 000427 1787144118.167753 write 0x00a960 thru 0x00b53b (0xbdc bytes) 000428 1787144118.173734 MIRROR_RESYNC 3 mirrors 000429 1787144119.033651 MIRROR_EXTEND to 4 mirrors 000430 1787144119.669434 trunc from 0x00cda8 to 0x010244 (0x349d bytes) 000431 1787144119.735080 trunc from 0x010245 to 0x020341 (0x100fd bytes) 000432 1787144119.800216 trunc from 0x020342 to 0x036a9c (0x1675b bytes) 000433 1787144119.823224 mapwrite 0x00a32f thru 0x0134d8 (0x91aa bytes) 000434 1787144119.893784 MIRROR_RESYNC 4 mirrors 000435 1787144120.888662 MIRROR_RESYNC 4 mirrors 000436 1787144121.300183 mapwrite 0x024036 thru 0x025319 (0x12e4 bytes) 000437 1787144121.405792 write 0x03b71a thru 0x03ffff (0x48e6 bytes) 000438 1787144121.407410 mapread 0x037954 thru 0x039210 (0x18bd bytes) 000439 1787144121.421071 mapwrite 0x015cf0 thru 0x018a51 (0x2d62 bytes) 000440 1787144121.458973 fallocate 0x03ff1c thru 0x03ffff (0xe4 bytes) 000441 1787144121.465454 punch from 0x020f39 to 0x02f5dc (0xe6a4 bytes) 000444 1787144121.474340 mapwrite 0x012085 thru 0x0214ce (0xf44a bytes) 000445 1787144121.512380 read 0x01bea5 thru 0x021c3c (0x5d98 bytes) 000446 1787144121.513767 mapread 0x01a48f thru 0x01c318 (0x1e8a bytes) 000447 1787144121.515023 trunc from 0x029609 to 0x02b934 (0x232c bytes) 000448 1787144121.521907 fallocate 0x0194e4 thru 0x01971a (0x237 bytes) 000449 1787144121.527160 read 0x031086 thru 0x0354ff (0x447a bytes) 000450 1787144121.527781 fallocate 0x02b792 thru 0x034659 (0x8ec8 bytes) 000451 1787144121.533033 mapwrite 0x032e59 thru 0x03485c (0x1a04 bytes) 000452 1787144121.544781 write 0x037824 thru 0x038c9e (0x147b bytes) 000454 1787144121.546129 fallocate 0x01e73c thru 0x023deb (0x56b0 bytes) 000455 1787144121.559799 MIRROR_RESYNC 4 mirrors 000456 1787144122.988961 trunc from 0x009b35 to 0x015b14 (0xbfe0 bytes) 000457 1787144123.058214 MIRROR_RESYNC 4 mirrors 000458 1787144124.490701 mapread 0x02d028 thru 0x0321d9 (0x51b2 bytes) 000459 1787144124.548628 MIRROR_RESYNC 4 mirrors 000460 1787144125.199149 write 0x027f56 thru 0x02a41e (0x24c9 bytes) 000462 1787144125.253053 trunc from 0x040000 to 0x0170c8 (0x28f37 bytes) 000463 1787144125.292459 mapread 0x010c75 thru 0x0170c8 (0x6454 bytes) 000465 1787144125.321033 mapwrite 0x024409 thru 0x027ed7 (0x3acf bytes) 000466 1787144125.387595 trunc from 0x00404a to 0x012e37 (0xedee bytes) 000467 1787144125.398366 fallocate 0x006573 thru 0x0072a8 (0xd36 bytes) 000468 1787144125.406526 trunc from 0x012c41 to 0x01d91f (0xacdf bytes) 000469 1787144125.419132 read 0x00058b thru 0x005a76 (0x54ec bytes) 000470 1787144125.435986 MIRROR_RESYNC 4 mirrors 000471 1787144126.229196 write 0x01fefc thru 0x020d0d (0xe12 bytes) 000472 1787144126.290636 write 0x01c98d thru 0x01f4bb (0x2b2f bytes) 000473 1787144126.308085 mapwrite 0x01893d thru 0x01ee9f (0x6563 bytes) 000474 1787144126.365899 read 0x017a38 thru 0x023cf4 (0xc2bd bytes) 000475 1787144126.374692 mapread 0x022db8 thru 0x027ed7 (0x5120 bytes) 000476 1787144126.390325 punch from 0x011667 to 0x012718 (0x10b2 bytes) 000478 1787144126.395873 fallocate 0x01e11e thru 0x02ce18 (0xecfb bytes) 000479 1787144126.401313 read 0x00434a thru 0x008b98 (0x484f bytes) 000480 1787144126.415887 trunc from 0x027ed8 to 0x011dcc (0x1610b bytes) 000481 1787144126.461126 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11dcd): Mirror 3: 0x5b7e5dd3 (stale) Mirror 4: 0x5b7e5dd3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11dcd): Mirror 3: 0x5b7e5dd3 (stale) Mirror 4: 0x5b7e5dd3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11dcd: skipped 000482 1787144126.839359 punch from 0x008cbf to 0x011dcc (0x910e bytes) 000485 1787144126.847437 read 0x0109d1 thru 0x011dcc (0x13fc bytes) 000486 1787144126.853888 punch from 0x0096e9 to 0x011dcc (0x86e4 bytes) 000487 1787144126.860384 mapwrite 0x021a69 thru 0x0237a1 (0x1d39 bytes) 000488 1787144126.924162 mapread 0x00d10c thru 0x01c7a5 (0xf69a bytes) 000489 1787144126.954272 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x237a2): Mirror 4: 0xb6c0977a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x237a2): Mirror 4: 0xb6c0977a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x237a2: skipped 000490 1787144127.283667 fallocate 0x032add thru 0x03904c (0x6570 bytes) 000491 1787144127.296476 trunc from 0x0023b7 to 0x00f787 (0xd3d1 bytes) 000492 1787144127.303790 punch from 0x010e8c to 0x01f31f (0xe494 bytes) 000493 1787144127.312621 mapread 0x014194 thru 0x01bc3e (0x7aab bytes) 000494 1787144127.343493 read 0x004b22 thru 0x00ee1a (0xa2f9 bytes) 000495 1787144127.352363 trunc from 0x00a19f to 0x01334a (0x91ac bytes) 000496 1787144127.357766 punch from 0x00f7c0 to 0x00ff9e (0x7df bytes) 000497 1787144127.362480 trunc from 0x0059d6 to 0x010f2c (0xb557 bytes) 000498 1787144127.367493 mapread 0x0163a6 thru 0x01df49 (0x7ba4 bytes) 000499 1787144127.369186 mapwrite 0x023b2c thru 0x031193 (0xd668 bytes) 000500 1787144127.454853 trunc from 0x031194 to 0x01eeb2 (0x122e1 bytes) 000501 1787144127.478467 trunc from 0x00f1c7 to 0x01527b (0x60b5 bytes) 000502 1787144127.488745 MIRROR_RESYNC 2 mirrors 000503 1787144128.065891 MIRROR_EXTEND to 3 mirrors 000504 1787144128.916596 trunc from 0x01eeb3 to 0x024ee7 (0x6035 bytes) 000505 1787144128.967742 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24ee8): Mirror 4: 0x78ce384a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24ee8): Mirror 4: 0x78ce384a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24ee8: skipped 000506 1787144129.447568 write 0x02dff7 thru 0x037011 (0x901b bytes) 000507 1787144129.460639 trunc from 0x035c8a to 0x035f5c (0x2d3 bytes) 000508 1787144129.495871 write 0x016e88 thru 0x021ced (0xae66 bytes) 000509 1787144129.510336 write 0x03eee4 thru 0x03ffff (0x111c bytes) 000510 1787144129.516321 MIRROR_RESYNC 2 mirrors 000512 1787144130.353646 read 0x038c85 thru 0x03ffff (0x737b bytes) 000513 1787144130.387598 MIRROR_EXTEND to 3 mirrors 000514 1787144131.327183 MIRROR_EXTEND to 4 mirrors 000515 1787144132.144132 fallocate 0x021337 thru 0x02512f (0x3df9 bytes) 000516 1787144132.197314 write 0x03f9aa thru 0x03ffff (0x656 bytes) 000517 1787144132.212462 trunc from 0x040000 to 0x0213f0 (0x1ec0f bytes) 000518 1787144132.243288 fallocate 0x039c5a thru 0x03ffff (0x63a6 bytes) 000519 1787144132.254481 trunc from 0x006dfd to 0x011c20 (0xae24 bytes) 000520 1787144132.260501 mapread 0x009c8c thru 0x0130f4 (0x9469 bytes) 000521 1787144132.281878 write 0x0151a5 thru 0x0213bc (0xc218 bytes) 000523 1787144132.295718 fallocate 0x033f2c thru 0x03b14a (0x721f bytes) 000524 1787144132.303867 trunc from 0x040000 to 0x016aae (0x29551 bytes) 000525 1787144132.324893 punch from 0x0161c0 to 0x016aae (0x8ef bytes) 000526 1787144132.349370 trunc from 0x016aaf to 0x02f8af (0x18e01 bytes) 000527 1787144132.380709 trunc from 0x02f8b0 to 0x01daaa (0x11e05 bytes) 000528 1787144132.418116 trunc from 0x013786 to 0x0194ba (0x5d35 bytes) 000529 1787144132.423762 fallocate 0x011bc9 thru 0x018443 (0x687b bytes) 000530 1787144132.432223 mapwrite 0x02a551 thru 0x02b324 (0xdd4 bytes) 000531 1787144132.498977 write 0x0367de thru 0x03db83 (0x73a6 bytes) 000532 1787144132.500792 trunc from 0x03db84 to 0x017316 (0x2686d bytes) 000533 1787144132.525035 trunc from 0x012c2c to 0x017316 (0x46eb bytes) 000534 1787144132.548989 MIRROR_RESYNC 4 mirrors 000535 1787144133.575367 trunc from 0x01685b to 0x017316 (0xabc bytes) 000536 1787144133.674911 punch from 0x00c3b3 to 0x017316 (0xaf64 bytes) 000538 1787144133.690363 mapread 0x0101f3 thru 0x017316 (0x7124 bytes) 000539 1787144133.717742 trunc from 0x017317 to 0x01171a (0x5bfc bytes) 000540 1787144133.746627 mapwrite 0x0323d2 thru 0x032fd1 (0xc00 bytes) 000541 1787144133.810715 punch from 0x01a337 to 0x021e8f (0x7b59 bytes) 000542 1787144133.820575 trunc from 0x032fd2 to 0x00e72c (0x248a5 bytes) 000543 1787144133.845311 mapwrite 0x02899c thru 0x033646 (0xacab bytes) 000544 1787144133.943507 trunc from 0x033647 to 0x02f332 (0x4314 bytes) 000545 1787144133.971432 mapread 0x018c1e thru 0x023cfc (0xb0df bytes) 000546 1787144133.995266 write 0x02a7b8 thru 0x02c155 (0x199e bytes) 000547 1787144133.998219 trunc from 0x017436 to 0x01d3a9 (0x5f74 bytes) 000548 1787144134.011735 trunc from 0x02f333 to 0x016b70 (0x187c2 bytes) 000549 1787144134.033201 punch from 0x006d11 to 0x009d57 (0x3047 bytes) 000550 1787144134.039735 mapread 0x002ab3 thru 0x00a45b (0x79a9 bytes) 000551 1787144134.056510 write 0x03b354 thru 0x03ecf6 (0x39a3 bytes) 000552 1787144134.064544 MIRROR_RESYNC 4 mirrors 000553 1787144134.994196 punch from 0x024227 to 0x024e98 (0xc72 bytes) 000554 1787144135.049847 read 0x031cf2 thru 0x037336 (0x5645 bytes) 000555 1787144135.080725 punch from 0x0315cf to 0x03abff (0x9631 bytes) 000556 1787144135.085628 mapwrite 0x03832e thru 0x03cf68 (0x4c3b bytes) 000557 1787144135.141183 trunc from 0x03ecf7 to 0x002678 (0x3c67e bytes) 000558 1787144135.212718 read 0x002338 thru 0x002678 (0x341 bytes) 000559 1787144135.222808 mapread 0x0006b5 thru 0x002678 (0x1fc4 bytes) 000560 1787144135.223705 mapwrite 0x0206a3 thru 0x026a70 (0x63ce bytes) 000561 1787144135.332556 fallocate 0x0048fb thru 0x00a696 (0x5d9c bytes) 000562 1787144135.343386 write 0x01ad87 thru 0x01bd7e (0xff8 bytes) 000563 1787144135.346873 trunc from 0x0169d1 to 0x01ce47 (0x6477 bytes) 000564 1787144135.373329 fallocate 0x011595 thru 0x01c477 (0xaee3 bytes) 000565 1787144135.379555 punch from 0x01e37c to 0x025e65 (0x7aea bytes) 000566 1787144135.385285 trunc from 0x026a71 to 0x0136f4 (0x1337c bytes) 000567 1787144135.414634 mapwrite 0x011a1e thru 0x01865a (0x6c3d bytes) 000568 1787144135.508079 write 0x02cded thru 0x032bb1 (0x5dc5 bytes) 000569 1787144135.513896 read 0x00fad2 thru 0x0178ee (0x7e1d bytes) 000570 1787144135.514781 write 0x008343 thru 0x00f6b8 (0x7376 bytes) 000571 1787144135.518460 trunc from 0x032bb2 to 0x0097a6 (0x2940b bytes) 000572 1787144135.564617 mapwrite 0x00c515 thru 0x01a4a4 (0xdf90 bytes) 000573 1787144135.676476 trunc from 0x005663 to 0x00f623 (0x9fc1 bytes) 000574 1787144135.694904 MIRROR_RESYNC 4 mirrors 000575 1787144136.722192 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000576 1787144137.204571 write 0x01e059 thru 0x02773f (0x96e7 bytes) 000577 1787144137.266375 write 0x014e0c thru 0x01a18a (0x537f bytes) 000578 1787144137.276913 read 0x0184cf thru 0x0209f1 (0x8523 bytes) 000579 1787144137.282895 mapwrite 0x01093a thru 0x01c51f (0xbbe6 bytes) 000580 1787144137.332885 trunc from 0x0071f0 to 0x00a49a (0x32ab bytes) 000581 1787144137.347134 MIRROR_RESYNC 3 mirrors 000582 1787144138.253743 fallocate 0x014489 thru 0x020b77 (0xc6ef bytes) 000583 1787144138.311536 fallocate 0x0367a1 thru 0x03ffff (0x985f bytes) 000584 1787144138.319396 write 0x0376de thru 0x03c2bc (0x4bdf bytes) 000585 1787144138.322189 read 0x022fb2 thru 0x02dd23 (0xad72 bytes) 000586 1787144138.332867 read 0x00c4a7 thru 0x00ee39 (0x2993 bytes) 000587 1787144138.345610 write 0x02e582 thru 0x0360cd (0x7b4c bytes) 000588 1787144138.347558 trunc from 0x03c2bd to 0x02b653 (0x10c69 bytes) 000589 1787144138.396083 mapread 0x005769 thru 0x013135 (0xd9cd bytes) 000590 1787144138.398243 punch from 0x01ff8f to 0x02b653 (0xb6c5 bytes) 000591 1787144138.405168 write 0x02c291 thru 0x0313cc (0x513c bytes) 000592 1787144138.406589 mapwrite 0x00ec62 thru 0x012fc5 (0x4364 bytes) 000593 1787144138.438887 read 0x00f4a5 thru 0x013709 (0x4265 bytes) 000594 1787144138.439767 mapread 0x0007bd thru 0x00bea9 (0xb6ed bytes) 000595 1787144138.441521 write 0x024a3c thru 0x031152 (0xc717 bytes) 000596 1787144138.452688 fallocate 0x035012 thru 0x03ab3f (0x5b2e bytes) 000597 1787144138.460516 mapread 0x01a1b9 thru 0x01afbe (0xe06 bytes) 000598 1787144138.467905 write 0x022ecc thru 0x031f23 (0xf058 bytes) 000599 1787144138.471537 read 0x023a76 thru 0x02ad8e (0x7319 bytes) 000600 1787144138.476576 write 0x01cc7c thru 0x02795d (0xace2 bytes) 000601 1787144138.481037 mapwrite 0x026473 thru 0x02a177 (0x3d05 bytes) 000602 1787144138.514182 read 0x00f470 thru 0x0148a1 (0x5432 bytes) 000603 1787144138.515348 mapwrite 0x02d568 thru 0x0364c1 (0x8f5a bytes) 000604 1787144138.567315 trunc from 0x0364c2 to 0x02b076 (0xb44b bytes) 000605 1787144138.601316 trunc from 0x01f91d to 0x02790f (0x7ff3 bytes) 000606 1787144138.613268 mapread 0x008160 thru 0x010dbe (0x8c5f bytes) 000607 1787144138.620504 fallocate 0x008929 thru 0x01848c (0xfb64 bytes) 000608 1787144138.629544 mapwrite 0x01d95f thru 0x0253ac (0x7a4e bytes) 000609 1787144138.695830 read 0x028418 thru 0x02b076 (0x2c5f bytes) 000610 1787144138.696933 read 0x014a7b thru 0x01ec4d (0xa1d3 bytes) 000611 1787144138.699216 write 0x014279 thru 0x01b6f7 (0x747f bytes) 000612 1787144138.701404 trunc from 0x02b077 to 0x039222 (0xe1ac bytes) 000613 1787144138.725925 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39223): Mirror 6: 0x945ed2f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39223): Mirror 6: 0x945ed2f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39223: skipped 000614 1787144139.073540 punch from 0x019d22 to 0x01ce37 (0x3116 bytes) 000615 1787144139.084906 trunc from 0x039223 to 0x01b4e1 (0x1dd41 bytes) 000616 1787144139.116960 write 0x00f0fd thru 0x01ef7e (0xfe82 bytes) 000617 1787144139.124685 trunc from 0x01eb70 to 0x01ef7e (0x40f bytes) 000618 1787144139.144432 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000619 1787144139.280827 mapwrite 0x031c18 thru 0x03b9dd (0x9dc6 bytes) 000620 1787144139.369293 mapwrite 0x00949e thru 0x015ca7 (0xc80a bytes) 000621 1787144139.454467 punch from 0x02e1af to 0x03b9dd (0xd82f bytes) 000623 1787144139.460920 write 0x034dd3 thru 0x0379a4 (0x2bd2 bytes) 000624 1787144139.471475 write 0x01a6de thru 0x01dcc0 (0x35e3 bytes) 000625 1787144139.477997 trunc from 0x03b9de to 0x0120c1 (0x2991c bytes) 000626 1787144139.499145 MIRROR_EXTEND to 2 mirrors 000627 1787144140.274848 mapread 0x00aa44 thru 0x0120c1 (0x767e bytes) 000628 1787144140.311818 MIRROR_RESYNC 2 mirrors 000629 1787144140.804719 read 0x0057f7 thru 0x00f1b5 (0x99bf bytes) 000630 1787144140.805252 fallocate 0x000d4a thru 0x00ce50 (0xc107 bytes) 000631 1787144140.878339 MIRROR_RESYNC 2 mirrors 000632 1787144141.673523 punch from 0x001464 to 0x00bbf2 (0xa78f bytes) 000633 1787144141.715352 read 0x00e636 thru 0x0120c1 (0x3a8c bytes) 000634 1787144141.730731 read 0x00f0cb thru 0x01100f (0x1f45 bytes) 000635 1787144141.736822 write 0x01d4cf thru 0x025eaa (0x89dc bytes) 000636 1787144141.742610 mapwrite 0x023b62 thru 0x02df81 (0xa420 bytes) 000638 1787144141.859462 write 0x005125 thru 0x012298 (0xd174 bytes) 000639 1787144141.862542 trunc from 0x00f68d to 0x0157c3 (0x6137 bytes) 000640 1787144141.882842 punch from 0x020aba to 0x02df81 (0xd4c8 bytes) 000641 1787144141.890440 write 0x01151f thru 0x017f90 (0x6a72 bytes) 000643 1787144141.900602 write 0x02972d thru 0x033de7 (0xa6bb bytes) 000644 1787144141.913909 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33de8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33de8): Mirror 2: 0xe63d2c38 (stale) CRC-32 checksum value for chunk [0, 0x33de8): Mirror 1: 0x12590a6d Mirror 3: 0x12590a6d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33de8: skipped 000645 1787144143.156519 write 0x02d2cf thru 0x03306f (0x5da1 bytes) 000647 1787144143.243874 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33de8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33de8): Mirror 2: 0xe63d2c38 (stale) Mirror 3: 0x12590a6d (stale) CRC-32 checksum value for chunk [0, 0x33de8): Mirror 1: 0x53682b39 Mirror 4: 0x53682b39 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33de8: skipped 000648 1787144144.649372 mapread 0x017a20 thru 0x021feb (0xa5cc bytes) 000649 1787144144.682765 mapread 0x0196b1 thru 0x025a1b (0xc36b bytes) 000650 1787144144.700309 write 0x028856 thru 0x03361b (0xadc6 bytes) 000651 1787144144.777214 fallocate 0x01b008 thru 0x0266e4 (0xb6dd bytes) 000652 1787144144.782525 mapwrite 0x000023 thru 0x00df47 (0xdf25 bytes) 000655 1787144144.830350 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33de8): Mirror 3: 0x12590a6d (stale) Mirror 4: 0x53682b39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33de8): Mirror 3: 0x12590a6d (stale) Mirror 4: 0x53682b39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33de8: skipped 000657 1787144145.317163 mapread 0x00c9ab thru 0x0103bc (0x3a12 bytes) 000658 1787144145.344335 read 0x028726 thru 0x02c6d1 (0x3fac bytes) 000659 1787144145.355200 mapread 0x022b12 thru 0x02f472 (0xc961 bytes) 000660 1787144145.385642 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33de8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33de8): Mirror 3: 0x12590a6d (stale) Mirror 4: 0x53682b39 (stale) CRC-32 checksum value for chunk [0, 0x33de8): Mirror 1: 0x253e615f Mirror 5: 0x253e615f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33de8: skipped 000661 1787144146.782890 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33de8): Mirror 4: 0x53682b39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33de8): Mirror 4: 0x53682b39 (stale) CRC-32 checksum value for chunk [0, 0x33de8): Mirror 1: 0x253e615f Mirror 5: 0x253e615f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33de8: skipped 000662 1787144147.800647 mapread 0x01b869 thru 0x023218 (0x79b0 bytes) 000663 1787144147.888045 read 0x02e647 thru 0x033de7 (0x57a1 bytes) 000665 1787144147.930983 read 0x025e27 thru 0x033de7 (0xdfc1 bytes) 000666 1787144148.002187 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33de8): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33de8): Mirror 4: 0x53682b39 (stale) CRC-32 checksum value for chunk [0, 0x33de8): Mirror 1: 0x253e615f Mirror 5: 0x253e615f Mirror 6: 0x253e615f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33de8: skipped 000667 1787144149.143562 write 0x011dd0 thru 0x016329 (0x455a bytes) 000668 1787144149.194919 trunc from 0x033de8 to 0x03efd8 (0xb1f1 bytes) 000669 1787144149.213527 write 0x03c4e4 thru 0x03ffff (0x3b1c bytes) 000670 1787144149.219861 write 0x010fe8 thru 0x012e53 (0x1e6c bytes) 000671 1787144149.226769 punch from 0x00992b to 0x00fee6 (0x65bc bytes) 000672 1787144149.232638 punch from 0x0218cc to 0x022dd3 (0x1508 bytes) 000673 1787144149.237331 trunc from 0x017813 to 0x01ee1c (0x760a bytes) 000674 1787144149.242397 punch from 0x004f93 to 0x012271 (0xd2df bytes) 000675 1787144149.259247 punch from 0x037b1a to 0x03ffff (0x84e6 bytes) 000676 1787144149.264005 mapwrite 0x0234ec thru 0x02883c (0x5351 bytes) 000677 1787144149.300549 read 0x01cfb4 thru 0x02866e (0xb6bb bytes) 000678 1787144149.307123 fallocate 0x035e0a thru 0x03ffff (0xa1f6 bytes) 000680 1787144149.314503 read 0x004d6e thru 0x00bbd3 (0x6e66 bytes) 000681 1787144149.332552 read 0x0095d3 thru 0x010123 (0x6b51 bytes) 000682 1787144149.333313 write 0x03b3c0 thru 0x03ffff (0x4c40 bytes) 000683 1787144149.334915 trunc from 0x028b54 to 0x02aa80 (0x1f2d bytes) 000684 1787144149.340776 read 0x0128ac thru 0x019dfc (0x7551 bytes) 000686 1787144149.342259 mapread 0x035ea0 thru 0x03ffbc (0xa11d bytes) 000687 1787144149.344158 mapread 0x0170c7 thru 0x01e4fa (0x7434 bytes) 000688 1787144149.349653 MIRROR_RESYNC 4 mirrors 000689 1787144151.691125 write 0x01abef thru 0x028c9c (0xe0ae bytes) 000690 1787144151.744090 mapread 0x03b0f3 thru 0x03ffff (0x4f0d bytes) 000691 1787144151.781097 trunc from 0x037b80 to 0x03ffff (0x8480 bytes) 000692 1787144151.787240 fallocate 0x012ec0 thru 0x016bdc (0x3d1d bytes) 000693 1787144151.796568 mapwrite 0x002917 thru 0x00786c (0x4f56 bytes) 000694 1787144151.842063 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x92ca8da2 (stale) Mirror 5: 0x92ca8da2 (stale) Mirror 6: 0x92ca8da2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x98ba5adc Mirror 7: 0x98ba5adc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000695 1787144153.185331 mapwrite 0x01d756 thru 0x020b16 (0x33c1 bytes) 000696 1787144153.290267 read 0x02a8bf thru 0x0322e1 (0x7a23 bytes) 000697 1787144153.298067 trunc from 0x040000 to 0x0193d2 (0x26c2d bytes) 000698 1787144153.331048 write 0x035c9a thru 0x03a11b (0x4482 bytes) 000699 1787144153.334226 mapread 0x00fbc9 thru 0x011e5d (0x2295 bytes) 000700 1787144153.353441 trunc from 0x03a11c to 0x01f891 (0x1a88a bytes) 000702 1787144153.375694 read 0x00be0c thru 0x01a179 (0xe36e bytes) 000703 1787144153.389900 mapwrite 0x02fed2 thru 0x0393a1 (0x94d0 bytes) 000704 1787144153.453282 trunc from 0x00f748 to 0x01373e (0x3ff7 bytes) 000705 1787144153.459716 punch from 0x01b46a to 0x02a11e (0xecb5 bytes) 000706 1787144153.470173 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x393a2): Mirror 5: 0xf5c0767e (stale) Mirror 6: 0xf5c0767e (stale) Mirror 7: 0x33f76e7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x393a2): Mirror 5: 0xf5c0767e (stale) Mirror 6: 0xf5c0767e (stale) Mirror 7: 0x33f76e7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x393a2: skipped 000707 1787144153.912690 write 0x004268 thru 0x012606 (0xe39f bytes) 000710 1787144153.937033 punch from 0x005605 to 0x0062ab (0xca7 bytes) 000711 1787144153.955282 write 0x00fe70 thru 0x01cc82 (0xce13 bytes) 000712 1787144153.963523 trunc from 0x0393a2 to 0x03cf18 (0x3b77 bytes) 000713 1787144153.983338 trunc from 0x03cf19 to 0x008573 (0x349a5 bytes) 000715 1787144154.005583 write 0x031f84 thru 0x03d6ef (0xb76c bytes) 000716 1787144154.008710 punch from 0x038f13 to 0x03d6ef (0x47dd bytes) 000717 1787144154.023629 write 0x00da8d thru 0x0111ea (0x375e bytes) 000718 1787144154.036497 mapread 0x02578f thru 0x02b5aa (0x5e1c bytes) 000719 1787144154.057715 fallocate 0x01df27 thru 0x025484 (0x755e bytes) 000720 1787144154.064301 trunc from 0x03d6f0 to 0x001491 (0x3c25e bytes) 000721 1787144154.080935 read 0x0001f9 thru 0x001491 (0x1299 bytes) 000722 1787144154.089587 fallocate 0x03ac3a thru 0x03b608 (0x9cf bytes) 000723 1787144154.101723 punch from 0x0181f9 to 0x023b81 (0xb989 bytes) 000724 1787144154.107569 mapwrite 0x03e69b thru 0x03ffff (0x1965 bytes) 000725 1787144154.158601 read 0x01fa80 thru 0x02b572 (0xbaf3 bytes) 000726 1787144154.194203 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x92ca8da2 (stale) Mirror 7: 0x98ba5adc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x92ca8da2 (stale) Mirror 7: 0x98ba5adc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000727 1787144154.724599 trunc from 0x032d21 to 0x034388 (0x1668 bytes) 000728 1787144154.734449 trunc from 0x040000 to 0x03f05e (0xfa1 bytes) 000729 1787144154.760216 read 0x0397c4 thru 0x03b637 (0x1e74 bytes) 000730 1787144154.769103 trunc from 0x033e11 to 0x0340d3 (0x2c3 bytes) 000731 1787144154.774354 trunc from 0x03f05f to 0x0093f0 (0x35c6e bytes) 000732 1787144154.793283 write 0x00cf59 thru 0x00ea22 (0x1aca bytes) 000733 1787144154.794553 mapwrite 0x0006ed thru 0x00cbc1 (0xc4d5 bytes) 000734 1787144154.847290 mapwrite 0x024726 thru 0x031807 (0xd0e2 bytes) 000735 1787144154.912603 trunc from 0x0268ec to 0x031807 (0xaf1c bytes) 000736 1787144154.921324 read 0x016cbb thru 0x023018 (0xc35e bytes) 000737 1787144154.936661 punch from 0x02e079 to 0x031807 (0x378f bytes) 000738 1787144154.943593 trunc from 0x002c05 to 0x009c6d (0x7069 bytes) 000739 1787144154.953216 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31808): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31808): Mirror 6: 0xf8b1aeb1 (stale) Mirror 7: 0xcb12781c (stale) CRC-32 checksum value for chunk [0, 0x31808): Mirror 1: 0xe320cb67 Mirror 8: 0xe320cb67 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31808: skipped 000740 1787144156.269149 mapwrite 0x03b04f thru 0x03ffff (0x4fb1 bytes) 000741 1787144156.419525 trunc from 0x040000 to 0x0120e3 (0x2df1c bytes) 000742 1787144156.448657 mapwrite 0x00b98b thru 0x015e43 (0xa4b9 bytes) 000744 1787144156.512755 trunc from 0x015e44 to 0x00f736 (0x670d bytes) 000745 1787144156.534780 mapread 0x008b56 thru 0x00b251 (0x26fc bytes) 000746 1787144156.543116 trunc from 0x001610 to 0x004dba (0x37ab bytes) 000747 1787144156.552855 trunc from 0x00f737 to 0x0213ec (0x11cb6 bytes) 000748 1787144156.596774 trunc from 0x0213ed to 0x039975 (0x18589 bytes) 000749 1787144156.622717 fallocate 0x02ab31 thru 0x034e8c (0xa35c bytes) 000750 1787144156.635693 trunc from 0x039976 to 0x037c1d (0x1d58 bytes) 000751 1787144156.701255 mapwrite 0x006b98 thru 0x008aaf (0x1f18 bytes) 000752 1787144156.736288 mapwrite 0x00d4dd thru 0x01157b (0x409f bytes) 000753 1787144156.780637 trunc from 0x037c1e to 0x01f628 (0x185f5 bytes) 000754 1787144156.805451 trunc from 0x010bf9 to 0x011571 (0x979 bytes) 000755 1787144156.812306 fallocate 0x014430 thru 0x0158f8 (0x14c9 bytes) 000756 1787144156.821758 trunc from 0x01f629 to 0x03ccea (0x1d6c2 bytes) 000757 1787144156.848625 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cceb): Mirror 6: 0x5a5a907c (stale) Mirror 7: 0xe5ccfab1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cceb): Mirror 6: 0x5a5a907c (stale) Mirror 7: 0xe5ccfab1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cceb: skipped 000758 1787144157.386775 trunc from 0x02892d to 0x038148 (0xf81c bytes) 000760 1787144157.395660 read 0x012cec thru 0x01744f (0x4764 bytes) 000761 1787144157.410108 MIRROR_RESYNC 3 mirrors 000762 1787144158.726194 trunc from 0x03cceb to 0x02320f (0x19adb bytes) 000763 1787144158.775842 punch from 0x016322 to 0x01f589 (0x9268 bytes) 000764 1787144158.783583 mapwrite 0x02a1de thru 0x0322d8 (0x80fb bytes) 000765 1787144158.868426 trunc from 0x0322d9 to 0x033d99 (0x1ac1 bytes) 000766 1787144158.890911 mapwrite 0x01688b thru 0x0222cc (0xba42 bytes) 000767 1787144158.977860 write 0x01ae33 thru 0x0255a1 (0xa76f bytes) 000768 1787144158.984915 read 0x011f96 thru 0x01f969 (0xd9d4 bytes) 000769 1787144158.986611 trunc from 0x00df54 to 0x018bfd (0xacaa bytes) 000770 1787144158.994288 mapread 0x00f652 thru 0x01248b (0x2e3a bytes) 000771 1787144159.005243 trunc from 0x033d9a to 0x03f07a (0xb2e1 bytes) 000772 1787144159.023510 trunc from 0x0342a8 to 0x03b644 (0x739d bytes) 000773 1787144159.030749 read 0x02ff38 thru 0x036645 (0x670e bytes) 000774 1787144159.055222 trunc from 0x03f07b to 0x0008bf (0x3e7bb bytes) 000775 1787144159.083318 MIRROR_RESYNC 3 mirrors 000776 1787144159.959174 read 0x00043d thru 0x0008bf (0x483 bytes) 000777 1787144159.969038 trunc from 0x000076 to 0x0008bf (0x84a bytes) 000778 1787144160.008793 read 0x0007fd thru 0x0008bf (0xc3 bytes) 000779 1787144160.017768 trunc from 0x0008c0 to 0x01e227 (0x1d968 bytes) 000780 1787144160.067049 mapread 0x014b1a thru 0x01c1b1 (0x7698 bytes) 000781 1787144160.104761 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1e228): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e228): Mirror 6: 0x99377ba1 (stale) Mirror 7: 0x99377ba1 (stale) CRC-32 checksum value for chunk [0, 0x1e228): Mirror 1: 0x2c13645d Mirror 8: 0x2c13645d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e228: skipped 000782 1787144161.084224 mapwrite 0x019776 thru 0x01b138 (0x19c3 bytes) 000783 1787144161.202832 mapwrite 0x0360dd thru 0x03ffff (0x9f23 bytes) 000784 1787144161.315545 fallocate 0x00fff5 thru 0x01f7f7 (0xf803 bytes) 000785 1787144161.327121 read 0x03a8c6 thru 0x03ccba (0x23f5 bytes) 000786 1787144161.327722 mapwrite 0x028203 thru 0x031c9a (0x9a98 bytes) 000787 1787144161.370797 write 0x019e6d thru 0x01af91 (0x1125 bytes) 000788 1787144161.372387 punch from 0x03ddce to 0x03ffff (0x2232 bytes) 000789 1787144161.385045 MIRROR_RESYNC 4 mirrors 000790 1787144163.074792 trunc from 0x040000 to 0x03cc48 (0x33b7 bytes) 000791 1787144163.129772 punch from 0x00c96e to 0x01ab85 (0xe218 bytes) 000792 1787144163.137192 fallocate 0x009ba1 thru 0x00b911 (0x1d71 bytes) 000793 1787144163.153094 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cc49): Mirror 6: 0xe6b7867b (stale) Mirror 8: 0xe6b7867b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc49): Mirror 6: 0xe6b7867b (stale) Mirror 8: 0xe6b7867b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc49: skipped 000795 1787144163.659012 mapwrite 0x02293f thru 0x02e376 (0xba38 bytes) 000796 1787144163.737725 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cc49): Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc49): Mirror 6: 0xe6b7867b (stale) Mirror 8: 0xe6b7867b (stale) CRC-32 checksum value for chunk [0, 0x3cc49): Mirror 1: 0x5d4df84e Mirror 9: 0x5d4df84e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc49: skipped 000797 1787144165.190662 punch from 0x029046 to 0x038448 (0xf403 bytes) 000798 1787144165.286094 mapwrite 0x03010b thru 0x036b6d (0x6a63 bytes) 000799 1787144165.330397 punch from 0x02a416 to 0x032324 (0x7f0f bytes) 000800 1787144165.336504 fallocate 0x002a52 thru 0x0115dc (0xeb8b bytes) 000801 1787144165.342033 write 0x02411c thru 0x02a54b (0x6430 bytes) 000802 1787144165.363644 trunc from 0x03cc49 to 0x02520a (0x17a3e bytes) 000803 1787144165.386090 punch from 0x018148 to 0x01dfd6 (0x5e8f bytes) 000805 1787144165.399710 read 0x010fc0 thru 0x01dfc9 (0xd00a bytes) 000806 1787144165.432833 trunc from 0x02520b to 0x031a87 (0xc87d bytes) 000807 1787144165.473487 mapwrite 0x02f179 thru 0x0362e6 (0x716e bytes) 000808 1787144165.559880 trunc from 0x0362e7 to 0x0248ea (0x119fc bytes) 000809 1787144165.592357 trunc from 0x0248eb to 0x039363 (0x14a79 bytes) 000810 1787144165.620892 write 0x020954 thru 0x020f5b (0x608 bytes) 000813 1787144165.629918 write 0x01fcf6 thru 0x02c96a (0xcc75 bytes) 000814 1787144165.643904 trunc from 0x039364 to 0x00d524 (0x2be3f bytes) 000815 1787144165.662930 read 0x0094a9 thru 0x00d524 (0x407c bytes) 000816 1787144165.677607 trunc from 0x00af8a to 0x00d524 (0x259b bytes) 000817 1787144165.687505 MIRROR_RESYNC 4 mirrors 000818 1787144166.829042 read 0x007e52 thru 0x00d524 (0x56d3 bytes) 000819 1787144166.842534 trunc from 0x0014b8 to 0x00165b (0x1a4 bytes) 000821 1787144166.936735 write 0x0209b0 thru 0x027f4e (0x759f bytes) 000822 1787144166.941327 mapread 0x01df6e thru 0x027f4e (0x9fe1 bytes) 000823 1787144166.966483 mapread 0x006b62 thru 0x0120d7 (0xb576 bytes) 000824 1787144167.001928 trunc from 0x013cfc to 0x018f35 (0x523a bytes) 000825 1787144167.017264 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27f4f): Mirror 8: 0x8a93fed5 (stale) Mirror 9: 0x8a93fed5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27f4f): Mirror 8: 0x8a93fed5 (stale) Mirror 9: 0x8a93fed5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27f4f: skipped 000826 1787144167.368492 punch from 0x0202f6 to 0x0244e1 (0x41ec bytes) 000827 1787144167.378844 trunc from 0x003e42 to 0x012878 (0xea37 bytes) 000828 1787144167.387931 mapwrite 0x00746a thru 0x0093cd (0x1f64 bytes) 000830 1787144167.424800 mapread 0x00df98 thru 0x011f45 (0x3fae bytes) 000831 1787144167.425987 punch from 0x026d67 to 0x027f4e (0x11e8 bytes) 000832 1787144167.431665 trunc from 0x027f4f to 0x0059b2 (0x2259c bytes) 000833 1787144167.452062 mapread 0x001567 thru 0x0059b2 (0x444c bytes) 000834 1787144167.458041 trunc from 0x0059b3 to 0x02fdde (0x2a42c bytes) 000835 1787144167.482830 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fddf): Mirror 9: 0xdd28c434 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fddf): Mirror 9: 0xdd28c434 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fddf: skipped 000836 1787144167.779908 trunc from 0x01acd9 to 0x027a7f (0xcda7 bytes) 000837 1787144167.791425 MIRROR_RESYNC 2 mirrors 000838 1787144168.652446 read 0x010112 thru 0x019393 (0x9282 bytes) 000839 1787144168.686193 write 0x039b7e thru 0x03ffff (0x6482 bytes) 000840 1787144168.763534 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x1218311d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe32966ff Mirror 10: 0xe32966ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000841 1787144170.205806 read 0x0057ff thru 0x00a139 (0x493b bytes) 000842 1787144170.232847 fallocate 0x036c24 thru 0x03e14f (0x752c bytes) 000844 1787144170.315259 trunc from 0x040000 to 0x03864b (0x79b4 bytes) 000845 1787144170.365094 read 0x00d679 thru 0x0197a7 (0xc12f bytes) 000846 1787144170.407280 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3864c): Mirror 10: 0x9c34dc53 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3864c): Mirror 10: 0x9c34dc53 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3864c: skipped 000847 1787144170.838054 mapwrite 0x00e90f thru 0x01b99a (0xd08c bytes) 000848 1787144170.900107 mapwrite 0x03849c thru 0x03d2bb (0x4e20 bytes) 000849 1787144170.969751 mapwrite 0x002a78 thru 0x00ef60 (0xc4e9 bytes) 000850 1787144171.042463 trunc from 0x03d2bc to 0x015e60 (0x2745b bytes) 000851 1787144171.070220 mapread 0x004204 thru 0x008fab (0x4da8 bytes) 000852 1787144171.071377 punch from 0x0015a8 to 0x006fe9 (0x5a42 bytes) 000853 1787144171.081554 fallocate 0x03db4d thru 0x03ffff (0x24b3 bytes) 000854 1787144171.089427 punch from 0x0114d6 to 0x015e60 (0x498b bytes) 000855 1787144171.101839 trunc from 0x005cc9 to 0x014951 (0xec89 bytes) 000856 1787144171.111707 trunc from 0x015e61 to 0x014aa9 (0x13b7 bytes) 000857 1787144171.159024 write 0x019b31 thru 0x027ed5 (0xe3a5 bytes) 000858 1787144171.167158 punch from 0x0228d2 to 0x02795a (0x5089 bytes) 000859 1787144171.190856 read 0x021d4c thru 0x027ed5 (0x618a bytes) 000860 1787144171.210703 trunc from 0x00ce7e to 0x01aa7a (0xdbfd bytes) 000861 1787144171.222488 trunc from 0x008fc7 to 0x018711 (0xf74b bytes) 000862 1787144171.227677 punch from 0x00a202 to 0x00c010 (0x1e0f bytes) 000863 1787144171.232660 read 0x026294 thru 0x027ed5 (0x1c42 bytes) 000864 1787144171.233591 trunc from 0x00a1f6 to 0x00a68f (0x49a bytes) 000865 1787144171.238887 fallocate 0x017e7a thru 0x01bad2 (0x3c59 bytes) 000866 1787144171.247022 punch from 0x02518a to 0x027ed5 (0x2d4c bytes) 000867 1787144171.254348 mapwrite 0x012991 thru 0x014c4c (0x22bc bytes) 000868 1787144171.277668 trunc from 0x025e10 to 0x027ed5 (0x20c6 bytes) 000869 1787144171.282860 mapread 0x00e15a thru 0x00feb5 (0x1d5c bytes) 000870 1787144171.284320 mapwrite 0x02afd2 thru 0x02ecfd (0x3d2c bytes) 000871 1787144171.329725 mapwrite 0x03659c thru 0x03a215 (0x3c7a bytes) 000872 1787144171.379865 mapread 0x02e7e0 thru 0x031083 (0x28a4 bytes) 000873 1787144171.381164 punch from 0x039d22 to 0x03a215 (0x4f4 bytes) 000875 1787144171.400682 punch from 0x000014 to 0x007b9e (0x7b8b bytes) 000876 1787144171.407696 mapread 0x02dddf thru 0x030112 (0x2334 bytes) 000877 1787144171.409654 write 0x0018f1 thru 0x004250 (0x2960 bytes) 000878 1787144171.431588 write 0x02268d thru 0x02fc1f (0xd593 bytes) 000879 1787144171.436308 punch from 0x034021 to 0x036867 (0x2847 bytes) 000880 1787144171.444273 trunc from 0x00c06c to 0x01bf65 (0xfefa bytes) 000881 1787144171.458986 trunc from 0x03a216 to 0x012e06 (0x2740f bytes) 000882 1787144171.490284 read 0x00c957 thru 0x00cd5b (0x405 bytes) 000883 1787144171.503636 punch from 0x007b55 to 0x00e69b (0x6b47 bytes) 000885 1787144171.515593 trunc from 0x012e07 to 0x0073cc (0xba3a bytes) 000886 1787144171.536247 write 0x006ce1 thru 0x012461 (0xb781 bytes) 000887 1787144171.538624 trunc from 0x001690 to 0x006200 (0x4b71 bytes) 000888 1787144171.564174 read 0x000cf0 thru 0x007e60 (0x7171 bytes) 000889 1787144171.583893 read 0x00fb12 thru 0x012461 (0x2950 bytes) 000890 1787144171.585213 read 0x012372 thru 0x012461 (0xf0 bytes) 000891 1787144171.586235 mapwrite 0x03466e thru 0x038f4d (0x48e0 bytes) 000892 1787144171.671438 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38f4e): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f4e): Mirror 10: 0x7396429f (stale) CRC-32 checksum value for chunk [0, 0x38f4e): Mirror 1: 0x71825a40 Mirror 11: 0x71825a40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f4e: skipped 000893 1787144172.909746 write 0x0099a5 thru 0x018b19 (0xf175 bytes) 000894 1787144172.976889 mapwrite 0x018185 thru 0x01a4c4 (0x2340 bytes) 000896 1787144173.009720 write 0x026a2e thru 0x029762 (0x2d35 bytes) 000897 1787144173.019473 read 0x033035 thru 0x034cd7 (0x1ca3 bytes) 000898 1787144173.025241 trunc from 0x010e0f to 0x01196d (0xb5f bytes) 000899 1787144173.030317 mapread 0x027e40 thru 0x030a8a (0x8c4b bytes) 000900 1787144173.040018 mapwrite 0x020010 thru 0x0278b2 (0x78a3 bytes) 000901 1787144173.064773 mapwrite 0x000540 thru 0x004c14 (0x46d5 bytes) 000902 1787144173.094309 trunc from 0x038e3a to 0x038f4d (0x114 bytes) 000903 1787144173.104892 write 0x02362f thru 0x027d92 (0x4764 bytes) 000904 1787144173.107444 trunc from 0x038f4e to 0x000c4b (0x38302 bytes) 000905 1787144173.130681 trunc from 0x0001a6 to 0x000c4b (0xaa6 bytes) 000906 1787144173.137210 write 0x00b7b7 thru 0x01ac26 (0xf470 bytes) 000907 1787144173.144535 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1ac27): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ac27): Mirror 10: 0xed0c33df (stale) Mirror 11: 0x481ce4ea (stale) CRC-32 checksum value for chunk [0, 0x1ac27): Mirror 1: 0xc79772c9 Mirror 12: 0xc79772c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ac27: skipped 000908 1787144174.035807 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ac27): Mirror 11: 0x481ce4ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ac27): Mirror 11: 0x481ce4ea (stale) CRC-32 checksum value for chunk [0, 0x1ac27): Mirror 1: 0xc79772c9 Mirror 12: 0xc79772c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ac27: skipped 000909 1787144174.833247 read 0x01a606 thru 0x01ac26 (0x621 bytes) 000910 1787144174.873353 mapwrite 0x028626 thru 0x034a0d (0xc3e8 bytes) 000911 1787144175.040651 fallocate 0x01a865 thru 0x024c75 (0xa411 bytes) 000912 1787144175.048750 write 0x03cff9 thru 0x03ffff (0x3007 bytes) 000913 1787144175.050970 mapread 0x022f94 thru 0x028c32 (0x5c9f bytes) 000914 1787144175.062413 write 0x02972e thru 0x03688d (0xd160 bytes) 000915 1787144175.068786 write 0x03661f thru 0x038bc2 (0x25a4 bytes) 000916 1787144175.074925 punch from 0x03f10b to 0x03ffff (0xef5 bytes) 000918 1787144175.094272 mapwrite 0x036d06 thru 0x03ffff (0x92fa bytes) 000919 1787144175.123326 read 0x014ef4 thru 0x01548c (0x599 bytes) 000920 1787144175.136782 fallocate 0x01e860 thru 0x01f0e3 (0x884 bytes) 000921 1787144175.142836 write 0x039708 thru 0x03ffff (0x68f8 bytes) 000922 1787144175.147565 mapread 0x033cd4 thru 0x037ddc (0x4109 bytes) 000925 1787144175.153004 fallocate 0x00f1d8 thru 0x0191b1 (0x9fda bytes) 000926 1787144175.163210 trunc from 0x040000 to 0x03fa93 (0x56c bytes) 000927 1787144175.199258 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fa94): Mirror 11: 0xac0c90eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa94): Mirror 11: 0xac0c90eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa94: skipped 000928 1787144175.512808 trunc from 0x03fa94 to 0x00de81 (0x31c12 bytes) 000929 1787144175.535748 trunc from 0x00de82 to 0x01e2a9 (0x10428 bytes) 000930 1787144175.553280 mapread 0x008c72 thru 0x0175e6 (0xe975 bytes) 000931 1787144175.587965 mapwrite 0x0320c5 thru 0x03f678 (0xd5b4 bytes) 000932 1787144175.668595 punch from 0x03f3f5 to 0x03f678 (0x284 bytes) 000933 1787144175.678299 write 0x020035 thru 0x024014 (0x3fe0 bytes) 000934 1787144175.708704 mapread 0x022f53 thru 0x02ac45 (0x7cf3 bytes) 000935 1787144175.730034 write 0x01e31b thru 0x02db9f (0xf885 bytes) 000936 1787144175.736217 fallocate 0x02491d thru 0x031560 (0xcc44 bytes) 000937 1787144175.756694 fallocate 0x01f3d7 thru 0x020f5e (0x1b88 bytes) 000938 1787144175.781032 punch from 0x037420 to 0x03f678 (0x8259 bytes) 000940 1787144175.793192 mapwrite 0x01cc61 thru 0x01d440 (0x7e0 bytes) 000941 1787144175.860236 mapwrite 0x011287 thru 0x01dc78 (0xc9f2 bytes) 000944 1787144175.901285 write 0x02db6e thru 0x02e097 (0x52a bytes) 000947 1787144175.909862 read 0x011f56 thru 0x01dec9 (0xbf74 bytes) 000948 1787144175.919181 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000949 1787144176.108220 read 0x00e587 thru 0x019315 (0xad8f bytes) 000950 1787144176.129674 write 0x0354bb thru 0x03ffff (0xab45 bytes) 000951 1787144176.145346 mapread 0x034c7a thru 0x03cf61 (0x82e8 bytes) 000952 1787144176.161931 fallocate 0x0198d2 thru 0x0222b6 (0x89e5 bytes) 000953 1787144176.167649 fallocate 0x030762 thru 0x03e225 (0xdac4 bytes) 000954 1787144176.176576 write 0x03ce91 thru 0x03ffff (0x316f bytes) 000955 1787144176.178928 trunc from 0x040000 to 0x01a4ce (0x25b31 bytes) 000956 1787144176.213333 read 0x0037d6 thru 0x00db4d (0xa378 bytes) 000957 1787144176.240496 mapwrite 0x0205de thru 0x024a52 (0x4475 bytes) 000958 1787144176.362001 punch from 0x00f9c9 to 0x01db2e (0xe166 bytes) 000960 1787144176.370789 mapwrite 0x03417c thru 0x034626 (0x4ab bytes) 000961 1787144176.472409 punch from 0x01edc2 to 0x027749 (0x8988 bytes) 000962 1787144176.485563 MIRROR_EXTEND to 2 mirrors 000963 1787144177.143886 trunc from 0x034627 to 0x006cbc (0x2d96a bytes) 000964 1787144177.262714 mapwrite 0x03d8e2 thru 0x03ffff (0x271e bytes) 000965 1787144177.315212 punch from 0x016290 to 0x023229 (0xcf9a bytes) 000966 1787144177.329528 read 0x02c491 thru 0x02dd09 (0x1879 bytes) 000967 1787144177.340252 fallocate 0x035454 thru 0x037dc7 (0x2974 bytes) 000968 1787144177.345768 fallocate 0x02faf6 thru 0x03c00b (0xc516 bytes) 000969 1787144177.351813 read 0x03d5fe thru 0x03ffff (0x2a02 bytes) 000970 1787144177.352427 read 0x03eb01 thru 0x03ec71 (0x171 bytes) 000971 1787144177.352980 trunc from 0x040000 to 0x020116 (0x1fee9 bytes) 000972 1787144177.372802 fallocate 0x015d34 thru 0x01ab50 (0x4e1d bytes) 000973 1787144177.378595 mapwrite 0x022db4 thru 0x0254a6 (0x26f3 bytes) 000975 1787144177.417247 mapwrite 0x00e9d0 thru 0x01c24b (0xd87c bytes) 000976 1787144177.449806 read 0x011758 thru 0x01b0dd (0x9986 bytes) 000977 1787144177.451845 trunc from 0x00d09b to 0x018f5b (0xbec1 bytes) 000978 1787144177.459910 trunc from 0x0254a7 to 0x0351c1 (0xfd1b bytes) 000979 1787144177.489193 read 0x00eb44 thru 0x018ca5 (0xa162 bytes) 000980 1787144177.503707 punch from 0x014ab6 to 0x01c8b4 (0x7dff bytes) 000981 1787144177.509338 fallocate 0x014c14 thru 0x0178bd (0x2caa bytes) 000982 1787144177.514803 trunc from 0x014258 to 0x01ba80 (0x7829 bytes) 000983 1787144177.519605 mapwrite 0x020d6f thru 0x024c7f (0x3f11 bytes) 000986 1787144177.540007 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000987 1787144177.715444 fallocate 0x01d99f thru 0x02194e (0x3fb0 bytes) 000988 1787144177.722486 mapread 0x00c4b1 thru 0x01c427 (0xff77 bytes) 000989 1787144177.761048 punch from 0x013dc6 to 0x01ee72 (0xb0ad bytes) 000990 1787144177.766615 mapwrite 0x032885 thru 0x0366c9 (0x3e45 bytes) 000991 1787144177.825600 read 0x003c8f thru 0x008fdd (0x534f bytes) 000993 1787144177.834574 read 0x02845e thru 0x02d89a (0x543d bytes) 000994 1787144177.844599 fallocate 0x03e3b4 thru 0x03ffff (0x1c4c bytes) 000995 1787144177.849531 read 0x017c94 thru 0x01c406 (0x4773 bytes) 000996 1787144177.850560 punch from 0x00e16a to 0x00f1e1 (0x1078 bytes) 000997 1787144177.855538 mapwrite 0x00a985 thru 0x0195f8 (0xec74 bytes) 000998 1787144177.877664 write 0x01d12d thru 0x02c70a (0xf5de bytes) 000999 1787144177.879365 trunc from 0x0366ca to 0x028055 (0xe674 bytes) 001000 1787144177.904104 mapread 0x017531 thru 0x023538 (0xc008 bytes) All operations completed A-OK! PASS 70a (90s) == sanity-flr test 71: check mirror extend parent fid ==== 08:56:23 (1787144183) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x92:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 08:56:33 (1787144193) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.284623 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098643 s, 10.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 08:56:43 (1787144203) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre2 Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4118368 .. Extending file size to 4055872 .. Extending file size to 5040864 .. Extending file size to 6513728 .. Extending file size to 6363072 .. Extending file size to 6778976 .. Extending file size to 8517600 .. Extending file size to 7700640 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8669280 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8863104 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9180256 .. Waiting 61264 61265 61267 61269 61270 done 192.168.204.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre2 (opts:) 192.168.204.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (90s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:58:13 (1787144293) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre2 Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9071712 .. Extending file size to 6624960 .. Extending file size to 8119424 .. Extending file size to 8481856 .. Extending file size to 9184416 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9173888 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62869 62870 62872 62873 62877 done 192.168.204.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre2 (opts:) 192.168.204.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:59:34 (1787144374) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1269240 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25788 3581232 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17900 3589120 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43688 7170352 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2064 1269240 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25788 3581232 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17900 3589120 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43688 7170352 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113665 s, 18.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195643 s, 10.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6c33aa1e (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:59:54 (1787144394) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.183093 s, 11.5 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 09:00:02 (1787144402) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa1:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:00:10 (1787144410) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.179103 s, 11.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:00:20 (1787144420) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107616 s, 19.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:00:28 (1787144428) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142845 s, 14.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e7:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:01:02 (1787144462) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg401-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126613 s, 8.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e5:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:01:57 (1787144517) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg401-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105512 s, 19.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:02:51 (1787144571) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg401-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg401-server: Pool lustre.test_204f created Adding targets to pool oleg401-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg401-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168171 s, 12.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg401-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg401-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg401-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:04:11 (1787144651) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:04:19 (1787144659) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:04:27 (1787144667) oleg401-server: Pool lustre.test_206 created Adding targets to pool oleg401-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg401-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg401-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg401-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg401-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:04:56 (1787144696) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100009 s, 10.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374085 s, 28.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xbd1d2ba9 Mirror 3: 0xbd1d2ba9 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:05:09 (1787144709) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:05:14 (1787144714) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:05:18 (1787144718) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0531258 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1020000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1480000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1480000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1480000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1950000 Read this (4 bytes): BBBB 192.168.204.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:05:59 (1787144759) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00823269 s, 995 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:06:06 (1787144766) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423053 s, 24.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc0 lmm_fid: [0x200000405:0xc0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc0 lmm_fid: [0x200000405:0xc0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611328 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-19 09:06:08.000000000 -0400 Modify: 2026-08-19 09:06:08.000000000 -0400 Change: 2026-08-19 09:06:09.000000000 -0400 Birth: 2026-08-19 09:06:07.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:06:15 (1787144775) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037759 s, 27.8 MB/s oleg401-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg401-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg401-server: O/280000401/d29/15d: File not found by ext2_lookup PASS 210b (26s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:06:40 (1787144800) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.231743 s, 177 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0249073 s, 164 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 19 09:06 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xc5:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 19 09:06 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:06:50 (1787144810) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (11s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:07:01 (1787144821) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1889 sec =========== 09:07:12 (1787144832) === sanity-flr: start cleanup 09:07:14 (1787144834) === === sanity-flr: finish cleanup 09:07:23 (1787144843) ===