-----============= acceptance-small: sanity-flr ============----- Sun Jul 26 10:43:11 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg336-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 10:43:28 (1785077008) === oleg336-client.virtnet: executing check_config_client /mnt/lustre oleg336-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg336-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c2e11bd5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c2e11bd5800.idle_timeout=debug disable quota as required oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:44:02 (1785077042) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8238] == sanity-flr test 0a: lfs mirror create with -N option == 10:44:04 (1785077044) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:44:20 (1785077060) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:44:23 (1785077063) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:44:25 (1785077065) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:44:48 (1785077088) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:44:51 (1785077091) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 10:44:55 (1785077095) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c2e11bd5800.stats=clear osc.lustre-OST0001-osc-ffff9c2e11bd5800.stats=clear PASS 0g (48s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:45:43 (1785077143) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:45:55 (1785077155) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:46:03 (1785077163) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:46:06 (1785077166) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:46:15 (1785077175) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 10:46:24 (1785077184) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:46:32 (1785077192) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:46:39 (1785077199) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:46:47 (1785077207) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:46:55 (1785077215) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:47:02 (1785077222) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 1.61246 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117598 s, 8.9 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:47:12 (1785077232) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0889098 s, 35.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:47:20 (1785077240) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:47:31 (1785077251) Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2e11bd5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2e11bd5800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg336-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg336-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg336-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg336-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg336-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg336-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg336-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg336-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg336-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg336-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg336-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg336-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg336-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg336-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg336-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg336-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg336-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg336-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg336-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg336-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg336-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg336-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg336-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c2e11bd5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2e11bd5800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg336-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (132s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:49:43 (1785077383) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.538785 s, 19.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2e11bd5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2e11bd5800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (39s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:50:22 (1785077422) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2e11bd5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2e11bd5800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c2e11bd5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2e11bd5800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg336-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (76s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:51:38 (1785077498) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.05095 s, 17.2 MB/s -rw-r--r-- 1 root root 52428800 Jul 26 10:51 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.960s user 0m0.001s sys 0m0.174s Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2e11bd5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2e11bd5800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.057s user 0m0.001s sys 0m0.170s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c2e11bd5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2e11bd5800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.373s user 0m0.002s sys 0m0.173s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg336-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c2e11bd5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2e11bd5800.ost_server_uuid in FULL state after 0 sec PASS 33b (91s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:53:09 (1785077589) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:53:13 (1785077593) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:53:16 (1785077596) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 10:53:20 (1785077600) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 10:53:26 (1785077606) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.619506 s, 20.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:53:39 (1785077619) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376128 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040306 s, 26.0 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:53:46 (1785077626) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:53:54 (1785077634) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00317538 s, 322 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113795 s, 900 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00137047 s, 747 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00279481 s, 366 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 10:54:03 (1785077643) create test files with size 15755813 .. aa96ae39a26aff731b0c5ace7f7264d8 - 948093f076b9a46b36f3863225178c1f - ffb6e723c3efa8318febc4a3c74850b0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg336-client.virtnet /mnt/lustre (opts:) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 10:54:24 (1785077664) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:54:37 (1785077677) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:54:44 (1785077684) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116342 s, 9.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348918 s, 30.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 10:54:52 (1785077692) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.080014 s, 26.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.392212 s, 10.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.241822 s, 8.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 10:55:08 (1785077708) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 10:55:12 (1785077712) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:55:15 (1785077715) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305541 s, 34.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 10:55:25 (1785077725) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.296083 s, 10.6 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:55:36 (1785077736) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xca:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:55:46 (1785077746) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.342696 s, 30.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg336-client.virtnet /mnt/lustre (opts:) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:55:58 (1785077758) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9c2e0d005000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.372425 s, 28.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg336-client.virtnet /mnt/lustre (opts:) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9c2e1885d800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:56:14 (1785077774) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:56:52 (1785077812) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:57:25 (1785077845) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 10:57:28 (1785077848) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.278038 s, 11.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.183989 s, 17.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd7809ba5 (nosync,stale) Mirror 2: 0xd7809ba5 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 253609b57073fdae0b603bf380253ac3 - mirror 1 checksum: 26f235b2f585351d4db493140d961376 - mirror 2 checksum: 253609b57073fdae0b603bf380253ac3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-26T10:57:32-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-26T10:57:32-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 253609b57073fdae0b603bf380253ac3 - mirror 1 checksum: 26f235b2f585351d4db493140d961376 - mirror 2 checksum: 26f235b2f585351d4db493140d961376 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-26T10:57:32-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-26T10:57:32-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 10:57:38 (1785077858) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281872 s, 37.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:57:46 (1785077866) Failing mds1 on oleg336-server Stopping /mnt/lustre-mds1 (opts:) on oleg336-server 10:58:00 (1785077880) shut down facet: mds1 facet_host: oleg336-server facet_failover_host: oleg336-server Failover mds1 to oleg336-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0000 10:58:34 (1785077914) targets are mounted 10:58:34 (1785077914) facet_failover done oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443443 s, 23.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd8:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:58:50 (1785077930) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0509714 s, 402 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0187852 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:59:01 (1785077941) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0506176 s, 405 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0248237 s, 825 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:59:10 (1785077950) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00828785 s, 2.0 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:59:16 (1785077956) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.040284 s, 508 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:59:23 (1785077963) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0117843 s, 348 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0489586 s, 83.7 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:59:30 (1785077970) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:59:54 (1785077994) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:00:22 (1785078022) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:00:45 (1785078045) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:01:08 (1785078068) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 11:01:15 (1785078075) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57589 mirror split pid 57591 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 57589 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 57591 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 11:02:24 (1785078144) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 8146 000001 1785078146.516956 mapwrite 0x03a74f thru 0x03ffff (0x58b1 bytes) 000004 1785078146.647850 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3d5cf 000005 1785078147.280825 trunc from 0x040000 to 0x03d5ce (0x2a31 bytes) 000006 1785078147.419060 write 0x030b81 thru 0x030ff2 (0x472 bytes) 000007 1785078147.439461 trunc from 0x03d5cf to 0x03cf6e (0x660 bytes) 000008 1785078147.486409 write 0x001464 thru 0x00710d (0x5caa bytes) 000009 1785078147.512861 trunc from 0x03cf6f to 0x000664 (0x3c90a bytes) 000010 1785078147.553618 mapread 0x000507 thru 0x000664 (0x15e bytes) 000011 1785078147.571434 mapwrite 0x017891 thru 0x026491 (0xec01 bytes) 000012 1785078147.681889 trunc from 0x026492 to 0x023124 (0x336d bytes) 000013 1785078147.721459 MIRROR_RESYNC 2 mirrors 000014 1785078148.645331 trunc from 0x023125 to 0x00a164 (0x18fc0 bytes) 000016 1785078148.730856 punch from 0x008649 to 0x00a164 (0x1b1c bytes) 000017 1785078148.741345 fallocate 0x00ac20 thru 0x00c63d (0x1a1e bytes) 000019 1785078148.750300 mapwrite 0x01795c thru 0x01da27 (0x60cc bytes) 000020 1785078148.889830 read 0x01b886 thru 0x01da27 (0x21a2 bytes) 000022 1785078148.903477 trunc from 0x01da28 to 0x022c63 (0x523c bytes) 000023 1785078148.932788 trunc from 0x015e3b to 0x015e4f (0x15 bytes) 000024 1785078148.942773 trunc from 0x022c64 to 0x01d242 (0x5a21 bytes) 000025 1785078148.978987 fallocate 0x00fd19 thru 0x0131d2 (0x34ba bytes) 000026 1785078148.993939 trunc from 0x007f11 to 0x0159e6 (0xdad6 bytes) 000027 1785078149.000385 punch from 0x00d4bb to 0x017ea5 (0xa9eb bytes) 000028 1785078149.018383 punch from 0x0009e5 to 0x003767 (0x2d83 bytes) 000029 1785078149.026901 read 0x0140fa thru 0x01d242 (0x9149 bytes) 000031 1785078149.066542 punch from 0x01c3d6 to 0x01d242 (0xe6d bytes) 000032 1785078149.074402 trunc from 0x01d243 to 0x026de4 (0x9ba2 bytes) 000033 1785078149.112506 mapread 0x002253 thru 0x00a769 (0x8517 bytes) 000034 1785078149.113724 write 0x0071e9 thru 0x01327d (0xc095 bytes) 000035 1785078149.132213 mapwrite 0x03b3e9 thru 0x03ffff (0x4c17 bytes) 000036 1785078149.199593 fallocate 0x029364 thru 0x02f7a1 (0x643e bytes) 000037 1785078149.214049 mapread 0x02ab64 thru 0x02c922 (0x1dbf bytes) 000038 1785078149.221458 write 0x02982f thru 0x032a0f (0x91e1 bytes) zero_range to largest ever: 0x40000 000040 1785078149.224202 trunc from 0x03e9f9 to 0x03ffff (0x1607 bytes) 000041 1785078149.239429 fallocate 0x03cd41 thru 0x03cd61 (0x21 bytes) 000042 1785078149.244897 read 0x036f0f thru 0x03ffff (0x90f1 bytes) 000043 1785078149.262016 punch from 0x01798a to 0x01b51f (0x3b96 bytes) 000044 1785078149.270342 punch from 0x015569 to 0x01f507 (0x9f9f bytes) 000045 1785078149.278961 write 0x01440c thru 0x014a03 (0x5f8 bytes) 000046 1785078149.280834 mapread 0x036dfb thru 0x03e2d8 (0x74de bytes) 000047 1785078149.285118 mapread 0x0194cf thru 0x01c085 (0x2bb7 bytes) 000048 1785078149.315305 fallocate 0x02b298 thru 0x02beec (0xc55 bytes) 000049 1785078149.341549 read 0x03766a thru 0x03b497 (0x3e2e bytes) 000050 1785078149.342331 punch from 0x028f55 to 0x032267 (0x9313 bytes) 000051 1785078149.363669 fallocate 0x02f8e8 thru 0x035e7e (0x6597 bytes) 000052 1785078149.379440 read 0x035b7e thru 0x037176 (0x15f9 bytes) 000054 1785078149.382848 read 0x030142 thru 0x034496 (0x4355 bytes) 000055 1785078149.396804 punch from 0x00e599 to 0x011177 (0x2bdf bytes) 000056 1785078149.427440 MIRROR_RESYNC 2 mirrors 000057 1785078150.166788 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000058 1785078150.323979 trunc from 0x014f6c to 0x0182fd (0x3392 bytes) 000059 1785078150.343661 trunc from 0x0021ef to 0x00bf21 (0x9d33 bytes) 000060 1785078150.348935 read 0x0389f2 thru 0x03ffff (0x760e bytes) 000061 1785078150.358873 mapwrite 0x01acee thru 0x01dd3b (0x304e bytes) 000064 1785078150.402484 trunc from 0x03dfc2 to 0x03ffff (0x203e bytes) 000065 1785078150.409810 fallocate 0x0234de thru 0x027184 (0x3ca7 bytes) 000066 1785078150.426540 trunc from 0x040000 to 0x025a1a (0x1a5e5 bytes) 000067 1785078150.496503 trunc from 0x01b03c to 0x025a1a (0xa9df bytes) 000068 1785078150.501895 mapread 0x00d715 thru 0x0165d1 (0x8ebd bytes) 000069 1785078150.525546 write 0x012746 thru 0x013077 (0x932 bytes) 000070 1785078150.526660 mapread 0x023422 thru 0x025a1a (0x25f9 bytes) 000071 1785078150.535758 mapwrite 0x000e90 thru 0x002a11 (0x1b82 bytes) 000074 1785078150.571123 mapread 0x015e03 thru 0x015e9a (0x98 bytes) 000075 1785078150.574461 fallocate 0x01657c thru 0x023253 (0xccd8 bytes) 000076 1785078150.587522 fallocate 0x03d42f thru 0x03f984 (0x2556 bytes) 000077 1785078150.597383 write 0x0292a2 thru 0x02e1cc (0x4f2b bytes) 000078 1785078150.598986 write 0x02b263 thru 0x036f6f (0xbd0d bytes) 000079 1785078150.600575 mapread 0x005456 thru 0x013b00 (0xe6ab bytes) 000080 1785078150.611572 MIRROR_EXTEND to 2 mirrors 000082 1785078151.381787 punch from 0x0296b2 to 0x036f6f (0xd8be bytes) 000083 1785078151.444529 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000084 1785078151.597475 punch from 0x015227 to 0x0206a0 (0xb47a bytes) 000086 1785078151.613011 mapread 0x0270fa thru 0x02dc61 (0x6b68 bytes) 000087 1785078151.632664 punch from 0x02cd82 to 0x02e557 (0x17d6 bytes) 000089 1785078151.641834 fallocate 0x03c425 thru 0x03ffff (0x3bdb bytes) 000090 1785078151.648672 trunc from 0x036f70 to 0x016663 (0x2090c bytes) 000091 1785078151.673584 trunc from 0x0111eb to 0x016663 (0x5479 bytes) 000092 1785078151.679212 fallocate 0x016fec thru 0x01d2cb (0x62e0 bytes) 000093 1785078151.683877 punch from 0x00b740 to 0x00b904 (0x1c5 bytes) 000094 1785078151.689244 fallocate 0x01d33f thru 0x02cbee (0xf8b0 bytes) 000096 1785078151.695335 write 0x018c60 thru 0x01f92a (0x6ccb bytes) 000097 1785078151.703732 mapwrite 0x01a6a3 thru 0x02027e (0x5bdc bytes) 000098 1785078151.774104 mapread 0x00c38c thru 0x00f953 (0x35c8 bytes) 000099 1785078151.794066 read 0x01c2a7 thru 0x02027e (0x3fd8 bytes) 000100 1785078151.803436 MIRROR_EXTEND to 2 mirrors 000101 1785078152.549675 mapwrite 0x017352 thru 0x022a7d (0xb72c bytes) 000102 1785078152.767086 MIRROR_RESYNC 2 mirrors 000103 1785078153.747720 MIRROR_RESYNC 2 mirrors 000104 1785078154.134326 mapwrite 0x00d3a7 thru 0x00f59e (0x21f8 bytes) 000105 1785078154.307548 MIRROR_RESYNC 2 mirrors 000106 1785078155.340561 MIRROR_EXTEND to 3 mirrors 000107 1785078156.313619 mapwrite 0x03fe76 thru 0x03ffff (0x18a bytes) 000108 1785078156.492786 write 0x01577c thru 0x0225e7 (0xce6c bytes) 000109 1785078156.506723 punch from 0x027043 to 0x0299aa (0x2968 bytes) 000110 1785078156.520996 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5108cfe5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5108cfe5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000112 1785078157.117966 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000113 1785078157.344313 read 0x0262eb thru 0x029542 (0x3258 bytes) 000114 1785078157.358576 punch from 0x03bdc8 to 0x03ffff (0x4238 bytes) 000115 1785078157.365648 punch from 0x02843f to 0x035b77 (0xd739 bytes) 000116 1785078157.373702 mapwrite 0x0243d7 thru 0x02eb47 (0xa771 bytes) 000118 1785078157.470701 mapread 0x005d1e thru 0x01322a (0xd50d bytes) 000119 1785078157.541232 read 0x008dfb thru 0x013423 (0xa629 bytes) 000121 1785078157.569583 MIRROR_EXTEND to 2 mirrors 000122 1785078159.020597 trunc from 0x040000 to 0x03e83a (0x17c5 bytes) 000123 1785078159.101650 punch from 0x028c5a to 0x036be9 (0xdf90 bytes) 000124 1785078159.108970 fallocate 0x017a52 thru 0x02334f (0xb8fe bytes) 000125 1785078159.116787 trunc from 0x03e83b to 0x02c624 (0x12216 bytes) 000126 1785078159.141841 read 0x015108 thru 0x021f2f (0xce28 bytes) 000127 1785078159.167740 write 0x03e9ce thru 0x03ffff (0x1632 bytes) 000128 1785078159.172612 punch from 0x021515 to 0x030664 (0xf150 bytes) 000129 1785078159.178328 fallocate 0x009e39 thru 0x0133b6 (0x957e bytes) 000130 1785078159.184710 mapread 0x03d27a thru 0x03ffff (0x2d86 bytes) 000131 1785078159.190644 write 0x0034ab thru 0x00f151 (0xbca7 bytes) 000132 1785078159.214986 punch from 0x0012db to 0x00746d (0x6193 bytes) 000133 1785078159.253751 trunc from 0x040000 to 0x01bd91 (0x2426e bytes) 000134 1785078159.276657 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1bd92): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bd92): Mirror 2: 0xa49109c4 (stale) CRC-32 checksum value for chunk [0, 0x1bd92): Mirror 1: 0x7588bc21 Mirror 3: 0x7588bc21 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bd92: skipped 000135 1785078160.417730 MIRROR_RESYNC 3 mirrors 000136 1785078161.403872 mapwrite 0x02365e thru 0x02f922 (0xc2c5 bytes) 000137 1785078161.631784 mapwrite 0x01800e thru 0x01e9b6 (0x69a9 bytes) 000138 1785078161.710894 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f923): Mirror 3: 0x9a32a055 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f923): Mirror 3: 0x9a32a055 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f923: skipped 000139 1785078162.086894 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000140 1785078162.229957 trunc from 0x02f923 to 0x018169 (0x177b9 bytes) 000141 1785078162.269142 write 0x02b7f9 thru 0x039f6c (0xe774 bytes) 000142 1785078162.279890 read 0x0185ec thru 0x01d398 (0x4dad bytes) 000143 1785078162.301718 punch from 0x020083 to 0x02740c (0x738a bytes) 000144 1785078162.309086 mapwrite 0x028ef5 thru 0x02de4b (0x4f57 bytes) 000146 1785078162.385797 read 0x03264f thru 0x039f6c (0x791e bytes) 000147 1785078162.399172 trunc from 0x039f6d to 0x01366c (0x26900 bytes) 000148 1785078162.421311 read 0x00027d thru 0x009765 (0x94e9 bytes) 000149 1785078162.440025 trunc from 0x01366d to 0x009697 (0x9fd5 bytes) 000150 1785078162.476576 trunc from 0x00474a to 0x009697 (0x4f4e bytes) 000151 1785078162.495356 fallocate 0x01a667 thru 0x023871 (0x920b bytes) 000152 1785078162.504706 mapread 0x0075ec thru 0x013b43 (0xc558 bytes) 000153 1785078162.638474 mapwrite 0x027021 thru 0x02ef6d (0x7f4d bytes) 000154 1785078162.729595 fallocate 0x03f059 thru 0x03ffff (0xfa7 bytes) 000155 1785078162.753773 write 0x01fce0 thru 0x022351 (0x2672 bytes) 000159 1785078162.766560 trunc from 0x02ef6e to 0x02e424 (0xb49 bytes) 000161 1785078162.797940 read 0x02288f thru 0x026f25 (0x4697 bytes) 000162 1785078162.798653 fallocate 0x004fa9 thru 0x00e838 (0x9890 bytes) 000163 1785078162.806759 mapwrite 0x01aa11 thru 0x026521 (0xbb11 bytes) 000164 1785078162.840229 mapwrite 0x025f28 thru 0x026e15 (0xeee bytes) 000165 1785078162.860570 fallocate 0x0300a7 thru 0x03e88f (0xe7e9 bytes) 000166 1785078162.870238 punch from 0x036195 to 0x03e88f (0x86fb bytes) 000167 1785078162.878821 trunc from 0x03e890 to 0x003a4e (0x3ae41 bytes) 000168 1785078162.912248 write 0x01fd72 thru 0x02de2a (0xe0b9 bytes) 000169 1785078162.916197 mapwrite 0x02958e thru 0x02d5a0 (0x4013 bytes) 000170 1785078162.936552 mapwrite 0x03aa6b thru 0x03ffff (0x5595 bytes) 000171 1785078162.982427 fallocate 0x01fa5c thru 0x02eab5 (0xf05a bytes) 000173 1785078162.989537 write 0x0175e8 thru 0x019c1c (0x2635 bytes) 000174 1785078162.991562 mapread 0x01a1f3 thru 0x024db2 (0xabc0 bytes) 000175 1785078162.993121 trunc from 0x040000 to 0x026be4 (0x1941b bytes) 000176 1785078163.018765 read 0x01efc1 thru 0x026be4 (0x7c24 bytes) 000177 1785078163.019913 write 0x00358a thru 0x007482 (0x3ef9 bytes) 000178 1785078163.022124 mapwrite 0x000826 thru 0x0082a6 (0x7a81 bytes) 000179 1785078163.046600 mapread 0x001a0d thru 0x010241 (0xe835 bytes) 000180 1785078163.051291 trunc from 0x026be5 to 0x00d4f1 (0x196f3 bytes) 000181 1785078163.086626 mapread 0x0096c7 thru 0x00d4f1 (0x3e2b bytes) 000182 1785078163.090929 write 0x0008b5 thru 0x0089b1 (0x80fd bytes) 000185 1785078163.093551 punch from 0x00814b to 0x00d4f1 (0x53a7 bytes) 000186 1785078163.141875 fallocate 0x031566 thru 0x03dbaa (0xc645 bytes) 000187 1785078163.152736 mapwrite 0x01ee72 thru 0x02ac44 (0xbdd3 bytes) 000188 1785078163.230027 punch from 0x007d34 to 0x017708 (0xf9d5 bytes) 000189 1785078163.242943 MIRROR_EXTEND to 2 mirrors 000190 1785078164.140923 trunc from 0x02ac45 to 0x00ee8f (0x1bdb5 bytes) 000191 1785078164.310515 mapwrite 0x033efd thru 0x036136 (0x223a bytes) 000192 1785078164.364011 mapwrite 0x022de0 thru 0x028671 (0x5892 bytes) 000193 1785078164.416863 mapwrite 0x01daa7 thru 0x01dde6 (0x340 bytes) 000195 1785078164.446237 punch from 0x020142 to 0x02e9f8 (0xe8b7 bytes) 000196 1785078164.463670 write 0x017f02 thru 0x024dec (0xceeb bytes) 000197 1785078164.499934 trunc from 0x01ea4d to 0x02e92c (0xfee0 bytes) 000198 1785078164.536591 trunc from 0x02b328 to 0x036136 (0xae0f bytes) 000199 1785078164.546836 mapwrite 0x00f975 thru 0x01022c (0x8b8 bytes) 000200 1785078164.594019 punch from 0x0108b3 to 0x01bdf3 (0xb541 bytes) 000201 1785078164.608434 trunc from 0x036137 to 0x015640 (0x20af6 bytes) 000202 1785078164.642733 read 0x00afec thru 0x00ef73 (0x3f88 bytes) 000203 1785078164.645091 trunc from 0x015641 to 0x030ebc (0x1b87c bytes) 000204 1785078164.668560 mapread 0x0068d4 thru 0x011aa0 (0xb1cd bytes) 000205 1785078164.699470 mapwrite 0x02f5b2 thru 0x03df6a (0xe9b9 bytes) 000207 1785078164.845503 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000208 1785078165.142246 trunc from 0x02de81 to 0x0353a2 (0x7522 bytes) 000209 1785078165.164738 MIRROR_EXTEND to 2 mirrors 000210 1785078166.531117 fallocate 0x026463 thru 0x02e8e9 (0x8487 bytes) 000211 1785078166.628997 mapread 0x01c2ac thru 0x01ed37 (0x2a8c bytes) 000212 1785078166.665629 mapread 0x00c107 thru 0x00dcc2 (0x1bbc bytes) 000213 1785078166.684294 mapread 0x012d13 thru 0x01856a (0x5858 bytes) 000214 1785078166.695562 trunc from 0x00b143 to 0x00b32e (0x1ec bytes) 000215 1785078166.715748 read 0x03c22e thru 0x03df6a (0x1d3d bytes) 000216 1785078166.723536 mapread 0x02d2d3 thru 0x036810 (0x953e bytes) 000217 1785078166.751554 trunc from 0x03df6b to 0x00fd22 (0x2e248 bytes) 000218 1785078166.780381 fallocate 0x02442a thru 0x024495 (0x6c bytes) 000220 1785078166.787922 trunc from 0x024496 to 0x0088e7 (0x1bbae bytes) 000221 1785078166.814696 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x88e8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x88e8): Mirror 2: 0x418a5621 (stale) CRC-32 checksum value for chunk [0, 0x88e8): Mirror 1: 0x418a5621 Mirror 3: 0x418a5621 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x88e8: skipped 000222 1785078168.139868 punch from 0x00352b to 0x0088e7 (0x53bd bytes) 000223 1785078168.186750 fallocate 0x0252cf thru 0x0313b5 (0xc0e7 bytes) 000224 1785078168.189851 mapwrite 0x00057e thru 0x000da4 (0x827 bytes) 000225 1785078168.203833 mapread 0x02af46 thru 0x0313b5 (0x6470 bytes) 000226 1785078168.238398 write 0x025bee thru 0x031174 (0xb587 bytes) 000227 1785078168.253936 read 0x00c5b4 thru 0x01a1c9 (0xdc16 bytes) 000228 1785078168.261065 trunc from 0x0313b6 to 0x005bbb (0x2b7fa bytes) 000229 1785078168.279998 punch from 0x004184 to 0x005bbb (0x1a38 bytes) 000230 1785078168.291634 read 0x001448 thru 0x005bbb (0x4774 bytes) 000231 1785078168.300515 trunc from 0x005bbc to 0x016fba (0x113ff bytes) 000232 1785078168.327338 punch from 0x014c40 to 0x016fba (0x237b bytes) 000233 1785078168.334086 mapwrite 0x02d023 thru 0x038639 (0xb617 bytes) 000234 1785078168.368636 mapread 0x035167 thru 0x037828 (0x26c2 bytes) 000235 1785078168.370386 trunc from 0x0102cb to 0x01517a (0x4eb0 bytes) 000236 1785078168.376009 trunc from 0x03863a to 0x01571d (0x22f1c bytes) 000237 1785078168.389771 punch from 0x010574 to 0x01571d (0x51aa bytes) 000238 1785078168.395543 write 0x02f227 thru 0x03cd87 (0xdb61 bytes) 000239 1785078168.398132 read 0x00902b thru 0x009c7b (0xc51 bytes) 000240 1785078168.399375 fallocate 0x030f7b thru 0x03f33f (0xe3c5 bytes) 000241 1785078168.415768 read 0x032f9a thru 0x03aeb2 (0x7f19 bytes) 000242 1785078168.418062 punch from 0x023121 to 0x025560 (0x2440 bytes) 000243 1785078168.427802 fallocate 0x03df8d thru 0x03ffff (0x2073 bytes) 000244 1785078168.438236 trunc from 0x040000 to 0x0303a8 (0xfc57 bytes) 000245 1785078168.467525 trunc from 0x0303a9 to 0x01bfff (0x143a9 bytes) 000247 1785078168.506603 mapwrite 0x01b056 thru 0x024715 (0x96c0 bytes) 000248 1785078168.573957 read 0x00e7ff thru 0x01351e (0x4d20 bytes) 000249 1785078168.575450 trunc from 0x013e6c to 0x0186d2 (0x4867 bytes) 000251 1785078168.592138 MIRROR_RESYNC 3 mirrors 000252 1785078169.925602 punch from 0x000b71 to 0x009dd5 (0x9265 bytes) 000254 1785078169.996953 trunc from 0x024716 to 0x00e791 (0x15f84 bytes) 000255 1785078170.069102 trunc from 0x00e792 to 0x01f4ec (0x10d5b bytes) 000256 1785078170.146152 trunc from 0x01efc6 to 0x01f4ec (0x527 bytes) 000258 1785078170.178890 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f4ed): Mirror 3: 0xb105f603 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f4ed): Mirror 3: 0xb105f603 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f4ed: skipped 000259 1785078170.709958 mapwrite 0x03fbf6 thru 0x03ffff (0x40a bytes) 000260 1785078170.807901 mapread 0x02faf7 thru 0x039927 (0x9e31 bytes) 000261 1785078170.892572 write 0x01cb97 thru 0x027786 (0xabf0 bytes) 000262 1785078170.923759 punch from 0x039474 to 0x03e504 (0x5091 bytes) 000264 1785078170.931139 mapwrite 0x015093 thru 0x020e65 (0xbdd3 bytes) 000265 1785078171.010237 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x67ad40f5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2ab19f7 Mirror 4: 0x2ab19f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000266 1785078172.364244 write 0x027701 thru 0x030188 (0x8a88 bytes) 000267 1785078172.427440 trunc from 0x007bd9 to 0x00a7c3 (0x2beb bytes) 000268 1785078172.432243 trunc from 0x032559 to 0x032f64 (0xa0c bytes) 000270 1785078172.437542 read 0x003c3a thru 0x00ae7a (0x7241 bytes) 000271 1785078172.446487 punch from 0x02211e to 0x0320c7 (0xffaa bytes) 000272 1785078172.470479 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x67ad40f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x67ad40f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000273 1785078173.067545 punch from 0x037c03 to 0x038d8f (0x118d bytes) 000274 1785078173.074405 mapwrite 0x00bafc thru 0x01770d (0xbc12 bytes) 000275 1785078173.181830 trunc from 0x012b5f to 0x01e02c (0xb4ce bytes) 000276 1785078173.196512 mapread 0x0182dc thru 0x020d8f (0x8ab4 bytes) 000277 1785078173.215109 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000279 1785078173.459609 read 0x01d74f thru 0x020551 (0x2e03 bytes) 000280 1785078173.479469 mapread 0x0369b2 thru 0x03fb41 (0x9190 bytes) 000281 1785078173.514500 MIRROR_EXTEND to 2 mirrors 000282 1785078174.899589 mapread 0x03baff thru 0x03ffff (0x4501 bytes) 000283 1785078174.938780 punch from 0x039dbe to 0x03ffff (0x6242 bytes) 000285 1785078175.026873 trunc from 0x0264b4 to 0x02fb52 (0x969f bytes) 000286 1785078175.063002 MIRROR_RESYNC 2 mirrors 000287 1785078176.425338 trunc from 0x040000 to 0x01bbf8 (0x24407 bytes) 000288 1785078176.537345 trunc from 0x00c071 to 0x00d5be (0x154e bytes) 000289 1785078176.553924 mapwrite 0x027a36 thru 0x033118 (0xb6e3 bytes) 000290 1785078176.733256 write 0x01daf4 thru 0x02cb2e (0xf03b bytes) 000291 1785078176.753928 trunc from 0x014caa to 0x01f9d5 (0xad2c bytes) 000292 1785078176.796355 write 0x002287 thru 0x00e1c7 (0xbf41 bytes) 000293 1785078176.827149 trunc from 0x033119 to 0x034f05 (0x1ded bytes) 000294 1785078176.890311 MIRROR_RESYNC 2 mirrors 000295 1785078178.216108 write 0x02eb26 thru 0x03e4b8 (0xf993 bytes) 000296 1785078178.333668 read 0x00d32c thru 0x01a869 (0xd53e bytes) 000297 1785078178.345476 mapwrite 0x03b225 thru 0x03ffff (0x4ddb bytes) 000298 1785078178.485647 mapwrite 0x02f79b thru 0x03f116 (0xf97c bytes) 000299 1785078178.566595 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x559ab34f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1f793ea4 Mirror 3: 0x1f793ea4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000300 1785078179.777802 trunc from 0x040000 to 0x02d230 (0x12dcf bytes) 000301 1785078179.895267 MIRROR_RESYNC 3 mirrors 000302 1785078181.030917 write 0x012e32 thru 0x01d04a (0xa219 bytes) 000303 1785078181.082787 trunc from 0x01ff6b to 0x027705 (0x779b bytes) 000304 1785078181.094291 fallocate 0x022298 thru 0x028fc8 (0x6d31 bytes) 000306 1785078181.106198 read 0x02a5e2 thru 0x02d230 (0x2c4f bytes) 000307 1785078181.131506 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d231): Mirror 3: 0x3daa7135 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d231): Mirror 3: 0x3daa7135 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d231: skipped 000308 1785078181.498780 mapwrite 0x02ee3a thru 0x03c7fd (0xd9c4 bytes) 000309 1785078181.658389 trunc from 0x03c7fe to 0x0255be (0x1723f bytes) 000310 1785078181.705651 mapread 0x00fc07 thru 0x00febe (0x2b8 bytes) 000311 1785078181.725126 mapwrite 0x00cdcc thru 0x019d9a (0xcfcf bytes) 000312 1785078181.775058 fallocate 0x009970 thru 0x009c1e (0x2af bytes) 000313 1785078181.789124 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000314 1785078181.969711 fallocate 0x023610 thru 0x0317a3 (0xe194 bytes) 000315 1785078181.976548 write 0x0395a0 thru 0x03ffff (0x6a60 bytes) 000316 1785078181.983510 punch from 0x0067ab to 0x0113fa (0xac50 bytes) 000317 1785078181.990308 punch from 0x01b144 to 0x02685b (0xb718 bytes) 000318 1785078181.997503 fallocate 0x00bc42 thru 0x00cf9a (0x1359 bytes) 000319 1785078182.009798 write 0x03312e thru 0x03f71e (0xc5f1 bytes) 000320 1785078182.022751 MIRROR_EXTEND to 2 mirrors 000321 1785078183.038287 MIRROR_EXTEND to 3 mirrors 000322 1785078184.523381 mapread 0x018aff thru 0x01c803 (0x3d05 bytes) 000323 1785078184.588561 write 0x03e264 thru 0x03ffff (0x1d9c bytes) 000324 1785078184.654001 mapwrite 0x028ecf thru 0x035005 (0xc137 bytes) 000326 1785078184.753644 mapread 0x03bc0d thru 0x03ffff (0x43f3 bytes) 000327 1785078184.764574 write 0x02aa4d thru 0x039b2e (0xf0e2 bytes) 000328 1785078184.767656 trunc from 0x040000 to 0x00962f (0x369d0 bytes) 000329 1785078184.790536 read 0x0078fb thru 0x009449 (0x1b4f bytes) 000331 1785078184.798561 read 0x003af6 thru 0x00962f (0x5b3a bytes) 000332 1785078184.806653 trunc from 0x009630 to 0x0256ac (0x1c07d bytes) 000333 1785078184.825548 trunc from 0x005c29 to 0x0109c7 (0xad9f bytes) 000334 1785078184.837956 punch from 0x005d6d to 0x009139 (0x33cd bytes) 000335 1785078184.848277 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x256ad): Mirror 3: 0x33a6a121 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x256ad): Mirror 3: 0x33a6a121 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x256ad: skipped 000336 1785078185.088096 write 0x008e5e thru 0x016525 (0xd6c8 bytes) 000337 1785078185.107091 fallocate 0x027ca2 thru 0x029fef (0x234e bytes) 000338 1785078185.113117 fallocate 0x0326bd thru 0x039b73 (0x74b7 bytes) 000339 1785078185.118636 mapread 0x0211cb thru 0x0252d9 (0x410f bytes) 000340 1785078185.141083 punch from 0x001acf to 0x009b4c (0x807e bytes) 000341 1785078185.159770 MIRROR_RESYNC 2 mirrors 000342 1785078185.897805 read 0x01b96b thru 0x01c05f (0x6f5 bytes) 000343 1785078185.914537 mapread 0x0259b6 thru 0x029fef (0x463a bytes) 000344 1785078185.965380 mapread 0x015702 thru 0x0246f8 (0xeff7 bytes) 000345 1785078185.998655 fallocate 0x02e69a thru 0x02fbdb (0x1542 bytes) 000347 1785078186.077483 read 0x0027ea thru 0x00e814 (0xc02b bytes) 000348 1785078186.091724 write 0x00d467 thru 0x016047 (0x8be1 bytes) 000349 1785078186.093701 fallocate 0x0193f7 thru 0x01c137 (0x2d41 bytes) 000350 1785078186.112308 trunc from 0x029ff0 to 0x0012dc (0x28d13 bytes) 000351 1785078186.224346 trunc from 0x0012dd to 0x012497 (0x111bb bytes) 000352 1785078186.254312 read 0x00bfe4 thru 0x012497 (0x64b4 bytes) 000353 1785078186.270600 punch from 0x0015b5 to 0x004c31 (0x367d bytes) 000355 1785078186.277764 write 0x00ad43 thru 0x018a96 (0xdd54 bytes) 000356 1785078186.282284 mapwrite 0x03232f thru 0x03bbd3 (0x98a5 bytes) 000357 1785078186.344461 trunc from 0x02c506 to 0x032d9f (0x689a bytes) 000358 1785078186.357511 mapwrite 0x01986f thru 0x024e43 (0xb5d5 bytes) 000359 1785078186.388842 trunc from 0x00aebc to 0x010d94 (0x5ed9 bytes) 000360 1785078186.412645 read 0x01d2cc thru 0x029f2e (0xcc63 bytes) 000361 1785078186.414733 fallocate 0x005147 thru 0x00a166 (0x5020 bytes) 000363 1785078186.428445 mapread 0x00ec90 thru 0x0140be (0x542f bytes) 000364 1785078186.460852 write 0x012da4 thru 0x01fdfa (0xd057 bytes) 000365 1785078186.462501 read 0x02bbe8 thru 0x02f9d5 (0x3dee bytes) 000366 1785078186.481432 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bbd4): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bbd4): Mirror 3: 0x759cbce5 (stale) CRC-32 checksum value for chunk [0, 0x3bbd4): Mirror 1: 0xe2a17d3c Mirror 4: 0xe2a17d3c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bbd4: skipped 000367 1785078187.779937 mapread 0x001f27 thru 0x0023f4 (0x4ce bytes) 000368 1785078187.814012 trunc from 0x03bbd4 to 0x031fcf (0x9c04 bytes) 000370 1785078187.933764 read 0x02f086 thru 0x030218 (0x1193 bytes) 000371 1785078187.963875 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31fd0): Mirror 4: 0xcb6278cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31fd0): Mirror 4: 0xcb6278cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31fd0: skipped 000373 1785078188.564731 read 0x01a9f8 thru 0x01ce68 (0x2471 bytes) 000374 1785078188.572003 trunc from 0x0176ea to 0x01db33 (0x644a bytes) 000375 1785078188.579056 trunc from 0x031fd0 to 0x01e1ee (0x13de1 bytes) 000376 1785078188.612539 read 0x0064a3 thru 0x011df3 (0xb951 bytes) 000377 1785078188.628097 write 0x0397c6 thru 0x03afae (0x17e9 bytes) 000378 1785078188.629759 mapread 0x029363 thru 0x036d7d (0xda1b bytes) 000379 1785078188.637679 read 0x01af2d thru 0x01ff81 (0x5055 bytes) 000380 1785078188.638405 punch from 0x00376c to 0x003954 (0x1e9 bytes) 000381 1785078188.645149 trunc from 0x03afaf to 0x030d77 (0xa237 bytes) 000382 1785078188.665094 punch from 0x02e3ef to 0x030d77 (0x2989 bytes) 000383 1785078188.670624 read 0x015516 thru 0x01f359 (0x9e44 bytes) 000385 1785078188.676937 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000386 1785078188.883136 mapread 0x01087a thru 0x019aed (0x9274 bytes) 000388 1785078188.931389 fallocate 0x020a2b thru 0x02ad61 (0xa337 bytes) 000389 1785078188.945641 mapread 0x00948b thru 0x017bd9 (0xe74f bytes) 000391 1785078188.960413 mapread 0x003504 thru 0x0088c9 (0x53c6 bytes) 000392 1785078188.962539 trunc from 0x030d78 to 0x00df1a (0x22e5d bytes) 000393 1785078188.992752 fallocate 0x019f0c thru 0x01dcc0 (0x3db5 bytes) 000394 1785078188.998063 fallocate 0x03fe93 thru 0x03ffff (0x16d bytes) 000395 1785078189.002884 trunc from 0x014b9e to 0x01648f (0x18f2 bytes) 000397 1785078189.013213 mapwrite 0x019482 thru 0x0236eb (0xa26a bytes) 000399 1785078189.061560 trunc from 0x003908 to 0x0102a1 (0xc99a bytes) 000400 1785078189.069490 fallocate 0x03a93f thru 0x03ffff (0x56c1 bytes) 000401 1785078189.074332 trunc from 0x014a73 to 0x0202fd (0xb88b bytes) 000402 1785078189.079697 mapread 0x00ec8b thru 0x0182dc (0x9652 bytes) 000403 1785078189.094714 trunc from 0x0209cf to 0x02dafb (0xd12d bytes) 000405 1785078189.101207 trunc from 0x040000 to 0x0211d7 (0x1ee28 bytes) 000406 1785078189.121654 read 0x0138de thru 0x0211d7 (0xd8fa bytes) 000407 1785078189.126720 fallocate 0x002bfc thru 0x010422 (0xd827 bytes) 000409 1785078189.133429 mapread 0x02051d thru 0x0211d7 (0xcbb bytes) 000410 1785078189.135397 mapread 0x01a4fa thru 0x0211d7 (0x6cde bytes) 000411 1785078189.140642 MIRROR_EXTEND to 2 mirrors 000412 1785078189.845873 read 0x01515a thru 0x0206f5 (0xb59c bytes) 000413 1785078189.859630 punch from 0x0190c5 to 0x01e473 (0x53af bytes) 000414 1785078189.914684 mapwrite 0x01face thru 0x026349 (0x687c bytes) 000415 1785078189.996385 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000416 1785078190.149599 trunc from 0x0158e8 to 0x021233 (0xb94c bytes) 000417 1785078190.157481 fallocate 0x00f72b thru 0x015948 (0x621e bytes) 000418 1785078190.165402 write 0x031ae4 thru 0x034b3e (0x305b bytes) 000419 1785078190.172079 mapread 0x031338 thru 0x034b3e (0x3807 bytes) 000421 1785078190.176646 mapwrite 0x0002c8 thru 0x00087a (0x5b3 bytes) 000422 1785078190.226547 read 0x0103a2 thru 0x013f5d (0x3bbc bytes) 000424 1785078190.235050 punch from 0x0132f9 to 0x021edd (0xebe5 bytes) 000425 1785078190.244099 read 0x00b576 thru 0x017fcf (0xca5a bytes) 000426 1785078190.272487 punch from 0x022781 to 0x0306e5 (0xdf65 bytes) 000427 1785078190.282565 trunc from 0x0250cc to 0x031107 (0xc03c bytes) 000428 1785078190.296928 punch from 0x0050ad to 0x0123d7 (0xd32b bytes) 000429 1785078190.309394 punch from 0x00f783 to 0x019325 (0x9ba3 bytes) 000430 1785078190.314020 punch from 0x003dc1 to 0x008e85 (0x50c5 bytes) 000432 1785078190.327244 mapread 0x016909 thru 0x02253c (0xbc34 bytes) 000433 1785078190.371902 trunc from 0x034b3f to 0x013945 (0x211f9 bytes) 000434 1785078190.403050 read 0x01093e thru 0x013908 (0x2fcb bytes) 000436 1785078190.481097 trunc from 0x013946 to 0x00549f (0xe4a6 bytes) 000437 1785078190.524966 fallocate 0x035ebd thru 0x03d8a9 (0x79ed bytes) 000438 1785078190.546044 mapwrite 0x01d3fd thru 0x020ccd (0x38d1 bytes) 000439 1785078190.616903 mapwrite 0x03444d thru 0x03d964 (0x9518 bytes) 000440 1785078190.712615 punch from 0x022512 to 0x02d384 (0xae73 bytes) 000441 1785078190.719549 mapread 0x02a702 thru 0x033089 (0x8988 bytes) 000442 1785078190.749438 trunc from 0x03d965 to 0x03c737 (0x122d bytes) 000443 1785078190.768266 mapwrite 0x003688 thru 0x00613d (0x2ab6 bytes) 000444 1785078190.800513 punch from 0x00e13c to 0x017b9a (0x9a5f bytes) 000445 1785078190.809100 read 0x029648 thru 0x032299 (0x8c52 bytes) 000446 1785078190.827764 mapwrite 0x01a076 thru 0x02435e (0xa2e9 bytes) 000447 1785078190.871573 trunc from 0x00452f to 0x0074fb (0x2fcd bytes) 000448 1785078190.887126 trunc from 0x03c738 to 0x025e37 (0x16900 bytes) 000450 1785078190.932078 fallocate 0x00a7b5 thru 0x0136e9 (0x8f35 bytes) 000453 1785078190.964806 trunc from 0x010a5a to 0x01d741 (0xcce8 bytes) 000454 1785078190.979354 write 0x004e83 thru 0x009002 (0x4180 bytes) 000455 1785078190.993926 mapread 0x022d34 thru 0x025e37 (0x3104 bytes) 000457 1785078190.995781 punch from 0x0053f5 to 0x012f0a (0xdb16 bytes) 000458 1785078191.066685 trunc from 0x025e38 to 0x03b5f4 (0x157bd bytes) 000459 1785078191.125935 MIRROR_EXTEND to 2 mirrors 000460 1785078191.903139 write 0x02fd4d thru 0x0312f3 (0x15a7 bytes) 000461 1785078191.964292 fallocate 0x0284e8 thru 0x02c621 (0x413a bytes) 000462 1785078191.972134 mapread 0x01a14a thru 0x02070c (0x65c3 bytes) 000463 1785078192.042406 MIRROR_RESYNC 2 mirrors 000464 1785078193.132580 punch from 0x029252 to 0x02e454 (0x5203 bytes) 000465 1785078193.190017 read 0x005fe6 thru 0x012e48 (0xce63 bytes) 000466 1785078193.218314 mapwrite 0x02b931 thru 0x02ea2b (0x30fb bytes) 000467 1785078193.306554 mapwrite 0x014663 thru 0x014f6e (0x90c bytes) 000468 1785078193.353972 trunc from 0x030217 to 0x037481 (0x726b bytes) 000469 1785078193.366147 mapread 0x02a6db thru 0x03395e (0x9284 bytes) 000470 1785078193.381787 punch from 0x02c0e5 to 0x02d960 (0x187c bytes) 000471 1785078193.386643 write 0x039a84 thru 0x03f856 (0x5dd3 bytes) 000472 1785078193.393051 fallocate 0x0274cd thru 0x027877 (0x3ab bytes) 000473 1785078193.401289 fallocate 0x034795 thru 0x03f89a (0xb106 bytes) 000474 1785078193.424900 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f857): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f857): Mirror 2: 0xa3627ede (stale) CRC-32 checksum value for chunk [0, 0x3f857): Mirror 1: 0xabcdb6bc Mirror 3: 0xabcdb6bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f857: skipped 000476 1785078194.566286 trunc from 0x03f857 to 0x026ab9 (0x18d9d bytes) 000477 1785078194.750656 fallocate 0x027798 thru 0x03090d (0x9176 bytes) 000478 1785078194.774200 trunc from 0x03090e to 0x01d563 (0x133aa bytes) 000480 1785078194.854394 mapwrite 0x00d8c6 thru 0x010399 (0x2ad4 bytes) 000481 1785078194.986117 write 0x0397fc thru 0x03ffff (0x6804 bytes) 000482 1785078194.987599 mapread 0x00c292 thru 0x01a455 (0xe1c4 bytes) 000483 1785078195.005627 fallocate 0x032110 thru 0x0345e3 (0x24d4 bytes) 000484 1785078195.022150 mapwrite 0x00cbd8 thru 0x00d896 (0xcbf bytes) 000485 1785078195.053078 punch from 0x01a17f to 0x027a16 (0xd898 bytes) 000486 1785078195.061777 read 0x031b12 thru 0x03ffff (0xe4ee bytes) 000487 1785078195.087537 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc0db4671 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc0db4671 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000488 1785078195.481418 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000489 1785078195.669840 trunc from 0x040000 to 0x037675 (0x898a bytes) 000490 1785078195.691795 mapread 0x01db63 thru 0x0213cf (0x386d bytes) 000491 1785078195.712011 trunc from 0x037676 to 0x00d6d0 (0x29fa5 bytes) 000492 1785078195.729809 read 0x005f96 thru 0x00c277 (0x62e2 bytes) 000495 1785078195.744636 mapwrite 0x03f5a3 thru 0x03ffff (0xa5d bytes) 000496 1785078195.799943 mapwrite 0x031007 thru 0x037eae (0x6ea8 bytes) 000497 1785078195.846706 punch from 0x010cca to 0x01e238 (0xd56f bytes) 000498 1785078195.851512 punch from 0x018a8f to 0x028072 (0xf5e4 bytes) 000500 1785078195.859353 punch from 0x00ceac to 0x01126c (0x43c1 bytes) 000501 1785078195.864715 trunc from 0x0321c6 to 0x033b09 (0x1944 bytes) 000503 1785078195.871221 mapread 0x032c52 thru 0x03a39a (0x7749 bytes) 000504 1785078195.877780 trunc from 0x030fb9 to 0x033235 (0x227d bytes) 000505 1785078195.884227 trunc from 0x040000 to 0x01bda7 (0x24258 bytes) 000506 1785078195.894845 mapwrite 0x02be6e thru 0x02fcc8 (0x3e5b bytes) 000507 1785078195.965007 read 0x02fa94 thru 0x02fcc8 (0x235 bytes) 000508 1785078195.969016 read 0x01db57 thru 0x026f11 (0x93bb bytes) 000509 1785078195.979703 trunc from 0x013ce9 to 0x01ce41 (0x9159 bytes) 000510 1785078195.990578 MIRROR_EXTEND to 2 mirrors 000511 1785078197.111754 MIRROR_EXTEND to 3 mirrors 000512 1785078198.304787 mapread 0x004020 thru 0x006b0d (0x2aee bytes) 000513 1785078198.365590 fallocate 0x00721f thru 0x015bf5 (0xe9d7 bytes) 000514 1785078198.418139 fallocate 0x00be58 thru 0x00d672 (0x181b bytes) 000515 1785078198.426526 fallocate 0x016571 thru 0x018ee2 (0x2972 bytes) 000516 1785078198.434337 mapread 0x002c87 thru 0x009698 (0x6a12 bytes) 000517 1785078198.453147 write 0x02f2c2 thru 0x03f0e0 (0xfe1f bytes) 000518 1785078198.474478 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f0e1): Mirror 3: 0x88d464a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f0e1): Mirror 3: 0x88d464a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f0e1: skipped 000519 1785078198.851430 read 0x035c32 thru 0x03f0e0 (0x94af bytes) 000520 1785078198.859792 trunc from 0x01fe8e to 0x024a10 (0x4b83 bytes) 000521 1785078198.865330 punch from 0x0310e1 to 0x033a1f (0x293f bytes) 000523 1785078198.872534 read 0x00d0a5 thru 0x0184dc (0xb438 bytes) 000524 1785078198.884764 trunc from 0x03f0e1 to 0x02bc0c (0x134d4 bytes) 000525 1785078198.903395 trunc from 0x02bc0d to 0x01c6c5 (0xf547 bytes) 000526 1785078198.919016 write 0x030002 thru 0x039a82 (0x9a81 bytes) 000527 1785078198.920848 read 0x030b8e thru 0x039a82 (0x8ef5 bytes) 000528 1785078198.921885 mapread 0x003fc5 thru 0x0134f1 (0xf52d bytes) 000530 1785078198.923649 write 0x00ab9d thru 0x00ca6e (0x1ed2 bytes) 000531 1785078198.924817 punch from 0x01e685 to 0x02cab5 (0xe431 bytes) 000532 1785078198.929575 write 0x01ca37 thru 0x01ce36 (0x400 bytes) 000533 1785078198.931231 write 0x00329b thru 0x003e70 (0xbd6 bytes) 000534 1785078198.936885 MIRROR_RESYNC 2 mirrors 000535 1785078199.849940 write 0x025b63 thru 0x02b9c2 (0x5e60 bytes) 000536 1785078199.933348 read 0x034af9 thru 0x036e56 (0x235e bytes) 000537 1785078199.945034 punch from 0x035b93 to 0x039a82 (0x3ef0 bytes) 000538 1785078199.961514 mapread 0x036f3e thru 0x038c9a (0x1d5d bytes) 000539 1785078200.001410 fallocate 0x032854 thru 0x033a42 (0x11ef bytes) 000540 1785078200.032235 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39a83): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a83): Mirror 3: 0x89d80063 (stale) CRC-32 checksum value for chunk [0, 0x39a83): Mirror 1: 0xb683a5e Mirror 4: 0xb683a5e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a83: skipped 000541 1785078201.466861 trunc from 0x039a83 to 0x03727c (0x2806 bytes) 000542 1785078201.559838 trunc from 0x012d6b to 0x021e1e (0xf0b4 bytes) 000543 1785078201.568237 mapread 0x0241ca thru 0x033e4e (0xfc85 bytes) 000544 1785078201.619603 trunc from 0x03727d to 0x016aba (0x207c2 bytes) 000546 1785078201.661442 trunc from 0x016abb to 0x008ccb (0xddef bytes) 000547 1785078201.704481 punch from 0x005843 to 0x008ccb (0x3489 bytes) 000548 1785078201.716418 trunc from 0x0014c4 to 0x008ccb (0x7808 bytes) 000549 1785078201.730393 fallocate 0x01a3c5 thru 0x027889 (0xd4c5 bytes) 000550 1785078201.741498 mapwrite 0x006be4 thru 0x00f458 (0x8875 bytes) 000552 1785078201.847770 trunc from 0x00f459 to 0x019367 (0x9f0f bytes) 000553 1785078201.879949 write 0x03416d thru 0x03ffff (0xbe93 bytes) 000554 1785078201.887338 mapread 0x01c8ab thru 0x025aab (0x9201 bytes) 000555 1785078201.914020 read 0x0340c0 thru 0x0378e7 (0x3828 bytes) 000557 1785078201.916787 punch from 0x00f597 to 0x01278b (0x31f5 bytes) 000558 1785078201.938539 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6227068f (stale) Mirror 4: 0x8ab001f4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9396cad3 Mirror 5: 0x9396cad3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000559 1785078203.299748 fallocate 0x02c4f5 thru 0x02ce75 (0x981 bytes) 000560 1785078203.362246 read 0x0123e3 thru 0x012e24 (0xa42 bytes) 000561 1785078203.371893 mapwrite 0x00ad90 thru 0x017f2c (0xd19d bytes) 000562 1785078203.489070 write 0x03507d thru 0x03a4db (0x545f bytes) 000563 1785078203.528087 write 0x03c9a1 thru 0x03dc57 (0x12b7 bytes) 000564 1785078203.545087 mapwrite 0x005a28 thru 0x0064d3 (0xaac bytes) 000565 1785078203.583076 mapwrite 0x028fc6 thru 0x02b11a (0x2155 bytes) 000566 1785078203.608581 trunc from 0x040000 to 0x01e67f (0x21980 bytes) 000567 1785078203.632529 write 0x023fd7 thru 0x0240d3 (0xfd bytes) 000568 1785078203.634445 trunc from 0x0240d4 to 0x004d7c (0x1f357 bytes) 000569 1785078203.656974 read 0x0044d9 thru 0x004d7c (0x8a4 bytes) 000570 1785078203.670831 mapwrite 0x00fc4d thru 0x017fb0 (0x8364 bytes) 000571 1785078203.782756 MIRROR_RESYNC 4 mirrors 000572 1785078205.104410 trunc from 0x017fb1 to 0x03de2a (0x25e7a bytes) 000573 1785078205.171417 write 0x00963f thru 0x00d4ed (0x3eaf bytes) 000574 1785078205.186801 write 0x00104b thru 0x00920d (0x81c3 bytes) 000575 1785078205.205417 trunc from 0x03de2b to 0x003093 (0x3ad97 bytes) 000576 1785078205.250232 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3094): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3094): Mirror 3: 0xb68a8d14 (stale) Mirror 4: 0xb68a8d14 (stale) Mirror 5: 0xb68a8d14 (stale) CRC-32 checksum value for chunk [0, 0x3094): Mirror 1: 0x4157c511 Mirror 6: 0x4157c511 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3094: skipped 000577 1785078206.251077 punch from 0x0024ed to 0x003093 (0xba7 bytes) 000578 1785078206.310596 write 0x03f756 thru 0x03ffff (0x8aa bytes) 000579 1785078206.324750 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x12b33387 (stale) Mirror 4: 0x12b33387 (stale) Mirror 5: 0x12b33387 (stale) Mirror 6: 0xe98d9f62 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8dd0a778 Mirror 7: 0x8dd0a778 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000580 1785078207.784957 trunc from 0x040000 to 0x012541 (0x2dabe bytes) 000581 1785078207.885649 trunc from 0x006b7c to 0x00eb59 (0x7fde bytes) 000582 1785078207.893140 write 0x005665 thru 0x00a479 (0x4e15 bytes) 000584 1785078207.908049 mapwrite 0x009229 thru 0x00b9d5 (0x27ad bytes) 000585 1785078207.937561 punch from 0x00c727 to 0x010577 (0x3e51 bytes) 000586 1785078207.943491 punch from 0x010cd6 to 0x012541 (0x186c bytes) 000587 1785078207.949386 read 0x004a21 thru 0x00dee7 (0x94c7 bytes) 000588 1785078207.958966 trunc from 0x0119f7 to 0x012541 (0xb4b bytes) 000589 1785078207.974801 MIRROR_RESYNC 6 mirrors 000590 1785078209.454450 mapwrite 0x00eefa thru 0x0157f7 (0x68fe bytes) 000591 1785078209.525512 fallocate 0x03363e thru 0x03e665 (0xb028 bytes) 000592 1785078209.531437 trunc from 0x03bf0e to 0x03e665 (0x2758 bytes) 000593 1785078209.536313 trunc from 0x02a6cb to 0x02e0eb (0x3a21 bytes) 000594 1785078209.542349 fallocate 0x014c2f thru 0x015488 (0x85a bytes) 000595 1785078209.548803 mapread 0x017d84 thru 0x023dd7 (0xc054 bytes) 000596 1785078209.570324 mapread 0x026fa0 thru 0x034190 (0xd1f1 bytes) 000597 1785078209.590685 punch from 0x0232f3 to 0x02fc34 (0xc942 bytes) 000599 1785078209.598372 trunc from 0x0073c7 to 0x00e504 (0x713e bytes) 000600 1785078209.614585 write 0x02a47f thru 0x032a9c (0x861e bytes) 000601 1785078209.627649 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e666): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e666): Mirror 3: 0x237b3981 (stale) Mirror 4: 0x237b3981 (stale) Mirror 5: 0x237b3981 (stale) Mirror 6: 0x237b3981 (stale) Mirror 7: 0x237b3981 (stale) CRC-32 checksum value for chunk [0, 0x3e666): Mirror 1: 0x27b89d36 Mirror 8: 0x27b89d36 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e666: skipped 000602 1785078210.870859 trunc from 0x038628 to 0x03e665 (0x603e bytes) 000603 1785078210.919301 write 0x039e7a thru 0x03c970 (0x2af7 bytes) 000604 1785078210.934575 read 0x013e2d thru 0x01dbd9 (0x9dad bytes) 000605 1785078210.945626 write 0x02dc9d thru 0x034479 (0x67dd bytes) 000607 1785078210.956996 read 0x02c979 thru 0x037388 (0xaa10 bytes) 000608 1785078210.966097 fallocate 0x03d8ce thru 0x03ffff (0x2732 bytes) 000609 1785078210.972531 read 0x02e3d9 thru 0x039296 (0xaebe bytes) 000611 1785078210.976393 mapwrite 0x03f986 thru 0x03ffff (0x67a bytes) 000612 1785078211.013339 read 0x02667b thru 0x0361ad (0xfb33 bytes) 000614 1785078211.015466 trunc from 0x040000 to 0x02bad2 (0x1452d bytes) 000615 1785078211.038856 read 0x01f03e thru 0x01f712 (0x6d5 bytes) 000616 1785078211.039939 read 0x01c9e0 thru 0x0265ce (0x9bef bytes) 000617 1785078211.041241 mapwrite 0x014fb7 thru 0x01f5d5 (0xa61f bytes) 000618 1785078211.059884 fallocate 0x032148 thru 0x033ff2 (0x1eab bytes) 000619 1785078211.065945 trunc from 0x013bc8 to 0x020fc6 (0xd3ff bytes) 000620 1785078211.073851 mapread 0x006d44 thru 0x014466 (0xd723 bytes) 000621 1785078211.086004 mapwrite 0x0291bf thru 0x02b9fe (0x2840 bytes) 000622 1785078211.112090 mapwrite 0x001f10 thru 0x011e88 (0xff79 bytes) 000623 1785078211.139770 mapread 0x009179 thru 0x0092d2 (0x15a bytes) 000624 1785078211.146318 read 0x02b6dd thru 0x02bad2 (0x3f6 bytes) 000626 1785078211.150099 write 0x00a33a thru 0x01035f (0x6026 bytes) 000627 1785078211.175089 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bad3): Mirror 4: 0x268d7ae3 (stale) Mirror 5: 0x268d7ae3 (stale) Mirror 6: 0x268d7ae3 (stale) Mirror 7: 0x268d7ae3 (stale) Mirror 8: 0xeed7782 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bad3): Mirror 4: 0x268d7ae3 (stale) Mirror 5: 0x268d7ae3 (stale) Mirror 6: 0x268d7ae3 (stale) Mirror 7: 0x268d7ae3 (stale) Mirror 8: 0xeed7782 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bad3: skipped 000628 1785078211.729631 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bad3): Mirror 5: 0x268d7ae3 (stale) Mirror 6: 0x268d7ae3 (stale) Mirror 7: 0x268d7ae3 (stale) Mirror 8: 0xeed7782 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bad3): Mirror 5: 0x268d7ae3 (stale) Mirror 6: 0x268d7ae3 (stale) Mirror 7: 0x268d7ae3 (stale) Mirror 8: 0xeed7782 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bad3: skipped 000629 1785078212.150541 fallocate 0x009c49 thru 0x00ae8a (0x1242 bytes) 000630 1785078212.159440 punch from 0x01055f to 0x01a9e6 (0xa488 bytes) 000631 1785078212.174427 punch from 0x021bb2 to 0x022267 (0x6b6 bytes) 000632 1785078212.181507 mapread 0x0212c0 thru 0x02ab6a (0x98ab bytes) 000633 1785078212.226447 write 0x0222a2 thru 0x02ea72 (0xc7d1 bytes) 000634 1785078212.236848 trunc from 0x02ea73 to 0x035d56 (0x72e4 bytes) 000635 1785078212.286227 mapwrite 0x007a26 thru 0x00dc0a (0x61e5 bytes) 000636 1785078212.351347 trunc from 0x017450 to 0x024ba0 (0xd751 bytes) 000637 1785078212.361261 trunc from 0x035d57 to 0x0315a0 (0x47b6 bytes) 000639 1785078212.404259 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x315a1): Mirror 5: 0x942a146 (stale) Mirror 6: 0x942a146 (stale) Mirror 7: 0x942a146 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x315a1): Mirror 5: 0x942a146 (stale) Mirror 6: 0x942a146 (stale) Mirror 7: 0x942a146 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x315a1: skipped 000640 1785078212.799211 fallocate 0x038d8b thru 0x039dc0 (0x1036 bytes) 000641 1785078212.805956 read 0x010642 thru 0x01ade8 (0xa7a7 bytes) 000642 1785078212.830864 trunc from 0x0094e5 to 0x00dd9b (0x48b7 bytes) 000643 1785078212.846254 mapwrite 0x03b5dd thru 0x03ca57 (0x147b bytes) 000644 1785078212.961810 mapread 0x031638 thru 0x03ca57 (0xb420 bytes) 000645 1785078212.993866 punch from 0x00c09e to 0x01a3d4 (0xe337 bytes) 000646 1785078213.003704 trunc from 0x0271e1 to 0x02e8a6 (0x76c6 bytes) 000647 1785078213.015086 punch from 0x00f7c5 to 0x0119d7 (0x2213 bytes) 000648 1785078213.036976 write 0x01216f thru 0x016437 (0x42c9 bytes) 000649 1785078213.055886 mapwrite 0x014847 thru 0x022ce9 (0xe4a3 bytes) 000650 1785078213.115780 fallocate 0x02a003 thru 0x036dca (0xcdc8 bytes) 000652 1785078213.134150 read 0x0127d1 thru 0x01fe16 (0xd646 bytes) 000653 1785078213.142718 MIRROR_RESYNC 4 mirrors 000654 1785078214.381395 mapread 0x016733 thru 0x01c3a3 (0x5c71 bytes) 000655 1785078214.405246 mapread 0x00899c thru 0x0132fa (0xa95f bytes) 000656 1785078214.417103 mapwrite 0x000329 thru 0x00b4a8 (0xb180 bytes) 000657 1785078214.555528 mapread 0x0157c1 thru 0x01d220 (0x7a60 bytes) 000658 1785078214.570797 MIRROR_RESYNC 4 mirrors 000660 1785078216.021281 MIRROR_RESYNC 4 mirrors 000661 1785078216.579701 mapwrite 0x0133a6 thru 0x0192d1 (0x5f2c bytes) 000662 1785078216.713742 fallocate 0x01337a thru 0x01ac18 (0x789f bytes) 000663 1785078216.726677 trunc from 0x02a49b to 0x030a58 (0x65be bytes) 000666 1785078216.738369 punch from 0x02e694 to 0x03a916 (0xc283 bytes) 000667 1785078216.744704 fallocate 0x033aee thru 0x03933e (0x5851 bytes) 000668 1785078216.762709 MIRROR_RESYNC 4 mirrors 000669 1785078218.215575 mapwrite 0x019a88 thru 0x026df4 (0xd36d bytes) 000670 1785078218.421597 trunc from 0x03ca58 to 0x01d97f (0x1f0d8 bytes) 000671 1785078218.457805 mapread 0x004058 thru 0x00e928 (0xa8d1 bytes) 000672 1785078218.482992 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1d980): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d980): Mirror 5: 0xde64d224 (stale) Mirror 6: 0xde64d224 (stale) Mirror 7: 0xde64d224 (stale) CRC-32 checksum value for chunk [0, 0x1d980): Mirror 1: 0xab8d9ed5 Mirror 8: 0xab8d9ed5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d980: skipped 000673 1785078219.475964 mapwrite 0x002741 thru 0x0067b6 (0x4076 bytes) 000674 1785078219.632289 trunc from 0x01d980 to 0x00d477 (0x10508 bytes) 000675 1785078219.693421 mapwrite 0x013fd4 thru 0x021bca (0xdbf7 bytes) 000676 1785078219.766324 read 0x00ad37 thru 0x012685 (0x794f bytes) 000677 1785078219.768001 read 0x012432 thru 0x01f079 (0xcc48 bytes) 000678 1785078219.769467 mapwrite 0x02328e thru 0x02e67b (0xb3ee bytes) 000679 1785078219.843613 read 0x00b328 thru 0x00bb37 (0x810 bytes) 000680 1785078219.844802 mapwrite 0x00e819 thru 0x01e19a (0xf982 bytes) 000681 1785078219.904089 mapwrite 0x0125c2 thru 0x016944 (0x4383 bytes) 000682 1785078219.944778 mapread 0x005eb5 thru 0x00b9d8 (0x5b24 bytes) 000683 1785078219.949498 mapwrite 0x01caea thru 0x023299 (0x67b0 bytes) 000684 1785078219.986309 write 0x026af7 thru 0x02ff3d (0x9447 bytes) 000685 1785078219.988078 mapwrite 0x02602e thru 0x035211 (0xf1e4 bytes) 000686 1785078220.031259 read 0x000e65 thru 0x008e85 (0x8021 bytes) 000687 1785078220.032304 read 0x00a800 thru 0x0106bc (0x5ebd bytes) 000689 1785078220.039839 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35212): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35212): Mirror 5: 0x83ce2be7 (stale) Mirror 6: 0x83ce2be7 (stale) Mirror 7: 0x83ce2be7 (stale) Mirror 8: 0x13dd85f1 (stale) CRC-32 checksum value for chunk [0, 0x35212): Mirror 1: 0xa7a9929b Mirror 9: 0xa7a9929b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35212: skipped 000690 1785078221.184601 read 0x00e4a2 thru 0x010c6e (0x27cd bytes) 000691 1785078221.202290 MIRROR_RESYNC 6 mirrors 000692 1785078222.369348 trunc from 0x022f60 to 0x024d3d (0x1dde bytes) 000693 1785078222.438415 trunc from 0x035212 to 0x035357 (0x146 bytes) 000694 1785078222.527341 fallocate 0x034018 thru 0x03f206 (0xb1ef bytes) 000695 1785078222.561357 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35358): Mirror 5: 0xcded087 (stale) Mirror 6: 0xcded087 (stale) Mirror 7: 0xcded087 (stale) Mirror 8: 0xcded087 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35358): Mirror 5: 0xcded087 (stale) Mirror 6: 0xcded087 (stale) Mirror 7: 0xcded087 (stale) Mirror 8: 0xcded087 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35358: skipped 000696 1785078223.236436 punch from 0x023a88 to 0x031b76 (0xe0ef bytes) 000697 1785078223.246115 mapwrite 0x024ba5 thru 0x0261a9 (0x1605 bytes) 000698 1785078223.300149 write 0x0145ca thru 0x01d0bf (0x8af6 bytes) 000700 1785078223.356936 punch from 0x00d4a3 to 0x017fa3 (0xab01 bytes) 000702 1785078223.410700 write 0x039195 thru 0x03d18f (0x3ffb bytes) 000703 1785078223.418411 trunc from 0x01626a to 0x01eec5 (0x8c5c bytes) 000704 1785078223.425411 trunc from 0x03d190 to 0x022a19 (0x1a776 bytes) 000705 1785078223.459440 trunc from 0x000a59 to 0x00c868 (0xbe10 bytes) 000706 1785078223.468509 mapwrite 0x0129e9 thru 0x01889d (0x5eb5 bytes) 000707 1785078223.522096 trunc from 0x001236 to 0x005378 (0x4143 bytes) 000708 1785078223.533366 fallocate 0x02b020 thru 0x036e6a (0xbe4b bytes) 000709 1785078223.546113 trunc from 0x006bee to 0x00d28a (0x669d bytes) 000710 1785078223.566509 write 0x005836 thru 0x011098 (0xb863 bytes) 000711 1785078223.584936 trunc from 0x036e6b to 0x0235a9 (0x138c1 bytes) 000712 1785078223.637214 punch from 0x0113bb to 0x01bac2 (0xa708 bytes) 000713 1785078223.679554 read 0x00b4c9 thru 0x019d49 (0xe881 bytes) 000714 1785078223.696224 punch from 0x014a1f to 0x021220 (0xc802 bytes) 000715 1785078223.707727 fallocate 0x031366 thru 0x03759f (0x623a bytes) 000716 1785078223.715575 trunc from 0x018cb6 to 0x01fe7a (0x71c5 bytes) 000717 1785078223.721641 mapwrite 0x0069c0 thru 0x01313a (0xc77b bytes) 000718 1785078223.778804 read 0x011fd2 thru 0x014285 (0x22b4 bytes) 000719 1785078223.791995 punch from 0x02310b to 0x0235a9 (0x49f bytes) 000720 1785078223.801782 trunc from 0x013b26 to 0x02072b (0xcc06 bytes) 000721 1785078223.816639 mapread 0x00f524 thru 0x01ae19 (0xb8f6 bytes) 000723 1785078223.886718 trunc from 0x0235aa to 0x00f6d9 (0x13ed0 bytes) 000724 1785078223.912027 trunc from 0x00dfa6 to 0x00e07a (0xd5 bytes) 000725 1785078223.919801 trunc from 0x00f6da to 0x009a13 (0x5cc6 bytes) 000726 1785078223.960534 mapread 0x007e5c thru 0x009a13 (0x1bb8 bytes) 000727 1785078223.963622 mapwrite 0x0154a8 thru 0x024848 (0xf3a1 bytes) 000728 1785078224.131079 fallocate 0x0074a1 thru 0x00831c (0xe7c bytes) 000729 1785078224.144971 mapwrite 0x01dcfd thru 0x022fad (0x52b1 bytes) 000730 1785078224.170524 read 0x018d99 thru 0x021f5e (0x91c6 bytes) 000731 1785078224.171550 fallocate 0x016179 thru 0x01c44b (0x62d3 bytes) 000732 1785078224.183480 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24849): Mirror 5: 0xe213f515 (stale) Mirror 6: 0xe213f515 (stale) Mirror 8: 0xe213f515 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24849): Mirror 5: 0xe213f515 (stale) Mirror 6: 0xe213f515 (stale) Mirror 8: 0xe213f515 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24849: skipped 000733 1785078224.622281 trunc from 0x024849 to 0x005e40 (0x1ea08 bytes) 000734 1785078224.658921 trunc from 0x003144 to 0x005a8e (0x294b bytes) 000735 1785078224.667397 mapread 0x000d31 thru 0x005e40 (0x5110 bytes) 000736 1785078224.738305 mapread 0x0011f4 thru 0x005e40 (0x4c4d bytes) 000737 1785078224.740763 trunc from 0x003d4d to 0x005e40 (0x20f4 bytes) 000738 1785078224.747380 punch from 0x0057bf to 0x005e40 (0x682 bytes) 000739 1785078224.759049 write 0x008a54 thru 0x00dbd2 (0x517f bytes) 000740 1785078224.772019 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdbd3): Mirror 6: 0x5280dd3f (stale) Mirror 8: 0x5280dd3f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdbd3): Mirror 6: 0x5280dd3f (stale) Mirror 8: 0x5280dd3f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdbd3: skipped 000741 1785078225.214705 punch from 0x001f59 to 0x00dbd2 (0xbc7a bytes) 000742 1785078225.233899 mapread 0x00ba8c thru 0x00ce71 (0x13e6 bytes) 000743 1785078225.250647 fallocate 0x02e9e2 thru 0x031710 (0x2d2f bytes) 000744 1785078225.260081 read 0x00bd4c thru 0x01a7ca (0xea7f bytes) 000745 1785078225.272152 trunc from 0x0174c6 to 0x01ea8f (0x75ca bytes) 000746 1785078225.287599 MIRROR_RESYNC 3 mirrors 000748 1785078226.177138 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000749 1785078226.537528 trunc from 0x02ca00 to 0x031710 (0x4d11 bytes) 000750 1785078226.609102 punch from 0x02a3a4 to 0x031710 (0x736d bytes) 000751 1785078226.616951 mapwrite 0x032f5c thru 0x03b9e1 (0x8a86 bytes) 000752 1785078226.690890 punch from 0x00c037 to 0x014f41 (0x8f0b bytes) 000753 1785078226.696940 mapwrite 0x00c071 thru 0x013edb (0x7e6b bytes) 000754 1785078226.734301 punch from 0x02fa82 to 0x031b0b (0x208a bytes) 000755 1785078226.740292 trunc from 0x03b9e2 to 0x028dee (0x12bf3 bytes) 000756 1785078226.764184 mapread 0x010d5b thru 0x0191a0 (0x8446 bytes) 000757 1785078226.797806 MIRROR_RESYNC 2 mirrors 000758 1785078227.580506 write 0x03ca19 thru 0x03ffff (0x35e7 bytes) 000759 1785078227.612969 mapwrite 0x03940d thru 0x03fb1c (0x6710 bytes) 000760 1785078227.660859 trunc from 0x040000 to 0x013da9 (0x2c256 bytes) 000761 1785078227.692100 mapwrite 0x000d09 thru 0x006919 (0x5c11 bytes) 000762 1785078227.741260 mapread 0x010dd1 thru 0x013da9 (0x2fd9 bytes) 000763 1785078227.773006 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000764 1785078227.976835 write 0x0264f1 thru 0x031d87 (0xb897 bytes) 000765 1785078227.985439 mapwrite 0x001ff3 thru 0x00ed12 (0xcd20 bytes) 000766 1785078228.039979 fallocate 0x012872 thru 0x0171e9 (0x4978 bytes) 000767 1785078228.045143 mapwrite 0x0109a1 thru 0x011202 (0x862 bytes) 000768 1785078228.071438 trunc from 0x01aac9 to 0x02436d (0x98a5 bytes) 000769 1785078228.078822 mapwrite 0x03dfdd thru 0x03ffff (0x2023 bytes) 000770 1785078228.129901 fallocate 0x00bf1d thru 0x0159ee (0x9ad2 bytes) 000772 1785078228.147707 MIRROR_EXTEND to 2 mirrors 000773 1785078229.152726 trunc from 0x01dff7 to 0x0298a4 (0xb8ae bytes) 000775 1785078229.208780 write 0x0142ba thru 0x01df68 (0x9caf bytes) 000776 1785078229.224009 punch from 0x016ed1 to 0x0228e6 (0xba16 bytes) 000777 1785078229.237376 trunc from 0x02a4bc to 0x039f63 (0xfaa8 bytes) 000778 1785078229.243607 punch from 0x00670d to 0x016705 (0xfff9 bytes) 000779 1785078229.249077 trunc from 0x02cba2 to 0x035a73 (0x8ed2 bytes) 000780 1785078229.254356 trunc from 0x040000 to 0x02498a (0x1b675 bytes) 000781 1785078229.294894 fallocate 0x03b2b7 thru 0x03e23d (0x2f87 bytes) 000784 1785078229.304897 fallocate 0x0041f0 thru 0x00a2b7 (0x60c8 bytes) 000785 1785078229.316469 fallocate 0x03d1d6 thru 0x03ffff (0x2e2a bytes) 000786 1785078229.327217 mapread 0x023fa0 thru 0x03066d (0xc6ce bytes) 000787 1785078229.358967 read 0x034cf9 thru 0x03f604 (0xa90c bytes) 000788 1785078229.367031 mapread 0x009d63 thru 0x011e33 (0x80d1 bytes) 000789 1785078229.377510 trunc from 0x040000 to 0x00046b (0x3fb94 bytes) 000790 1785078229.396215 fallocate 0x021c37 thru 0x02a175 (0x853f bytes) 000791 1785078229.401658 trunc from 0x00046c to 0x008bc8 (0x875d bytes) 000792 1785078229.422362 fallocate 0x0320fe thru 0x03d31b (0xb21e bytes) 000793 1785078229.427455 write 0x00a1da thru 0x019970 (0xf797 bytes) 000794 1785078229.430578 mapwrite 0x02d7ea thru 0x02f324 (0x1b3b bytes) 000795 1785078229.492491 mapread 0x01e4ac thru 0x025bef (0x7744 bytes) 000796 1785078229.503437 punch from 0x00ff3a to 0x019f44 (0xa00b bytes) 000797 1785078229.509249 punch from 0x0282df to 0x02b14f (0x2e71 bytes) 000798 1785078229.515968 trunc from 0x0150a4 to 0x020aad (0xba0a bytes) 000799 1785078229.522053 read 0x00a0df thru 0x01552f (0xb451 bytes) 000801 1785078229.536140 trunc from 0x02f325 to 0x03cad8 (0xd7b4 bytes) 000802 1785078229.555316 write 0x03cf73 thru 0x03ffff (0x308d bytes) 000803 1785078229.566901 fallocate 0x02e54c thru 0x034b8b (0x6640 bytes) 000804 1785078229.572995 mapwrite 0x025391 thru 0x029a41 (0x46b1 bytes) 000805 1785078229.601641 MIRROR_RESYNC 2 mirrors 000806 1785078230.596799 trunc from 0x040000 to 0x03ebe6 (0x1419 bytes) 000807 1785078230.649373 mapwrite 0x009e44 thru 0x00bf1d (0x20da bytes) 000809 1785078230.682276 write 0x03e51e thru 0x03e99f (0x482 bytes) 000810 1785078230.688428 trunc from 0x01a3f6 to 0x0229cc (0x85d7 bytes) 000811 1785078230.693489 mapread 0x03ea33 thru 0x03ebe6 (0x1b4 bytes) 000812 1785078230.694803 read 0x008504 thru 0x00e8eb (0x63e8 bytes) 000813 1785078230.696383 trunc from 0x03ebe7 to 0x033603 (0xb5e3 bytes) 000814 1785078230.716449 mapwrite 0x02053b thru 0x0220cd (0x1b93 bytes) 000815 1785078230.750685 mapread 0x02046b thru 0x029fca (0x9b60 bytes) 000816 1785078230.763905 write 0x00936c thru 0x00962d (0x2c2 bytes) 000818 1785078230.767036 trunc from 0x033604 to 0x002166 (0x3149d bytes) 000819 1785078230.795031 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000820 1785078231.014126 write 0x036d74 thru 0x038197 (0x1424 bytes) 000821 1785078231.024108 mapread 0x02f77e thru 0x038197 (0x8a1a bytes) 000822 1785078231.061879 write 0x023983 thru 0x029f2b (0x65a9 bytes) 000823 1785078231.073501 mapread 0x01ef11 thru 0x02c170 (0xd260 bytes) 000824 1785078231.082967 fallocate 0x01bd31 thru 0x024476 (0x8746 bytes) 000825 1785078231.097332 mapwrite 0x03d382 thru 0x03ffff (0x2c7e bytes) 000826 1785078231.185367 MIRROR_EXTEND to 2 mirrors 000827 1785078232.376122 trunc from 0x040000 to 0x0343f8 (0xbc07 bytes) 000828 1785078232.518841 read 0x0168ef thru 0x0246e8 (0xddfa bytes) 000829 1785078232.530788 fallocate 0x038661 thru 0x03a38e (0x1d2e bytes) 000831 1785078232.539384 trunc from 0x013a3d to 0x01f3c7 (0xb98b bytes) 000833 1785078232.563683 punch from 0x031d71 to 0x0343f8 (0x2688 bytes) 000834 1785078232.581977 fallocate 0x01d39c thru 0x0212d5 (0x3f3a bytes) 000835 1785078232.592118 mapread 0x00f990 thru 0x013c8b (0x42fc bytes) 000837 1785078232.637521 write 0x01c608 thru 0x01e1f5 (0x1bee bytes) 000838 1785078232.648746 trunc from 0x0343f9 to 0x02d773 (0x6c85 bytes) 000839 1785078232.682020 trunc from 0x001404 to 0x00c152 (0xad4f bytes) 000840 1785078232.691483 MIRROR_RESYNC 2 mirrors 000841 1785078233.774550 read 0x01e472 thru 0x02a291 (0xbe20 bytes) 000842 1785078233.785595 trunc from 0x02d774 to 0x01e482 (0xf2f1 bytes) 000843 1785078233.857959 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000844 1785078234.114417 punch from 0x00289c to 0x005161 (0x28c6 bytes) 000845 1785078234.140151 mapwrite 0x02d5ea thru 0x03ae58 (0xd86f bytes) 000846 1785078234.287222 punch from 0x0224a6 to 0x031232 (0xed8d bytes) 000848 1785078234.301551 mapread 0x02df53 thru 0x037c5e (0x9d0c bytes) 000849 1785078234.319303 punch from 0x02916f to 0x031650 (0x84e2 bytes) 000850 1785078234.329906 trunc from 0x019d30 to 0x0264d3 (0xc7a4 bytes) 000851 1785078234.354813 trunc from 0x03ae59 to 0x017dc6 (0x23092 bytes) 000852 1785078234.376460 read 0x00feed thru 0x011023 (0x1137 bytes) 000853 1785078234.380819 trunc from 0x017dc7 to 0x011761 (0x6665 bytes) 000854 1785078234.399981 punch from 0x006ae5 to 0x00ccdf (0x61fb bytes) 000855 1785078234.408665 punch from 0x00b2ad to 0x011761 (0x64b5 bytes) 000857 1785078234.420411 trunc from 0x0032c4 to 0x00a38a (0x70c7 bytes) 000858 1785078234.441559 trunc from 0x011762 to 0x022886 (0x11125 bytes) 000859 1785078234.463251 mapwrite 0x03ccac thru 0x03ffff (0x3354 bytes) 000860 1785078234.674315 MIRROR_EXTEND to 2 mirrors 000861 1785078235.748958 mapread 0x01c06a thru 0x020cf6 (0x4c8d bytes) 000863 1785078235.785815 trunc from 0x040000 to 0x038ea9 (0x7156 bytes) 000864 1785078235.883244 punch from 0x009db4 to 0x009f9f (0x1ec bytes) 000865 1785078235.896926 trunc from 0x0389ef to 0x038ea9 (0x4bb bytes) 000866 1785078235.906003 mapread 0x02a5f4 thru 0x02eab3 (0x44c0 bytes) 000867 1785078235.942921 mapread 0x035aae thru 0x038ea9 (0x33fc bytes) 000869 1785078235.955894 fallocate 0x00a6da thru 0x00e3a0 (0x3cc7 bytes) 000870 1785078235.968495 mapread 0x01a2f6 thru 0x029898 (0xf5a3 bytes) 000871 1785078235.985809 mapread 0x007dee thru 0x00cde4 (0x4ff7 bytes) 000872 1785078236.010364 fallocate 0x00ae3d thru 0x00fa9e (0x4c62 bytes) 000873 1785078236.047027 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000875 1785078236.212272 mapread 0x01425f thru 0x01e19f (0x9f41 bytes) 000877 1785078236.245092 write 0x01edd0 thru 0x02899f (0x9bd0 bytes) 000878 1785078236.258212 trunc from 0x013887 to 0x0222d4 (0xea4e bytes) 000879 1785078236.278467 read 0x02ab40 thru 0x033f48 (0x9409 bytes) 000881 1785078236.291267 mapwrite 0x02509f thru 0x02fe28 (0xad8a bytes) 000882 1785078236.322734 trunc from 0x038eaa to 0x02cebf (0xbfea bytes) 000883 1785078236.344380 trunc from 0x02590a to 0x027f19 (0x2610 bytes) 000884 1785078236.350680 punch from 0x01a448 to 0x01b929 (0x14e2 bytes) 000885 1785078236.358355 write 0x037302 thru 0x03ad5a (0x3a59 bytes) 000886 1785078236.359722 mapwrite 0x02f59e thru 0x035b01 (0x6564 bytes) 000887 1785078236.395713 mapwrite 0x02e8b1 thru 0x03482d (0x5f7d bytes) 000890 1785078236.440479 mapwrite 0x021b49 thru 0x027b24 (0x5fdc bytes) 000891 1785078236.489395 read 0x0124e3 thru 0x01b823 (0x9341 bytes) 000892 1785078236.510410 read 0x02b278 thru 0x031071 (0x5dfa bytes) 000893 1785078236.511661 mapwrite 0x0232f9 thru 0x028eb1 (0x5bb9 bytes) 000895 1785078236.537808 trunc from 0x031c52 to 0x038612 (0x69c1 bytes) 000896 1785078236.547549 punch from 0x026f9f to 0x02dc0e (0x6c70 bytes) 000897 1785078236.561875 write 0x00d4ac thru 0x019a75 (0xc5ca bytes) 000898 1785078236.564766 mapwrite 0x0211b4 thru 0x025450 (0x429d bytes) 000899 1785078236.590833 trunc from 0x02f5ee to 0x034e47 (0x585a bytes) 000900 1785078236.596505 trunc from 0x03ad5b to 0x02af6d (0xfded bytes) 000901 1785078236.612885 trunc from 0x02af6e to 0x03c530 (0x115c3 bytes) 000902 1785078236.634088 MIRROR_EXTEND to 2 mirrors 000903 1785078237.517632 mapwrite 0x027bdf thru 0x0331b5 (0xb5d7 bytes) 000905 1785078237.647366 punch from 0x022aff to 0x02ecdd (0xc1df bytes) 000906 1785078237.662457 MIRROR_RESYNC 2 mirrors 000907 1785078238.573042 mapwrite 0x01b863 thru 0x0276c0 (0xbe5e bytes) 000908 1785078238.705899 fallocate 0x02c218 thru 0x03b6c5 (0xf4ae bytes) 000909 1785078238.715437 fallocate 0x02821f thru 0x03197d (0x975f bytes) 000910 1785078238.726851 fallocate 0x01613f thru 0x01cdb6 (0x6c78 bytes) 000911 1785078238.735716 fallocate 0x009aaf thru 0x00b06b (0x15bd bytes) 000912 1785078238.743659 fallocate 0x019639 thru 0x01e61a (0x4fe2 bytes) 000914 1785078238.753092 fallocate 0x02fc45 thru 0x0320e7 (0x24a3 bytes) 000915 1785078238.759263 write 0x03b49e thru 0x03ffff (0x4b62 bytes) 000916 1785078238.766845 mapwrite 0x0373da thru 0x03ffff (0x8c26 bytes) 000917 1785078238.803104 write 0x00fadb thru 0x017816 (0x7d3c bytes) 000918 1785078238.810818 fallocate 0x0165f3 thru 0x022c04 (0xc612 bytes) 000920 1785078238.823721 MIRROR_RESYNC 2 mirrors 000922 1785078239.802129 mapread 0x011700 thru 0x0183b3 (0x6cb4 bytes) 000923 1785078239.828916 trunc from 0x031744 to 0x03a5c7 (0x8e84 bytes) 000924 1785078239.873434 fallocate 0x033f00 thru 0x03cf79 (0x907a bytes) 000925 1785078239.891798 mapwrite 0x01e097 thru 0x028ac1 (0xaa2b bytes) 000927 1785078239.953630 write 0x01c7c2 thru 0x01eb7d (0x23bc bytes) 000928 1785078239.956795 trunc from 0x040000 to 0x0155f0 (0x2aa0f bytes) 000929 1785078239.983410 write 0x00d986 thru 0x00ee29 (0x14a4 bytes) 000930 1785078239.998078 trunc from 0x0155f1 to 0x019615 (0x4025 bytes) 000931 1785078240.055033 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000933 1785078240.366423 trunc from 0x0062a9 to 0x0068b4 (0x60c bytes) 000934 1785078240.373026 write 0x00ca66 thru 0x00e225 (0x17c0 bytes) 000935 1785078240.391911 mapread 0x00d5ac thru 0x0193a2 (0xbdf7 bytes) 000936 1785078240.429801 punch from 0x00a84c to 0x00c672 (0x1e27 bytes) 000937 1785078240.446345 fallocate 0x0047da thru 0x00a1b1 (0x59d8 bytes) 000938 1785078240.461771 mapwrite 0x01358a thru 0x022faf (0xfa26 bytes) 000939 1785078240.568023 read 0x01b23e thru 0x022faf (0x7d72 bytes) 000940 1785078240.569043 mapwrite 0x0014a4 thru 0x00a4fb (0x9058 bytes) 000941 1785078240.631959 write 0x01e269 thru 0x023e13 (0x5bab bytes) 000944 1785078240.639618 write 0x035be3 thru 0x03ffff (0xa41d bytes) 000945 1785078240.644311 trunc from 0x017452 to 0x023c9c (0xc84b bytes) 000946 1785078240.686629 punch from 0x02dff5 to 0x02e43f (0x44b bytes) 000948 1785078240.711854 MIRROR_EXTEND to 2 mirrors 000949 1785078241.790681 mapread 0x0194b1 thru 0x0203a8 (0x6ef8 bytes) 000950 1785078241.828975 trunc from 0x02ea89 to 0x03e841 (0xfdb9 bytes) 000951 1785078241.889442 trunc from 0x040000 to 0x039f6f (0x6090 bytes) 000952 1785078241.925845 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39f70): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39f70): Mirror 2: 0xcf33bdb0 (stale) CRC-32 checksum value for chunk [0, 0x39f70): Mirror 1: 0x90b3710f Mirror 3: 0x90b3710f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39f70: skipped 000953 1785078243.206471 mapread 0x00ed36 thru 0x00fa77 (0xd42 bytes) 000956 1785078243.295403 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39f70): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39f70): Mirror 2: 0xcf33bdb0 (stale) CRC-32 checksum value for chunk [0, 0x39f70): Mirror 1: 0x90b3710f Mirror 3: 0x90b3710f Mirror 4: 0x90b3710f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39f70: skipped 000957 1785078244.511511 MIRROR_RESYNC 4 mirrors 000958 1785078245.550348 mapwrite 0x03eccc thru 0x03ffff (0x1334 bytes) 000959 1785078245.625402 punch from 0x00a503 to 0x0121f8 (0x7cf6 bytes) 000960 1785078245.631622 mapwrite 0x00dfd1 thru 0x01dd31 (0xfd61 bytes) 000961 1785078245.674394 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd0bc4df0 (stale) Mirror 3: 0xd0bc4df0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd0bc4df0 (stale) Mirror 3: 0xd0bc4df0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000962 1785078246.047241 trunc from 0x040000 to 0x02e3f6 (0x11c09 bytes) 000963 1785078246.075567 trunc from 0x023438 to 0x0236bf (0x288 bytes) 000964 1785078246.081395 mapread 0x003521 thru 0x00eb62 (0xb642 bytes) 000965 1785078246.100397 punch from 0x0045e9 to 0x01375e (0xf176 bytes) 000966 1785078246.106143 write 0x03a875 thru 0x03ffff (0x578b bytes) 000967 1785078246.107696 trunc from 0x040000 to 0x029bc9 (0x16436 bytes) 000968 1785078246.126282 punch from 0x0136d2 to 0x022ae8 (0xf417 bytes) 000969 1785078246.131881 fallocate 0x00a2e4 thru 0x013a63 (0x9780 bytes) 000970 1785078246.137340 fallocate 0x012a80 thru 0x012ad4 (0x55 bytes) 000971 1785078246.148238 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x29bca): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29bca): Mirror 2: 0x64be8378 (stale) Mirror 3: 0x64be8378 (stale) CRC-32 checksum value for chunk [0, 0x29bca): Mirror 1: 0xa293fef3 Mirror 4: 0xa293fef3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29bca: skipped 000972 1785078247.091331 mapread 0x019ec2 thru 0x027a28 (0xdb67 bytes) 000973 1785078247.136860 mapwrite 0x02cc66 thru 0x037f62 (0xb2fd bytes) 000974 1785078247.259747 trunc from 0x037f63 to 0x012bd8 (0x2538a bytes) 000975 1785078247.297058 trunc from 0x012bd9 to 0x03b4e7 (0x2890f bytes) 000976 1785078247.361354 write 0x03a325 thru 0x03e336 (0x4012 bytes) 000978 1785078247.377095 mapread 0x009f0d thru 0x01676f (0xc863 bytes) 000979 1785078247.425081 fallocate 0x011669 thru 0x021525 (0xfebd bytes) 000980 1785078247.436194 read 0x0346c5 thru 0x03e336 (0x9c72 bytes) 000981 1785078247.448857 read 0x02d6e2 thru 0x03c71a (0xf039 bytes) 000982 1785078247.462324 trunc from 0x03e337 to 0x03bd18 (0x261e bytes) 000983 1785078247.481999 mapread 0x03a651 thru 0x03bd18 (0x16c8 bytes) 000984 1785078247.489889 mapwrite 0x02d478 thru 0x02d52a (0xb3 bytes) 000985 1785078247.517429 punch from 0x02e3e6 to 0x03121b (0x2e36 bytes) 000986 1785078247.565041 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bd19): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd19): Mirror 2: 0x157ceb58 (stale) Mirror 3: 0x157ceb58 (stale) Mirror 4: 0xadf8aaf3 (stale) CRC-32 checksum value for chunk [0, 0x3bd19): Mirror 1: 0xad373ba4 Mirror 5: 0xad373ba4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd19: skipped 000987 1785078248.791945 fallocate 0x0059e9 thru 0x010e15 (0xb42d bytes) 000989 1785078248.843379 mapwrite 0x00a69c thru 0x017b96 (0xd4fb bytes) 000990 1785078248.908039 trunc from 0x0288fa to 0x02e779 (0x5e80 bytes) 000991 1785078248.914549 trunc from 0x008c91 to 0x018bb2 (0xff22 bytes) 000992 1785078248.926619 fallocate 0x01bc2a thru 0x01f95d (0x3d34 bytes) 000993 1785078248.931845 mapread 0x03aeeb thru 0x03bd18 (0xe2e bytes) 000994 1785078248.945848 trunc from 0x039582 to 0x03bd18 (0x2797 bytes) 000995 1785078248.955986 punch from 0x01cb52 to 0x02bca5 (0xf154 bytes) 000996 1785078248.973208 read 0x01e06d thru 0x02b4b0 (0xd444 bytes) 000997 1785078248.981782 mapwrite 0x024ae8 thru 0x0289a1 (0x3eba bytes) 000998 1785078248.997280 punch from 0x030188 to 0x03871a (0x8593 bytes) 000999 1785078249.002737 mapread 0x026611 thru 0x02af60 (0x4950 bytes) 001000 1785078249.004807 trunc from 0x03bd19 to 0x00db41 (0x2e1d7 bytes) All operations completed A-OK! PASS 70a (110s) == sanity-flr test 71: check mirror extend parent fid ==== 11:04:14 (1785078254) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x8c:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 11:04:25 (1785078265) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195853 s, 10.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147142 s, 7.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e6:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 11:04:34 (1785078274) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8126848 .. Extending file size to 2643936 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8456928 .. failed Extending file size to 5085664 .. Extending file size to 8994208 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7851456 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8357632 .. Extending file size to 8771328 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61382 61383 61385 61387 61389 done 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) 192.168.203.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:05:55 (1785078355) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6832160 .. Extending file size to 3082112 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8314592 .. Extending file size to 6340544 .. Extending file size to 7294496 .. Extending file size to 8825120 .. Extending file size to 7642784 .. Extending file size to 8800416 .. Extending file size to 9030720 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9108928 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 62748 62749 62750 62754 62755 failed 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) 192.168.203.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:07:21 (1785078441) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1992 1269312 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24524 3582496 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18716 3588304 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43240 7170800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1992 1269312 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24524 3582496 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18716 3588304 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43240 7170800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135545 s, 15.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.213547 s, 9.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xbd96d7de (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:07:39 (1785078459) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142287 s, 14.7 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 11:07:46 (1785078466) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fd:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fd:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fe:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x2fb:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:07:54 (1785078474) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0966804 s, 21.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:08:01 (1785078481) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131599 s, 15.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x301:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fe:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x301:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fe:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ff:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:08:09 (1785078489) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128953 s, 16.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x300:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x300:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x301:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:08:44 (1785078524) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg336-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065863 s, 15.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x303:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x302:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x303:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x302:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x304:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:09:38 (1785078578) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg336-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x305:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145121 s, 14.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x305:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x305:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x305:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x305:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:10:32 (1785078632) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg336-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg336-server: Pool lustre.test_204f created Adding targets to pool oleg336-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg336-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.188675 s, 11.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x306:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x306:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg336-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg336-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg336-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:11:52 (1785078712) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:11:59 (1785078719) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:12:07 (1785078727) oleg336-server: Pool lustre.test_206 created Adding targets to pool oleg336-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg336-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg336-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg336-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg336-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:12:36 (1785078756) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980071 s, 10.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644152 s, 16.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xefadf2ad Mirror 3: 0xefadf2ad PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:12:47 (1785078767) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:12:50 (1785078770) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:12:54 (1785078774) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0910401 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1890000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1050000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1050000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1050000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2120000 Read this (4 bytes): BBBB 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:13:38 (1785078818) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0425914 s, 192 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:13:46 (1785078826) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09469 s, 11.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1c1 lmm_fid: [0x200000405:0x1c1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1c1 lmm_fid: [0x200000405:0x1c1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x311:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611585 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-26 11:13:47.000000000 -0400 Modify: 2026-07-26 11:13:47.000000000 -0400 Change: 2026-07-26 11:13:49.000000000 -0400 Birth: 2026-07-26 11:13:48.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:13:54 (1785078834) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784761 s, 13.4 MB/s oleg336-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg336-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg336-server: O/280000401/d16/310: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:14:16 (1785078856) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.234625 s, 175 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0294832 s, 139 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 26 11:14 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c6:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 26 11:14 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:14:24 (1785078864) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:14:32 (1785078872) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1888 sec =========== 11:14:41 (1785078881) === sanity-flr: start cleanup 11:14:43 (1785078883) === === sanity-flr: finish cleanup 11:14:52 (1785078892) ===