-----============= acceptance-small: sanity-flr ============----- Fri Sep 4 11:26:48 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg656-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 11:27:05 (1788535625) === oleg656-client.virtnet: executing check_config_client /mnt/lustre oleg656-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg656-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9934d1590000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9934d1590000.idle_timeout=debug disable quota as required oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 11:27:46 (1788535666) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8242] == sanity-flr test 0a: lfs mirror create with -N option == 11:27:49 (1788535669) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:28:08 (1788535688) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:28:12 (1788535692) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:28:16 (1788535696) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:28:46 (1788535726) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:28:50 (1788535730) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 11:28:53 (1788535733) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9934d1590000.stats=clear osc.lustre-OST0001-osc-ffff9934d1590000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:29:02 (1788535742) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:29:15 (1788535755) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:29:25 (1788535765) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 11:29:29 (1788535769) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:29:38 (1788535778) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 11:29:48 (1788535788) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:29:55 (1788535795) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:30:03 (1788535803) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:30:13 (1788535813) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:30:24 (1788535824) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:30:32 (1788535832) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.623238 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165836 s, 6.3 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:30:42 (1788535842) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.466415 s, 38.2 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 11:30:52 (1788535852) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:31:03 (1788535863) Stopping /mnt/lustre-ost1 (opts:) on oleg656-server oleg656-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9934d1590000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9934d1590000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg656-server: oleg656-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg656-server oleg656-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9934d1590000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9934d1590000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg656-server: oleg656-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg656-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (135s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:33:18 (1788535998) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.37204 s, 16.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg656-server oleg656-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9934d1590000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9934d1590000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg656-server: oleg656-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg656-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (38s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:33:57 (1788536037) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg656-server oleg656-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9934d1590000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9934d1590000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg656-server: oleg656-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg656-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg656-server oleg656-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9934d1590000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9934d1590000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg656-server: oleg656-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg656-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (76s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:35:13 (1788536113) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.72855 s, 19.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 4 11:35 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.755s user 0m0.001s sys 0m0.194s Stopping /mnt/lustre-ost1 (opts:) on oleg656-server oleg656-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9934d1590000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9934d1590000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.114s user 0m0.001s sys 0m0.191s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg656-server: oleg656-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg656-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg656-server oleg656-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9934d1590000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9934d1590000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.222s user 0m0.000s sys 0m0.209s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg656-server: oleg656-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg656-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg656-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9934d1590000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9934d1590000.ost_server_uuid in FULL state after 0 sec PASS 33b (91s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:36:44 (1788536204) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:36:47 (1788536207) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:36:51 (1788536211) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 11:36:54 (1788536214) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 11:37:03 (1788536223) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.476606 s, 26.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:37:17 (1788536237) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582222 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312748 s, 33.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:37:25 (1788536245) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:37:33 (1788536253) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000835332 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105942 s, 967 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000907716 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000765512 s, 1.3 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 11:37:43 (1788536263) create test files with size 15730280 .. 89042d5d68fdd0c6e5dba5359971adfe - 89feb10a5f00d55cbb99860575633b75 - 70b896ab7b888490ddb09ceeb60381bd - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg656-client.virtnet /mnt/lustre (opts:) Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 11:38:07 (1788536287) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:38:23 (1788536303) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:38:30 (1788536310) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113522 s, 9.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xab:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418774 s, 25.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xac:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xad:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 11:38:40 (1788536320) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108534 s, 19.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.347018 s, 12.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170753 s, 12.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 11:38:55 (1788536335) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 11:38:59 (1788536339) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:39:02 (1788536342) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457763 s, 22.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 11:39:11 (1788536351) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.280903 s, 11.2 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:39:23 (1788536363) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:39:35 (1788536375) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.287357 s, 36.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg656-client.virtnet /mnt/lustre (opts:) Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:39:48 (1788536388) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9934c893b800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.376028 s, 27.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg656-client.virtnet /mnt/lustre (opts:) Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9934d0f51800.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:40:06 (1788536406) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:40:52 (1788536452) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:41:28 (1788536488) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 11:41:30 (1788536490) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.223698 s, 14.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.089893 s, 35.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xc984530f (nosync,stale) Mirror 2: 0xc984530f (nosync,stale) ** verify mirror 2 doesn't change original checksum: d3a6ee2fa3cec586fdb71ee7efba8277 - mirror 1 checksum: 2312ae89f46735151e69ded5262b8f59 - mirror 2 checksum: d3a6ee2fa3cec586fdb71ee7efba8277 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T11:41:34-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T11:41:34-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d3a6ee2fa3cec586fdb71ee7efba8277 - mirror 1 checksum: 2312ae89f46735151e69ded5262b8f59 - mirror 2 checksum: 2312ae89f46735151e69ded5262b8f59 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T11:41:34-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T11:41:34-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 11:41:40 (1788536500) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413024 s, 25.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:41:47 (1788536507) Failing mds1 on oleg656-server Stopping /mnt/lustre-mds1 (opts:) on oleg656-server 11:42:00 (1788536520) shut down facet: mds1 facet_host: oleg656-server facet_failover_host: oleg656-server Failover mds1 to oleg656-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-MDT0000 11:42:24 (1788536544) targets are mounted 11:42:24 (1788536544) facet_failover done oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386664 s, 27.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } PASS 50A (54s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:42:41 (1788536561) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0283365 s, 723 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0144455 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:42:51 (1788536571) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0502507 s, 408 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0117425 s, 1.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:43:00 (1788536580) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00886978 s, 1.8 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:43:08 (1788536588) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0376983 s, 543 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:43:15 (1788536595) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00818901 s, 500 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0659761 s, 62.1 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:43:23 (1788536603) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:43:46 (1788536626) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:44:16 (1788536656) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3e:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:44:42 (1788536682) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:45:06 (1788536706) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 11:45:14 (1788536714) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57599 mirror split pid 57600 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7845: 57599 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7845: 57600 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 11:46:21 (1788536781) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 6783 skipping zero length punch hole 000003 1788536783.830746 write 0x03525c thru 0x03ffff (0xada4 bytes) truncating to largest ever: 0x278d4 000004 1788536783.834001 trunc from 0x040000 to 0x0278d3 (0x1872c bytes) 000005 1788536783.854927 read 0x01e832 thru 0x025a61 (0x7230 bytes) 000006 1788536783.862423 mapread 0x00611c thru 0x007cc8 (0x1bad bytes) 000007 1788536783.881178 trunc from 0x003991 to 0x00db6b (0xa1db bytes) 000008 1788536783.888799 trunc from 0x0278d4 to 0x0039c5 (0x23f0e bytes) 000009 1788536783.912416 write 0x027ec9 thru 0x02ac64 (0x2d9c bytes) 000010 1788536783.922508 MIRROR_EXTEND to 2 mirrors 000011 1788536784.800683 punch from 0x01c351 to 0x021648 (0x52f8 bytes) 000012 1788536784.841346 punch from 0x00af47 to 0x010134 (0x51ee bytes) 000013 1788536784.853692 MIRROR_RESYNC 2 mirrors 000014 1788536785.610514 MIRROR_EXTEND to 3 mirrors 000015 1788536786.192027 mapread 0x0106c2 thru 0x01a455 (0x9d94 bytes) 000016 1788536786.224470 MIRROR_RESYNC 3 mirrors 000017 1788536786.711966 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000020 1788536787.292431 punch from 0x009136 to 0x0137a4 (0xa66f bytes) 000021 1788536787.397935 punch from 0x019616 to 0x026669 (0xd054 bytes) 000022 1788536787.409082 read 0x00b9cd thru 0x00bfcc (0x600 bytes) 000023 1788536787.419375 mapwrite 0x03c241 thru 0x03ffff (0x3dbf bytes) 000024 1788536787.578704 punch from 0x03c43e to 0x03ffff (0x3bc2 bytes) fallocating to largest ever: 0x2a6c6 000025 1788536787.594527 fallocate 0x01a7c8 thru 0x02a6c5 (0xfefe bytes) zero_range to largest ever: 0x3b860 000026 1788536787.616910 trunc from 0x0350a9 to 0x03b85f (0x67b7 bytes) 000027 1788536787.621691 trunc from 0x040000 to 0x006c15 (0x393ea bytes) 000028 1788536787.653409 trunc from 0x004f78 to 0x006c15 (0x1c9e bytes) 000029 1788536787.661425 trunc from 0x000efc to 0x004edd (0x3fe2 bytes) 000030 1788536787.670235 trunc from 0x001a4c to 0x006c15 (0x51ca bytes) 000031 1788536787.687244 punch from 0x001e49 to 0x003dba (0x1f72 bytes) 000032 1788536787.700621 mapwrite 0x02cc7e thru 0x038b95 (0xbf18 bytes) 000033 1788536787.831544 mapwrite 0x03fb36 thru 0x03ffff (0x4ca bytes) 000034 1788536787.900303 punch from 0x03053a to 0x031a9f (0x1566 bytes) 000035 1788536787.905546 mapwrite 0x016caa thru 0x01ba02 (0x4d59 bytes) 000037 1788536787.932040 mapread 0x03d5ae thru 0x03d99b (0x3ee bytes) 000038 1788536787.938685 MIRROR_RESYNC 2 mirrors 000039 1788536788.927561 write 0x0249f4 thru 0x033b56 (0xf163 bytes) 000040 1788536789.001913 mapread 0x011720 thru 0x016fea (0x58cb bytes) 000041 1788536789.034314 fallocate 0x027d40 thru 0x033f1b (0xc1dc bytes) 000042 1788536789.040245 trunc from 0x02686f to 0x02771e (0xeb0 bytes) 000043 1788536789.100097 punch from 0x00cda5 to 0x00fa19 (0x2c75 bytes) 000044 1788536789.112347 trunc from 0x040000 to 0x02d44a (0x12bb5 bytes) 000045 1788536789.156444 trunc from 0x009d77 to 0x013cd0 (0x9f5a bytes) 000046 1788536789.164324 fallocate 0x008106 thru 0x00dd83 (0x5c7e bytes) 000047 1788536789.172114 punch from 0x02ab11 to 0x02d44a (0x293a bytes) 000048 1788536789.180093 mapread 0x01ba28 thru 0x022b74 (0x714d bytes) 000049 1788536789.198275 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d44b): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d44b): Mirror 3: 0xca38c765 (stale) CRC-32 checksum value for chunk [0, 0x2d44b): Mirror 1: 0x1dd6e228 Mirror 4: 0x1dd6e228 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d44b: skipped 000050 1788536790.622873 trunc from 0x02d44b to 0x00ae99 (0x225b1 bytes) 000051 1788536790.696005 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xae9a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xae9a): Mirror 3: 0x77da1604 (stale) Mirror 4: 0x77da1604 (stale) CRC-32 checksum value for chunk [0, 0xae9a): Mirror 1: 0x77da1604 Mirror 5: 0x77da1604 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xae9a: skipped 000052 1788536791.406492 fallocate 0x01c4b5 thru 0x02ae06 (0xe952 bytes) 000053 1788536791.448879 read 0x00f8a0 thru 0x01152f (0x1c90 bytes) 000054 1788536791.455589 trunc from 0x02ae07 to 0x020183 (0xac83 bytes) 000055 1788536791.477492 write 0x007b78 thru 0x014688 (0xcb11 bytes) 000056 1788536791.491002 punch from 0x0011fb to 0x00d6ab (0xc4b1 bytes) 000057 1788536791.516839 write 0x012532 thru 0x01d716 (0xb1e5 bytes) 000059 1788536791.526347 fallocate 0x00d545 thru 0x0154ae (0x7f6a bytes) 000061 1788536791.533081 trunc from 0x020184 to 0x00adfb (0x15388 bytes) 000062 1788536791.554335 mapwrite 0x01cca7 thru 0x021798 (0x4af2 bytes) 000063 1788536791.682382 trunc from 0x010364 to 0x01cbf9 (0xc896 bytes) fallocating to largest ever: 0x3dcf2 000064 1788536791.689316 fallocate 0x03b614 thru 0x03dcf1 (0x26de bytes) 000065 1788536791.719320 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x21799): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21799): Mirror 3: 0x8f98801c (stale) Mirror 4: 0x8f98801c (stale) Mirror 5: 0xd82ce36a (stale) CRC-32 checksum value for chunk [0, 0x21799): Mirror 1: 0x294a46fa Mirror 6: 0x294a46fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21799: skipped 000066 1788536793.459370 mapwrite 0x034fe4 thru 0x03dbc6 (0x8be3 bytes) 000067 1788536793.561319 fallocate 0x01ebbe thru 0x024559 (0x599c bytes) 000068 1788536793.582791 mapread 0x03d15d thru 0x03dbc6 (0xa6a bytes) 000069 1788536793.586595 trunc from 0x0259db to 0x02f871 (0x9e97 bytes) 000070 1788536793.599211 read 0x004dc6 thru 0x01210a (0xd345 bytes) 000071 1788536793.609591 read 0x024fbf thru 0x03223f (0xd281 bytes) 000072 1788536793.623756 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dbc7): Mirror 3: 0x4ae1549b (stale) Mirror 4: 0x715fecac (stale) Mirror 5: 0x78f3934c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dbc7): Mirror 3: 0x4ae1549b (stale) Mirror 4: 0x715fecac (stale) Mirror 5: 0x78f3934c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dbc7: skipped 000073 1788536794.066602 trunc from 0x030252 to 0x03c19a (0xbf49 bytes) 000074 1788536794.073465 trunc from 0x01785f to 0x025907 (0xe0a9 bytes) 000075 1788536794.077721 trunc from 0x03dbc7 to 0x03b090 (0x2b36 bytes) 000078 1788536794.112305 mapwrite 0x01a6e7 thru 0x022ca7 (0x85c1 bytes) 000079 1788536794.159802 mapread 0x033f93 thru 0x03557d (0x15eb bytes) 000080 1788536794.170939 punch from 0x01d7a9 to 0x02c28c (0xeae4 bytes) 000081 1788536794.179085 fallocate 0x006c8f thru 0x006dc2 (0x134 bytes) 000082 1788536794.187566 MIRROR_RESYNC 4 mirrors 000083 1788536795.461909 read 0x003174 thru 0x0070bb (0x3f48 bytes) 000084 1788536795.493069 trunc from 0x006535 to 0x0079e3 (0x14af bytes) 000085 1788536795.571834 fallocate 0x00afd0 thru 0x00f66d (0x469e bytes) 000086 1788536795.584103 mapwrite 0x03117f thru 0x039648 (0x84ca bytes) 000087 1788536795.695235 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b091): Mirror 4: 0x3b5e19cd (stale) Mirror 5: 0x3b5e19cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b091): Mirror 4: 0x3b5e19cd (stale) Mirror 5: 0x3b5e19cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b091: skipped 000088 1788536796.226478 trunc from 0x026c73 to 0x035f75 (0xf303 bytes) 000089 1788536796.233388 trunc from 0x037017 to 0x03b090 (0x407a bytes) 000090 1788536796.240160 trunc from 0x00a6d8 to 0x011f82 (0x78ab bytes) 000091 1788536796.258329 write 0x0110df thru 0x020c1e (0xfb40 bytes) 000092 1788536796.285694 write 0x035e27 thru 0x03b584 (0x575e bytes) 000093 1788536796.298500 fallocate 0x00c2bc thru 0x01298c (0x66d1 bytes) 000094 1788536796.312214 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b585): Mirror 5: 0xff04a0ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b585): Mirror 5: 0xff04a0ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b585: skipped 000095 1788536796.782688 write 0x0203a4 thru 0x0286d2 (0x832f bytes) 000096 1788536796.804411 mapread 0x00ddfe thru 0x013762 (0x5965 bytes) 000097 1788536796.853326 MIRROR_RESYNC 2 mirrors 000098 1788536797.653700 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000099 1788536797.851239 read 0x01d582 thru 0x028c9f (0xb71e bytes) 000100 1788536797.878488 trunc from 0x03b585 to 0x035b17 (0x5a6d bytes) 000102 1788536797.940310 punch from 0x00361f to 0x0076eb (0x40cd bytes) 000103 1788536797.946706 trunc from 0x014542 to 0x01bab7 (0x7576 bytes) 000104 1788536797.954962 fallocate 0x022e1d thru 0x025b42 (0x2d26 bytes) 000105 1788536797.961325 punch from 0x0129b6 to 0x015ca8 (0x32f3 bytes) 000106 1788536797.969465 trunc from 0x035b18 to 0x03bd68 (0x6251 bytes) 000107 1788536797.993475 punch from 0x0045d5 to 0x010379 (0xbda5 bytes) 000108 1788536798.001064 write 0x01e3e4 thru 0x027643 (0x9260 bytes) 000109 1788536798.024964 mapwrite 0x03fa8c thru 0x03ffff (0x574 bytes) 000111 1788536798.124600 read 0x019014 thru 0x01f140 (0x612d bytes) 000113 1788536798.147342 read 0x023280 thru 0x030920 (0xd6a1 bytes) 000114 1788536798.148020 trunc from 0x040000 to 0x030c4f (0xf3b0 bytes) 000115 1788536798.179042 write 0x01c555 thru 0x0201d2 (0x3c7e bytes) 000116 1788536798.180282 write 0x03b1d8 thru 0x03ffff (0x4e28 bytes) 000117 1788536798.185018 write 0x0063e7 thru 0x008a03 (0x261d bytes) 000120 1788536798.190270 write 0x00daeb thru 0x00e074 (0x58a bytes) zero_range to largest ever: 0x40000 000121 1788536798.215297 trunc from 0x03a817 to 0x03ffff (0x57e9 bytes) 000123 1788536798.250176 mapwrite 0x038462 thru 0x038543 (0xe2 bytes) 000124 1788536798.316618 read 0x0354e0 thru 0x03b9c3 (0x64e4 bytes) 000125 1788536798.317289 punch from 0x02297a to 0x02c176 (0x97fd bytes) 000126 1788536798.332691 mapread 0x039d7f thru 0x03ffff (0x6281 bytes) 000128 1788536798.334007 mapwrite 0x0309b6 thru 0x03a01e (0x9669 bytes) 000129 1788536798.410368 mapwrite 0x02900a thru 0x0303c3 (0x73ba bytes) 000130 1788536798.478318 read 0x027fe1 thru 0x029c1f (0x1c3f bytes) 000131 1788536798.480037 punch from 0x026e14 to 0x034aab (0xdc98 bytes) 000133 1788536798.497411 punch from 0x0059e2 to 0x014b08 (0xf127 bytes) 000134 1788536798.510631 punch from 0x01d5aa to 0x02ad53 (0xd7aa bytes) 000135 1788536798.532443 trunc from 0x03a054 to 0x03b30b (0x12b8 bytes) 000136 1788536798.558785 MIRROR_EXTEND to 2 mirrors 000137 1788536799.663270 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000138 1788536799.902267 fallocate 0x00be94 thru 0x0121eb (0x6358 bytes) 000139 1788536799.927482 read 0x019d08 thru 0x025f70 (0xc269 bytes) 000141 1788536799.951984 punch from 0x03646f to 0x03ffff (0x9b91 bytes) 000142 1788536799.963858 write 0x00acfb thru 0x017dc8 (0xd0ce bytes) 000143 1788536800.010308 mapwrite 0x019403 thru 0x026da4 (0xd9a2 bytes) 000144 1788536800.083265 write 0x008c56 thru 0x00e5ab (0x5956 bytes) 000145 1788536800.094049 fallocate 0x01ed50 thru 0x02b133 (0xc3e4 bytes) 000146 1788536800.109437 punch from 0x02f14b to 0x03da36 (0xe8ec bytes) 000147 1788536800.125796 mapwrite 0x020e00 thru 0x021609 (0x80a bytes) 000148 1788536800.163188 read 0x027a2a thru 0x02fb65 (0x813c bytes) 000150 1788536800.200024 write 0x01c3ee thru 0x027eba (0xbacd bytes) 000151 1788536800.215006 trunc from 0x022fbb to 0x032b70 (0xfbb6 bytes) 000152 1788536800.266867 fallocate 0x02dd33 thru 0x031083 (0x3351 bytes) 000153 1788536800.274657 mapwrite 0x000327 thru 0x003dc9 (0x3aa3 bytes) 000154 1788536800.323090 trunc from 0x040000 to 0x001d64 (0x3e29b bytes) 000155 1788536800.369417 read 0x001a49 thru 0x001d64 (0x31c bytes) 000156 1788536800.372612 trunc from 0x000576 to 0x001d64 (0x17ef bytes) 000158 1788536800.391956 trunc from 0x000185 to 0x001d64 (0x1be0 bytes) 000159 1788536800.402933 mapread 0x000537 thru 0x001d64 (0x182e bytes) 000160 1788536800.426172 trunc from 0x000ee8 to 0x001d64 (0xe7d bytes) 000161 1788536800.441910 read 0x0006d1 thru 0x001d64 (0x1694 bytes) 000162 1788536800.455286 write 0x00a990 thru 0x016c69 (0xc2da bytes) 000163 1788536800.477793 MIRROR_EXTEND to 2 mirrors 000164 1788536801.821452 write 0x021fcf thru 0x030509 (0xe53b bytes) 000165 1788536801.959298 read 0x015d43 thru 0x024a8d (0xed4b bytes) 000166 1788536801.975244 fallocate 0x030b11 thru 0x038dd2 (0x82c2 bytes) 000167 1788536801.987278 trunc from 0x038dd3 to 0x00b522 (0x2d8b0 bytes) 000168 1788536802.038645 read 0x002b52 thru 0x00b522 (0x89d1 bytes) 000169 1788536802.056922 trunc from 0x00597d to 0x00b522 (0x5ba6 bytes) 000170 1788536802.072552 fallocate 0x028153 thru 0x02b16c (0x301a bytes) 000171 1788536802.096745 trunc from 0x00b523 to 0x006e4d (0x46d5 bytes) 000172 1788536802.147117 trunc from 0x006e4e to 0x02b572 (0x24725 bytes) 000173 1788536802.174497 mapwrite 0x03622a thru 0x03c48c (0x6263 bytes) 000174 1788536802.300980 fallocate 0x03ea12 thru 0x03ffff (0x15ee bytes) 000175 1788536802.315721 trunc from 0x006013 to 0x00c138 (0x6126 bytes) 000177 1788536802.341848 trunc from 0x03c48d to 0x038695 (0x3df7 bytes) 000178 1788536802.413626 fallocate 0x035178 thru 0x03b498 (0x6321 bytes) 000179 1788536802.434547 write 0x00480e thru 0x00e444 (0x9c37 bytes) 000180 1788536802.441911 trunc from 0x02d653 to 0x038695 (0xb043 bytes) 000181 1788536802.454084 punch from 0x00a359 to 0x016913 (0xc5bb bytes) 000183 1788536802.501807 write 0x00f65f thru 0x01e6c7 (0xf069 bytes) 000184 1788536802.512108 trunc from 0x038696 to 0x032189 (0x650c bytes) 000185 1788536802.551917 punch from 0x015042 to 0x01a29c (0x525b bytes) 000187 1788536802.596703 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000188 1788536802.939251 MIRROR_EXTEND to 2 mirrors 000189 1788536804.453229 fallocate 0x021483 thru 0x02ec38 (0xd7b6 bytes) 000190 1788536804.483933 trunc from 0x016924 to 0x01e541 (0x7c1e bytes) 000191 1788536804.502044 mapread 0x02b170 thru 0x032189 (0x701a bytes) 000192 1788536804.548090 trunc from 0x03218a to 0x00e3ee (0x23d9b bytes) 000194 1788536804.582599 punch from 0x001860 to 0x00e177 (0xc918 bytes) 000195 1788536804.589920 write 0x011eeb thru 0x017baf (0x5cc5 bytes) 000196 1788536804.592109 punch from 0x00f943 to 0x012423 (0x2ae1 bytes) 000197 1788536804.610638 mapwrite 0x028d88 thru 0x030298 (0x7511 bytes) 000198 1788536804.706481 read 0x01dc86 thru 0x024819 (0x6b94 bytes) 000199 1788536804.717105 mapread 0x001882 thru 0x00518e (0x390d bytes) 000200 1788536804.742928 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30299): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30299): Mirror 2: 0x24d9fe3a (stale) CRC-32 checksum value for chunk [0, 0x30299): Mirror 1: 0x3b3022e2 Mirror 3: 0x3b3022e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30299: skipped 000201 1788536806.095764 trunc from 0x030299 to 0x01d238 (0x13060 bytes) 000202 1788536806.182921 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1d239): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d239): Mirror 2: 0xe8eae0da (stale) Mirror 3: 0xb62a2739 (stale) CRC-32 checksum value for chunk [0, 0x1d239): Mirror 1: 0xb62a2739 Mirror 4: 0xb62a2739 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d239: skipped 000203 1788536807.309390 write 0x00a069 thru 0x01700c (0xcfa4 bytes) 000204 1788536807.445992 mapread 0x010d22 thru 0x013579 (0x2858 bytes) 000205 1788536807.447224 mapwrite 0x021349 thru 0x02dd43 (0xc9fb bytes) 000206 1788536807.630399 MIRROR_RESYNC 4 mirrors 000207 1788536808.634201 punch from 0x0232d6 to 0x02c11e (0x8e49 bytes) 000209 1788536808.663523 mapread 0x004d7c thru 0x008059 (0x32de bytes) 000210 1788536808.678512 trunc from 0x00a3f3 to 0x01009b (0x5ca9 bytes) 000212 1788536808.685275 punch from 0x014697 to 0x01c2a5 (0x7c0f bytes) 000213 1788536808.690836 trunc from 0x0231a1 to 0x02dd43 (0xaba3 bytes) 000215 1788536808.699630 trunc from 0x026260 to 0x02989a (0x363b bytes) 000216 1788536808.703359 trunc from 0x00019e to 0x00a2b4 (0xa117 bytes) 000217 1788536808.708391 punch from 0x019c22 to 0x022a84 (0x8e63 bytes) 000218 1788536808.713256 read 0x02a898 thru 0x02dd43 (0x34ac bytes) 000219 1788536808.719396 fallocate 0x0226f9 thru 0x024592 (0x1e9a bytes) 000220 1788536808.725176 fallocate 0x01da62 thru 0x02aba8 (0xd147 bytes) 000221 1788536808.729769 fallocate 0x022896 thru 0x02a8a8 (0x8013 bytes) 000222 1788536808.734115 trunc from 0x013e47 to 0x01fc0d (0xbdc7 bytes) 000223 1788536808.739358 fallocate 0x00aa94 thru 0x00e5ea (0x3b57 bytes) 000225 1788536808.746054 write 0x01a285 thru 0x021521 (0x729d bytes) 000226 1788536808.757375 trunc from 0x00d099 to 0x016a88 (0x99f0 bytes) 000227 1788536808.762155 fallocate 0x03a171 thru 0x03ffff (0x5e8f bytes) 000228 1788536808.766755 mapwrite 0x017e6f thru 0x02433a (0xc4cc bytes) 000229 1788536808.808813 trunc from 0x040000 to 0x03d262 (0x2d9d bytes) 000230 1788536808.845619 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d263): Mirror 2: 0x9a9779ef (stale) Mirror 3: 0x9a9779ef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d263): Mirror 2: 0x9a9779ef (stale) Mirror 3: 0x9a9779ef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d263: skipped 000231 1788536809.224578 read 0x0298ce thru 0x02b991 (0x20c4 bytes) 000232 1788536809.236902 mapwrite 0x02e477 thru 0x030a69 (0x25f3 bytes) 000233 1788536809.275883 trunc from 0x025515 to 0x02e5fd (0x90e9 bytes) 000234 1788536809.305001 write 0x0283cb thru 0x0372f7 (0xef2d bytes) 000235 1788536809.324798 mapread 0x01d87b thru 0x02cba1 (0xf327 bytes) 000236 1788536809.451464 mapwrite 0x0385ac thru 0x03ffff (0x7a54 bytes) 000237 1788536809.562017 fallocate 0x0140fe thru 0x014c48 (0xb4b bytes) 000238 1788536809.569965 punch from 0x02a828 to 0x02e30c (0x3ae5 bytes) 000240 1788536809.580036 fallocate 0x00d192 thru 0x019395 (0xc204 bytes) 000241 1788536809.587287 mapwrite 0x02986d thru 0x03949e (0xfc32 bytes) 000242 1788536809.714009 MIRROR_RESYNC 3 mirrors 000243 1788536810.812488 write 0x003669 thru 0x009dca (0x6762 bytes) 000244 1788536810.850567 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd59de656 (stale) Mirror 3: 0xd59de656 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5917ed21 Mirror 4: 0x5917ed21 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000245 1788536811.944847 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd59de656 (stale) Mirror 3: 0xd59de656 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5917ed21 Mirror 4: 0x5917ed21 Mirror 5: 0x5917ed21 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000246 1788536813.150762 mapread 0x017ba4 thru 0x021ac5 (0x9f22 bytes) 000247 1788536813.183627 punch from 0x001276 to 0x00cd8e (0xbb19 bytes) 000249 1788536813.275277 fallocate 0x0073bb thru 0x00e711 (0x7357 bytes) 000250 1788536813.288216 read 0x039c6a thru 0x03ffff (0x6396 bytes) 000251 1788536813.325817 trunc from 0x040000 to 0x02659b (0x19a64 bytes) 000252 1788536813.385606 punch from 0x008828 to 0x00b179 (0x2952 bytes) 000253 1788536813.396063 fallocate 0x004527 thru 0x011b9c (0xd676 bytes) 000254 1788536813.414388 mapwrite 0x0323bd thru 0x037fd7 (0x5c1b bytes) 000255 1788536813.536156 fallocate 0x035fad thru 0x03f357 (0x93ab bytes) 000256 1788536813.546352 mapread 0x01c470 thru 0x029678 (0xd209 bytes) 000257 1788536813.600459 trunc from 0x037fd8 to 0x020d30 (0x172a7 bytes) 000258 1788536813.628261 read 0x0163f1 thru 0x020d30 (0xa940 bytes) 000259 1788536813.639802 MIRROR_RESYNC 5 mirrors 000260 1788536814.746184 MIRROR_EXTEND to 6 mirrors 000261 1788536815.438936 punch from 0x0079ec to 0x015233 (0xd848 bytes) 000262 1788536815.510645 fallocate 0x03007e thru 0x033848 (0x37cb bytes) 000263 1788536815.516102 read 0x00096b thru 0x00b6c7 (0xad5d bytes) 000264 1788536815.526733 write 0x00d01f thru 0x010c03 (0x3be5 bytes) 000265 1788536815.544857 trunc from 0x00b615 to 0x012c26 (0x7612 bytes) 000266 1788536815.560539 mapwrite 0x02c9fb thru 0x034b59 (0x815f bytes) 000268 1788536815.653079 mapread 0x002e38 thru 0x00dd93 (0xaf5c bytes) 000270 1788536815.673966 read 0x0147f8 thru 0x016739 (0x1f42 bytes) 000271 1788536815.692191 mapread 0x024997 thru 0x02e1cd (0x9837 bytes) 000272 1788536815.698314 MIRROR_RESYNC 6 mirrors 000273 1788536817.342228 mapread 0x00c421 thru 0x00cc90 (0x870 bytes) 000274 1788536817.346855 trunc from 0x034b5a to 0x026130 (0xea29 bytes) 000275 1788536817.394787 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x26131): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26131): Mirror 2: 0x4e511160 (stale) Mirror 3: 0x4e511160 (stale) Mirror 4: 0x4e511160 (stale) Mirror 5: 0x4e511160 (stale) Mirror 6: 0x4e511160 (stale) CRC-32 checksum value for chunk [0, 0x26131): Mirror 1: 0x4e511160 Mirror 7: 0x4e511160 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26131: skipped 000276 1788536818.383709 punch from 0x0207f4 to 0x026130 (0x593d bytes) 000277 1788536818.450157 trunc from 0x006ae5 to 0x00d5b3 (0x6acf bytes) 000278 1788536818.461093 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26131): Mirror 2: 0x4e511160 (stale) Mirror 4: 0x4e511160 (stale) Mirror 5: 0x4e511160 (stale) Mirror 6: 0x4e511160 (stale) Mirror 7: 0x4e511160 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26131): Mirror 2: 0x4e511160 (stale) Mirror 4: 0x4e511160 (stale) Mirror 5: 0x4e511160 (stale) Mirror 6: 0x4e511160 (stale) Mirror 7: 0x4e511160 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26131: skipped 000279 1788536818.979721 write 0x01fe25 thru 0x02568e (0x586a bytes) 000280 1788536819.027993 punch from 0x0079a2 to 0x0170b8 (0xf717 bytes) 000281 1788536819.044937 MIRROR_RESYNC 6 mirrors 000282 1788536820.551633 read 0x002d4b thru 0x0088dd (0x5b93 bytes) 000283 1788536820.570314 trunc from 0x026131 to 0x013394 (0x12d9c bytes) 000284 1788536820.640996 trunc from 0x013395 to 0x01cc77 (0x98e3 bytes) 000285 1788536820.665262 mapwrite 0x030d19 thru 0x03581c (0x4b04 bytes) 000286 1788536820.750434 mapread 0x00b9b8 thru 0x016efc (0xb545 bytes) 000287 1788536820.780223 trunc from 0x03581d to 0x009874 (0x2bfa8 bytes) 000288 1788536820.802889 fallocate 0x0153b7 thru 0x01c79e (0x73e8 bytes) 000289 1788536820.808330 write 0x0192d4 thru 0x02061d (0x734a bytes) 000290 1788536820.832989 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2061e): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2061e): Mirror 2: 0x918aea55 (stale) Mirror 4: 0x918aea55 (stale) Mirror 5: 0x918aea55 (stale) Mirror 6: 0x918aea55 (stale) Mirror 7: 0x918aea55 (stale) CRC-32 checksum value for chunk [0, 0x2061e): Mirror 1: 0x997bcd51 Mirror 8: 0x997bcd51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2061e: skipped 000291 1788536821.939948 punch from 0x009dd0 to 0x018933 (0xeb64 bytes) 000292 1788536821.991668 mapwrite 0x031c8a thru 0x03bbf2 (0x9f69 bytes) 000293 1788536822.066108 read 0x00fd66 thru 0x019f0f (0xa1aa bytes) 000294 1788536822.076992 mapread 0x00f94a thru 0x01f0b5 (0xf76c bytes) 000295 1788536822.086973 write 0x0058d6 thru 0x0131c3 (0xd8ee bytes) 000296 1788536822.093101 read 0x004011 thru 0x010341 (0xc331 bytes) 000297 1788536822.102297 fallocate 0x02117b thru 0x02d492 (0xc318 bytes) 000298 1788536822.108023 write 0x021008 thru 0x02af1f (0x9f18 bytes) 000299 1788536822.109622 trunc from 0x03bbf3 to 0x03ce9e (0x12ac bytes) 000300 1788536822.130538 mapread 0x02f94f thru 0x031bec (0x229e bytes) 000301 1788536822.133002 mapwrite 0x00976a thru 0x00a717 (0xfae bytes) 000302 1788536822.166333 trunc from 0x02dea8 to 0x038e63 (0xafbc bytes) 000304 1788536822.174487 write 0x03fc18 thru 0x03fc1f (0x8 bytes) 000305 1788536822.176594 read 0x039a10 thru 0x03fc1f (0x6210 bytes) 000306 1788536822.188722 trunc from 0x03fc20 to 0x00b002 (0x34c1d bytes) 000307 1788536822.224711 read 0x007456 thru 0x00b002 (0x3bad bytes) 000308 1788536822.225464 trunc from 0x00b003 to 0x02bba2 (0x20ba0 bytes) 000309 1788536822.254241 read 0x00a822 thru 0x015d8b (0xb56a bytes) 000310 1788536822.275658 punch from 0x00114f to 0x009f36 (0x8de8 bytes) 000311 1788536822.283045 fallocate 0x03e4d6 thru 0x03ffff (0x1b2a bytes) 000312 1788536822.292331 mapwrite 0x007d00 thru 0x0093c4 (0x16c5 bytes) 000313 1788536822.337751 trunc from 0x00873e to 0x010b73 (0x8436 bytes) 000315 1788536822.356014 trunc from 0x02bba3 to 0x024580 (0x7622 bytes) 000316 1788536822.389930 punch from 0x01e9ec to 0x024580 (0x5b95 bytes) 000318 1788536822.404188 trunc from 0x02214e to 0x024580 (0x2433 bytes) 000319 1788536822.412184 trunc from 0x015b2f to 0x01e13a (0x860c bytes) 000320 1788536822.433401 MIRROR_RESYNC 7 mirrors 000321 1788536824.738096 fallocate 0x00b550 thru 0x0117e3 (0x6294 bytes) 000323 1788536824.825108 mapread 0x019b44 thru 0x01efbc (0x5479 bytes) 000324 1788536824.847259 fallocate 0x019200 thru 0x020674 (0x7475 bytes) 000325 1788536824.855481 mapwrite 0x02db94 thru 0x02dc68 (0xd5 bytes) 000326 1788536824.939779 trunc from 0x02dc69 to 0x006b0a (0x2715e bytes) 000327 1788536824.958838 punch from 0x00044f to 0x006b0a (0x66bc bytes) 000328 1788536824.966363 fallocate 0x015623 thru 0x015db3 (0x791 bytes) 000329 1788536824.980608 mapwrite 0x0327cd thru 0x03ffff (0xd833 bytes) 000330 1788536825.086232 mapread 0x021f08 thru 0x026082 (0x417b bytes) 000331 1788536825.112151 fallocate 0x028430 thru 0x036213 (0xdde4 bytes) 000333 1788536825.130491 trunc from 0x040000 to 0x01fe9c (0x20163 bytes) 000334 1788536825.179177 MIRROR_RESYNC 7 mirrors 000335 1788536826.547607 trunc from 0x01fe9d to 0x01e490 (0x1a0c bytes) 000336 1788536826.584444 mapwrite 0x022e0e thru 0x031d59 (0xef4c bytes) 000337 1788536826.651852 read 0x0122c5 thru 0x01ffc4 (0xdd00 bytes) 000338 1788536826.665470 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31d5a): Mirror 2: 0x5f34e67d (stale) Mirror 4: 0x5f34e67d (stale) Mirror 6: 0x5f34e67d (stale) Mirror 7: 0x5f34e67d (stale) Mirror 8: 0x5f34e67d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d5a): Mirror 2: 0x5f34e67d (stale) Mirror 4: 0x5f34e67d (stale) Mirror 6: 0x5f34e67d (stale) Mirror 7: 0x5f34e67d (stale) Mirror 8: 0x5f34e67d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d5a: skipped 000339 1788536827.153279 mapwrite 0x03b02a thru 0x03c03f (0x1016 bytes) 000341 1788536827.213960 mapread 0x015e21 thru 0x02531b (0xf4fb bytes) 000342 1788536827.247040 write 0x021879 thru 0x02b9a4 (0xa12c bytes) 000344 1788536827.254569 write 0x03aae7 thru 0x03ad79 (0x293 bytes) 000345 1788536827.261218 write 0x02839b thru 0x032a57 (0xa6bd bytes) 000346 1788536827.269542 fallocate 0x03edd6 thru 0x03ffff (0x122a bytes) 000347 1788536827.274478 write 0x002f5f thru 0x010b47 (0xdbe9 bytes) 000348 1788536827.281068 trunc from 0x03c040 to 0x0049c1 (0x3767e bytes) 000349 1788536827.297773 read 0x00166b thru 0x0049c1 (0x3357 bytes) 000350 1788536827.304217 punch from 0x003978 to 0x0049c1 (0x104a bytes) 000351 1788536827.314050 write 0x03d707 thru 0x03e4df (0xdd9 bytes) 000352 1788536827.315375 write 0x034e14 thru 0x0389af (0x3b9c bytes) 000353 1788536827.324813 write 0x003326 thru 0x012ff1 (0xfccc bytes) 000354 1788536827.334179 mapread 0x001eee thru 0x009631 (0x7744 bytes) 000355 1788536827.335775 fallocate 0x03a5a8 thru 0x03ffff (0x5a58 bytes) 000356 1788536827.340661 fallocate 0x01ebc5 thru 0x02a68f (0xbacb bytes) 000357 1788536827.345585 mapwrite 0x02fd16 thru 0x03e3e5 (0xe6d0 bytes) 000358 1788536827.401374 read 0x002977 thru 0x011751 (0xeddb bytes) 000359 1788536827.402102 trunc from 0x03f60b to 0x03ffff (0x9f5 bytes) 000360 1788536827.407885 mapread 0x01ec4c thru 0x0288f3 (0x9ca8 bytes) 000361 1788536827.419257 trunc from 0x00b7ca to 0x011d62 (0x6599 bytes) 000363 1788536827.425596 trunc from 0x0197f6 to 0x0272c2 (0xdacd bytes) 000364 1788536827.433478 punch from 0x014de7 to 0x0157ad (0x9c7 bytes) 000365 1788536827.440892 fallocate 0x03a7a9 thru 0x03ffff (0x5857 bytes) 000366 1788536827.446833 read 0x02f049 thru 0x034881 (0x5839 bytes) 000367 1788536827.448229 write 0x012150 thru 0x017094 (0x4f45 bytes) 000368 1788536827.450934 mapread 0x00655b thru 0x00ee99 (0x893f bytes) 000369 1788536827.467760 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000370 1788536828.059735 read 0x030ea9 thru 0x03ffff (0xf157 bytes) 000371 1788536828.074445 write 0x01196e thru 0x012430 (0xac3 bytes) 000373 1788536828.098066 trunc from 0x004b7c to 0x00eeb8 (0xa33d bytes) 000374 1788536828.104406 read 0x03b75f thru 0x03ffff (0x48a1 bytes) 000376 1788536828.105421 fallocate 0x01dfba thru 0x02896d (0xa9b4 bytes) 000377 1788536828.110081 mapread 0x02f651 thru 0x03d59e (0xdf4e bytes) 000379 1788536828.147011 trunc from 0x040000 to 0x0346d3 (0xb92c bytes) 000380 1788536828.176945 fallocate 0x00a961 thru 0x00f4ea (0x4b8a bytes) 000381 1788536828.185650 read 0x01c4eb thru 0x029c90 (0xd7a6 bytes) 000382 1788536828.187673 read 0x004d9d thru 0x012169 (0xd3cd bytes) 000383 1788536828.188878 write 0x0178d2 thru 0x01b060 (0x378f bytes) 000384 1788536828.192365 fallocate 0x01c5f1 thru 0x01e606 (0x2016 bytes) 000385 1788536828.202178 trunc from 0x0346d4 to 0x00419b (0x30538 bytes) 000386 1788536828.235552 trunc from 0x00342f to 0x00419b (0xd6d bytes) 000387 1788536828.249173 fallocate 0x016c0c thru 0x017116 (0x50b bytes) 000388 1788536828.261317 punch from 0x002266 to 0x00419b (0x1f36 bytes) 000389 1788536828.287215 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x419c): Mirror 6: 0x4e65f596 (stale) Mirror 7: 0x4e65f596 (stale) Mirror 8: 0x4e65f596 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x419c): Mirror 6: 0x4e65f596 (stale) Mirror 7: 0x4e65f596 (stale) Mirror 8: 0x4e65f596 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x419c: skipped 000390 1788536828.682578 MIRROR_RESYNC 4 mirrors 000391 1788536829.991157 fallocate 0x00cd14 thru 0x011b1e (0x4e0b bytes) 000392 1788536830.037180 trunc from 0x00ecb3 to 0x011b1e (0x2e6c bytes) 000394 1788536830.043059 punch from 0x011053 to 0x011b1e (0xacc bytes) 000395 1788536830.048848 mapwrite 0x02acf1 thru 0x038665 (0xd975 bytes) 000396 1788536830.140318 mapread 0x02f2fe thru 0x030673 (0x1376 bytes) 000397 1788536830.147227 punch from 0x033711 to 0x034638 (0xf28 bytes) 000398 1788536830.163341 trunc from 0x038666 to 0x022b11 (0x15b54 bytes) 000399 1788536830.204711 trunc from 0x022b12 to 0x01913c (0x99d5 bytes) 000400 1788536830.239564 trunc from 0x00ebaf to 0x01913c (0xa58e bytes) 000401 1788536830.244293 trunc from 0x01913d to 0x03e552 (0x25416 bytes) 000402 1788536830.287298 fallocate 0x01cade thru 0x0257bb (0x8cde bytes) 000404 1788536830.298364 mapread 0x0071dd thru 0x012322 (0xb146 bytes) 000406 1788536830.353969 write 0x022afb thru 0x023022 (0x528 bytes) 000407 1788536830.396361 trunc from 0x03e553 to 0x030a96 (0xdabc bytes) 000408 1788536830.438595 punch from 0x00ced7 to 0x01abfd (0xdd27 bytes) 000410 1788536830.447974 mapread 0x016643 thru 0x023ced (0xd6ab bytes) 000411 1788536830.480113 punch from 0x02974d to 0x03098d (0x7241 bytes) 000412 1788536830.487301 trunc from 0x00a9fe to 0x015d5d (0xb360 bytes) 000413 1788536830.494908 write 0x02a8a6 thru 0x03a244 (0xf99f bytes) 000414 1788536830.503212 mapread 0x02e95f thru 0x03a244 (0xb8e6 bytes) 000416 1788536830.511678 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a245): Mirror 7: 0x89d3c69 (stale) Mirror 8: 0x89d3c69 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a245): Mirror 7: 0x89d3c69 (stale) Mirror 8: 0x89d3c69 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a245: skipped 000417 1788536830.862407 punch from 0x033b7d to 0x039bd3 (0x6057 bytes) 000418 1788536830.866286 read 0x028e1d thru 0x02db21 (0x4d05 bytes) 000420 1788536830.888073 mapwrite 0x01abe3 thru 0x020ca7 (0x60c5 bytes) 000421 1788536830.973030 trunc from 0x03a245 to 0x03022c (0xa018 bytes) 000422 1788536831.025258 mapread 0x01cb38 thru 0x02741e (0xa8e7 bytes) 000423 1788536831.044665 punch from 0x00b91a to 0x00c1d9 (0x8c0 bytes) 000424 1788536831.049792 mapread 0x01d544 thru 0x0276cb (0xa188 bytes) 000425 1788536831.053741 trunc from 0x017355 to 0x0235de (0xc28a bytes) 000427 1788536831.067987 fallocate 0x023d9f thru 0x032579 (0xe7db bytes) 000429 1788536831.089515 fallocate 0x01783b thru 0x01c70e (0x4ed4 bytes) 000430 1788536831.099036 fallocate 0x019c70 thru 0x01a171 (0x502 bytes) 000431 1788536831.104458 mapwrite 0x02ba0c thru 0x036a8b (0xb080 bytes) 000432 1788536831.152762 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36a8c): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a8c): Mirror 7: 0x334ead92 (stale) Mirror 8: 0x334ead92 (stale) CRC-32 checksum value for chunk [0, 0x36a8c): Mirror 1: 0x53e17746 Mirror 9: 0x53e17746 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a8c: skipped 000433 1788536832.309516 write 0x02b93d thru 0x02faf2 (0x41b6 bytes) 000434 1788536832.396393 mapread 0x025185 thru 0x0322a2 (0xd11e bytes) 000435 1788536832.456067 mapwrite 0x01e15f thru 0x020dcc (0x2c6e bytes) 000436 1788536832.553465 trunc from 0x036a8c to 0x02b9f9 (0xb092 bytes) 000437 1788536832.670308 mapread 0x01936b thru 0x01e275 (0x4f0b bytes) 000438 1788536832.671537 read 0x02a9cb thru 0x02b9f9 (0x102f bytes) 000439 1788536832.680828 mapread 0x00844b thru 0x00b24c (0x2e02 bytes) 000440 1788536832.702603 mapread 0x003eff thru 0x010a64 (0xcb66 bytes) 000441 1788536832.711247 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b9fa): Mirror 7: 0x1703f2b3 (stale) Mirror 8: 0x1703f2b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b9fa): Mirror 7: 0x1703f2b3 (stale) Mirror 8: 0x1703f2b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b9fa: skipped 000442 1788536833.063008 fallocate 0x02f425 thru 0x034e51 (0x5a2d bytes) 000443 1788536833.068799 mapwrite 0x0393c2 thru 0x039cd9 (0x918 bytes) 000444 1788536833.100078 read 0x0253ac thru 0x033312 (0xdf67 bytes) 000445 1788536833.110085 write 0x013510 thru 0x019b60 (0x6651 bytes) 000446 1788536833.119420 trunc from 0x035858 to 0x0386d5 (0x2e7e bytes) 000447 1788536833.123731 punch from 0x039173 to 0x039cd9 (0xb67 bytes) 000448 1788536833.132875 MIRROR_RESYNC 3 mirrors 000449 1788536834.247275 trunc from 0x00825f to 0x015f50 (0xdcf2 bytes) 000450 1788536834.298638 punch from 0x01a7e3 to 0x025ac1 (0xb2df bytes) 000451 1788536834.310260 mapread 0x01fa3a thru 0x028e88 (0x944f bytes) 000452 1788536834.353050 trunc from 0x036d37 to 0x039cd9 (0x2fa3 bytes) 000453 1788536834.364077 punch from 0x01bb97 to 0x028bb6 (0xd020 bytes) 000454 1788536834.378040 fallocate 0x00b01d thru 0x01af97 (0xff7b bytes) 000455 1788536834.389591 write 0x03f445 thru 0x03ffff (0xbbb bytes) 000456 1788536834.406721 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x1fab43cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x1fab43cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000457 1788536834.681384 trunc from 0x006692 to 0x00b619 (0x4f88 bytes) 000458 1788536834.686904 mapwrite 0x00a16a thru 0x0146d9 (0xa570 bytes) 000459 1788536834.724718 mapwrite 0x000bc5 thru 0x009ede (0x931a bytes) 000461 1788536834.751077 fallocate 0x031817 thru 0x037659 (0x5e43 bytes) 000462 1788536834.756088 punch from 0x023d79 to 0x031d86 (0xe00e bytes) 000464 1788536834.761473 punch from 0x001a8c to 0x0059d2 (0x3f47 bytes) 000465 1788536834.767504 trunc from 0x027141 to 0x03561b (0xe4db bytes) 000466 1788536834.774300 fallocate 0x00100d thru 0x006162 (0x5156 bytes) 000467 1788536834.779977 trunc from 0x0234f0 to 0x02475f (0x1270 bytes) 000468 1788536834.793978 fallocate 0x001f7d thru 0x0114d4 (0xf558 bytes) 000469 1788536834.800677 mapread 0x015926 thru 0x023b97 (0xe272 bytes) 000471 1788536834.822989 trunc from 0x040000 to 0x010d78 (0x2f287 bytes) 000473 1788536834.882247 read 0x0010cc thru 0x00a307 (0x923c bytes) 000474 1788536834.896142 mapread 0x0074ca thru 0x00b0cc (0x3c03 bytes) 000475 1788536834.898223 read 0x003d45 thru 0x00832a (0x45e6 bytes) 000476 1788536834.898887 trunc from 0x00886e to 0x009d53 (0x14e6 bytes) 000477 1788536834.914666 read 0x006874 thru 0x008f99 (0x2726 bytes) 000478 1788536834.919975 write 0x031924 thru 0x038257 (0x6934 bytes) 000479 1788536834.923261 write 0x02a625 thru 0x0345a8 (0x9f84 bytes) 000480 1788536834.945177 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38258): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38258): Mirror 8: 0x6d8b02e7 (stale) CRC-32 checksum value for chunk [0, 0x38258): Mirror 1: 0x87c2d38f Mirror 9: 0x87c2d38f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38258: skipped 000481 1788536835.855761 punch from 0x0228f6 to 0x022e04 (0x50f bytes) 000482 1788536835.931002 mapwrite 0x03e84e thru 0x03ffff (0x17b2 bytes) 000483 1788536836.013646 trunc from 0x00083a to 0x00cf5b (0xc722 bytes) 000484 1788536836.026461 punch from 0x02a656 to 0x02b92c (0x12d7 bytes) 000485 1788536836.034108 trunc from 0x03c69e to 0x03ffff (0x3962 bytes) 000488 1788536836.054356 fallocate 0x03e095 thru 0x03ffff (0x1f6b bytes) 000489 1788536836.065875 mapwrite 0x017473 thru 0x02211d (0xacab bytes) 000490 1788536836.157046 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x1fab43cd (stale) Mirror 9: 0x831d0ad5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6b5bd9af Mirror 10: 0x6b5bd9af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000491 1788536837.198223 read 0x01d79a thru 0x02669a (0x8f01 bytes) 000492 1788536837.220291 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x1fab43cd (stale) Mirror 9: 0x831d0ad5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x1fab43cd (stale) Mirror 9: 0x831d0ad5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000493 1788536837.678505 write 0x0051d4 thru 0x012359 (0xd186 bytes) 000494 1788536837.706022 read 0x003393 thru 0x0049f5 (0x1663 bytes) 000495 1788536837.712111 write 0x00d073 thru 0x01cb9e (0xfb2c bytes) 000497 1788536837.718061 trunc from 0x040000 to 0x00b685 (0x3497a bytes) 000498 1788536837.756653 mapwrite 0x01a5ce thru 0x01f134 (0x4b67 bytes) 000499 1788536837.825230 mapread 0x017aa1 thru 0x01f134 (0x7694 bytes) 000501 1788536837.826645 mapwrite 0x030e26 thru 0x032ed0 (0x20ab bytes) 000503 1788536837.887493 trunc from 0x032ed1 to 0x012207 (0x20cc9 bytes) 000504 1788536837.921063 read 0x002a9c thru 0x008e9a (0x63ff bytes) 000505 1788536837.942962 trunc from 0x002ee4 to 0x007f2d (0x504a bytes) 000506 1788536837.957110 mapread 0x005ba8 thru 0x00df69 (0x83c2 bytes) 000507 1788536837.996862 mapwrite 0x02b86c thru 0x037d1d (0xc4b2 bytes) 000508 1788536838.149514 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37d1e): Mirror 8: 0xb1a977b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37d1e): Mirror 8: 0xb1a977b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37d1e: skipped 000509 1788536838.589651 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000510 1788536838.755350 read 0x0140d4 thru 0x01475c (0x689 bytes) 000511 1788536838.767484 MIRROR_EXTEND to 2 mirrors 000512 1788536839.729649 trunc from 0x037d1e to 0x00cbce (0x2b14f bytes) 000514 1788536839.774898 punch from 0x0065a7 to 0x00cbce (0x6628 bytes) 000515 1788536839.780735 fallocate 0x01c4af thru 0x026543 (0xa095 bytes) 000516 1788536839.786590 mapread 0x002a27 thru 0x00cbce (0xa1a8 bytes) 000517 1788536839.806624 mapread 0x00b25d thru 0x00cbce (0x1972 bytes) 000518 1788536839.808069 fallocate 0x018664 thru 0x01b04b (0x29e8 bytes) 000519 1788536839.813078 read 0x0174d3 thru 0x01b04b (0x3b79 bytes) 000520 1788536839.821703 mapwrite 0x03e05b thru 0x03ffff (0x1fa5 bytes) 000521 1788536839.856993 mapwrite 0x00704a thru 0x00e9be (0x7975 bytes) 000522 1788536839.888420 read 0x01bd7c thru 0x01e284 (0x2509 bytes) 000523 1788536839.897997 write 0x0174e9 thru 0x018ef9 (0x1a11 bytes) 000524 1788536839.899059 write 0x01c6c7 thru 0x0291e7 (0xcb21 bytes) 000525 1788536839.901507 write 0x0207bc thru 0x024b9f (0x43e4 bytes) 000526 1788536839.903477 mapread 0x002fab thru 0x003284 (0x2da bytes) 000527 1788536839.904896 trunc from 0x040000 to 0x00b8d0 (0x3472f bytes) 000529 1788536839.922778 trunc from 0x00b8d1 to 0x018965 (0xd095 bytes) 000530 1788536839.945920 mapread 0x00cb84 thru 0x018965 (0xbde2 bytes) 000531 1788536840.013969 mapread 0x002903 thru 0x00c915 (0xa013 bytes) 000532 1788536840.020771 MIRROR_RESYNC 2 mirrors 000533 1788536840.729178 fallocate 0x031705 thru 0x03c840 (0xb13c bytes) 000534 1788536840.770655 trunc from 0x018966 to 0x0356ac (0x1cd47 bytes) 000535 1788536840.832110 trunc from 0x0160de to 0x01641b (0x33e bytes) 000536 1788536840.843030 mapread 0x020764 thru 0x02509b (0x4938 bytes) 000537 1788536840.896829 MIRROR_RESYNC 2 mirrors 000538 1788536841.719671 trunc from 0x0356ad to 0x01ac63 (0x1aa49 bytes) 000539 1788536841.782182 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000540 1788536841.977161 read 0x00c54a thru 0x01ac63 (0xe71a bytes) 000541 1788536841.988101 read 0x0153c1 thru 0x01ac63 (0x58a3 bytes) 000542 1788536841.988982 trunc from 0x01ac64 to 0x0248d5 (0x9c72 bytes) 000543 1788536842.007336 fallocate 0x01c037 thru 0x02478b (0x8755 bytes) 000544 1788536842.011973 trunc from 0x014f89 to 0x018a75 (0x3aed bytes) 000546 1788536842.017289 fallocate 0x01f89f thru 0x0259f1 (0x6153 bytes) 000547 1788536842.021802 punch from 0x0031ff to 0x00df0f (0xad11 bytes) 000548 1788536842.026489 mapread 0x01e9f8 thru 0x022662 (0x3c6b bytes) 000550 1788536842.037318 mapwrite 0x00b9b2 thru 0x012be3 (0x7232 bytes) 000551 1788536842.054470 trunc from 0x0259f2 to 0x016aa3 (0xef4e bytes) 000552 1788536842.069905 mapwrite 0x0245b4 thru 0x02cdbc (0x8809 bytes) 000555 1788536842.117990 write 0x039eb1 thru 0x03ffff (0x614f bytes) 000556 1788536842.119497 trunc from 0x0263b2 to 0x02fe10 (0x9a5f bytes) 000558 1788536842.126452 write 0x02296f thru 0x0305f4 (0xdc86 bytes) 000559 1788536842.135295 mapwrite 0x015b30 thru 0x01849f (0x2970 bytes) 000560 1788536842.174629 trunc from 0x040000 to 0x0370a8 (0x8f57 bytes) 000561 1788536842.252967 mapread 0x0351d8 thru 0x0357ea (0x613 bytes) 000562 1788536842.254688 punch from 0x01d423 to 0x024656 (0x7234 bytes) 000563 1788536842.266646 write 0x007c7e thru 0x008de0 (0x1163 bytes) 000564 1788536842.268720 write 0x00f2b2 thru 0x016f7f (0x7cce bytes) 000566 1788536842.273258 fallocate 0x0283cc thru 0x02a5ec (0x2221 bytes) 000567 1788536842.284396 write 0x0374c5 thru 0x03ffff (0x8b3b bytes) 000569 1788536842.289614 read 0x00cc1f thru 0x015b8a (0x8f6c bytes) 000570 1788536842.292926 read 0x00b145 thru 0x00cb41 (0x19fd bytes) 000571 1788536842.295324 trunc from 0x040000 to 0x036019 (0x9fe6 bytes) 000572 1788536842.328048 punch from 0x02f04f to 0x036019 (0x6fcb bytes) 000573 1788536842.338406 write 0x0080bf thru 0x00c3f4 (0x4336 bytes) 000575 1788536842.346852 read 0x02bfa9 thru 0x036019 (0xa071 bytes) 000576 1788536842.361960 fallocate 0x00a6ec thru 0x00b7ea (0x10ff bytes) 000577 1788536842.371436 mapread 0x0354d3 thru 0x036019 (0xb47 bytes) 000578 1788536842.373013 mapread 0x009a6c thru 0x00fff2 (0x6587 bytes) 000579 1788536842.375110 fallocate 0x009db2 thru 0x013e5e (0xa0ad bytes) 000580 1788536842.388175 trunc from 0x01c952 to 0x022a76 (0x6125 bytes) 000581 1788536842.401200 trunc from 0x03601a to 0x0124e4 (0x23b35 bytes) 000582 1788536842.471639 punch from 0x0000a3 to 0x00f810 (0xf76e bytes) 000583 1788536842.502970 write 0x01c9e7 thru 0x0258ed (0x8f07 bytes) 000584 1788536842.507092 punch from 0x0100ae to 0x01825d (0x81b0 bytes) 000585 1788536842.512513 write 0x01029b thru 0x01e62a (0xe390 bytes) 000586 1788536842.518925 mapread 0x005d15 thru 0x00744a (0x1736 bytes) 000587 1788536842.538037 fallocate 0x02b3b1 thru 0x03572e (0xa37e bytes) 000588 1788536842.549388 mapwrite 0x01339e thru 0x01daaa (0xa70d bytes) 000589 1788536842.573455 mapread 0x001dc0 thru 0x008619 (0x685a bytes) 000591 1788536842.610008 fallocate 0x02a653 thru 0x02ac61 (0x60f bytes) 000593 1788536842.619629 fallocate 0x01db60 thru 0x029e7c (0xc31d bytes) 000594 1788536842.624757 read 0x006765 thru 0x0139f9 (0xd295 bytes) 000596 1788536842.633777 fallocate 0x01cca9 thru 0x022bdc (0x5f34 bytes) 000597 1788536842.639423 fallocate 0x023667 thru 0x02a8fd (0x7297 bytes) 000599 1788536842.646947 write 0x023e44 thru 0x029dbb (0x5f78 bytes) 000601 1788536842.649226 punch from 0x0131f0 to 0x022036 (0xee47 bytes) 000602 1788536842.654484 trunc from 0x0250ed to 0x02a9f3 (0x5907 bytes) 000603 1788536842.676910 punch from 0x031027 to 0x034aa8 (0x3a82 bytes) 000604 1788536842.681855 write 0x02a3ad thru 0x0335ec (0x9240 bytes) 000605 1788536842.691183 mapwrite 0x005cbe thru 0x00f10a (0x944d bytes) 000606 1788536842.713112 mapread 0x012e07 thru 0x017749 (0x4943 bytes) 000607 1788536842.738625 trunc from 0x025381 to 0x02d5d7 (0x8257 bytes) 000608 1788536842.743817 write 0x01f759 thru 0x02606d (0x6915 bytes) 000609 1788536842.752550 read 0x013268 thru 0x01a321 (0x70ba bytes) 000611 1788536842.766250 write 0x00d0b8 thru 0x018244 (0xb18d bytes) 000612 1788536842.768202 fallocate 0x00f14f thru 0x0195bb (0xa46d bytes) 000614 1788536842.773716 trunc from 0x03572f to 0x02f045 (0x66e9 bytes) 000615 1788536842.791578 mapwrite 0x00a2f9 thru 0x00b2c0 (0xfc8 bytes) 000616 1788536842.816061 read 0x009aea thru 0x01496b (0xae82 bytes) 000617 1788536842.817093 mapread 0x02a4e9 thru 0x02e0e1 (0x3bf9 bytes) 000618 1788536842.819103 read 0x004445 thru 0x00ab92 (0x674e bytes) 000619 1788536842.819877 punch from 0x01f782 to 0x028d77 (0x95f6 bytes) 000622 1788536842.830445 trunc from 0x02e403 to 0x02f045 (0xc43 bytes) 000623 1788536842.836160 fallocate 0x02d97e thru 0x034c20 (0x72a3 bytes) 000624 1788536842.845767 punch from 0x02cab2 to 0x034c20 (0x816f bytes) 000625 1788536842.853867 trunc from 0x01863a to 0x01a5c5 (0x1f8c bytes) 000626 1788536842.872369 mapread 0x026523 thru 0x02c6a6 (0x6184 bytes) 000627 1788536842.881083 trunc from 0x034c21 to 0x02bae5 (0x913b bytes) 000628 1788536842.926923 MIRROR_EXTEND to 2 mirrors 000629 1788536843.978442 trunc from 0x025eb5 to 0x026988 (0xad4 bytes) 000630 1788536844.033569 trunc from 0x018504 to 0x01a926 (0x2423 bytes) 000631 1788536844.044994 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000632 1788536844.335168 mapwrite 0x009d99 thru 0x00c035 (0x229d bytes) 000633 1788536844.406578 punch from 0x02b3ed to 0x02bae5 (0x6f9 bytes) 000634 1788536844.426509 read 0x02116c thru 0x024248 (0x30dd bytes) 000635 1788536844.443211 fallocate 0x00e3ab thru 0x01b365 (0xcfbb bytes) 000636 1788536844.450377 mapread 0x01ea1a thru 0x02bae5 (0xd0cc bytes) 000637 1788536844.489030 trunc from 0x02bae6 to 0x008220 (0x238c5 bytes) 000638 1788536844.540419 write 0x02cc60 thru 0x03c129 (0xf4ca bytes) 000639 1788536844.546419 write 0x020619 thru 0x02513c (0x4b24 bytes) 000640 1788536844.565716 mapwrite 0x03f48f thru 0x03ffff (0xb71 bytes) 000641 1788536844.640825 trunc from 0x040000 to 0x009688 (0x36977 bytes) 000642 1788536844.667863 trunc from 0x009689 to 0x01023e (0x6bb6 bytes) 000643 1788536844.692345 fallocate 0x014e20 thru 0x01e90e (0x9aef bytes) 000644 1788536844.700289 fallocate 0x030a05 thru 0x037e23 (0x741f bytes) 000645 1788536844.710259 mapwrite 0x004b99 thru 0x00c925 (0x7d8d bytes) 000646 1788536844.737888 punch from 0x0011e5 to 0x003519 (0x2335 bytes) 000647 1788536844.742157 punch from 0x0097d4 to 0x01309d (0x98ca bytes) 000648 1788536844.747021 trunc from 0x01e90f to 0x03d6f1 (0x1ede3 bytes) 000650 1788536844.760740 trunc from 0x03d6f2 to 0x03f518 (0x1e27 bytes) 000651 1788536844.775924 trunc from 0x00580f to 0x00e844 (0x9036 bytes) 000652 1788536844.780539 write 0x0096b5 thru 0x011c8b (0x85d7 bytes) 000653 1788536844.794435 fallocate 0x01bbb0 thru 0x01ca52 (0xea3 bytes) 000654 1788536844.802338 mapread 0x039683 thru 0x03f518 (0x5e96 bytes) 000655 1788536844.812880 read 0x017ee8 thru 0x020c9a (0x8db3 bytes) 000656 1788536844.828068 punch from 0x0023c7 to 0x00db59 (0xb793 bytes) 000657 1788536844.849593 trunc from 0x03f519 to 0x015f04 (0x29614 bytes) 000658 1788536844.869658 read 0x015a06 thru 0x015f04 (0x4ff bytes) 000659 1788536844.870256 mapwrite 0x03d350 thru 0x03ffff (0x2cb0 bytes) 000660 1788536844.905659 write 0x00deb3 thru 0x019885 (0xb9d3 bytes) 000661 1788536844.908064 write 0x01d1f8 thru 0x02cfde (0xfde7 bytes) 000662 1788536844.910650 fallocate 0x03367b thru 0x03f702 (0xc088 bytes) 000663 1788536844.916098 read 0x03357f thru 0x034891 (0x1313 bytes) 000665 1788536844.918871 read 0x02beee thru 0x03469a (0x87ad bytes) 000666 1788536844.919626 punch from 0x0356f4 to 0x0395f2 (0x3eff bytes) 000667 1788536844.924201 fallocate 0x02101b thru 0x0259ff (0x49e5 bytes) 000668 1788536844.929481 punch from 0x00ff92 to 0x0106c2 (0x731 bytes) 000669 1788536844.955683 read 0x021aab thru 0x02c159 (0xa6af bytes) 000671 1788536844.957795 trunc from 0x0184a3 to 0x018590 (0xee bytes) 000672 1788536844.972908 fallocate 0x002f44 thru 0x00a426 (0x74e3 bytes) 000673 1788536844.986836 read 0x0191a6 thru 0x019739 (0x594 bytes) 000674 1788536844.987673 read 0x032724 thru 0x03f457 (0xcd34 bytes) 000675 1788536845.006601 mapread 0x00e5e6 thru 0x01504a (0x6a65 bytes) 000677 1788536845.015336 mapread 0x0271ec thru 0x030480 (0x9295 bytes) 000679 1788536845.017080 punch from 0x010943 to 0x0138a2 (0x2f60 bytes) 000680 1788536845.028799 trunc from 0x024cb2 to 0x02974f (0x4a9e bytes) 000681 1788536845.038373 punch from 0x00821a to 0x008a59 (0x840 bytes) 000682 1788536845.049511 read 0x016d45 thru 0x020704 (0x99c0 bytes) 000683 1788536845.052430 mapwrite 0x028f36 thru 0x034d1b (0xbde6 bytes) 000685 1788536845.118871 punch from 0x01f832 to 0x02148b (0x1c5a bytes) 000686 1788536845.133816 write 0x0330bb thru 0x0334c7 (0x40d bytes) 000687 1788536845.134827 write 0x00433c thru 0x0095d1 (0x5296 bytes) 000688 1788536845.138488 mapwrite 0x0218f5 thru 0x023659 (0x1d65 bytes) 000689 1788536845.179348 fallocate 0x01a98a thru 0x01ed5e (0x43d5 bytes) 000690 1788536845.187876 read 0x0090fc thru 0x0154a5 (0xc3aa bytes) 000692 1788536845.195791 trunc from 0x040000 to 0x01119a (0x2ee65 bytes) 000693 1788536845.235202 mapwrite 0x003781 thru 0x006192 (0x2a12 bytes) 000694 1788536845.268396 punch from 0x0070e2 to 0x008b00 (0x1a1f bytes) 000695 1788536845.279106 punch from 0x0060bf to 0x006e10 (0xd52 bytes) 000696 1788536845.294182 punch from 0x004e3c to 0x01119a (0xc35f bytes) 000697 1788536845.301094 punch from 0x00d4cf to 0x01119a (0x3ccc bytes) 000698 1788536845.307083 punch from 0x00dc09 to 0x01119a (0x3592 bytes) 000699 1788536845.311002 read 0x0110a2 thru 0x01119a (0xf9 bytes) 000700 1788536845.321837 read 0x001d3f thru 0x00a425 (0x86e7 bytes) 000701 1788536845.323166 fallocate 0x02d8be thru 0x02e728 (0xe6b bytes) 000702 1788536845.331647 read 0x008c80 thru 0x01119a (0x851b bytes) 000703 1788536845.333838 punch from 0x00dd01 to 0x00dfbe (0x2be bytes) 000704 1788536845.339869 fallocate 0x038bf5 thru 0x03fe61 (0x726d bytes) 000705 1788536845.346536 fallocate 0x011fa0 thru 0x0200b7 (0xe118 bytes) 000706 1788536845.353218 mapwrite 0x02a3a5 thru 0x030a14 (0x6670 bytes) 000707 1788536845.402517 trunc from 0x029f9f to 0x0362c9 (0xc32b bytes) 000708 1788536845.410593 trunc from 0x03fe62 to 0x00d52a (0x32937 bytes) 000711 1788536845.439089 trunc from 0x0092a8 to 0x00d52a (0x4283 bytes) 000712 1788536845.445919 fallocate 0x0338de thru 0x0397e9 (0x5f0c bytes) 000713 1788536845.451884 mapwrite 0x037d3a thru 0x03ffff (0x82c6 bytes) 000714 1788536845.563571 punch from 0x000acb to 0x00fc5d (0xf193 bytes) 000715 1788536845.570963 mapread 0x00ce05 thru 0x015a43 (0x8c3f bytes) 000716 1788536845.629253 mapwrite 0x036240 thru 0x03ffff (0x9dc0 bytes) 000717 1788536845.657841 trunc from 0x03e11a to 0x03ffff (0x1ee6 bytes) 000718 1788536845.664836 fallocate 0x020d22 thru 0x02623f (0x551e bytes) 000719 1788536845.670725 trunc from 0x040000 to 0x006dd1 (0x3922e bytes) 000720 1788536845.707433 punch from 0x0061b2 to 0x006dd1 (0xc20 bytes) 000721 1788536845.716814 read 0x002847 thru 0x006dd1 (0x458b bytes) 000723 1788536845.742378 punch from 0x0056e9 to 0x006dd1 (0x16e9 bytes) 000726 1788536845.754894 punch from 0x00466b to 0x006dd1 (0x2767 bytes) 000727 1788536845.779525 mapread 0x00371a thru 0x006dd1 (0x36b8 bytes) 000728 1788536845.815606 trunc from 0x006dd2 to 0x01b6e6 (0x14915 bytes) 000729 1788536845.845054 punch from 0x00ffda to 0x019435 (0x945c bytes) 000731 1788536845.851982 write 0x033610 thru 0x037e1b (0x480c bytes) 000732 1788536845.853199 trunc from 0x037e12 to 0x037e1b (0xa bytes) 000734 1788536845.873662 fallocate 0x02a286 thru 0x031145 (0x6ec0 bytes) 000735 1788536845.881030 trunc from 0x037e1c to 0x026536 (0x118e5 bytes) 000737 1788536845.912315 write 0x03a175 thru 0x03ffff (0x5e8b bytes) 000739 1788536845.923480 read 0x03dd94 thru 0x03f2a8 (0x1515 bytes) 000740 1788536845.924661 read 0x013082 thru 0x0227c3 (0xf742 bytes) 000741 1788536845.945387 trunc from 0x00d648 to 0x019077 (0xba30 bytes) 000743 1788536845.952739 write 0x031781 thru 0x03ffff (0xe87f bytes) 000744 1788536845.954900 trunc from 0x040000 to 0x009cee (0x36311 bytes) 000745 1788536845.970918 read 0x0067bc thru 0x009cee (0x3533 bytes) 000746 1788536845.971639 fallocate 0x03038d thru 0x0331ab (0x2e1f bytes) 000747 1788536845.977018 punch from 0x02e967 to 0x02f5f4 (0xc8e bytes) 000748 1788536845.981440 mapread 0x007f57 thru 0x00ab4f (0x2bf9 bytes) 000749 1788536845.991988 mapwrite 0x01f2ab thru 0x0206d0 (0x1426 bytes) 000750 1788536846.004609 mapwrite 0x016d69 thru 0x0266d9 (0xf971 bytes) 000752 1788536846.024968 mapwrite 0x013de4 thru 0x01b84a (0x7a67 bytes) 000753 1788536846.042734 punch from 0x03047a to 0x032171 (0x1cf8 bytes) 000754 1788536846.046838 fallocate 0x03842d thru 0x03ffff (0x7bd3 bytes) 000756 1788536846.052916 write 0x03920a thru 0x03ffff (0x6df6 bytes) 000759 1788536846.064632 read 0x03d8d3 thru 0x03ffff (0x272d bytes) 000760 1788536846.066054 write 0x0283ba thru 0x037f92 (0xfbd9 bytes) 000761 1788536846.068770 fallocate 0x025803 thru 0x02a3e8 (0x4be6 bytes) 000763 1788536846.074917 trunc from 0x040000 to 0x026759 (0x198a6 bytes) 000765 1788536846.096859 trunc from 0x02675a to 0x039598 (0x12e3f bytes) 000766 1788536846.111406 read 0x027377 thru 0x032996 (0xb620 bytes) 000767 1788536846.120255 trunc from 0x039599 to 0x02ca04 (0xcb94 bytes) 000769 1788536846.139971 write 0x01c65d thru 0x01fca1 (0x3645 bytes) 000770 1788536846.145831 trunc from 0x02ca05 to 0x02709a (0x596a bytes) 000771 1788536846.166352 fallocate 0x00886e thru 0x00afbb (0x274e bytes) 000772 1788536846.171608 mapread 0x010e28 thru 0x019612 (0x87eb bytes) 000773 1788536846.172489 mapwrite 0x0070ee thru 0x00d797 (0x66aa bytes) 000774 1788536846.204088 trunc from 0x02709b to 0x02de47 (0x6dad bytes) 000775 1788536846.250470 trunc from 0x02de48 to 0x02fdde (0x1f97 bytes) 000776 1788536846.289257 fallocate 0x002d35 thru 0x00f17d (0xc449 bytes) 000779 1788536846.299363 fallocate 0x00469a thru 0x00c6fe (0x8065 bytes) 000780 1788536846.313801 fallocate 0x026f44 thru 0x0276b7 (0x774 bytes) 000781 1788536846.329443 trunc from 0x00d81d to 0x012cbb (0x549f bytes) 000782 1788536846.340617 trunc from 0x02fddf to 0x005d2b (0x2a0b3 bytes) 000783 1788536846.370630 fallocate 0x03ea32 thru 0x03ffff (0x15ce bytes) 000784 1788536846.376978 mapread 0x00218c thru 0x005d2b (0x3ba0 bytes) 000785 1788536846.379209 trunc from 0x0018a2 to 0x005d2b (0x448a bytes) 000787 1788536846.392216 MIRROR_EXTEND to 2 mirrors 000789 1788536847.045097 read 0x001f9e thru 0x005d2b (0x3d8e bytes) 000790 1788536847.067787 MIRROR_EXTEND to 3 mirrors 000791 1788536847.706776 read 0x004b10 thru 0x005d2b (0x121c bytes) 000793 1788536847.717048 read 0x003975 thru 0x005d2b (0x23b7 bytes) 000794 1788536847.733019 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000795 1788536848.080864 trunc from 0x005d2c to 0x01020a (0xa4df bytes) 000796 1788536848.228968 write 0x035d30 thru 0x03a8e7 (0x4bb8 bytes) 000798 1788536848.230653 mapread 0x012d51 thru 0x01436a (0x161a bytes) 000799 1788536848.261902 trunc from 0x03a8e8 to 0x03f765 (0x4e7e bytes) 000800 1788536848.288110 trunc from 0x03f766 to 0x002d28 (0x3ca3d bytes) 000802 1788536848.303521 mapread 0x0000e7 thru 0x002d28 (0x2c42 bytes) 000803 1788536848.309856 write 0x017406 thru 0x027181 (0xfd7c bytes) 000804 1788536848.321708 mapread 0x0102d1 thru 0x014a4e (0x477e bytes) 000805 1788536848.329540 fallocate 0x034f52 thru 0x03a96f (0x5a1e bytes) 000806 1788536848.335321 mapwrite 0x01127c thru 0x0132d3 (0x2058 bytes) 000807 1788536848.363734 fallocate 0x039571 thru 0x03ffff (0x6a8f bytes) 000808 1788536848.374110 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000809 1788536848.544909 trunc from 0x027bbc to 0x02abb9 (0x2ffe bytes) 000810 1788536848.551838 mapread 0x038419 thru 0x03ffff (0x7be7 bytes) 000811 1788536848.580068 trunc from 0x040000 to 0x03bba4 (0x445b bytes) 000812 1788536848.609108 mapread 0x020a01 thru 0x02e77b (0xdd7b bytes) 000813 1788536848.638864 punch from 0x0174cc to 0x026c03 (0xf738 bytes) 000814 1788536848.647648 trunc from 0x00e9ee to 0x01c2a3 (0xd8b6 bytes) 000815 1788536848.653045 fallocate 0x006e2f thru 0x0135d7 (0xc7a9 bytes) 000816 1788536848.657590 read 0x019c74 thru 0x028e08 (0xf195 bytes) 000818 1788536848.667236 trunc from 0x031e85 to 0x032278 (0x3f4 bytes) 000819 1788536848.671898 trunc from 0x0309c9 to 0x03a4b1 (0x9ae9 bytes) 000820 1788536848.681116 mapread 0x0189bc thru 0x019eeb (0x1530 bytes) 000821 1788536848.690328 trunc from 0x03bba5 to 0x01c858 (0x1f34c bytes) 000822 1788536848.704540 write 0x01790b thru 0x01c5cf (0x4cc5 bytes) 000825 1788536848.707706 write 0x02eeb8 thru 0x03d8ac (0xe9f5 bytes) 000826 1788536848.714113 MIRROR_EXTEND to 2 mirrors 000827 1788536849.598551 mapread 0x039f18 thru 0x03c3cc (0x24b5 bytes) 000828 1788536849.659752 write 0x034507 thru 0x03ffff (0xbaf9 bytes) 000829 1788536849.727449 trunc from 0x040000 to 0x03d120 (0x2edf bytes) 000830 1788536849.770160 punch from 0x00a7de to 0x013a61 (0x9284 bytes) 000831 1788536849.779100 punch from 0x0228b5 to 0x02c2d9 (0x9a25 bytes) 000832 1788536849.786896 mapwrite 0x0375b0 thru 0x03aa39 (0x348a bytes) 000833 1788536849.829117 read 0x020951 thru 0x025e88 (0x5538 bytes) 000834 1788536849.838863 mapwrite 0x03529f thru 0x03af36 (0x5c98 bytes) 000835 1788536849.876311 trunc from 0x033e90 to 0x03b0b3 (0x7224 bytes) 000836 1788536849.884268 read 0x03505a thru 0x03d120 (0x80c7 bytes) 000837 1788536849.926012 mapwrite 0x018244 thru 0x024a72 (0xc82f bytes) 000838 1788536849.978001 MIRROR_RESYNC 2 mirrors 000839 1788536850.925750 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000840 1788536851.112063 punch from 0x006f3d to 0x01234c (0xb410 bytes) 000842 1788536851.132603 mapwrite 0x019a31 thru 0x01fa79 (0x6049 bytes) 000844 1788536851.167462 punch from 0x0262f4 to 0x0351db (0xeee8 bytes) 000846 1788536851.173816 punch from 0x03b34c to 0x03d120 (0x1dd5 bytes) 000847 1788536851.179972 punch from 0x02e098 to 0x02efbb (0xf24 bytes) 000848 1788536851.184925 trunc from 0x0397f2 to 0x03d120 (0x392f bytes) 000849 1788536851.194039 punch from 0x01257c to 0x021a99 (0xf51e bytes) 000851 1788536851.200969 write 0x012387 thru 0x0152e3 (0x2f5d bytes) 000852 1788536851.215649 write 0x03e265 thru 0x03ffff (0x1d9b bytes) 000853 1788536851.217907 fallocate 0x00cf46 thru 0x019403 (0xc4be bytes) 000854 1788536851.226817 punch from 0x007ea0 to 0x00abaf (0x2d10 bytes) 000855 1788536851.232088 read 0x0348bf thru 0x03c54a (0x7c8c bytes) 000856 1788536851.242598 mapread 0x0205f0 thru 0x02e298 (0xdca9 bytes) 000857 1788536851.258957 trunc from 0x0214bd to 0x02f661 (0xe1a5 bytes) 000859 1788536851.270029 fallocate 0x0189d1 thru 0x01cf9f (0x45cf bytes) 000860 1788536851.278935 trunc from 0x03be72 to 0x03ffff (0x418e bytes) 000861 1788536851.297908 read 0x026eff thru 0x0311ec (0xa2ee bytes) 000862 1788536851.328789 write 0x006205 thru 0x009e1f (0x3c1b bytes) 000863 1788536851.330432 mapwrite 0x03fa09 thru 0x03ffff (0x5f7 bytes) 000864 1788536851.358272 mapread 0x027d4a thru 0x0318b8 (0x9b6f bytes) 000865 1788536851.359840 read 0x01ed43 thru 0x02638f (0x764d bytes) 000866 1788536851.362191 trunc from 0x040000 to 0x02bd34 (0x142cb bytes) 000867 1788536851.386529 MIRROR_EXTEND to 2 mirrors 000868 1788536852.172036 write 0x02cad9 thru 0x035239 (0x8761 bytes) 000870 1788536852.227957 write 0x015ca6 thru 0x023997 (0xdcf2 bytes) 000871 1788536852.241906 fallocate 0x010606 thru 0x01301c (0x2a17 bytes) 000872 1788536852.253959 mapread 0x02bacf thru 0x035239 (0x976b bytes) 000873 1788536852.271949 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3523a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3523a): Mirror 2: 0x8b2334a3 (stale) CRC-32 checksum value for chunk [0, 0x3523a): Mirror 1: 0x26aa263b Mirror 3: 0x26aa263b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3523a: skipped 000874 1788536853.129418 read 0x003ed3 thru 0x0086b9 (0x47e7 bytes) 000875 1788536853.151314 write 0x0269d7 thru 0x0368ae (0xfed8 bytes) 000876 1788536853.209751 fallocate 0x017a0b thru 0x0238b5 (0xbeab bytes) 000877 1788536853.216755 write 0x033744 thru 0x03de15 (0xa6d2 bytes) 000878 1788536853.219569 fallocate 0x031b93 thru 0x03cd5d (0xb1cb bytes) 000879 1788536853.225491 trunc from 0x02d185 to 0x02da35 (0x8b1 bytes) 000880 1788536853.243111 punch from 0x024a59 to 0x02b3b7 (0x695f bytes) 000881 1788536853.249562 punch from 0x02f846 to 0x03662b (0x6de6 bytes) 000882 1788536853.257368 write 0x01a92d thru 0x025997 (0xb06b bytes) 000883 1788536853.275014 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3de16): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de16): Mirror 2: 0xd4d11642 (stale) Mirror 3: 0x63660e51 (stale) CRC-32 checksum value for chunk [0, 0x3de16): Mirror 1: 0xf64dc58f Mirror 4: 0xf64dc58f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de16: skipped 000884 1788536854.774797 MIRROR_RESYNC 4 mirrors 000885 1788536856.119316 fallocate 0x000fee thru 0x00b389 (0xa39c bytes) 000886 1788536856.169484 fallocate 0x00cd78 thru 0x014999 (0x7c22 bytes) 000887 1788536856.183714 mapread 0x0111dd thru 0x01daec (0xc910 bytes) 000888 1788536856.209104 mapread 0x01f0a8 thru 0x027700 (0x8659 bytes) 000889 1788536856.220401 trunc from 0x016765 to 0x0198d9 (0x3175 bytes) 000890 1788536856.226153 read 0x013511 thru 0x01a558 (0x7048 bytes) 000891 1788536856.235966 mapwrite 0x025a30 thru 0x03257c (0xcb4d bytes) 000892 1788536856.270922 mapread 0x0076da thru 0x012f71 (0xb898 bytes) 000893 1788536856.289582 punch from 0x032003 to 0x0335e9 (0x15e7 bytes) 000894 1788536856.296959 punch from 0x035c03 to 0x03d5f1 (0x79ef bytes) 000895 1788536856.305736 trunc from 0x03de16 to 0x020b7a (0x1d29b bytes) 000896 1788536856.357525 write 0x015f50 thru 0x01d52e (0x75df bytes) 000897 1788536856.358791 punch from 0x004807 to 0x0138a3 (0xf09d bytes) 000899 1788536856.391649 punch from 0x00a208 to 0x00adb7 (0xbb0 bytes) 000901 1788536856.398916 write 0x00bfd0 thru 0x0132f1 (0x7322 bytes) 000903 1788536856.433988 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20b7b): Mirror 2: 0xdbb505c4 (stale) Mirror 3: 0xdbb505c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20b7b): Mirror 2: 0xdbb505c4 (stale) Mirror 3: 0xdbb505c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20b7b: skipped 000905 1788536856.845787 trunc from 0x020b7b to 0x031faa (0x11430 bytes) 000906 1788536856.898817 trunc from 0x031fab to 0x02f72e (0x287c bytes) 000907 1788536856.933916 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f72f): Mirror 3: 0x5cbf4169 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f72f): Mirror 3: 0x5cbf4169 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f72f: skipped 000908 1788536857.174988 mapread 0x0250f1 thru 0x02db10 (0x8a20 bytes) 000909 1788536857.193171 trunc from 0x02f72f to 0x01d25c (0x124d2 bytes) 000910 1788536857.218546 read 0x0176b5 thru 0x01d25c (0x5ba8 bytes) 000911 1788536857.231604 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1d25d): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d25d): Mirror 3: 0xd16b8924 (stale) CRC-32 checksum value for chunk [0, 0x1d25d): Mirror 1: 0x319e7fa8 Mirror 4: 0x319e7fa8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d25d: skipped 000912 1788536857.917086 read 0x0174f1 thru 0x01d25c (0x5d6c bytes) 000913 1788536857.945702 mapread 0x00643f thru 0x012269 (0xbe2b bytes) 000914 1788536857.989079 trunc from 0x00432f to 0x00603b (0x1d0d bytes) 000915 1788536858.037783 read 0x005acd thru 0x007d63 (0x2297 bytes) 000916 1788536858.057923 trunc from 0x002c5b to 0x006c52 (0x3ff8 bytes) 000917 1788536858.064682 fallocate 0x03daf1 thru 0x03ffff (0x250f bytes) 000919 1788536858.069993 read 0x0216c2 thru 0x02eb6d (0xd4ac bytes) 000921 1788536858.085046 punch from 0x0134e5 to 0x01df64 (0xaa80 bytes) 000922 1788536858.096818 mapread 0x024f66 thru 0x03179c (0xc837 bytes) 000923 1788536858.097863 trunc from 0x02edc8 to 0x030beb (0x1e24 bytes) 000924 1788536858.102810 fallocate 0x0158f6 thru 0x016fd7 (0x16e2 bytes) 000925 1788536858.107918 punch from 0x03ab43 to 0x03eeef (0x43ad bytes) 000926 1788536858.116551 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb2bd3ca5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb2bd3ca5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000927 1788536858.403707 punch from 0x033576 to 0x0397de (0x6269 bytes) 000928 1788536858.414952 trunc from 0x040000 to 0x030937 (0xf6c8 bytes) 000929 1788536858.468609 mapwrite 0x033276 thru 0x0332ea (0x75 bytes) 000930 1788536858.547577 mapread 0x031877 thru 0x0332ea (0x1a74 bytes) 000932 1788536858.563094 mapwrite 0x00ecbe thru 0x01a7e5 (0xbb28 bytes) 000933 1788536858.617086 read 0x014c85 thru 0x01f23a (0xa5b6 bytes) 000934 1788536858.617781 mapwrite 0x0378fa thru 0x03ffff (0x8706 bytes) 000935 1788536858.706241 punch from 0x03b9dc to 0x03ffff (0x4624 bytes) 000938 1788536858.713604 trunc from 0x040000 to 0x020d39 (0x1f2c6 bytes) 000939 1788536858.729894 punch from 0x018475 to 0x01a5a6 (0x2132 bytes) 000940 1788536858.736894 fallocate 0x02d0f0 thru 0x035a3d (0x894e bytes) 000941 1788536858.742306 punch from 0x0172ee to 0x01821d (0xf30 bytes) 000942 1788536858.748004 trunc from 0x020d3a to 0x02a473 (0x973a bytes) 000943 1788536858.771011 mapread 0x011bc0 thru 0x014105 (0x2546 bytes) 000944 1788536858.772039 trunc from 0x00e23f to 0x016e0d (0x8bcf bytes) 000945 1788536858.782911 fallocate 0x037912 thru 0x03ffff (0x86ee bytes) 000947 1788536858.799809 trunc from 0x02fc8d to 0x031198 (0x150c bytes) 000949 1788536858.809557 trunc from 0x040000 to 0x023b98 (0x1c467 bytes) 000950 1788536858.827734 read 0x012bb8 thru 0x018ab4 (0x5efd bytes) 000951 1788536858.846847 punch from 0x01f46e to 0x0232d7 (0x3e6a bytes) 000952 1788536858.867515 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000953 1788536859.056811 trunc from 0x00157d to 0x00b2e8 (0x9d6c bytes) 000954 1788536859.065038 trunc from 0x016565 to 0x01d511 (0x6fad bytes) 000955 1788536859.071097 write 0x03fb0d thru 0x03ffff (0x4f3 bytes) 000957 1788536859.101750 read 0x02b22b thru 0x02d475 (0x224b bytes) 000958 1788536859.109772 trunc from 0x02271c to 0x025a68 (0x334d bytes) 000959 1788536859.119819 punch from 0x027c8e to 0x02bf02 (0x4275 bytes) 000960 1788536859.141581 MIRROR_EXTEND to 2 mirrors 000962 1788536860.213356 write 0x0312dc thru 0x032d16 (0x1a3b bytes) 000963 1788536860.301975 read 0x03c266 thru 0x03ffff (0x3d9a bytes) 000964 1788536860.310998 mapread 0x02c00a thru 0x033211 (0x7208 bytes) 000966 1788536860.348571 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000967 1788536860.610826 fallocate 0x0330d8 thru 0x03ffff (0xcf28 bytes) 000969 1788536860.621471 trunc from 0x02c688 to 0x02f516 (0x2e8f bytes) 000970 1788536860.628050 trunc from 0x01bb93 to 0x024d79 (0x91e7 bytes) 000971 1788536860.636625 read 0x0265bf thru 0x02ac19 (0x465b bytes) 000972 1788536860.653997 punch from 0x03d3be to 0x03ffff (0x2c42 bytes) 000973 1788536860.660001 trunc from 0x007bb4 to 0x00c841 (0x4c8e bytes) 000974 1788536860.667503 trunc from 0x006f9e to 0x00d47b (0x64de bytes) 000975 1788536860.673884 punch from 0x0232a6 to 0x027618 (0x4373 bytes) 000976 1788536860.680007 trunc from 0x02f1df to 0x03f0a1 (0xfec3 bytes) 000977 1788536860.686786 mapwrite 0x03da8a thru 0x03ffff (0x2576 bytes) 000978 1788536860.725871 mapread 0x011846 thru 0x01779c (0x5f57 bytes) 000979 1788536860.738592 write 0x014c78 thru 0x019f25 (0x52ae bytes) 000980 1788536860.744686 write 0x035258 thru 0x035b99 (0x942 bytes) 000981 1788536860.751975 punch from 0x02938a to 0x036da8 (0xda1f bytes) 000982 1788536860.769519 trunc from 0x040000 to 0x024d70 (0x1b28f bytes) 000983 1788536860.815940 mapwrite 0x00581e thru 0x011fc4 (0xc7a7 bytes) 000985 1788536860.845000 trunc from 0x00d73f to 0x013f5d (0x681f bytes) 000986 1788536860.851611 trunc from 0x024d71 to 0x030be2 (0xbe72 bytes) 000987 1788536860.869094 punch from 0x0101b1 to 0x01c514 (0xc364 bytes) 000988 1788536860.874205 trunc from 0x01cacf to 0x01f6ad (0x2bdf bytes) 000989 1788536860.879568 write 0x011887 thru 0x0141ba (0x2934 bytes) 000990 1788536860.890618 read 0x00894d thru 0x015df8 (0xd4ac bytes) 000991 1788536860.902918 trunc from 0x030be3 to 0x0287e3 (0x83ff bytes) 000992 1788536860.923019 read 0x01d019 thru 0x0287e3 (0xb7cb bytes) 000993 1788536860.924768 punch from 0x00de96 to 0x00eb47 (0xcb2 bytes) 000994 1788536860.930534 trunc from 0x0287e4 to 0x00256a (0x26279 bytes) 000995 1788536860.949023 write 0x01d7ce thru 0x02063f (0x2e72 bytes) 000997 1788536860.952613 trunc from 0x020640 to 0x02fda1 (0xf762 bytes) 000998 1788536860.972781 trunc from 0x017c5f to 0x01eb62 (0x6f04 bytes) 000999 1788536860.983831 mapwrite 0x034439 thru 0x03ad47 (0x690f bytes) 001000 1788536861.085301 mapwrite 0x00b7a7 thru 0x014068 (0x88c2 bytes) All operations completed A-OK! PASS 70a (86s) == sanity-flr test 71: check mirror extend parent fid ==== 11:47:47 (1788536867) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x64:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 11:47:57 (1788536877) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.207747 s, 10.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124642 s, 8.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 11:48:07 (1788536887) Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre2 Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1563136 .. Extending file size to 660096 .. failed Extending file size to 3639424 .. Extending file size to 7451904 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5719744 .. Extending file size to 8604448 .. Extending file size to 7694592 .. Extending file size to 7802912 .. Extending file size to 9033568 .. Extending file size to 8620192 .. Extending file size to 9397184 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 61307 61308 61309 61313 61314 failed 192.168.206.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg656-client.virtnet /mnt/lustre2 (opts:) 192.168.206.156@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg656-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:49:27 (1788536967) Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre2 Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3487936 .. Extending file size to 1209056 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3339136 .. Extending file size to 4185664 .. Extending file size to 4052384 .. Extending file size to 5369536 .. Extending file size to 5952992 .. Extending file size to 8671680 .. Extending file size to 8309312 .. Extending file size to 8689504 .. Extending file size to 8654848 .. Extending file size to 8773376 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8919680 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 62972 62974 62976 62977 62979 failed 192.168.206.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg656-client.virtnet /mnt/lustre2 (opts:) 192.168.206.156@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg656-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:50:45 (1788537045) Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1268960 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2068 1269236 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17540 3589480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25284 3581736 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42824 7171216 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1268960 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2068 1269236 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17540 3589480 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25284 3581736 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42824 7171216 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105543 s, 19.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178259 s, 11.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa14a89e6 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg656-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:51:05 (1788537065) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162014 s, 12.9 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 11:51:13 (1788537073) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x73:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x141:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x141:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x142:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x123:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:51:20 (1788537080) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0962628 s, 21.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x143:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x125:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:51:29 (1788537089) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18272 s, 11.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x144:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x145:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x144:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x145:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:51:38 (1788537098) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138763 s, 15.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x146:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x128:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x146:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x128:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x129:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:52:13 (1788537133) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg656-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133381 s, 7.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x201:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x202:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x201:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x201:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x202:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x201:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x202:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:53:09 (1788537189) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg656-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x147:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x148:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124715 s, 16.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x147:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x148:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x147:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x148:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:54:04 (1788537244) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg656-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg656-server: Pool lustre.test_204f created Adding targets to pool oleg656-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg656-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103528 s, 20.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg656-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg656-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg656-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:55:17 (1788537317) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:55:26 (1788537326) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:55:34 (1788537334) oleg656-server: Pool lustre.test_206 created Adding targets to pool oleg656-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg656-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg656-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg656-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg656-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:56:00 (1788537360) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657504 s, 15.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508219 s, 20.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4fb49980 Mirror 3: 0x4fb49980 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:56:12 (1788537372) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:56:15 (1788537375) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:56:19 (1788537379) Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0722501 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1900000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1de0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1de0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1de0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2170000 Read this (4 bytes): BBBB 192.168.206.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg656-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:57:01 (1788537421) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0278222 s, 294 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:57:08 (1788537428) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846623 s, 12.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x96 lmm_fid: [0x200000405:0x96:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x96 lmm_fid: [0x200000405:0x96:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611286 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 11:57:10.000000000 -0400 Modify: 2026-09-04 11:57:10.000000000 -0400 Change: 2026-09-04 11:57:12.000000000 -0400 Birth: 2026-09-04 11:57:11.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:57:17 (1788537437) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385061 s, 27.2 MB/s oleg656-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg656-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg656-server: O/280000401/d19/153: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:57:41 (1788537461) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.261053 s, 157 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0215362 s, 190 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 4 11:57 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 4 11:57 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:57:51 (1788537471) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:57:59 (1788537479) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1878 sec =========== 11:58:08 (1788537488) === sanity-flr: start cleanup 11:58:10 (1788537490) === === sanity-flr: finish cleanup 11:58:19 (1788537499) ===