-----============= acceptance-small: sanity-flr ============----- Wed Sep 30 10:45:56 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg224-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 10:46:14 (1790779574) === oleg224-client.virtnet: executing check_config_client /mnt/lustre oleg224-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg224-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c3f49d21000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c3f49d21000.idle_timeout=debug disable quota as required oleg224-server: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:46:49 (1790779609) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8181] == sanity-flr test 0a: lfs mirror create with -N option == 10:46:51 (1790779611) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:47:07 (1790779627) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:47:10 (1790779630) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:47:13 (1790779633) PASS 0d (31s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:47:44 (1790779664) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:47:48 (1790779668) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 10:47:52 (1790779672) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8c3f49d21000.stats=clear osc.lustre-OST0001-osc-ffff8c3f49d21000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:48:00 (1790779680) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:48:11 (1790779691) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:48:18 (1790779698) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:48:21 (1790779701) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:48:27 (1790779707) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 10:48:34 (1790779714) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:48:39 (1790779719) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:48:46 (1790779726) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:48:53 (1790779733) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:49:02 (1790779742) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:49:08 (1790779748) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137431 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133779 s, 7.8 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:49:15 (1790779755) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.2034 s, 36.1 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:49:23 (1790779763) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:49:34 (1790779774) Stopping /mnt/lustre-ost1 (opts:) on oleg224-server oleg224-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c3f49d21000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c3f49d21000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg224-server: oleg224-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg224-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg224-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg224-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg224-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg224-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg224-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg224-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg224-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg224-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg224-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg224-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg224-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg224-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg224-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg224-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg224-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg224-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg224-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg224-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg224-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg224-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg224-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg224-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg224-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg224-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg224-server oleg224-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c3f49d21000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c3f49d21000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg224-server: oleg224-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg224-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (139s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:51:53 (1790779913) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.329197 s, 19.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg224-server oleg224-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c3f49d21000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c3f49d21000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg224-server: oleg224-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg224-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:52:34 (1790779954) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg224-server oleg224-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c3f49d21000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c3f49d21000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg224-server: oleg224-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg224-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg224-server oleg224-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c3f49d21000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c3f49d21000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg224-server: oleg224-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg224-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:53:59 (1790780039) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.09746 s, 16.9 MB/s -rw-r--r-- 1 root root 52428800 Sep 30 10:54 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.993s user 0m0.001s sys 0m0.208s Stopping /mnt/lustre-ost1 (opts:) on oleg224-server oleg224-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c3f49d21000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c3f49d21000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.960s user 0m0.002s sys 0m0.180s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg224-server: oleg224-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg224-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg224-server oleg224-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c3f49d21000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c3f49d21000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.199s user 0m0.003s sys 0m0.179s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg224-server: oleg224-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg224-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg224-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c3f49d21000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c3f49d21000.ost_server_uuid in FULL state after 0 sec PASS 33b (104s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:55:44 (1790780144) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:55:47 (1790780147) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:55:51 (1790780151) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 10:55:56 (1790780156) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 10:56:06 (1790780166) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.525424 s, 23.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:56:19 (1790780179) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321487 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307334 s, 34.1 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:56:26 (1790780186) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:56:34 (1790780194) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000800459 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00138517 s, 739 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00293293 s, 349 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00106167 s, 965 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 10:56:44 (1790780204) create test files with size 15729881 .. 589794a16da3c6bd3191ecc78fae340a - 18593edba51f6b37eb4963b02d870077 - 52f0403920a0fc3432be1a209a977050 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg224-client.virtnet /mnt/lustre (opts:) Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 10:57:10 (1790780230) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:57:24 (1790780244) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:57:32 (1790780252) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772768 s, 13.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266595 s, 39.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 10:57:40 (1790780260) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0807172 s, 26.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.34888 s, 12.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.215742 s, 9.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 10:57:59 (1790780279) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 10:58:02 (1790780282) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:58:04 (1790780284) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812438 s, 12.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 10:58:13 (1790780293) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.206634 s, 15.2 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:58:25 (1790780305) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:58:38 (1790780318) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.206971 s, 50.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg224-client.virtnet /mnt/lustre (opts:) Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:58:53 (1790780333) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8c3f43dac000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.367389 s, 28.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg224-client.virtnet /mnt/lustre (opts:) Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8c3f4bf3c800.xattr_cache=1 PASS 44e (18s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 10:59:11 (1790780351) error lfs split: create victim file failed: File exists PASS 44f (22s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:59:33 (1790780373) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:00:11 (1790780411) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:00:51 (1790780451) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 11:00:54 (1790780454) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.331471 s, 9.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.149192 s, 21.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5d0f49ed (nosync,stale) Mirror 2: 0x5d0f49ed (nosync,stale) ** verify mirror 2 doesn't change original checksum: 7e4c9fafe0fced9075f8d1f5f420651c - mirror 1 checksum: 45f6a93c843e781e2c8aa33720786a23 - mirror 2 checksum: 7e4c9fafe0fced9075f8d1f5f420651c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-30T11:00:56-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5f lmm_fid: [0x200000405:0x5f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-30T11:00:56-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5f lmm_fid: [0x200000405:0x5f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7e4c9fafe0fced9075f8d1f5f420651c - mirror 1 checksum: 45f6a93c843e781e2c8aa33720786a23 - mirror 2 checksum: 45f6a93c843e781e2c8aa33720786a23 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-30T11:00:56-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5f lmm_fid: [0x200000405:0x5f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-30T11:00:56-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5f lmm_fid: [0x200000405:0x5f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf9:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 11:01:06 (1790780466) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054927 s, 19.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:01:14 (1790780474) Failing mds1 on oleg224-server Stopping /mnt/lustre-mds1 (opts:) on oleg224-server 11:01:22 (1790780482) shut down facet: mds1 facet_host: oleg224-server facet_failover_host: oleg224-server Failover mds1 to oleg224-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 11:01:57 (1790780517) targets are mounted 11:01:57 (1790780517) facet_failover done oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272255 s, 38.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000404 lmm_object_id: 0x8 lmm_fid: [0x240000404:0x8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:02:14 (1790780534) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0441357 s, 464 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0208361 s, 983 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:02:24 (1790780544) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0607003 s, 337 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0147972 s, 1.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:02:34 (1790780554) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00707164 s, 2.3 MB/s PASS 50c (11s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:02:45 (1790780565) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0580736 s, 353 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:02:54 (1790780574) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00518126 s, 791 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0519618 s, 78.8 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:03:02 (1790780582) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x71:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:03:24 (1790780604) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:03:52 (1790780632) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:04:17 (1790780657) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:04:43 (1790780683) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 11:04:50 (1790780690) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 58240 mirror split pid 58241 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 58240 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 58241 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 11:05:59 (1790780759) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 762 skipping zero size read 000002 1790780762.476055 mapwrite 0x0173ff thru 0x02121d (0x9e1f bytes) 000003 1790780762.563851 read 0x006b18 thru 0x00833c (0x1825 bytes) 000004 1790780762.576201 punch from 0x020079 to 0x02121d (0x11a5 bytes) 000006 1790780762.591167 mapwrite 0x00d12a thru 0x00f366 (0x223d bytes) 000007 1790780762.625350 mapread 0x013f3b thru 0x017eb8 (0x3f7e bytes) fallocating to largest ever: 0x40000 000008 1790780762.626259 fallocate 0x038b44 thru 0x03ffff (0x74bc bytes) 000009 1790780762.637769 punch from 0x01549a to 0x01a453 (0x4fba bytes) 000010 1790780762.647856 mapwrite 0x0254f6 thru 0x02ed46 (0x9851 bytes) 000011 1790780762.730028 trunc from 0x040000 to 0x022804 (0x1d7fb bytes) 000012 1790780762.753153 fallocate 0x03fd87 thru 0x03ffff (0x279 bytes) 000014 1790780762.765394 mapread 0x006d28 thru 0x012ab4 (0xbd8d bytes) 000016 1790780762.767090 mapwrite 0x0293a6 thru 0x029e0a (0xa65 bytes) 000018 1790780762.837249 trunc from 0x029e0b to 0x02a622 (0x818 bytes) 000019 1790780762.861833 MIRROR_EXTEND to 2 mirrors 000020 1790780764.056474 MIRROR_RESYNC 2 mirrors 000021 1790780764.383145 trunc from 0x02a623 to 0x035b81 (0xb55f bytes) 000022 1790780764.452973 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35b82): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b82): Mirror 2: 0x8d2ee5ca (stale) CRC-32 checksum value for chunk [0, 0x35b82): Mirror 1: 0x8d2ee5ca Mirror 3: 0x8d2ee5ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b82: skipped 000023 1790780765.413806 mapwrite 0x014e90 thru 0x01e857 (0x99c8 bytes) 000024 1790780765.533687 mapwrite 0x02963c thru 0x03946b (0xfe30 bytes) 000025 1790780765.622137 read 0x004570 thru 0x009641 (0x50d2 bytes) 000026 1790780765.632908 trunc from 0x03946c to 0x00556b (0x33f00 bytes) 000028 1790780765.658528 write 0x00d4fd thru 0x00d66f (0x173 bytes) 000029 1790780765.664928 MIRROR_RESYNC 3 mirrors 000030 1790780766.378432 write 0x008b22 thru 0x013745 (0xac24 bytes) 000031 1790780766.416508 mapwrite 0x01bb04 thru 0x021653 (0x5b50 bytes) 000032 1790780766.478775 trunc from 0x021654 to 0x00dc73 (0x139e0 bytes) 000033 1790780766.492184 write 0x00622c thru 0x00a416 (0x41eb bytes) 000034 1790780766.501108 trunc from 0x00dc74 to 0x01fa1f (0x11dac bytes) 000035 1790780766.516456 punch from 0x01eb19 to 0x01fa1f (0xf07 bytes) 000036 1790780766.520568 mapread 0x007e42 thru 0x0161a4 (0xe363 bytes) 000037 1790780766.529566 punch from 0x0132cd to 0x01fa1f (0xc753 bytes) 000038 1790780766.534347 read 0x017238 thru 0x01fa1f (0x87e8 bytes) 000039 1790780766.543430 read 0x00e62f thru 0x01a919 (0xc2eb bytes) 000040 1790780766.551427 write 0x023d16 thru 0x030871 (0xcb5c bytes) 000041 1790780766.552698 read 0x0276b3 thru 0x030871 (0x91bf bytes) 000042 1790780766.553561 mapwrite 0x013c63 thru 0x020ac7 (0xce65 bytes) 000043 1790780766.574576 fallocate 0x020d9c thru 0x0285a6 (0x780b bytes) 000044 1790780766.579953 mapread 0x02553c thru 0x030871 (0xb336 bytes) 000045 1790780766.580860 punch from 0x01ad9e to 0x01d306 (0x2569 bytes) 000046 1790780766.588282 trunc from 0x000803 to 0x00ff7d (0xf77b bytes) 000047 1790780766.594163 write 0x002570 thru 0x00aa14 (0x84a5 bytes) 000048 1790780766.606101 trunc from 0x0206d2 to 0x02cd26 (0xc655 bytes) 000049 1790780766.612241 trunc from 0x030872 to 0x017225 (0x1964c bytes) 000050 1790780766.636129 fallocate 0x02a9e8 thru 0x02d0b8 (0x26d1 bytes) 000051 1790780766.644079 mapwrite 0x02bfc2 thru 0x0338cc (0x790b bytes) 000052 1790780766.721201 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x338cd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x338cd): Mirror 2: 0x468de2b0 (stale) Mirror 3: 0x468de2b0 (stale) CRC-32 checksum value for chunk [0, 0x338cd): Mirror 1: 0x3e89592e Mirror 4: 0x3e89592e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x338cd: skipped 000053 1790780767.559731 mapwrite 0x00e90e thru 0x01c4fe (0xdbf1 bytes) 000054 1790780767.741016 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x338cd): Mirror 2: 0x468de2b0 (stale) Mirror 3: 0x468de2b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x338cd): Mirror 2: 0x468de2b0 (stale) Mirror 3: 0x468de2b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x338cd: skipped 000055 1790780768.170355 trunc from 0x0338cd to 0x005ad2 (0x2ddfa bytes) 000056 1790780768.200195 trunc from 0x003085 to 0x005ad2 (0x2a4e bytes) 000057 1790780768.213904 mapread 0x004d67 thru 0x005ad2 (0xd6c bytes) 000058 1790780768.242477 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5ad3): Mirror 2: 0x24e04467 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5ad3): Mirror 2: 0x24e04467 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5ad3: skipped 000059 1790780768.577172 mapread 0x0037f6 thru 0x005ad2 (0x22dd bytes) 000060 1790780768.595440 trunc from 0x002757 to 0x005ad2 (0x337c bytes) 000061 1790780768.607130 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) 000062 1790780768.747295 trunc from 0x005ad3 to 0x03feda (0x3a408 bytes) 000063 1790780768.774266 punch from 0x019103 to 0x01ba17 (0x2915 bytes) 000064 1790780768.780193 punch from 0x0192a7 to 0x01a98e (0x16e8 bytes) 000065 1790780768.785493 punch from 0x0239b2 to 0x0257de (0x1e2d bytes) 000067 1790780768.796302 trunc from 0x00b12e to 0x016dc6 (0xbc99 bytes) 000068 1790780768.803016 trunc from 0x03fedb to 0x03c950 (0x358a bytes) 000069 1790780768.823182 fallocate 0x02603c thru 0x02eede (0x8ea3 bytes) 000070 1790780768.828420 trunc from 0x03c951 to 0x02f007 (0xd949 bytes) 000071 1790780768.841591 write 0x004f6d thru 0x00b056 (0x60ea bytes) 000072 1790780768.854126 mapread 0x002121 thru 0x002fb9 (0xe99 bytes) 000073 1790780768.859609 punch from 0x0233bf to 0x02b813 (0x8455 bytes) 000074 1790780768.864679 read 0x024c9c thru 0x02f007 (0xa36c bytes) 000075 1790780768.881565 MIRROR_EXTEND to 2 mirrors 000076 1790780769.934776 punch from 0x026543 to 0x02b189 (0x4c47 bytes) 000077 1790780770.036693 mapread 0x02c2a9 thru 0x02f007 (0x2d5f bytes) 000078 1790780770.082532 mapread 0x0056d9 thru 0x00716f (0x1a97 bytes) 000079 1790780770.096034 write 0x021817 thru 0x02f1ea (0xd9d4 bytes) 000080 1790780770.120171 trunc from 0x02f1eb to 0x00c765 (0x22a85 bytes) 000081 1790780770.157437 mapread 0x001a01 thru 0x00b35b (0x995b bytes) 000082 1790780770.159453 punch from 0x00c226 to 0x00c765 (0x540 bytes) 000083 1790780770.168146 mapwrite 0x029a6c thru 0x032f4e (0x94e3 bytes) 000084 1790780770.263376 mapread 0x008b34 thru 0x0147ad (0xbc7a bytes) 000085 1790780770.277394 MIRROR_RESYNC 2 mirrors 000086 1790780770.996566 mapwrite 0x021223 thru 0x02a04c (0x8e2a bytes) 000087 1790780771.176400 read 0x00ad1b thru 0x00f8a3 (0x4b89 bytes) 000088 1790780771.188527 fallocate 0x0192bc thru 0x02740c (0xe151 bytes) 000089 1790780771.196900 write 0x030e15 thru 0x03415b (0x3347 bytes) 000090 1790780771.207748 punch from 0x02e6f6 to 0x02faa0 (0x13ab bytes) 000091 1790780771.216062 trunc from 0x008774 to 0x011296 (0x8b23 bytes) 000092 1790780771.224553 read 0x001c3c thru 0x003537 (0x18fc bytes) 000093 1790780771.262307 read 0x02c952 thru 0x03415b (0x780a bytes) 000094 1790780771.263843 mapwrite 0x005646 thru 0x0154a0 (0xfe5b bytes) 000095 1790780771.346233 write 0x00827f thru 0x00d755 (0x54d7 bytes) 000096 1790780771.351300 punch from 0x01043f to 0x01b053 (0xac15 bytes) 000097 1790780771.364478 fallocate 0x0383ff thru 0x03a0a3 (0x1ca5 bytes) 000099 1790780771.385139 MIRROR_RESYNC 2 mirrors 000100 1790780772.227982 punch from 0x031a0a to 0x03a0a3 (0x869a bytes) 000101 1790780772.267319 fallocate 0x010e31 thru 0x01ae19 (0x9fe9 bytes) 000103 1790780772.272669 trunc from 0x001aed to 0x00236c (0x880 bytes) 000104 1790780772.280305 punch from 0x00130a to 0x002075 (0xd6c bytes) 000105 1790780772.285550 punch from 0x00a009 to 0x0138a0 (0x9898 bytes) 000107 1790780772.294287 punch from 0x0029f2 to 0x00fdf6 (0xd405 bytes) 000108 1790780772.311608 trunc from 0x03a0a4 to 0x03dfa1 (0x3efe bytes) 000109 1790780772.410773 mapwrite 0x03f997 thru 0x03ffff (0x669 bytes) 000110 1790780772.501028 mapread 0x02c92d thru 0x034d19 (0x83ed bytes) 000111 1790780772.538403 write 0x017db6 thru 0x026e76 (0xf0c1 bytes) 000112 1790780772.548391 read 0x003bf1 thru 0x012cef (0xf0ff bytes) 000113 1790780772.574239 mapread 0x006905 thru 0x00ae53 (0x454f bytes) 000114 1790780772.576163 trunc from 0x03680c to 0x03ffff (0x97f4 bytes) 000115 1790780772.598002 write 0x03bb79 thru 0x03ffff (0x4487 bytes) 000116 1790780772.610418 read 0x03f1ed thru 0x03ffff (0xe13 bytes) 000117 1790780772.621397 MIRROR_RESYNC 2 mirrors 000118 1790780773.451952 write 0x001cf7 thru 0x008b02 (0x6e0c bytes) 000119 1790780773.510350 trunc from 0x040000 to 0x02e018 (0x11fe7 bytes) 000120 1790780773.541384 MIRROR_RESYNC 2 mirrors 000121 1790780774.463473 write 0x01826f thru 0x020bb0 (0x8942 bytes) 000122 1790780774.516682 trunc from 0x02e019 to 0x0396e9 (0xb6d1 bytes) 000123 1790780774.543596 read 0x0020f3 thru 0x007795 (0x56a3 bytes) 000124 1790780774.555477 trunc from 0x01ee7a to 0x0295c3 (0xa74a bytes) 000125 1790780774.588835 trunc from 0x0396ea to 0x02d045 (0xc6a4 bytes) 000126 1790780774.641842 write 0x01eae7 thru 0x02dde2 (0xf2fc bytes) 000127 1790780774.659062 punch from 0x00c47e to 0x015a85 (0x9608 bytes) 000128 1790780774.677477 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) 000129 1790780774.954344 write 0x036b73 thru 0x03b47e (0x490c bytes) 000131 1790780774.963176 write 0x039ee0 thru 0x03ffff (0x6120 bytes) 000132 1790780774.965226 fallocate 0x0089e6 thru 0x00f1f2 (0x680d bytes) 000134 1790780774.975062 trunc from 0x040000 to 0x02772f (0x188d0 bytes) 000135 1790780775.005225 mapread 0x026bf0 thru 0x02772f (0xb40 bytes) 000136 1790780775.022993 punch from 0x014789 to 0x0218e1 (0xd159 bytes) 000137 1790780775.031549 mapwrite 0x0016e7 thru 0x00c9d1 (0xb2eb bytes) 000138 1790780775.081085 write 0x00a40a thru 0x00b9cb (0x15c2 bytes) 000139 1790780775.082165 mapwrite 0x00f466 thru 0x019ff4 (0xab8f bytes) 000140 1790780775.123608 read 0x004ac5 thru 0x00abf5 (0x6131 bytes) 000141 1790780775.124583 write 0x03ccdf thru 0x03ffff (0x3321 bytes) 000142 1790780775.129474 trunc from 0x040000 to 0x022735 (0x1d8ca bytes) 000143 1790780775.150664 read 0x012f17 thru 0x020498 (0xd582 bytes) 000144 1790780775.152113 write 0x01a373 thru 0x0266c0 (0xc34e bytes) 000145 1790780775.156839 read 0x018d24 thru 0x025f75 (0xd252 bytes) 000146 1790780775.157604 trunc from 0x0266c1 to 0x03ac45 (0x14585 bytes) 000147 1790780775.179064 fallocate 0x014543 thru 0x0241ef (0xfcad bytes) 000149 1790780775.185694 mapwrite 0x019dc5 thru 0x028805 (0xea41 bytes) 000150 1790780775.222514 fallocate 0x02c082 thru 0x03b8d2 (0xf851 bytes) 000151 1790780775.228404 mapread 0x0283f9 thru 0x0318f7 (0x94ff bytes) 000153 1790780775.229931 trunc from 0x026891 to 0x03354c (0xccbc bytes) 000154 1790780775.236829 mapwrite 0x038e89 thru 0x03ffff (0x7177 bytes) 000155 1790780775.311737 punch from 0x005959 to 0x00babe (0x6166 bytes) 000156 1790780775.327175 read 0x00e811 thru 0x01cfb9 (0xe7a9 bytes) 000157 1790780775.328142 read 0x023dfd thru 0x02f5f1 (0xb7f5 bytes) 000159 1790780775.358507 read 0x0364c8 thru 0x03e39e (0x7ed7 bytes) 000160 1790780775.359191 trunc from 0x040000 to 0x0271ff (0x18e00 bytes) 000161 1790780775.393392 punch from 0x0108eb to 0x01b560 (0xac76 bytes) 000162 1790780775.404172 write 0x033466 thru 0x033912 (0x4ad bytes) 000163 1790780775.405618 mapread 0x010d1c thru 0x01e455 (0xd73a bytes) 000164 1790780775.430283 trunc from 0x01b46b to 0x01cc94 (0x182a bytes) 000165 1790780775.435750 read 0x001c71 thru 0x00ab35 (0x8ec5 bytes) 000166 1790780775.441748 trunc from 0x00f002 to 0x0173bb (0x83ba bytes) 000167 1790780775.449749 punch from 0x005d3c to 0x010251 (0xa516 bytes) 000168 1790780775.455059 fallocate 0x032849 thru 0x03b006 (0x87be bytes) 000170 1790780775.460367 trunc from 0x030c0d to 0x0330ff (0x24f3 bytes) 000171 1790780775.487457 trunc from 0x033913 to 0x00b2b9 (0x28659 bytes) 000172 1790780775.509937 write 0x0313ea thru 0x032260 (0xe77 bytes) 000173 1790780775.511441 punch from 0x01c997 to 0x025f9e (0x9608 bytes) 000174 1790780775.516182 read 0x012f6f thru 0x018f17 (0x5fa9 bytes) 000175 1790780775.529717 trunc from 0x032261 to 0x019462 (0x18dfe bytes) 000176 1790780775.548780 trunc from 0x019463 to 0x00e9b2 (0xaab0 bytes) 000177 1790780775.565125 trunc from 0x00e9b3 to 0x011c21 (0x326f bytes) 000178 1790780775.583043 trunc from 0x011c22 to 0x038e63 (0x27242 bytes) 000181 1790780775.605200 mapread 0x011cae thru 0x019d8a (0x80dd bytes) 000182 1790780775.666162 mapwrite 0x031ec5 thru 0x03667c (0x47b8 bytes) 000183 1790780775.685124 mapwrite 0x02c281 thru 0x035e95 (0x9c15 bytes) 000184 1790780775.731462 write 0x03af5e thru 0x03ffff (0x50a2 bytes) 000185 1790780775.737405 MIRROR_EXTEND to 2 mirrors 000186 1790780776.542882 read 0x00ec4d thru 0x01c657 (0xda0b bytes) 000187 1790780776.574625 mapwrite 0x03b53a thru 0x03ffff (0x4ac6 bytes) 000188 1790780776.738507 mapread 0x02c554 thru 0x039076 (0xcb23 bytes) 000189 1790780776.754072 mapread 0x01b57f thru 0x025786 (0xa208 bytes) 000190 1790780776.772900 punch from 0x000f0b to 0x004be5 (0x3cdb bytes) 000191 1790780776.786518 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) 000192 1790780777.014509 mapread 0x022fb8 thru 0x02fac8 (0xcb11 bytes) 000193 1790780777.094316 trunc from 0x040000 to 0x03774b (0x88b4 bytes) 000194 1790780777.146153 punch from 0x00f6e3 to 0x00f91c (0x23a bytes) 000195 1790780777.152406 trunc from 0x03774c to 0x00a9ea (0x2cd61 bytes) 000196 1790780777.189733 fallocate 0x03c74b thru 0x03ffff (0x38b5 bytes) 000197 1790780777.199835 trunc from 0x040000 to 0x03be69 (0x4196 bytes) 000199 1790780777.243562 mapwrite 0x008ea2 thru 0x00cb06 (0x3c65 bytes) 000200 1790780777.294610 mapread 0x0302fd thru 0x038b2f (0x8833 bytes) 000201 1790780777.324463 read 0x00c2ad thru 0x01a6c6 (0xe41a bytes) 000203 1790780777.351600 fallocate 0x01f005 thru 0x028531 (0x952d bytes) 000204 1790780777.381861 MIRROR_EXTEND to 2 mirrors 000206 1790780778.154289 write 0x03e6ba thru 0x03f653 (0xf9a bytes) 000207 1790780778.206825 trunc from 0x03f654 to 0x0081b3 (0x374a0 bytes) 000208 1790780778.299009 punch from 0x007fb7 to 0x0081b3 (0x1fd bytes) 000210 1790780778.306586 read 0x000474 thru 0x002ddb (0x2968 bytes) 000211 1790780778.316590 mapread 0x000da6 thru 0x0064bd (0x5718 bytes) 000212 1790780778.343708 mapwrite 0x03ec1b thru 0x03ffff (0x13e5 bytes) 000213 1790780778.407462 fallocate 0x019a2f thru 0x01d967 (0x3f39 bytes) 000214 1790780778.414559 trunc from 0x035f28 to 0x03ffff (0xa0d8 bytes) 000215 1790780778.420113 punch from 0x017898 to 0x026743 (0xeeac bytes) 000216 1790780778.429469 fallocate 0x027b15 thru 0x034884 (0xcd70 bytes) 000217 1790780778.436492 read 0x038d5b thru 0x03e454 (0x56fa bytes) 000218 1790780778.465363 mapwrite 0x0176a5 thru 0x025a06 (0xe362 bytes) 000219 1790780778.505303 mapread 0x01dd5a thru 0x02717d (0x9424 bytes) 000221 1790780778.507890 read 0x03004a thru 0x03dac2 (0xda79 bytes) 000222 1790780778.508731 trunc from 0x040000 to 0x035c64 (0xa39b bytes) 000223 1790780778.527073 write 0x02a49e thru 0x034cfc (0xa85f bytes) 000224 1790780778.532123 read 0x000705 thru 0x0073ee (0x6cea bytes) 000225 1790780778.533093 mapwrite 0x026cf6 thru 0x03532d (0xe638 bytes) 000226 1790780778.556070 read 0x009b35 thru 0x00a26b (0x737 bytes) 000229 1790780778.558987 mapread 0x026493 thru 0x033869 (0xd3d7 bytes) 000231 1790780778.560692 read 0x00c0ab thru 0x014e7d (0x8dd3 bytes) 000232 1790780778.562137 trunc from 0x035c65 to 0x0025a6 (0x336be bytes) 000233 1790780778.588087 punch from 0x0014e4 to 0x0025a6 (0x10c3 bytes) 000234 1790780778.606576 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) 000237 1790780778.770258 fallocate 0x010fa6 thru 0x01daee (0xcb49 bytes) 000238 1790780778.778513 punch from 0x00a0c7 to 0x00f5cf (0x5509 bytes) 000240 1790780778.785514 punch from 0x0018e2 to 0x00279c (0xebb bytes) 000242 1790780778.791178 write 0x01776a thru 0x023ee5 (0xc77c bytes) 000243 1790780778.803077 fallocate 0x0179db thru 0x019c3a (0x2260 bytes) 000244 1790780778.808394 mapread 0x015b84 thru 0x019a8c (0x3f09 bytes) 000246 1790780778.822281 mapwrite 0x00dc23 thru 0x0152c2 (0x76a0 bytes) 000247 1790780778.874163 write 0x01dc6f thru 0x02d69d (0xfa2f bytes) 000248 1790780778.877241 mapread 0x02528c thru 0x02d69d (0x8412 bytes) 000249 1790780778.879061 write 0x00e467 thru 0x0130b6 (0x4c50 bytes) 000250 1790780778.880923 trunc from 0x00202d to 0x002575 (0x549 bytes) 000252 1790780778.893535 write 0x000183 thru 0x008953 (0x87d1 bytes) 000253 1790780778.901292 mapwrite 0x02be64 thru 0x03a82d (0xe9ca bytes) 000255 1790780779.027238 write 0x013f65 thru 0x020d34 (0xcdd0 bytes) 000256 1790780779.029016 trunc from 0x0166e9 to 0x025a56 (0xf36e bytes) 000257 1790780779.068497 trunc from 0x00019d to 0x00e810 (0xe674 bytes) 000258 1790780779.074697 fallocate 0x036d0f thru 0x03a1f3 (0x34e5 bytes) 000259 1790780779.082266 read 0x02a703 thru 0x037296 (0xcb94 bytes) 000260 1790780779.086543 trunc from 0x03a82e to 0x0275cb (0x13262 bytes) 000261 1790780779.124619 mapwrite 0x01eefc thru 0x027a49 (0x8b4e bytes) 000262 1790780779.209926 mapwrite 0x01660c thru 0x01acf5 (0x46ea bytes) 000264 1790780779.245743 trunc from 0x027a4a to 0x00843d (0x1f60c bytes) 000265 1790780779.266363 mapwrite 0x013854 thru 0x02066b (0xce18 bytes) 000266 1790780779.360273 MIRROR_EXTEND to 2 mirrors 000267 1790780779.904260 trunc from 0x02066c to 0x03c3ef (0x1bd84 bytes) 000268 1790780779.989232 mapwrite 0x036a93 thru 0x03ffff (0x956d bytes) 000269 1790780780.052608 trunc from 0x02d959 to 0x034fe1 (0x7689 bytes) 000270 1790780780.064197 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) 000271 1790780780.215207 punch from 0x011c24 to 0x01bb0e (0x9eeb bytes) 000272 1790780780.224295 mapwrite 0x020529 thru 0x023de9 (0x38c1 bytes) 000273 1790780780.261944 read 0x02ccf2 thru 0x02e713 (0x1a22 bytes) 000274 1790780780.274677 MIRROR_EXTEND to 2 mirrors 000275 1790780780.951216 write 0x004c79 thru 0x01277d (0xdb05 bytes) 000276 1790780781.014579 punch from 0x03a1b7 to 0x03b362 (0x11ac bytes) 000277 1790780781.024507 read 0x026f04 thru 0x028f31 (0x202e bytes) 000278 1790780781.030100 mapwrite 0x01764a thru 0x020edf (0x9896 bytes) 000280 1790780781.122708 trunc from 0x040000 to 0x01c2e4 (0x23d1b bytes) 000281 1790780781.288306 read 0x00c004 thru 0x00efd8 (0x2fd5 bytes) 000282 1790780781.289535 fallocate 0x01774d thru 0x01b4fd (0x3db1 bytes) 000283 1790780781.305181 trunc from 0x01c2e5 to 0x01ace9 (0x15fb bytes) 000284 1790780781.355234 read 0x0157cd thru 0x01ace9 (0x551d bytes) 000285 1790780781.364314 MIRROR_RESYNC 2 mirrors 000286 1790780782.185019 read 0x00ec1f thru 0x010b69 (0x1f4b bytes) 000288 1790780782.206756 mapread 0x00c8f8 thru 0x00f0aa (0x27b3 bytes) 000289 1790780782.227038 mapwrite 0x03b241 thru 0x03f6b4 (0x4474 bytes) 000290 1790780782.421396 write 0x022cbd thru 0x027417 (0x475b bytes) 000291 1790780782.448921 punch from 0x03c513 to 0x03f6b4 (0x31a2 bytes) 000292 1790780782.459026 punch from 0x01f6bd to 0x021215 (0x1b59 bytes) 000293 1790780782.473114 fallocate 0x000032 thru 0x00af10 (0xaedf bytes) 000294 1790780782.498995 mapread 0x039b9d thru 0x03e28d (0x46f1 bytes) 000295 1790780782.509357 write 0x008d3d thru 0x00b348 (0x260c bytes) 000296 1790780782.547299 trunc from 0x023287 to 0x028dee (0x5b68 bytes) 000297 1790780782.605911 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f6b5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6b5): Mirror 2: 0x73b4ab2 (stale) CRC-32 checksum value for chunk [0, 0x3f6b5): Mirror 1: 0x3aa7bda9 Mirror 3: 0x3aa7bda9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6b5: skipped 000299 1790780784.412243 mapread 0x03cf90 thru 0x03f6b4 (0x2725 bytes) 000300 1790780784.444508 trunc from 0x03f6b5 to 0x021a09 (0x1dcab bytes) 000301 1790780784.539201 mapwrite 0x02f1c8 thru 0x031710 (0x2549 bytes) 000302 1790780784.650736 trunc from 0x031711 to 0x025904 (0xbe0c bytes) 000303 1790780784.675385 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x25905): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25905): Mirror 2: 0x9ae86be3 (stale) Mirror 3: 0x93089fbe (stale) CRC-32 checksum value for chunk [0, 0x25905): Mirror 1: 0x3a1419aa Mirror 4: 0x3a1419aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25905: skipped 000304 1790780785.985158 fallocate 0x02c9e5 thru 0x035e52 (0x946e bytes) 000305 1790780786.023817 punch from 0x000440 to 0x0081b0 (0x7d71 bytes) 000306 1790780786.031130 write 0x006392 thru 0x008206 (0x1e75 bytes) 000307 1790780786.045187 fallocate 0x02e881 thru 0x03d224 (0xe9a4 bytes) 000309 1790780786.050102 write 0x0046e9 thru 0x00cde2 (0x86fa bytes) 000310 1790780786.058728 trunc from 0x03d225 to 0x001dde (0x3b446 bytes) 000311 1790780786.078340 read 0x000235 thru 0x001dde (0x1baa bytes) 000312 1790780786.084591 fallocate 0x0067cd thru 0x01635b (0xfb8f bytes) 000313 1790780786.091472 write 0x03b648 thru 0x03ffff (0x49b8 bytes) 000315 1790780786.093377 read 0x031f91 thru 0x03e952 (0xc9c2 bytes) 000316 1790780786.103457 fallocate 0x00a343 thru 0x011640 (0x72fe bytes) 000317 1790780786.109595 trunc from 0x03a77b to 0x03ffff (0x5885 bytes) 000318 1790780786.123440 write 0x039c98 thru 0x03ffff (0x6368 bytes) 000319 1790780786.125332 read 0x02b911 thru 0x0360a4 (0xa794 bytes) 000320 1790780786.126222 trunc from 0x01c050 to 0x027322 (0xb2d3 bytes) 000321 1790780786.132184 fallocate 0x009b10 thru 0x00db0d (0x3ffe bytes) 000322 1790780786.137262 fallocate 0x005d49 thru 0x012324 (0xc5dc bytes) 000323 1790780786.142577 trunc from 0x040000 to 0x01d3ca (0x22c35 bytes) 000324 1790780786.158421 trunc from 0x007392 to 0x00ce46 (0x5ab5 bytes) 000325 1790780786.164405 mapwrite 0x01bea0 thru 0x026dbb (0xaf1c bytes) 000326 1790780786.219417 punch from 0x00e152 to 0x019455 (0xb304 bytes) 000327 1790780786.227525 punch from 0x0228f7 to 0x026dbb (0x44c5 bytes) 000328 1790780786.235189 mapread 0x003d4a thru 0x004140 (0x3f7 bytes) 000329 1790780786.236502 punch from 0x00bfef to 0x01b993 (0xf9a5 bytes) 000330 1790780786.244597 mapwrite 0x005d9c thru 0x00da51 (0x7cb6 bytes) 000331 1790780786.314422 trunc from 0x00cdf0 to 0x00ed24 (0x1f35 bytes) 000332 1790780786.320564 write 0x001551 thru 0x004511 (0x2fc1 bytes) 000333 1790780786.323222 read 0x008062 thru 0x00ab87 (0x2b26 bytes) 000334 1790780786.325188 trunc from 0x026dbc to 0x03271b (0xb960 bytes) 000335 1790780786.356983 mapwrite 0x006695 thru 0x0147a1 (0xe10d bytes) 000336 1790780786.403325 fallocate 0x009393 thru 0x00deb4 (0x4b22 bytes) 000337 1790780786.416093 read 0x00f492 thru 0x0185cc (0x913b bytes) 000338 1790780786.416726 fallocate 0x03b1b6 thru 0x03f8f2 (0x473d bytes) 000339 1790780786.427261 trunc from 0x03271c to 0x0298ac (0x8e6f bytes) 000341 1790780786.452243 write 0x021887 thru 0x022814 (0xf8e bytes) 000342 1790780786.453206 mapread 0x020a15 thru 0x025cd7 (0x52c3 bytes) 000343 1790780786.454244 trunc from 0x010410 to 0x01ac9d (0xa88e bytes) 000344 1790780786.459538 read 0x02924f thru 0x0298ac (0x65e bytes) 000345 1790780786.460260 fallocate 0x00c5a9 thru 0x014834 (0x828c bytes) 000346 1790780786.464686 mapwrite 0x036a38 thru 0x03ffff (0x95c8 bytes) 000347 1790780786.524589 write 0x021650 thru 0x02b6fb (0xa0ac bytes) 000348 1790780786.527196 write 0x0306fb thru 0x03c371 (0xbc77 bytes) 000349 1790780786.529206 read 0x02eb3e thru 0x030103 (0x15c6 bytes) 000350 1790780786.530567 punch from 0x006721 to 0x00a3cb (0x3cab bytes) 000352 1790780786.542327 MIRROR_RESYNC 4 mirrors 000353 1790780788.714984 write 0x02232d thru 0x022a1c (0x6f0 bytes) 000354 1790780788.757234 trunc from 0x0339ac to 0x035618 (0x1c6d bytes) 000355 1790780788.766271 fallocate 0x030759 thru 0x03d7ae (0xd056 bytes) 000356 1790780788.772371 punch from 0x02eb0f to 0x030de9 (0x22db bytes) 000357 1790780788.778475 mapwrite 0x0294a9 thru 0x02fe7b (0x69d3 bytes) 000358 1790780788.826059 trunc from 0x01e2aa to 0x029027 (0xad7e bytes) 000359 1790780788.833507 mapread 0x0350d6 thru 0x035b25 (0xa50 bytes) 000361 1790780788.848964 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x42481cfe (stale) Mirror 3: 0x42481cfe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x42481cfe (stale) Mirror 3: 0x42481cfe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000364 1790780789.214481 fallocate 0x01d3af thru 0x02162a (0x427c bytes) 000366 1790780789.223469 mapread 0x00e84c thru 0x0146c3 (0x5e78 bytes) 000367 1790780789.249220 mapwrite 0x035bf1 thru 0x03a244 (0x4654 bytes) 000368 1790780789.293124 punch from 0x039391 to 0x03ddb6 (0x4a26 bytes) 000369 1790780789.308875 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x42481cfe (stale) Mirror 3: 0x42481cfe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd57d3d6f Mirror 4: 0xd57d3d6f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000370 1790780790.252355 trunc from 0x040000 to 0x01eeb3 (0x2114c bytes) 000371 1790780790.354086 write 0x003c8a thru 0x008d17 (0x508e bytes) 000372 1790780790.368393 MIRROR_RESYNC 4 mirrors 000373 1790780791.409378 read 0x01df17 thru 0x01eeb3 (0xf9d bytes) 000374 1790780791.416871 write 0x01c315 thru 0x027ee2 (0xbbce bytes) 000376 1790780791.452118 trunc from 0x018310 to 0x0229d8 (0xa6c9 bytes) 000377 1790780791.473354 mapread 0x01f883 thru 0x01fec2 (0x640 bytes) 000378 1790780791.479213 fallocate 0x01cb58 thru 0x02b663 (0xeb0c bytes) 000379 1790780791.486952 fallocate 0x022693 thru 0x02c14f (0x9abd bytes) 000380 1790780791.492852 mapread 0x01f2f1 thru 0x02b62b (0xc33b bytes) 000381 1790780791.520829 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b664): Mirror 2: 0xe9bb10bc (stale) Mirror 3: 0xe9bb10bc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b664): Mirror 2: 0xe9bb10bc (stale) Mirror 3: 0xe9bb10bc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b664: skipped 000384 1790780791.834168 mapread 0x01dbac thru 0x02b663 (0xdab8 bytes) 000385 1790780791.870481 punch from 0x02a32b to 0x02b663 (0x1339 bytes) 000387 1790780791.876141 mapwrite 0x026368 thru 0x02e537 (0x81d0 bytes) 000388 1790780791.937001 write 0x005e20 thru 0x012823 (0xca04 bytes) 000389 1790780791.948464 write 0x036e56 thru 0x0393d6 (0x2581 bytes) 000390 1790780791.949852 trunc from 0x0393d7 to 0x0145ed (0x24de9 bytes) 000391 1790780791.966970 punch from 0x003fa5 to 0x0090ae (0x510a bytes) 000393 1790780791.996907 punch from 0x001f13 to 0x006d35 (0x4e23 bytes) 000394 1790780792.009849 read 0x00938d thru 0x014178 (0xadec bytes) 000395 1790780792.021428 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x145ee): Mirror 3: 0xbefcee6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x145ee): Mirror 3: 0xbefcee6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x145ee: skipped 000396 1790780792.348394 write 0x004896 thru 0x00b843 (0x6fae bytes) 000397 1790780792.368387 read 0x0130c7 thru 0x0145ed (0x1527 bytes) 000398 1790780792.373524 write 0x035fd4 thru 0x03ffff (0xa02c bytes) 000399 1790780792.376503 read 0x016eec thru 0x01d2a5 (0x63ba bytes) 000400 1790780792.392281 read 0x00dcf7 thru 0x01bdef (0xe0f9 bytes) 000401 1790780792.393028 write 0x00673c thru 0x00aa62 (0x4327 bytes) 000402 1790780792.400423 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) 000403 1790780792.613022 MIRROR_EXTEND to 2 mirrors 000404 1790780793.464323 punch from 0x00ada4 to 0x013096 (0x82f3 bytes) 000405 1790780793.500495 read 0x02e8a0 thru 0x032a2a (0x418b bytes) 000406 1790780793.509209 write 0x03f2eb thru 0x03ffff (0xd15 bytes) 000407 1790780793.524772 mapwrite 0x02b825 thru 0x032b16 (0x72f2 bytes) 000408 1790780793.564470 mapread 0x02ecb9 thru 0x03e6f4 (0xfa3c bytes) 000409 1790780793.572391 trunc from 0x0285a2 to 0x035754 (0xd1b3 bytes) 000410 1790780793.583349 MIRROR_RESYNC 2 mirrors 000413 1790780794.386466 read 0x038656 thru 0x03c039 (0x39e4 bytes) 000414 1790780794.395773 mapread 0x01bf74 thru 0x022867 (0x68f4 bytes) 000415 1790780794.434180 MIRROR_RESYNC 2 mirrors 000416 1790780794.903427 write 0x00c224 thru 0x016785 (0xa562 bytes) 000417 1790780794.964080 read 0x015f38 thru 0x01ca94 (0x6b5d bytes) 000418 1790780794.974099 read 0x026501 thru 0x02d8bd (0x73bd bytes) 000419 1790780794.985916 fallocate 0x00580e thru 0x009117 (0x390a bytes) 000420 1790780794.991096 read 0x032dd9 thru 0x03b40a (0x8632 bytes) 000421 1790780794.991879 trunc from 0x040000 to 0x01e7dd (0x21822 bytes) 000422 1790780795.036532 punch from 0x009675 to 0x011bba (0x8546 bytes) 000423 1790780795.057621 write 0x01c44f thru 0x01d76f (0x1321 bytes) 000425 1790780795.059041 trunc from 0x002b12 to 0x009acd (0x6fbc bytes) 000426 1790780795.066432 fallocate 0x01c558 thru 0x01e6a3 (0x214c bytes) 000427 1790780795.081399 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1e7de): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e7de): Mirror 2: 0x9e0ce3aa (stale) CRC-32 checksum value for chunk [0, 0x1e7de): Mirror 1: 0xd0d03c26 Mirror 3: 0xd0d03c26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e7de: skipped 000430 1790780795.959044 fallocate 0x03cdce thru 0x03ffff (0x3232 bytes) 000431 1790780796.033521 write 0x0044e2 thru 0x0104de (0xbffd bytes) 000432 1790780796.051012 punch from 0x02be4d to 0x0376c4 (0xb878 bytes) 000433 1790780796.056434 fallocate 0x0302f9 thru 0x03f81f (0xf527 bytes) 000435 1790780796.067733 mapread 0x021ca1 thru 0x03089e (0xebfe bytes) 000436 1790780796.133298 trunc from 0x031443 to 0x03e8ef (0xd4ad bytes) 000437 1790780796.142383 read 0x00d733 thru 0x01a25f (0xcb2d bytes) 000438 1790780796.172984 fallocate 0x031e31 thru 0x03eb93 (0xcd63 bytes) 000439 1790780796.184189 punch from 0x0315e4 to 0x03dc20 (0xc63d bytes) 000440 1790780796.195543 punch from 0x024a7c to 0x03111b (0xc6a0 bytes) 000441 1790780796.205997 read 0x0075fb thru 0x0142ce (0xccd4 bytes) 000442 1790780796.207037 write 0x03551d thru 0x03ffff (0xaae3 bytes) 000443 1790780796.246702 MIRROR_RESYNC 3 mirrors 000444 1790780797.249193 trunc from 0x040000 to 0x00278e (0x3d871 bytes) 000445 1790780797.330035 punch from 0x000d0a to 0x00278e (0x1a85 bytes) 000447 1790780797.336422 trunc from 0x00278f to 0x017c7a (0x154ec bytes) 000448 1790780797.380526 write 0x03076c thru 0x030ee2 (0x777 bytes) 000449 1790780797.381571 mapwrite 0x029b4d thru 0x02dce1 (0x4195 bytes) 000450 1790780797.426260 write 0x015ee7 thru 0x0225f5 (0xc70f bytes) 000452 1790780797.439973 mapread 0x02895d thru 0x030ee2 (0x8586 bytes) 000453 1790780797.440942 trunc from 0x01238a to 0x017ec2 (0x5b39 bytes) 000454 1790780797.458466 write 0x0072c2 thru 0x00f3a0 (0x80df bytes) 000455 1790780797.477061 MIRROR_RESYNC 3 mirrors 000456 1790780798.804781 write 0x01cb48 thru 0x02a2df (0xd798 bytes) 000457 1790780798.895377 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30ee3): Mirror 2: 0x83f56450 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30ee3): Mirror 2: 0x83f56450 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30ee3: skipped 000458 1790780799.468215 mapread 0x004874 thru 0x0097fa (0x4f87 bytes) 000459 1790780799.493936 read 0x0166cd thru 0x01f7ae (0x90e2 bytes) 000460 1790780799.505798 write 0x022d19 thru 0x03060d (0xd8f5 bytes) 000461 1790780799.522200 punch from 0x0020df to 0x006379 (0x429b bytes) 000462 1790780799.526753 fallocate 0x029c5c thru 0x02aa78 (0xe1d bytes) 000463 1790780799.535400 mapwrite 0x03b9a2 thru 0x03ffff (0x465e bytes) 000464 1790780799.580456 trunc from 0x000312 to 0x005642 (0x5331 bytes) 000465 1790780799.589002 fallocate 0x02e0ad thru 0x03b278 (0xd1cc bytes) 000466 1790780799.601330 mapwrite 0x021a88 thru 0x02d87a (0xbdf3 bytes) 000467 1790780799.642598 mapread 0x015c6b thru 0x016d51 (0x10e7 bytes) 000470 1790780799.661820 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x24788901 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x80670808 Mirror 3: 0x80670808 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000471 1790780800.559303 fallocate 0x023abb thru 0x0305fc (0xcb42 bytes) 000472 1790780800.608853 MIRROR_RESYNC 3 mirrors 000473 1790780801.663780 trunc from 0x03da0a to 0x03ffff (0x25f6 bytes) 000474 1790780801.718347 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x80670808 (stale) Mirror 3: 0x80670808 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe1248f7e Mirror 4: 0xe1248f7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000475 1790780802.809421 write 0x0186f2 thru 0x019920 (0x122f bytes) 000476 1790780802.854436 punch from 0x01de22 to 0x025cc8 (0x7ea7 bytes) 000477 1790780802.863398 trunc from 0x00d4fb to 0x01133a (0x3e40 bytes) 000478 1790780802.869167 mapread 0x0256cb thru 0x026313 (0xc49 bytes) 000479 1790780802.878850 mapread 0x0194a7 thru 0x02263c (0x9196 bytes) 000480 1790780802.895718 write 0x01385b thru 0x01988d (0x6033 bytes) 000481 1790780802.896905 trunc from 0x040000 to 0x038d95 (0x726a bytes) 000483 1790780802.926329 MIRROR_RESYNC 4 mirrors 000484 1790780804.338625 mapwrite 0x022d78 thru 0x03034d (0xd5d6 bytes) 000485 1790780804.515402 trunc from 0x01fa45 to 0x028170 (0x872c bytes) 000486 1790780804.524524 read 0x02b2fc thru 0x036744 (0xb449 bytes) 000487 1790780804.544320 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38d96): Mirror 3: 0x88c6025a (stale) Mirror 4: 0x88c6025a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d96): Mirror 3: 0x88c6025a (stale) Mirror 4: 0x88c6025a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d96: skipped 000488 1790780805.100352 trunc from 0x038d96 to 0x029313 (0xfa82 bytes) 000489 1790780805.129336 write 0x02dca7 thru 0x0301e2 (0x253c bytes) 000490 1790780805.131341 trunc from 0x004a1a to 0x00b759 (0x6d40 bytes) 000491 1790780805.139601 trunc from 0x0301e3 to 0x027d6f (0x8473 bytes) 000492 1790780805.165812 trunc from 0x027d70 to 0x02f70d (0x799e bytes) 000493 1790780805.200993 punch from 0x02174c to 0x02cf89 (0xb83e bytes) 000494 1790780805.213992 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f70e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f70e): Mirror 3: 0x2d73f57c (stale) Mirror 4: 0x2d73f57c (stale) CRC-32 checksum value for chunk [0, 0x2f70e): Mirror 1: 0xcafaeb01 Mirror 5: 0xcafaeb01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f70e: skipped 000497 1790780806.361225 trunc from 0x02f70e to 0x034518 (0x4e0b bytes) 000498 1790780806.434323 mapread 0x013f6f thru 0x015d5c (0x1dee bytes) 000499 1790780806.466190 mapwrite 0x01a620 thru 0x024fad (0xa98e bytes) 000500 1790780806.520944 mapread 0x020639 thru 0x02720f (0x6bd7 bytes) 000501 1790780806.541663 fallocate 0x01233d thru 0x021310 (0xefd4 bytes) 000502 1790780806.556285 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34519): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34519): Mirror 3: 0x851e0ffd (stale) Mirror 4: 0x851e0ffd (stale) Mirror 5: 0xd8f5a3d3 (stale) CRC-32 checksum value for chunk [0, 0x34519): Mirror 1: 0xffdab0b4 Mirror 6: 0xffdab0b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34519: skipped 000503 1790780808.197508 read 0x00a826 thru 0x0146a4 (0x9e7f bytes) 000504 1790780808.239185 write 0x03fd59 thru 0x03ffff (0x2a7 bytes) 000505 1790780808.322537 fallocate 0x004646 thru 0x00aea5 (0x6860 bytes) 000506 1790780808.346207 write 0x0083c3 thru 0x01459a (0xc1d8 bytes) 000507 1790780808.376007 write 0x01f16a thru 0x024033 (0x4eca bytes) 000508 1790780808.403112 trunc from 0x040000 to 0x01b131 (0x24ece bytes) 000509 1790780808.453409 mapread 0x001766 thru 0x00c46f (0xad0a bytes) 000510 1790780808.490146 MIRROR_RESYNC 5 mirrors 000511 1790780810.228457 mapread 0x010580 thru 0x01b131 (0xabb2 bytes) 000512 1790780810.287791 MIRROR_EXTEND to 6 mirrors 000513 1790780811.156273 read 0x00b2ec thru 0x0149d3 (0x96e8 bytes) 000514 1790780811.177284 trunc from 0x012b31 to 0x01b131 (0x8601 bytes) 000515 1790780811.238954 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1b132): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b132): Mirror 3: 0x855fb89c (stale) Mirror 4: 0x855fb89c (stale) Mirror 5: 0x855fb89c (stale) Mirror 6: 0x855fb89c (stale) Mirror 7: 0x855fb89c (stale) CRC-32 checksum value for chunk [0, 0x1b132): Mirror 1: 0x497d85e1 Mirror 8: 0x497d85e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b132: skipped 000516 1790780812.366158 punch from 0x013871 to 0x016b42 (0x32d2 bytes) 000517 1790780812.448347 fallocate 0x03fe2b thru 0x03ffff (0x1d5 bytes) 000518 1790780812.458030 trunc from 0x005c50 to 0x00c8a8 (0x6c59 bytes) 000519 1790780812.482030 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b132): Mirror 3: 0x855fb89c (stale) Mirror 4: 0x855fb89c (stale) Mirror 5: 0x855fb89c (stale) Mirror 6: 0x855fb89c (stale) Mirror 7: 0x855fb89c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b132): Mirror 3: 0x855fb89c (stale) Mirror 4: 0x855fb89c (stale) Mirror 5: 0x855fb89c (stale) Mirror 6: 0x855fb89c (stale) Mirror 7: 0x855fb89c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b132: skipped 000520 1790780812.911173 mapread 0x005106 thru 0x008e49 (0x3d44 bytes) 000521 1790780812.928328 trunc from 0x019946 to 0x01b131 (0x17ec bytes) 000522 1790780812.939115 fallocate 0x03fae2 thru 0x03ffff (0x51e bytes) 000523 1790780812.954969 trunc from 0x03d906 to 0x03f1b2 (0x18ad bytes) 000524 1790780812.963118 trunc from 0x040000 to 0x006c5c (0x393a3 bytes) 000525 1790780812.992001 trunc from 0x006b8c to 0x006c5c (0xd1 bytes) 000527 1790780813.019792 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6c5d): Mirror 4: 0x74babf0c (stale) Mirror 5: 0x74babf0c (stale) Mirror 6: 0x74babf0c (stale) Mirror 7: 0x74babf0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6c5d): Mirror 4: 0x74babf0c (stale) Mirror 5: 0x74babf0c (stale) Mirror 6: 0x74babf0c (stale) Mirror 7: 0x74babf0c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6c5d: skipped 000528 1790780813.325323 read 0x00084e thru 0x006c5c (0x640f bytes) 000529 1790780813.332380 write 0x009d9d thru 0x012614 (0x8878 bytes) 000530 1790780813.338238 punch from 0x007784 to 0x0113d3 (0x9c50 bytes) 000531 1790780813.351628 mapwrite 0x01006c thru 0x014fe0 (0x4f75 bytes) 000532 1790780813.412953 mapwrite 0x01eccf thru 0x028ba6 (0x9ed8 bytes) 000533 1790780813.463366 mapread 0x016b69 thru 0x01ca65 (0x5efd bytes) 000534 1790780813.482321 MIRROR_RESYNC 5 mirrors 000535 1790780814.618836 punch from 0x0247d5 to 0x028ba6 (0x43d2 bytes) 000537 1790780814.664768 MIRROR_RESYNC 5 mirrors 000538 1790780816.178597 MIRROR_EXTEND to 6 mirrors 000539 1790780817.449627 write 0x03a0f3 thru 0x03ffff (0x5f0d bytes) 000540 1790780817.542692 read 0x02d9f9 thru 0x02f4f2 (0x1afa bytes) 000541 1790780817.581157 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x329e0d3e (stale) Mirror 5: 0x329e0d3e (stale) Mirror 6: 0x329e0d3e (stale) Mirror 7: 0x329e0d3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x329e0d3e (stale) Mirror 5: 0x329e0d3e (stale) Mirror 6: 0x329e0d3e (stale) Mirror 7: 0x329e0d3e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000542 1790780818.048962 trunc from 0x01c738 to 0x029cb2 (0xd57b bytes) 000543 1790780818.054024 punch from 0x0140e4 to 0x022fea (0xef07 bytes) 000544 1790780818.059254 write 0x00088c thru 0x002f2e (0x26a3 bytes) 000545 1790780818.077896 MIRROR_RESYNC 5 mirrors 000547 1790780819.858127 trunc from 0x040000 to 0x0234ab (0x1cb54 bytes) 000548 1790780819.913034 MIRROR_RESYNC 5 mirrors 000549 1790780821.002112 punch from 0x018be3 to 0x01c232 (0x3650 bytes) 000550 1790780821.047470 trunc from 0x01effc to 0x0234ab (0x44b0 bytes) 000551 1790780821.059277 punch from 0x0055e0 to 0x0063b1 (0xdd2 bytes) 000552 1790780821.073261 mapwrite 0x006659 thru 0x013a64 (0xd40c bytes) 000554 1790780821.132415 fallocate 0x0391bb thru 0x03ffff (0x6e45 bytes) 000556 1790780821.139506 trunc from 0x040000 to 0x0398d3 (0x672c bytes) 000557 1790780821.175285 read 0x007f37 thru 0x00f152 (0x721c bytes) 000558 1790780821.175926 mapwrite 0x003216 thru 0x006bd5 (0x39c0 bytes) 000559 1790780821.195218 fallocate 0x01a8bf thru 0x01f2fe (0x4a40 bytes) 000560 1790780821.199996 trunc from 0x0398d4 to 0x008ec9 (0x30a0a bytes) 000561 1790780821.222115 MIRROR_RESYNC 5 mirrors 000564 1790780821.874973 trunc from 0x00264d to 0x008ec9 (0x687d bytes) 000565 1790780821.906884 trunc from 0x008eca to 0x02af2a (0x22061 bytes) 000566 1790780821.932333 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2af2b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2af2b): Mirror 4: 0xb822f68 (stale) Mirror 5: 0xb822f68 (stale) Mirror 6: 0xb822f68 (stale) Mirror 7: 0xb822f68 (stale) CRC-32 checksum value for chunk [0, 0x2af2b): Mirror 1: 0xd4a9c057 Mirror 8: 0xd4a9c057 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2af2b: skipped 000567 1790780822.839484 trunc from 0x02af2b to 0x03fcff (0x14dd5 bytes) 000568 1790780822.900059 read 0x03a7bd thru 0x03fcff (0x5543 bytes) 000569 1790780822.910440 fallocate 0x02d179 thru 0x038cc1 (0xbb49 bytes) 000570 1790780822.915317 trunc from 0x00c74b to 0x0169c2 (0xa278 bytes) 000571 1790780822.920608 trunc from 0x000096 to 0x004caf (0x4c1a bytes) 000572 1790780822.926156 trunc from 0x0217db to 0x02a5c9 (0x8def bytes) 000573 1790780822.931478 mapwrite 0x027e6f thru 0x02b1fd (0x338f bytes) 000574 1790780822.972480 write 0x0345af thru 0x03ffff (0xba51 bytes) 000575 1790780822.978238 read 0x02e5e9 thru 0x02f434 (0xe4c bytes) 000576 1790780822.978862 trunc from 0x02f70a to 0x033516 (0x3e0d bytes) 000577 1790780822.986897 write 0x0188f7 thru 0x026f39 (0xe643 bytes) 000578 1790780823.010499 mapread 0x031086 thru 0x031321 (0x29c bytes) 000579 1790780823.011552 mapread 0x00b5bb thru 0x019181 (0xdbc7 bytes) 000580 1790780823.012938 trunc from 0x040000 to 0x00153a (0x3eac5 bytes) 000581 1790780823.040820 mapwrite 0x02001f thru 0x029b3b (0x9b1d bytes) 000582 1790780823.141793 MIRROR_RESYNC 6 mirrors 000583 1790780824.174355 trunc from 0x001cec to 0x002c88 (0xf9d bytes) 000584 1790780824.214929 mapwrite 0x01a934 thru 0x0217e6 (0x6eb3 bytes) 000585 1790780824.251392 read 0x0194e4 thru 0x023faa (0xaac7 bytes) 000586 1790780824.252090 trunc from 0x0093ff to 0x015a21 (0xc623 bytes) 000587 1790780824.258189 mapwrite 0x0138c5 thru 0x022d85 (0xf4c1 bytes) 000588 1790780824.288477 trunc from 0x012260 to 0x01e16d (0xbf0e bytes) 000589 1790780824.294548 mapread 0x016fff thru 0x026ef1 (0xfef3 bytes) 000591 1790780824.315898 write 0x00d49f thru 0x016c1c (0x977e bytes) 000593 1790780824.318831 write 0x03630c thru 0x03943f (0x3134 bytes) 000594 1790780824.320512 fallocate 0x02eb88 thru 0x03d8a6 (0xed1f bytes) 000595 1790780824.325828 write 0x03ff92 thru 0x03ffff (0x6e bytes) 000596 1790780824.327441 write 0x03deee thru 0x03ffff (0x2112 bytes) 000597 1790780824.332854 mapread 0x0092ee thru 0x01544f (0xc162 bytes) 000598 1790780824.340791 trunc from 0x00daca to 0x01a64d (0xcb84 bytes) 000599 1790780824.357704 trunc from 0x040000 to 0x03372a (0xc8d5 bytes) 000600 1790780824.413475 punch from 0x017191 to 0x01a3ef (0x325f bytes) 000601 1790780824.418832 write 0x005f1b thru 0x00ea08 (0x8aee bytes) 000602 1790780824.424757 trunc from 0x03372b to 0x022d03 (0x10a27 bytes) 000603 1790780824.453753 MIRROR_RESYNC 6 mirrors 000604 1790780825.724935 punch from 0x0064a5 to 0x0085c7 (0x2123 bytes) 000605 1790780825.765441 fallocate 0x01075f thru 0x015672 (0x4f14 bytes) 000606 1790780825.771499 read 0x007538 thru 0x011eed (0xa9b6 bytes) 000607 1790780825.792031 punch from 0x0184ba to 0x0194b5 (0xffc bytes) 000608 1790780825.799192 trunc from 0x0048b2 to 0x00cf84 (0x86d3 bytes) 000609 1790780825.806571 write 0x024981 thru 0x027b26 (0x31a6 bytes) 000610 1790780825.808160 write 0x011a2f thru 0x01bf5e (0xa530 bytes) 000611 1790780825.815072 mapwrite 0x02afca thru 0x02fffe (0x5035 bytes) 000612 1790780825.941218 read 0x00bdd2 thru 0x0113be (0x55ed bytes) 000613 1790780825.975242 read 0x00c5a9 thru 0x019cb0 (0xd708 bytes) 000614 1790780825.976859 punch from 0x008f6b to 0x0122b8 (0x934e bytes) 000615 1790780826.004878 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ffff): Mirror 5: 0x8ed86a96 (stale) Mirror 6: 0x8ed86a96 (stale) Mirror 7: 0x8ed86a96 (stale) Mirror 8: 0x8ed86a96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffff): Mirror 5: 0x8ed86a96 (stale) Mirror 6: 0x8ed86a96 (stale) Mirror 7: 0x8ed86a96 (stale) Mirror 8: 0x8ed86a96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffff: skipped 000616 1790780826.582336 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ffff): Mirror 6: 0x8ed86a96 (stale) Mirror 7: 0x8ed86a96 (stale) Mirror 8: 0x8ed86a96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffff): Mirror 6: 0x8ed86a96 (stale) Mirror 7: 0x8ed86a96 (stale) Mirror 8: 0x8ed86a96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffff: skipped 000617 1790780827.037354 punch from 0x024177 to 0x02fffe (0xbe88 bytes) 000618 1790780827.047536 write 0x018bf6 thru 0x022d8f (0xa19a bytes) 000620 1790780827.066897 trunc from 0x01e310 to 0x0273c4 (0x90b5 bytes) 000621 1790780827.079567 mapread 0x013c3a thru 0x01d792 (0x9b59 bytes) 000622 1790780827.107610 read 0x0229e1 thru 0x0285d6 (0x5bf6 bytes) 000623 1790780827.132917 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ffff): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffff): Mirror 6: 0x8ed86a96 (stale) Mirror 7: 0x8ed86a96 (stale) Mirror 8: 0x8ed86a96 (stale) CRC-32 checksum value for chunk [0, 0x2ffff): Mirror 1: 0x6d5b7c50 Mirror 9: 0x6d5b7c50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffff: skipped 000624 1790780827.980872 trunc from 0x02f8ab to 0x02fffe (0x754 bytes) 000625 1790780828.035027 mapread 0x02fba5 thru 0x02fffe (0x45a bytes) 000626 1790780828.041523 trunc from 0x00f674 to 0x00fc3b (0x5c8 bytes) 000627 1790780828.048235 write 0x004eea thru 0x00ac4e (0x5d65 bytes) 000628 1790780828.069584 fallocate 0x001e73 thru 0x0053b3 (0x3541 bytes) 000629 1790780828.079158 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ffff): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffff): Mirror 6: 0x8ed86a96 (stale) Mirror 7: 0x8ed86a96 (stale) Mirror 8: 0x8ed86a96 (stale) Mirror 9: 0x6d5b7c50 (stale) CRC-32 checksum value for chunk [0, 0x2ffff): Mirror 1: 0xa265c48c Mirror 10: 0xa265c48c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffff: skipped 000630 1790780829.470935 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ffff): Mirror 7: 0x8ed86a96 (stale) Mirror 8: 0x8ed86a96 (stale) Mirror 9: 0x6d5b7c50 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffff): Mirror 7: 0x8ed86a96 (stale) Mirror 8: 0x8ed86a96 (stale) Mirror 9: 0x6d5b7c50 (stale) CRC-32 checksum value for chunk [0, 0x2ffff): Mirror 1: 0xa265c48c Mirror 10: 0xa265c48c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffff: skipped 000631 1790780830.529935 fallocate 0x020dae thru 0x026133 (0x5386 bytes) 000633 1790780830.632959 read 0x014153 thru 0x01f4af (0xb35d bytes) 000634 1790780830.656540 trunc from 0x02ccad to 0x02fffe (0x3352 bytes) 000635 1790780830.670036 write 0x03b4b7 thru 0x03ffff (0x4b49 bytes) 000636 1790780830.678296 write 0x004799 thru 0x004cc2 (0x52a bytes) 000637 1790780830.685921 trunc from 0x040000 to 0x02f0b3 (0x10f4c bytes) 000638 1790780830.749071 mapwrite 0x02577f thru 0x029f04 (0x4786 bytes) 000639 1790780830.803185 trunc from 0x02f0b4 to 0x02db21 (0x1592 bytes) 000640 1790780830.831563 fallocate 0x00bb10 thru 0x0191c9 (0xd6ba bytes) 000641 1790780830.843399 read 0x01f247 thru 0x0218a3 (0x265d bytes) 000642 1790780830.844983 write 0x02575d thru 0x030e5b (0xb6ff bytes) 000643 1790780830.860113 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x30e5c): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30e5c): Mirror 7: 0xb598439 (stale) Mirror 8: 0xb598439 (stale) Mirror 9: 0x79a18ddf (stale) Mirror 10: 0xfa6f1321 (stale) CRC-32 checksum value for chunk [0, 0x30e5c): Mirror 1: 0xc2865e07 Mirror 11: 0xc2865e07 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30e5c: skipped 000644 1790780832.080806 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30e5c): Mirror 7: 0xb598439 (stale) Mirror 9: 0x79a18ddf (stale) Mirror 10: 0xfa6f1321 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30e5c): Mirror 7: 0xb598439 (stale) Mirror 9: 0x79a18ddf (stale) Mirror 10: 0xfa6f1321 (stale) CRC-32 checksum value for chunk [0, 0x30e5c): Mirror 1: 0xc2865e07 Mirror 11: 0xc2865e07 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30e5c: skipped 000645 1790780832.839892 read 0x00e5fb thru 0x01dfdf (0xf9e5 bytes) 000646 1790780832.886997 mapread 0x01605b thru 0x01a765 (0x470b bytes) 000647 1790780832.931456 write 0x02ac57 thru 0x033ab9 (0x8e63 bytes) 000648 1790780833.017748 trunc from 0x012859 to 0x020de4 (0xe58c bytes) 000649 1790780833.023792 write 0x00e909 thru 0x0159c6 (0x70be bytes) 000650 1790780833.034622 fallocate 0x01a01d thru 0x0253e8 (0xb3cc bytes) 000651 1790780833.040867 write 0x0229b6 thru 0x032582 (0xfbcd bytes) 000652 1790780833.047541 trunc from 0x02346f to 0x028146 (0x4cd8 bytes) 000653 1790780833.064069 fallocate 0x0263a6 thru 0x027174 (0xdcf bytes) 000656 1790780833.069686 mapwrite 0x03d278 thru 0x03ffff (0x2d88 bytes) 000657 1790780833.151063 mapread 0x02c17b thru 0x02e09f (0x1f25 bytes) 000658 1790780833.151998 mapread 0x02f479 thru 0x032f02 (0x3a8a bytes) 000659 1790780833.165624 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xe1dde9a0 (stale) Mirror 10: 0x17a3fa74 (stale) Mirror 11: 0x5dfce907 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xe1dde9a0 (stale) Mirror 10: 0x17a3fa74 (stale) Mirror 11: 0x5dfce907 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000660 1790780833.513747 mapwrite 0x02e2d6 thru 0x03a1dc (0xbf07 bytes) 000661 1790780833.564378 fallocate 0x01cb3f thru 0x0241d8 (0x769a bytes) 000662 1790780833.568896 mapwrite 0x0201a0 thru 0x02f6ed (0xf54e bytes) 000663 1790780833.598049 mapwrite 0x01ccb1 thru 0x029138 (0xc488 bytes) 000664 1790780833.617331 punch from 0x036069 to 0x03ffff (0x9f97 bytes) 000665 1790780833.626505 mapwrite 0x03567a thru 0x03ffff (0xa986 bytes) 000666 1790780833.677804 mapread 0x000387 thru 0x010153 (0xfdcd bytes) 000667 1790780833.700826 fallocate 0x02b257 thru 0x0337cb (0x8575 bytes) 000668 1790780833.709812 trunc from 0x003cce to 0x005d4e (0x2081 bytes) 000669 1790780833.729817 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x17a3fa74 (stale) Mirror 11: 0x5dfce907 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x17a3fa74 (stale) Mirror 11: 0x5dfce907 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000670 1790780834.075906 mapwrite 0x025908 thru 0x0340a2 (0xe79b bytes) 000671 1790780834.119099 trunc from 0x02cfec to 0x03a7fa (0xd80f bytes) 000672 1790780834.124413 fallocate 0x03ecfb thru 0x03ffff (0x1305 bytes) 000673 1790780834.132100 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x5dfce907 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x5dfce907 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000674 1790780834.442849 MIRROR_RESYNC 2 mirrors 000676 1790780835.476990 mapread 0x02f6ec thru 0x03638b (0x6ca0 bytes) 000677 1790780835.511606 mapwrite 0x01bc23 thru 0x02b341 (0xf71f bytes) 000678 1790780835.671677 trunc from 0x01046d to 0x014c7b (0x480f bytes) 000679 1790780835.682440 write 0x039245 thru 0x03ffff (0x6dbb bytes) 000680 1790780835.695704 mapwrite 0x0153e6 thru 0x02484d (0xf468 bytes) 000681 1790780835.734398 fallocate 0x03ab1a thru 0x03ffff (0x54e6 bytes) 000683 1790780835.739311 fallocate 0x03aa37 thru 0x03ffff (0x55c9 bytes) 000685 1790780835.745926 read 0x018493 thru 0x020a07 (0x8575 bytes) 000686 1790780835.746790 mapread 0x023d0b thru 0x033bf4 (0xfeea bytes) 000687 1790780835.760062 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) 000688 1790780835.929105 fallocate 0x0154a3 thru 0x017d25 (0x2883 bytes) 000689 1790780835.938613 mapread 0x0119bc thru 0x01a922 (0x8f67 bytes) 000690 1790780835.966480 trunc from 0x040000 to 0x00a579 (0x35a86 bytes) 000691 1790780835.985991 mapread 0x005cc2 thru 0x006ac5 (0xe04 bytes) 000692 1790780835.998298 write 0x035761 thru 0x036185 (0xa25 bytes) 000693 1790780835.999355 fallocate 0x036942 thru 0x03ffff (0x96be bytes) 000694 1790780836.004807 write 0x02dd3d thru 0x038851 (0xab15 bytes) 000696 1790780836.017102 read 0x008ced thru 0x00fea5 (0x71b9 bytes) 000697 1790780836.030245 mapread 0x02cb16 thru 0x03b9be (0xeea9 bytes) 000698 1790780836.047330 write 0x01aa76 thru 0x022bda (0x8165 bytes) 000699 1790780836.056227 write 0x0226af thru 0x02e47b (0xbdcd bytes) 000700 1790780836.058451 punch from 0x015081 to 0x015160 (0xe0 bytes) 000701 1790780836.063587 punch from 0x008a16 to 0x01008b (0x7676 bytes) 000702 1790780836.068978 fallocate 0x003f69 thru 0x00eb64 (0xabfc bytes) 000703 1790780836.074736 mapread 0x0367ff thru 0x03a9c8 (0x41ca bytes) 000704 1790780836.076100 punch from 0x02e318 to 0x0354bd (0x71a6 bytes) 000705 1790780836.093041 read 0x0210ad thru 0x021fb9 (0xf0d bytes) 000706 1790780836.093671 mapwrite 0x01c414 thru 0x01ce20 (0xa0d bytes) 000709 1790780836.106501 write 0x0214a1 thru 0x0299e5 (0x8545 bytes) 000710 1790780836.108880 mapread 0x016df8 thru 0x023d74 (0xcf7d bytes) 000711 1790780836.129745 read 0x02c3f6 thru 0x02e599 (0x21a4 bytes) 000712 1790780836.130312 write 0x0362e0 thru 0x03e17a (0x7e9b bytes) 000714 1790780836.132764 write 0x02df5c thru 0x03393f (0x59e4 bytes) 000715 1790780836.134381 mapwrite 0x0206f6 thru 0x025f48 (0x5853 bytes) 000716 1790780836.152056 trunc from 0x01e307 to 0x02e095 (0xfd8f bytes) 000717 1790780836.157012 mapwrite 0x0014fd thru 0x00c59e (0xb0a2 bytes) 000718 1790780836.175732 mapwrite 0x034403 thru 0x03ffff (0xbbfd bytes) 000720 1790780836.199222 write 0x014687 thru 0x016d44 (0x26be bytes) 000721 1790780836.200539 fallocate 0x01af06 thru 0x02187b (0x6976 bytes) 000722 1790780836.205363 fallocate 0x015850 thru 0x025499 (0xfc4a bytes) 000724 1790780836.211584 trunc from 0x040000 to 0x00ed35 (0x312ca bytes) 000726 1790780836.230859 trunc from 0x00ed36 to 0x009520 (0x5815 bytes) 000727 1790780836.257408 fallocate 0x001e79 thru 0x00fe14 (0xdf9c bytes) 000729 1790780836.264926 fallocate 0x03af71 thru 0x03ffff (0x508f bytes) 000730 1790780836.271364 fallocate 0x009ea9 thru 0x00e630 (0x4788 bytes) 000731 1790780836.282802 read 0x03ae06 thru 0x03ffff (0x51fa bytes) 000732 1790780836.299442 punch from 0x00473b to 0x010bdf (0xc4a5 bytes) 000733 1790780836.310369 punch from 0x0073f4 to 0x00b41f (0x402c bytes) 000734 1790780836.316100 mapread 0x026127 thru 0x035942 (0xf81c bytes) 000735 1790780836.321281 MIRROR_EXTEND to 2 mirrors 000736 1790780837.060922 mapread 0x02bc06 thru 0x034819 (0x8c14 bytes) 000737 1790780837.103441 trunc from 0x040000 to 0x00e217 (0x31de8 bytes) 000738 1790780837.202516 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) 000739 1790780837.361000 MIRROR_EXTEND to 2 mirrors 000740 1790780838.202868 trunc from 0x0016ee to 0x00e217 (0xcb2a bytes) 000741 1790780838.231225 trunc from 0x00e218 to 0x01e57b (0x10364 bytes) 000742 1790780838.246429 trunc from 0x01e57c to 0x02b9ad (0xd432 bytes) 000743 1790780838.271441 trunc from 0x02b9ae to 0x01a34b (0x11662 bytes) 000744 1790780838.293092 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1a34c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a34c): Mirror 2: 0xd0a39950 (stale) CRC-32 checksum value for chunk [0, 0x1a34c): Mirror 1: 0xe9b26ec0 Mirror 3: 0xe9b26ec0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a34c: skipped 000745 1790780839.283547 mapread 0x012f9d thru 0x0175d5 (0x4639 bytes) 000746 1790780839.324457 trunc from 0x01a34c to 0x01122d (0x911e bytes) 000747 1790780839.428975 fallocate 0x0018f2 thru 0x0073eb (0x5afa bytes) 000748 1790780839.444583 MIRROR_RESYNC 3 mirrors 000750 1790780840.146138 trunc from 0x005213 to 0x00e8e7 (0x96d5 bytes) 000751 1790780840.213256 read 0x006657 thru 0x00fdf7 (0x97a1 bytes) 000752 1790780840.227862 punch from 0x002932 to 0x01122d (0xe8fc bytes) 000753 1790780840.233510 trunc from 0x006a76 to 0x01063f (0x9bca bytes) 000754 1790780840.250944 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1122e): Mirror 3: 0x5431a694 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1122e): Mirror 3: 0x5431a694 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1122e: skipped 000755 1790780840.655601 read 0x00c123 thru 0x01122d (0x510b bytes) 000757 1790780840.683996 mapwrite 0x017c23 thru 0x019304 (0x16e2 bytes) 000758 1790780840.771484 MIRROR_RESYNC 2 mirrors 000759 1790780841.649872 trunc from 0x019305 to 0x03e15a (0x24e56 bytes) 000760 1790780841.709621 MIRROR_RESYNC 2 mirrors 000761 1790780842.449177 trunc from 0x030294 to 0x038559 (0x82c6 bytes) 000762 1790780842.524982 MIRROR_RESYNC 2 mirrors 000763 1790780843.487425 mapwrite 0x012e7c thru 0x01c063 (0x91e8 bytes) 000765 1790780843.607565 write 0x00d52d thru 0x0145fe (0x70d2 bytes) 000766 1790780843.611026 trunc from 0x03e15b to 0x026c47 (0x17513 bytes) 000767 1790780843.651328 trunc from 0x0024c3 to 0x00993b (0x7479 bytes) 000768 1790780843.660323 write 0x004332 thru 0x00cecc (0x8b9b bytes) 000769 1790780843.684972 read 0x01d6d7 thru 0x026c47 (0x9571 bytes) 000771 1790780843.702983 mapwrite 0x022771 thru 0x02e46c (0xbcfc bytes) 000772 1790780843.815496 write 0x0330f5 thru 0x034392 (0x129e bytes) 000773 1790780843.819044 write 0x0316f4 thru 0x03ffff (0xe90c bytes) 000774 1790780843.847925 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) 000775 1790780844.108974 fallocate 0x035702 thru 0x03dfea (0x88e9 bytes) 000776 1790780844.115434 mapwrite 0x02abb0 thru 0x0363fd (0xb84e bytes) 000777 1790780844.207167 punch from 0x01e0fe to 0x027cc0 (0x9bc3 bytes) 000778 1790780844.214842 write 0x004554 thru 0x006ff0 (0x2a9d bytes) 000779 1790780844.252412 mapwrite 0x02a9bb thru 0x0372bb (0xc901 bytes) 000780 1790780844.288447 write 0x0227f6 thru 0x0258e0 (0x30eb bytes) 000781 1790780844.298836 read 0x03222b thru 0x03d3bc (0xb192 bytes) 000782 1790780844.301387 fallocate 0x01fa8e thru 0x021978 (0x1eeb bytes) 000784 1790780844.311778 punch from 0x00b967 to 0x0165c7 (0xac61 bytes) 000785 1790780844.316128 read 0x03e61f thru 0x03ffff (0x19e1 bytes) 000786 1790780844.329534 mapread 0x025190 thru 0x028337 (0x31a8 bytes) 000789 1790780844.334490 fallocate 0x0016d0 thru 0x00fbbd (0xe4ee bytes) 000790 1790780844.346579 trunc from 0x004c42 to 0x0096fe (0x4abd bytes) 000791 1790780844.366542 fallocate 0x01dfb1 thru 0x02374b (0x579b bytes) 000792 1790780844.385321 trunc from 0x01d927 to 0x023b3e (0x6218 bytes) 000793 1790780844.422545 write 0x01ab39 thru 0x0214e5 (0x69ad bytes) 000794 1790780844.437541 read 0x03a7bb thru 0x03ffff (0x5845 bytes) 000795 1790780844.439145 write 0x002e0f thru 0x00f107 (0xc2f9 bytes) 000796 1790780844.444515 write 0x01e38d thru 0x022272 (0x3ee6 bytes) 000797 1790780844.464338 read 0x019c1c thru 0x026d67 (0xd14c bytes) 000798 1790780844.465684 read 0x01f87a thru 0x025bae (0x6335 bytes) 000799 1790780844.467655 fallocate 0x024f15 thru 0x028727 (0x3813 bytes) 000800 1790780844.492041 MIRROR_EXTEND to 2 mirrors 000801 1790780845.486878 trunc from 0x040000 to 0x03d76c (0x2893 bytes) 000802 1790780845.557105 mapwrite 0x039ad7 thru 0x03a500 (0xa2a bytes) 000803 1790780845.577147 punch from 0x02acca to 0x02cc3e (0x1f75 bytes) 000804 1790780845.581769 write 0x01a8fc thru 0x0263bc (0xbac1 bytes) 000805 1790780845.594679 mapwrite 0x02e7f6 thru 0x03936d (0xab78 bytes) 000806 1790780845.645994 trunc from 0x02c9ad to 0x03059f (0x3bf3 bytes) 000807 1790780845.655370 mapread 0x012f9e thru 0x0141c1 (0x1224 bytes) 000808 1790780845.674276 fallocate 0x00127f thru 0x001494 (0x216 bytes) 000809 1790780845.683200 fallocate 0x0228a3 thru 0x026429 (0x3b87 bytes) 000810 1790780845.696150 mapwrite 0x0214da thru 0x02b588 (0xa0af bytes) 000811 1790780845.729042 mapwrite 0x03719e thru 0x03ffff (0x8e62 bytes) 000812 1790780845.775540 punch from 0x0353b2 to 0x03bea5 (0x6af4 bytes) 000813 1790780845.786617 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) 000814 1790780846.005021 mapwrite 0x021e30 thru 0x02b881 (0x9a52 bytes) 000815 1790780846.065427 read 0x010502 thru 0x0157a9 (0x52a8 bytes) 000816 1790780846.076822 write 0x00248a thru 0x003db3 (0x192a bytes) 000817 1790780846.090989 punch from 0x021950 to 0x0310a1 (0xf752 bytes) 000818 1790780846.097669 punch from 0x014d80 to 0x02010e (0xb38f bytes) 000819 1790780846.103062 read 0x0205b8 thru 0x026515 (0x5f5e bytes) 000820 1790780846.118113 mapread 0x00a341 thru 0x015f42 (0xbc02 bytes) 000821 1790780846.120026 trunc from 0x040000 to 0x035a8c (0xa573 bytes) 000822 1790780846.146808 trunc from 0x0016c8 to 0x00996f (0x82a8 bytes) 000823 1790780846.158340 trunc from 0x035a8d to 0x007883 (0x2e209 bytes) 000824 1790780846.178325 read 0x004311 thru 0x007883 (0x3573 bytes) 000825 1790780846.187621 mapread 0x0004a8 thru 0x0053b5 (0x4f0e bytes) 000827 1790780846.190687 fallocate 0x029b97 thru 0x029d5f (0x1c9 bytes) 000828 1790780846.196262 mapread 0x005674 thru 0x007802 (0x218f bytes) 000829 1790780846.197766 read 0x002521 thru 0x004a56 (0x2536 bytes) 000832 1790780846.201477 write 0x0202e1 thru 0x028cf8 (0x8a18 bytes) 000833 1790780846.203859 trunc from 0x028cf9 to 0x002f0b (0x25ded bytes) 000834 1790780846.224365 punch from 0x002b5c to 0x002f0b (0x3b0 bytes) 000835 1790780846.232342 trunc from 0x002f0c to 0x008cef (0x5de4 bytes) 000836 1790780846.254595 trunc from 0x008cf0 to 0x0155dd (0xc8ee bytes) 000837 1790780846.282477 mapwrite 0x01424b thru 0x014886 (0x63c bytes) 000838 1790780846.306930 trunc from 0x010184 to 0x0155dd (0x545a bytes) 000839 1790780846.312984 mapread 0x01543c thru 0x0155dd (0x1a2 bytes) 000840 1790780846.318927 mapread 0x00dabe thru 0x0155dd (0x7b20 bytes) 000841 1790780846.345253 trunc from 0x0155de to 0x00e615 (0x6fc8 bytes) 000843 1790780846.362313 punch from 0x007462 to 0x00e615 (0x71b4 bytes) 000845 1790780846.369370 trunc from 0x00e616 to 0x024052 (0x15a3d bytes) 000846 1790780846.391093 trunc from 0x0128ef to 0x0139be (0x10d0 bytes) 000847 1790780846.398801 fallocate 0x02fcf6 thru 0x0333c5 (0x36d0 bytes) 000848 1790780846.406237 trunc from 0x012f9f to 0x01400d (0x106f bytes) 000849 1790780846.411123 mapwrite 0x02da9b thru 0x038d12 (0xb278 bytes) 000851 1790780846.462354 write 0x03fde5 thru 0x03ffff (0x21b bytes) 000852 1790780846.463715 read 0x00bcc6 thru 0x01b18f (0xf4ca bytes) 000853 1790780846.473421 trunc from 0x040000 to 0x021718 (0x1e8e7 bytes) 000854 1790780846.489827 mapread 0x02133a thru 0x021718 (0x3df bytes) 000855 1790780846.491408 mapwrite 0x0218f7 thru 0x02191c (0x26 bytes) 000856 1790780846.531100 trunc from 0x02191d to 0x01a0fe (0x781e bytes) 000858 1790780846.549574 trunc from 0x01a0ff to 0x008308 (0x11df6 bytes) 000859 1790780846.568403 write 0x03dfc2 thru 0x03ffff (0x203e bytes) 000860 1790780846.570744 fallocate 0x01c55a thru 0x029fc8 (0xda6f bytes) 000861 1790780846.576105 read 0x017f6f thru 0x0267d0 (0xe862 bytes) 000863 1790780846.588839 read 0x03da27 thru 0x03ffff (0x25d9 bytes) 000864 1790780846.590472 write 0x022319 thru 0x03094f (0xe637 bytes) 000866 1790780846.594055 trunc from 0x040000 to 0x00dbaf (0x32450 bytes) 000868 1790780846.615924 punch from 0x000992 to 0x00bcf9 (0xb368 bytes) 000870 1790780846.621756 read 0x00caa5 thru 0x00cf52 (0x4ae bytes) 000871 1790780846.622338 punch from 0x00dacd to 0x00dbaf (0xe3 bytes) 000872 1790780846.626208 mapread 0x002f72 thru 0x00447d (0x150c bytes) 000874 1790780846.641933 write 0x00335a thru 0x004b70 (0x1817 bytes) 000875 1790780846.642983 read 0x004b61 thru 0x008303 (0x37a3 bytes) 000876 1790780846.643546 read 0x00cda5 thru 0x00dbaf (0xe0b bytes) 000877 1790780846.644845 fallocate 0x039d84 thru 0x03c9ff (0x2c7c bytes) 000880 1790780846.650847 trunc from 0x01308e to 0x01de30 (0xada3 bytes) 000881 1790780846.654620 mapwrite 0x00e716 thru 0x0104b5 (0x1da0 bytes) 000884 1790780846.686455 trunc from 0x02b812 to 0x0306a6 (0x4e95 bytes) 000885 1790780846.718401 read 0x024727 thru 0x033833 (0xf10d bytes) 000886 1790780846.763226 mapread 0x01c4cf thru 0x01f4de (0x3010 bytes) 000887 1790780846.764851 punch from 0x02a185 to 0x03201c (0x7e98 bytes) 000888 1790780846.787069 fallocate 0x00c3bf thru 0x017148 (0xad8a bytes) 000889 1790780846.806864 read 0x011774 thru 0x019a14 (0x82a1 bytes) 000890 1790780846.812819 mapwrite 0x03b7db thru 0x03c7c1 (0xfe7 bytes) 000891 1790780846.855228 MIRROR_EXTEND to 2 mirrors 000892 1790780847.962498 fallocate 0x0371f9 thru 0x03e0ec (0x6ef4 bytes) 000893 1790780847.990454 mapwrite 0x036495 thru 0x03b2e9 (0x4e55 bytes) 000894 1790780848.040406 trunc from 0x03ca00 to 0x008951 (0x340ae bytes) 000895 1790780848.073675 mapread 0x005dbd thru 0x008951 (0x2b95 bytes) 000896 1790780848.100860 mapwrite 0x011a47 thru 0x01b761 (0x9d1b bytes) 000897 1790780848.202336 trunc from 0x00348c to 0x008cc1 (0x5836 bytes) 000898 1790780848.209024 trunc from 0x01706b to 0x01b761 (0x46f7 bytes) 000899 1790780848.215262 write 0x02cbd0 thru 0x03789a (0xaccb bytes) 000900 1790780848.218103 trunc from 0x01260e to 0x012f5a (0x94d bytes) 000901 1790780848.223527 write 0x02a338 thru 0x02dd65 (0x3a2e bytes) 000903 1790780848.232526 fallocate 0x000af2 thru 0x001ebc (0x13cb bytes) 000904 1790780848.238470 mapread 0x0079ad thru 0x016d8e (0xf3e2 bytes) 000905 1790780848.258014 read 0x02cbea thru 0x037280 (0xa697 bytes) 000906 1790780848.258552 mapwrite 0x01f170 thru 0x02a198 (0xb029 bytes) 000907 1790780848.297821 read 0x003ab2 thru 0x00cc59 (0x91a8 bytes) 000908 1790780848.298908 trunc from 0x03789b to 0x01a244 (0x1d656 bytes) 000909 1790780848.315668 mapread 0x004966 thru 0x00d44d (0x8ae8 bytes) 000911 1790780848.318022 write 0x011982 thru 0x0194ed (0x7b6c bytes) 000912 1790780848.326455 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) 000913 1790780848.476431 mapwrite 0x00e36d thru 0x019b57 (0xb7eb bytes) 000915 1790780848.534323 mapwrite 0x0382f9 thru 0x03ffff (0x7d07 bytes) 000916 1790780848.587904 trunc from 0x040000 to 0x01a1d8 (0x25e27 bytes) 000917 1790780848.604433 trunc from 0x01a1d9 to 0x0379a5 (0x1d7cd bytes) 000918 1790780848.621203 write 0x01f85d thru 0x0215de (0x1d82 bytes) 000919 1790780848.631433 mapwrite 0x02ff95 thru 0x035352 (0x53be bytes) 000920 1790780848.661590 fallocate 0x025ec8 thru 0x033377 (0xd4b0 bytes) 000921 1790780848.667222 trunc from 0x0379a6 to 0x030cef (0x6cb6 bytes) 000922 1790780848.682228 punch from 0x0126a8 to 0x018244 (0x5b9d bytes) 000923 1790780848.692234 MIRROR_EXTEND to 2 mirrors 000924 1790780849.979405 punch from 0x01b863 to 0x0205fb (0x4d99 bytes) 000925 1790780850.019868 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30cf0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30cf0): Mirror 2: 0x666a396c (stale) CRC-32 checksum value for chunk [0, 0x30cf0): Mirror 1: 0x2c108482 Mirror 3: 0x2c108482 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30cf0: skipped 000926 1790780851.482056 fallocate 0x02bfa7 thru 0x035f6f (0x9fc9 bytes) 000927 1790780851.592409 mapread 0x015eb2 thru 0x0252fa (0xf449 bytes) 000928 1790780851.666510 trunc from 0x02576e to 0x031ccb (0xc55e bytes) 000929 1790780851.672907 mapread 0x00713a thru 0x01084b (0x9712 bytes) 000930 1790780851.686854 write 0x0187a0 thru 0x021e72 (0x96d3 bytes) 000931 1790780851.694329 mapread 0x00455b thru 0x00fd4d (0xb7f3 bytes) 000932 1790780851.695533 write 0x01725a thru 0x01e6c1 (0x7468 bytes) 000933 1790780851.698669 read 0x01283a thru 0x017f9e (0x5765 bytes) 000934 1790780851.706317 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35f70): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f70): Mirror 2: 0x83de10dd (stale) Mirror 3: 0x99069ff5 (stale) CRC-32 checksum value for chunk [0, 0x35f70): Mirror 1: 0xd0c47432 Mirror 4: 0xd0c47432 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f70: skipped 000935 1790780853.293150 fallocate 0x035cc8 thru 0x03747c (0x17b5 bytes) 000936 1790780853.349852 write 0x011d14 thru 0x01ec4f (0xcf3c bytes) 000937 1790780853.370079 punch from 0x00af32 to 0x0188e6 (0xd9b5 bytes) 000938 1790780853.388237 fallocate 0x03568d thru 0x03c54c (0x6ec0 bytes) 000939 1790780853.393472 punch from 0x01a7f6 to 0x025054 (0xa85f bytes) 000940 1790780853.399334 trunc from 0x0171d0 to 0x0177c5 (0x5f6 bytes) 000941 1790780853.405880 punch from 0x01d1e8 to 0x01f1c4 (0x1fdd bytes) 000942 1790780853.418312 MIRROR_RESYNC 4 mirrors 000943 1790780854.840000 mapwrite 0x00863b thru 0x016265 (0xdc2b bytes) 000944 1790780855.136547 write 0x023abb thru 0x025400 (0x1946 bytes) 000945 1790780855.174403 read 0x032614 thru 0x03c54c (0x9f39 bytes) 000946 1790780855.208426 MIRROR_RESYNC 4 mirrors 000947 1790780856.906572 trunc from 0x0364d6 to 0x03c54c (0x6077 bytes) 000949 1790780856.938314 trunc from 0x02c34d to 0x02d76a (0x141e bytes) 000950 1790780856.944390 fallocate 0x0399c7 thru 0x03ffff (0x6639 bytes) 000951 1790780856.951750 trunc from 0x03c54d to 0x03514d (0x73ff bytes) 000952 1790780856.979907 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3514e): Mirror 2: 0x58ac8506 (stale) Mirror 3: 0x58ac8506 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3514e): Mirror 2: 0x58ac8506 (stale) Mirror 3: 0x58ac8506 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3514e: skipped 000953 1790780857.348151 mapread 0x03402f thru 0x03514d (0x111f bytes) 000954 1790780857.366009 mapwrite 0x00436b thru 0x009147 (0x4ddd bytes) 000955 1790780857.435181 trunc from 0x03514e to 0x016f8e (0x1e1bf bytes) 000956 1790780857.465971 mapread 0x000a50 thru 0x0027b3 (0x1d64 bytes) 000957 1790780857.474837 punch from 0x013ac6 to 0x016f8e (0x34c9 bytes) 000958 1790780857.481589 fallocate 0x00b4ae thru 0x0185db (0xd12e bytes) 000959 1790780857.488829 write 0x00a119 thru 0x00f12f (0x5017 bytes) 000960 1790780857.493542 mapread 0x006f74 thru 0x00ae78 (0x3f05 bytes) 000961 1790780857.495151 mapwrite 0x01509e thru 0x016f78 (0x1edb bytes) 000962 1790780857.562562 punch from 0x017a3b to 0x0185db (0xba1 bytes) 000964 1790780857.572967 fallocate 0x02ca8d thru 0x02f52a (0x2a9e bytes) 000965 1790780857.591193 fallocate 0x0283dd thru 0x033e33 (0xba57 bytes) 000966 1790780857.619819 read 0x00e429 thru 0x011786 (0x335e bytes) 000967 1790780857.626183 mapwrite 0x01b0da thru 0x0266d8 (0xb5ff bytes) 000968 1790780857.737465 mapread 0x018943 thru 0x01c93c (0x3ffa bytes) 000969 1790780857.740986 read 0x00d550 thru 0x014cc0 (0x7771 bytes) 000970 1790780857.756284 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x266d9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x266d9): Mirror 2: 0x350aa1be (stale) Mirror 3: 0x350aa1be (stale) CRC-32 checksum value for chunk [0, 0x266d9): Mirror 1: 0x347d36d7 Mirror 4: 0x347d36d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x266d9: skipped 000971 1790780859.170072 trunc from 0x0135e5 to 0x018656 (0x5072 bytes) 000972 1790780859.231271 trunc from 0x01816e to 0x024d6b (0xcbfe bytes) 000973 1790780859.238665 fallocate 0x023cbf thru 0x02b60a (0x794c bytes) 000974 1790780859.250556 punch from 0x0035d3 to 0x00e738 (0xb166 bytes) 000975 1790780859.262188 fallocate 0x010bc7 thru 0x01931b (0x8755 bytes) 000976 1790780859.267751 mapread 0x01b02f thru 0x0269c9 (0xb99b bytes) 000977 1790780859.325452 MIRROR_RESYNC 4 mirrors 000978 1790780861.025327 fallocate 0x01d3ed thru 0x02011c (0x2d30 bytes) 000979 1790780861.063857 trunc from 0x02b60b to 0x029f6e (0x169c bytes) 000980 1790780861.104931 mapread 0x0147d7 thru 0x0198b0 (0x50da bytes) 000981 1790780861.138188 fallocate 0x023413 thru 0x031e95 (0xea83 bytes) 000983 1790780861.154994 mapread 0x00469d thru 0x004a92 (0x3f6 bytes) 000984 1790780861.178483 trunc from 0x031e96 to 0x012438 (0x1fa5d bytes) 000985 1790780861.203724 mapwrite 0x01de24 thru 0x023691 (0x586e bytes) 000986 1790780861.300173 mapwrite 0x031970 thru 0x03d54e (0xbbdf bytes) 000987 1790780861.387289 write 0x013b75 thru 0x021c04 (0xe090 bytes) 000988 1790780861.410083 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d54f): Mirror 2: 0x4daf3a74 (stale) Mirror 4: 0x4daf3a74 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d54f): Mirror 2: 0x4daf3a74 (stale) Mirror 4: 0x4daf3a74 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d54f: skipped 000989 1790780861.874609 trunc from 0x03d54f to 0x031d22 (0xb82c bytes) 000990 1790780861.908533 write 0x02d7b9 thru 0x0345fb (0x6e43 bytes) 000991 1790780861.919976 MIRROR_RESYNC 3 mirrors 000993 1790780863.178827 mapwrite 0x029256 thru 0x038f2a (0xfcd5 bytes) 000995 1790780863.414798 read 0x02f063 thru 0x032582 (0x3520 bytes) 000997 1790780863.415892 punch from 0x014ef6 to 0x022eb0 (0xdfbb bytes) 000998 1790780863.420822 fallocate 0x00398c thru 0x00efcc (0xb641 bytes) 000999 1790780863.429859 read 0x024c91 thru 0x029eb6 (0x5226 bytes) All operations completed A-OK! PASS 70a (110s) == sanity-flr test 71: check mirror extend parent fid ==== 11:07:49 (1790780869) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1a1:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 11:07:59 (1790780879) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.326963 s, 6.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779595 s, 13.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 11:08:09 (1790780889) Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre2 Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3499808 .. Extending file size to 5043712 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6972096 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7712928 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7788992 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8498208 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8271424 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8634016 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8638496 .. Extending file size to 9156000 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 61893 61894 61895 61899 61900 failed 192.168.202.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg224-client.virtnet /mnt/lustre2 (opts:) 192.168.202.124@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg224-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:09:29 (1790780969) Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre2 Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5179136 .. Extending file size to 4896736 .. Extending file size to 6635744 .. Extending file size to 7058336 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6890816 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8597696 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8390048 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8992544 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8713792 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 63407 63408 63410 63412 63414 192.168.202.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg224-client.virtnet /mnt/lustre2 (opts:) 192.168.202.124@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg224-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:10:46 (1790781046) Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2216 1269088 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1269140 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24384 3582636 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18740 3588280 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43124 7170916 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2216 1269088 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2164 1269140 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24384 3582636 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18740 3588280 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43124 7170916 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0864728 s, 24.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1636 s, 12.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xb0eec56 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg224-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:11:03 (1790781063) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0775736 s, 27.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 11:11:10 (1790781070) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xb4:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:11:19 (1790781079) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132675 s, 15.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1de:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:11:30 (1790781090) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114779 s, 18.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:11:41 (1790781101) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.200048 s, 10.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:12:15 (1790781135) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg224-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103454 s, 10.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x153:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x153:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x154:0x0] } PASS 204d (49s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:13:04 (1790781184) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg224-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.14725 s, 14.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:14:02 (1790781242) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg224-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg224-server: Pool lustre.test_204f created Adding targets to pool oleg224-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg224-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192255 s, 10.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x155:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x155:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg224-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg224-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg224-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:15:25 (1790781325) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:15:34 (1790781334) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:15:42 (1790781342) oleg224-server: Pool lustre.test_206 created Adding targets to pool oleg224-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg224-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg224-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg224-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg224-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:16:12 (1790781372) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571929 s, 18.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886619 s, 11.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x8a51841e Mirror 3: 0x8a51841e PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:16:24 (1790781384) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:16:28 (1790781388) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:16:32 (1790781392) Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0429164 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x910000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xd70000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xd70000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xd70000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1fa0000 Read this (4 bytes): BBBB 192.168.202.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg224-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:17:15 (1790781435) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0411091 s, 199 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:17:25 (1790781445) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696482 s, 15.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd5 lmm_fid: [0x200000405:0xd5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd5 lmm_fid: [0x200000405:0xd5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611349 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 11:17:27.000000000 -0400 Modify: 2026-09-30 11:17:27.000000000 -0400 Change: 2026-09-30 11:17:29.000000000 -0400 Birth: 2026-09-30 11:17:26.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:17:35 (1790781455) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302616 s, 34.7 MB/s oleg224-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg224-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg224-server: O/280000401/d30/15e: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:18:00 (1790781480) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.302639 s, 135 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0349586 s, 117 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 30 11:18 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xda:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 30 11:18 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:18:08 (1790781488) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:18:16 (1790781496) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1947 sec =========== 11:18:25 (1790781505) === sanity-flr: start cleanup 11:18:27 (1790781507) === === sanity-flr: finish cleanup 11:18:36 (1790781516) ===