-----============= acceptance-small: sanity-flr ============----- Mon Jun 1 13:19:00 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg411-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 13:19:19 (1780334359) === oleg411-client.virtnet: executing check_config_client /mnt/lustre oleg411-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg411-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b82191ac800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b82191ac800.idle_timeout=debug disable quota as required oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 13:19:52 (1780334392) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7573] == sanity-flr test 0a: lfs mirror create with -N option == 13:19:54 (1780334394) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:20:11 (1780334411) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:20:15 (1780334415) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:20:18 (1780334418) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:20:46 (1780334446) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:20:51 (1780334451) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 13:20:54 (1780334454) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b82191ac800.stats=clear osc.lustre-OST0001-osc-ffff9b82191ac800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:21:04 (1780334464) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:21:18 (1780334478) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:21:29 (1780334489) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:21:33 (1780334493) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:21:41 (1780334501) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 13:21:51 (1780334511) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:21:58 (1780334518) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:22:01 (1780334521) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:22:08 (1780334528) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:22:17 (1780334537) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:22:25 (1780334545) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.98703 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113254 s, 9.3 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:22:36 (1780334556) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.576556 s, 34.6 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:22:45 (1780334565) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:22:54 (1780334574) Stopping /mnt/lustre-ost1 (opts:) on oleg411-server oleg411-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b82191ac800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b82191ac800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg411-server: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg411-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg411-server oleg411-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b82191ac800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b82191ac800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg411-server: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg411-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (66s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:24:00 (1780334640) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.366144 s, 20.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg411-server oleg411-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b82191ac800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b82191ac800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg411-server: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg411-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (38s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:24:38 (1780334678) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg411-server oleg411-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b82191ac800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b82191ac800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg411-server: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg411-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg411-server oleg411-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b82191ac800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b82191ac800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg411-server: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg411-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (72s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:25:51 (1780334751) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.00271 s, 10.5 MB/s -rw-r--r-- 1 root root 52428800 Jun 1 13:25 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.376s user 0m0.000s sys 0m0.225s Stopping /mnt/lustre-ost1 (opts:) on oleg411-server oleg411-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b82191ac800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b82191ac800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.375s user 0m0.002s sys 0m0.154s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg411-server: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg411-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg411-server oleg411-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b82191ac800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b82191ac800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.377s user 0m0.000s sys 0m0.182s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg411-server: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg411-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b82191ac800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b82191ac800.ost_server_uuid in FULL state after 0 sec PASS 33b (91s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:27:22 (1780334842) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:27:25 (1780334845) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:27:29 (1780334849) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 13:27:32 (1780334852) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 13:27:41 (1780334861) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.492716 s, 25.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:27:55 (1780334875) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544931 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816629 s, 12.8 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:28:04 (1780334884) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:28:12 (1780334892) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000793324 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00148902 s, 688 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00587264 s, 174 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00300865 s, 340 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 13:28:25 (1780334905) create test files with size 15744264 .. 0e9650a213faacd57523c68c4826d567 - 089cdae8bf0a1ead3f45ca46ad5f832e - 44c9b9d45a73a70b9afff787914b43fe - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg411-client.virtnet /mnt/lustre (opts:) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 13:28:51 (1780334931) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:29:04 (1780334944) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:29:10 (1780334950) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273185 s, 3.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703992 s, 14.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 13:29:21 (1780334961) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150042 s, 14.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.348697 s, 12.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.246972 s, 8.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 13:29:39 (1780334979) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 13:29:44 (1780334984) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:29:48 (1780334988) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246857 s, 42.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 13:29:56 (1780334996) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:29:59 (1780334999) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:30:11 (1780335011) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.395224 s, 26.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg411-client.virtnet /mnt/lustre (opts:) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:30:23 (1780335023) llite.lustre-ffff9b82029f8800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.34428 s, 30.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg411-client.virtnet /mnt/lustre (opts:) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9b82181fd800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:30:39 (1780335039) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:31:17 (1780335077) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:31:52 (1780335112) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 13:31:55 (1780335115) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.240264 s, 13.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.1881 s, 16.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd6caeecb (nosync,stale) Mirror 2: 0xd6caeecb (nosync,stale) ** verify mirror 2 doesn't change original checksum: d277924559cec65a24c7116175cf54ed - mirror 1 checksum: af17e8eee0987a55190b9d1f904192b7 - mirror 2 checksum: d277924559cec65a24c7116175cf54ed - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 13:31:59 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 13:31:59 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d277924559cec65a24c7116175cf54ed - mirror 1 checksum: af17e8eee0987a55190b9d1f904192b7 - mirror 2 checksum: af17e8eee0987a55190b9d1f904192b7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 13:31:59 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 13:31:59 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:32:07 (1780335127) Failing mds1 on oleg411-server Stopping /mnt/lustre-mds1 (opts:) on oleg411-server 13:32:14 (1780335134) shut down facet: mds1 facet_host: oleg411-server facet_failover_host: oleg411-server Failover mds1 to oleg411-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-MDT0000 13:32:45 (1780335165) targets are mounted 13:32:45 (1780335165) facet_failover done oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295471 s, 35.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } PASS 50A (57s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:33:04 (1780335184) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:33:07 (1780335187) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:33:11 (1780335191) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0130898 s, 1.3 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:33:18 (1780335198) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:33:22 (1780335202) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:33:25 (1780335205) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:33:49 (1780335229) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:34:19 (1780335259) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:34:43 (1780335283) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:35:07 (1780335307) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 13:35:17 (1780335317) mirror create pid 53531 mirror split pid 53532 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53531 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53532 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 13:36:26 (1780335386) Seed set to 5388 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1780335388.511044 mapwrite 0x01ec64 thru 0x02154a (0x28e7 bytes) 000004 1780335388.764845 mapread 0x00624f thru 0x00f755 (0x9507 bytes) 000005 1780335388.787352 mapread 0x00b8d1 thru 0x00eca7 (0x33d7 bytes) 000006 1780335388.790667 mapwrite 0x012f92 thru 0x01c064 (0x90d3 bytes) 000007 1780335388.908865 mapread 0x001b62 thru 0x004da5 (0x3244 bytes) 000009 1780335388.916682 mapread 0x017645 thru 0x02154a (0x9f06 bytes) 000012 1780335388.924528 mapread 0x00a3c7 thru 0x0152ed (0xaf27 bytes) 000014 1780335388.929670 mapread 0x0093e9 thru 0x012173 (0x8d8b bytes) 000017 1780335388.934363 mapwrite 0x00f6f8 thru 0x00f82c (0x135 bytes) 000023 1780335389.016991 mapread 0x0118c7 thru 0x019f2d (0x8667 bytes) 000028 1780335389.032574 mapread 0x00e896 thru 0x010964 (0x20cf bytes) 000029 1780335389.035915 write 0x02da8a thru 0x030cdd (0x3254 bytes) truncating to largest ever: 0x3f0da 000031 1780335389.040202 trunc from 0x030cde to 0x03f0d9 (0xe3fc bytes) 000032 1780335389.076236 MIRROR_EXTEND to 2 mirrors 000033 1780335390.063762 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) 000034 1780335390.372484 mapwrite 0x011d72 thru 0x01b2ce (0x955d bytes) 000036 1780335390.592986 mapwrite 0x02d9aa thru 0x030583 (0x2bda bytes) 000040 1780335390.756218 read 0x0071a2 thru 0x013b28 (0xc987 bytes) 000041 1780335390.777056 read 0x01f830 thru 0x029cb3 (0xa484 bytes) 000042 1780335390.814478 MIRROR_EXTEND to 2 mirrors 000043 1780335391.941029 write 0x02b28b thru 0x02f727 (0x449d bytes) 000044 1780335392.086319 mapwrite 0x00d737 thru 0x019c15 (0xc4df bytes) 000045 1780335392.202658 trunc from 0x03f0da to 0x0264bc (0x18c1d bytes) 000046 1780335392.231826 read 0x00387f thru 0x0132cc (0xfa4e bytes) 000047 1780335392.243025 mapwrite 0x00bc4d thru 0x013ba7 (0x7f5b bytes) 000049 1780335392.338927 MIRROR_RESYNC 2 mirrors 000050 1780335393.713734 mapread 0x01d83d thru 0x0264bc (0x8c80 bytes) 000051 1780335393.746278 read 0x01cbe1 thru 0x02091e (0x3d3e bytes) 000052 1780335393.750048 read 0x00c4f4 thru 0x018780 (0xc28d bytes) 000054 1780335393.763540 mapread 0x009f69 thru 0x01540d (0xb4a5 bytes) 000057 1780335393.764969 trunc from 0x0264bd to 0x0256c5 (0xdf7 bytes) 000059 1780335394.013881 MIRROR_RESYNC 2 mirrors 000061 1780335395.066606 MIRROR_EXTEND to 3 mirrors 000062 1780335396.298126 mapwrite 0x002c8e thru 0x00e3ac (0xb71f bytes) 000063 1780335396.657155 mapwrite 0x00393e thru 0x012b1e (0xf1e1 bytes) 000068 1780335396.731327 trunc from 0x0256c6 to 0x037500 (0x11e3b bytes) 000069 1780335396.754202 trunc from 0x037501 to 0x035b1a (0x19e6 bytes) 000070 1780335396.782795 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35b1b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b1b): Mirror 2: 0x7d6811e7 (stale) Mirror 3: 0x7d6811e7 (stale) CRC-32 checksum value for chunk [0, 0x35b1b): Mirror 1: 0x59a03803 Mirror 4: 0x59a03803 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b1b: skipped 000072 1780335398.041975 mapread 0x01df17 thru 0x029935 (0xba1f bytes) 000076 1780335398.140300 write 0x006d02 thru 0x00ad98 (0x4097 bytes) 000077 1780335398.315646 read 0x00fc89 thru 0x017c55 (0x7fcd bytes) 000078 1780335398.343575 trunc from 0x035b1b to 0x0298e7 (0xc233 bytes) 000079 1780335398.398271 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x298e8): Mirror 2: 0xc21b6251 (stale) Mirror 3: 0xc21b6251 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x298e8): Mirror 2: 0xc21b6251 (stale) Mirror 3: 0xc21b6251 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x298e8: skipped 000089 1780335398.892636 write 0x008ca2 thru 0x013142 (0xa4a1 bytes) 000091 1780335398.910224 read 0x023b26 thru 0x02543d (0x1918 bytes) 000092 1780335398.917799 trunc from 0x0298e8 to 0x03cebc (0x135d5 bytes) 000093 1780335398.937421 write 0x01cef6 thru 0x023cc1 (0x6dcc bytes) 000095 1780335398.948582 read 0x0236ff thru 0x032317 (0xec19 bytes) 000096 1780335398.959567 read 0x000f6a thru 0x010192 (0xf229 bytes) 000098 1780335398.960994 trunc from 0x03cebd to 0x03d941 (0xa85 bytes) 000099 1780335398.980435 mapwrite 0x00b62e thru 0x011bfa (0x65cd bytes) 000100 1780335399.066574 write 0x004286 thru 0x0047c2 (0x53d bytes) 000101 1780335399.071165 trunc from 0x03d942 to 0x003e25 (0x39b1c bytes) 000102 1780335399.118556 mapread 0x003346 thru 0x003e25 (0xae0 bytes) 000104 1780335399.120285 trunc from 0x003e26 to 0x00e864 (0xaa3f bytes) 000109 1780335399.190052 mapwrite 0x02a799 thru 0x02acf2 (0x55a bytes) 000111 1780335399.433600 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2acf3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2acf3): Mirror 2: 0x20dcc7eb (stale) Mirror 3: 0x20dcc7eb (stale) CRC-32 checksum value for chunk [0, 0x2acf3): Mirror 1: 0x929bf37d Mirror 4: 0x929bf37d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2acf3: skipped 000114 1780335400.819474 MIRROR_RESYNC 4 mirrors 000115 1780335402.150583 MIRROR_RESYNC 4 mirrors 000116 1780335402.754352 MIRROR_EXTEND to 5 mirrors 000118 1780335403.611542 read 0x015552 thru 0x01b696 (0x6145 bytes) 000119 1780335403.644000 mapwrite 0x018587 thru 0x024e2d (0xc8a7 bytes) 000120 1780335403.874269 write 0x02a50a thru 0x02dedc (0x39d3 bytes) 000124 1780335403.879416 trunc from 0x02dedd to 0x005022 (0x28eba bytes) 000126 1780335403.942936 trunc from 0x005023 to 0x032624 (0x2d602 bytes) 000128 1780335403.972268 mapwrite 0x03f176 thru 0x03ffff (0xe8a bytes) 000129 1780335404.088548 read 0x002658 thru 0x0060f4 (0x3a9d bytes) 000130 1780335404.103599 mapread 0x03846d thru 0x03fc39 (0x77cd bytes) 000133 1780335404.114107 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc0062036 (stale) Mirror 3: 0xc0062036 (stale) Mirror 5: 0xc0062036 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc0062036 (stale) Mirror 3: 0xc0062036 (stale) Mirror 5: 0xc0062036 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000135 1780335404.622636 write 0x03c85f thru 0x03ffff (0x37a1 bytes) 000138 1780335404.642231 mapread 0x03d779 thru 0x03e71d (0xfa5 bytes) 000139 1780335404.644186 read 0x027238 thru 0x035586 (0xe34f bytes) 000140 1780335404.654908 trunc from 0x040000 to 0x037631 (0x89ce bytes) 000143 1780335404.695307 mapread 0x02d7e0 thru 0x033da3 (0x65c4 bytes) 000144 1780335404.707154 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37632): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37632): Mirror 2: 0x317c858f (stale) Mirror 3: 0x317c858f (stale) Mirror 5: 0x317c858f (stale) CRC-32 checksum value for chunk [0, 0x37632): Mirror 1: 0x8296db1b Mirror 6: 0x8296db1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37632: skipped 000145 1780335405.803241 write 0x008d03 thru 0x00a9ef (0x1ced bytes) 000146 1780335405.918348 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37632): Mirror 2: 0x317c858f (stale) Mirror 5: 0x317c858f (stale) Mirror 6: 0x8296db1b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37632): Mirror 2: 0x317c858f (stale) Mirror 5: 0x317c858f (stale) Mirror 6: 0x8296db1b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37632: skipped 000151 1780335406.458930 write 0x02edf7 thru 0x02fd59 (0xf63 bytes) 000152 1780335406.482211 read 0x016082 thru 0x01c3ab (0x632a bytes) 000153 1780335406.503074 read 0x02f96c thru 0x037631 (0x7cc6 bytes) 000155 1780335406.518426 mapread 0x01ea6d thru 0x02a312 (0xb8a6 bytes) 000157 1780335406.521101 mapread 0x0283f1 thru 0x02e3b6 (0x5fc6 bytes) 000159 1780335406.523730 read 0x01eeef thru 0x029962 (0xaa74 bytes) 000160 1780335406.531001 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37632): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37632): Mirror 2: 0x317c858f (stale) Mirror 5: 0x317c858f (stale) Mirror 6: 0x8296db1b (stale) CRC-32 checksum value for chunk [0, 0x37632): Mirror 1: 0x9ba251c3 Mirror 7: 0x9ba251c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37632: skipped 000163 1780335407.898399 trunc from 0x037632 to 0x02e647 (0x8fea bytes) 000164 1780335408.068671 mapread 0x026535 thru 0x02e647 (0x8113 bytes) 000165 1780335408.093192 MIRROR_RESYNC 5 mirrors 000167 1780335409.300855 read 0x00bab1 thru 0x00c7f5 (0xd45 bytes) 000169 1780335409.312559 mapwrite 0x02f009 thru 0x0312a5 (0x229d bytes) 000170 1780335409.567330 read 0x02e5f8 thru 0x0312a5 (0x2cae bytes) 000172 1780335409.585950 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x312a6): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x312a6): Mirror 2: 0x4e40f214 (stale) Mirror 5: 0x4e40f214 (stale) Mirror 6: 0x4e40f214 (stale) Mirror 7: 0x4e40f214 (stale) CRC-32 checksum value for chunk [0, 0x312a6): Mirror 1: 0xc336d7a1 Mirror 8: 0xc336d7a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x312a6: skipped 000173 1780335410.844763 read 0x02e3ff thru 0x0312a5 (0x2ea7 bytes) 000174 1780335410.868348 trunc from 0x0312a6 to 0x022486 (0xee1f bytes) 000175 1780335411.014936 mapwrite 0x02ad55 thru 0x030a37 (0x5ce3 bytes) 000176 1780335411.135018 read 0x009b24 thru 0x00e5ba (0x4a97 bytes) 000177 1780335411.158582 mapwrite 0x005b18 thru 0x0112d9 (0xb7c2 bytes) 000178 1780335411.292520 mapwrite 0x0130ad thru 0x0167e6 (0x373a bytes) 000179 1780335411.424020 read 0x00652f thru 0x009f57 (0x3a29 bytes) 000180 1780335411.426385 trunc from 0x030a38 to 0x007fae (0x28a89 bytes) 000181 1780335411.476925 trunc from 0x007faf to 0x001c88 (0x6326 bytes) 000183 1780335411.520590 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1c89): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c89): Mirror 2: 0x9433064a (stale) Mirror 5: 0x9433064a (stale) Mirror 6: 0x9433064a (stale) Mirror 7: 0x9433064a (stale) Mirror 8: 0x9433064a (stale) CRC-32 checksum value for chunk [0, 0x1c89): Mirror 1: 0x9433064a Mirror 9: 0x9433064a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c89: skipped truncating to largest ever: 0x3f8d3 000184 1780335412.406958 trunc from 0x001c89 to 0x03f8d2 (0x3dc4a bytes) 000185 1780335412.505989 read 0x0258bc thru 0x0279fc (0x2141 bytes) 000187 1780335412.517045 mapwrite 0x02c7ee thru 0x038e5d (0xc670 bytes) 000188 1780335412.630307 trunc from 0x03f8d3 to 0x02f0f3 (0x107df bytes) 000189 1780335412.666248 read 0x00d78a thru 0x00e8a9 (0x1120 bytes) 000190 1780335412.699755 mapwrite 0x00ecda thru 0x01294d (0x3c74 bytes) 000191 1780335412.750200 trunc from 0x02f0f4 to 0x02b76e (0x3985 bytes) 000192 1780335412.770312 write 0x031952 thru 0x03d0a7 (0xb756 bytes) 000195 1780335412.778299 mapwrite 0x024eab thru 0x0331b6 (0xe30c bytes) 000197 1780335412.870609 mapwrite 0x030dbe thru 0x036352 (0x5595 bytes) 000200 1780335412.946260 mapread 0x0145b8 thru 0x01f91d (0xb366 bytes) 000201 1780335412.950163 read 0x00ac61 thru 0x013051 (0x83f1 bytes) 000202 1780335412.958890 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d0a8): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d0a8): Mirror 2: 0x5ec77b88 (stale) Mirror 5: 0x5ec77b88 (stale) Mirror 6: 0x5ec77b88 (stale) Mirror 7: 0x5ec77b88 (stale) Mirror 8: 0xcdde1619 (stale) Mirror 9: 0x427827d4 (stale) CRC-32 checksum value for chunk [0, 0x3d0a8): Mirror 1: 0x70db4766 Mirror 10: 0x70db4766 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d0a8: skipped 000203 1780335414.175292 trunc from 0x03d0a8 to 0x039c6f (0x3438 bytes) 000207 1780335414.331417 trunc from 0x039c70 to 0x001d1d (0x37f52 bytes) 000209 1780335414.377892 mapread 0x000b64 thru 0x001d1d (0x11ba bytes) 000211 1780335414.398931 trunc from 0x001d1e to 0x010e25 (0xf108 bytes) 000212 1780335414.433642 read 0x008d9e thru 0x010e25 (0x8088 bytes) 000215 1780335414.464389 MIRROR_RESYNC 8 mirrors 000216 1780335416.367046 trunc from 0x010e26 to 0x03007a (0x1f255 bytes) 000218 1780335416.493617 MIRROR_RESYNC 8 mirrors 000220 1780335418.436612 trunc from 0x03007b to 0x01a898 (0x157e2 bytes) 000222 1780335418.546643 mapread 0x0116fc thru 0x01a898 (0x919d bytes) 000223 1780335418.580957 read 0x0134ea thru 0x018a23 (0x553a bytes) 000225 1780335418.584072 trunc from 0x01a899 to 0x01ac7d (0x3e5 bytes) 000226 1780335418.623163 mapread 0x003afc thru 0x00ca7a (0x8f7f bytes) 000228 1780335418.653626 MIRROR_RESYNC 8 mirrors 000232 1780335420.329608 mapwrite 0x0341f4 thru 0x03ffff (0xbe0c bytes) 000233 1780335420.584963 read 0x02269a thru 0x02a2d5 (0x7c3c bytes) 000234 1780335420.596846 mapwrite 0x0299d9 thru 0x02d20b (0x3833 bytes) 000235 1780335420.717880 mapread 0x0143c0 thru 0x017e37 (0x3a78 bytes) 000238 1780335420.741215 mapwrite 0x00ea6d thru 0x01089d (0x1e31 bytes) 000243 1780335420.833951 write 0x02fd63 thru 0x0304af (0x74d bytes) 000244 1780335420.840199 mapwrite 0x03a21e thru 0x03ffff (0x5de2 bytes) 000245 1780335420.936395 write 0x02d65d thru 0x035e7e (0x8822 bytes) 000246 1780335420.954132 MIRROR_RESYNC 8 mirrors 000247 1780335423.132339 MIRROR_RESYNC 8 mirrors 000248 1780335423.672814 mapwrite 0x00cf5d thru 0x016865 (0x9909 bytes) 000250 1780335423.927916 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8f01349d (stale) Mirror 5: 0x8f01349d (stale) Mirror 6: 0x8f01349d (stale) Mirror 7: 0x8f01349d (stale) Mirror 8: 0x8f01349d (stale) Mirror 9: 0x8f01349d (stale) Mirror 10: 0x8f01349d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbbc69838 Mirror 11: 0xbbc69838 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000252 1780335425.296144 read 0x004bc0 thru 0x013487 (0xe8c8 bytes) 000253 1780335425.324583 mapwrite 0x00d43d thru 0x01426c (0x6e30 bytes) 000254 1780335425.618856 read 0x01ebe8 thru 0x025644 (0x6a5d bytes) 000257 1780335425.637452 mapread 0x03f97c thru 0x03ffff (0x684 bytes) 000258 1780335425.638463 read 0x00ef49 thru 0x00fd45 (0xdfd bytes) 000260 1780335425.655664 MIRROR_RESYNC 9 mirrors 000261 1780335427.453022 read 0x010ec1 thru 0x01edaa (0xdeea bytes) 000262 1780335427.474600 write 0x012b78 thru 0x01abc6 (0x804f bytes) 000263 1780335427.598052 mapread 0x024541 thru 0x0269da (0x249a bytes) 000265 1780335427.630273 trunc from 0x040000 to 0x01cf76 (0x23089 bytes) 000268 1780335427.678955 trunc from 0x01cf77 to 0x038cee (0x1bd78 bytes) 000269 1780335427.740866 mapwrite 0x004b02 thru 0x00f3ec (0xa8eb bytes) 000271 1780335427.847162 write 0x02305a thru 0x02b57b (0x8522 bytes) 000273 1780335427.864879 mapread 0x03426f thru 0x034c33 (0x9c5 bytes) 000275 1780335427.877832 read 0x027872 thru 0x030b63 (0x92f2 bytes) 000276 1780335427.890798 read 0x021d8a thru 0x02c649 (0xa8c0 bytes) 000277 1780335427.891666 mapwrite 0x0074cd thru 0x00bd2f (0x4863 bytes) 000278 1780335427.980920 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x38cef): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38cef): Mirror 2: 0x3da686d1 (stale) Mirror 5: 0x3da686d1 (stale) Mirror 6: 0x3da686d1 (stale) Mirror 7: 0x3da686d1 (stale) Mirror 8: 0x3da686d1 (stale) Mirror 9: 0x3da686d1 (stale) Mirror 10: 0x3da686d1 (stale) Mirror 11: 0x3da686d1 (stale) CRC-32 checksum value for chunk [0, 0x38cef): Mirror 1: 0x2fc2f9ee Mirror 12: 0x2fc2f9ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38cef: skipped 000281 1780335429.306368 write 0x031260 thru 0x031989 (0x72a bytes) 000282 1780335429.436777 read 0x03132a thru 0x038cee (0x79c5 bytes) 000283 1780335429.446780 write 0x03e017 thru 0x03ffff (0x1fe9 bytes) 000284 1780335429.447919 read 0x019e30 thru 0x024594 (0xa765 bytes) 000285 1780335429.492230 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xca30f91a (stale) Mirror 5: 0xca30f91a (stale) Mirror 6: 0xca30f91a (stale) Mirror 7: 0xca30f91a (stale) Mirror 8: 0xca30f91a (stale) Mirror 9: 0xca30f91a (stale) Mirror 10: 0xca30f91a (stale) Mirror 11: 0xca30f91a (stale) Mirror 12: 0x19ade7fd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8c56feb Mirror 13: 0x8c56feb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000287 1780335430.670901 mapwrite 0x032634 thru 0x032729 (0xf6 bytes) 000288 1780335430.842334 trunc from 0x040000 to 0x02c0ce (0x13f31 bytes) 000290 1780335430.898056 trunc from 0x02c0cf to 0x0348df (0x8811 bytes) 000292 1780335430.921065 read 0x027d1e thru 0x02c187 (0x446a bytes) 000293 1780335430.931811 mapread 0x000daa thru 0x00e3c7 (0xd61e bytes) 000295 1780335430.946221 read 0x024ebd thru 0x027dee (0x2f32 bytes) 000296 1780335430.957683 write 0x008c6f thru 0x013596 (0xa928 bytes) 000298 1780335430.960586 mapread 0x007ba3 thru 0x00a144 (0x25a2 bytes) 000299 1780335430.962124 mapread 0x02ccfe thru 0x0348df (0x7be2 bytes) 000300 1780335430.963354 write 0x00eb6d thru 0x01db97 (0xf02b bytes) 000301 1780335430.967028 write 0x017885 thru 0x01a63e (0x2dba bytes) 000302 1780335430.968165 read 0x02e773 thru 0x032c2d (0x44bb bytes) 000303 1780335430.969958 mapread 0x0100ab thru 0x01bbce (0xbb24 bytes) 000304 1780335430.971595 mapread 0x01161c thru 0x016b63 (0x5548 bytes) 000307 1780335430.974671 mapread 0x01492b thru 0x02277d (0xde53 bytes) 000308 1780335430.976449 trunc from 0x0348e0 to 0x026bda (0xdd05 bytes) 000309 1780335431.017159 mapread 0x01cdb1 thru 0x023864 (0x6ab4 bytes) 000311 1780335431.019240 read 0x0161ca thru 0x017c1b (0x1a52 bytes) 000313 1780335431.029644 mapwrite 0x01caa0 thru 0x023ae1 (0x7042 bytes) 000314 1780335431.137693 mapread 0x00968b thru 0x013b4c (0xa4c2 bytes) 000315 1780335431.139540 write 0x013fb0 thru 0x01a3da (0x642b bytes) 000316 1780335431.141153 read 0x00a45a thru 0x015d2d (0xb8d4 bytes) 000317 1780335431.142086 mapwrite 0x02a9f3 thru 0x0365b0 (0xbbbe bytes) 000320 1780335431.279776 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x365b1): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x365b1): Mirror 2: 0xa6bd7c43 (stale) Mirror 5: 0xa6bd7c43 (stale) Mirror 6: 0xa6bd7c43 (stale) Mirror 7: 0xa6bd7c43 (stale) Mirror 8: 0xa6bd7c43 (stale) Mirror 9: 0xa6bd7c43 (stale) Mirror 10: 0xa6bd7c43 (stale) Mirror 11: 0xa6bd7c43 (stale) Mirror 12: 0x705d1706 (stale) Mirror 13: 0xb622f92e (stale) CRC-32 checksum value for chunk [0, 0x365b1): Mirror 1: 0xbb222de6 Mirror 14: 0xbb222de6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x365b1: skipped 000322 1780335432.442226 read 0x0111af thru 0x014ead (0x3cff bytes) 000323 1780335432.453231 mapread 0x01dff8 thru 0x0274e3 (0x94ec bytes) 000324 1780335432.482138 write 0x03cfb8 thru 0x03faff (0x2b48 bytes) 000325 1780335432.571581 mapread 0x008dbe thru 0x0105ad (0x77f0 bytes) 000326 1780335432.584993 read 0x037323 thru 0x03a777 (0x3455 bytes) 000327 1780335432.598441 mapwrite 0x0157b3 thru 0x020efa (0xb748 bytes) 000328 1780335432.694203 read 0x03d00e thru 0x03faff (0x2af2 bytes) 000329 1780335432.698905 write 0x00ebe8 thru 0x01a154 (0xb56d bytes) 000333 1780335432.706901 write 0x030cc6 thru 0x032019 (0x1354 bytes) 000336 1780335432.710445 trunc from 0x03fb00 to 0x027a1b (0x180e4 bytes) 000337 1780335432.771678 mapread 0x0215e3 thru 0x027a1b (0x6439 bytes) 000338 1780335432.774181 read 0x00073e thru 0x00722d (0x6af0 bytes) 000339 1780335432.776616 mapwrite 0x001811 thru 0x009d88 (0x8578 bytes) 000340 1780335432.834908 mapwrite 0x004d5c thru 0x00e838 (0x9add bytes) 000341 1780335432.895297 write 0x0015f1 thru 0x00de5e (0xc86e bytes) 000342 1780335432.896664 read 0x01666a thru 0x01c483 (0x5e1a bytes) 000343 1780335432.897343 trunc from 0x027a1c to 0x02fbcf (0x81b4 bytes) 000346 1780335432.923657 MIRROR_RESYNC 12 mirrors 000347 1780335434.991972 mapwrite 0x00f7e8 thru 0x01e08e (0xe8a7 bytes) 000348 1780335435.195103 trunc from 0x02fbd0 to 0x01b351 (0x1487e bytes) 000349 1780335435.231657 mapread 0x012f6e thru 0x018245 (0x52d8 bytes) 000350 1780335435.241662 MIRROR_RESYNC 12 mirrors 000351 1780335437.072819 mapread 0x000014 thru 0x00e0d7 (0xe0c4 bytes) 000352 1780335437.115635 write 0x016402 thru 0x01cea1 (0x6aa0 bytes) 000354 1780335437.224830 mapread 0x00e2d5 thru 0x0166d4 (0x8400 bytes) 000356 1780335437.248632 write 0x02c687 thru 0x0318cb (0x5245 bytes) 000357 1780335437.251883 trunc from 0x0318cc to 0x00d082 (0x24849 bytes) 000358 1780335437.286935 mapread 0x004fae thru 0x00b666 (0x66b9 bytes) 000361 1780335437.302167 write 0x00a3d7 thru 0x00a5de (0x208 bytes) 000362 1780335437.305388 write 0x01ab62 thru 0x024a20 (0x9ebf bytes) 000364 1780335437.308514 mapwrite 0x0072e6 thru 0x01711f (0xfe3a bytes) 000367 1780335437.461914 mapread 0x01328a thru 0x014944 (0x16bb bytes) 000369 1780335437.468694 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24a21): Mirror 2: 0x3ab480a8 (stale) Mirror 5: 0x3ab480a8 (stale) Mirror 6: 0x3ab480a8 (stale) Mirror 7: 0x3ab480a8 (stale) Mirror 9: 0x3ab480a8 (stale) Mirror 10: 0x3ab480a8 (stale) Mirror 11: 0x3ab480a8 (stale) Mirror 12: 0x3ab480a8 (stale) Mirror 13: 0x3ab480a8 (stale) Mirror 14: 0x3ab480a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24a21): Mirror 2: 0x3ab480a8 (stale) Mirror 5: 0x3ab480a8 (stale) Mirror 6: 0x3ab480a8 (stale) Mirror 7: 0x3ab480a8 (stale) Mirror 9: 0x3ab480a8 (stale) Mirror 10: 0x3ab480a8 (stale) Mirror 11: 0x3ab480a8 (stale) Mirror 12: 0x3ab480a8 (stale) Mirror 13: 0x3ab480a8 (stale) Mirror 14: 0x3ab480a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24a21: skipped 000372 1780335438.180090 MIRROR_RESYNC 11 mirrors 000377 1780335439.862572 trunc from 0x024a21 to 0x0184fa (0xc526 bytes) 000379 1780335439.969019 trunc from 0x0184fb to 0x01382a (0x4cd0 bytes) 000380 1780335439.989144 write 0x036173 thru 0x03ffff (0x9e8d bytes) 000382 1780335439.992101 write 0x0099ed thru 0x010ff8 (0x760c bytes) 000384 1780335440.014384 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x286be6a (stale) Mirror 5: 0x286be6a (stale) Mirror 6: 0x286be6a (stale) Mirror 7: 0x286be6a (stale) Mirror 9: 0x286be6a (stale) Mirror 10: 0x286be6a (stale) Mirror 11: 0x286be6a (stale) Mirror 12: 0x286be6a (stale) Mirror 13: 0x286be6a (stale) Mirror 14: 0x286be6a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1bb6c0e Mirror 15: 0x1bb6c0e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000385 1780335441.180908 mapread 0x0214db thru 0x02b1ec (0x9d12 bytes) 000388 1780335441.240586 trunc from 0x040000 to 0x037815 (0x87ea bytes) 000391 1780335441.405250 read 0x032425 thru 0x037815 (0x53f1 bytes) 000392 1780335441.418864 mapwrite 0x025523 thru 0x029f7d (0x4a5b bytes) 000394 1780335441.583893 read 0x01ecbc thru 0x02c27c (0xd5c1 bytes) 000395 1780335441.587445 write 0x00ddc1 thru 0x010b1a (0x2d5a bytes) 000396 1780335441.605006 mapwrite 0x0332d3 thru 0x03a1ee (0x6f1c bytes) 000397 1780335441.708484 mapwrite 0x00bb59 thru 0x01ad1f (0xf1c7 bytes) 000398 1780335441.773254 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3a1ef): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1ef): Mirror 2: 0x3e896e67 (stale) Mirror 5: 0x3e896e67 (stale) Mirror 6: 0x3e896e67 (stale) Mirror 7: 0x3e896e67 (stale) Mirror 9: 0x3e896e67 (stale) Mirror 10: 0x3e896e67 (stale) Mirror 11: 0x3e896e67 (stale) Mirror 12: 0x3e896e67 (stale) Mirror 13: 0x3e896e67 (stale) Mirror 14: 0x3e896e67 (stale) Mirror 15: 0x23fa12fe (stale) CRC-32 checksum value for chunk [0, 0x3a1ef): Mirror 1: 0x46e68ea4 Mirror 16: 0x46e68ea4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1ef: skipped 000401 1780335443.144993 trunc from 0x03a1ef to 0x023be7 (0x16607 bytes) 000402 1780335443.375008 trunc from 0x023be8 to 0x0277fd (0x3c16 bytes) 000404 1780335443.437987 read 0x00df0e thru 0x0103ec (0x24df bytes) 000405 1780335443.447339 write 0x0312dc thru 0x03a258 (0x8f7d bytes) 000407 1780335443.449365 mapread 0x01ba8c thru 0x01dd45 (0x22ba bytes) 000408 1780335443.480296 mapwrite 0x033d1c thru 0x038674 (0x4959 bytes) 000411 1780335443.597427 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a259): Mirror 2: 0x4b74de09 (stale) Mirror 5: 0x4b74de09 (stale) Mirror 6: 0x4b74de09 (stale) Mirror 7: 0x4b74de09 (stale) Mirror 9: 0x4b74de09 (stale) Mirror 10: 0x4b74de09 (stale) Mirror 11: 0x4b74de09 (stale) Mirror 12: 0x4b74de09 (stale) Mirror 13: 0x4b74de09 (stale) Mirror 14: 0x4b74de09 (stale) Mirror 16: 0x9754b541 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a259): Mirror 2: 0x4b74de09 (stale) Mirror 5: 0x4b74de09 (stale) Mirror 6: 0x4b74de09 (stale) Mirror 7: 0x4b74de09 (stale) Mirror 9: 0x4b74de09 (stale) Mirror 10: 0x4b74de09 (stale) Mirror 11: 0x4b74de09 (stale) Mirror 12: 0x4b74de09 (stale) Mirror 13: 0x4b74de09 (stale) Mirror 14: 0x4b74de09 (stale) Mirror 16: 0x9754b541 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a259: skipped 000413 1780335444.373921 write 0x026d28 thru 0x02aaa5 (0x3d7e bytes) 000414 1780335444.385955 mapread 0x013c7d thru 0x01ad69 (0x70ed bytes) 000415 1780335444.415693 MIRROR_RESYNC 12 mirrors 000417 1780335446.942746 write 0x01b2e7 thru 0x0248e7 (0x9601 bytes) 000418 1780335447.072185 read 0x01508f thru 0x022d06 (0xdc78 bytes) 000419 1780335447.093604 write 0x014c16 thru 0x019102 (0x44ed bytes) 000420 1780335447.114783 mapwrite 0x039e7d thru 0x03ffff (0x6183 bytes) 000424 1780335447.296980 mapwrite 0x02dffb thru 0x03a7d2 (0xc7d8 bytes) 000427 1780335447.438641 read 0x00480b thru 0x00cf0f (0x8705 bytes) 000428 1780335447.451853 write 0x012527 thru 0x01f239 (0xcd13 bytes) 000429 1780335447.459938 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1abe0040 (stale) Mirror 5: 0x1abe0040 (stale) Mirror 6: 0x1abe0040 (stale) Mirror 7: 0x1abe0040 (stale) Mirror 9: 0x1abe0040 (stale) Mirror 10: 0x1abe0040 (stale) Mirror 11: 0x1abe0040 (stale) Mirror 12: 0x1abe0040 (stale) Mirror 13: 0x1abe0040 (stale) Mirror 14: 0x1abe0040 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1abe0040 (stale) Mirror 5: 0x1abe0040 (stale) Mirror 6: 0x1abe0040 (stale) Mirror 7: 0x1abe0040 (stale) Mirror 9: 0x1abe0040 (stale) Mirror 10: 0x1abe0040 (stale) Mirror 11: 0x1abe0040 (stale) Mirror 12: 0x1abe0040 (stale) Mirror 13: 0x1abe0040 (stale) Mirror 14: 0x1abe0040 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000431 1780335448.427338 write 0x01e912 thru 0x024783 (0x5e72 bytes) 000432 1780335448.454071 trunc from 0x040000 to 0x03550b (0xaaf4 bytes) 000433 1780335448.488978 mapread 0x00ea46 thru 0x0161ef (0x77aa bytes) 000434 1780335448.515257 trunc from 0x03550c to 0x02d8d9 (0x7c32 bytes) 000435 1780335448.569997 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2d8da): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d8da): Mirror 2: 0x454e84bd (stale) Mirror 5: 0x454e84bd (stale) Mirror 6: 0x454e84bd (stale) Mirror 7: 0x454e84bd (stale) Mirror 9: 0x454e84bd (stale) Mirror 10: 0x454e84bd (stale) Mirror 11: 0x454e84bd (stale) Mirror 12: 0x454e84bd (stale) Mirror 13: 0x454e84bd (stale) Mirror 14: 0x454e84bd (stale) CRC-32 checksum value for chunk [0, 0x2d8da): Mirror 1: 0x3f49ed14 Mirror 15: 0x3f49ed14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d8da: skipped 000436 1780335450.194984 read 0x00e463 thru 0x01243a (0x3fd8 bytes) 000438 1780335450.269310 mapread 0x0072d9 thru 0x015b6c (0xe894 bytes) 000448 1780335450.492969 trunc from 0x02d8da to 0x018160 (0x15779 bytes) 000452 1780335450.686431 write 0x00bd97 thru 0x0160b6 (0xa320 bytes) 000453 1780335450.731486 mapwrite 0x001da9 thru 0x00c3cc (0xa624 bytes) 000454 1780335450.919530 trunc from 0x018161 to 0x0176cf (0xa91 bytes) 000455 1780335450.975221 write 0x000c86 thru 0x004dbc (0x4137 bytes) 000456 1780335450.990786 read 0x00edfc thru 0x0176cf (0x88d4 bytes) 000458 1780335451.005234 write 0x00b3fd thru 0x01426c (0x8e70 bytes) 000460 1780335451.007161 read 0x00ad61 thru 0x0176cf (0xc96f bytes) 000461 1780335451.014543 write 0x02796d thru 0x02dd10 (0x63a4 bytes) 000462 1780335451.016632 mapwrite 0x033a0b thru 0x03e02c (0xa622 bytes) 000466 1780335451.162389 write 0x0377df thru 0x037ecf (0x6f1 bytes) 000467 1780335451.163866 trunc from 0x03e02d to 0x002f9f (0x3b08d bytes) 000468 1780335451.184959 read 0x001d06 thru 0x002f9f (0x129a bytes) 000469 1780335451.192200 MIRROR_RESYNC 12 mirrors 000470 1780335453.232860 mapwrite 0x02f7cd thru 0x036440 (0x6c74 bytes) 000471 1780335453.427591 mapwrite 0x02a236 thru 0x031d0a (0x7ad5 bytes) 000472 1780335453.494916 MIRROR_RESYNC 12 mirrors 000475 1780335455.263363 mapwrite 0x03f53f thru 0x03ffff (0xac1 bytes) 000476 1780335455.516441 read 0x00ea77 thru 0x0138cc (0x4e56 bytes) 000478 1780335455.549231 write 0x02ed36 thru 0x032848 (0x3b13 bytes) 000479 1780335455.583747 mapread 0x035f3b thru 0x03ffff (0xa0c5 bytes) 000481 1780335455.610115 read 0x03e9a3 thru 0x03ffff (0x165d bytes) 000482 1780335455.610889 read 0x02b69a thru 0x03768a (0xbff1 bytes) 000485 1780335455.614107 mapread 0x036548 thru 0x03c112 (0x5bcb bytes) 000486 1780335455.615404 mapread 0x02b58d thru 0x0338cc (0x8340 bytes) 000487 1780335455.617458 mapwrite 0x012377 thru 0x017fa8 (0x5c32 bytes) 000488 1780335455.769187 write 0x018cb1 thru 0x01d1ac (0x44fc bytes) 000492 1780335455.780564 trunc from 0x040000 to 0x003e0c (0x3c1f3 bytes) 000494 1780335455.830338 trunc from 0x003e0d to 0x03f2f5 (0x3b4e9 bytes) 000496 1780335455.870568 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3f2f6): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2f6): Mirror 2: 0x14e0c21d (stale) Mirror 5: 0x14e0c21d (stale) Mirror 6: 0x14e0c21d (stale) Mirror 7: 0x14e0c21d (stale) Mirror 9: 0x14e0c21d (stale) Mirror 10: 0x14e0c21d (stale) Mirror 11: 0x14e0c21d (stale) Mirror 12: 0x14e0c21d (stale) Mirror 13: 0x14e0c21d (stale) Mirror 14: 0x14e0c21d (stale) Mirror 15: 0x14e0c21d (stale) CRC-32 checksum value for chunk [0, 0x3f2f6): Mirror 1: 0xf65092d Mirror 16: 0xf65092d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2f6: skipped 000502 1780335457.199081 trunc from 0x03f2f6 to 0x003b2a (0x3b7cb bytes) 000504 1780335457.306857 mapwrite 0x0018ed thru 0x00396d (0x2081 bytes) 000505 1780335457.422246 MIRROR_RESYNC 13 mirrors 000506 1780335459.455073 mapwrite 0x017cff thru 0x01e982 (0x6c84 bytes) 000507 1780335459.690633 mapwrite 0x020e58 thru 0x025839 (0x49e2 bytes) 000508 1780335459.792406 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2583a): Mirror 5: 0xa221f80a (stale) Mirror 6: 0xa221f80a (stale) Mirror 7: 0xa221f80a (stale) Mirror 9: 0xa221f80a (stale) Mirror 10: 0xa221f80a (stale) Mirror 11: 0xa221f80a (stale) Mirror 12: 0xa221f80a (stale) Mirror 13: 0xa221f80a (stale) Mirror 14: 0xa221f80a (stale) Mirror 15: 0xa221f80a (stale) Mirror 16: 0xa221f80a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2583a): Mirror 5: 0xa221f80a (stale) Mirror 6: 0xa221f80a (stale) Mirror 7: 0xa221f80a (stale) Mirror 9: 0xa221f80a (stale) Mirror 10: 0xa221f80a (stale) Mirror 11: 0xa221f80a (stale) Mirror 12: 0xa221f80a (stale) Mirror 13: 0xa221f80a (stale) Mirror 14: 0xa221f80a (stale) Mirror 15: 0xa221f80a (stale) Mirror 16: 0xa221f80a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2583a: skipped 000510 1780335460.565616 write 0x005ce8 thru 0x012da0 (0xd0b9 bytes) 000511 1780335460.612857 mapwrite 0x02d1ff thru 0x038474 (0xb276 bytes) 000512 1780335460.721575 write 0x024d65 thru 0x02839f (0x363b bytes) 000513 1780335460.745315 mapread 0x027071 thru 0x034467 (0xd3f7 bytes) 000514 1780335460.746766 trunc from 0x038475 to 0x0126f8 (0x25d7c bytes) 000515 1780335460.776050 mapread 0x00cbde thru 0x012368 (0x578b bytes) 000516 1780335460.801982 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x126f9): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x126f9): Mirror 5: 0x6d7c848 (stale) Mirror 6: 0x6d7c848 (stale) Mirror 7: 0x6d7c848 (stale) Mirror 9: 0x6d7c848 (stale) Mirror 10: 0x6d7c848 (stale) Mirror 11: 0x6d7c848 (stale) Mirror 12: 0x6d7c848 (stale) Mirror 13: 0x6d7c848 (stale) Mirror 14: 0x6d7c848 (stale) Mirror 15: 0x6d7c848 (stale) Mirror 16: 0x6d7c848 (stale) CRC-32 checksum value for chunk [0, 0x126f9): Mirror 1: 0x97138bc0 Mirror 17: 0x97138bc0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x126f9: skipped 000517 1780335462.497050 write 0x03775b thru 0x03ffff (0x88a5 bytes) 000518 1780335462.631336 trunc from 0x040000 to 0x019099 (0x26f66 bytes) 000521 1780335462.689372 mapread 0x016d00 thru 0x019099 (0x239a bytes) 000522 1780335462.719976 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1909a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1909a): Mirror 5: 0x4d302826 (stale) Mirror 6: 0x4d302826 (stale) Mirror 7: 0x4d302826 (stale) Mirror 9: 0x4d302826 (stale) Mirror 10: 0x4d302826 (stale) Mirror 11: 0x4d302826 (stale) Mirror 12: 0x4d302826 (stale) Mirror 13: 0x4d302826 (stale) Mirror 14: 0x4d302826 (stale) Mirror 15: 0x4d302826 (stale) Mirror 16: 0x4d302826 (stale) Mirror 17: 0x2f83589c (stale) CRC-32 checksum value for chunk [0, 0x1909a): Mirror 1: 0x2f83589c Mirror 18: 0x2f83589c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1909a: skipped 000523 1780335464.227404 read 0x0175b7 thru 0x019099 (0x1ae3 bytes) 000524 1780335464.262589 write 0x0022bf thru 0x005b37 (0x3879 bytes) 000526 1780335464.447133 MIRROR_SPLIT mirror 11 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1909a): Mirror 5: 0x4d302826 (stale) Mirror 6: 0x4d302826 (stale) Mirror 7: 0x4d302826 (stale) Mirror 9: 0x4d302826 (stale) Mirror 10: 0x4d302826 (stale) Mirror 11: 0x4d302826 (stale) Mirror 12: 0x4d302826 (stale) Mirror 13: 0x4d302826 (stale) Mirror 14: 0x4d302826 (stale) Mirror 15: 0x4d302826 (stale) Mirror 17: 0x2f83589c (stale) Mirror 18: 0x2f83589c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1909a): Mirror 5: 0x4d302826 (stale) Mirror 6: 0x4d302826 (stale) Mirror 7: 0x4d302826 (stale) Mirror 9: 0x4d302826 (stale) Mirror 10: 0x4d302826 (stale) Mirror 11: 0x4d302826 (stale) Mirror 12: 0x4d302826 (stale) Mirror 13: 0x4d302826 (stale) Mirror 14: 0x4d302826 (stale) Mirror 15: 0x4d302826 (stale) Mirror 17: 0x2f83589c (stale) Mirror 18: 0x2f83589c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1909a: skipped 000527 1780335465.453579 MIRROR_RESYNC 13 mirrors 000530 1780335468.111536 trunc from 0x01909a to 0x0102c6 (0x8dd3 bytes) 000532 1780335468.236390 read 0x001e8a thru 0x00d677 (0xb7ee bytes) 000533 1780335468.248964 write 0x03cb1e thru 0x03ffff (0x34e2 bytes) 000534 1780335468.250476 trunc from 0x040000 to 0x01180c (0x2e7f3 bytes) 000535 1780335468.288996 write 0x003c51 thru 0x0117a6 (0xdb56 bytes) 000536 1780335468.297337 trunc from 0x01180d to 0x03947e (0x27c72 bytes) 000537 1780335468.323584 mapwrite 0x00624d thru 0x014af2 (0xe8a6 bytes) 000538 1780335468.462797 MIRROR_RESYNC 13 mirrors 000539 1780335471.438446 mapwrite 0x000d32 thru 0x00ade0 (0xa0af bytes) 000542 1780335471.686640 mapwrite 0x033119 thru 0x03c487 (0x936f bytes) 000543 1780335471.852828 trunc from 0x03c488 to 0x0149e9 (0x27a9e bytes) 000545 1780335471.885320 write 0x02fe9a thru 0x03e508 (0xe66f bytes) 000550 1780335471.894460 mapwrite 0x000d62 thru 0x010cb1 (0xff50 bytes) 000551 1780335472.030380 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e509): Mirror 5: 0xb379745b (stale) Mirror 6: 0xb379745b (stale) Mirror 9: 0xb379745b (stale) Mirror 10: 0xb379745b (stale) Mirror 11: 0xb379745b (stale) Mirror 12: 0xb379745b (stale) Mirror 13: 0xb379745b (stale) Mirror 14: 0xb379745b (stale) Mirror 15: 0xb379745b (stale) Mirror 17: 0xb379745b (stale) Mirror 18: 0xb379745b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e509): Mirror 5: 0xb379745b (stale) Mirror 6: 0xb379745b (stale) Mirror 9: 0xb379745b (stale) Mirror 10: 0xb379745b (stale) Mirror 11: 0xb379745b (stale) Mirror 12: 0xb379745b (stale) Mirror 13: 0xb379745b (stale) Mirror 14: 0xb379745b (stale) Mirror 15: 0xb379745b (stale) Mirror 17: 0xb379745b (stale) Mirror 18: 0xb379745b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e509: skipped 000552 1780335472.927335 write 0x00d72c thru 0x014f16 (0x77eb bytes) 000553 1780335472.940416 trunc from 0x03e509 to 0x02e14e (0x103ba bytes) 000554 1780335472.967266 mapread 0x01c473 thru 0x02a449 (0xdfd7 bytes) 000555 1780335473.022077 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e14f): Mirror 5: 0x97f2187b (stale) Mirror 6: 0x97f2187b (stale) Mirror 9: 0x97f2187b (stale) Mirror 10: 0x97f2187b (stale) Mirror 11: 0x97f2187b (stale) Mirror 12: 0x97f2187b (stale) Mirror 13: 0x97f2187b (stale) Mirror 14: 0x97f2187b (stale) Mirror 15: 0x97f2187b (stale) Mirror 18: 0x97f2187b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e14f): Mirror 5: 0x97f2187b (stale) Mirror 6: 0x97f2187b (stale) Mirror 9: 0x97f2187b (stale) Mirror 10: 0x97f2187b (stale) Mirror 11: 0x97f2187b (stale) Mirror 12: 0x97f2187b (stale) Mirror 13: 0x97f2187b (stale) Mirror 14: 0x97f2187b (stale) Mirror 15: 0x97f2187b (stale) Mirror 18: 0x97f2187b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e14f: skipped 000556 1780335473.881515 write 0x03ef16 thru 0x03ffff (0x10ea bytes) 000558 1780335473.887159 read 0x03ce3a thru 0x03d562 (0x729 bytes) 000561 1780335473.902506 mapwrite 0x03524b thru 0x03ffff (0xadb5 bytes) 000563 1780335474.081075 mapread 0x03a932 thru 0x03ffff (0x56ce bytes) 000564 1780335474.083022 mapread 0x028ac5 thru 0x02e11b (0x5657 bytes) 000566 1780335474.095858 read 0x005b8b thru 0x0060a7 (0x51d bytes) 000571 1780335474.118998 trunc from 0x040000 to 0x0074e3 (0x38b1c bytes) 000573 1780335474.141484 mapread 0x006fa1 thru 0x0074e3 (0x543 bytes) 000575 1780335474.144777 trunc from 0x0074e4 to 0x0254eb (0x1e008 bytes) 000576 1780335474.172604 read 0x007b99 thru 0x007e0a (0x272 bytes) 000577 1780335474.197867 MIRROR_RESYNC 11 mirrors 000580 1780335476.112867 read 0x02135a thru 0x0254eb (0x4192 bytes) 000581 1780335476.122527 trunc from 0x0254ec to 0x0234df (0x200c bytes) 000582 1780335476.239516 mapwrite 0x0398c3 thru 0x03ffff (0x673d bytes) 000585 1780335476.357183 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x815b96e5 (stale) Mirror 6: 0x815b96e5 (stale) Mirror 9: 0x815b96e5 (stale) Mirror 10: 0x815b96e5 (stale) Mirror 11: 0x815b96e5 (stale) Mirror 12: 0x815b96e5 (stale) Mirror 13: 0x815b96e5 (stale) Mirror 14: 0x815b96e5 (stale) Mirror 15: 0x815b96e5 (stale) Mirror 18: 0x815b96e5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xedc402af Mirror 19: 0xedc402af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000587 1780335477.710584 mapread 0x03a182 thru 0x03d7d6 (0x3655 bytes) 000590 1780335477.747276 mapwrite 0x028ad2 thru 0x02c5ab (0x3ada bytes) 000592 1780335477.973555 read 0x03c1a9 thru 0x03ffff (0x3e57 bytes) 000595 1780335477.997396 trunc from 0x040000 to 0x02cd58 (0x132a7 bytes) 000597 1780335478.076938 trunc from 0x02cd59 to 0x0265b5 (0x67a3 bytes) 000600 1780335478.103753 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x265b6): Mirror 5: 0xf3df9910 (stale) Mirror 6: 0xf3df9910 (stale) Mirror 9: 0xf3df9910 (stale) Mirror 10: 0xf3df9910 (stale) Mirror 11: 0xf3df9910 (stale) Mirror 12: 0xf3df9910 (stale) Mirror 13: 0xf3df9910 (stale) Mirror 14: 0xf3df9910 (stale) Mirror 18: 0xf3df9910 (stale) Mirror 19: 0xf3df9910 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x265b6): Mirror 5: 0xf3df9910 (stale) Mirror 6: 0xf3df9910 (stale) Mirror 9: 0xf3df9910 (stale) Mirror 10: 0xf3df9910 (stale) Mirror 11: 0xf3df9910 (stale) Mirror 12: 0xf3df9910 (stale) Mirror 13: 0xf3df9910 (stale) Mirror 14: 0xf3df9910 (stale) Mirror 18: 0xf3df9910 (stale) Mirror 19: 0xf3df9910 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x265b6: skipped 000602 1780335478.888817 write 0x0262b3 thru 0x030e0b (0xab59 bytes) 000605 1780335478.899820 read 0x01f50a thru 0x02903b (0x9b32 bytes) 000606 1780335478.911088 mapread 0x01da11 thru 0x0202fe (0x28ee bytes) 000607 1780335478.936696 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30e0c): Mirror 5: 0x20937b55 (stale) Mirror 6: 0x20937b55 (stale) Mirror 9: 0x20937b55 (stale) Mirror 10: 0x20937b55 (stale) Mirror 11: 0x20937b55 (stale) Mirror 12: 0x20937b55 (stale) Mirror 13: 0x20937b55 (stale) Mirror 14: 0x20937b55 (stale) Mirror 19: 0x20937b55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30e0c): Mirror 5: 0x20937b55 (stale) Mirror 6: 0x20937b55 (stale) Mirror 9: 0x20937b55 (stale) Mirror 10: 0x20937b55 (stale) Mirror 11: 0x20937b55 (stale) Mirror 12: 0x20937b55 (stale) Mirror 13: 0x20937b55 (stale) Mirror 14: 0x20937b55 (stale) Mirror 19: 0x20937b55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30e0c: skipped 000611 1780335479.720863 mapread 0x026da9 thru 0x02eb8c (0x7de4 bytes) 000612 1780335479.774075 read 0x025ae9 thru 0x02d288 (0x77a0 bytes) 000613 1780335479.783126 MIRROR_RESYNC 10 mirrors 000618 1780335482.050822 MIRROR_RESYNC 10 mirrors 000620 1780335482.807156 read 0x001c8a thru 0x006041 (0x43b8 bytes) 000623 1780335482.821328 mapwrite 0x017049 thru 0x01e4d3 (0x748b bytes) 000624 1780335483.045167 trunc from 0x030e0c to 0x021b6f (0xf29c bytes) 000626 1780335483.075039 trunc from 0x021b70 to 0x001192 (0x209dd bytes) 000629 1780335483.104754 write 0x03f982 thru 0x03ffff (0x67e bytes) 000632 1780335483.119339 MIRROR_RESYNC 10 mirrors 000633 1780335484.872822 trunc from 0x040000 to 0x01f97b (0x20684 bytes) 000634 1780335484.990106 mapread 0x006091 thru 0x00e48b (0x83fb bytes) 000635 1780335485.016269 mapread 0x000090 thru 0x0098b2 (0x9823 bytes) 000637 1780335485.023014 read 0x003db5 thru 0x0075eb (0x3837 bytes) 000639 1780335485.028967 MIRROR_RESYNC 10 mirrors 000641 1780335486.890862 read 0x0026eb thru 0x003661 (0xf77 bytes) 000643 1780335486.897258 read 0x006fb2 thru 0x00b39e (0x43ed bytes) 000645 1780335486.911096 mapread 0x006757 thru 0x011dc8 (0xb672 bytes) 000650 1780335486.912983 mapread 0x01c478 thru 0x01f97b (0x3504 bytes) 000651 1780335486.915607 read 0x00f62a thru 0x010e3c (0x1813 bytes) 000652 1780335486.916146 read 0x019cd5 thru 0x01a434 (0x760 bytes) 000654 1780335486.917242 write 0x008783 thru 0x00bf11 (0x378f bytes) 000656 1780335487.021406 mapwrite 0x037b5f thru 0x03f0ff (0x75a1 bytes) 000657 1780335487.177563 mapwrite 0x00534b thru 0x00d585 (0x823b bytes) 000658 1780335487.250933 write 0x02e0fd thru 0x0333fb (0x52ff bytes) 000659 1780335487.252949 write 0x037905 thru 0x03c39c (0x4a98 bytes) 000662 1780335487.257392 read 0x03624f thru 0x03f0ff (0x8eb1 bytes) 000664 1780335487.259369 trunc from 0x03f100 to 0x02e3a7 (0x10d58 bytes) 000665 1780335487.283638 write 0x028cd8 thru 0x02fd5f (0x7088 bytes) 000668 1780335487.288171 mapread 0x01bdaa thru 0x029c5f (0xdeb6 bytes) 000669 1780335487.290197 read 0x01c6d9 thru 0x02616e (0x9a96 bytes) 000670 1780335487.291597 mapread 0x014c93 thru 0x017315 (0x2683 bytes) 000671 1780335487.292918 write 0x030a8b thru 0x034957 (0x3ecd bytes) 000673 1780335487.296173 write 0x006706 thru 0x00e546 (0x7e41 bytes) 000675 1780335487.298688 mapwrite 0x01731a thru 0x01d360 (0x6047 bytes) 000676 1780335487.371250 write 0x00cb2c thru 0x00d319 (0x7ee bytes) 000679 1780335487.377282 read 0x00084a thru 0x00a3f5 (0x9bac bytes) 000681 1780335487.382351 trunc from 0x034958 to 0x011bc0 (0x22d97 bytes) 000682 1780335487.414995 write 0x035d02 thru 0x03dead (0x81ac bytes) 000684 1780335487.416946 trunc from 0x03deae to 0x029749 (0x14764 bytes) 000685 1780335487.444436 read 0x026812 thru 0x029749 (0x2f38 bytes) 000686 1780335487.463198 trunc from 0x02974a to 0x011f27 (0x17822 bytes) 000687 1780335487.488962 trunc from 0x011f28 to 0x034c1c (0x22cf5 bytes) 000689 1780335487.517200 write 0x0111e5 thru 0x01e888 (0xd6a4 bytes) 000690 1780335487.541374 read 0x00cb42 thru 0x017c02 (0xb0c1 bytes) 000691 1780335487.543586 read 0x002a4b thru 0x00f94a (0xcf00 bytes) 000693 1780335487.557234 read 0x013a78 thru 0x021652 (0xdbdb bytes) 000695 1780335487.578588 mapread 0x0089d2 thru 0x012160 (0x978f bytes) 000696 1780335487.583115 write 0x0225bd thru 0x02ed36 (0xc77a bytes) 000697 1780335487.584600 read 0x01acde thru 0x025b8f (0xaeb2 bytes) 000699 1780335487.587229 trunc from 0x034c1d to 0x02da23 (0x71f9 bytes) 000700 1780335487.631265 mapwrite 0x01c3d3 thru 0x0250f1 (0x8d1f bytes) 000703 1780335487.718390 trunc from 0x02da24 to 0x03b6ad (0xdc8a bytes) 000705 1780335487.745256 read 0x0274b2 thru 0x02c0cf (0x4c1e bytes) 000707 1780335487.747997 trunc from 0x03b6ae to 0x03c946 (0x1299 bytes) 000708 1780335487.782540 read 0x00b38e thru 0x00eb8b (0x37fe bytes) 000710 1780335487.786320 write 0x012018 thru 0x01ad1f (0x8d08 bytes) 000714 1780335487.790018 mapwrite 0x0290ac thru 0x02cbcf (0x3b24 bytes) 000715 1780335487.852760 mapwrite 0x018f3b thru 0x01e892 (0x5958 bytes) 000718 1780335487.975077 write 0x0390a9 thru 0x03ffff (0x6f57 bytes) 000719 1780335487.986133 read 0x0253f4 thru 0x03351e (0xe12b bytes) 000722 1780335487.987974 mapwrite 0x021342 thru 0x02ad20 (0x99df bytes) 000723 1780335488.109675 write 0x010bcd thru 0x0184c7 (0x78fb bytes) 000724 1780335488.113365 mapwrite 0x00d073 thru 0x01a087 (0xd015 bytes) 000730 1780335488.256971 trunc from 0x040000 to 0x0122df (0x2dd20 bytes) 000731 1780335488.302342 mapread 0x008248 thru 0x0122df (0xa098 bytes) 000735 1780335488.308188 read 0x0109da thru 0x0122df (0x1906 bytes) 000736 1780335488.309317 read 0x006ad2 thru 0x0122df (0xb80e bytes) 000737 1780335488.310738 write 0x031449 thru 0x03ffff (0xebb7 bytes) 000738 1780335488.312818 write 0x0057f4 thru 0x013d91 (0xe59e bytes) 000739 1780335488.341327 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x401c9c95 (stale) Mirror 6: 0x401c9c95 (stale) Mirror 9: 0x401c9c95 (stale) Mirror 10: 0x401c9c95 (stale) Mirror 11: 0x401c9c95 (stale) Mirror 12: 0x401c9c95 (stale) Mirror 14: 0x401c9c95 (stale) Mirror 19: 0x401c9c95 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x401c9c95 (stale) Mirror 6: 0x401c9c95 (stale) Mirror 9: 0x401c9c95 (stale) Mirror 10: 0x401c9c95 (stale) Mirror 11: 0x401c9c95 (stale) Mirror 12: 0x401c9c95 (stale) Mirror 14: 0x401c9c95 (stale) Mirror 19: 0x401c9c95 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000742 1780335489.218284 mapwrite 0x02d0d1 thru 0x0375e6 (0xa516 bytes) 000744 1780335489.333920 write 0x01849c thru 0x01d7bb (0x5320 bytes) 000745 1780335489.346940 mapwrite 0x018fa7 thru 0x01f8c2 (0x691c bytes) 000746 1780335489.416257 write 0x036dff thru 0x03b9e1 (0x4be3 bytes) 000750 1780335489.433469 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x401c9c95 (stale) Mirror 6: 0x401c9c95 (stale) Mirror 9: 0x401c9c95 (stale) Mirror 10: 0x401c9c95 (stale) Mirror 11: 0x401c9c95 (stale) Mirror 12: 0x401c9c95 (stale) Mirror 14: 0x401c9c95 (stale) Mirror 19: 0x401c9c95 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x10f6179b Mirror 20: 0x10f6179b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000751 1780335490.370312 mapread 0x00d27a thru 0x01c0e5 (0xee6c bytes) 000753 1780335490.415529 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x401c9c95 (stale) Mirror 6: 0x401c9c95 (stale) Mirror 9: 0x401c9c95 (stale) Mirror 10: 0x401c9c95 (stale) Mirror 11: 0x401c9c95 (stale) Mirror 12: 0x401c9c95 (stale) Mirror 14: 0x401c9c95 (stale) Mirror 19: 0x401c9c95 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x10f6179b Mirror 20: 0x10f6179b Mirror 21: 0x10f6179b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000758 1780335491.739352 mapwrite 0x01d709 thru 0x023c33 (0x652b bytes) 000759 1780335492.092354 write 0x02bdb8 thru 0x02c5c6 (0x80f bytes) 000761 1780335492.115542 read 0x018743 thru 0x021677 (0x8f35 bytes) 000762 1780335492.116234 read 0x0213bd thru 0x02ac96 (0x98da bytes) 000765 1780335492.120069 write 0x01483a thru 0x01cf05 (0x86cc bytes) 000766 1780335492.168929 read 0x039435 thru 0x03ffff (0x6bcb bytes) 000770 1780335492.183538 mapwrite 0x03cabe thru 0x03ffff (0x3542 bytes) 000772 1780335492.358934 trunc from 0x040000 to 0x011017 (0x2efe8 bytes) 000774 1780335492.461236 mapwrite 0x036885 thru 0x03ffff (0x977b bytes) 000775 1780335492.700589 mapwrite 0x01d46c thru 0x02843e (0xafd3 bytes) 000776 1780335492.837313 mapread 0x026968 thru 0x02bc35 (0x52ce bytes) 000778 1780335492.838943 trunc from 0x040000 to 0x015025 (0x2afda bytes) 000780 1780335492.883872 trunc from 0x015026 to 0x0381c9 (0x231a4 bytes) 000781 1780335492.906798 trunc from 0x0381ca to 0x03a1d5 (0x200c bytes) 000782 1780335492.943436 mapwrite 0x029a21 thru 0x02c55a (0x2b3a bytes) 000783 1780335493.045500 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a1d6): Mirror 5: 0x7aee0a80 (stale) Mirror 6: 0x7aee0a80 (stale) Mirror 9: 0x7aee0a80 (stale) Mirror 11: 0x7aee0a80 (stale) Mirror 12: 0x7aee0a80 (stale) Mirror 14: 0x7aee0a80 (stale) Mirror 19: 0x7aee0a80 (stale) Mirror 20: 0xb83dfacc (stale) Mirror 21: 0xb83dfacc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1d6): Mirror 5: 0x7aee0a80 (stale) Mirror 6: 0x7aee0a80 (stale) Mirror 9: 0x7aee0a80 (stale) Mirror 11: 0x7aee0a80 (stale) Mirror 12: 0x7aee0a80 (stale) Mirror 14: 0x7aee0a80 (stale) Mirror 19: 0x7aee0a80 (stale) Mirror 20: 0xb83dfacc (stale) Mirror 21: 0xb83dfacc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1d6: skipped 000787 1780335493.940071 mapread 0x0150af thru 0x01e8c1 (0x9813 bytes) 000788 1780335493.998431 read 0x020d60 thru 0x024626 (0x38c7 bytes) 000790 1780335494.054952 MIRROR_RESYNC 10 mirrors 000791 1780335496.471456 mapread 0x01aa54 thru 0x02675d (0xbd0a bytes) 000792 1780335496.545140 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) 000793 1780335497.391372 mapread 0x024b8b thru 0x0335ee (0xea64 bytes) 000794 1780335497.465783 read 0x0224a0 thru 0x031978 (0xf4d9 bytes) 000796 1780335497.484008 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) 000797 1780335498.304213 trunc from 0x03a1d6 to 0x00a857 (0x2f97e bytes) 000801 1780335498.495288 read 0x0076f7 thru 0x00a857 (0x3161 bytes) 000805 1780335498.511212 mapwrite 0x00d016 thru 0x014457 (0x7442 bytes) 000806 1780335498.727256 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x14458): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14458): Mirror 5: 0x91e89c41 (stale) Mirror 6: 0x91e89c41 (stale) Mirror 11: 0x91e89c41 (stale) Mirror 12: 0x91e89c41 (stale) Mirror 14: 0x91e89c41 (stale) Mirror 19: 0x91e89c41 (stale) Mirror 21: 0x91e89c41 (stale) CRC-32 checksum value for chunk [0, 0x14458): Mirror 1: 0x263fc42f Mirror 22: 0x263fc42f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14458: skipped 000807 1780335500.185486 write 0x003c21 thru 0x01137f (0xd75f bytes) 000811 1780335500.291500 mapwrite 0x02e8b9 thru 0x033019 (0x4761 bytes) 000812 1780335500.400832 mapread 0x012f22 thru 0x018e71 (0x5f50 bytes) 000813 1780335500.441944 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3301a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3301a): Mirror 5: 0xa6fa19e9 (stale) Mirror 6: 0xa6fa19e9 (stale) Mirror 11: 0xa6fa19e9 (stale) Mirror 12: 0xa6fa19e9 (stale) Mirror 14: 0xa6fa19e9 (stale) Mirror 19: 0xa6fa19e9 (stale) Mirror 21: 0xa6fa19e9 (stale) Mirror 22: 0xad77979b (stale) CRC-32 checksum value for chunk [0, 0x3301a): Mirror 1: 0x7ec9ca6e Mirror 23: 0x7ec9ca6e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3301a: skipped 000815 1780335501.879943 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3301a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3301a): Mirror 5: 0xa6fa19e9 (stale) Mirror 6: 0xa6fa19e9 (stale) Mirror 11: 0xa6fa19e9 (stale) Mirror 12: 0xa6fa19e9 (stale) Mirror 14: 0xa6fa19e9 (stale) Mirror 19: 0xa6fa19e9 (stale) Mirror 21: 0xa6fa19e9 (stale) Mirror 22: 0xad77979b (stale) CRC-32 checksum value for chunk [0, 0x3301a): Mirror 1: 0x7ec9ca6e Mirror 23: 0x7ec9ca6e Mirror 24: 0x7ec9ca6e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3301a: skipped 000816 1780335503.256513 mapread 0x00de4e thru 0x01cfad (0xf160 bytes) 000818 1780335503.316886 mapwrite 0x02cda7 thru 0x0374b5 (0xa70f bytes) 000819 1780335503.752432 mapwrite 0x039f62 thru 0x03ffff (0x609e bytes) 000821 1780335503.944289 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x79533cbf (stale) Mirror 6: 0x79533cbf (stale) Mirror 12: 0x79533cbf (stale) Mirror 14: 0x79533cbf (stale) Mirror 19: 0x79533cbf (stale) Mirror 21: 0x79533cbf (stale) Mirror 22: 0xc6fd8b5c (stale) Mirror 23: 0x70196144 (stale) Mirror 24: 0x70196144 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x79533cbf (stale) Mirror 6: 0x79533cbf (stale) Mirror 12: 0x79533cbf (stale) Mirror 14: 0x79533cbf (stale) Mirror 19: 0x79533cbf (stale) Mirror 21: 0x79533cbf (stale) Mirror 22: 0xc6fd8b5c (stale) Mirror 23: 0x70196144 (stale) Mirror 24: 0x70196144 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000822 1780335505.155544 read 0x03983e thru 0x03ffff (0x67c2 bytes) 000823 1780335505.174309 read 0x0053cf thru 0x0117c3 (0xc3f5 bytes) 000824 1780335505.197835 trunc from 0x040000 to 0x0128e6 (0x2d719 bytes) 000825 1780335505.229612 mapread 0x001999 thru 0x0043c0 (0x2a28 bytes) 000827 1780335505.247508 MIRROR_RESYNC 10 mirrors 000828 1780335507.692612 read 0x00697a thru 0x006d20 (0x3a7 bytes) 000830 1780335507.705945 mapread 0x00fcf2 thru 0x0128e6 (0x2bf5 bytes) 000831 1780335507.742196 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) 000833 1780335508.353652 write 0x011bcc thru 0x01cd24 (0xb159 bytes) 000834 1780335508.470918 mapwrite 0x03368a thru 0x037694 (0x400b bytes) 000835 1780335508.610947 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x37695): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37695): Mirror 5: 0x7d415dc9 (stale) Mirror 6: 0x7d415dc9 (stale) Mirror 12: 0x7d415dc9 (stale) Mirror 14: 0x7d415dc9 (stale) Mirror 19: 0x7d415dc9 (stale) Mirror 22: 0x7d415dc9 (stale) Mirror 23: 0x7d415dc9 (stale) Mirror 24: 0x7d415dc9 (stale) CRC-32 checksum value for chunk [0, 0x37695): Mirror 1: 0x69542826 Mirror 25: 0x69542826 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37695: skipped 000837 1780335510.095345 mapread 0x00d2d1 thru 0x015f00 (0x8c30 bytes) 000838 1780335510.158205 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37695): Mirror 5: 0x7d415dc9 (stale) Mirror 6: 0x7d415dc9 (stale) Mirror 12: 0x7d415dc9 (stale) Mirror 14: 0x7d415dc9 (stale) Mirror 19: 0x7d415dc9 (stale) Mirror 22: 0x7d415dc9 (stale) Mirror 23: 0x7d415dc9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37695): Mirror 5: 0x7d415dc9 (stale) Mirror 6: 0x7d415dc9 (stale) Mirror 12: 0x7d415dc9 (stale) Mirror 14: 0x7d415dc9 (stale) Mirror 19: 0x7d415dc9 (stale) Mirror 22: 0x7d415dc9 (stale) Mirror 23: 0x7d415dc9 (stale) CRC-32 checksum value for chunk [0, 0x37695): Mirror 1: 0x69542826 Mirror 25: 0x69542826 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37695: skipped 000842 1780335511.253361 read 0x023def thru 0x02e51a (0xa72c bytes) 000843 1780335511.306761 mapwrite 0x0021ca thru 0x00faf6 (0xd92d bytes) 000844 1780335511.554460 mapread 0x026958 thru 0x0358d2 (0xef7b bytes) 000845 1780335511.586008 mapwrite 0x0146f8 thru 0x01f11d (0xaa26 bytes) 000846 1780335511.698056 read 0x0006d4 thru 0x00ca3f (0xc36c bytes) 000848 1780335511.727240 MIRROR_RESYNC 9 mirrors 000849 1780335513.477959 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) 000850 1780335514.133521 read 0x00cb7b thru 0x012567 (0x59ed bytes) 000851 1780335514.149746 read 0x01e1ec thru 0x02c395 (0xe1aa bytes) 000854 1780335514.173068 trunc from 0x037695 to 0x03b01b (0x3987 bytes) 000855 1780335514.282424 trunc from 0x03b01c to 0x0199ea (0x21631 bytes) 000856 1780335514.307939 mapread 0x006124 thru 0x011841 (0xb71e bytes) 000857 1780335514.378313 read 0x00f097 thru 0x015416 (0x6380 bytes) 000858 1780335514.390448 read 0x019351 thru 0x0199ea (0x69a bytes) 000859 1780335514.391734 mapwrite 0x0187b7 thru 0x026a58 (0xe2a2 bytes) 000860 1780335514.518203 trunc from 0x026a59 to 0x038c23 (0x121cb bytes) 000862 1780335514.550332 trunc from 0x038c24 to 0x0099bb (0x2f268 bytes) 000863 1780335514.580543 read 0x000f5b thru 0x0099bb (0x8a61 bytes) 000866 1780335514.589647 MIRROR_RESYNC 8 mirrors 000868 1780335516.268516 write 0x013a75 thru 0x015a0c (0x1f98 bytes) 000869 1780335516.329151 read 0x00d888 thru 0x00e840 (0xfb9 bytes) 000870 1780335516.340795 mapwrite 0x0248cf thru 0x02d90b (0x903d bytes) 000871 1780335516.472264 read 0x01d139 thru 0x022f0d (0x5dd5 bytes) 000873 1780335516.483582 mapwrite 0x01e3c3 thru 0x024f8d (0x6bcb bytes) 000874 1780335516.539567 write 0x008092 thru 0x00edd9 (0x6d48 bytes) 000876 1780335516.547444 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d90c): Mirror 6: 0x98ad0372 (stale) Mirror 12: 0x98ad0372 (stale) Mirror 14: 0x98ad0372 (stale) Mirror 19: 0x98ad0372 (stale) Mirror 22: 0x98ad0372 (stale) Mirror 23: 0x98ad0372 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d90c): Mirror 6: 0x98ad0372 (stale) Mirror 12: 0x98ad0372 (stale) Mirror 14: 0x98ad0372 (stale) Mirror 19: 0x98ad0372 (stale) Mirror 22: 0x98ad0372 (stale) Mirror 23: 0x98ad0372 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d90c: skipped 000877 1780335517.182793 write 0x01866f thru 0x01b7bc (0x314e bytes) 000879 1780335517.212740 mapwrite 0x011795 thru 0x018988 (0x71f4 bytes) 000881 1780335517.300625 write 0x013481 thru 0x01e780 (0xb300 bytes) 000882 1780335517.307904 read 0x017b99 thru 0x026810 (0xec78 bytes) 000885 1780335517.324922 trunc from 0x02d90c to 0x0396b3 (0xbda8 bytes) 000886 1780335517.347634 trunc from 0x0396b4 to 0x010a31 (0x28c82 bytes) 000887 1780335517.367214 write 0x02df8f thru 0x03b137 (0xd1a9 bytes) 000888 1780335517.375403 mapwrite 0x03dee3 thru 0x03ffff (0x211d bytes) 000892 1780335517.489852 mapread 0x029ff3 thru 0x02c350 (0x235e bytes) 000894 1780335517.516970 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x96f6ea36 (stale) Mirror 12: 0x96f6ea36 (stale) Mirror 14: 0x96f6ea36 (stale) Mirror 19: 0x96f6ea36 (stale) Mirror 22: 0x96f6ea36 (stale) Mirror 23: 0x96f6ea36 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xea4a94f5 Mirror 24: 0xea4a94f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000897 1780335519.076934 mapwrite 0x009e53 thru 0x01704f (0xd1fd bytes) 000898 1780335519.325238 trunc from 0x040000 to 0x01c4a9 (0x23b56 bytes) 000900 1780335519.356166 mapwrite 0x02f172 thru 0x038764 (0x95f3 bytes) 000901 1780335519.545767 mapwrite 0x012eb6 thru 0x0150e4 (0x222f bytes) 000903 1780335519.635692 write 0x039e5a thru 0x03ebb2 (0x4d59 bytes) 000904 1780335519.639060 mapwrite 0x004c3b thru 0x012a73 (0xde39 bytes) 000905 1780335519.717020 read 0x012622 thru 0x017f57 (0x5936 bytes) 000906 1780335519.717906 read 0x0206d7 thru 0x020a54 (0x37e bytes) 000907 1780335519.729583 mapread 0x027d42 thru 0x0349af (0xcc6e bytes) 000908 1780335519.731174 read 0x02644f thru 0x0351eb (0xed9d bytes) 000909 1780335519.733411 read 0x035df0 thru 0x03ebb2 (0x8dc3 bytes) 000913 1780335519.739500 mapread 0x03a5c1 thru 0x03ebb2 (0x45f2 bytes) 000916 1780335519.744612 trunc from 0x03ebb3 to 0x02b51c (0x13696 bytes) 000917 1780335519.779118 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2b51d): Mirror 6: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b51d): Mirror 6: 0x4dfb505f (stale) Mirror 12: 0x4dfb505f (stale) Mirror 14: 0x4dfb505f (stale) Mirror 19: 0x4dfb505f (stale) Mirror 22: 0x4dfb505f (stale) Mirror 23: 0x4dfb505f (stale) Mirror 24: 0x10dd3361 (stale) CRC-32 checksum value for chunk [0, 0x2b51d): Mirror 1: 0xc7044fcd Mirror 25: 0xc7044fcd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b51d: skipped 000919 1780335521.354146 mapread 0x010eec thru 0x012180 (0x1295 bytes) 000920 1780335521.444051 trunc from 0x02b51d to 0x0053db (0x26141 bytes) 000922 1780335521.656915 trunc from 0x0053dc to 0x0219ef (0x1c614 bytes) 000923 1780335521.696182 mapread 0x010473 thru 0x01c6d5 (0xc263 bytes) 000925 1780335521.754637 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x219f0): Mirror 6: 0xc6663d02 (stale) Mirror 12: 0xc6663d02 (stale) Mirror 14: 0xc6663d02 (stale) Mirror 19: 0xc6663d02 (stale) Mirror 22: 0xc6663d02 (stale) Mirror 23: 0xc6663d02 (stale) Mirror 25: 0xf7f30de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x219f0): Mirror 6: 0xc6663d02 (stale) Mirror 12: 0xc6663d02 (stale) Mirror 14: 0xc6663d02 (stale) Mirror 19: 0xc6663d02 (stale) Mirror 22: 0xc6663d02 (stale) Mirror 23: 0xc6663d02 (stale) Mirror 25: 0xf7f30de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x219f0: skipped 000926 1780335522.579401 mapread 0x0015b3 thru 0x009fb1 (0x89ff bytes) 000927 1780335522.606510 mapread 0x005784 thru 0x012341 (0xcbbe bytes) 000930 1780335522.617888 mapread 0x018e2a thru 0x01a3d7 (0x15ae bytes) 000931 1780335522.628607 mapread 0x01dfe6 thru 0x0219ef (0x3a0a bytes) 000932 1780335522.629949 mapwrite 0x029ba0 thru 0x02eb81 (0x4fe2 bytes) 000933 1780335522.748767 mapwrite 0x018171 thru 0x01e919 (0x67a9 bytes) 000936 1780335522.819866 write 0x03d06b thru 0x03ffff (0x2f95 bytes) 000940 1780335522.825126 read 0x032cb0 thru 0x0365f4 (0x3945 bytes) 000943 1780335522.840877 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x96f6ea36 (stale) Mirror 12: 0x96f6ea36 (stale) Mirror 14: 0x96f6ea36 (stale) Mirror 22: 0x96f6ea36 (stale) Mirror 23: 0x96f6ea36 (stale) Mirror 25: 0x6cc5ce3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x96f6ea36 (stale) Mirror 12: 0x96f6ea36 (stale) Mirror 14: 0x96f6ea36 (stale) Mirror 22: 0x96f6ea36 (stale) Mirror 23: 0x96f6ea36 (stale) Mirror 25: 0x6cc5ce3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000945 1780335523.406464 mapread 0x020acc thru 0x02afd4 (0xa509 bytes) 000946 1780335523.471143 mapwrite 0x026a8f thru 0x033b00 (0xd072 bytes) 000950 1780335523.579840 MIRROR_RESYNC 7 mirrors 000951 1780335525.318613 write 0x003c1f thru 0x0131e5 (0xf5c7 bytes) 000954 1780335525.468399 mapwrite 0x022eeb thru 0x02db8d (0xaca3 bytes) 000957 1780335525.635000 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4f57105c (stale) Mirror 12: 0x4f57105c (stale) Mirror 14: 0x4f57105c (stale) Mirror 22: 0x4f57105c (stale) Mirror 23: 0x4f57105c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4f57105c (stale) Mirror 12: 0x4f57105c (stale) Mirror 14: 0x4f57105c (stale) Mirror 22: 0x4f57105c (stale) Mirror 23: 0x4f57105c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000958 1780335526.317612 mapread 0x007970 thru 0x009429 (0x1aba bytes) 000959 1780335526.355089 mapwrite 0x00765b thru 0x00a8d2 (0x3278 bytes) 000961 1780335526.420346 write 0x010799 thru 0x017dbc (0x7624 bytes) 000963 1780335526.432603 mapread 0x035899 thru 0x03695e (0x10c6 bytes) 000964 1780335526.442131 mapread 0x0362a8 thru 0x03ffff (0x9d58 bytes) 000966 1780335526.478283 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x4f57105c (stale) Mirror 14: 0x4f57105c (stale) Mirror 22: 0x4f57105c (stale) Mirror 23: 0x4f57105c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x4f57105c (stale) Mirror 14: 0x4f57105c (stale) Mirror 22: 0x4f57105c (stale) Mirror 23: 0x4f57105c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000967 1780335526.938790 write 0x024548 thru 0x02f619 (0xb0d2 bytes) 000970 1780335526.950999 read 0x02a042 thru 0x02f702 (0x56c1 bytes) 000971 1780335526.951655 trunc from 0x040000 to 0x016cb8 (0x29347 bytes) 000972 1780335526.979468 read 0x00b8e6 thru 0x01375c (0x7e77 bytes) 000974 1780335526.992549 mapread 0x00edf5 thru 0x016cb8 (0x7ec4 bytes) 000977 1780335527.002695 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x16cb9): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16cb9): Mirror 12: 0x6bb46f7 (stale) Mirror 14: 0x6bb46f7 (stale) Mirror 22: 0x6bb46f7 (stale) Mirror 23: 0x6bb46f7 (stale) CRC-32 checksum value for chunk [0, 0x16cb9): Mirror 1: 0x23b97cbf Mirror 24: 0x23b97cbf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16cb9: skipped 000978 1780335528.141670 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16cb9): Mirror 12: 0x6bb46f7 (stale) Mirror 14: 0x6bb46f7 (stale) Mirror 23: 0x6bb46f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16cb9): Mirror 12: 0x6bb46f7 (stale) Mirror 14: 0x6bb46f7 (stale) Mirror 23: 0x6bb46f7 (stale) CRC-32 checksum value for chunk [0, 0x16cb9): Mirror 1: 0x23b97cbf Mirror 24: 0x23b97cbf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16cb9: skipped 000979 1780335528.973471 trunc from 0x016cb9 to 0x03cf09 (0x26251 bytes) 000980 1780335529.165574 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cf0a): Mirror 14: 0x3a5f2d8e (stale) Mirror 23: 0x3a5f2d8e (stale) Mirror 24: 0xdfc783d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf0a): Mirror 14: 0x3a5f2d8e (stale) Mirror 23: 0x3a5f2d8e (stale) Mirror 24: 0xdfc783d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf0a: skipped 000981 1780335529.635956 mapread 0x0366cd thru 0x03ac2b (0x455f bytes) 000983 1780335529.679806 mapwrite 0x03fe97 thru 0x03ffff (0x169 bytes) 000984 1780335529.763048 mapread 0x0247e4 thru 0x02c1e9 (0x7a06 bytes) 000985 1780335529.780678 read 0x02a626 thru 0x034171 (0x9b4c bytes) 000986 1780335529.803981 mapwrite 0x0326a6 thru 0x03ffff (0xd95a bytes) 000987 1780335529.860369 mapwrite 0x006aba thru 0x014a0b (0xdf52 bytes) 000992 1780335529.983663 mapwrite 0x022c6c thru 0x02c631 (0x99c6 bytes) 000993 1780335530.062549 mapwrite 0x0379db thru 0x03c238 (0x485e bytes) 000994 1780335530.114678 read 0x026c55 thru 0x027cb7 (0x1063 bytes) 000996 1780335530.121175 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x4f57105c (stale) Mirror 24: 0xf5700f75 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x4f57105c (stale) Mirror 24: 0xf5700f75 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000997 1780335530.590493 trunc from 0x040000 to 0x01da7b (0x22584 bytes) 000998 1780335530.611621 mapread 0x004620 thru 0x009244 (0x4c25 bytes) 001000 1780335530.637663 read 0x009b6e thru 0x00ccdf (0x3172 bytes) All operations completed A-OK! PASS 70a (150s) == sanity-flr test 71: check mirror extend parent fid ==== 13:38:56 (1780335536) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 13:39:01 (1780335541) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x280:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.331798 s, 6.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x280:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x281:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232627 s, 4.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x280:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x281:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x280:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x281:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x282:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 13:39:11 (1780335551) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 685888 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1399776 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5277472 .. Extending file size to 4444992 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8314336 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5984832 .. Extending file size to 6253824 .. Extending file size to 7007168 .. Extending file size to 7230944 .. Extending file size to 7694848 .. Extending file size to 9069632 .. Extending file size to 8396864 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8495392 .. Extending file size to 9023744 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 56570 56571 56574 56575 56577 done 192.168.204.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre2 (opts:) 192.168.204.111@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:40:39 (1780335639) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 797632 .. Extending file size to 2356064 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4736640 .. Extending file size to 5058176 .. failed Extending file size to 7885312 .. Extending file size to 8325920 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8561920 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8671840 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9238112 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8960768 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58028 58029 58030 58031 58034 done 192.168.204.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre2 (opts:) 192.168.204.111@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:42:04 (1780335724) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0999165 s, 21.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.445056 s, 4.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x8066c872 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:42:25 (1780335745) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138301 s, 15.2 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 13:42:33 (1780335753) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x293:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x293:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x294:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x297:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:42:39 (1780335759) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.14123 s, 14.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x295:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x298:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x299:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:42:48 (1780335768) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194608 s, 10.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x296:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x297:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x296:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x297:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29b:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:42:59 (1780335779) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.454525 s, 4.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x298:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x298:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29d:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:43:34 (1780335814) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg411-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166701 s, 6.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x299:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x299:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29a:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:44:36 (1780335876) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg411-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.380199 s, 5.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (59s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:45:35 (1780335935) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg411-server mds-ost sync done. Creating new pool test_204f oleg411-server: Pool lustre.test_204f created Adding targets to pool oleg411-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg411-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.184288 s, 11.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg411-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg411-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg411-server: Pool lustre.test_204f destroyed PASS 204f (85s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:47:00 (1780336020) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:47:07 (1780336027) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:47:15 (1780336035) oleg411-server: Pool lustre.test_206 created Adding targets to pool oleg411-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg411-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg411-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg411-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg411-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:47:40 (1780336060) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060924 s, 17.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452308 s, 23.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x72bac47a Mirror 3: 0x72bac47a PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:47:50 (1780336070) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:47:54 (1780336074) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:47:58 (1780336078) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.169857 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x11e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x20d0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x20d0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x20d0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xb70000 Read this (4 bytes): BBBB 192.168.204.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:48:38 (1780336118) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0492685 s, 166 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:48:48 (1780336128) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572312 s, 18.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1c4 lmm_fid: [0x200000404:0x1c4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1c4 lmm_fid: [0x200000404:0x1c4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834372 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 13:48:50.000000000 -0400 Modify: 2026-06-01 13:48:50.000000000 -0400 Change: 2026-06-01 13:48:52.000000000 -0400 Birth: 2026-06-01 13:48:51.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:48:59 (1780336139) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:49:03 (1780336143) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.559605 s, 73.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.106503 s, 38.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 1 13:49 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c6:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 1 13:49 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 13:49:13 (1780336153) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 13:49:21 (1780336161) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1829 sec =========== 13:49:31 (1780336171) === sanity-flr: start cleanup 13:49:33 (1780336173) === === sanity-flr: finish cleanup 13:49:40 (1780336180) ===