-----============= acceptance-small: sanity-flr ============----- Mon Jun 1 11:17:00 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg230-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 11:17:17 (1780327037) === oleg230-client.virtnet: executing check_config_client /mnt/lustre oleg230-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg230-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c628bae5000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c628bae5000.idle_timeout=debug disable quota as required oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 11:17:48 (1780327068) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8187] == sanity-flr test 0a: lfs mirror create with -N option == 11:17:50 (1780327070) PASS 0a (12s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:18:02 (1780327082) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:18:05 (1780327085) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:18:07 (1780327087) PASS 0d (20s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:18:27 (1780327107) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:18:30 (1780327110) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 11:18:33 (1780327113) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8c628bae5000.stats=clear osc.lustre-OST0001-osc-ffff8c628bae5000.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:18:39 (1780327119) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:18:48 (1780327128) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:18:54 (1780327134) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 11:18:57 (1780327137) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:19:03 (1780327143) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 11:19:10 (1780327150) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:19:16 (1780327156) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:19:23 (1780327163) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:19:29 (1780327169) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:19:38 (1780327178) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:19:44 (1780327184) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.171731 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543193 s, 19.3 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:19:51 (1780327191) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.10437 s, 50.2 MB/s PASS 22 (5s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 11:19:57 (1780327197) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:20:06 (1780327206) Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c628bae5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c628bae5000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c628bae5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c628bae5000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg230-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (57s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:21:03 (1780327263) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.232471 s, 13.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c628bae5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c628bae5000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (34s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:21:37 (1780327297) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c628bae5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c628bae5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c628bae5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c628bae5000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg230-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (57s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:22:34 (1780327354) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.30403 s, 22.8 MB/s -rw-r--r-- 1 root root 52428800 Jun 1 11:22 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.874s user 0m0.001s sys 0m0.176s Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c628bae5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c628bae5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.899s user 0m0.001s sys 0m0.170s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c628bae5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c628bae5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.934s user 0m0.000s sys 0m0.161s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg230-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c628bae5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c628bae5000.ost_server_uuid in FULL state after 0 sec PASS 33b (65s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:23:39 (1780327419) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:23:42 (1780327422) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:23:44 (1780327424) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 11:23:46 (1780327426) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 11:23:52 (1780327432) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.352859 s, 35.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (8s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:24:00 (1780327440) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206076 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434928 s, 24.1 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:24:06 (1780327446) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:24:12 (1780327452) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00123518 s, 829 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00273631 s, 374 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00119875 s, 854 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00222513 s, 460 kB/s flv=10 olv=10 PASS 36d (7s) == sanity-flr test 37: mirror I/O API verification ======= 11:24:19 (1780327459) create test files with size 15745505 .. cc79073d5f9baeb573217c69cddd0b5e - 68c5697bad8277912bbdc94047456c21 - 32eab94035ab9c848833d2cf2fae0682 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre PASS 37 (17s) == sanity-flr test 38: resync ============================ 11:24:36 (1780327476) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (8s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:24:44 (1780327484) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:24:49 (1780327489) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381703 s, 27.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263978 s, 39.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 11:24:54 (1780327494) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0583536 s, 35.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.188865 s, 22.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0906264 s, 23.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (9s) == sanity-flr test 42: lfs mirror verify ================= 11:25:03 (1780327503) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 11:25:05 (1780327505) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:25:07 (1780327507) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164768 s, 63.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 11:25:11 (1780327511) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.162188 s, 19.4 MB/s PASS 44a (6s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:25:17 (1780327517) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:25:24 (1780327524) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.19445 s, 53.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (7s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:25:31 (1780327531) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8c6288178000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.160696 s, 65.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8c628ac98000.xattr_cache=1 PASS 44e (10s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:25:41 (1780327541) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (24s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:26:05 (1780327565) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (22s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:26:27 (1780327587) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 11:26:29 (1780327589) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.116643 s, 27.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.139542 s, 22.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x6202162b (nosync,stale) Mirror 2: 0x6202162b (nosync,stale) ** verify mirror 2 doesn't change original checksum: bf17d1f4405adce6ee0aafcddfd927a2 - mirror 1 checksum: 6635b3230e897699534cfcc2a37e4e7e - mirror 2 checksum: bf17d1f4405adce6ee0aafcddfd927a2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 11:26:31 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 11:26:31 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: bf17d1f4405adce6ee0aafcddfd927a2 - mirror 1 checksum: 6635b3230e897699534cfcc2a37e4e7e - mirror 2 checksum: 6635b3230e897699534cfcc2a37e4e7e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 11:26:31 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 11:26:31 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 48 (7s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 11:26:36 (1780327596) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238292 s, 44.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (5s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:26:41 (1780327601) Failing mds1 on oleg230-server Stopping /mnt/lustre-mds1 (opts:) on oleg230-server 11:26:51 (1780327611) shut down facet: mds1 facet_host: oleg230-server facet_failover_host: oleg230-server Failover mds1 to oleg230-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-MDT0000 11:27:09 (1780327629) targets are mounted 11:27:09 (1780327629) facet_failover done oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193152 s, 54.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 50A (38s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:27:19 (1780327639) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0584357 s, 350 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0137959 s, 1.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (6s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:27:25 (1780327645) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0465655 s, 440 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00918019 s, 2.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (6s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:27:31 (1780327651) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.007926 s, 2.1 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:27:35 (1780327655) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0301492 s, 679 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:27:40 (1780327660) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0061755 s, 663 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0261992 s, 156 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:27:44 (1780327664) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (13s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:27:57 (1780327677) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (23s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:28:20 (1780327700) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:28:39 (1780327719) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (20s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:28:59 (1780327739) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 11:29:03 (1780327743) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57099 mirror split pid 57100 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57099 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57100 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 11:30:07 (1780327807) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 7809 truncating to largest ever: 0x33f57 000001 1780327809.258799 trunc from 00000000 to 0x033f56 (0x33f57 bytes) 000002 1780327809.271946 trunc from 0x033f57 to 0x0276df (0xc877 bytes) 000003 1780327809.285966 fallocate 0x01a31f thru 0x01b4f5 (0x11d7 bytes) 000004 1780327809.292309 punch from 0x019caf to 0x022862 (0x8bb4 bytes) 000005 1780327809.297960 trunc from 0x003540 to 0x013380 (0xfe41 bytes) 000006 1780327809.303042 mapwrite 0x03d03b thru 0x03e017 (0xfdd bytes) 000007 1780327809.349130 fallocate 0x01fc8e thru 0x02eb9c (0xef0f bytes) fallocating to largest ever: 0x40000 000008 1780327809.354371 fallocate 0x03b824 thru 0x03ffff (0x47dc bytes) 000009 1780327809.359326 write 0x034401 thru 0x03b32f (0x6f2f bytes) 000010 1780327809.369657 mapread 0x018236 thru 0x0192d6 (0x10a1 bytes) 000013 1780327809.380596 mapwrite 0x038d00 thru 0x03ffff (0x7300 bytes) 000014 1780327809.401246 fallocate 0x03b83a thru 0x03fe7f (0x4646 bytes) 000015 1780327809.405407 mapread 0x017948 thru 0x022ff6 (0xb6af bytes) 000016 1780327809.413434 read 0x029d6a thru 0x02d1db (0x3472 bytes) 000017 1780327809.418409 read 0x03a16b thru 0x03ffff (0x5e95 bytes) 000018 1780327809.419644 trunc from 0x01470d to 0x0192ca (0x4bbe bytes) 000019 1780327809.432322 read 0x02ac32 thru 0x033539 (0x8908 bytes) 000020 1780327809.433369 trunc from 0x040000 to 0x020235 (0x1fdca bytes) 000021 1780327809.468058 MIRROR_EXTEND to 2 mirrors 000022 1780327809.975088 trunc from 0x01844e to 0x01a48e (0x2041 bytes) 000023 1780327810.008847 trunc from 0x018694 to 0x01e978 (0x62e5 bytes) 000024 1780327810.014170 trunc from 0x020236 to 0x010eda (0xf35b bytes) 000025 1780327810.065371 mapread 0x01079f thru 0x010eda (0x73c bytes) 000026 1780327810.072655 write 0x02d2fc thru 0x03086d (0x3572 bytes) 000027 1780327810.074278 write 0x013dbf thru 0x01a97b (0x6bbd bytes) 000028 1780327810.081654 mapread 0x01aca4 thru 0x028a10 (0xdd6d bytes) 000029 1780327810.091560 trunc from 0x01c0bc to 0x024b72 (0x8ab7 bytes) 000030 1780327810.094855 read 0x02d565 thru 0x02fbcf (0x266b bytes) 000031 1780327810.095828 trunc from 0x03086e to 0x0377de (0x6f71 bytes) 000032 1780327810.107110 MIRROR_RESYNC 2 mirrors 000033 1780327810.574318 mapread 0x01917b thru 0x026a85 (0xd90b bytes) 000035 1780327810.604467 mapwrite 0x02bc22 thru 0x02f712 (0x3af1 bytes) 000036 1780327810.664538 write 0x0152ae thru 0x024944 (0xf697 bytes) 000037 1780327810.682748 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) 000038 1780327810.797502 read 0x01c861 thru 0x024edc (0x867c bytes) 000039 1780327810.804655 punch from 0x0122d1 to 0x019c8b (0x79bb bytes) 000040 1780327810.811498 mapread 0x02cc9e thru 0x031747 (0x4aaa bytes) 000041 1780327810.831278 write 0x00e467 thru 0x01295d (0x44f7 bytes) 000042 1780327810.846076 trunc from 0x02c563 to 0x03605e (0x9afc bytes) 000043 1780327810.851836 mapread 0x00c5b3 thru 0x00ef51 (0x299f bytes) 000044 1780327810.853761 mapwrite 0x01c317 thru 0x01d285 (0xf6f bytes) 000045 1780327810.871449 punch from 0x00cb60 to 0x01827b (0xb71c bytes) 000046 1780327810.886490 MIRROR_EXTEND to 2 mirrors 000047 1780327811.443553 fallocate 0x02c636 thru 0x02eb05 (0x24d0 bytes) 000048 1780327811.465442 punch from 0x02da08 to 0x0377de (0x9dd7 bytes) 000049 1780327811.477265 write 0x021df3 thru 0x028fdd (0x71eb bytes) 000050 1780327811.488656 mapread 0x02a1fc thru 0x0377de (0xd5e3 bytes) 000051 1780327811.505474 read 0x027cb9 thru 0x030828 (0x8b70 bytes) 000052 1780327811.506006 trunc from 0x032d47 to 0x0377de (0x4a98 bytes) 000053 1780327811.510988 mapread 0x017678 thru 0x017c4d (0x5d6 bytes) 000054 1780327811.527248 mapwrite 0x011913 thru 0x01ab25 (0x9213 bytes) 000055 1780327811.564380 mapwrite 0x0231bf thru 0x030f0f (0xdd51 bytes) 000056 1780327811.594364 mapread 0x031b80 thru 0x035035 (0x34b6 bytes) 000057 1780327811.600714 mapread 0x005622 thru 0x009d92 (0x4771 bytes) 000058 1780327811.616229 MIRROR_RESYNC 2 mirrors 000059 1780327811.945461 MIRROR_EXTEND to 3 mirrors 000060 1780327812.346851 mapwrite 0x035929 thru 0x03ffff (0xa6d7 bytes) 000062 1780327812.449436 write 0x01cc49 thru 0x029586 (0xc93e bytes) 000063 1780327812.466344 trunc from 0x03d0ae to 0x03ffff (0x2f52 bytes) 000064 1780327812.473762 trunc from 0x03bfd1 to 0x03ffff (0x402f bytes) 000065 1780327812.478167 fallocate 0x03292e thru 0x034346 (0x1a19 bytes) 000066 1780327812.486249 fallocate 0x01f7c3 thru 0x01f87d (0xbb bytes) 000067 1780327812.495417 trunc from 0x01079e to 0x017f0c (0x776f bytes) 000068 1780327812.499496 write 0x037935 thru 0x03ffff (0x86cb bytes) 000069 1780327812.501038 mapread 0x011b44 thru 0x01fbee (0xe0ab bytes) 000070 1780327812.512630 read 0x00bbbd thru 0x0111cd (0x5611 bytes) 000071 1780327812.513605 fallocate 0x00f8fd thru 0x016334 (0x6a38 bytes) 000072 1780327812.517537 punch from 0x03281d to 0x03ffff (0xd7e3 bytes) 000073 1780327812.534306 punch from 0x01f3b1 to 0x029aaa (0xa6fa bytes) 000074 1780327812.540137 trunc from 0x040000 to 0x023224 (0x1cddb bytes) 000075 1780327812.555934 fallocate 0x034c67 thru 0x038c23 (0x3fbd bytes) 000076 1780327812.559968 punch from 0x0001fd to 0x00aaab (0xa8af bytes) 000077 1780327812.566851 trunc from 0x038c24 to 0x0206ff (0x18524 bytes) 000078 1780327812.580393 trunc from 0x01e0ea to 0x0206ff (0x2616 bytes) 000079 1780327812.584346 mapread 0x010d64 thru 0x01d861 (0xcafe bytes) 000080 1780327812.585470 trunc from 0x005878 to 0x00e36a (0x8af3 bytes) 000081 1780327812.589816 trunc from 0x020700 to 0x01788e (0x8e71 bytes) 000082 1780327812.600278 punch from 0x00a0f6 to 0x00f512 (0x541d bytes) 000083 1780327812.606143 fallocate 0x0205bf thru 0x026bcd (0x660f bytes) 000084 1780327812.609263 punch from 0x002fac to 0x00e946 (0xb99b bytes) 000085 1780327812.611789 trunc from 0x01788f to 0x015693 (0x21fb bytes) 000086 1780327812.631187 read 0x00df4d thru 0x015693 (0x7747 bytes) 000089 1780327812.643474 mapread 0x012855 thru 0x015693 (0x2e3f bytes) 000090 1780327812.645152 mapwrite 0x02525b thru 0x026bf2 (0x1998 bytes) 000091 1780327812.707088 punch from 0x00e7fa to 0x01387d (0x5084 bytes) 000092 1780327812.720598 fallocate 0x00470d thru 0x00fdea (0xb6de bytes) 000093 1780327812.726828 fallocate 0x00b71a thru 0x019e03 (0xe6ea bytes) 000094 1780327812.732533 trunc from 0x026bf3 to 0x00d4a6 (0x1974c bytes) 000095 1780327812.747807 fallocate 0x02b5c2 thru 0x02c7c9 (0x1208 bytes) 000096 1780327812.756045 mapwrite 0x01a14f thru 0x01a1b3 (0x65 bytes) 000097 1780327812.794836 read 0x010e8c thru 0x01a1b3 (0x9328 bytes) 000099 1780327812.807733 punch from 0x00d196 to 0x00f7fa (0x2665 bytes) 000100 1780327812.818601 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a1b4): Mirror 3: 0x59da72a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a1b4): Mirror 3: 0x59da72a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a1b4: skipped 000101 1780327812.995876 trunc from 0x001499 to 0x00909c (0x7c04 bytes) 000102 1780327813.003448 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) 000103 1780327813.101273 fallocate 0x0111a1 thru 0x0122b8 (0x1118 bytes) 000104 1780327813.106910 trunc from 0x005310 to 0x008438 (0x3129 bytes) 000105 1780327813.113309 fallocate 0x00d22f thru 0x0135a5 (0x6377 bytes) 000107 1780327813.117729 mapread 0x009b12 thru 0x011bd5 (0x80c4 bytes) 000108 1780327813.130071 write 0x007c4f thru 0x0169bd (0xed6f bytes) 000109 1780327813.139397 trunc from 0x01a1b4 to 0x006a49 (0x1376a bytes) 000110 1780327813.154114 fallocate 0x033c03 thru 0x03fcb5 (0xc0b3 bytes) 000111 1780327813.158891 fallocate 0x014c59 thru 0x01e67d (0x9a25 bytes) 000112 1780327813.162171 write 0x038273 thru 0x03dcdf (0x5a6d bytes) 000113 1780327813.163270 read 0x01c4ff thru 0x0230ca (0x6bcc bytes) 000115 1780327813.169214 fallocate 0x0017f9 thru 0x005567 (0x3d6f bytes) 000116 1780327813.172320 mapread 0x0360a2 thru 0x03dcdf (0x7c3e bytes) 000117 1780327813.178450 read 0x03ab7a thru 0x03dcdf (0x3166 bytes) 000118 1780327813.179167 fallocate 0x01276c thru 0x021d3d (0xf5d2 bytes) 000119 1780327813.182968 trunc from 0x03dce0 to 0x0114dd (0x2c802 bytes) 000120 1780327813.196546 fallocate 0x00e9de thru 0x01e0fc (0xf71f bytes) 000121 1780327813.211398 punch from 0x012dbc to 0x01e0fc (0xb341 bytes) 000122 1780327813.215824 fallocate 0x03267b thru 0x03ab90 (0x8516 bytes) 000123 1780327813.222341 mapread 0x0062f3 thru 0x013b2a (0xd838 bytes) 000124 1780327813.235276 trunc from 0x01a516 to 0x01e0fc (0x3be7 bytes) 000125 1780327813.239406 mapwrite 0x025dad thru 0x028bba (0x2e0e bytes) 000126 1780327813.282367 mapwrite 0x00765d thru 0x00bb6b (0x450f bytes) 000127 1780327813.306388 mapwrite 0x025126 thru 0x02f35a (0xa235 bytes) 000130 1780327813.350247 read 0x029ec0 thru 0x02f35a (0x549b bytes) 000131 1780327813.350900 mapread 0x02cd40 thru 0x02d147 (0x408 bytes) 000132 1780327813.352100 trunc from 0x020e57 to 0x023988 (0x2b32 bytes) 000133 1780327813.356165 trunc from 0x02f35b to 0x023f8a (0xb3d0 bytes) 000134 1780327813.367513 mapwrite 0x020873 thru 0x026700 (0x5e8e bytes) 000136 1780327813.397911 fallocate 0x02404f thru 0x02783e (0x37f0 bytes) 000137 1780327813.403909 trunc from 0x014008 to 0x01f030 (0xb029 bytes) 000138 1780327813.413444 mapread 0x01f682 thru 0x02783e (0x81bd bytes) 000139 1780327813.421244 trunc from 0x02783f to 0x000378 (0x274c6 bytes) 000141 1780327813.434384 mapwrite 0x02d59d thru 0x0346e6 (0x714a bytes) 000142 1780327813.483176 fallocate 0x010885 thru 0x0146dc (0x3e58 bytes) 000143 1780327813.486655 mapwrite 0x006655 thru 0x012698 (0xc044 bytes) 000145 1780327813.550568 fallocate 0x016aa5 thru 0x018730 (0x1c8c bytes) 000146 1780327813.554905 mapwrite 0x023b37 thru 0x02918f (0x5659 bytes) 000147 1780327813.591104 mapwrite 0x015b53 thru 0x0259ab (0xfe59 bytes) 000148 1780327813.662468 punch from 0x0026ba to 0x00a627 (0x7f6e bytes) 000149 1780327813.666707 punch from 0x021309 to 0x026d5d (0x5a55 bytes) 000150 1780327813.670135 fallocate 0x00fced thru 0x0114ab (0x17bf bytes) 000151 1780327813.675345 mapwrite 0x00b308 thru 0x01a48a (0xf183 bytes) 000152 1780327813.711405 write 0x00a21c thru 0x0112f8 (0x70dd bytes) 000153 1780327813.715223 read 0x01c4d4 thru 0x020611 (0x413e bytes) 000154 1780327813.715901 write 0x02d126 thru 0x037eb2 (0xad8d bytes) 000156 1780327813.717572 trunc from 0x01496c to 0x023716 (0xedab bytes) 000157 1780327813.721010 trunc from 0x02e02e to 0x037eb2 (0x9e85 bytes) 000158 1780327813.731898 write 0x008f8c thru 0x01397c (0xa9f1 bytes) 000159 1780327813.743616 fallocate 0x0280c9 thru 0x02ab22 (0x2a5a bytes) 000160 1780327813.749031 fallocate 0x00fa68 thru 0x016983 (0x6f1c bytes) 000161 1780327813.754180 fallocate 0x02d8a6 thru 0x02fb7d (0x22d8 bytes) 000162 1780327813.764113 MIRROR_EXTEND to 2 mirrors 000164 1780327814.241609 trunc from 0x027917 to 0x02fac2 (0x81ac bytes) 000165 1780327814.265371 read 0x0132f4 thru 0x0155c6 (0x22d3 bytes) 000166 1780327814.273664 fallocate 0x013fc5 thru 0x018a94 (0x4ad0 bytes) 000167 1780327814.283050 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) 000168 1780327814.400211 MIRROR_EXTEND to 2 mirrors 000169 1780327814.812145 write 0x03d062 thru 0x03d2bb (0x25a bytes) 000170 1780327814.845901 mapwrite 0x01a012 thru 0x022930 (0x891f bytes) 000171 1780327814.882033 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) 000172 1780327814.981827 trunc from 0x02e4f7 to 0x031326 (0x2e30 bytes) 000173 1780327814.985903 read 0x02854f thru 0x0383ad (0xfe5f bytes) 000174 1780327814.995502 fallocate 0x021322 thru 0x02cd9a (0xba79 bytes) 000175 1780327815.001373 mapwrite 0x03c06a thru 0x03ffff (0x3f96 bytes) 000176 1780327815.041987 trunc from 0x01f77e to 0x0268f9 (0x717c bytes) 000178 1780327815.048405 read 0x0224d8 thru 0x0266e9 (0x4212 bytes) 000179 1780327815.055299 punch from 0x03bb13 to 0x03ffff (0x44ed bytes) 000180 1780327815.058723 trunc from 0x02106f to 0x02521c (0x41ae bytes) 000181 1780327815.066349 fallocate 0x03dccc thru 0x03f21b (0x1550 bytes) 000182 1780327815.074632 trunc from 0x040000 to 0x00b1d3 (0x34e2c bytes) 000183 1780327815.098850 MIRROR_EXTEND to 2 mirrors 000184 1780327815.474021 trunc from 0x000817 to 0x00b1d3 (0xa9bd bytes) 000185 1780327815.502957 trunc from 0x00ac5d to 0x00b1d3 (0x577 bytes) 000186 1780327815.509594 trunc from 0x0025b8 to 0x00aa7d (0x84c6 bytes) 000187 1780327815.512712 trunc from 0x004a89 to 0x00b1d3 (0x674b bytes) 000188 1780327815.516638 mapwrite 0x02a957 thru 0x033c76 (0x9320 bytes) 000189 1780327815.599461 trunc from 0x033c77 to 0x01a06d (0x19c09 bytes) 000190 1780327815.611934 trunc from 0x01a06e to 0x03fc67 (0x25bfa bytes) 000191 1780327815.626841 trunc from 0x01569b to 0x021d06 (0xc66c bytes) 000192 1780327815.639064 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) 000193 1780327815.743229 fallocate 0x011eb6 thru 0x01305d (0x11a8 bytes) 000194 1780327815.748646 fallocate 0x0158bf thru 0x017012 (0x1754 bytes) 000195 1780327815.755949 punch from 0x004c2d to 0x005dc9 (0x119d bytes) 000196 1780327815.766967 punch from 0x027a91 to 0x031b89 (0xa0f9 bytes) 000197 1780327815.774508 trunc from 0x0160d6 to 0x016757 (0x682 bytes) 000198 1780327815.780187 read 0x010d57 thru 0x01b887 (0xab31 bytes) 000199 1780327815.786471 write 0x02ed5b thru 0x0373a4 (0x864a bytes) 000200 1780327815.809272 read 0x00e6c4 thru 0x0110d1 (0x2a0e bytes) 000201 1780327815.822955 mapread 0x0282f3 thru 0x02bc9c (0x39aa bytes) 000202 1780327815.823724 fallocate 0x00aded thru 0x014834 (0x9a48 bytes) 000203 1780327815.828279 punch from 0x0329d3 to 0x033267 (0x895 bytes) 000204 1780327815.840624 trunc from 0x021ec0 to 0x022d8e (0xecf bytes) 000205 1780327815.844359 mapread 0x0170f7 thru 0x022bfe (0xbb08 bytes) 000206 1780327815.852253 read 0x018f6b thru 0x01e94d (0x59e3 bytes) 000207 1780327815.852626 trunc from 0x01272a to 0x013140 (0xa17 bytes) 000208 1780327815.855584 fallocate 0x00142d thru 0x00ba8e (0xa662 bytes) 000209 1780327815.859897 mapread 0x00bf2c thru 0x01a5b5 (0xe68a bytes) 000210 1780327815.866315 punch from 0x031027 to 0x03b604 (0xa5de bytes) 000211 1780327815.871240 punch from 0x036bdf to 0x03d66e (0x6a90 bytes) 000212 1780327815.876013 mapread 0x0144c9 thru 0x01d36d (0x8ea5 bytes) 000213 1780327815.877407 punch from 0x027659 to 0x02c349 (0x4cf1 bytes) 000214 1780327815.885272 MIRROR_EXTEND to 2 mirrors 000215 1780327816.324049 trunc from 0x01526d to 0x020c94 (0xba28 bytes) 000217 1780327816.344189 read 0x02d1bb thru 0x03369b (0x64e1 bytes) 000218 1780327816.352045 punch from 0x032f99 to 0x03883f (0x58a7 bytes) 000220 1780327816.357580 read 0x02a94a thru 0x034df7 (0xa4ae bytes) 000221 1780327816.373865 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) 000222 1780327816.447271 mapwrite 0x0234c9 thru 0x03097e (0xd4b6 bytes) 000223 1780327816.481017 MIRROR_EXTEND to 2 mirrors 000224 1780327816.869078 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) 000225 1780327816.974600 fallocate 0x033ae9 thru 0x0373fd (0x3915 bytes) 000226 1780327816.990916 MIRROR_EXTEND to 2 mirrors 000227 1780327817.479138 mapwrite 0x0010df thru 0x0013d2 (0x2f4 bytes) 000228 1780327817.543364 trunc from 0x00ea33 to 0x01dc2e (0xf1fc bytes) 000229 1780327817.551164 mapwrite 0x007569 thru 0x016803 (0xf29b bytes) 000230 1780327817.598641 fallocate 0x02c062 thru 0x03419d (0x813c bytes) 000231 1780327817.605815 fallocate 0x01f8d6 thru 0x029a94 (0xa1bf bytes) 000232 1780327817.615645 read 0x01e5bb thru 0x02d9e6 (0xf42c bytes) 000233 1780327817.627905 mapread 0x02e93a thru 0x034142 (0x5809 bytes) 000234 1780327817.635933 mapread 0x00f23e thru 0x012482 (0x3245 bytes) 000235 1780327817.639050 trunc from 0x03fc68 to 0x0131cd (0x2ca9a bytes) 000236 1780327817.663769 trunc from 0x002623 to 0x004256 (0x1c34 bytes) 000237 1780327817.668838 mapread 0x006c48 thru 0x009c78 (0x3031 bytes) 000238 1780327817.670223 trunc from 0x00e0bf to 0x0131cd (0x510f bytes) 000239 1780327817.680296 trunc from 0x0131ce to 0x03a9c7 (0x277fa bytes) 000240 1780327817.692208 fallocate 0x03e611 thru 0x03ffff (0x19ef bytes) 000241 1780327817.696112 mapwrite 0x020122 thru 0x02ee66 (0xed45 bytes) 000242 1780327817.734106 trunc from 0x023acf to 0x02fe4c (0xc37e bytes) 000243 1780327817.743308 read 0x01fae4 thru 0x0265de (0x6afb bytes) 000244 1780327817.752933 fallocate 0x02aec8 thru 0x0350a5 (0xa1de bytes) 000245 1780327817.760717 MIRROR_RESYNC 2 mirrors 000246 1780327818.310042 read 0x000ce4 thru 0x00eb25 (0xde42 bytes) 000247 1780327818.323796 trunc from 0x028bd7 to 0x02d38e (0x47b8 bytes) 000248 1780327818.343585 write 0x028f62 thru 0x0356af (0xc74e bytes) 000249 1780327818.350540 write 0x02ad0e thru 0x0398d7 (0xebca bytes) 000250 1780327818.358025 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc83050a4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8eafbe95 Mirror 3: 0x8eafbe95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000251 1780327819.141777 write 0x00d29d thru 0x00eb94 (0x18f8 bytes) 000252 1780327819.219480 trunc from 0x040000 to 0x0182f7 (0x27d08 bytes) 000253 1780327819.257890 punch from 0x00feba to 0x012ed8 (0x301f bytes) 000254 1780327819.262882 write 0x007201 thru 0x0120ca (0xaeca bytes) 000255 1780327819.302410 trunc from 0x001240 to 0x009d3c (0x8afd bytes) 000257 1780327819.322342 mapread 0x00ca38 thru 0x00e68e (0x1c57 bytes) 000258 1780327819.324023 trunc from 0x002ea0 to 0x00e594 (0xb6f5 bytes) 000260 1780327819.337428 read 0x007565 thru 0x009faf (0x2a4b bytes) 000261 1780327819.344320 fallocate 0x0074bc thru 0x010090 (0x8bd5 bytes) 000262 1780327819.350368 fallocate 0x008084 thru 0x0144dd (0xc45a bytes) 000263 1780327819.359980 mapwrite 0x01f2fc thru 0x0245d6 (0x52db bytes) 000264 1780327819.414471 mapwrite 0x002710 thru 0x00b796 (0x9087 bytes) 000265 1780327819.439300 mapwrite 0x03da44 thru 0x03f01e (0x15db bytes) 000266 1780327819.475275 mapread 0x018a34 thru 0x0205da (0x7ba7 bytes) 000267 1780327819.476884 punch from 0x0367ef to 0x03f01e (0x8830 bytes) 000268 1780327819.482268 trunc from 0x03f01f to 0x028316 (0x16d08 bytes) 000269 1780327819.495422 write 0x006d30 thru 0x014bd8 (0xdea9 bytes) 000270 1780327819.499949 write 0x0223f3 thru 0x02258b (0x199 bytes) 000271 1780327819.500882 fallocate 0x020779 thru 0x025f9c (0x5824 bytes) 000272 1780327819.504736 mapwrite 0x03899a thru 0x03ffff (0x7666 bytes) 000273 1780327819.540386 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8eafbe95 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8eafbe95 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000274 1780327819.732502 mapwrite 0x0386a6 thru 0x03ffff (0x795a bytes) 000275 1780327819.771276 write 0x003e30 thru 0x00c63c (0x880d bytes) 000276 1780327819.780470 punch from 0x0285d2 to 0x033798 (0xb1c7 bytes) 000278 1780327819.784221 mapwrite 0x00767e thru 0x008f52 (0x18d5 bytes) 000280 1780327819.797790 MIRROR_RESYNC 2 mirrors 000281 1780327820.185378 trunc from 0x040000 to 0x035a6f (0xa590 bytes) 000282 1780327820.214484 trunc from 0x035a70 to 0x02eecb (0x6ba4 bytes) 000283 1780327820.229594 read 0x02b2eb thru 0x02eecb (0x3be1 bytes) 000284 1780327820.234952 punch from 0x01a0ca to 0x023ca8 (0x9bdf bytes) 000285 1780327820.238103 mapwrite 0x03cf7c thru 0x03df88 (0x100d bytes) 000286 1780327820.279313 fallocate 0x0367c6 thru 0x03ffff (0x983a bytes) 000287 1780327820.285060 mapread 0x031fcb thru 0x03b741 (0x9777 bytes) 000288 1780327820.304488 mapwrite 0x0283f1 thru 0x0345fb (0xc20b bytes) 000289 1780327820.331597 fallocate 0x02771e thru 0x02d1a9 (0x5a8c bytes) 000290 1780327820.335739 read 0x02c5fb thru 0x02e666 (0x206c bytes) 000291 1780327820.336336 mapread 0x0318fa thru 0x033032 (0x1739 bytes) 000292 1780327820.337216 punch from 0x00fe9e to 0x01f81b (0xf97e bytes) 000293 1780327820.342736 read 0x00f521 thru 0x01391f (0x43ff bytes) 000294 1780327820.355741 trunc from 0x0022ba to 0x0033cb (0x1112 bytes) 000295 1780327820.360970 write 0x02b799 thru 0x02db86 (0x23ee bytes) 000296 1780327820.362182 fallocate 0x00e591 thru 0x013473 (0x4ee3 bytes) 000297 1780327820.374996 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) 000298 1780327820.539847 mapwrite 0x0036f2 thru 0x012734 (0xf043 bytes) 000299 1780327820.597029 fallocate 0x008bae thru 0x00f970 (0x6dc3 bytes) 000301 1780327820.602246 read 0x002190 thru 0x00f366 (0xd1d7 bytes) 000302 1780327820.603025 punch from 0x00e76e to 0x014e9e (0x6731 bytes) 000306 1780327820.608993 fallocate 0x022ec6 thru 0x02584f (0x298a bytes) 000307 1780327820.612506 trunc from 0x040000 to 0x01838a (0x27c75 bytes) 000308 1780327820.630924 write 0x00662c thru 0x01611a (0xfaef bytes) 000309 1780327820.632961 write 0x036f09 thru 0x037355 (0x44d bytes) 000310 1780327820.635400 punch from 0x016aea to 0x01a974 (0x3e8b bytes) 000311 1780327820.646812 fallocate 0x01843a thru 0x018a06 (0x5cd bytes) 000313 1780327820.651362 trunc from 0x00ff64 to 0x016d2e (0x6dcb bytes) 000314 1780327820.657292 mapread 0x01b9c1 thru 0x0282d6 (0xc916 bytes) 000315 1780327820.670969 mapread 0x02a69a thru 0x036895 (0xc1fc bytes) 000316 1780327820.680934 mapwrite 0x027bf9 thru 0x02b33d (0x3745 bytes) 000317 1780327820.692451 punch from 0x0250e3 to 0x02f19f (0xa0bd bytes) 000318 1780327820.696031 mapwrite 0x01020b thru 0x018e36 (0x8c2c bytes) 000319 1780327820.714813 write 0x013a8d thru 0x01ccef (0x9263 bytes) 000320 1780327820.715857 trunc from 0x01d4e9 to 0x02a6ca (0xd1e2 bytes) 000321 1780327820.718977 trunc from 0x012d71 to 0x020f08 (0xe198 bytes) 000322 1780327820.732483 punch from 0x0019ab to 0x00b5f6 (0x9c4c bytes) 000324 1780327820.738121 read 0x01b1a1 thru 0x027d92 (0xcbf2 bytes) 000325 1780327820.745970 read 0x002f02 thru 0x00cbde (0x9cdd bytes) 000326 1780327820.746756 punch from 0x016309 to 0x024dcc (0xeac4 bytes) 000327 1780327820.750910 fallocate 0x02d038 thru 0x03c1ed (0xf1b6 bytes) 000330 1780327820.760335 MIRROR_EXTEND to 2 mirrors 000331 1780327821.332549 mapread 0x017380 thru 0x0192a4 (0x1f25 bytes) 000332 1780327821.349358 trunc from 0x03c1ee to 0x03dd93 (0x1ba6 bytes) 000333 1780327821.391489 read 0x009f4a thru 0x0192e7 (0xf39e bytes) 000334 1780327821.403343 read 0x00dde7 thru 0x00dfaa (0x1c4 bytes) 000335 1780327821.404365 mapwrite 0x00487d thru 0x012dc4 (0xe548 bytes) 000336 1780327821.448154 trunc from 0x033705 to 0x03dd93 (0xa68f bytes) 000337 1780327821.453640 punch from 0x02cbc3 to 0x034c26 (0x8064 bytes) 000338 1780327821.463790 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3dd94): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd94): Mirror 2: 0xda9144f8 (stale) CRC-32 checksum value for chunk [0, 0x3dd94): Mirror 1: 0x1ef96151 Mirror 3: 0x1ef96151 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd94: skipped 000339 1780327822.135522 trunc from 0x031cf0 to 0x03c533 (0xa844 bytes) 000340 1780327822.175789 MIRROR_RESYNC 3 mirrors 000341 1780327822.837043 MIRROR_RESYNC 3 mirrors 000342 1780327823.140158 mapread 0x01e04e thru 0x025d76 (0x7d29 bytes) 000343 1780327823.158473 mapwrite 0x0104f9 thru 0x01231f (0x1e27 bytes) 000344 1780327823.218983 write 0x03eb97 thru 0x03ffff (0x1469 bytes) 000345 1780327823.220142 punch from 0x02d14c to 0x02f107 (0x1fbc bytes) 000346 1780327823.226026 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x18af5fbe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x18af5fbe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000347 1780327823.391977 mapwrite 0x00c36a thru 0x00fd02 (0x3999 bytes) 000348 1780327823.432150 mapwrite 0x019a36 thru 0x01e8ed (0x4eb8 bytes) 000349 1780327823.459956 read 0x0271bc thru 0x033460 (0xc2a5 bytes) 000350 1780327823.472906 mapwrite 0x00ba91 thru 0x01962a (0xdb9a bytes) 000352 1780327823.503824 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x18af5fbe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6a907cd4 Mirror 4: 0x6a907cd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000353 1780327824.306950 trunc from 0x040000 to 0x0325b1 (0xda4e bytes) 000355 1780327824.395269 punch from 0x01d08c to 0x020fb2 (0x3f27 bytes) 000357 1780327824.402800 write 0x005950 thru 0x00b22e (0x58df bytes) 000358 1780327824.413151 punch from 0x024101 to 0x0279db (0x38db bytes) 000359 1780327824.418952 fallocate 0x01e78d thru 0x02ad69 (0xc5dd bytes) 000361 1780327824.424870 write 0x02a4a0 thru 0x02e7d4 (0x4335 bytes) 000362 1780327824.436223 write 0x002b3a thru 0x0111e4 (0xe6ab bytes) 000363 1780327824.450167 read 0x017f13 thru 0x02679f (0xe88d bytes) 000364 1780327824.459494 trunc from 0x0325b2 to 0x016d81 (0x1b830 bytes) 000365 1780327824.486134 MIRROR_RESYNC 3 mirrors 000366 1780327824.970425 mapread 0x0055cf thru 0x00cc00 (0x7632 bytes) 000367 1780327824.998213 punch from 0x00c74d to 0x00fc2e (0x34e2 bytes) 000368 1780327825.041588 mapread 0x014930 thru 0x016d81 (0x2452 bytes) 000369 1780327825.057623 mapwrite 0x01d96c thru 0x025832 (0x7ec7 bytes) 000370 1780327825.132735 trunc from 0x006f75 to 0x00e7dd (0x7869 bytes) 000372 1780327825.138572 fallocate 0x03e593 thru 0x03ffff (0x1a6d bytes) 000373 1780327825.142854 mapread 0x003b83 thru 0x00bd29 (0x81a7 bytes) 000374 1780327825.158855 trunc from 0x025833 to 0x00c661 (0x191d1 bytes) 000375 1780327825.174216 mapwrite 0x003b5b thru 0x00a31c (0x67c2 bytes) 000376 1780327825.204818 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc662): Mirror 4: 0xfdf3cba8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc662): Mirror 4: 0xfdf3cba8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc662: skipped 000377 1780327825.677238 write 0x0054b8 thru 0x011998 (0xc4e1 bytes) 000378 1780327825.709351 read 0x00faeb thru 0x011998 (0x1eae bytes) 000379 1780327825.710183 trunc from 0x005314 to 0x00769e (0x238b bytes) 000380 1780327825.726623 write 0x03b07e thru 0x03ffff (0x4f82 bytes) 000381 1780327825.732918 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x117638d1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x46f1003d Mirror 5: 0x46f1003d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000383 1780327826.478320 punch from 0x023def to 0x032560 (0xe772 bytes) 000384 1780327826.521535 mapwrite 0x00fb0f thru 0x01df72 (0xe464 bytes) 000385 1780327826.572137 mapwrite 0x01dd95 thru 0x02af82 (0xd1ee bytes) 000386 1780327826.596534 trunc from 0x02e3fc to 0x02e88b (0x490 bytes) 000387 1780327826.600868 trunc from 0x005aec to 0x005b3d (0x52 bytes) 000388 1780327826.605127 read 0x0295cb thru 0x038e64 (0xf89a bytes) 000390 1780327826.613662 punch from 0x00cbcf to 0x015040 (0x8472 bytes) 000391 1780327826.618422 fallocate 0x02188b thru 0x02e6b3 (0xce29 bytes) 000393 1780327826.623347 punch from 0x028d2d to 0x0350cb (0xc39f bytes) 000394 1780327826.628175 trunc from 0x0274c5 to 0x034461 (0xcf9d bytes) 000395 1780327826.633060 fallocate 0x03e3b2 thru 0x03ffff (0x1c4e bytes) 000396 1780327826.637037 trunc from 0x040000 to 0x03fb90 (0x46f bytes) 000397 1780327826.655455 mapread 0x0198b3 thru 0x02563c (0xbd8a bytes) 000398 1780327826.657699 read 0x036cfb thru 0x03a07b (0x3381 bytes) 000399 1780327826.672464 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fb91): Mirror 4: 0x508aa6da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb91): Mirror 4: 0x508aa6da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb91: skipped 000400 1780327826.901748 write 0x038cbd thru 0x03ffff (0x7343 bytes) 000401 1780327826.914882 punch from 0x019f60 to 0x025b33 (0xbbd4 bytes) 000402 1780327826.923202 trunc from 0x040000 to 0x018b66 (0x27499 bytes) 000403 1780327826.937664 read 0x0145e1 thru 0x018b66 (0x4586 bytes) 000404 1780327826.948213 trunc from 0x018b67 to 0x03cb7b (0x24015 bytes) 000405 1780327826.972313 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3cb7c): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb7c): Mirror 4: 0x3ed84587 (stale) CRC-32 checksum value for chunk [0, 0x3cb7c): Mirror 1: 0xbbfac546 Mirror 5: 0xbbfac546 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb7c: skipped 000406 1780327828.006404 fallocate 0x038822 thru 0x03ffff (0x77de bytes) 000407 1780327828.065211 write 0x03ef8a thru 0x03ffff (0x1076 bytes) 000408 1780327828.092918 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x117638d1 (stale) Mirror 5: 0x3887d555 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6b442772 Mirror 6: 0x6b442772 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000409 1780327829.380328 fallocate 0x00ce50 thru 0x00e6cf (0x1880 bytes) 000410 1780327829.423420 punch from 0x01e62f to 0x022a26 (0x43f8 bytes) 000413 1780327829.434437 write 0x00e536 thru 0x01454e (0x6019 bytes) 000414 1780327829.454587 trunc from 0x00cbcd to 0x00da99 (0xecd bytes) 000415 1780327829.462070 trunc from 0x040000 to 0x03db23 (0x24dc bytes) 000416 1780327829.482736 trunc from 0x033ba6 to 0x036259 (0x26b4 bytes) 000417 1780327829.488110 mapwrite 0x02c23f thru 0x033e1a (0x7bdc bytes) 000418 1780327829.533862 mapwrite 0x00b8f6 thru 0x0160db (0xa7e6 bytes) 000419 1780327829.577376 write 0x00f353 thru 0x019374 (0xa022 bytes) 000420 1780327829.583827 mapread 0x0044e2 thru 0x007903 (0x3422 bytes) 000421 1780327829.585761 mapread 0x002f2e thru 0x0046cb (0x179e bytes) 000422 1780327829.601507 read 0x0202b6 thru 0x02fcef (0xfa3a bytes) 000423 1780327829.615855 fallocate 0x03e9cf thru 0x03ffff (0x1631 bytes) 000424 1780327829.628381 punch from 0x01d300 to 0x027318 (0xa019 bytes) 000425 1780327829.634787 read 0x024e51 thru 0x02d424 (0x85d4 bytes) 000426 1780327829.656507 MIRROR_RESYNC 4 mirrors 000427 1780327831.402489 read 0x029dff thru 0x0331d1 (0x93d3 bytes) 000428 1780327831.412643 write 0x028e56 thru 0x0337cd (0xa978 bytes) 000429 1780327831.480635 mapwrite 0x02a721 thru 0x02b850 (0x1130 bytes) 000431 1780327831.531066 fallocate 0x011288 thru 0x01de92 (0xcc0b bytes) 000432 1780327831.539659 write 0x032a8f thru 0x03b36c (0x88de bytes) 000433 1780327831.567082 mapread 0x00bf4a thru 0x00f63d (0x36f4 bytes) 000434 1780327831.601641 mapwrite 0x00418b thru 0x013b79 (0xf9ef bytes) 000435 1780327831.705383 trunc from 0x02ef6b to 0x039471 (0xa507 bytes) 000436 1780327831.723147 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3db24): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db24): Mirror 4: 0x23c4c0f9 (stale) Mirror 5: 0x23c4c0f9 (stale) Mirror 6: 0x23c4c0f9 (stale) CRC-32 checksum value for chunk [0, 0x3db24): Mirror 1: 0x337e9ac7 Mirror 7: 0x337e9ac7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db24: skipped 000437 1780327833.302365 fallocate 0x028e32 thru 0x036994 (0xdb63 bytes) 000438 1780327833.341367 write 0x0348c4 thru 0x03f83a (0xaf77 bytes) 000439 1780327833.352254 trunc from 0x03330f to 0x03f83a (0xc52c bytes) 000440 1780327833.372299 mapread 0x01f035 thru 0x027ca2 (0x8c6e bytes) 000441 1780327833.391233 fallocate 0x039c81 thru 0x03ffff (0x637f bytes) 000442 1780327833.396959 mapwrite 0x0384b2 thru 0x03ffff (0x7b4e bytes) 000443 1780327833.448176 read 0x028276 thru 0x030045 (0x7dd0 bytes) 000444 1780327833.458951 fallocate 0x007f37 thru 0x00bea8 (0x3f72 bytes) 000445 1780327833.464233 mapwrite 0x000d09 thru 0x009ce5 (0x8fdd bytes) 000446 1780327833.494873 trunc from 0x040000 to 0x006ff9 (0x39006 bytes) 000447 1780327833.521573 mapread 0x004fad thru 0x006ff9 (0x204d bytes) 000448 1780327833.523415 write 0x003cce thru 0x0080dc (0x440f bytes) 000449 1780327833.529909 mapread 0x003901 thru 0x0080dc (0x47dc bytes) 000450 1780327833.533457 read 0x00215c thru 0x0080dc (0x5f81 bytes) 000451 1780327833.535205 write 0x03e366 thru 0x03ffff (0x1c9a bytes) 000452 1780327833.536898 mapread 0x01aadc thru 0x01df24 (0x3449 bytes) 000453 1780327833.550011 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6f37351d (stale) Mirror 5: 0x6f37351d (stale) Mirror 6: 0x6f37351d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6f37351d (stale) Mirror 5: 0x6f37351d (stale) Mirror 6: 0x6f37351d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000454 1780327834.039493 trunc from 0x040000 to 0x025d80 (0x1a27f bytes) 000455 1780327834.086158 mapread 0x010f9f thru 0x01d216 (0xc278 bytes) 000456 1780327834.200987 fallocate 0x033683 thru 0x0382a9 (0x4c27 bytes) 000457 1780327834.212960 trunc from 0x02c9dc to 0x0382a9 (0xb8ce bytes) 000458 1780327834.263114 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x382aa): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x382aa): Mirror 4: 0xe346a0f2 (stale) Mirror 5: 0xe346a0f2 (stale) Mirror 6: 0xe346a0f2 (stale) CRC-32 checksum value for chunk [0, 0x382aa): Mirror 1: 0xec8a3966 Mirror 7: 0xec8a3966 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x382aa: skipped 000459 1780327836.382019 write 0x025654 thru 0x030b3e (0xb4eb bytes) 000460 1780327836.537520 fallocate 0x012a14 thru 0x018a19 (0x6006 bytes) 000461 1780327836.564727 mapread 0x016f7a thru 0x01c3b0 (0x5437 bytes) 000462 1780327836.611283 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x382aa): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x382aa): Mirror 4: 0xe346a0f2 (stale) Mirror 5: 0xe346a0f2 (stale) Mirror 6: 0xe346a0f2 (stale) Mirror 7: 0xec8a3966 (stale) CRC-32 checksum value for chunk [0, 0x382aa): Mirror 1: 0x84651a72 Mirror 8: 0x84651a72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x382aa: skipped 000463 1780327838.312658 mapread 0x02e3f1 thru 0x0382a9 (0x9eb9 bytes) 000464 1780327838.359343 MIRROR_RESYNC 6 mirrors 000465 1780327840.962092 trunc from 0x0382aa to 0x01e0f9 (0x1a1b0 bytes) 000466 1780327841.000473 mapread 0x012388 thru 0x01aa63 (0x86dc bytes) 000467 1780327841.021973 punch from 0x012b3b to 0x015d77 (0x323d bytes) 000468 1780327841.027983 fallocate 0x000ddf thru 0x002cb6 (0x1ed8 bytes) 000469 1780327841.035176 read 0x016c1f thru 0x01a11a (0x34fc bytes) 000470 1780327841.036503 write 0x0232c7 thru 0x0317fa (0xe534 bytes) 000471 1780327841.041242 mapread 0x022519 thru 0x022f2d (0xa15 bytes) 000472 1780327841.053217 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x317fb): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x317fb): Mirror 4: 0xa705f796 (stale) Mirror 5: 0xa705f796 (stale) Mirror 6: 0xa705f796 (stale) Mirror 7: 0xa705f796 (stale) Mirror 8: 0xa705f796 (stale) CRC-32 checksum value for chunk [0, 0x317fb): Mirror 1: 0xeda1ebf1 Mirror 9: 0xeda1ebf1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x317fb: skipped 000473 1780327842.492298 mapread 0x025fde thru 0x02b2e7 (0x530a bytes) 000475 1780327842.557509 trunc from 0x02bcf1 to 0x02ff57 (0x4267 bytes) 000477 1780327842.630091 MIRROR_RESYNC 7 mirrors 000478 1780327844.138291 MIRROR_RESYNC 7 mirrors 000479 1780327844.848086 mapread 0x02694d thru 0x02a4d6 (0x3b8a bytes) 000480 1780327844.884165 trunc from 0x0317fb to 0x01cc32 (0x14bc8 bytes) 000481 1780327844.959493 trunc from 0x01cc33 to 0x016fd4 (0x5c5e bytes) 000482 1780327845.050293 trunc from 0x016fd5 to 0x01f495 (0x84c1 bytes) 000483 1780327845.109116 punch from 0x00e146 to 0x00f3d7 (0x1292 bytes) 000484 1780327845.134422 read 0x01a756 thru 0x01f495 (0x4d40 bytes) 000485 1780327845.155206 mapwrite 0x005189 thru 0x007add (0x2955 bytes) 000486 1780327845.205072 mapread 0x019576 thru 0x01e6bd (0x5148 bytes) 000487 1780327845.226293 mapwrite 0x02005f thru 0x02d5e2 (0xd584 bytes) 000488 1780327845.394718 mapread 0x027dd9 thru 0x02d5e2 (0x580a bytes) 000490 1780327845.397841 mapwrite 0x01e434 thru 0x02e29d (0xfe6a bytes) 000491 1780327845.471703 write 0x01f104 thru 0x023900 (0x47fd bytes) 000492 1780327845.473080 mapwrite 0x005301 thru 0x00cacf (0x77cf bytes) 000493 1780327845.505941 MIRROR_RESYNC 7 mirrors 000494 1780327847.582715 mapread 0x00b522 thru 0x014c69 (0x9748 bytes) 000495 1780327847.601123 mapwrite 0x027b70 thru 0x0364b7 (0xe948 bytes) 000497 1780327847.691267 mapwrite 0x02a2e3 thru 0x02b80f (0x152d bytes) 000499 1780327847.767510 read 0x02e313 thru 0x032fe0 (0x4cce bytes) 000500 1780327847.768388 mapwrite 0x03b46b thru 0x03ffff (0x4b95 bytes) 000501 1780327847.816415 punch from 0x032b8d to 0x036c9a (0x410e bytes) 000502 1780327847.825172 MIRROR_RESYNC 7 mirrors 000503 1780327849.943250 punch from 0x0274ca to 0x02c025 (0x4b5c bytes) 000504 1780327850.008571 fallocate 0x032a57 thru 0x03c193 (0x973d bytes) 000505 1780327850.024455 punch from 0x00abdc to 0x012ea8 (0x82cd bytes) 000506 1780327850.061034 MIRROR_RESYNC 7 mirrors 000507 1780327855.251925 punch from 0x00fa62 to 0x0165ab (0x6b4a bytes) 000508 1780327855.485989 punch from 0x017a8a to 0x026355 (0xe8cc bytes) 000509 1780327855.493124 mapwrite 0x034759 thru 0x03f99c (0xb244 bytes) 000510 1780327855.584010 MIRROR_RESYNC 7 mirrors 000511 1780327860.111018 mapread 0x024471 thru 0x028afd (0x468d bytes) 000512 1780327860.144033 write 0x025b12 thru 0x027cd1 (0x21c0 bytes) 000513 1780327860.190912 MIRROR_RESYNC 7 mirrors 000514 1780327861.904984 mapread 0x0070e1 thru 0x0155b0 (0xe4d0 bytes) 000515 1780327861.957265 MIRROR_EXTEND to 8 mirrors 000516 1780327863.395428 trunc from 0x02a9fa to 0x032656 (0x7c5d bytes) 000517 1780327863.447489 mapread 0x03b6dd thru 0x03ffff (0x4923 bytes) 000519 1780327863.465524 trunc from 0x040000 to 0x002b65 (0x3d49a bytes) 000520 1780327863.488015 punch from 0x001b9b to 0x002b65 (0xfcb bytes) 000521 1780327863.494055 punch from 0x001194 to 0x002b65 (0x19d2 bytes) 000522 1780327863.498119 trunc from 0x002b66 to 0x019228 (0x166c3 bytes) 000523 1780327863.516301 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19229): Mirror 4: 0xd832e9ae (stale) Mirror 5: 0xd832e9ae (stale) Mirror 6: 0xd832e9ae (stale) Mirror 8: 0xd832e9ae (stale) Mirror 9: 0xd832e9ae (stale) Mirror 10: 0xd832e9ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19229): Mirror 4: 0xd832e9ae (stale) Mirror 5: 0xd832e9ae (stale) Mirror 6: 0xd832e9ae (stale) Mirror 8: 0xd832e9ae (stale) Mirror 9: 0xd832e9ae (stale) Mirror 10: 0xd832e9ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19229: skipped 000524 1780327863.925394 fallocate 0x0175a1 thru 0x0254b2 (0xdf12 bytes) 000525 1780327863.934995 MIRROR_RESYNC 7 mirrors 000526 1780327865.693313 mapread 0x00e2a4 thru 0x01051d (0x227a bytes) 000527 1780327865.718972 write 0x019740 thru 0x01da97 (0x4358 bytes) 000528 1780327865.767034 write 0x01737b thru 0x0242a3 (0xcf29 bytes) 000529 1780327865.770024 trunc from 0x00ff37 to 0x014446 (0x4510 bytes) 000530 1780327865.774881 mapread 0x0128d7 thru 0x01efac (0xc6d6 bytes) 000532 1780327865.792310 read 0x01c910 thru 0x01cffe (0x6ef bytes) 000533 1780327865.793418 trunc from 0x00e9d8 to 0x0104f4 (0x1b1d bytes) 000534 1780327865.798631 read 0x006c7c thru 0x01443a (0xd7bf bytes) 000535 1780327865.809215 write 0x024397 thru 0x02d849 (0x94b3 bytes) 000536 1780327865.812061 trunc from 0x009c39 to 0x00c324 (0x26ec bytes) 000537 1780327865.817862 punch from 0x0151d0 to 0x02489f (0xf6d0 bytes) 000538 1780327865.836143 write 0x0165e1 thru 0x017e0c (0x182c bytes) 000540 1780327865.856927 trunc from 0x02d84a to 0x031c57 (0x440e bytes) 000541 1780327865.888147 trunc from 0x02a22b to 0x03098d (0x6763 bytes) 000542 1780327865.892830 mapwrite 0x0103a1 thru 0x01f4ef (0xf14f bytes) 000543 1780327865.954253 mapread 0x014349 thru 0x0241e9 (0xfea1 bytes) 000544 1780327865.976687 trunc from 0x031c58 to 0x0101b5 (0x21aa2 bytes) 000545 1780327865.994543 mapread 0x002d71 thru 0x00e43e (0xb6ce bytes) 000546 1780327866.018448 trunc from 0x00bd05 to 0x0101b5 (0x44b1 bytes) 000547 1780327866.024903 write 0x03055e thru 0x0326c1 (0x2164 bytes) 000548 1780327866.031903 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x326c2): Mirror 4: 0x8674b3e4 (stale) Mirror 5: 0x8674b3e4 (stale) Mirror 6: 0x8674b3e4 (stale) Mirror 9: 0x8674b3e4 (stale) Mirror 10: 0x8674b3e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326c2): Mirror 4: 0x8674b3e4 (stale) Mirror 5: 0x8674b3e4 (stale) Mirror 6: 0x8674b3e4 (stale) Mirror 9: 0x8674b3e4 (stale) Mirror 10: 0x8674b3e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326c2: skipped 000549 1780327867.104054 read 0x030046 thru 0x0326c1 (0x267c bytes) 000550 1780327867.136465 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x326c2): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326c2): Mirror 4: 0x8674b3e4 (stale) Mirror 5: 0x8674b3e4 (stale) Mirror 6: 0x8674b3e4 (stale) Mirror 9: 0x8674b3e4 (stale) Mirror 10: 0x8674b3e4 (stale) CRC-32 checksum value for chunk [0, 0x326c2): Mirror 1: 0xcabe2692 Mirror 11: 0xcabe2692 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326c2: skipped 000551 1780327868.697866 mapread 0x0195c6 thru 0x01e0fd (0x4b38 bytes) 000552 1780327868.768127 write 0x020723 thru 0x02508d (0x496b bytes) 000553 1780327868.925025 trunc from 0x0326c2 to 0x007193 (0x2b52e bytes) 000554 1780327868.944508 mapwrite 0x03e2eb thru 0x03f071 (0xd87 bytes) 000555 1780327868.976282 fallocate 0x02e205 thru 0x03ca4f (0xe84b bytes) 000557 1780327868.983125 write 0x009315 thru 0x00bf47 (0x2c33 bytes) 000559 1780327868.997147 write 0x00e481 thru 0x017e02 (0x9982 bytes) 000560 1780327869.002984 mapwrite 0x03d03e thru 0x03ffff (0x2fc2 bytes) 000561 1780327869.036190 write 0x005482 thru 0x013b19 (0xe698 bytes) 000562 1780327869.038149 mapwrite 0x024378 thru 0x02db22 (0x97ab bytes) 000563 1780327869.093547 fallocate 0x01f32d thru 0x027619 (0x82ed bytes) 000564 1780327869.099084 write 0x029f3e thru 0x02b9ba (0x1a7d bytes) 000565 1780327869.105078 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xbd713bde (stale) Mirror 6: 0xbd713bde (stale) Mirror 9: 0xbd713bde (stale) Mirror 10: 0xbd713bde (stale) Mirror 11: 0x32ecbe0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xbd713bde (stale) Mirror 6: 0xbd713bde (stale) Mirror 9: 0xbd713bde (stale) Mirror 10: 0xbd713bde (stale) Mirror 11: 0x32ecbe0c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000566 1780327869.834304 mapread 0x02c46b thru 0x02f375 (0x2f0b bytes) 000567 1780327869.852332 mapwrite 0x03688f thru 0x0371e1 (0x953 bytes) 000568 1780327869.886458 read 0x01b544 thru 0x029067 (0xdb24 bytes) 000569 1780327870.022784 mapread 0x02fa9f thru 0x03c5e1 (0xcb43 bytes) 000570 1780327870.026449 read 0x03a8ef thru 0x03ffff (0x5711 bytes) 000571 1780327870.028381 fallocate 0x00c550 thru 0x0180e0 (0xbb91 bytes) 000572 1780327870.036440 fallocate 0x00edbe thru 0x012c69 (0x3eac bytes) 000573 1780327870.044333 fallocate 0x028bb0 thru 0x032b50 (0x9fa1 bytes) 000574 1780327870.051320 trunc from 0x02b7ef to 0x038808 (0xd01a bytes) 000575 1780327870.064035 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xbd713bde (stale) Mirror 6: 0xbd713bde (stale) Mirror 9: 0xbd713bde (stale) Mirror 11: 0x32ecbe0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xbd713bde (stale) Mirror 6: 0xbd713bde (stale) Mirror 9: 0xbd713bde (stale) Mirror 11: 0x32ecbe0c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000576 1780327870.454974 write 0x01905a thru 0x020c19 (0x7bc0 bytes) 000577 1780327870.471173 trunc from 0x03c482 to 0x03fd89 (0x3908 bytes) 000578 1780327870.477833 read 0x01fb0f thru 0x028c84 (0x9176 bytes) 000579 1780327870.486837 trunc from 0x040000 to 0x02ee17 (0x111e8 bytes) 000580 1780327870.506905 mapread 0x02737b thru 0x02ee17 (0x7a9d bytes) 000581 1780327870.515119 write 0x02b977 thru 0x03870e (0xcd98 bytes) 000582 1780327870.517939 mapread 0x001280 thru 0x004e81 (0x3c02 bytes) 000583 1780327870.530581 MIRROR_RESYNC 5 mirrors 000584 1780327874.254885 mapread 0x025cb5 thru 0x033a4b (0xdd97 bytes) 000585 1780327874.341578 trunc from 0x037aad to 0x03870e (0xc62 bytes) 000586 1780327874.451227 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3870f): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3870f): Mirror 4: 0xdf916fab (stale) Mirror 6: 0xdf916fab (stale) Mirror 9: 0xdf916fab (stale) Mirror 11: 0xdf916fab (stale) CRC-32 checksum value for chunk [0, 0x3870f): Mirror 1: 0x62760b93 Mirror 12: 0x62760b93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3870f: skipped 000587 1780327876.969214 trunc from 0x03870f to 0x0252d0 (0x1343e bytes) 000588 1780327877.076201 trunc from 0x01464e to 0x018bd2 (0x4585 bytes) 000589 1780327877.091917 read 0x01c1b3 thru 0x020f88 (0x4dd6 bytes) 000590 1780327877.106860 fallocate 0x0313ca thru 0x0392ba (0x7ef1 bytes) 000592 1780327877.131175 punch from 0x022cba to 0x0252d0 (0x2617 bytes) 000593 1780327877.137125 mapwrite 0x02984c thru 0x036ce2 (0xd497 bytes) 000594 1780327877.362725 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36ce3): Mirror 4: 0x5cb3ebcc (stale) Mirror 9: 0x5cb3ebcc (stale) Mirror 11: 0x5cb3ebcc (stale) Mirror 12: 0x5cb3ebcc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ce3): Mirror 4: 0x5cb3ebcc (stale) Mirror 9: 0x5cb3ebcc (stale) Mirror 11: 0x5cb3ebcc (stale) Mirror 12: 0x5cb3ebcc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ce3: skipped 000595 1780327878.499024 trunc from 0x03443c to 0x036ce2 (0x28a7 bytes) 000596 1780327878.518441 trunc from 0x033350 to 0x036ce2 (0x3993 bytes) 000597 1780327878.549309 mapread 0x02a748 thru 0x031f67 (0x7820 bytes) 000598 1780327878.607097 mapread 0x0183a8 thru 0x023df1 (0xba4a bytes) 000599 1780327878.673258 write 0x02e7cb thru 0x0341a5 (0x59db bytes) 000600 1780327878.703019 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36ce3): Mirror 4: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ce3): Mirror 4: 0x5cb3ebcc (stale) Mirror 9: 0x5cb3ebcc (stale) Mirror 11: 0x5cb3ebcc (stale) Mirror 12: 0x5cb3ebcc (stale) CRC-32 checksum value for chunk [0, 0x36ce3): Mirror 1: 0xf7db01c7 Mirror 13: 0xf7db01c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ce3: skipped 000601 1780327881.148163 read 0x0213c0 thru 0x02e500 (0xd141 bytes) 000602 1780327881.203437 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x36ce3): Mirror 4: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ce3): Mirror 4: 0x5cb3ebcc (stale) Mirror 9: 0x5cb3ebcc (stale) Mirror 11: 0x5cb3ebcc (stale) Mirror 12: 0x5cb3ebcc (stale) CRC-32 checksum value for chunk [0, 0x36ce3): Mirror 1: 0xf7db01c7 Mirror 13: 0xf7db01c7 Mirror 14: 0xf7db01c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ce3: skipped 000603 1780327883.633284 mapread 0x035d4a thru 0x036ce2 (0xf99 bytes) 000604 1780327883.719372 trunc from 0x036ce3 to 0x01a8ab (0x1c437 bytes) 000605 1780327883.832213 fallocate 0x030f35 thru 0x034425 (0x34f1 bytes) 000606 1780327883.844258 mapwrite 0x00073a thru 0x00928f (0x8b56 bytes) 000608 1780327883.902398 mapread 0x01305b thru 0x01a8ab (0x7851 bytes) 000609 1780327883.918423 read 0x009537 thru 0x012b94 (0x965e bytes) 000610 1780327883.919585 write 0x02ae95 thru 0x03526e (0xa3da bytes) 000611 1780327883.935994 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3526f): Mirror 9: 0x515f9a8d (stale) Mirror 11: 0x515f9a8d (stale) Mirror 12: 0x515f9a8d (stale) Mirror 13: 0xb74c8c11 (stale) Mirror 14: 0xb74c8c11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3526f): Mirror 9: 0x515f9a8d (stale) Mirror 11: 0x515f9a8d (stale) Mirror 12: 0x515f9a8d (stale) Mirror 13: 0xb74c8c11 (stale) Mirror 14: 0xb74c8c11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3526f: skipped 000612 1780327884.773935 trunc from 0x03526f to 0x03abde (0x5970 bytes) 000614 1780327884.814541 read 0x025fd6 thru 0x026c63 (0xc8e bytes) 000615 1780327884.830326 write 0x03b2f3 thru 0x03ffff (0x4d0d bytes) 000616 1780327884.838364 mapread 0x03553b thru 0x03ffff (0xaac5 bytes) 000617 1780327884.930368 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x6376496b (stale) Mirror 11: 0x6376496b (stale) Mirror 12: 0x9f6d3f07 (stale) Mirror 13: 0x345ea0f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x6376496b (stale) Mirror 11: 0x6376496b (stale) Mirror 12: 0x9f6d3f07 (stale) Mirror 13: 0x345ea0f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000619 1780327885.830286 fallocate 0x035de0 thru 0x03ffff (0xa220 bytes) 000620 1780327885.837270 trunc from 0x040000 to 0x030200 (0xfdff bytes) 000621 1780327885.888774 write 0x03f870 thru 0x03ffff (0x790 bytes) 000622 1780327885.890601 read 0x012f9c thru 0x01d6ee (0xa753 bytes) 000624 1780327885.901579 trunc from 0x040000 to 0x02898e (0x17671 bytes) 000625 1780327885.934942 trunc from 0x02898f to 0x007f49 (0x20a45 bytes) 000626 1780327885.968860 write 0x039386 thru 0x03cc8e (0x3909 bytes) 000628 1780327885.972793 punch from 0x0251dd to 0x034a64 (0xf888 bytes) 000629 1780327885.977616 trunc from 0x0160ff to 0x025b3f (0xfa41 bytes) 000630 1780327885.983936 trunc from 0x03cc8f to 0x00baba (0x311d4 bytes) 000631 1780327886.023792 mapread 0x004b84 thru 0x00baba (0x6f37 bytes) 000632 1780327886.055498 trunc from 0x00babb to 0x00339c (0x871e bytes) 000633 1780327886.102369 read 0x0012ca thru 0x00339c (0x20d3 bytes) 000635 1780327886.118419 mapread 0x001fa7 thru 0x00339c (0x13f6 bytes) 000636 1780327886.132705 MIRROR_RESYNC 5 mirrors 000637 1780327887.743773 punch from 0x002a3e to 0x00339c (0x95f bytes) 000638 1780327887.778966 trunc from 0x00339d to 0x00a15f (0x6dc3 bytes) 000639 1780327887.810274 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xa160): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa160): Mirror 9: 0x446f2b76 (stale) Mirror 11: 0x446f2b76 (stale) Mirror 12: 0x446f2b76 (stale) Mirror 13: 0x446f2b76 (stale) CRC-32 checksum value for chunk [0, 0xa160): Mirror 1: 0xa4daba64 Mirror 14: 0xa4daba64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa160: skipped 000640 1780327888.832206 mapwrite 0x011e95 thru 0x0132a0 (0x140c bytes) 000641 1780327889.002385 punch from 0x000194 to 0x003fae (0x3e1b bytes) 000642 1780327889.014463 mapwrite 0x003715 thru 0x011b1b (0xe407 bytes) 000645 1780327889.119840 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x132a1): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x132a1): Mirror 9: 0xfd946208 (stale) Mirror 11: 0xfd946208 (stale) Mirror 12: 0xfd946208 (stale) Mirror 13: 0xfd946208 (stale) Mirror 14: 0x6821acae (stale) CRC-32 checksum value for chunk [0, 0x132a1): Mirror 1: 0x2023d692 Mirror 15: 0x2023d692 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x132a1: skipped 000646 1780327890.734995 trunc from 0x0132a1 to 0x008005 (0xb29b bytes) 000648 1780327890.924119 mapread 0x006ccc thru 0x008005 (0x133a bytes) 000649 1780327890.990917 mapread 0x004976 thru 0x008005 (0x3690 bytes) 000650 1780327890.996957 trunc from 0x00060b to 0x0047f2 (0x41e8 bytes) 000651 1780327891.007642 read 0x004a43 thru 0x007f84 (0x3542 bytes) 000652 1780327891.017898 trunc from 0x002d99 to 0x002df4 (0x5c bytes) 000653 1780327891.024362 write 0x00daf5 thru 0x015dd0 (0x82dc bytes) 000654 1780327891.027160 write 0x00dc4f thru 0x00ebf1 (0xfa3 bytes) 000655 1780327891.030123 write 0x0178fe thru 0x023d69 (0xc46c bytes) 000656 1780327891.035197 mapread 0x0180b6 thru 0x023d69 (0xbcb4 bytes) 000657 1780327891.037194 trunc from 0x005679 to 0x01178e (0xc116 bytes) 000658 1780327891.066424 trunc from 0x01e278 to 0x023d69 (0x5af2 bytes) 000659 1780327891.083941 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23d6a): Mirror 11: 0xc45211aa (stale) Mirror 12: 0xc45211aa (stale) Mirror 13: 0xc45211aa (stale) Mirror 14: 0x6d782d91 (stale) Mirror 15: 0x1eea44f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23d6a): Mirror 11: 0xc45211aa (stale) Mirror 12: 0xc45211aa (stale) Mirror 13: 0xc45211aa (stale) Mirror 14: 0x6d782d91 (stale) Mirror 15: 0x1eea44f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23d6a: skipped 000661 1780327891.853452 mapread 0x006ac0 thru 0x014f09 (0xe44a bytes) 000662 1780327891.909610 trunc from 0x023d6a to 0x01d442 (0x6927 bytes) 000663 1780327891.971114 mapread 0x002433 thru 0x00fed9 (0xdaa7 bytes) 000664 1780327891.981510 punch from 0x01b30f to 0x01d442 (0x2134 bytes) 000665 1780327891.999650 fallocate 0x02d68f thru 0x02df6f (0x8e1 bytes) 000666 1780327892.026796 MIRROR_RESYNC 6 mirrors 000667 1780327895.223872 trunc from 0x01d443 to 0x00dd43 (0xf6ff bytes) 000668 1780327895.273189 write 0x00d2d4 thru 0x01295f (0x568c bytes) 000669 1780327895.279627 trunc from 0x012960 to 0x01679e (0x3e3f bytes) 000670 1780327895.302521 trunc from 0x01679f to 0x03f836 (0x29098 bytes) 000671 1780327895.325570 trunc from 0x035732 to 0x03dcd6 (0x85a5 bytes) 000672 1780327895.332316 write 0x0296e4 thru 0x037762 (0xe07f bytes) 000673 1780327895.344754 trunc from 0x03f837 to 0x034b92 (0xaca4 bytes) 000674 1780327895.363462 read 0x031bcc thru 0x034b92 (0x2fc7 bytes) 000675 1780327895.370380 MIRROR_RESYNC 6 mirrors 000676 1780327897.759416 write 0x035bf0 thru 0x03ffff (0xa410 bytes) 000677 1780327897.823858 mapread 0x016451 thru 0x0208df (0xa48f bytes) 000678 1780327897.864282 punch from 0x029ab5 to 0x0320c4 (0x8610 bytes) 000679 1780327897.874801 trunc from 0x040000 to 0x03bbc1 (0x443e bytes) 000680 1780327897.906837 trunc from 0x03877f to 0x03b481 (0x2d03 bytes) 000681 1780327897.940827 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bbc2): Mirror 11: 0xa6d4adec (stale) Mirror 12: 0xa6d4adec (stale) Mirror 14: 0xa6d4adec (stale) Mirror 15: 0xa6d4adec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bbc2): Mirror 11: 0xa6d4adec (stale) Mirror 12: 0xa6d4adec (stale) Mirror 14: 0xa6d4adec (stale) Mirror 15: 0xa6d4adec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bbc2: skipped 000682 1780327898.447355 mapwrite 0x002cd0 thru 0x00a785 (0x7ab6 bytes) 000683 1780327898.495180 write 0x02c2a4 thru 0x0396b1 (0xd40e bytes) 000685 1780327898.521423 punch from 0x0372d7 to 0x03bbc1 (0x48eb bytes) 000686 1780327898.538490 punch from 0x02a4f7 to 0x02f0d1 (0x4bdb bytes) 000687 1780327898.544507 mapwrite 0x01b7e1 thru 0x026ee7 (0xb707 bytes) 000688 1780327898.595638 punch from 0x007ee8 to 0x008ee9 (0x1002 bytes) 000689 1780327898.615777 MIRROR_RESYNC 5 mirrors 000690 1780327900.247795 mapread 0x021ff8 thru 0x02bfb6 (0x9fbf bytes) 000691 1780327900.276945 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) 000692 1780327900.650320 trunc from 0x03bbc2 to 0x00df28 (0x2dc99 bytes) 000693 1780327900.704042 punch from 0x00603f to 0x009a83 (0x3a45 bytes) 000694 1780327900.710482 trunc from 0x00df29 to 0x02b6c0 (0x1d798 bytes) 000695 1780327900.735806 mapwrite 0x012817 thru 0x01b8df (0x90c9 bytes) 000696 1780327900.777717 read 0x014c87 thru 0x021229 (0xc5a3 bytes) 000697 1780327900.787235 trunc from 0x018848 to 0x01b20e (0x29c7 bytes) 000698 1780327900.793584 punch from 0x017111 to 0x01f6e8 (0x85d8 bytes) 000699 1780327900.804161 MIRROR_RESYNC 4 mirrors 000701 1780327903.586393 fallocate 0x01112d thru 0x020d81 (0xfc55 bytes) 000702 1780327903.639060 MIRROR_RESYNC 4 mirrors 000703 1780327905.792849 trunc from 0x02b6c1 to 0x03bf44 (0x10884 bytes) 000704 1780327905.878013 trunc from 0x03bf45 to 0x02edde (0xd166 bytes) 000705 1780327905.969723 trunc from 0x02eddf to 0x010048 (0x1ed96 bytes) 000706 1780327906.009028 trunc from 0x00cf2c to 0x010048 (0x311d bytes) 000707 1780327906.020258 read 0x0043ba thru 0x010048 (0xbc8f bytes) 000709 1780327906.062497 fallocate 0x020dde thru 0x021d9e (0xfc1 bytes) 000710 1780327906.075244 trunc from 0x010049 to 0x031eec (0x21ea4 bytes) 000711 1780327906.129216 punch from 0x003252 to 0x003b8f (0x93e bytes) 000712 1780327906.140539 fallocate 0x01cfec thru 0x01f71b (0x2730 bytes) 000713 1780327906.151651 read 0x012610 thru 0x020b2e (0xe51f bytes) 000714 1780327906.206892 trunc from 0x021832 to 0x02d904 (0xc0d3 bytes) 000715 1780327906.224451 punch from 0x02894e to 0x02a36a (0x1a1d bytes) 000716 1780327906.234243 mapwrite 0x00b360 thru 0x012ed5 (0x7b76 bytes) 000718 1780327906.280650 trunc from 0x031eed to 0x031c03 (0x2e9 bytes) 000719 1780327906.320114 trunc from 0x031c04 to 0x002bc2 (0x2f041 bytes) 000720 1780327906.359427 mapwrite 0x023115 thru 0x023cde (0xbca bytes) 000721 1780327906.454211 fallocate 0x0035ee thru 0x0113e1 (0xddf4 bytes) 000722 1780327906.461575 punch from 0x004afc to 0x00de2c (0x9331 bytes) 000723 1780327906.487361 read 0x00ea7c thru 0x014dd3 (0x6358 bytes) 000724 1780327906.505069 trunc from 0x0108b4 to 0x012f46 (0x2693 bytes) 000726 1780327906.540711 trunc from 0x023cdf to 0x0314b6 (0xd7d8 bytes) 000728 1780327906.575093 punch from 0x028cf1 to 0x0314b6 (0x87c6 bytes) 000729 1780327906.596288 fallocate 0x02213e thru 0x02bf1d (0x9de0 bytes) 000730 1780327906.617199 mapwrite 0x016a94 thru 0x0241cd (0xd73a bytes) 000731 1780327906.720737 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x314b7): Mirror 14: 0xcfd6eb8f (stale) Mirror 15: 0xcfd6eb8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x314b7): Mirror 14: 0xcfd6eb8f (stale) Mirror 15: 0xcfd6eb8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x314b7: skipped 000732 1780327907.444427 read 0x0167e6 thru 0x01b674 (0x4e8f bytes) 000733 1780327907.474090 read 0x01cae4 thru 0x01e9d4 (0x1ef1 bytes) 000734 1780327907.508711 trunc from 0x0314b7 to 0x01f1f1 (0x122c5 bytes) 000735 1780327907.582202 MIRROR_RESYNC 3 mirrors 000736 1780327909.141856 trunc from 0x01f1f2 to 0x03e04a (0x1ee59 bytes) 000737 1780327909.266733 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e04b): Mirror 14: 0xfe375c25 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e04b): Mirror 14: 0xfe375c25 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e04b: skipped 000738 1780327909.930369 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) 000739 1780327910.155364 mapread 0x036385 thru 0x03e04a (0x7cc6 bytes) 000740 1780327910.198467 trunc from 0x030550 to 0x03a19b (0x9c4c bytes) 000741 1780327910.204127 trunc from 0x03d948 to 0x03e04a (0x703 bytes) 000742 1780327910.209001 punch from 0x0157b5 to 0x022f1d (0xd769 bytes) 000743 1780327910.214591 mapwrite 0x0197db thru 0x0236c8 (0x9eee bytes) 000745 1780327910.257562 write 0x0171c7 thru 0x0225d9 (0xb413 bytes) 000746 1780327910.272530 punch from 0x02fa05 to 0x039ea9 (0xa4a5 bytes) 000747 1780327910.281035 trunc from 0x007e30 to 0x011346 (0x9517 bytes) 000748 1780327910.289346 trunc from 0x01e240 to 0x0214ce (0x328f bytes) 000749 1780327910.326893 trunc from 0x02fa48 to 0x03c241 (0xc7fa bytes) 000751 1780327910.349438 fallocate 0x013925 thru 0x01f7e4 (0xbec0 bytes) 000752 1780327910.355919 read 0x0238b9 thru 0x02d5cc (0x9d14 bytes) 000753 1780327910.368575 trunc from 0x03e04b to 0x0098a4 (0x347a6 bytes) 000754 1780327910.399156 punch from 0x0088af to 0x0098a4 (0xff6 bytes) 000755 1780327910.405584 trunc from 0x0098a5 to 0x01ae49 (0x115a5 bytes) 000756 1780327910.426105 mapread 0x00494f thru 0x00833b (0x39ed bytes) 000757 1780327910.446172 write 0x03b0e7 thru 0x03ffff (0x4f19 bytes) 000758 1780327910.448746 fallocate 0x007927 thru 0x008854 (0xf2e bytes) 000759 1780327910.459388 trunc from 0x040000 to 0x02e41a (0x11be5 bytes) 000760 1780327910.509850 trunc from 0x0070ab to 0x0135a5 (0xc4fb bytes) 000761 1780327910.517835 trunc from 0x02e41b to 0x004024 (0x2a3f6 bytes) 000762 1780327910.551902 trunc from 0x002031 to 0x004024 (0x1ff4 bytes) 000764 1780327910.565453 read 0x002408 thru 0x00306b (0xc64 bytes) 000765 1780327910.584106 read 0x000191 thru 0x004024 (0x3e94 bytes) 000768 1780327910.586402 mapwrite 0x02ead6 thru 0x033c57 (0x5182 bytes) 000769 1780327910.651616 write 0x028ed0 thru 0x0302c4 (0x73f5 bytes) 000770 1780327910.655370 write 0x01f815 thru 0x025568 (0x5d54 bytes) 000771 1780327910.658631 trunc from 0x00f4dd to 0x018c4d (0x9771 bytes) 000772 1780327910.667056 mapwrite 0x02a6c5 thru 0x034d41 (0xa67d bytes) 000773 1780327910.733025 write 0x02036e thru 0x022ba3 (0x2836 bytes) 000774 1780327910.734309 trunc from 0x02d7a5 to 0x0345db (0x6e37 bytes) 000775 1780327910.752320 mapwrite 0x02c514 thru 0x02fbc8 (0x36b5 bytes) 000776 1780327910.814379 read 0x022e07 thru 0x023239 (0x433 bytes) 000777 1780327910.815562 mapread 0x034777 thru 0x034d41 (0x5cb bytes) 000778 1780327910.837585 write 0x02cc4e thru 0x033ca1 (0x7054 bytes) 000779 1780327910.842556 read 0x0270ae thru 0x03049c (0x93ef bytes) 000780 1780327910.853390 MIRROR_EXTEND to 2 mirrors 000781 1780327912.260342 mapwrite 0x011d7c thru 0x014637 (0x28bc bytes) 000782 1780327912.547538 read 0x0278c9 thru 0x030706 (0x8e3e bytes) 000783 1780327912.605489 MIRROR_RESYNC 2 mirrors 000784 1780327914.556253 mapwrite 0x021827 thru 0x02f1af (0xd989 bytes) 000785 1780327914.820427 fallocate 0x03c1e0 thru 0x03f14b (0x2f6c bytes) 000786 1780327914.839257 fallocate 0x002008 thru 0x00fada (0xdad3 bytes) 000787 1780327914.854393 trunc from 0x030770 to 0x039908 (0x9199 bytes) 000788 1780327914.874374 mapread 0x01862b thru 0x022653 (0xa029 bytes) 000789 1780327914.906329 read 0x016277 thru 0x023632 (0xd3bc bytes) 000790 1780327914.907379 mapread 0x006422 thru 0x0085a2 (0x2181 bytes) 000791 1780327914.941594 read 0x0203c2 thru 0x020f64 (0xba3 bytes) 000792 1780327914.942349 trunc from 0x03f14c to 0x03ca26 (0x2725 bytes) 000793 1780327915.004304 trunc from 0x03ca27 to 0x01f12e (0x1d8f8 bytes) 000794 1780327915.068574 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) 000795 1780327915.410333 trunc from 0x01a3b9 to 0x01bef8 (0x1b40 bytes) 000796 1780327915.423137 punch from 0x0175b0 to 0x0189ad (0x13fe bytes) 000797 1780327915.434156 read 0x01b763 thru 0x01de92 (0x2730 bytes) 000798 1780327915.456171 trunc from 0x01f12f to 0x005256 (0x19ed8 bytes) 000800 1780327915.522180 punch from 0x003186 to 0x005256 (0x20d1 bytes) 000801 1780327915.547923 trunc from 0x000e54 to 0x005256 (0x4403 bytes) 000802 1780327915.556924 trunc from 0x003c2f to 0x005256 (0x1628 bytes) 000804 1780327915.571300 mapwrite 0x03e411 thru 0x03ffff (0x1bef bytes) 000805 1780327915.652559 trunc from 0x02e494 to 0x035ef1 (0x7a5e bytes) 000806 1780327915.664894 trunc from 0x040000 to 0x006a5e (0x395a1 bytes) 000807 1780327915.688027 mapwrite 0x00d1ca thru 0x00f997 (0x27ce bytes) 000809 1780327915.768074 fallocate 0x022854 thru 0x0306f1 (0xde9e bytes) 000810 1780327915.790338 trunc from 0x0306f2 to 0x031e06 (0x1715 bytes) 000811 1780327915.827893 trunc from 0x031e07 to 0x024d5c (0xd0aa bytes) 000812 1780327915.856649 punch from 0x01397f to 0x02159c (0xdc1e bytes) 000813 1780327915.864344 fallocate 0x025a41 thru 0x028461 (0x2a21 bytes) 000814 1780327915.871712 trunc from 0x028462 to 0x012107 (0x1635a bytes) 000815 1780327915.909501 trunc from 0x012108 to 0x03289b (0x20794 bytes) 000816 1780327915.961327 mapwrite 0x008694 thru 0x00c2ee (0x3c5b bytes) 000817 1780327916.022963 trunc from 0x03289c to 0x033c7e (0x13e3 bytes) 000818 1780327916.048507 punch from 0x032043 to 0x033c7e (0x1c3c bytes) 000819 1780327916.054252 read 0x012885 thru 0x019cf9 (0x7475 bytes) 000820 1780327916.068337 mapwrite 0x015cfe thru 0x02448f (0xe792 bytes) 000821 1780327916.104343 trunc from 0x003c51 to 0x013a34 (0xfde4 bytes) 000822 1780327916.112228 mapread 0x01f0da thru 0x02c4f6 (0xd41d bytes) 000823 1780327916.115399 trunc from 0x033c7f to 0x01bf80 (0x17cfe bytes) 000824 1780327916.143005 write 0x028ec8 thru 0x037635 (0xe76e bytes) 000825 1780327916.153242 MIRROR_EXTEND to 2 mirrors 000826 1780327917.727488 read 0x024b6f thru 0x02afe4 (0x6476 bytes) 000827 1780327917.745413 mapread 0x031108 thru 0x0332ca (0x21c3 bytes) 000828 1780327917.787263 punch from 0x02dec6 to 0x037635 (0x9770 bytes) 000829 1780327917.868090 punch from 0x0029ea to 0x009918 (0x6f2f bytes) 000830 1780327917.882616 mapread 0x010e01 thru 0x020775 (0xf975 bytes) 000831 1780327917.933566 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37636): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37636): Mirror 2: 0xf96bf644 (stale) CRC-32 checksum value for chunk [0, 0x37636): Mirror 1: 0x522df87b Mirror 3: 0x522df87b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37636: skipped 000832 1780327919.561887 write 0x03f7c3 thru 0x03ffff (0x83d bytes) 000833 1780327919.608619 read 0x01c3e4 thru 0x02a3ad (0xdfca bytes) 000834 1780327919.619568 mapread 0x01949c thru 0x026436 (0xcf9b bytes) 000835 1780327919.638170 write 0x024660 thru 0x02e5dd (0x9f7e bytes) 000836 1780327919.645538 trunc from 0x040000 to 0x0359dd (0xa622 bytes) 000837 1780327919.671406 fallocate 0x010b0b thru 0x01c821 (0xbd17 bytes) 000838 1780327919.678907 write 0x011296 thru 0x01aab7 (0x9822 bytes) 000839 1780327919.696244 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x359de): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x359de): Mirror 2: 0xa8405328 (stale) Mirror 3: 0xa174ad70 (stale) CRC-32 checksum value for chunk [0, 0x359de): Mirror 1: 0x9a67bb58 Mirror 4: 0x9a67bb58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x359de: skipped 000840 1780327921.290307 mapwrite 0x01d801 thru 0x01ea27 (0x1227 bytes) 000841 1780327921.468493 trunc from 0x0359de to 0x038cfe (0x3321 bytes) 000842 1780327921.535271 fallocate 0x01f511 thru 0x02341f (0x3f0f bytes) 000843 1780327921.543995 write 0x02be57 thru 0x038825 (0xc9cf bytes) 000844 1780327921.586797 write 0x03575f thru 0x03c743 (0x6fe5 bytes) 000845 1780327921.588628 trunc from 0x03c744 to 0x01c781 (0x1ffc2 bytes) 000847 1780327921.663397 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c782): Mirror 2: 0x4902dd6d (stale) Mirror 4: 0xa23a836e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c782): Mirror 2: 0x4902dd6d (stale) Mirror 4: 0xa23a836e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c782: skipped 000849 1780327922.202243 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1c782): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c782): Mirror 2: 0x4902dd6d (stale) Mirror 4: 0xa23a836e (stale) CRC-32 checksum value for chunk [0, 0x1c782): Mirror 1: 0xa23a836e Mirror 5: 0xa23a836e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c782: skipped 000850 1780327923.507256 mapwrite 0x02a422 thru 0x0356bd (0xb29c bytes) 000851 1780327923.602054 trunc from 0x018715 to 0x01b146 (0x2a32 bytes) 000852 1780327923.642034 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x356be): Mirror 2: 0xc2f1c365 (stale) Mirror 5: 0xccc3dc43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x356be): Mirror 2: 0xc2f1c365 (stale) Mirror 5: 0xccc3dc43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x356be: skipped 000853 1780327924.134617 write 0x02156f thru 0x028616 (0x70a8 bytes) 000854 1780327924.153635 mapwrite 0x03a91c thru 0x03ffff (0x56e4 bytes) 000855 1780327924.236056 trunc from 0x015010 to 0x01dce5 (0x8cd6 bytes) 000856 1780327924.246045 trunc from 0x040000 to 0x023642 (0x1c9bd bytes) 000858 1780327924.274047 trunc from 0x023643 to 0x03999e (0x1635c bytes) 000859 1780327924.296046 punch from 0x01884a to 0x0260d8 (0xd88f bytes) 000860 1780327924.303054 write 0x03cac2 thru 0x03ffff (0x353e bytes) 000861 1780327924.307522 trunc from 0x00d791 to 0x00d84f (0xbf bytes) 000862 1780327924.318623 punch from 0x0325c6 to 0x03ffff (0xda3a bytes) 000863 1780327924.349172 read 0x01cdce thru 0x01f114 (0x2347 bytes) 000864 1780327924.360940 write 0x03426a thru 0x037aff (0x3896 bytes) 000865 1780327924.386256 trunc from 0x040000 to 0x03ee73 (0x118c bytes) 000866 1780327924.415295 mapread 0x039f27 thru 0x03ee73 (0x4f4d bytes) 000867 1780327924.430052 mapread 0x0317be thru 0x035a99 (0x42dc bytes) 000868 1780327924.433933 trunc from 0x03ee74 to 0x000e69 (0x3e00a bytes) 000869 1780327924.456457 write 0x0312d8 thru 0x035a8e (0x47b7 bytes) 000870 1780327924.459789 mapread 0x010a6e thru 0x01ebde (0xe171 bytes) 000872 1780327924.492135 read 0x0059b2 thru 0x0144af (0xeafe bytes) 000873 1780327924.510631 MIRROR_RESYNC 3 mirrors 000874 1780327925.067576 read 0x015c79 thru 0x01c659 (0x69e1 bytes) 000875 1780327925.075628 mapread 0x0138dc thru 0x015cc3 (0x23e8 bytes) 000876 1780327925.085263 write 0x03026e thru 0x037ae0 (0x7873 bytes) 000877 1780327925.137418 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ae1): Mirror 5: 0xbf61d3fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ae1): Mirror 5: 0xbf61d3fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ae1: skipped 000878 1780327925.433546 fallocate 0x015280 thru 0x019048 (0x3dc9 bytes) 000880 1780327925.437489 read 0x01cdfc thru 0x021542 (0x4747 bytes) 000881 1780327925.445230 mapread 0x009e4d thru 0x016f5e (0xd112 bytes) 000882 1780327925.471306 trunc from 0x032fe3 to 0x036bca (0x3be8 bytes) 000883 1780327925.480348 trunc from 0x037ae1 to 0x013586 (0x2455a bytes) 000884 1780327925.530196 mapwrite 0x034ddd thru 0x03752f (0x2753 bytes) 000885 1780327925.618738 punch from 0x00074f to 0x009b8d (0x943f bytes) 000886 1780327925.633836 mapread 0x0225bc thru 0x02cda4 (0xa7e9 bytes) 000887 1780327925.664110 trunc from 0x010815 to 0x010910 (0xfc bytes) 000888 1780327925.674460 read 0x020f53 thru 0x02f9d9 (0xea87 bytes) 000889 1780327925.675577 mapread 0x02d9ad thru 0x03752f (0x9b83 bytes) 000890 1780327925.679333 punch from 0x02bea1 to 0x02dd93 (0x1ef3 bytes) 000891 1780327925.688263 punch from 0x0025e7 to 0x00f018 (0xca32 bytes) 000892 1780327925.693924 punch from 0x03375c to 0x03752f (0x3dd4 bytes) 000893 1780327925.700356 mapread 0x01509e thru 0x016f4b (0x1eae bytes) 000894 1780327925.712849 punch from 0x02cbbb to 0x03752f (0xa975 bytes) 000895 1780327925.718754 read 0x02e880 thru 0x0359e0 (0x7161 bytes) 000896 1780327925.728370 read 0x033729 thru 0x03752f (0x3e07 bytes) 000897 1780327925.729385 fallocate 0x00f330 thru 0x01c21b (0xceec bytes) 000898 1780327925.735766 punch from 0x00639d to 0x010736 (0xa39a bytes) 000899 1780327925.745957 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) 000900 1780327925.889604 trunc from 0x01f454 to 0x0274ef (0x809c bytes) 000901 1780327925.896621 fallocate 0x02a8e4 thru 0x02fd60 (0x547d bytes) 000902 1780327925.903125 write 0x0070f2 thru 0x00f85a (0x8769 bytes) 000903 1780327925.922249 write 0x0053c1 thru 0x012774 (0xd3b4 bytes) 000904 1780327925.946043 write 0x00b092 thru 0x00b2b5 (0x224 bytes) 000905 1780327925.948137 trunc from 0x005acd to 0x00a0d2 (0x4606 bytes) 000906 1780327925.964439 mapread 0x02ab31 thru 0x02de5f (0x332f bytes) 000907 1780327925.977452 mapwrite 0x00a428 thru 0x00eccd (0x48a6 bytes) 000908 1780327926.008642 trunc from 0x027ffa to 0x03272c (0xa733 bytes) 000909 1780327926.013657 mapwrite 0x021444 thru 0x02a9de (0x959b bytes) 000910 1780327926.087588 MIRROR_EXTEND to 2 mirrors 000911 1780327927.372534 fallocate 0x006533 thru 0x012d03 (0xc7d1 bytes) 000912 1780327927.448995 write 0x039e82 thru 0x03ffff (0x617e bytes) 000913 1780327927.463037 read 0x024e2a thru 0x03315b (0xe332 bytes) 000914 1780327927.498010 fallocate 0x00ac18 thru 0x00dac4 (0x2ead bytes) 000915 1780327927.506348 punch from 0x037199 to 0x03ffff (0x8e67 bytes) 000916 1780327927.528427 trunc from 0x0202a6 to 0x029698 (0x93f3 bytes) 000917 1780327927.534404 mapread 0x035b55 thru 0x03de2d (0x82d9 bytes) 000918 1780327927.551935 write 0x0259d0 thru 0x0261eb (0x81c bytes) 000919 1780327927.567193 mapwrite 0x020c3d thru 0x02228d (0x1651 bytes) 000920 1780327927.592192 trunc from 0x0273bd to 0x034f12 (0xdb56 bytes) 000921 1780327927.602908 read 0x01c112 thru 0x020f1d (0x4e0c bytes) 000922 1780327927.618515 write 0x020b39 thru 0x02c70b (0xbbd3 bytes) 000923 1780327927.625605 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2c7:0x0]: Invalid argument (22) 000924 1780327927.776810 trunc from 0x005156 to 0x011e82 (0xcd2d bytes) 000925 1780327927.785135 trunc from 0x040000 to 0x00f7d9 (0x30826 bytes) 000927 1780327927.816399 fallocate 0x018909 thru 0x01df93 (0x568b bytes) 000928 1780327927.822642 trunc from 0x00853e to 0x00ecd7 (0x679a bytes) 000929 1780327927.834417 trunc from 0x00d3e0 to 0x00f7d9 (0x23fa bytes) 000930 1780327927.839282 fallocate 0x037928 thru 0x03ffff (0x86d8 bytes) 000931 1780327927.844216 trunc from 0x00f7da to 0x0276e4 (0x17f0b bytes) 000933 1780327927.858010 punch from 0x016ed4 to 0x022469 (0xb596 bytes) 000934 1780327927.862334 mapread 0x0163dc thru 0x01989c (0x34c1 bytes) 000936 1780327927.889781 fallocate 0x017eb6 thru 0x01fa2a (0x7b75 bytes) 000937 1780327927.899139 read 0x005d63 thru 0x0069a5 (0xc43 bytes) 000938 1780327927.904571 read 0x019446 thru 0x023c5c (0xa817 bytes) 000939 1780327927.926736 write 0x02b188 thru 0x02f034 (0x3ead bytes) 000940 1780327927.930390 punch from 0x020429 to 0x02d43c (0xd014 bytes) 000941 1780327927.948314 write 0x021ffd thru 0x0241fd (0x2201 bytes) 000943 1780327927.976237 MIRROR_EXTEND to 2 mirrors 000944 1780327928.590189 mapwrite 0x029f87 thru 0x038ed9 (0xef53 bytes) 000946 1780327928.693342 write 0x032fe3 thru 0x03a773 (0x7791 bytes) 000947 1780327928.695497 mapread 0x02376b thru 0x0334ae (0xfd44 bytes) 000948 1780327928.703499 trunc from 0x03a774 to 0x038421 (0x2352 bytes) 000949 1780327928.716565 mapwrite 0x00acdb thru 0x015558 (0xa87e bytes) 000950 1780327928.749442 fallocate 0x03421f thru 0x03b5ad (0x738f bytes) 000951 1780327928.753610 fallocate 0x020fa4 thru 0x026d62 (0x5dbf bytes) 000952 1780327928.758190 trunc from 0x01e051 to 0x01ea75 (0xa25 bytes) 000953 1780327928.761889 read 0x028dca thru 0x03463c (0xb873 bytes) 000956 1780327928.764107 trunc from 0x01a25d to 0x01e5bc (0x4360 bytes) 000957 1780327928.771158 trunc from 0x038422 to 0x0211c6 (0x1725b bytes) 000958 1780327928.784391 mapwrite 0x018010 thru 0x027e0a (0xfdfb bytes) 000959 1780327928.839989 MIRROR_RESYNC 2 mirrors 000960 1780327929.680269 mapwrite 0x0302cd thru 0x03f711 (0xf445 bytes) 000961 1780327929.818307 trunc from 0x03f712 to 0x0205dc (0x1f135 bytes) 000962 1780327929.850223 read 0x018f45 thru 0x01d107 (0x41c3 bytes) 000963 1780327929.860157 punch from 0x00d13a to 0x01740d (0xa2d4 bytes) 000964 1780327929.866434 mapread 0x0077e9 thru 0x014c97 (0xd4af bytes) 000965 1780327929.895320 MIRROR_RESYNC 2 mirrors 000966 1780327930.637554 write 0x02631c thru 0x02e403 (0x80e8 bytes) 000967 1780327930.675825 fallocate 0x008e04 thru 0x0092ee (0x4eb bytes) 000968 1780327930.681143 mapwrite 0x003058 thru 0x00c2de (0x9287 bytes) 000969 1780327930.740490 write 0x0384ba thru 0x03f8fe (0x7445 bytes) 000970 1780327930.742089 trunc from 0x0138c1 to 0x0170ed (0x382d bytes) 000971 1780327930.758172 punch from 0x00c444 to 0x013377 (0x6f34 bytes) 000972 1780327930.771538 mapread 0x0072ac thru 0x010e0d (0x9b62 bytes) 000973 1780327930.791375 mapwrite 0x025084 thru 0x0326fa (0xd677 bytes) 000974 1780327930.856907 punch from 0x002ecf to 0x007849 (0x497b bytes) 000975 1780327930.862646 read 0x033e5a thru 0x03ad4a (0x6ef1 bytes) 000976 1780327930.863644 mapwrite 0x00fac2 thru 0x015c32 (0x6171 bytes) 000977 1780327930.890183 mapwrite 0x024ace thru 0x02c8c9 (0x7dfc bytes) 000978 1780327930.916122 fallocate 0x027609 thru 0x030674 (0x906c bytes) 000979 1780327930.922857 trunc from 0x03f8ff to 0x0298cf (0x1602f bytes) 000980 1780327930.975141 mapread 0x006217 thru 0x014cc4 (0xeaae bytes) 000981 1780327930.994124 write 0x0323fb thru 0x03bccd (0x98d3 bytes) 000982 1780327930.998749 punch from 0x0042ff to 0x00564d (0x134f bytes) 000983 1780327931.005079 mapwrite 0x03c4a3 thru 0x03ffff (0x3b5d bytes) 000984 1780327931.050866 mapread 0x02ccd2 thru 0x033729 (0x6a58 bytes) 000985 1780327931.058372 fallocate 0x01d00a thru 0x0266d3 (0x96ca bytes) 000986 1780327931.064661 write 0x029197 thru 0x02a256 (0x10c0 bytes) 000987 1780327931.071055 MIRROR_RESYNC 2 mirrors 000988 1780327931.715966 read 0x039d4c thru 0x03ffff (0x62b4 bytes) 000989 1780327931.739253 MIRROR_EXTEND to 3 mirrors 000990 1780327932.677869 mapwrite 0x033caf thru 0x03a574 (0x68c6 bytes) 000991 1780327932.796388 trunc from 0x040000 to 0x0254f7 (0x1ab08 bytes) 000992 1780327932.840216 fallocate 0x0097ab thru 0x011a97 (0x82ed bytes) 000993 1780327932.849244 trunc from 0x01c0c7 to 0x0254f7 (0x9431 bytes) 000994 1780327932.859826 write 0x0244d0 thru 0x025eb3 (0x19e4 bytes) 000995 1780327932.899316 MIRROR_RESYNC 3 mirrors 000996 1780327934.242075 trunc from 0x025eb4 to 0x003643 (0x22870 bytes) 000997 1780327934.330290 punch from 0x00266f to 0x003643 (0xfd5 bytes) 000998 1780327934.345853 read 0x0022f0 thru 0x003643 (0x1354 bytes) 001000 1780327934.355071 read 0x00229c thru 0x003643 (0x13a8 bytes) All operations completed A-OK! PASS 70a (133s) == sanity-flr test 71: check mirror extend parent fid ==== 11:32:20 (1780327940) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x310:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 11:32:31 (1780327951) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206922 s, 10.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195718 s, 5.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 11:32:41 (1780327961) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8213408 .. Extending file size to 2808000 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4110400 .. Extending file size to 8384800 .. Extending file size to 8520896 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8531456 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8551296 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9309952 .. Extending file size to 9188640 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61695 61696 61698 61701 61702 done 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) 192.168.202.130@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:34:03 (1780328043) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6423712 .. Extending file size to 7323712 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8774976 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7462112 .. done Extending file size to 8206080 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9341632 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8597472 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8781440 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 63171 63172 63173 63176 63178 done 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) 192.168.202.130@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:35:27 (1780328127) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2308 1268996 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2392 1268912 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26076 3580944 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17960 3589060 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44036 7170004 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2308 1268996 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2392 1268912 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 26076 3580944 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17960 3589060 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44036 7170004 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121897 s, 17.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.23554 s, 8.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xffb2d652 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:35:45 (1780328145) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103118 s, 20.3 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 11:35:53 (1780328153) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:36:00 (1780328160) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131442 s, 16.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:36:09 (1780328169) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.144137 s, 14.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:36:19 (1780328179) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17507 s, 12.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e1:0x0] } PASS 204c (39s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:36:59 (1780328219) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg230-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129081 s, 8.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3e3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3e3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3e4:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:37:58 (1780328278) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg230-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145442 s, 14.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (50s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:38:48 (1780328328) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg230-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg230-server: Pool lustre.test_204f created Adding targets to pool oleg230-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg230-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126716 s, 16.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg230-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg230-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg230-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:40:08 (1780328408) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:40:16 (1780328416) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:40:24 (1780328424) oleg230-server: Pool lustre.test_206 created Adding targets to pool oleg230-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg230-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg230-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg230-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg230-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:40:54 (1780328454) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127821 s, 8.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368397 s, 28.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xdf9c909f Mirror 3: 0xdf9c909f PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:41:06 (1780328466) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:41:10 (1780328470) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:41:13 (1780328473) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0846259 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x940000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xaa0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xaa0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xaa0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1f70000 Read this (4 bytes): BBBB 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) PASS 209a (45s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:41:58 (1780328518) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0297121 s, 276 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:42:08 (1780328528) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428068 s, 24.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611265 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 11:42:09.000000000 -0400 Modify: 2026-06-01 11:42:09.000000000 -0400 Change: 2026-06-01 11:42:11.000000000 -0400 Birth: 2026-06-01 11:42:10.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:42:17 (1780328537) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438842 s, 23.9 MB/s oleg230-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg230-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg230-server: O/280000401/d28/13c: File not found by ext2_lookup PASS 210b (27s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:42:45 (1780328565) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.328658 s, 125 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0451731 s, 90.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 1 11:42 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x86:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 1 11:42 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:42:54 (1780328574) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:43:03 (1780328583) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1571 sec =========== 11:43:13 (1780328593) === sanity-flr: start cleanup 11:43:15 (1780328595) === === sanity-flr: finish cleanup 11:43:26 (1780328606) ===