-----============= acceptance-small: sanity-flr ============----- Mon Sep 21 09:08:29 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg311-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 09:08:46 (1789996126) === oleg311-client.virtnet: executing check_config_client /mnt/lustre oleg311-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg311-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c0403050000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c0403050000.idle_timeout=debug disable quota as required oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 09:09:29 (1789996169) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8230] == sanity-flr test 0a: lfs mirror create with -N option == 09:09:32 (1789996172) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:09:51 (1789996191) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:09:55 (1789996195) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:09:58 (1789996198) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:10:23 (1789996223) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:10:26 (1789996226) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 09:10:30 (1789996230) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8c0403050000.stats=clear osc.lustre-OST0001-osc-ffff8c0403050000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:10:37 (1789996237) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:10:48 (1789996248) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:10:56 (1789996256) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:10:59 (1789996259) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:11:07 (1789996267) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 09:11:17 (1789996277) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:11:25 (1789996285) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:11:33 (1789996293) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:11:41 (1789996301) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:11:51 (1789996311) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:11:58 (1789996318) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.08608 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137539 s, 7.6 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:12:07 (1789996327) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0758864 s, 55.3 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:12:15 (1789996335) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:12:25 (1789996345) Stopping /mnt/lustre-ost1 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c0403050000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c0403050000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg311-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c0403050000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c0403050000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg311-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (73s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:13:38 (1789996418) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.596329 s, 17.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c0403050000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c0403050000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg311-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:14:20 (1789996460) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c0403050000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c0403050000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg311-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c0403050000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c0403050000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg311-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:15:43 (1789996543) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.86857 s, 18.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 21 09:15 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.057s user 0m0.000s sys 0m0.198s Stopping /mnt/lustre-ost1 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c0403050000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c0403050000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.980s user 0m0.001s sys 0m0.197s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg311-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c0403050000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c0403050000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.366s user 0m0.000s sys 0m0.162s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg311-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg311-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c0403050000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c0403050000.ost_server_uuid in FULL state after 0 sec PASS 33b (95s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:17:19 (1789996639) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:17:22 (1789996642) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:17:26 (1789996646) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 09:17:29 (1789996649) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 09:17:36 (1789996656) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.679356 s, 18.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:17:49 (1789996669) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262624 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241249 s, 43.5 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:17:58 (1789996678) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:18:07 (1789996687) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00423484 s, 242 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000931987 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101513 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00108968 s, 940 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 09:18:17 (1789996697) create test files with size 15752050 .. d5e4463cfee3af787d5cc9a6a1917e69 - 818f0cfdd4490d0bf61dea39d8e8e08f - 8f1906b18732e10fa541e180ab71ecf5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg311-client.virtnet /mnt/lustre (opts:) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 09:18:44 (1789996724) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:19:00 (1789996740) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:19:08 (1789996748) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114196 s, 9.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287066 s, 36.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 09:19:18 (1789996758) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122451 s, 17.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.511969 s, 8.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.219353 s, 9.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 09:19:35 (1789996775) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 09:19:39 (1789996779) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:19:42 (1789996782) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218706 s, 47.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 09:19:50 (1789996790) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.417526 s, 7.5 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:20:01 (1789996801) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:20:10 (1789996810) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.397237 s, 26.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg311-client.virtnet /mnt/lustre (opts:) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:20:21 (1789996821) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8c041163a000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.284245 s, 36.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg311-client.virtnet /mnt/lustre (opts:) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8c041133b000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 09:20:34 (1789996834) error lfs split: create victim file failed: File exists PASS 44f (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:20:52 (1789996852) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (49s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:21:41 (1789996901) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:22:19 (1789996939) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 09:22:22 (1789996942) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.233118 s, 13.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.174567 s, 18.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xcb8e483e (nosync,stale) Mirror 2: 0xcb8e483e (nosync,stale) ** verify mirror 2 doesn't change original checksum: ab964077903dc692fd56505b4f0be05e - mirror 1 checksum: 20505b44cc56f726e4d1d8dc4c6028ca - mirror 2 checksum: ab964077903dc692fd56505b4f0be05e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-21T09:22:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xfe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-21T09:22:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xff:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: ab964077903dc692fd56505b4f0be05e - mirror 1 checksum: 20505b44cc56f726e4d1d8dc4c6028ca - mirror 2 checksum: 20505b44cc56f726e4d1d8dc4c6028ca - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-21T09:22:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xfe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-21T09:22:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xff:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:22:33 (1789996953) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608952 s, 17.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:22:40 (1789996960) Failing mds1 on oleg311-server Stopping /mnt/lustre-mds1 (opts:) on oleg311-server 09:22:46 (1789996966) shut down facet: mds1 facet_host: oleg311-server facet_failover_host: oleg311-server Failover mds1 to oleg311-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0000 09:23:17 (1789996997) targets are mounted 09:23:17 (1789996997) facet_failover done oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x8:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319835 s, 32.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000405 lmm_object_id: 0x8 lmm_fid: [0x240000405:0x8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e:0x0] } PASS 50A (52s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:23:32 (1789997012) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0405027 s, 506 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0142116 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:23:42 (1789997022) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0515194 s, 398 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00860406 s, 2.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:23:51 (1789997031) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0078969 s, 2.1 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:23:59 (1789997039) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0602561 s, 340 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:24:06 (1789997046) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00350259 s, 1.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0430355 s, 95.2 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:24:13 (1789997053) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:24:34 (1789997074) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:25:01 (1789997101) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:25:23 (1789997123) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:25:45 (1789997145) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 09:25:52 (1789997152) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 58248 mirror split pid 58249 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 58248 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 58249 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 09:26:58 (1789997218) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 7220 skipping zero size read fallocating to largest ever: 0x99fb 000002 1789997220.351423 fallocate 0x002c03 thru 0x0099fa (0x6df8 bytes) fallocating to largest ever: 0x9b83 000003 1789997220.360312 fallocate 0x0075af thru 0x009b82 (0x25d4 bytes) 000004 1789997220.364698 write 0x016268 thru 0x01bad2 (0x586b bytes) 000005 1789997220.365536 punch from 0x00c287 to 0x00ea38 (0x27b2 bytes) 000007 1789997220.370435 trunc from 0x00501a to 0x00b3d8 (0x63bf bytes) 000008 1789997220.375851 mapwrite 0x02d00f thru 0x038505 (0xb4f7 bytes) fallocating to largest ever: 0x32ae3 000009 1789997220.436792 fallocate 0x02bf28 thru 0x032ae2 (0x6bbb bytes) 000010 1789997220.441066 fallocate 0x02753e thru 0x027f48 (0xa0b bytes) fallocating to largest ever: 0x36acc 000011 1789997220.450153 fallocate 0x030f3e thru 0x036acb (0x5b8e bytes) 000012 1789997220.454817 read 0x010c0c thru 0x01fffb (0xf3f0 bytes) 000013 1789997220.472349 write 0x0282c4 thru 0x02ad10 (0x2a4d bytes) 000014 1789997220.474048 mapread 0x01abbc thru 0x022ac0 (0x7f05 bytes) 000015 1789997220.487994 trunc from 0x038506 to 0x028639 (0xfecc bytes) 000016 1789997220.512198 punch from 0x01c13c to 0x028639 (0xc4fe bytes) 000017 1789997220.524747 mapread 0x00b977 thru 0x00fae0 (0x416a bytes) 000018 1789997220.540561 write 0x028aba thru 0x02fc7f (0x71c6 bytes) 000019 1789997220.551967 mapread 0x01ea4f thru 0x02e877 (0xfe29 bytes) 000021 1789997220.582701 write 0x0071ce thru 0x01704d (0xfe80 bytes) 000022 1789997220.589174 MIRROR_EXTEND to 2 mirrors 000023 1789997221.212726 write 0x004d6b thru 0x00fde4 (0xb07a bytes) 000024 1789997221.266524 write 0x0322fb thru 0x0368c1 (0x45c7 bytes) 000025 1789997221.267936 mapwrite 0x01ab36 thru 0x02999e (0xee69 bytes) 000027 1789997221.319948 mapwrite 0x02e3a0 thru 0x03303d (0x4c9e bytes) 000028 1789997221.343729 write 0x01f085 thru 0x028b9d (0x9b19 bytes) 000029 1789997221.345219 trunc from 0x0368c2 to 0x00892e (0x2df93 bytes) 000030 1789997221.379985 mapwrite 0x008304 thru 0x00876d (0x46a bytes) 000031 1789997221.397313 trunc from 0x002aa2 to 0x00892e (0x5e8d bytes) 000032 1789997221.402246 mapread 0x007544 thru 0x00892e (0x13eb bytes) 000033 1789997221.408931 trunc from 0x00892f to 0x01e79d (0x15e6f bytes) 000034 1789997221.422968 punch from 0x01e4bc to 0x01e79d (0x2e2 bytes) fallocating to largest ever: 0x3b38e 000035 1789997221.431547 fallocate 0x02ccbf thru 0x03b38d (0xe6cf bytes) 000036 1789997221.436671 mapwrite 0x02b0de thru 0x02c626 (0x1549 bytes) 000037 1789997221.468298 punch from 0x013575 to 0x01a50f (0x6f9b bytes) 000038 1789997221.473550 mapread 0x0059d1 thru 0x00cc64 (0x7294 bytes) 000039 1789997221.485675 trunc from 0x02c627 to 0x02e8e4 (0x22be bytes) 000040 1789997221.499352 punch from 0x029a21 to 0x02e8e4 (0x4ec4 bytes) 000042 1789997221.504643 mapwrite 0x03cc4c thru 0x03d722 (0xad7 bytes) 000043 1789997221.540640 punch from 0x00b7be to 0x018c3d (0xd480 bytes) 000046 1789997221.546257 trunc from 0x013e89 to 0x01e587 (0xa6ff bytes) 000047 1789997221.550358 mapread 0x00fc43 thru 0x01aa72 (0xae30 bytes) 000049 1789997221.562005 trunc from 0x0006da to 0x007e62 (0x7789 bytes) 000050 1789997221.576899 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) 000051 1789997221.720147 fallocate 0x00788d thru 0x00b59f (0x3d13 bytes) 000053 1789997221.728018 mapread 0x02e518 thru 0x035cff (0x77e8 bytes) 000054 1789997221.754961 trunc from 0x016676 to 0x01c3c9 (0x5d54 bytes) 000055 1789997221.761612 trunc from 0x0228ec to 0x02671e (0x3e33 bytes) 000056 1789997221.766966 write 0x03addd thru 0x03f37a (0x459e bytes) 000057 1789997221.773475 mapread 0x0158d9 thru 0x021fc6 (0xc6ee bytes) 000058 1789997221.784529 punch from 0x000482 to 0x0066c2 (0x6241 bytes) 000059 1789997221.788958 trunc from 0x038960 to 0x03f37a (0x6a1b bytes) 000061 1789997221.807060 MIRROR_EXTEND to 2 mirrors 000062 1789997222.467143 mapwrite 0x0038d8 thru 0x004fa7 (0x16d0 bytes) 000063 1789997222.543245 mapread 0x03c44c thru 0x03f37a (0x2f2f bytes) 000064 1789997222.552990 mapwrite 0x014930 thru 0x020bdf (0xc2b0 bytes) 000065 1789997222.585079 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) 000066 1789997222.676370 write 0x017332 thru 0x01ea57 (0x7726 bytes) 000067 1789997222.704721 MIRROR_EXTEND to 2 mirrors 000069 1789997223.364806 mapread 0x02f892 thru 0x031d59 (0x24c8 bytes) 000070 1789997223.381972 mapread 0x018ce7 thru 0x024d25 (0xc03f bytes) 000071 1789997223.399656 write 0x039dbb thru 0x03ffff (0x6245 bytes) 000072 1789997223.437240 write 0x02f87e thru 0x0365c4 (0x6d47 bytes) 000073 1789997223.450578 trunc from 0x001886 to 0x00b54c (0x9cc7 bytes) 000074 1789997223.456604 fallocate 0x0312d2 thru 0x038529 (0x7258 bytes) 000075 1789997223.461105 trunc from 0x0020b7 to 0x002ae5 (0xa2f bytes) 000076 1789997223.465818 punch from 0x00cc7d to 0x00f521 (0x28a5 bytes) 000077 1789997223.480502 punch from 0x0165a6 to 0x019cd8 (0x3733 bytes) 000078 1789997223.488545 read 0x035537 thru 0x037925 (0x23ef bytes) 000079 1789997223.492825 fallocate 0x00f121 thru 0x017ccc (0x8bac bytes) 000080 1789997223.498537 mapread 0x00fd70 thru 0x013513 (0x37a4 bytes) 000081 1789997223.518625 mapwrite 0x02f222 thru 0x03cac1 (0xd8a0 bytes) 000082 1789997223.541363 punch from 0x01c5e7 to 0x028835 (0xc24f bytes) 000083 1789997223.546440 trunc from 0x035528 to 0x03e117 (0x8bf0 bytes) 000084 1789997223.552645 write 0x0195ae thru 0x02905c (0xfaaf bytes) 000085 1789997223.576825 mapwrite 0x031a9b thru 0x03f3b0 (0xd916 bytes) 000086 1789997223.603721 trunc from 0x040000 to 0x01437f (0x2bc80 bytes) 000087 1789997223.621696 mapwrite 0x018e21 thru 0x02674d (0xd92d bytes) 000088 1789997223.684217 trunc from 0x02674e to 0x00204b (0x24702 bytes) 000089 1789997223.707534 mapwrite 0x03dd6f thru 0x03ffff (0x2291 bytes) 000090 1789997223.761795 punch from 0x02feac to 0x033e8c (0x3fe1 bytes) 000091 1789997223.773591 read 0x032719 thru 0x037194 (0x4a7c bytes) 000092 1789997223.784237 write 0x02218b thru 0x0257a2 (0x3618 bytes) 000093 1789997223.785293 punch from 0x02076b to 0x0254ef (0x4d85 bytes) 000094 1789997223.801030 write 0x019ee6 thru 0x025a36 (0xbb51 bytes) 000095 1789997223.809566 trunc from 0x012301 to 0x0162c2 (0x3fc2 bytes) 000096 1789997223.814765 trunc from 0x040000 to 0x03303d (0xcfc2 bytes) 000097 1789997223.842082 fallocate 0x00c36e thru 0x01715a (0xaded bytes) 000098 1789997223.849688 mapwrite 0x00d8af thru 0x013636 (0x5d88 bytes) 000099 1789997223.884297 trunc from 0x023bb1 to 0x02ea0d (0xae5d bytes) 000100 1789997223.889668 read 0x019504 thru 0x028703 (0xf200 bytes) 000101 1789997223.903225 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) 000102 1789997224.083860 fallocate 0x022c58 thru 0x02915f (0x6508 bytes) 000103 1789997224.089245 read 0x00d4b1 thru 0x00e04a (0xb9a bytes) 000104 1789997224.095678 fallocate 0x01c02c thru 0x02844d (0xc422 bytes) 000105 1789997224.101882 trunc from 0x03303e to 0x036f7b (0x3f3e bytes) 000106 1789997224.128956 punch from 0x012c44 to 0x01dd22 (0xb0df bytes) 000107 1789997224.135582 mapread 0x033aab thru 0x036f7b (0x34d1 bytes) 000109 1789997224.164875 fallocate 0x028c70 thru 0x035f4c (0xd2dd bytes) 000110 1789997224.170284 fallocate 0x01d034 thru 0x025eb7 (0x8e84 bytes) 000111 1789997224.178963 punch from 0x02f067 to 0x0312c6 (0x2260 bytes) 000112 1789997224.187693 trunc from 0x036f7c to 0x00d373 (0x29c08 bytes) 000113 1789997224.206873 mapwrite 0x017ab0 thru 0x01cab2 (0x5003 bytes) 000114 1789997224.250936 mapread 0x00490a thru 0x00b023 (0x671a bytes) 000115 1789997224.283744 mapread 0x019e0d thru 0x01cab2 (0x2ca6 bytes) 000117 1789997224.289438 mapread 0x00cec3 thru 0x016893 (0x99d1 bytes) 000119 1789997224.296758 mapread 0x018a85 thru 0x01cab2 (0x402e bytes) 000120 1789997224.297774 trunc from 0x01cab3 to 0x037277 (0x1a7c5 bytes) 000121 1789997224.321332 mapwrite 0x02f4b0 thru 0x030cd5 (0x1826 bytes) 000122 1789997224.358164 fallocate 0x035813 thru 0x0381b4 (0x29a2 bytes) 000123 1789997224.365087 fallocate 0x01e0d8 thru 0x022961 (0x488a bytes) 000124 1789997224.378717 read 0x003832 thru 0x00f057 (0xb826 bytes) 000125 1789997224.402327 read 0x01f287 thru 0x02c272 (0xcfec bytes) 000126 1789997224.422329 mapread 0x01687f thru 0x0190a9 (0x282b bytes) 000127 1789997224.423850 fallocate 0x00c2c9 thru 0x016034 (0x9d6c bytes) 000128 1789997224.437896 trunc from 0x037278 to 0x038812 (0x159b bytes) 000129 1789997224.483929 MIRROR_EXTEND to 2 mirrors 000130 1789997225.446787 MIRROR_EXTEND to 3 mirrors 000132 1789997226.340346 MIRROR_EXTEND to 4 mirrors 000133 1789997227.143762 write 0x03f31f thru 0x03ffff (0xce1 bytes) 000134 1789997227.183260 trunc from 0x020a07 to 0x024a16 (0x4010 bytes) 000135 1789997227.189351 read 0x005cf6 thru 0x00b7d7 (0x5ae2 bytes) 000136 1789997227.202077 trunc from 0x040000 to 0x010355 (0x2fcaa bytes) 000137 1789997227.237322 write 0x025d10 thru 0x02b106 (0x53f7 bytes) 000138 1789997227.241097 mapwrite 0x014a2f thru 0x01c229 (0x77fb bytes) 000139 1789997227.348886 read 0x00b178 thru 0x00f104 (0x3f8d bytes) 000140 1789997227.358899 trunc from 0x00eab9 to 0x00f4d5 (0xa1d bytes) 000141 1789997227.366798 trunc from 0x016e79 to 0x018854 (0x19dc bytes) 000142 1789997227.372008 mapread 0x017049 thru 0x01bad3 (0x4a8b bytes) 000143 1789997227.384923 mapread 0x012ded thru 0x01f4d9 (0xc6ed bytes) 000144 1789997227.397572 punch from 0x01dce9 to 0x0295e0 (0xb8f8 bytes) 000145 1789997227.402525 write 0x0127d6 thru 0x01f152 (0xc97d bytes) 000146 1789997227.408867 fallocate 0x015664 thru 0x0220a0 (0xca3d bytes) 000147 1789997227.412766 mapread 0x016049 thru 0x01e86e (0x8826 bytes) 000149 1789997227.414135 trunc from 0x00bf47 to 0x00d290 (0x134a bytes) 000150 1789997227.430317 mapwrite 0x009600 thru 0x01239a (0x8d9b bytes) 000151 1789997227.538040 fallocate 0x0074ec thru 0x00b1d3 (0x3ce8 bytes) 000152 1789997227.542619 mapwrite 0x03b044 thru 0x03ffff (0x4fbc bytes) 000153 1789997227.585823 mapwrite 0x026837 thru 0x0295a9 (0x2d73 bytes) 000154 1789997227.628141 mapread 0x033673 thru 0x03cf70 (0x98fe bytes) 000155 1789997227.629263 trunc from 0x040000 to 0x012168 (0x2de97 bytes) 000156 1789997227.642743 trunc from 0x012169 to 0x03b29a (0x29132 bytes) 000157 1789997227.658581 trunc from 0x03b29b to 0x00906c (0x3222e bytes) 000158 1789997227.686593 mapwrite 0x03f32c thru 0x03ffff (0xcd4 bytes) 000159 1789997227.724971 read 0x01d498 thru 0x022b59 (0x56c2 bytes) fallocating to largest ever: 0x3d1d2 000160 1789997227.739559 fallocate 0x0396c8 thru 0x03d1d1 (0x3b0a bytes) truncating to largest ever: 0x3e59c 000162 1789997227.749772 trunc from 0x040000 to 0x03e59b (0x1a64 bytes) 000164 1789997227.771021 read 0x002873 thru 0x007c13 (0x53a1 bytes) 000165 1789997227.771760 mapwrite 0x010e3a thru 0x01eda6 (0xdf6d bytes) 000166 1789997227.794198 trunc from 0x03ddec to 0x03e59b (0x7b0 bytes) 000167 1789997227.804600 trunc from 0x03e59c to 0x030c0d (0xd98e bytes) 000168 1789997227.824346 trunc from 0x0283d8 to 0x030a86 (0x86af bytes) 000169 1789997227.840887 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30c0e): Mirror 3: 0xe006dd07 (stale) Mirror 4: 0xe006dd07 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30c0e): Mirror 3: 0xe006dd07 (stale) Mirror 4: 0xe006dd07 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30c0e: skipped 000170 1789997228.179371 mapread 0x002ef7 thru 0x00648c (0x3596 bytes) 000171 1789997228.196938 trunc from 0x01b0b6 to 0x01b1c4 (0x10f bytes) 000172 1789997228.201791 trunc from 0x015a57 to 0x02528c (0xf836 bytes) 000173 1789997228.214131 punch from 0x00f519 to 0x01e413 (0xeefb bytes) 000174 1789997228.226783 trunc from 0x030c0e to 0x006d2a (0x29ee3 bytes) 000175 1789997228.258872 write 0x02a830 thru 0x030a03 (0x61d4 bytes) 000176 1789997228.262795 mapread 0x0297b5 thru 0x02d57c (0x3dc8 bytes) 000177 1789997228.274917 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30a04): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30a04): Mirror 3: 0x754a78fe (stale) Mirror 4: 0x754a78fe (stale) CRC-32 checksum value for chunk [0, 0x30a04): Mirror 1: 0x2681dc0a Mirror 5: 0x2681dc0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30a04: skipped 000178 1789997229.110343 write 0x02756a thru 0x02f1a3 (0x7c3a bytes) 000179 1789997229.160964 mapwrite 0x01da07 thru 0x027e80 (0xa47a bytes) 000180 1789997229.215293 read 0x0052fb thru 0x00cd0d (0x7a13 bytes) 000181 1789997229.228080 MIRROR_RESYNC 4 mirrors 000182 1789997229.896013 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) 000183 1789997230.336938 punch from 0x000e6e to 0x0083d4 (0x7567 bytes) 000184 1789997230.443627 read 0x02f5c0 thru 0x030a03 (0x1444 bytes) 000185 1789997230.465363 mapread 0x0103b7 thru 0x01a173 (0x9dbd bytes) 000186 1789997230.513318 trunc from 0x030a04 to 0x0349a9 (0x3fa6 bytes) 000187 1789997230.577726 mapread 0x01dbe7 thru 0x02d35e (0xf778 bytes) 000189 1789997230.619839 trunc from 0x0349aa to 0x03ab59 (0x61b0 bytes) 000190 1789997230.647502 punch from 0x023307 to 0x0235a2 (0x29c bytes) 000191 1789997230.667408 read 0x00011d thru 0x00ff61 (0xfe45 bytes) 000192 1789997230.678869 mapread 0x025956 thru 0x02f87d (0x9f28 bytes) 000193 1789997230.687372 read 0x035330 thru 0x03ab59 (0x582a bytes) 000194 1789997230.688327 mapread 0x00b33f thru 0x016191 (0xae53 bytes) 000195 1789997230.689882 write 0x034e69 thru 0x036f86 (0x211e bytes) 000197 1789997230.695898 read 0x029e69 thru 0x02bdd4 (0x1f6c bytes) 000198 1789997230.700358 mapwrite 0x038f2e thru 0x03cc2d (0x3d00 bytes) 000199 1789997230.765336 trunc from 0x01d903 to 0x025f85 (0x8683 bytes) 000200 1789997230.769598 write 0x03e887 thru 0x03ffff (0x1779 bytes) 000202 1789997230.771806 trunc from 0x03a0e3 to 0x03f415 (0x5333 bytes) 000203 1789997230.789141 mapwrite 0x02face thru 0x039561 (0x9a94 bytes) 000204 1789997230.815356 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5c8ac25b (stale) Mirror 5: 0x5c8ac25b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5913bf8e Mirror 6: 0x5913bf8e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000205 1789997231.687327 write 0x01bf15 thru 0x027075 (0xb161 bytes) 000206 1789997231.728342 punch from 0x0091c3 to 0x015b59 (0xc997 bytes) 000207 1789997231.734632 read 0x02a7d3 thru 0x036357 (0xbb85 bytes) 000208 1789997231.742511 punch from 0x034b22 to 0x0398d3 (0x4db2 bytes) 000209 1789997231.746307 punch from 0x0139f0 to 0x01e676 (0xac87 bytes) 000210 1789997231.766121 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5c8ac25b (stale) Mirror 5: 0x5c8ac25b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5c8ac25b (stale) Mirror 5: 0x5c8ac25b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000211 1789997232.164741 punch from 0x01cc89 to 0x022df5 (0x616d bytes) 000212 1789997232.169642 mapread 0x025bb0 thru 0x02c5ae (0x69ff bytes) fallocating to largest ever: 0x40000 000213 1789997232.190497 fallocate 0x036a52 thru 0x03ffff (0x95ae bytes) 000214 1789997232.194996 mapwrite 0x03c006 thru 0x03e0aa (0x20a5 bytes) 000215 1789997232.221549 write 0x00b7aa thru 0x00d50f (0x1d66 bytes) 000216 1789997232.248992 write 0x02d28a thru 0x039db9 (0xcb30 bytes) 000218 1789997232.254439 write 0x030b90 thru 0x033bd7 (0x3048 bytes) 000219 1789997232.257324 read 0x01ad73 thru 0x02338b (0x8619 bytes) 000220 1789997232.297865 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5c8ac25b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5c8ac25b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000222 1789997232.737475 mapwrite 0x031b59 thru 0x03e72f (0xcbd7 bytes) 000223 1789997232.812620 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5c8ac25b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8f72ac91 Mirror 5: 0x8f72ac91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000224 1789997234.001606 read 0x01b1b9 thru 0x01bc85 (0xacd bytes) 000225 1789997234.015550 mapwrite 0x004f2a thru 0x005185 (0x25c bytes) 000226 1789997234.146378 punch from 0x0192f3 to 0x0208fc (0x760a bytes) 000227 1789997234.154039 mapread 0x00dd1a thru 0x011fcd (0x42b4 bytes) 000228 1789997234.175614 write 0x033d1a thru 0x03ffff (0xc2e6 bytes) 000229 1789997234.203712 trunc from 0x040000 to 0x022827 (0x1d7d8 bytes) 000230 1789997234.242251 mapwrite 0x012a81 thru 0x01d4b4 (0xaa34 bytes) 000232 1789997234.286782 read 0x002643 thru 0x011224 (0xebe2 bytes) 000233 1789997234.287895 read 0x01c434 thru 0x022827 (0x63f4 bytes) 000234 1789997234.289526 mapread 0x01f6c5 thru 0x022827 (0x3163 bytes) 000235 1789997234.293856 mapread 0x01ec98 thru 0x021d18 (0x3081 bytes) 000236 1789997234.297797 read 0x01dee1 thru 0x01ed29 (0xe49 bytes) 000237 1789997234.298520 write 0x01c075 thru 0x0246e2 (0x866e bytes) 000238 1789997234.306809 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x246e3): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x246e3): Mirror 4: 0xcf089c67 (stale) Mirror 5: 0xf04d6161 (stale) CRC-32 checksum value for chunk [0, 0x246e3): Mirror 1: 0x4515e30f Mirror 6: 0x4515e30f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x246e3: skipped 000239 1789997235.256979 trunc from 0x01532f to 0x01a55a (0x522c bytes) 000241 1789997235.301840 fallocate 0x01fc99 thru 0x02c41f (0xc787 bytes) 000242 1789997235.309545 trunc from 0x020967 to 0x0246e2 (0x3d7c bytes) 000243 1789997235.314242 write 0x0001d5 thru 0x007e0a (0x7c36 bytes) 000244 1789997235.333325 read 0x0231b5 thru 0x0246e2 (0x152e bytes) 000245 1789997235.341868 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x246e3): Mirror 4: 0xcf089c67 (stale) Mirror 5: 0xf04d6161 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x246e3): Mirror 4: 0xcf089c67 (stale) Mirror 5: 0xf04d6161 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x246e3: skipped 000246 1789997235.781774 mapread 0x00c5e2 thru 0x010191 (0x3bb0 bytes) 000247 1789997235.811734 mapread 0x0121ae thru 0x0215f2 (0xf445 bytes) 000248 1789997235.839153 fallocate 0x0247dd thru 0x0339ea (0xf20e bytes) 000249 1789997235.845437 mapread 0x01c78d thru 0x02b7eb (0xf05f bytes) 000250 1789997235.855861 punch from 0x0030f9 to 0x00b90e (0x8816 bytes) 000251 1789997235.860847 read 0x02f660 thru 0x0339ea (0x438b bytes) 000252 1789997235.875412 trunc from 0x0339eb to 0x0155ad (0x1e43d bytes) 000253 1789997235.942341 mapwrite 0x023305 thru 0x0276e1 (0x43dd bytes) 000254 1789997235.995951 fallocate 0x03e643 thru 0x03ffff (0x19bd bytes) 000255 1789997236.011202 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5c8ac25b (stale) Mirror 5: 0x8f72ac91 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9b44478e Mirror 6: 0x9b44478e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000256 1789997236.905632 fallocate 0x01df4b thru 0x029a25 (0xbadb bytes) 000257 1789997236.948052 trunc from 0x011b84 to 0x0155d5 (0x3a52 bytes) 000258 1789997236.953259 fallocate 0x027fd2 thru 0x031afb (0x9b2a bytes) 000259 1789997236.958556 mapwrite 0x01eea8 thru 0x02e3a1 (0xf4fa bytes) 000260 1789997237.026920 punch from 0x027b51 to 0x0286cc (0xb7c bytes) 000262 1789997237.032423 mapread 0x00f098 thru 0x00f306 (0x26f bytes) 000263 1789997237.038951 mapwrite 0x03455f thru 0x034b04 (0x5a6 bytes) 000264 1789997237.061371 trunc from 0x040000 to 0x028b1c (0x174e3 bytes) 000265 1789997237.088718 fallocate 0x030520 thru 0x03d7f8 (0xd2d9 bytes) 000266 1789997237.096544 mapread 0x0014cf thru 0x007cec (0x681e bytes) 000267 1789997237.112191 mapread 0x01c461 thru 0x01fbb4 (0x3754 bytes) 000268 1789997237.114058 trunc from 0x01488e to 0x01ba28 (0x719b bytes) 000269 1789997237.119886 mapread 0x01dc53 thru 0x028b1c (0xaeca bytes) 000270 1789997237.128172 mapread 0x00e9e8 thru 0x011387 (0x29a0 bytes) 000271 1789997237.131517 mapread 0x00231e thru 0x006f2d (0x4c10 bytes) 000272 1789997237.135337 mapread 0x00084c thru 0x005d39 (0x54ee bytes) 000273 1789997237.147429 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28b1d): Mirror 5: 0x1572a307 (stale) Mirror 6: 0xad168f18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28b1d): Mirror 5: 0x1572a307 (stale) Mirror 6: 0xad168f18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28b1d: skipped 000274 1789997237.570570 mapwrite 0x03e475 thru 0x03ffff (0x1b8b bytes) 000275 1789997237.630400 write 0x0378d9 thru 0x03ffff (0x8727 bytes) 000277 1789997237.639578 mapread 0x0295b3 thru 0x0360d5 (0xcb23 bytes) 000278 1789997237.659732 mapread 0x028153 thru 0x02a46d (0x231b bytes) 000279 1789997237.661162 trunc from 0x007e1e to 0x0138de (0xbac1 bytes) 000280 1789997237.682199 punch from 0x007044 to 0x00bf7e (0x4f3b bytes) 000281 1789997237.689106 fallocate 0x034ca3 thru 0x035fbd (0x131b bytes) 000282 1789997237.695713 mapwrite 0x03d5bb thru 0x03ffff (0x2a45 bytes) 000283 1789997237.708351 mapwrite 0x00a3d9 thru 0x00bb3f (0x1767 bytes) 000284 1789997237.727402 trunc from 0x040000 to 0x026ead (0x19152 bytes) 000285 1789997237.758540 mapwrite 0x02d909 thru 0x035e64 (0x855c bytes) 000286 1789997237.809665 read 0x035c2f thru 0x035e64 (0x236 bytes) 000288 1789997237.812720 trunc from 0x035e65 to 0x038bfd (0x2d99 bytes) 000289 1789997237.827175 mapwrite 0x03d328 thru 0x03ed9f (0x1a78 bytes) 000290 1789997237.868091 trunc from 0x03eda0 to 0x028d7e (0x16021 bytes) 000291 1789997237.888078 MIRROR_RESYNC 3 mirrors 000292 1789997238.666102 mapwrite 0x0045bd thru 0x012c65 (0xe6a9 bytes) 000293 1789997238.782590 mapread 0x011fd8 thru 0x01c073 (0xa09c bytes) 000294 1789997238.796367 trunc from 0x00327b to 0x007871 (0x45f7 bytes) 000295 1789997238.804265 punch from 0x0029fc to 0x00cfc4 (0xa5c9 bytes) 000296 1789997238.812072 fallocate 0x018dec thru 0x0269fd (0xdc12 bytes) 000298 1789997238.817716 fallocate 0x038e4e thru 0x03f755 (0x6908 bytes) 000299 1789997238.821524 fallocate 0x00eeb4 thru 0x01dd1b (0xee68 bytes) 000300 1789997238.827644 mapread 0x007c6a thru 0x00961c (0x19b3 bytes) 000301 1789997238.837829 mapread 0x007004 thru 0x00d6f3 (0x66f0 bytes) 000303 1789997238.843134 mapwrite 0x034986 thru 0x03c80e (0x7e89 bytes) 000305 1789997238.880071 fallocate 0x0293a4 thru 0x033882 (0xa4df bytes) 000306 1789997238.897117 MIRROR_RESYNC 3 mirrors 000307 1789997239.729266 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) 000308 1789997240.022079 read 0x02b161 thru 0x02edd6 (0x3c76 bytes) 000309 1789997240.029631 trunc from 0x03f756 to 0x0134fb (0x2c25a bytes) 000310 1789997240.086124 trunc from 0x00af68 to 0x0134fb (0x8594 bytes) 000311 1789997240.095736 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x134fc): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x134fc): Mirror 6: 0xb69b0c12 (stale) CRC-32 checksum value for chunk [0, 0x134fc): Mirror 1: 0x3cd99e80 Mirror 7: 0x3cd99e80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x134fc: skipped 000312 1789997240.920018 mapread 0x00a976 thru 0x00f21d (0x48a8 bytes) 000313 1789997240.936673 trunc from 0x0134fc to 0x01e124 (0xac29 bytes) 000314 1789997241.005063 mapwrite 0x0314a0 thru 0x03fc01 (0xe762 bytes) 000315 1789997241.066259 read 0x00a08a thru 0x00e0ce (0x4045 bytes) 000317 1789997241.075143 mapread 0x0369ce thru 0x03bc94 (0x52c7 bytes) 000318 1789997241.076090 write 0x034175 thru 0x03c3f6 (0x8282 bytes) 000319 1789997241.077681 punch from 0x02d8cb to 0x03c797 (0xeecd bytes) 000320 1789997241.102842 punch from 0x03b5ec to 0x03fc01 (0x4616 bytes) 000322 1789997241.109678 MIRROR_RESYNC 3 mirrors 000324 1789997241.839689 fallocate 0x017992 thru 0x0230d0 (0xb73f bytes) 000325 1789997241.898414 trunc from 0x03fc02 to 0x03dc28 (0x1fd9 bytes) 000326 1789997241.989206 fallocate 0x03bf84 thru 0x03ffff (0x407c bytes) 000327 1789997242.019085 MIRROR_RESYNC 3 mirrors 000329 1789997242.840759 mapread 0x032906 thru 0x038586 (0x5c81 bytes) 000330 1789997242.859831 mapread 0x00b4d2 thru 0x00f355 (0x3e84 bytes) 000331 1789997242.875144 trunc from 0x040000 to 0x029b75 (0x1648a bytes) 000332 1789997242.926019 mapread 0x012713 thru 0x01a446 (0x7d34 bytes) 000333 1789997242.938527 write 0x03161b thru 0x0316c8 (0xae bytes) 000334 1789997242.939474 mapread 0x012592 thru 0x01d358 (0xadc7 bytes) 000335 1789997242.952194 MIRROR_RESYNC 3 mirrors 000336 1789997243.634226 trunc from 0x0316c9 to 0x01ec7b (0x12a4d bytes) 000337 1789997243.693955 trunc from 0x008b10 to 0x00df1d (0x540e bytes) 000339 1789997243.702533 trunc from 0x01ec7c to 0x001f3c (0x1cd3f bytes) 000340 1789997243.729006 punch from 0x0003a6 to 0x001f3c (0x1b97 bytes) 000341 1789997243.736691 trunc from 0x001f3d to 0x038e7d (0x36f41 bytes) 000342 1789997243.755981 MIRROR_RESYNC 3 mirrors 000343 1789997244.369184 trunc from 0x02242e to 0x026f7b (0x4b4e bytes) 000344 1789997244.402413 trunc from 0x026ad2 to 0x0320a2 (0xb5d1 bytes) 000345 1789997244.406764 read 0x0245a2 thru 0x0284b8 (0x3f17 bytes) 000346 1789997244.415715 write 0x0396db thru 0x03ffff (0x6925 bytes) 000347 1789997244.417367 fallocate 0x01cbd9 thru 0x023fb8 (0x73e0 bytes) 000348 1789997244.423813 mapread 0x02e798 thru 0x03355d (0x4dc6 bytes) 000349 1789997244.452922 trunc from 0x040000 to 0x03dec8 (0x2137 bytes) 000350 1789997244.474956 mapwrite 0x038c29 thru 0x03b092 (0x246a bytes) 000351 1789997244.511213 mapread 0x037e7e thru 0x03dec8 (0x604b bytes) 000352 1789997244.513345 read 0x02f1da thru 0x0336ac (0x44d3 bytes) 000353 1789997244.518112 fallocate 0x01cf56 thru 0x028eec (0xbf97 bytes) 000354 1789997244.527757 read 0x01a0fc thru 0x01a746 (0x64b bytes) 000355 1789997244.539439 mapread 0x03469c thru 0x0385d0 (0x3f35 bytes) 000356 1789997244.540983 mapwrite 0x01be9a thru 0x0245e7 (0x874e bytes) 000357 1789997244.564812 punch from 0x015f49 to 0x021c92 (0xbd4a bytes) 000358 1789997244.571096 read 0x021303 thru 0x023090 (0x1d8e bytes) 000359 1789997244.584346 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dec9): Mirror 7: 0x4d72bfc8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dec9): Mirror 7: 0x4d72bfc8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dec9: skipped 000360 1789997244.884651 trunc from 0x01d79e to 0x01ea47 (0x12aa bytes) 000361 1789997244.890260 mapread 0x005147 thru 0x00c766 (0x7620 bytes) 000362 1789997244.930634 mapwrite 0x01dab1 thru 0x0281aa (0xa6fa bytes) 000363 1789997244.969512 read 0x02f582 thru 0x02fa5f (0x4de bytes) 000364 1789997244.973990 fallocate 0x000ec6 thru 0x00d0b6 (0xc1f1 bytes) 000365 1789997244.981041 trunc from 0x02c47e to 0x03b3e7 (0xef6a bytes) 000366 1789997244.987939 read 0x03a9b8 thru 0x03dec8 (0x3511 bytes) 000367 1789997245.003952 trunc from 0x03dec9 to 0x02d251 (0x10c77 bytes) 000368 1789997245.025111 MIRROR_RESYNC 2 mirrors 000369 1789997245.665012 punch from 0x00e37f to 0x016fa8 (0x8c2a bytes) 000370 1789997245.695300 punch from 0x005508 to 0x00fb50 (0xa649 bytes) 000371 1789997245.702655 write 0x039b7d thru 0x03ce7a (0x32fe bytes) 000372 1789997245.704056 write 0x01e222 thru 0x0291b7 (0xaf96 bytes) 000373 1789997245.722927 punch from 0x01bb04 to 0x01fffa (0x44f7 bytes) 000374 1789997245.747812 mapread 0x028bb3 thru 0x037d15 (0xf163 bytes) 000375 1789997245.783050 mapwrite 0x02ee16 thru 0x03e69f (0xf88a bytes) 000376 1789997245.888743 fallocate 0x02a8c7 thru 0x037c59 (0xd393 bytes) 000377 1789997245.900924 trunc from 0x00e062 to 0x010bce (0x2b6d bytes) 000378 1789997245.916176 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e6a0): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e6a0): Mirror 7: 0xb675b953 (stale) CRC-32 checksum value for chunk [0, 0x3e6a0): Mirror 1: 0x929d77fd Mirror 8: 0x929d77fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e6a0: skipped 000379 1789997246.858969 mapread 0x0327cc thru 0x037c50 (0x5485 bytes) 000380 1789997246.893941 fallocate 0x003301 thru 0x00b83e (0x853e bytes) 000381 1789997246.932090 read 0x004506 thru 0x013141 (0xec3c bytes) 000382 1789997246.947099 trunc from 0x03e6a0 to 0x00d8f4 (0x30dab bytes) 000383 1789997246.984049 write 0x00b4f1 thru 0x016c42 (0xb752 bytes) 000385 1789997246.986002 punch from 0x012b5c to 0x016c42 (0x40e7 bytes) 000386 1789997247.016307 mapread 0x008259 thru 0x008587 (0x32f bytes) 000388 1789997247.018060 read 0x0104da thru 0x016c42 (0x6769 bytes) 000389 1789997247.047289 trunc from 0x01179d to 0x016c42 (0x54a6 bytes) 000390 1789997247.053879 fallocate 0x030148 thru 0x03fcd3 (0xfb8c bytes) 000391 1789997247.065859 write 0x035ce4 thru 0x03cf2a (0x7247 bytes) 000392 1789997247.071350 trunc from 0x011fc1 to 0x016175 (0x41b5 bytes) 000393 1789997247.085083 write 0x032c0a thru 0x03edcc (0xc1c3 bytes) 000394 1789997247.093919 mapwrite 0x00f3c0 thru 0x013313 (0x3f54 bytes) 000395 1789997247.164730 fallocate 0x01a48f thru 0x020ea2 (0x6a14 bytes) 000396 1789997247.172543 trunc from 0x0219ca to 0x030353 (0xe98a bytes) 000397 1789997247.179563 write 0x013955 thru 0x018aa2 (0x514e bytes) 000398 1789997247.180736 read 0x001652 thru 0x00d3c0 (0xbd6f bytes) 000399 1789997247.181552 fallocate 0x02b7cf thru 0x03790d (0xc13f bytes) 000400 1789997247.187251 trunc from 0x016510 to 0x01f16f (0x8c60 bytes) 000401 1789997247.208640 mapwrite 0x00db3e thru 0x01690d (0x8dd0 bytes) 000402 1789997247.248361 mapread 0x0139c0 thru 0x0203aa (0xc9eb bytes) 000403 1789997247.255736 MIRROR_RESYNC 3 mirrors 000404 1789997248.119774 mapwrite 0x02041f thru 0x026a4b (0x662d bytes) 000405 1789997248.203008 read 0x030499 thru 0x0309fc (0x564 bytes) 000406 1789997248.207322 mapread 0x013b86 thru 0x016f26 (0x33a1 bytes) 000407 1789997248.215610 mapwrite 0x01d10f thru 0x027cf9 (0xabeb bytes) 000408 1789997248.240909 mapwrite 0x0013bb thru 0x0084aa (0x70f0 bytes) 000409 1789997248.265860 write 0x02e6eb thru 0x038a12 (0xa328 bytes) 000410 1789997248.289109 mapwrite 0x03b360 thru 0x03d5aa (0x224b bytes) 000412 1789997248.329835 read 0x00263b thru 0x00ea50 (0xc416 bytes) 000413 1789997248.335323 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3edcd): Mirror 8: 0x17e869b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3edcd): Mirror 8: 0x17e869b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3edcd: skipped 000414 1789997248.655128 trunc from 0x03edcd to 0x02403e (0x1ad8e bytes) 000415 1789997248.708741 read 0x000593 thru 0x009f24 (0x9992 bytes) 000416 1789997248.722876 mapread 0x00fa0d thru 0x01ce59 (0xd44d bytes) 000417 1789997248.770287 fallocate 0x0047d6 thru 0x006485 (0x1cb0 bytes) 000418 1789997248.778406 mapread 0x01220e thru 0x01aaaf (0x88a2 bytes) 000419 1789997248.780706 punch from 0x001b46 to 0x00fa97 (0xdf52 bytes) 000420 1789997248.788272 punch from 0x00d990 to 0x018e08 (0xb479 bytes) 000421 1789997248.793638 trunc from 0x00253b to 0x00f694 (0xd15a bytes) 000422 1789997248.798718 mapread 0x0058fa thru 0x0119ec (0xc0f3 bytes) 000423 1789997248.820232 mapread 0x004cc7 thru 0x006dac (0x20e6 bytes) 000424 1789997248.822108 fallocate 0x020259 thru 0x02764c (0x73f4 bytes) 000425 1789997248.826389 trunc from 0x02764d to 0x037fb4 (0x10968 bytes) 000426 1789997248.840697 read 0x0062ed thru 0x009594 (0x32a8 bytes) 000427 1789997248.841408 mapread 0x02ec07 thru 0x037fb4 (0x93ae bytes) 000428 1789997248.848576 punch from 0x0249e1 to 0x0316b3 (0xccd3 bytes) 000429 1789997248.853546 fallocate 0x022908 thru 0x02b714 (0x8e0d bytes) 000430 1789997248.862125 MIRROR_RESYNC 2 mirrors 000431 1789997249.419946 write 0x038caf thru 0x03ffff (0x7351 bytes) 000432 1789997249.447857 mapread 0x00be65 thru 0x01aed3 (0xf06f bytes) 000433 1789997249.489839 mapwrite 0x03e4d9 thru 0x03ffff (0x1b27 bytes) 000435 1789997249.516976 fallocate 0x02c097 thru 0x03644f (0xa3b9 bytes) 000436 1789997249.523431 read 0x0323d2 thru 0x0350ef (0x2d1e bytes) 000437 1789997249.531411 mapwrite 0x02b99d thru 0x02f970 (0x3fd4 bytes) 000438 1789997249.563706 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x5941504e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xede98c0c Mirror 9: 0xede98c0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000439 1789997250.383187 write 0x0396a3 thru 0x03b6ca (0x2028 bytes) 000440 1789997250.451980 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x5941504e (stale) Mirror 9: 0xede98c0c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2b290a90 Mirror 10: 0x2b290a90 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000441 1789997251.537601 mapread 0x0370b9 thru 0x03ffff (0x8f47 bytes) 000442 1789997251.598054 mapread 0x0003ed thru 0x008fb1 (0x8bc5 bytes) 000443 1789997251.669020 mapwrite 0x03923e thru 0x03ffff (0x6dc2 bytes) 000444 1789997251.779612 trunc from 0x040000 to 0x01b48f (0x24b70 bytes) 000445 1789997251.818899 trunc from 0x004572 to 0x00708c (0x2b1b bytes) 000446 1789997251.824547 fallocate 0x01e2fe thru 0x01ea95 (0x798 bytes) 000447 1789997251.830273 trunc from 0x01ea96 to 0x02c8fc (0xde67 bytes) 000449 1789997251.848705 trunc from 0x02c8fd to 0x02cb19 (0x21d bytes) 000450 1789997251.870617 MIRROR_RESYNC 4 mirrors 000451 1789997252.779785 read 0x02b94e thru 0x02cb19 (0x11cc bytes) 000452 1789997252.789601 fallocate 0x019c5a thru 0x01a869 (0xc10 bytes) 000453 1789997252.855796 read 0x023cf9 thru 0x029bd9 (0x5ee1 bytes) 000454 1789997252.872433 write 0x00d5d3 thru 0x00e37e (0xdac bytes) 000455 1789997252.873488 mapwrite 0x018a38 thru 0x026073 (0xd63c bytes) 000456 1789997252.915556 write 0x012077 thru 0x01e26a (0xc1f4 bytes) 000457 1789997252.917167 mapwrite 0x01cc62 thru 0x021151 (0x44f0 bytes) 000458 1789997252.934561 mapwrite 0x02f64b thru 0x03c95c (0xd312 bytes) 000460 1789997253.042825 mapwrite 0x02fb8f thru 0x0370be (0x7530 bytes) 000461 1789997253.078164 write 0x01a233 thru 0x02240c (0x81da bytes) 000462 1789997253.079689 write 0x036866 thru 0x039912 (0x30ad bytes) 000463 1789997253.085944 mapwrite 0x01bd96 thru 0x02baa2 (0xfd0d bytes) 000464 1789997253.115289 punch from 0x005c7b to 0x014893 (0xec19 bytes) 000466 1789997253.123092 mapread 0x000703 thru 0x002ce4 (0x25e2 bytes) 000468 1789997253.125953 punch from 0x01067c to 0x015b17 (0x549c bytes) 000469 1789997253.133139 mapwrite 0x02708f thru 0x02c77c (0x56ee bytes) 000470 1789997253.158601 MIRROR_RESYNC 4 mirrors 000471 1789997254.120295 mapwrite 0x03dff2 thru 0x03ffff (0x200e bytes) 000473 1789997254.280134 mapread 0x022097 thru 0x022986 (0x8f0 bytes) 000474 1789997254.292311 write 0x014b54 thru 0x01d026 (0x84d3 bytes) 000475 1789997254.297396 trunc from 0x040000 to 0x007b67 (0x38498 bytes) 000476 1789997254.321093 trunc from 0x007b68 to 0x02fced (0x28186 bytes) 000477 1789997254.343589 write 0x03c665 thru 0x03ffff (0x399b bytes) 000479 1789997254.345677 write 0x01b377 thru 0x02348c (0x8116 bytes) 000480 1789997254.361875 punch from 0x00b3a0 to 0x00d79e (0x23ff bytes) 000482 1789997254.372102 read 0x03da0a thru 0x03fd95 (0x238c bytes) 000483 1789997254.373171 read 0x02aaa7 thru 0x02ba56 (0xfb0 bytes) 000485 1789997254.399169 mapread 0x02d6be thru 0x03d461 (0xfda4 bytes) 000486 1789997254.400276 read 0x012c13 thru 0x01b9d6 (0x8dc4 bytes) 000487 1789997254.401352 fallocate 0x033955 thru 0x03ede1 (0xb48d bytes) 000488 1789997254.406670 punch from 0x012f3a to 0x019a4f (0x6b16 bytes) 000489 1789997254.414277 write 0x019d7a thru 0x02710f (0xd396 bytes) 000490 1789997254.425177 trunc from 0x040000 to 0x0065b6 (0x39a49 bytes) 000491 1789997254.455034 mapwrite 0x0001a9 thru 0x00473f (0x4597 bytes) 000492 1789997254.481901 write 0x008015 thru 0x016731 (0xe71d bytes) 000493 1789997254.483699 trunc from 0x016732 to 0x03769c (0x20f6b bytes) 000494 1789997254.498160 trunc from 0x03769d to 0x0158dd (0x21dbf bytes) 000495 1789997254.518411 read 0x0139a8 thru 0x0158dd (0x1f36 bytes) 000496 1789997254.519058 read 0x002e56 thru 0x011b06 (0xecb1 bytes) 000497 1789997254.542721 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x158de): Mirror 9: 0xa3a42f12 (stale) Mirror 10: 0xa3a42f12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x158de): Mirror 9: 0xa3a42f12 (stale) Mirror 10: 0xa3a42f12 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x158de: skipped 000498 1789997254.850497 read 0x0113e5 thru 0x0158dd (0x44f9 bytes) 000499 1789997254.858730 mapwrite 0x007d49 thru 0x010120 (0x83d8 bytes) 000500 1789997254.913014 trunc from 0x009811 to 0x0102af (0x6a9f bytes) 000501 1789997254.919054 write 0x033648 thru 0x03ffff (0xc9b8 bytes) 000502 1789997254.928379 write 0x00eacd thru 0x015dce (0x7302 bytes) 000504 1789997254.936776 trunc from 0x008971 to 0x017064 (0xe6f4 bytes) 000505 1789997254.957051 MIRROR_RESYNC 3 mirrors 000506 1789997255.812414 trunc from 0x040000 to 0x03165b (0xe9a4 bytes) 000507 1789997255.866366 trunc from 0x0048d9 to 0x00e0af (0x97d7 bytes) 000508 1789997255.877865 write 0x01b3cf thru 0x01cd0e (0x1940 bytes) 000510 1789997255.889069 trunc from 0x01c364 to 0x0273db (0xb078 bytes) 000511 1789997255.899050 mapread 0x017d2d thru 0x01c58f (0x4863 bytes) 000512 1789997255.922917 mapwrite 0x0138a8 thru 0x01d88e (0x9fe7 bytes) 000513 1789997255.949265 read 0x030566 thru 0x03165b (0x10f6 bytes) 000514 1789997255.955690 trunc from 0x03165c to 0x01c9c3 (0x14c98 bytes) 000515 1789997255.985186 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1c9c4): Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c9c4): Mirror 9: 0x7358eca1 (stale) Mirror 10: 0x7358eca1 (stale) CRC-32 checksum value for chunk [0, 0x1c9c4): Mirror 1: 0x7f9c24ec Mirror 11: 0x7f9c24ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c9c4: skipped 000516 1789997257.078943 mapwrite 0x00aaa5 thru 0x011003 (0x655f bytes) 000517 1789997257.226410 fallocate 0x029a18 thru 0x02a3f5 (0x9de bytes) 000518 1789997257.232636 mapwrite 0x00ee2e thru 0x014d79 (0x5f4c bytes) 000519 1789997257.266569 read 0x0164b7 thru 0x01d346 (0x6e90 bytes) 000520 1789997257.277041 read 0x012c85 thru 0x01396a (0xce6 bytes) 000521 1789997257.277916 read 0x002bba thru 0x00c960 (0x9da7 bytes) 000522 1789997257.287863 trunc from 0x02a3f6 to 0x033d2c (0x9937 bytes) 000523 1789997257.319613 mapread 0x032b1a thru 0x033d2c (0x1213 bytes) 000524 1789997257.342241 mapread 0x028653 thru 0x029181 (0xb2f bytes) 000525 1789997257.343326 mapwrite 0x018be2 thru 0x01eb27 (0x5f46 bytes) 000527 1789997257.365942 read 0x03163a thru 0x033d2c (0x26f3 bytes) 000528 1789997257.366749 mapread 0x0214d9 thru 0x0279a3 (0x64cb bytes) 000529 1789997257.369391 write 0x005f52 thru 0x0080e6 (0x2195 bytes) 000530 1789997257.371696 write 0x028e6f thru 0x02a2bc (0x144e bytes) 000532 1789997257.380954 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33d2d): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d2d): Mirror 9: 0xb0daa1a4 (stale) Mirror 10: 0xb0daa1a4 (stale) Mirror 11: 0x8f5bb5c (stale) CRC-32 checksum value for chunk [0, 0x33d2d): Mirror 1: 0x5306bea7 Mirror 12: 0x5306bea7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d2d: skipped 000533 1789997258.432684 trunc from 0x023599 to 0x02ac10 (0x7678 bytes) 000534 1789997258.508209 trunc from 0x006669 to 0x009d63 (0x36fb bytes) 000535 1789997258.518904 trunc from 0x033d2d to 0x00b4a6 (0x28886 bytes) 000536 1789997258.599018 mapwrite 0x021f2f thru 0x023fea (0x20bc bytes) 000537 1789997258.684886 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23feb): Mirror 9: 0xdcc3ae3b (stale) Mirror 10: 0xdcc3ae3b (stale) Mirror 12: 0xb913b37c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23feb): Mirror 9: 0xdcc3ae3b (stale) Mirror 10: 0xdcc3ae3b (stale) Mirror 12: 0xb913b37c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23feb: skipped 000538 1789997259.163709 mapread 0x006d8b thru 0x012c69 (0xbedf bytes) 000539 1789997259.209234 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x23feb): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23feb): Mirror 9: 0xdcc3ae3b (stale) Mirror 10: 0xdcc3ae3b (stale) Mirror 12: 0xb913b37c (stale) CRC-32 checksum value for chunk [0, 0x23feb): Mirror 1: 0x5f5d1ad0 Mirror 13: 0x5f5d1ad0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23feb: skipped 000540 1789997260.186240 trunc from 0x023feb to 0x0122be (0x11d2c bytes) 000542 1789997260.234084 trunc from 0x000d08 to 0x008d67 (0x8060 bytes) 000543 1789997260.240287 mapwrite 0x00aba5 thru 0x00b5ca (0xa26 bytes) 000544 1789997260.280601 mapwrite 0x013e6d thru 0x016fe1 (0x3175 bytes) 000545 1789997260.340041 trunc from 0x011128 to 0x015d85 (0x4c5e bytes) 000546 1789997260.351296 mapwrite 0x02ce17 thru 0x03637f (0x9569 bytes) 000547 1789997260.415977 write 0x006c24 thru 0x00cd67 (0x6144 bytes) 000548 1789997260.422356 mapread 0x010de3 thru 0x01d1b3 (0xc3d1 bytes) 000549 1789997260.439359 mapwrite 0x00fa08 thru 0x01d7f5 (0xddee bytes) 000550 1789997260.465158 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36380): Mirror 9: 0xc4f396e3 (stale) Mirror 10: 0xc4f396e3 (stale) Mirror 13: 0xb40d44d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36380): Mirror 9: 0xc4f396e3 (stale) Mirror 10: 0xc4f396e3 (stale) Mirror 13: 0xb40d44d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36380: skipped 000551 1789997260.834042 fallocate 0x007beb thru 0x012f2e (0xb344 bytes) 000552 1789997260.843541 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36380): Mirror 10: 0xc4f396e3 (stale) Mirror 13: 0xb40d44d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36380): Mirror 10: 0xc4f396e3 (stale) Mirror 13: 0xb40d44d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36380: skipped 000553 1789997261.251312 punch from 0x02a7af to 0x03637f (0xbbd1 bytes) 000554 1789997261.257375 fallocate 0x038db4 thru 0x03a884 (0x1ad1 bytes) 000555 1789997261.263235 punch from 0x0133c8 to 0x014f6c (0x1ba5 bytes) 000556 1789997261.269624 punch from 0x010391 to 0x0166ea (0x635a bytes) 000557 1789997261.281302 MIRROR_RESYNC 3 mirrors 000558 1789997262.426215 read 0x0124b4 thru 0x01e18d (0xbcda bytes) 000559 1789997262.435435 write 0x029dc4 thru 0x02e176 (0x43b3 bytes) 000561 1789997262.510311 trunc from 0x0382dd to 0x03a884 (0x25a8 bytes) 000562 1789997262.521911 mapread 0x035874 thru 0x03a884 (0x5011 bytes) 000563 1789997262.573728 mapread 0x02fbc3 thru 0x03322a (0x3668 bytes) 000564 1789997262.584675 trunc from 0x03a885 to 0x01867b (0x22209 bytes) 000565 1789997262.636441 punch from 0x00f166 to 0x01867b (0x9516 bytes) 000567 1789997262.649846 MIRROR_RESYNC 3 mirrors 000568 1789997263.570912 write 0x0374da thru 0x03a235 (0x2d5c bytes) 000570 1789997263.603016 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a236): Mirror 10: 0xc9526b60 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a236): Mirror 10: 0xc9526b60 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a236: skipped 000571 1789997263.896075 fallocate 0x01f92e thru 0x027dcc (0x849f bytes) 000572 1789997263.905573 trunc from 0x029af5 to 0x032fac (0x94b8 bytes) 000573 1789997263.912944 trunc from 0x03a236 to 0x013366 (0x26ecf bytes) 000574 1789997263.943250 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x13367): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13367): Mirror 10: 0x2f51f771 (stale) CRC-32 checksum value for chunk [0, 0x13367): Mirror 1: 0x2f51f771 Mirror 11: 0x2f51f771 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13367: skipped 000575 1789997264.869222 mapwrite 0x024a69 thru 0x029764 (0x4cfc bytes) 000576 1789997264.969199 trunc from 0x029765 to 0x030bfc (0x7498 bytes) 000577 1789997264.999932 write 0x023b94 thru 0x02ff22 (0xc38f bytes) 000578 1789997265.005881 fallocate 0x003f10 thru 0x00d714 (0x9805 bytes) 000579 1789997265.014384 punch from 0x007f5a to 0x00da33 (0x5ada bytes) 000580 1789997265.018903 fallocate 0x03ac28 thru 0x03ffff (0x53d8 bytes) 000581 1789997265.027374 MIRROR_RESYNC 3 mirrors 000582 1789997266.173640 MIRROR_RESYNC 3 mirrors 000583 1789997266.750036 mapwrite 0x011c6a thru 0x0127f1 (0xb88 bytes) 000584 1789997266.861656 write 0x035728 thru 0x03a774 (0x504d bytes) 000585 1789997266.864274 write 0x02efe2 thru 0x0307a0 (0x17bf bytes) 000586 1789997266.878080 trunc from 0x03a775 to 0x01a4e0 (0x20294 bytes) 000587 1789997266.901371 punch from 0x00fe3d to 0x015d78 (0x5f3c bytes) 000588 1789997266.909393 write 0x010f5c thru 0x01e030 (0xd0d5 bytes) 000589 1789997266.920110 read 0x0133ae thru 0x017f34 (0x4b87 bytes) 000590 1789997266.921759 punch from 0x007b0b to 0x00fc57 (0x814d bytes) 000591 1789997266.926367 write 0x02ecfb thru 0x02f6d5 (0x9db bytes) 000594 1789997266.929090 mapwrite 0x027f87 thru 0x03147f (0x94f9 bytes) 000595 1789997266.987846 fallocate 0x017f2d thru 0x0233e6 (0xb4ba bytes) 000596 1789997266.994053 write 0x03ef11 thru 0x03ffff (0x10ef bytes) 000597 1789997266.995538 fallocate 0x015394 thru 0x024dab (0xfa18 bytes) 000598 1789997267.000222 trunc from 0x040000 to 0x03cd0f (0x32f0 bytes) 000599 1789997267.016653 punch from 0x01b4d0 to 0x01b5dd (0x10e bytes) 000600 1789997267.022747 trunc from 0x026fee to 0x027be7 (0xbfa bytes) 000601 1789997267.028261 read 0x03912c thru 0x03cd0f (0x3be4 bytes) 000602 1789997267.044228 trunc from 0x03cd10 to 0x026eed (0x15e22 bytes) 000603 1789997267.075723 MIRROR_RESYNC 3 mirrors 000604 1789997268.043369 mapwrite 0x0255bc thru 0x02ce40 (0x7885 bytes) 000605 1789997268.194345 trunc from 0x02ce41 to 0x009915 (0x2352b bytes) 000606 1789997268.215726 trunc from 0x0068d9 to 0x0097c6 (0x2eee bytes) 000607 1789997268.221981 mapwrite 0x03b1cd thru 0x03b96c (0x7a0 bytes) 000609 1789997268.259322 mapwrite 0x010748 thru 0x01dbaf (0xd468 bytes) 000610 1789997268.325920 read 0x02a79e thru 0x0380a8 (0xd90b bytes) 000611 1789997268.340961 MIRROR_RESYNC 3 mirrors 000612 1789997269.121851 punch from 0x01f9cb to 0x02a6a6 (0xacdc bytes) 000613 1789997269.179827 read 0x0203c0 thru 0x02ee69 (0xeaaa bytes) 000614 1789997269.193413 trunc from 0x0332b8 to 0x03b96c (0x86b5 bytes) 000615 1789997269.205987 write 0x039c15 thru 0x03ffff (0x63eb bytes) 000616 1789997269.217666 mapread 0x013ab4 thru 0x01d14d (0x969a bytes) 000617 1789997269.249970 trunc from 0x01950f to 0x026f09 (0xd9fb bytes) 000618 1789997269.261700 trunc from 0x03c67a to 0x03ffff (0x3986 bytes) 000619 1789997269.294021 mapread 0x00199e thru 0x0048cc (0x2f2f bytes) 000620 1789997269.326212 trunc from 0x040000 to 0x03ca95 (0x356a bytes) 000621 1789997269.388700 read 0x00a92d thru 0x017c74 (0xd348 bytes) 000622 1789997269.390057 mapwrite 0x0074f8 thru 0x00bef2 (0x49fb bytes) 000623 1789997269.414677 write 0x02ae8b thru 0x02e172 (0x32e8 bytes) 000624 1789997269.417942 read 0x022595 thru 0x0269b0 (0x441c bytes) 000625 1789997269.442106 mapread 0x036d0e thru 0x037f13 (0x1206 bytes) 000626 1789997269.443050 write 0x034f90 thru 0x03ffff (0xb070 bytes) 000627 1789997269.444941 mapread 0x022216 thru 0x031613 (0xf3fe bytes) 000628 1789997269.451014 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xbaf1f039 (stale) Mirror 11: 0xbaf1f039 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5ce85896 Mirror 12: 0x5ce85896 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000629 1789997270.593057 write 0x01398e thru 0x0140e0 (0x753 bytes) 000630 1789997270.666994 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xbaf1f039 (stale) Mirror 12: 0x5ce85896 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xbaf1f039 (stale) Mirror 12: 0x5ce85896 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000631 1789997271.041387 read 0x019b9e thru 0x01b5ae (0x1a11 bytes) 000632 1789997271.049130 trunc from 0x040000 to 0x01c14b (0x23eb4 bytes) 000633 1789997271.069749 trunc from 0x01c14c to 0x02e4d8 (0x1238d bytes) 000634 1789997271.086882 punch from 0x01b9db to 0x025e49 (0xa46f bytes) 000635 1789997271.103980 MIRROR_RESYNC 3 mirrors 000636 1789997272.085061 write 0x017d8f thru 0x01bc9e (0x3f10 bytes) 000637 1789997272.123392 read 0x01db62 thru 0x01fa03 (0x1ea2 bytes) 000638 1789997272.131192 trunc from 0x02e4d9 to 0x02bc16 (0x28c2 bytes) 000639 1789997272.172324 mapwrite 0x020327 thru 0x02e572 (0xe24c bytes) 000641 1789997272.269281 trunc from 0x02e573 to 0x03c998 (0xe426 bytes) 000642 1789997272.297138 read 0x030b1a thru 0x035b73 (0x505a bytes) 000643 1789997272.310323 mapwrite 0x0036aa thru 0x005233 (0x1b8a bytes) 000644 1789997272.323533 trunc from 0x03c999 to 0x027fd5 (0x149c3 bytes) 000645 1789997272.352218 read 0x026a2f thru 0x027fd5 (0x15a7 bytes) 000646 1789997272.353071 mapwrite 0x002201 thru 0x004cb4 (0x2ab4 bytes) 000648 1789997272.370842 trunc from 0x027fd6 to 0x008fb3 (0x1f022 bytes) 000649 1789997272.407332 trunc from 0x008fb4 to 0x01ff46 (0x16f93 bytes) 000650 1789997272.423302 read 0x01b351 thru 0x01ff46 (0x4bf6 bytes) 000651 1789997272.435484 read 0x006294 thru 0x00bea7 (0x5c14 bytes) 000652 1789997272.436172 punch from 0x01ff35 to 0x01ff46 (0x12 bytes) 000653 1789997272.441940 write 0x033f6c thru 0x03510a (0x119f bytes) 000654 1789997272.443849 mapread 0x006240 thru 0x010831 (0xa5f2 bytes) 000655 1789997272.448232 fallocate 0x015866 thru 0x020aac (0xb247 bytes) 000656 1789997272.453663 fallocate 0x02c74d thru 0x03a719 (0xdfcd bytes) 000657 1789997272.462803 mapwrite 0x013c99 thru 0x023519 (0xf881 bytes) 000658 1789997272.499738 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a71a): Mirror 12: 0xa382643f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a71a): Mirror 12: 0xa382643f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a71a: skipped 000659 1789997272.896149 mapread 0x004b4a thru 0x0082a7 (0x375e bytes) 000660 1789997272.937250 mapwrite 0x00678c thru 0x00a3f8 (0x3c6d bytes) 000661 1789997272.958405 fallocate 0x01a673 thru 0x028ede (0xe86c bytes) 000662 1789997272.963474 read 0x02e049 thru 0x02e53b (0x4f3 bytes) 000663 1789997272.971911 MIRROR_RESYNC 2 mirrors 000664 1789997273.775958 trunc from 0x036c1a to 0x03a719 (0x3b00 bytes) 000665 1789997273.817343 trunc from 0x03a71a to 0x00b183 (0x2f596 bytes) 000666 1789997273.848411 fallocate 0x03520d thru 0x03ffff (0xadf3 bytes) 000667 1789997273.858436 fallocate 0x0020a4 thru 0x00dcc1 (0xbc1e bytes) 000668 1789997273.869309 MIRROR_RESYNC 2 mirrors 000670 1789997274.506075 MIRROR_EXTEND to 3 mirrors 000671 1789997275.242998 mapread 0x031dcc thru 0x03c05e (0xa293 bytes) 000672 1789997275.274312 trunc from 0x040000 to 0x032488 (0xdb77 bytes) 000674 1789997275.350069 trunc from 0x032489 to 0x0318ba (0xbce bytes) 000675 1789997275.389065 trunc from 0x020d2e to 0x02e231 (0xd504 bytes) 000676 1789997275.396399 read 0x029a19 thru 0x02e4d9 (0x4ac1 bytes) 000677 1789997275.415126 write 0x0398e4 thru 0x03ffff (0x671c bytes) 000678 1789997275.421258 MIRROR_RESYNC 3 mirrors 000679 1789997276.468854 trunc from 0x040000 to 0x01fbe3 (0x2041c bytes) 000680 1789997276.541743 punch from 0x0011c9 to 0x003ebe (0x2cf6 bytes) 000683 1789997276.552773 write 0x03d7da thru 0x03ffff (0x2826 bytes) 000684 1789997276.554199 trunc from 0x040000 to 0x037975 (0x868a bytes) 000685 1789997276.585742 fallocate 0x035715 thru 0x03c148 (0x6a34 bytes) 000686 1789997276.590890 mapread 0x0082e8 thru 0x0170db (0xedf4 bytes) 000687 1789997276.648232 trunc from 0x008cc3 to 0x00a762 (0x1aa0 bytes) 000689 1789997276.658893 punch from 0x00c42e to 0x016b4a (0xa71d bytes) 000690 1789997276.668158 write 0x03c441 thru 0x03ffff (0x3bbf bytes) 000691 1789997276.677269 trunc from 0x0227e8 to 0x02b43e (0x8c57 bytes) 000692 1789997276.688417 mapread 0x023b1d thru 0x023d86 (0x26a bytes) 000693 1789997276.697673 fallocate 0x025755 thru 0x02aff7 (0x58a3 bytes) 000694 1789997276.703171 read 0x0093d3 thru 0x01042a (0x7058 bytes) 000695 1789997276.712620 trunc from 0x040000 to 0x001e93 (0x3e16c bytes) 000696 1789997276.730127 trunc from 0x001240 to 0x001e93 (0xc54 bytes) 000697 1789997276.734861 read 0x0019e6 thru 0x001e93 (0x4ae bytes) 000698 1789997276.742746 read 0x000bf9 thru 0x001e93 (0x129b bytes) 000700 1789997276.744413 mapread 0x001819 thru 0x001e93 (0x67b bytes) 000702 1789997276.746030 write 0x027efd thru 0x02fb6b (0x7c6f bytes) 000703 1789997276.747854 read 0x010d19 thru 0x013d87 (0x306f bytes) 000704 1789997276.755270 fallocate 0x0318ea thru 0x033b40 (0x2257 bytes) 000705 1789997276.760128 mapread 0x01ddeb thru 0x02214f (0x4365 bytes) 000706 1789997276.761111 read 0x00b29c thru 0x01144e (0x61b3 bytes) 000707 1789997276.762070 mapread 0x00c651 thru 0x0135bb (0x6f6b bytes) 000708 1789997276.763147 write 0x03804d thru 0x03ae85 (0x2e39 bytes) 000709 1789997276.768826 MIRROR_RESYNC 3 mirrors 000710 1789997277.567184 mapread 0x01fa55 thru 0x021f68 (0x2514 bytes) 000711 1789997277.582628 punch from 0x031801 to 0x035f07 (0x4707 bytes) 000712 1789997277.618583 mapread 0x0211d8 thru 0x026805 (0x562e bytes) 000713 1789997277.632708 fallocate 0x00205a thru 0x00f0f2 (0xd099 bytes) 000714 1789997277.637585 mapwrite 0x01943f thru 0x0260fa (0xccbc bytes) 000715 1789997277.663323 read 0x003014 thru 0x00e113 (0xb100 bytes) 000716 1789997277.671334 mapwrite 0x027637 thru 0x03192d (0xa2f7 bytes) 000717 1789997277.702783 MIRROR_RESYNC 3 mirrors 000718 1789997278.641571 punch from 0x02cf93 to 0x035ea7 (0x8f15 bytes) 000719 1789997278.687579 fallocate 0x03b62a thru 0x03ffff (0x49d6 bytes) 000720 1789997278.693514 fallocate 0x031560 thru 0x03ab0a (0x95ab bytes) 000721 1789997278.703227 trunc from 0x029c79 to 0x03418d (0xa515 bytes) 000722 1789997278.712376 trunc from 0x03ae86 to 0x00472c (0x36759 bytes) 000723 1789997278.748197 punch from 0x003b42 to 0x00472c (0xbeb bytes) 000724 1789997278.763424 fallocate 0x032338 thru 0x03afca (0x8c93 bytes) 000725 1789997278.770044 mapwrite 0x000577 thru 0x007a44 (0x74ce bytes) 000726 1789997278.894058 trunc from 0x005ea1 to 0x007a44 (0x1ba4 bytes) 000727 1789997278.909349 MIRROR_RESYNC 3 mirrors 000728 1789997279.827560 MIRROR_EXTEND to 4 mirrors 000729 1789997280.641780 fallocate 0x0260b3 thru 0x034fdd (0xef2b bytes) 000730 1789997280.695008 fallocate 0x039206 thru 0x03ffff (0x6dfa bytes) 000731 1789997280.705278 mapwrite 0x0226d7 thru 0x02b26d (0x8b97 bytes) 000734 1789997280.753336 read 0x0310df thru 0x034603 (0x3525 bytes) 000735 1789997280.761519 write 0x010558 thru 0x018858 (0x8301 bytes) 000736 1789997280.782075 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34fde): Mirror 12: 0xc44b01ae (stale) Mirror 14: 0xc44b01ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34fde): Mirror 12: 0xc44b01ae (stale) Mirror 14: 0xc44b01ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34fde: skipped 000737 1789997281.116220 fallocate 0x011019 thru 0x01c502 (0xb4ea bytes) 000738 1789997281.120964 trunc from 0x016520 to 0x025f75 (0xfa56 bytes) 000739 1789997281.139688 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34fde): Mirror 14: 0xc44b01ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34fde): Mirror 14: 0xc44b01ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34fde: skipped 000741 1789997281.531227 mapwrite 0x01ff7b thru 0x02b62a (0xb6b0 bytes) 000742 1789997281.653965 mapwrite 0x00472a thru 0x005453 (0xd2a bytes) 000744 1789997281.691680 MIRROR_RESYNC 2 mirrors 000745 1789997282.663978 mapread 0x01fb3c thru 0x02695a (0x6e1f bytes) 000746 1789997282.702374 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) 000747 1789997282.860387 fallocate 0x01fd21 thru 0x024b5a (0x4e3a bytes) 000749 1789997282.883995 mapread 0x02d643 thru 0x034fdd (0x799b bytes) 000750 1789997282.904244 read 0x028bdd thru 0x034fdd (0xc401 bytes) 000751 1789997282.905180 fallocate 0x011bd5 thru 0x015394 (0x37c0 bytes) 000752 1789997282.910031 trunc from 0x034fde to 0x03575a (0x77d bytes) 000754 1789997282.931289 write 0x03afed thru 0x03ffff (0x5013 bytes) 000755 1789997282.933097 write 0x021b98 thru 0x02d3ea (0xb853 bytes) 000756 1789997282.939754 mapread 0x0018ae thru 0x006c7d (0x53d0 bytes) 000758 1789997282.950728 punch from 0x02372b to 0x0258dd (0x21b3 bytes) 000759 1789997282.964794 read 0x002c95 thru 0x00914c (0x64b8 bytes) 000760 1789997282.965348 trunc from 0x03721d to 0x037515 (0x2f9 bytes) 000761 1789997282.973139 trunc from 0x00c709 to 0x012320 (0x5c18 bytes) 000762 1789997282.979324 write 0x02ea06 thru 0x03a3c5 (0xb9c0 bytes) 000763 1789997282.982240 mapread 0x0306b2 thru 0x03b289 (0xabd8 bytes) 000764 1789997282.983837 fallocate 0x001a2a thru 0x00291a (0xef1 bytes) 000765 1789997282.993547 MIRROR_EXTEND to 2 mirrors 000766 1789997283.854479 punch from 0x016b96 to 0x01f33a (0x87a5 bytes) 000767 1789997283.888841 mapread 0x0213e0 thru 0x02262e (0x124f bytes) 000768 1789997283.900327 mapread 0x002b03 thru 0x010ff0 (0xe4ee bytes) 000769 1789997283.909261 trunc from 0x03acf2 to 0x03ffff (0x530e bytes) 000770 1789997283.915121 punch from 0x01d451 to 0x020655 (0x3205 bytes) 000771 1789997283.920747 write 0x0361a3 thru 0x03ffff (0x9e5d bytes) 000772 1789997283.932077 read 0x03ef60 thru 0x03ffff (0x10a0 bytes) 000773 1789997283.932816 trunc from 0x040000 to 0x02ec0a (0x113f5 bytes) 000775 1789997283.959086 read 0x0262d0 thru 0x02cf2e (0x6c5f bytes) 000776 1789997283.970866 trunc from 0x02ec0b to 0x026611 (0x85f9 bytes) 000777 1789997283.994193 MIRROR_RESYNC 2 mirrors 000778 1789997284.984849 punch from 0x01ea10 to 0x022993 (0x3f84 bytes) 000779 1789997285.021502 mapread 0x0105f5 thru 0x0122b5 (0x1cc1 bytes) 000780 1789997285.044171 trunc from 0x026612 to 0x020779 (0x5e98 bytes) 000781 1789997285.067144 trunc from 0x01b9ee to 0x020779 (0x4d8c bytes) 000782 1789997285.072330 mapwrite 0x031982 thru 0x03f738 (0xddb7 bytes) 000783 1789997285.130504 fallocate 0x0241ea thru 0x026b10 (0x2927 bytes) 000784 1789997285.140897 mapwrite 0x03de4a thru 0x03ffff (0x21b6 bytes) 000786 1789997285.174631 write 0x0093fd thru 0x00cfdc (0x3be0 bytes) 000787 1789997285.186598 MIRROR_RESYNC 2 mirrors 000788 1789997286.044696 trunc from 0x040000 to 0x02fb85 (0x1047a bytes) 000789 1789997286.122174 mapwrite 0x0033bb thru 0x0057b9 (0x23ff bytes) 000791 1789997286.157693 mapread 0x023f9b thru 0x02719f (0x3205 bytes) 000792 1789997286.169257 write 0x007d58 thru 0x00a284 (0x252d bytes) 000793 1789997286.170691 write 0x027bb9 thru 0x03163c (0x9a84 bytes) 000795 1789997286.173225 punch from 0x005edc to 0x009558 (0x367d bytes) 000796 1789997286.191307 write 0x03077c thru 0x034f52 (0x47d7 bytes) 000797 1789997286.192954 trunc from 0x00d6f0 to 0x014f65 (0x7876 bytes) 000798 1789997286.198363 write 0x00132f thru 0x007a2e (0x6700 bytes) 000800 1789997286.208991 punch from 0x02a599 to 0x034f52 (0xa9ba bytes) 000802 1789997286.251645 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34f53): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f53): Mirror 2: 0xb20e0952 (stale) CRC-32 checksum value for chunk [0, 0x34f53): Mirror 1: 0xc788e695 Mirror 3: 0xc788e695 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f53: skipped 000803 1789997287.140407 fallocate 0x039d4c thru 0x03ffff (0x62b4 bytes) 000804 1789997287.207920 fallocate 0x015c3d thru 0x01fe83 (0xa247 bytes) 000805 1789997287.221663 MIRROR_RESYNC 3 mirrors 000806 1789997288.246520 read 0x0094c0 thru 0x0166ce (0xd20f bytes) 000807 1789997288.262364 fallocate 0x01dfb6 thru 0x0291cf (0xb21a bytes) 000808 1789997288.335530 trunc from 0x034f53 to 0x03dab2 (0x8b60 bytes) 000811 1789997288.371606 punch from 0x02d013 to 0x02ea8d (0x1a7b bytes) 000812 1789997288.382295 trunc from 0x0143d6 to 0x020756 (0xc381 bytes) 000813 1789997288.392714 trunc from 0x03dab3 to 0x01d376 (0x2073c bytes) 000814 1789997288.417779 punch from 0x011c13 to 0x01d376 (0xb764 bytes) 000815 1789997288.424685 punch from 0x00ada2 to 0x01a49e (0xf6fd bytes) 000816 1789997288.430389 mapwrite 0x00a882 thru 0x011c2a (0x73a9 bytes) 000817 1789997288.477954 mapwrite 0x0120be thru 0x018a20 (0x6963 bytes) 000819 1789997288.512731 punch from 0x003d11 to 0x013693 (0xf983 bytes) 000820 1789997288.519195 mapread 0x0058c6 thru 0x00be28 (0x6563 bytes) 000821 1789997288.534104 mapread 0x003289 thru 0x00af34 (0x7cac bytes) 000822 1789997288.536083 mapwrite 0x03eedf thru 0x03ffff (0x1121 bytes) 000823 1789997288.619293 mapread 0x00fa57 thru 0x01f000 (0xf5aa bytes) 000824 1789997288.647054 fallocate 0x01727c thru 0x01bc9a (0x4a1f bytes) 000825 1789997288.655343 punch from 0x02b3f8 to 0x02ecb3 (0x38bc bytes) 000826 1789997288.667944 trunc from 0x040000 to 0x00dabb (0x32544 bytes) 000827 1789997288.699427 punch from 0x0079b8 to 0x00dabb (0x6104 bytes) 000828 1789997288.705679 mapread 0x00197c thru 0x00dabb (0xc140 bytes) 000829 1789997288.725558 write 0x02b328 thru 0x03829e (0xcf77 bytes) 000830 1789997288.733949 punch from 0x01a82b to 0x0287a3 (0xdf79 bytes) 000831 1789997288.740177 trunc from 0x03829f to 0x009706 (0x2eb98 bytes) 000832 1789997288.761270 punch from 0x00849a to 0x009706 (0x126d bytes) 000833 1789997288.769185 trunc from 0x0051ff to 0x009706 (0x4508 bytes) 000834 1789997288.779415 punch from 0x003100 to 0x005b35 (0x2a36 bytes) 000835 1789997288.786404 fallocate 0x00fb31 thru 0x0149dd (0x4ead bytes) 000836 1789997288.792342 trunc from 0x0149de to 0x033f21 (0x1f544 bytes) 000837 1789997288.815593 read 0x01ef3a thru 0x020ef3 (0x1fba bytes) 000838 1789997288.831940 MIRROR_RESYNC 3 mirrors 000839 1789997289.509656 read 0x004583 thru 0x0049cc (0x44a bytes) 000840 1789997289.522911 MIRROR_RESYNC 3 mirrors 000841 1789997289.883724 fallocate 0x0309d1 thru 0x03af81 (0xa5b1 bytes) 000842 1789997289.939786 trunc from 0x033f22 to 0x03a57b (0x665a bytes) 000843 1789997289.964898 trunc from 0x03a57c to 0x032dbc (0x77bf bytes) 000844 1789997289.987136 trunc from 0x032dbd to 0x00ad90 (0x2802c bytes) 000845 1789997290.008314 mapwrite 0x02fa5b thru 0x03cfb4 (0xd55a bytes) 000846 1789997290.094955 fallocate 0x012ff9 thru 0x01c15e (0x9166 bytes) 000847 1789997290.100766 punch from 0x0086ce to 0x00c6ea (0x401d bytes) 000849 1789997290.111730 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cfb5): Mirror 3: 0xc8291b00 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cfb5): Mirror 3: 0xc8291b00 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cfb5: skipped 000851 1789997290.405438 mapwrite 0x005dd3 thru 0x007d8b (0x1fb9 bytes) 000852 1789997290.460998 MIRROR_RESYNC 2 mirrors 000854 1789997291.089431 trunc from 0x01fe9e to 0x0288fd (0x8a60 bytes) 000855 1789997291.134769 read 0x019ad7 thru 0x01fb7f (0x60a9 bytes) 000856 1789997291.144761 mapwrite 0x038afe thru 0x039308 (0x80b bytes) 000857 1789997291.168344 mapread 0x02e3e1 thru 0x03cfb4 (0xebd4 bytes) 000858 1789997291.198340 trunc from 0x01cd76 to 0x01ec02 (0x1e8d bytes) 000859 1789997291.202395 mapwrite 0x00b2a3 thru 0x0112db (0x6039 bytes) 000860 1789997291.235250 trunc from 0x003c8e to 0x00d07e (0x93f1 bytes) 000861 1789997291.240606 punch from 0x029952 to 0x035c2e (0xc2dd bytes) 000862 1789997291.246198 trunc from 0x03cfb5 to 0x004f9f (0x38015 bytes) 000863 1789997291.291760 mapwrite 0x01fcd6 thru 0x02ba96 (0xbdc1 bytes) 000864 1789997291.356317 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) 000865 1789997291.524619 write 0x01c039 thru 0x01c724 (0x6ec bytes) 000866 1789997291.549055 mapwrite 0x02a7ce thru 0x03295b (0x818e bytes) 000867 1789997291.634238 trunc from 0x03295c to 0x0155fa (0x1d361 bytes) 000868 1789997291.652343 trunc from 0x0023da to 0x00e4b5 (0xc0dc bytes) 000869 1789997291.658122 mapread 0x00503c thru 0x00b722 (0x66e7 bytes) 000870 1789997291.676058 read 0x010275 thru 0x0155fa (0x5386 bytes) 000871 1789997291.684292 mapwrite 0x026915 thru 0x031e15 (0xb501 bytes) 000872 1789997291.729730 MIRROR_EXTEND to 2 mirrors 000873 1789997292.453692 MIRROR_EXTEND to 3 mirrors 000874 1789997293.237776 MIRROR_EXTEND to 4 mirrors 000875 1789997294.079911 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) 000879 1789997294.525898 read 0x00a6e1 thru 0x017c6d (0xd58d bytes) 000880 1789997294.539198 MIRROR_RESYNC 3 mirrors 000881 1789997294.885391 mapread 0x01d1bb thru 0x024279 (0x70bf bytes) 000882 1789997294.912928 mapwrite 0x018f31 thru 0x0257be (0xc88e bytes) 000883 1789997295.059835 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31e16): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e16): Mirror 3: 0x5da39615 (stale) Mirror 4: 0x5da39615 (stale) CRC-32 checksum value for chunk [0, 0x31e16): Mirror 1: 0xf39867bf Mirror 5: 0xf39867bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e16: skipped 000884 1789997296.066983 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x31e16): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e16): Mirror 3: 0x5da39615 (stale) Mirror 4: 0x5da39615 (stale) CRC-32 checksum value for chunk [0, 0x31e16): Mirror 1: 0xf39867bf Mirror 5: 0xf39867bf Mirror 6: 0xf39867bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e16: skipped 000885 1789997297.022728 mapread 0x013d3c thru 0x0170b1 (0x3376 bytes) 000886 1789997297.047899 punch from 0x031d02 to 0x031e15 (0x114 bytes) 000887 1789997297.109328 write 0x01dba6 thru 0x02cd54 (0xf1af bytes) 000888 1789997297.128056 mapread 0x00f7fa thru 0x01a4e6 (0xaced bytes) 000889 1789997297.150221 MIRROR_RESYNC 5 mirrors 000890 1789997298.122005 mapread 0x028ade thru 0x031e15 (0x9338 bytes) 000891 1789997298.170558 mapwrite 0x00c88a thru 0x016bad (0xa324 bytes) 000892 1789997298.322851 trunc from 0x031e16 to 0x005ac7 (0x2c34e bytes) 000893 1789997298.386954 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5ac8): Mirror 3: 0x3a06baeb (stale) Mirror 4: 0x3a06baeb (stale) Mirror 6: 0x3a06baeb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5ac8): Mirror 3: 0x3a06baeb (stale) Mirror 4: 0x3a06baeb (stale) Mirror 6: 0x3a06baeb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5ac8: skipped 000894 1789997298.782987 fallocate 0x00ef78 thru 0x01307c (0x4105 bytes) 000895 1789997298.788394 write 0x00e92d thru 0x01e4f5 (0xfbc9 bytes) 000896 1789997298.803097 read 0x00263c thru 0x005d00 (0x36c5 bytes) 000897 1789997298.811971 write 0x03a777 thru 0x03ffff (0x5889 bytes) 000898 1789997298.814681 fallocate 0x02e6d6 thru 0x038785 (0xa0b0 bytes) 000900 1789997298.819582 read 0x03d76b thru 0x03ffff (0x2895 bytes) 000901 1789997298.820143 punch from 0x0383af to 0x03ffff (0x7c51 bytes) 000902 1789997298.841585 trunc from 0x01e067 to 0x026a07 (0x89a1 bytes) 000903 1789997298.865416 fallocate 0x01bce1 thru 0x01fde6 (0x4106 bytes) 000904 1789997298.870987 read 0x034dbb thru 0x037dc2 (0x3008 bytes) 000905 1789997298.881928 trunc from 0x022eda to 0x02d84a (0xa971 bytes) 000906 1789997298.886420 write 0x02264b thru 0x02d054 (0xaa0a bytes) 000907 1789997298.895520 fallocate 0x0388b8 thru 0x03f939 (0x7082 bytes) 000908 1789997298.902937 trunc from 0x040000 to 0x012870 (0x2d78f bytes) 000910 1789997298.921269 fallocate 0x01a784 thru 0x029797 (0xf014 bytes) 000912 1789997298.931914 read 0x00f787 thru 0x011ece (0x2748 bytes) 000913 1789997298.936066 trunc from 0x029798 to 0x008eb7 (0x208e0 bytes) 000915 1789997298.966337 write 0x02eeb4 thru 0x03294d (0x3a9a bytes) 000916 1789997298.969394 mapwrite 0x013322 thru 0x01ef4a (0xbc29 bytes) 000918 1789997299.011322 punch from 0x027b6c to 0x02ea65 (0x6efa bytes) 000919 1789997299.018005 punch from 0x0016eb to 0x0104a4 (0xedba bytes) 000920 1789997299.024624 trunc from 0x03294e to 0x003ef9 (0x2ea54 bytes) 000921 1789997299.040601 read 0x00203f thru 0x003ef9 (0x1ebb bytes) 000922 1789997299.052101 mapwrite 0x010ae2 thru 0x017880 (0x6d9f bytes) 000923 1789997299.100688 mapwrite 0x026b78 thru 0x02ff09 (0x9392 bytes) 000924 1789997299.143296 mapread 0x00ebdd thru 0x01d49e (0xe8c2 bytes) 000925 1789997299.144715 mapread 0x001c81 thru 0x011130 (0xf4b0 bytes) 000926 1789997299.151357 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ff0a): Mirror 4: 0xcd76f7ee (stale) Mirror 6: 0xcd76f7ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ff0a): Mirror 4: 0xcd76f7ee (stale) Mirror 6: 0xcd76f7ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ff0a: skipped 000927 1789997299.450015 punch from 0x025336 to 0x02ff09 (0xabd4 bytes) 000928 1789997299.458041 read 0x014e6e thru 0x01cc08 (0x7d9b bytes) 000929 1789997299.481011 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ff0a): Mirror 4: 0xcd76f7ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ff0a): Mirror 4: 0xcd76f7ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ff0a: skipped 000930 1789997299.853005 trunc from 0x02ff0a to 0x030108 (0x1ff bytes) 000931 1789997299.877355 mapread 0x006089 thru 0x00b48d (0x5405 bytes) 000932 1789997299.901561 mapread 0x001ecb thru 0x0052d4 (0x340a bytes) 000933 1789997299.902785 trunc from 0x024123 to 0x030108 (0xbfe6 bytes) 000934 1789997299.907360 trunc from 0x030109 to 0x0259a6 (0xa762 bytes) 000935 1789997299.922856 mapwrite 0x018719 thru 0x0204cc (0x7db4 bytes) 000936 1789997299.949278 write 0x02632f thru 0x02fdd5 (0x9aa7 bytes) 000937 1789997299.951688 write 0x011489 thru 0x018cd5 (0x784d bytes) 000938 1789997299.953960 trunc from 0x0284a3 to 0x02fdd5 (0x7933 bytes) 000939 1789997299.973614 trunc from 0x02fdd6 to 0x03c60c (0xc837 bytes) 000940 1789997299.993573 read 0x002196 thru 0x00cbee (0xaa59 bytes) 000942 1789997299.994785 write 0x006b94 thru 0x00e4e8 (0x7955 bytes) 000943 1789997299.996087 write 0x025844 thru 0x02c1b9 (0x6976 bytes) 000945 1789997300.012116 write 0x024415 thru 0x031f81 (0xdb6d bytes) 000946 1789997300.022175 punch from 0x0395f6 to 0x03c60c (0x3017 bytes) 000947 1789997300.027079 trunc from 0x03c60d to 0x0184f6 (0x24116 bytes) 000950 1789997300.047350 fallocate 0x033ece thru 0x035433 (0x1566 bytes) 000951 1789997300.053308 read 0x025e0f thru 0x02bc0c (0x5dfe bytes) 000952 1789997300.067109 mapwrite 0x0101c7 thru 0x01bb40 (0xb97a bytes) 000953 1789997300.085422 mapwrite 0x0186e7 thru 0x022255 (0x9b6f bytes) 000954 1789997300.105037 read 0x02d76d thru 0x0318fc (0x4190 bytes) 000955 1789997300.106303 trunc from 0x035434 to 0x03c320 (0x6eed bytes) 000956 1789997300.122063 write 0x037a82 thru 0x03d321 (0x58a0 bytes) 000957 1789997300.131752 mapread 0x027f66 thru 0x02c364 (0x43ff bytes) 000958 1789997300.134214 read 0x036ebc thru 0x037666 (0x7ab bytes) 000959 1789997300.135325 read 0x009a07 thru 0x00d447 (0x3a41 bytes) 000960 1789997300.136250 punch from 0x02d910 to 0x0317d7 (0x3ec8 bytes) 000961 1789997300.140389 trunc from 0x03d322 to 0x03f8ab (0x258a bytes) 000962 1789997300.162870 trunc from 0x03f8ac to 0x00e9f7 (0x30eb4 bytes) 000963 1789997300.192757 fallocate 0x03e0eb thru 0x03ffff (0x1f15 bytes) 000964 1789997300.199399 mapread 0x00b259 thru 0x011076 (0x5e1e bytes) 000965 1789997300.231394 trunc from 0x016015 to 0x01d0cc (0x70b8 bytes) 000966 1789997300.236667 punch from 0x01af37 to 0x0276bf (0xc789 bytes) 000967 1789997300.247207 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) 000969 1789997300.420115 read 0x003b22 thru 0x00ad2b (0x720a bytes) 000970 1789997300.428856 mapwrite 0x01b7af thru 0x02484e (0x90a0 bytes) 000971 1789997300.470569 punch from 0x00a420 to 0x017099 (0xcc7a bytes) 000972 1789997300.476360 trunc from 0x00ecef to 0x01c705 (0xda17 bytes) 000973 1789997300.489350 fallocate 0x0083d8 thru 0x013f14 (0xbb3d bytes) 000974 1789997300.499134 fallocate 0x02f51c thru 0x030fe4 (0x1ac9 bytes) 000975 1789997300.508573 punch from 0x013ece to 0x01e8b8 (0xa9eb bytes) 000976 1789997300.521044 mapwrite 0x03a0cb thru 0x03ffff (0x5f35 bytes) 000979 1789997300.570368 fallocate 0x00164d thru 0x00707e (0x5a32 bytes) 000980 1789997300.576699 write 0x031b31 thru 0x0347b3 (0x2c83 bytes) 000981 1789997300.587270 trunc from 0x01e105 to 0x024ba2 (0x6a9e bytes) 000982 1789997300.593728 trunc from 0x02a17c to 0x02a750 (0x5d5 bytes) 000984 1789997300.599021 mapwrite 0x008c50 thru 0x014c3e (0xbfef bytes) 000985 1789997300.640051 fallocate 0x009462 thru 0x016cb1 (0xd850 bytes) 000987 1789997300.646743 punch from 0x01ec24 to 0x020720 (0x1afd bytes) 000989 1789997300.652348 trunc from 0x00dba8 to 0x00ded6 (0x32f bytes) 000990 1789997300.658165 trunc from 0x040000 to 0x00c78d (0x33872 bytes) 000991 1789997300.684161 read 0x003d1e thru 0x008a81 (0x4d64 bytes) 000992 1789997300.684982 mapwrite 0x029596 thru 0x02fe42 (0x68ad bytes) 000993 1789997300.741917 trunc from 0x02fe43 to 0x02ebb2 (0x1290 bytes) 000995 1789997300.764242 mapread 0x0079a6 thru 0x0085b3 (0xc0e bytes) 000996 1789997300.765158 trunc from 0x02ebb3 to 0x038e29 (0xa277 bytes) 000997 1789997300.777219 mapread 0x01ace5 thru 0x026f19 (0xc235 bytes) 000998 1789997300.778270 read 0x01a332 thru 0x01b6f5 (0x13c4 bytes) 000999 1789997300.779980 trunc from 0x01529a to 0x017414 (0x217b bytes) All operations completed A-OK! PASS 70a (87s) == sanity-flr test 71: check mirror extend parent fid ==== 09:28:25 (1789997305) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x1c4:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 72: check append on FLR file ========== 09:28:33 (1789997313) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x155:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.204755 s, 10.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x155:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082835 s, 12.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x156:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x155:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x156:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x155:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 09:28:40 (1789997320) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6553408 .. Extending file size to 7077152 .. Extending file size to 6775392 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7365632 .. Extending file size to 6983264 .. Extending file size to 9016768 .. Extending file size to 7657280 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7966016 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8121440 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8299616 .. Extending file size to 8552736 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 62140 62141 62142 62143 62146 192.168.203.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre2 (opts:) 192.168.203.111@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:29:59 (1789997399) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5855776 .. Extending file size to 3462848 .. failed Extending file size to 8233152 .. Extending file size to 6059040 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8526016 .. Extending file size to 9040640 .. Extending file size to 8787456 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9160384 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 63417 63418 63419 63421 63422 failed 192.168.203.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre2 (opts:) 192.168.203.111@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:31:15 (1789997475) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2352 1268952 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2100 1269204 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24720 3582300 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18824 3588196 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43544 7170496 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2352 1268952 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2100 1269204 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24720 3582300 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18824 3588196 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43544 7170496 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0948539 s, 22.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174822 s, 12.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x1b868aec (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:31:32 (1789997492) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0961288 s, 21.8 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 09:31:39 (1789997499) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xef:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x16c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x16c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x16d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x16c:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:31:45 (1789997505) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0915618 s, 22.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:31:50 (1789997510) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0786427 s, 26.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x16e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x16f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x16d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x16e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x16f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x16d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x16e:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:31:56 (1789997516) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0998068 s, 21.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x170:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x16f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x170:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x16f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x170:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:32:26 (1789997546) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg311-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469007 s, 22.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x251:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x250:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x251:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x250:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24d:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:33:09 (1789997589) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg311-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x171:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x172:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.087158 s, 24.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x171:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x172:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x171:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x171:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x172:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x171:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:33:54 (1789997634) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg311-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg311-server: Pool lustre.test_204f created Adding targets to pool oleg311-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg311-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0803277 s, 26.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x173:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x172:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x173:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x172:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg311-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg311-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg311-server: Pool lustre.test_204f destroyed PASS 204f (71s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:35:05 (1789997705) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:35:11 (1789997711) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:35:19 (1789997719) oleg311-server: Pool lustre.test_206 created Adding targets to pool oleg311-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg311-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg311-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg311-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg311-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:35:43 (1789997743) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619908 s, 16.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434602 s, 24.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xea2473f0 Mirror 3: 0xea2473f0 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:35:54 (1789997754) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:35:57 (1789997757) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:36:01 (1789997761) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0445044 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1fc0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x12f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x12f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x12f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x9a0000 Read this (4 bytes): BBBB 192.168.203.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre2 (opts:) PASS 209a (31s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:36:32 (1789997792) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0170058 s, 482 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:36:38 (1789997798) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431209 s, 24.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x110 lmm_fid: [0x200000405:0x110:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x110 lmm_fid: [0x200000405:0x110:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x17d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611408 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-21 09:36:40.000000000 -0400 Modify: 2026-09-21 09:36:40.000000000 -0400 Change: 2026-09-21 09:36:41.000000000 -0400 Birth: 2026-09-21 09:36:39.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:36:45 (1789997805) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275433 s, 38.1 MB/s oleg311-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg311-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg311-server: O/280000401/d29/17d: File not found by ext2_lookup PASS 210b (18s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:37:03 (1789997823) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.186502 s, 220 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0191692 s, 214 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 21 09:37 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x115:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 21 09:37 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:37:10 (1789997830) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:37:16 (1789997836) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1733 sec =========== 09:37:23 (1789997843) === sanity-flr: start cleanup 09:37:24 (1789997844) === === sanity-flr: finish cleanup 09:37:32 (1789997852) ===