-----============= acceptance-small: sanity-flr ============----- Tue May 12 15:21:52 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg431-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 15:22:14 (1778613734) === oleg431-client.virtnet: executing check_config_client /mnt/lustre oleg431-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg431-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b6d8515a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b6d8515a800.idle_timeout=debug disable quota as required oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 15:22:55 (1778613775) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8200] == sanity-flr test 0a: lfs mirror create with -N option == 15:22:58 (1778613778) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:23:17 (1778613797) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:23:21 (1778613801) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:23:25 (1778613805) PASS 0d (37s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:24:01 (1778613841) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:24:05 (1778613845) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 15:24:09 (1778613849) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b6d8515a800.stats=clear osc.lustre-OST0001-osc-ffff9b6d8515a800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:24:16 (1778613856) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:24:26 (1778613866) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:24:34 (1778613874) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:24:37 (1778613877) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:24:45 (1778613885) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 15:24:55 (1778613895) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:25:02 (1778613902) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:25:11 (1778613911) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:25:19 (1778613919) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:25:30 (1778613930) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:25:37 (1778613937) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.637131 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477883 s, 21.9 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:25:47 (1778613947) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.566981 s, 35.1 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:25:56 (1778613956) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:26:07 (1778613967) Stopping /mnt/lustre-ost1 (opts:) on oleg431-server oleg431-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b6d8515a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b6d8515a800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg431-server: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg431-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg431-server oleg431-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b6d8515a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b6d8515a800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg431-server: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg431-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (79s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:27:26 (1778614046) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.205884 s, 15.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg431-server oleg431-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b6d8515a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b6d8515a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg431-server: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg431-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:28:12 (1778614092) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg431-server oleg431-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b6d8515a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b6d8515a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg431-server: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg431-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg431-server oleg431-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b6d8515a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b6d8515a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg431-server: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg431-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:29:34 (1778614174) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.56446 s, 14.7 MB/s -rw-r--r-- 1 root root 52428800 May 12 15:29 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.036s user 0m0.001s sys 0m0.188s Stopping /mnt/lustre-ost1 (opts:) on oleg431-server oleg431-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b6d8515a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b6d8515a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.615s user 0m0.001s sys 0m0.173s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg431-server: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg431-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg431-server oleg431-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b6d8515a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b6d8515a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.706s user 0m0.001s sys 0m0.165s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg431-server: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg431-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b6d8515a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b6d8515a800.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:31:04 (1778614264) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:31:08 (1778614268) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:31:11 (1778614271) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 15:31:15 (1778614275) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 15:31:25 (1778614285) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.323594 s, 38.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:31:40 (1778614300) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373559 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308605 s, 34.0 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:31:48 (1778614308) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:31:58 (1778614318) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00116424 s, 880 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00117771 s, 869 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00082075 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00134573 s, 761 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 15:32:11 (1778614331) create test files with size 15759429 .. 7889901534fffb7ed1836430ac3a1f20 - a0ec4dc42a1b2e0bc5a7c814f0c4cefb - 9456c03a166de5c5d230b80d0c3216db - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 15:32:39 (1778614359) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:32:56 (1778614376) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:33:05 (1778614385) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136528 s, 7.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523788 s, 20.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 15:33:16 (1778614396) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0772816 s, 27.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.330639 s, 12.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224885 s, 9.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 15:33:35 (1778614415) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 15:33:39 (1778614419) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:33:43 (1778614423) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682983 s, 15.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 15:33:52 (1778614432) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.241181 s, 13.0 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:34:04 (1778614444) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (12s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:34:18 (1778614458) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.413248 s, 25.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:34:32 (1778614472) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9b6d9854c000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.439892 s, 23.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg431-client.virtnet /mnt/lustre (opts:) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9b6d8f161000.xattr_cache=1 PASS 44e (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:34:52 (1778614492) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:35:39 (1778614539) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:36:21 (1778614581) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 15:36:25 (1778614585) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.40833 s, 7.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.231492 s, 13.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x3edaa732 (stale) Mirror 2: 0x3edaa732 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 8541cc8652ba5fd40ba45efd8ff4693b - mirror 1 checksum: c14e8c711a0223dabc53c966d3cac4ec - mirror 2 checksum: 8541cc8652ba5fd40ba45efd8ff4693b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue May 12 15:36:29 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue May 12 15:36:29 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8541cc8652ba5fd40ba45efd8ff4693b - mirror 1 checksum: c14e8c711a0223dabc53c966d3cac4ec - mirror 2 checksum: c14e8c711a0223dabc53c966d3cac4ec - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue May 12 15:36:29 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue May 12 15:36:29 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 15:36:38 (1778614598) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03204 s, 32.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:36:46 (1778614606) Failing mds1 on oleg431-server Stopping /mnt/lustre-mds1 (opts:) on oleg431-server 15:36:59 (1778614619) shut down facet: mds1 facet_host: oleg431-server facet_failover_host: oleg431-server Failover mds1 to oleg431-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 15:37:36 (1778614656) targets are mounted 15:37:36 (1778614656) facet_failover done oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198924 s, 52.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:37:53 (1778614673) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.077467 s, 264 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.045495 s, 450 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (13s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:38:05 (1778614685) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0610634 s, 335 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.011073 s, 1.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:38:15 (1778614695) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00735874 s, 2.2 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:38:24 (1778614704) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0609333 s, 336 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:38:34 (1778614714) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0268291 s, 153 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0557274 s, 73.5 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:38:43 (1778614723) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:39:09 (1778614749) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:39:41 (1778614781) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:40:05 (1778614805) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:40:31 (1778614831) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 15:40:39 (1778614839) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57174 mirror split pid 57176 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57174 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57176 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 15:41:48 (1778614908) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 4910 skipping zero size read 000003 1778614911.088711 write 0x03b120 thru 0x03ffff (0x4ee0 bytes) fallocating to largest ever: 0x1f6ff 000004 1778614911.091241 fallocate 0x01c47d thru 0x01f6fe (0x3282 bytes) 000006 1778614911.098937 fallocate 0x010ece thru 0x016fc1 (0x60f4 bytes) 000007 1778614911.104336 punch from 0x030b65 to 0x035cb4 (0x5150 bytes) 000008 1778614911.110233 mapread 0x00505f thru 0x00804d (0x2fef bytes) 000009 1778614911.140906 punch from 0x003317 to 0x0112fa (0xdfe4 bytes) 000010 1778614911.146949 trunc from 0x040000 to 0x01a5c2 (0x25a3d bytes) 000011 1778614911.175257 mapread 0x015da7 thru 0x01a5c2 (0x481c bytes) 000012 1778614911.189360 read 0x00298e thru 0x00ad9e (0x8411 bytes) truncating to largest ever: 0x3719e 000013 1778614911.208059 trunc from 0x01a5c3 to 0x03719d (0x1cbdb bytes) 000014 1778614911.235462 write 0x02d03c thru 0x033c72 (0x6c37 bytes) 000015 1778614911.247750 write 0x037cf0 thru 0x03eeef (0x7200 bytes) 000016 1778614911.256607 fallocate 0x008611 thru 0x011817 (0x9207 bytes) 000018 1778614911.264294 read 0x00d868 thru 0x018291 (0xaa2a bytes) zero_range to largest ever: 0x38a04 000019 1778614911.278529 trunc from 0x03853e to 0x038a03 (0x4c6 bytes) 000020 1778614911.299974 mapwrite 0x02f2ee thru 0x038574 (0x9287 bytes) 000021 1778614911.329160 punch from 0x00acf4 to 0x017d41 (0xd04e bytes) 000023 1778614911.339289 punch from 0x033a1f to 0x037b90 (0x4172 bytes) 000024 1778614911.346246 trunc from 0x03eef0 to 0x00ef49 (0x2ffa6 bytes) 000025 1778614911.366650 trunc from 0x00ef4a to 0x00961a (0x592f bytes) 000026 1778614911.386184 trunc from 0x00961b to 0x000791 (0x8e89 bytes) 000027 1778614911.405772 fallocate 0x00ebea thru 0x01bcbe (0xd0d5 bytes) 000029 1778614911.417932 write 0x0086d4 thru 0x008909 (0x236 bytes) fallocating to largest ever: 0x40000 000031 1778614911.424170 fallocate 0x03e037 thru 0x03ffff (0x1fc9 bytes) 000032 1778614911.430834 mapwrite 0x0084d1 thru 0x0106c8 (0x81f8 bytes) 000033 1778614911.516455 punch from 0x0061b9 to 0x006599 (0x3e1 bytes) 000034 1778614911.522646 read 0x004af5 thru 0x007dd8 (0x32e4 bytes) 000035 1778614911.535181 trunc from 0x00c0ef to 0x0106c8 (0x45da bytes) 000036 1778614911.541990 mapread 0x00b19c thru 0x0106c8 (0x552d bytes) 000038 1778614911.579703 MIRROR_EXTEND to 2 mirrors 000039 1778614912.525336 punch from 0x004b52 to 0x0106c8 (0xbb77 bytes) 000040 1778614912.601133 trunc from 0x00fd98 to 0x0106c8 (0x931 bytes) 000042 1778614912.610232 trunc from 0x009ea1 to 0x0106c8 (0x6828 bytes) 000043 1778614912.620706 mapread 0x003b4d thru 0x008036 (0x44ea bytes) 000044 1778614912.666890 trunc from 0x0106c9 to 0x007e3e (0x888a bytes) 000045 1778614912.733800 trunc from 0x005a98 to 0x007e3e (0x23a7 bytes) 000046 1778614912.747271 trunc from 0x007e3f to 0x018047 (0x10209 bytes) 000047 1778614912.776740 trunc from 0x004be6 to 0x0133ff (0xe81a bytes) 000048 1778614912.782552 trunc from 0x018048 to 0x01767d (0x9ca bytes) 000049 1778614912.817357 mapwrite 0x017a5a thru 0x018fb5 (0x155c bytes) 000050 1778614912.896254 mapwrite 0x00b499 thru 0x00c7c2 (0x132a bytes) 000051 1778614912.939189 mapwrite 0x03ca84 thru 0x03ffff (0x357c bytes) 000052 1778614913.044652 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) 000053 1778614913.288506 mapread 0x0059d5 thru 0x00ec26 (0x9252 bytes) 000054 1778614913.309588 read 0x02d922 thru 0x035d75 (0x8454 bytes) 000055 1778614913.323442 MIRROR_EXTEND to 2 mirrors 000056 1778614914.301188 read 0x0350d6 thru 0x03b93a (0x6865 bytes) 000060 1778614914.326874 trunc from 0x040000 to 0x0352cf (0xad30 bytes) 000061 1778614914.424879 mapwrite 0x00e50e thru 0x016fb0 (0x8aa3 bytes) 000062 1778614914.485475 punch from 0x007789 to 0x012561 (0xadd9 bytes) 000063 1778614914.491222 read 0x017403 thru 0x0257d7 (0xe3d5 bytes) 000064 1778614914.505453 read 0x030c7d thru 0x0352cf (0x4653 bytes) 000065 1778614914.506181 trunc from 0x0352d0 to 0x034d9f (0x530 bytes) 000066 1778614914.525818 punch from 0x012107 to 0x01d8a2 (0xb79c bytes) 000067 1778614914.532784 read 0x032a55 thru 0x034d9f (0x234b bytes) 000068 1778614914.533505 mapwrite 0x019456 thru 0x024f38 (0xbae3 bytes) 000070 1778614914.577022 trunc from 0x034da0 to 0x018b68 (0x1c237 bytes) 000072 1778614914.601201 read 0x00bca7 thru 0x018b68 (0xcec2 bytes) 000073 1778614914.602049 mapread 0x0022d0 thru 0x0102e7 (0xe018 bytes) 000074 1778614914.604538 punch from 0x007206 to 0x0163dc (0xf1d7 bytes) 000075 1778614914.610043 punch from 0x000e61 to 0x001b2e (0xcce bytes) 000076 1778614914.615646 mapread 0x00841e thru 0x014559 (0xc13c bytes) 000077 1778614914.627621 punch from 0x00231f to 0x00b808 (0x94ea bytes) 000078 1778614914.632168 mapwrite 0x02c4b0 thru 0x0364a4 (0x9ff5 bytes) 000079 1778614914.692624 mapread 0x016d05 thru 0x01bb3f (0x4e3b bytes) 000080 1778614914.693834 write 0x0093c2 thru 0x012dd9 (0x9a18 bytes) 000082 1778614914.707820 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x364a5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364a5): Mirror 2: 0x4c563366 (stale) CRC-32 checksum value for chunk [0, 0x364a5): Mirror 1: 0x2af5cf9a Mirror 3: 0x2af5cf9a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364a5: skipped 000083 1778614916.122457 trunc from 0x0364a5 to 0x0204d6 (0x15fce bytes) 000084 1778614916.198520 mapread 0x0019fb thru 0x0070ff (0x5705 bytes) 000085 1778614916.237222 punch from 0x008983 to 0x015b1a (0xd198 bytes) 000087 1778614916.245634 fallocate 0x03602e thru 0x03c30f (0x62e2 bytes) 000088 1778614916.269259 mapwrite 0x02362a thru 0x0288a8 (0x527f bytes) 000089 1778614916.368681 read 0x024434 thru 0x0288a8 (0x4475 bytes) 000090 1778614916.372545 punch from 0x021530 to 0x0288a8 (0x7379 bytes) 000091 1778614916.395706 mapread 0x01bfe1 thru 0x0288a8 (0xc8c8 bytes) 000092 1778614916.477619 read 0x022331 thru 0x022d78 (0xa48 bytes) 000093 1778614916.478433 read 0x01b769 thru 0x01c8a5 (0x113d bytes) 000094 1778614916.479799 trunc from 0x0176a8 to 0x023c94 (0xc5ed bytes) 000095 1778614916.492354 read 0x024df0 thru 0x0288a8 (0x3ab9 bytes) 000097 1778614916.493568 punch from 0x0232e8 to 0x02726f (0x3f88 bytes) 000098 1778614916.527558 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x288a9): Mirror 3: 0x902c42e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x288a9): Mirror 3: 0x902c42e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x288a9: skipped 000099 1778614916.851193 trunc from 0x0288a9 to 0x004f89 (0x2391f bytes) 000100 1778614916.879575 fallocate 0x0148b2 thru 0x01f7dd (0xaf2c bytes) 000101 1778614916.885990 trunc from 0x00d157 to 0x019ad9 (0xc983 bytes) 000102 1778614916.892564 read 0x01dcdd thru 0x01f7dd (0x1b01 bytes) 000103 1778614916.900540 punch from 0x017abd to 0x01d67f (0x5bc3 bytes) 000104 1778614916.908462 punch from 0x01c172 to 0x01f7dd (0x366c bytes) 000105 1778614916.915070 trunc from 0x01bc6e to 0x01f7dd (0x3b70 bytes) 000106 1778614916.919692 punch from 0x00fc7b to 0x0137e1 (0x3b67 bytes) 000107 1778614916.924958 trunc from 0x011ef8 to 0x017cf7 (0x5e00 bytes) 000108 1778614916.929277 trunc from 0x01f7de to 0x019fa6 (0x5837 bytes) 000109 1778614916.953838 MIRROR_RESYNC 2 mirrors 000111 1778614918.070251 trunc from 0x019fa7 to 0x03c094 (0x220ee bytes) 000112 1778614918.171597 trunc from 0x03c095 to 0x0269ab (0x156e9 bytes) 000113 1778614918.215914 mapwrite 0x033b03 thru 0x03779a (0x3c98 bytes) 000114 1778614918.332263 trunc from 0x03779b to 0x0098f3 (0x2dea7 bytes) 000115 1778614918.371539 punch from 0x008e79 to 0x0098f3 (0xa7b bytes) 000116 1778614918.378035 write 0x00e0b4 thru 0x010b32 (0x2a7f bytes) 000117 1778614918.380050 mapwrite 0x038395 thru 0x03dea0 (0x5b0c bytes) 000119 1778614918.447584 trunc from 0x027fc4 to 0x0342c3 (0xc300 bytes) 000120 1778614918.457179 read 0x018b73 thru 0x025e0d (0xd29b bytes) 000121 1778614918.484964 mapread 0x00effc thru 0x018a7b (0x9a80 bytes) 000122 1778614918.534040 punch from 0x010182 to 0x01b78f (0xb60e bytes) 000124 1778614918.548026 trunc from 0x03dea1 to 0x03854e (0x5952 bytes) 000125 1778614918.582074 trunc from 0x03854f to 0x03e7f4 (0x62a6 bytes) 000126 1778614918.611073 read 0x027eb3 thru 0x02be16 (0x3f64 bytes) 000127 1778614918.635968 read 0x01c6ba thru 0x020e1b (0x4762 bytes) 000128 1778614918.643882 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e7f5): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e7f5): Mirror 3: 0xdf4f6b4b (stale) CRC-32 checksum value for chunk [0, 0x3e7f5): Mirror 1: 0x807797c2 Mirror 4: 0x807797c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e7f5: skipped 000129 1778614920.267424 punch from 0x02e170 to 0x03c79e (0xe62f bytes) 000131 1778614920.407944 mapread 0x01c55f thru 0x02a792 (0xe234 bytes) 000133 1778614920.499006 trunc from 0x006a98 to 0x0106c8 (0x9c31 bytes) 000134 1778614920.517683 mapwrite 0x02659f thru 0x02f89a (0x92fc bytes) 000135 1778614920.570879 mapread 0x02bcea thru 0x02d0b4 (0x13cb bytes) 000136 1778614920.574361 fallocate 0x034b52 thru 0x03c812 (0x7cc1 bytes) 000138 1778614920.591349 punch from 0x037848 to 0x039286 (0x1a3f bytes) 000139 1778614920.613832 read 0x026312 thru 0x02878d (0x247c bytes) 000140 1778614920.614512 punch from 0x038a36 to 0x03e587 (0x5b52 bytes) 000141 1778614920.647997 trunc from 0x03e7f5 to 0x031a9c (0xcd58 bytes) 000142 1778614920.720782 trunc from 0x031a9d to 0x01881c (0x19280 bytes) 000143 1778614920.756048 MIRROR_RESYNC 3 mirrors 000144 1778614922.126599 MIRROR_RESYNC 3 mirrors 000145 1778614922.640934 MIRROR_RESYNC 3 mirrors 000147 1778614922.894520 read 0x00b7b2 thru 0x0137f8 (0x8047 bytes) 000148 1778614922.908589 punch from 0x00bc4d to 0x01805c (0xc410 bytes) 000149 1778614922.934826 mapread 0x014a31 thru 0x01881c (0x3dec bytes) 000150 1778614922.961501 write 0x024e29 thru 0x02e510 (0x96e8 bytes) 000151 1778614922.969393 fallocate 0x0299bc thru 0x0333be (0x9a03 bytes) 000152 1778614922.976822 read 0x029e66 thru 0x02cfc5 (0x3160 bytes) 000153 1778614922.977670 punch from 0x009ffc to 0x014d09 (0xad0e bytes) 000155 1778614922.989427 write 0x01e366 thru 0x021fd5 (0x3c70 bytes) 000156 1778614923.019199 read 0x0163f6 thru 0x02166c (0xb277 bytes) 000157 1778614923.019899 read 0x01259b thru 0x01ef2c (0xc992 bytes) 000159 1778614923.021135 mapwrite 0x01a198 thru 0x029cbc (0xfb25 bytes) 000160 1778614923.065285 fallocate 0x0322a1 thru 0x03591b (0x367b bytes) 000161 1778614923.079757 read 0x005af1 thru 0x012570 (0xca80 bytes) 000163 1778614923.083789 read 0x024cc7 thru 0x033666 (0xe9a0 bytes) 000164 1778614923.084842 trunc from 0x03591c to 0x0149e8 (0x20f33 bytes) 000165 1778614923.131141 trunc from 0x0149e9 to 0x030375 (0x1b98d bytes) 000166 1778614923.153338 fallocate 0x00c978 thru 0x0132c9 (0x6952 bytes) 000168 1778614923.160903 fallocate 0x018fb4 thru 0x01b560 (0x25ad bytes) 000169 1778614923.166859 read 0x013297 thru 0x01409b (0xe05 bytes) 000170 1778614923.168425 trunc from 0x030376 to 0x025ac0 (0xa8b5 bytes) 000172 1778614923.193405 MIRROR_RESYNC 3 mirrors 000173 1778614924.120811 fallocate 0x00c05c thru 0x016974 (0xa919 bytes) 000174 1778614924.166661 write 0x0391d7 thru 0x03ffff (0x6e29 bytes) 000175 1778614924.171462 fallocate 0x02a007 thru 0x02ad02 (0xcfc bytes) 000176 1778614924.183519 read 0x03951f thru 0x03ffff (0x6ae1 bytes) 000177 1778614924.184969 punch from 0x00fe5a to 0x01c2c4 (0xc46b bytes) 000179 1778614924.191431 mapread 0x00729e thru 0x00963a (0x239d bytes) 000180 1778614924.239706 punch from 0x023446 to 0x02c7a2 (0x935d bytes) 000181 1778614924.247001 write 0x027c6b thru 0x0285a6 (0x93c bytes) 000182 1778614924.278930 trunc from 0x02cb47 to 0x03a8ee (0xdda8 bytes) 000183 1778614924.303221 trunc from 0x040000 to 0x0103f1 (0x2fc0e bytes) 000185 1778614924.333535 trunc from 0x0103f2 to 0x00f174 (0x127d bytes) 000188 1778614924.362961 punch from 0x00c303 to 0x00f174 (0x2e72 bytes) 000189 1778614924.368380 trunc from 0x007093 to 0x0088b0 (0x181e bytes) 000190 1778614924.378935 mapwrite 0x02afbd thru 0x02d040 (0x2084 bytes) 000191 1778614924.521941 MIRROR_RESYNC 3 mirrors 000192 1778614925.439904 trunc from 0x02831d to 0x02c8a1 (0x4585 bytes) 000193 1778614925.478665 punch from 0x00faad to 0x013b3e (0x4092 bytes) 000194 1778614925.494298 mapwrite 0x008c22 thru 0x010de5 (0x81c4 bytes) 000195 1778614925.597811 write 0x00ddc3 thru 0x017b25 (0x9d63 bytes) 000196 1778614925.606537 mapwrite 0x0164c0 thru 0x01b944 (0x5485 bytes) 000197 1778614925.693586 MIRROR_RESYNC 3 mirrors 000198 1778614926.671023 read 0x02a434 thru 0x02d040 (0x2c0d bytes) 000200 1778614926.678056 fallocate 0x02f340 thru 0x03920c (0x9ecd bytes) 000201 1778614926.709424 mapread 0x014ce0 thru 0x01f03f (0xa360 bytes) 000202 1778614926.762068 read 0x010db1 thru 0x01ffc7 (0xf217 bytes) 000203 1778614926.763074 read 0x0142df thru 0x01c287 (0x7fa9 bytes) 000204 1778614926.764676 fallocate 0x030dc0 thru 0x03579c (0x49dd bytes) 000206 1778614926.779460 write 0x014365 thru 0x01c3e2 (0x807e bytes) 000207 1778614926.780799 trunc from 0x003e0c to 0x00ac44 (0x6e39 bytes) 000208 1778614926.792001 read 0x022346 thru 0x02803f (0x5cfa bytes) 000210 1778614926.815334 write 0x036d87 thru 0x03af04 (0x417e bytes) 000211 1778614926.816574 trunc from 0x03af05 to 0x0386cc (0x2838 bytes) 000212 1778614926.860962 trunc from 0x0386cd to 0x0164a6 (0x22226 bytes) 000213 1778614926.887230 mapread 0x000e0c thru 0x00e751 (0xd946 bytes) 000214 1778614926.888713 write 0x03288c thru 0x032d83 (0x4f8 bytes) 000215 1778614926.893433 trunc from 0x002fc1 to 0x00602c (0x306c bytes) 000216 1778614926.898793 trunc from 0x032d84 to 0x01625c (0x1cb27 bytes) 000217 1778614926.964325 fallocate 0x024d87 thru 0x02ed8e (0xa008 bytes) 000219 1778614926.971947 read 0x00f8ab thru 0x01e43c (0xeb92 bytes) 000220 1778614926.991975 mapread 0x0255f2 thru 0x02b1ca (0x5bd9 bytes) 000222 1778614927.016537 MIRROR_RESYNC 3 mirrors 000224 1778614928.254942 fallocate 0x035ea2 thru 0x037de4 (0x1f43 bytes) 000225 1778614928.326212 fallocate 0x0037f5 thru 0x00598d (0x2199 bytes) 000226 1778614928.343047 write 0x01d032 thru 0x01e7cb (0x179a bytes) 000227 1778614928.385673 fallocate 0x00f610 thru 0x014f65 (0x5956 bytes) 000228 1778614928.396430 punch from 0x02c234 to 0x037de4 (0xbbb1 bytes) 000229 1778614928.408297 trunc from 0x037de5 to 0x011a1a (0x263ca bytes) 000230 1778614928.467519 trunc from 0x007e8e to 0x00a633 (0x27a6 bytes) 000231 1778614928.479851 punch from 0x006823 to 0x00faec (0x92ca bytes) 000232 1778614928.508368 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x11a1b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11a1b): Mirror 3: 0xf95c80d0 (stale) Mirror 4: 0xf95c80d0 (stale) CRC-32 checksum value for chunk [0, 0x11a1b): Mirror 1: 0x9233f916 Mirror 5: 0x9233f916 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11a1b: skipped 000233 1778614929.685795 punch from 0x009d7b to 0x011a1a (0x7ca0 bytes) 000234 1778614929.754395 punch from 0x001931 to 0x010f3b (0xf60b bytes) 000235 1778614929.759918 trunc from 0x002d29 to 0x011a1a (0xecf2 bytes) 000236 1778614929.768210 mapread 0x001404 thru 0x0061a9 (0x4da6 bytes) 000237 1778614929.819776 mapwrite 0x01b3e2 thru 0x02a67f (0xf29e bytes) 000238 1778614929.923864 mapread 0x0170f2 thru 0x0244bb (0xd3ca bytes) 000239 1778614929.926671 trunc from 0x0198e5 to 0x023578 (0x9c94 bytes) 000240 1778614929.931767 write 0x03aecb thru 0x03ffff (0x5135 bytes) 000241 1778614929.933890 mapread 0x013cbf thru 0x01ce6f (0x91b1 bytes) 000242 1778614929.957888 trunc from 0x040000 to 0x02cf8c (0x13073 bytes) 000243 1778614929.976322 mapwrite 0x0397f0 thru 0x03f01d (0x582e bytes) 000244 1778614930.070448 mapwrite 0x01b3ae thru 0x028cbb (0xd90e bytes) 000245 1778614930.097914 write 0x023c9d thru 0x02894f (0x4cb3 bytes) 000246 1778614930.099096 trunc from 0x02d4d7 to 0x034e0e (0x7938 bytes) 000248 1778614930.110721 trunc from 0x03f01e to 0x03ad10 (0x430d bytes) 000249 1778614930.132250 write 0x019e2e thru 0x029d52 (0xff25 bytes) 000250 1778614930.136262 trunc from 0x011806 to 0x01c57c (0xad77 bytes) 000251 1778614930.157361 read 0x02c348 thru 0x02cffe (0xcb7 bytes) 000252 1778614930.168161 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ad11): Mirror 3: 0xa434707d (stale) Mirror 5: 0xda01fc0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad11): Mirror 3: 0xa434707d (stale) Mirror 5: 0xda01fc0c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad11: skipped 000253 1778614930.673836 read 0x034506 thru 0x03ad10 (0x680b bytes) 000254 1778614930.699001 write 0x02b22d thru 0x03821c (0xcff0 bytes) 000255 1778614930.721311 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ad11): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad11): Mirror 3: 0xa434707d (stale) Mirror 5: 0xda01fc0c (stale) CRC-32 checksum value for chunk [0, 0x3ad11): Mirror 1: 0x9b1fc682 Mirror 6: 0x9b1fc682 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad11: skipped 000256 1778614932.397814 mapwrite 0x005f97 thru 0x011596 (0xb600 bytes) 000257 1778614932.561178 mapwrite 0x03eb6a thru 0x03ffff (0x1496 bytes) 000258 1778614932.639578 fallocate 0x00a611 thru 0x00f998 (0x5388 bytes) 000259 1778614932.654517 trunc from 0x040000 to 0x01b1fa (0x24e05 bytes) 000260 1778614932.703966 mapread 0x00606a thru 0x0139e1 (0xd978 bytes) 000261 1778614932.710561 write 0x03f3c6 thru 0x03ffff (0xc3a bytes) 000262 1778614932.713694 read 0x01b899 thru 0x01e1e4 (0x294c bytes) 000263 1778614932.724710 read 0x010d20 thru 0x01281d (0x1afe bytes) 000264 1778614932.726001 read 0x011e67 thru 0x01ef10 (0xd0aa bytes) 000265 1778614932.746646 mapread 0x0000a9 thru 0x00d737 (0xd68f bytes) 000266 1778614932.750693 punch from 0x03a5b5 to 0x03b733 (0x117f bytes) 000267 1778614932.764039 fallocate 0x001f89 thru 0x00dbb0 (0xbc28 bytes) 000268 1778614932.780575 fallocate 0x020bcd thru 0x029373 (0x87a7 bytes) 000269 1778614932.788628 mapwrite 0x02570e thru 0x02876a (0x305d bytes) 000270 1778614932.874068 mapread 0x0057df thru 0x008a47 (0x3269 bytes) 000271 1778614932.877790 mapwrite 0x00f4fc thru 0x010629 (0x112e bytes) 000274 1778614932.937275 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x93ace630 (stale) Mirror 6: 0x7ab1099d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x93ace630 (stale) Mirror 6: 0x7ab1099d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000275 1778614933.437577 mapread 0x016c2e thru 0x01f269 (0x863c bytes) 000276 1778614933.467709 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x93ace630 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x93ace630 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000277 1778614933.969958 mapread 0x0092c2 thru 0x017a33 (0xe772 bytes) 000278 1778614934.037243 write 0x0178e7 thru 0x022973 (0xb08d bytes) 000279 1778614934.052637 trunc from 0x027149 to 0x02ab0a (0x39c2 bytes) 000280 1778614934.057683 read 0x027b26 thru 0x031cce (0xa1a9 bytes) 000281 1778614934.066112 write 0x02fba2 thru 0x03e8a3 (0xed02 bytes) 000282 1778614934.075353 write 0x00a71a thru 0x00d9a8 (0x328f bytes) 000283 1778614934.076496 mapread 0x006b7f thru 0x01681b (0xfc9d bytes) 000284 1778614934.080830 mapread 0x0336ae thru 0x03b6e5 (0x8038 bytes) 000285 1778614934.083593 mapread 0x0397ab thru 0x03ce62 (0x36b8 bytes) 000286 1778614934.085728 trunc from 0x040000 to 0x00031d (0x3fce2 bytes) 000288 1778614934.116499 read 0x000004 thru 0x00031d (0x31a bytes) 000289 1778614934.133444 write 0x0086c5 thru 0x015532 (0xce6e bytes) 000290 1778614934.135226 mapwrite 0x014cd6 thru 0x01b208 (0x6533 bytes) 000291 1778614934.181538 read 0x017619 thru 0x01b208 (0x3bf0 bytes) 000292 1778614934.182223 mapread 0x01aa38 thru 0x01b208 (0x7d1 bytes) 000293 1778614934.183311 punch from 0x01625d to 0x01b208 (0x4fac bytes) 000294 1778614934.188442 mapwrite 0x033abe thru 0x03ffff (0xc542 bytes) 000295 1778614934.366637 MIRROR_RESYNC 2 mirrors 000296 1778614935.826005 trunc from 0x040000 to 0x033b7a (0xc485 bytes) 000297 1778614935.910416 fallocate 0x00c23f thru 0x01ab07 (0xe8c9 bytes) 000298 1778614935.919169 write 0x022dae thru 0x025324 (0x2577 bytes) 000299 1778614935.944501 trunc from 0x033b7b to 0x0253ea (0xe790 bytes) 000300 1778614936.000842 mapwrite 0x03ba09 thru 0x03ffff (0x45f7 bytes) 000302 1778614936.077744 fallocate 0x02e911 thru 0x030f59 (0x2649 bytes) 000304 1778614936.084646 trunc from 0x034f50 to 0x03cdcd (0x7e7e bytes) 000305 1778614936.091162 trunc from 0x040000 to 0x029fe1 (0x1601e bytes) 000306 1778614936.105233 read 0x007c58 thru 0x013793 (0xbb3c bytes) 000307 1778614936.126555 MIRROR_RESYNC 2 mirrors 000308 1778614937.329952 mapread 0x00fefa thru 0x0135ea (0x36f1 bytes) 000309 1778614937.375569 punch from 0x001cfe to 0x00f9ba (0xdcbd bytes) 000310 1778614937.437019 trunc from 0x029fe2 to 0x010cc3 (0x1931e bytes) 000312 1778614937.568867 write 0x003b72 thru 0x0058e4 (0x1d73 bytes) 000314 1778614937.588598 punch from 0x004b3e to 0x00e633 (0x9af6 bytes) 000315 1778614937.604921 write 0x0136b8 thru 0x0228ec (0xf235 bytes) 000316 1778614937.613259 mapread 0x00b86a thru 0x014ce2 (0x9479 bytes) 000317 1778614937.640480 mapwrite 0x022094 thru 0x026ff0 (0x4f5d bytes) 000320 1778614937.714839 trunc from 0x026ff1 to 0x02d129 (0x6139 bytes) 000321 1778614937.753734 write 0x0124c0 thru 0x017bdc (0x571d bytes) 000322 1778614937.758156 read 0x018e42 thru 0x022d67 (0x9f26 bytes) 000323 1778614937.760112 trunc from 0x02d12a to 0x00db16 (0x1f613 bytes) 000324 1778614937.783720 punch from 0x0070cb to 0x00c1ef (0x5125 bytes) 000325 1778614937.790050 read 0x0083dd thru 0x00db16 (0x573a bytes) 000326 1778614937.817653 mapread 0x0007d2 thru 0x00472d (0x3f5c bytes) 000327 1778614937.818830 mapread 0x007aa4 thru 0x009096 (0x15f3 bytes) 000328 1778614937.819719 mapread 0x0017da thru 0x005ab7 (0x42de bytes) 000330 1778614937.826606 mapwrite 0x032982 thru 0x03e413 (0xba92 bytes) 000331 1778614937.986615 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e414): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e414): Mirror 5: 0xb072ecb5 (stale) CRC-32 checksum value for chunk [0, 0x3e414): Mirror 1: 0xc521bca5 Mirror 6: 0xc521bca5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e414: skipped 000332 1778614939.248213 fallocate 0x01b52f thru 0x01e9ba (0x348c bytes) 000333 1778614939.350363 mapwrite 0x0372fc thru 0x03b58c (0x4291 bytes) 000334 1778614939.454582 fallocate 0x03ba0d thru 0x03efe5 (0x35d9 bytes) 000335 1778614939.467486 trunc from 0x00eb02 to 0x0153aa (0x68a9 bytes) 000336 1778614939.494398 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3efe6): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3efe6): Mirror 5: 0x71bea326 (stale) Mirror 6: 0x12e48ebf (stale) CRC-32 checksum value for chunk [0, 0x3efe6): Mirror 1: 0xd6e0106 Mirror 7: 0xd6e0106 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3efe6: skipped 000337 1778614940.619797 MIRROR_RESYNC 4 mirrors 000338 1778614941.482601 trunc from 0x03efe6 to 0x00358b (0x3ba5a bytes) 000339 1778614941.543499 mapread 0x000d8d thru 0x00358b (0x27ff bytes) 000340 1778614941.588630 write 0x01e2cf thru 0x023aac (0x57de bytes) 000341 1778614941.590779 write 0x009161 thru 0x00f5f9 (0x6499 bytes) 000342 1778614941.607944 punch from 0x019de6 to 0x01ed40 (0x4f5b bytes) 000343 1778614941.632743 read 0x02236e thru 0x023aac (0x173f bytes) 000344 1778614941.634046 mapread 0x007c67 thru 0x00fc2f (0x7fc9 bytes) 000345 1778614941.640499 punch from 0x0082c8 to 0x00afdf (0x2d18 bytes) 000346 1778614941.652629 read 0x01284a thru 0x01e5a7 (0xbd5e bytes) 000347 1778614941.674216 mapwrite 0x024bef thru 0x02b719 (0x6b2b bytes) 000348 1778614941.724431 write 0x0371d6 thru 0x03ffff (0x8e2a bytes) 000351 1778614941.729138 mapread 0x02d317 thru 0x03ca7f (0xf769 bytes) 000352 1778614941.738733 write 0x026692 thru 0x02ce9b (0x680a bytes) 000353 1778614941.740769 mapwrite 0x005932 thru 0x00efa2 (0x9671 bytes) 000354 1778614941.769442 mapread 0x00b48d thru 0x00de78 (0x29ec bytes) 000355 1778614941.771616 trunc from 0x040000 to 0x00c741 (0x338be bytes) 000356 1778614941.795556 write 0x014ab8 thru 0x01a0bc (0x5605 bytes) 000357 1778614941.804475 MIRROR_RESYNC 4 mirrors 000358 1778614943.163465 write 0x00a99b thru 0x018b15 (0xe17b bytes) 000359 1778614943.215966 read 0x014531 thru 0x01a0bc (0x5b8c bytes) 000361 1778614943.223654 punch from 0x0058fb to 0x014d48 (0xf44e bytes) 000363 1778614943.251492 trunc from 0x01a0bd to 0x022be6 (0x8b2a bytes) 000364 1778614943.279342 punch from 0x010840 to 0x01e085 (0xd846 bytes) 000365 1778614943.289422 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x22be7): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22be7): Mirror 5: 0x7ac8fbb9 (stale) Mirror 6: 0x7ac8fbb9 (stale) Mirror 7: 0x7ac8fbb9 (stale) CRC-32 checksum value for chunk [0, 0x22be7): Mirror 1: 0xe7d659e2 Mirror 8: 0xe7d659e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22be7: skipped 000366 1778614944.346900 trunc from 0x022be7 to 0x02151e (0x16c8 bytes) 000367 1778614944.408919 mapwrite 0x025b38 thru 0x02bbd8 (0x60a1 bytes) 000368 1778614944.469403 write 0x0105a3 thru 0x011ddd (0x183b bytes) 000369 1778614944.483979 punch from 0x026bf7 to 0x02bbd8 (0x4fe2 bytes) 000370 1778614944.492573 read 0x01019f thru 0x01ac3a (0xaa9c bytes) 000371 1778614944.502406 write 0x036337 thru 0x03ffff (0x9cc9 bytes) 000372 1778614944.512247 punch from 0x0125b0 to 0x013288 (0xcd9 bytes) 000373 1778614944.517775 mapread 0x035aad thru 0x03ffff (0xa553 bytes) 000375 1778614944.529939 mapread 0x0116f9 thru 0x01710a (0x5a12 bytes) 000376 1778614944.536552 trunc from 0x040000 to 0x03f645 (0x9ba bytes) 000377 1778614944.553760 write 0x02d20a thru 0x03d157 (0xff4e bytes) 000378 1778614944.565258 mapread 0x012be8 thru 0x019c7c (0x7095 bytes) 000379 1778614944.566492 mapread 0x01c8ba thru 0x02026c (0x39b3 bytes) 000380 1778614944.574195 mapwrite 0x010a32 thru 0x013d9b (0x336a bytes) 000381 1778614944.602509 punch from 0x008cd9 to 0x0096bc (0x9e4 bytes) 000382 1778614944.606890 mapread 0x02cf00 thru 0x037058 (0xa159 bytes) 000383 1778614944.608216 mapread 0x00270b thru 0x00a84f (0x8145 bytes) 000384 1778614944.626934 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f646): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f646): Mirror 5: 0xee33c310 (stale) Mirror 6: 0xee33c310 (stale) Mirror 7: 0xee33c310 (stale) Mirror 8: 0xbde70e8b (stale) CRC-32 checksum value for chunk [0, 0x3f646): Mirror 1: 0x2854bd58 Mirror 9: 0x2854bd58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f646: skipped 000385 1778614946.165549 trunc from 0x035ead to 0x03ce6a (0x6fbe bytes) 000386 1778614946.260259 write 0x03f915 thru 0x03ffff (0x6eb bytes) 000387 1778614946.284049 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbffa307b (stale) Mirror 6: 0xbffa307b (stale) Mirror 7: 0xbffa307b (stale) Mirror 9: 0xe9caa7be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbffa307b (stale) Mirror 6: 0xbffa307b (stale) Mirror 7: 0xbffa307b (stale) Mirror 9: 0xe9caa7be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000388 1778614946.693036 trunc from 0x017485 to 0x018970 (0x14ec bytes) 000389 1778614946.698767 punch from 0x03d664 to 0x03ffff (0x299c bytes) 000390 1778614946.706255 mapread 0x020261 thru 0x02b8ef (0xb68f bytes) 000391 1778614946.774965 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbffa307b (stale) Mirror 6: 0xbffa307b (stale) Mirror 7: 0xbffa307b (stale) Mirror 9: 0xe9caa7be (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3753ccb6 Mirror 10: 0x3753ccb6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000392 1778614948.207566 fallocate 0x03101f thru 0x03bc74 (0xac56 bytes) 000393 1778614948.286855 write 0x0170d8 thru 0x01828a (0x11b3 bytes) 000394 1778614948.309873 write 0x0152ea thru 0x01e17c (0x8e93 bytes) 000395 1778614948.347830 fallocate 0x00108d thru 0x00bf8c (0xaf00 bytes) 000396 1778614948.366418 MIRROR_RESYNC 6 mirrors 000397 1778614952.979603 read 0x0141d6 thru 0x01851a (0x4345 bytes) 000398 1778614953.004894 mapread 0x01fee4 thru 0x02fd75 (0xfe92 bytes) 000399 1778614953.060783 trunc from 0x031560 to 0x03ffff (0xeaa0 bytes) 000400 1778614953.140459 mapwrite 0x030471 thru 0x03b167 (0xacf7 bytes) 000401 1778614953.213069 mapwrite 0x037d38 thru 0x03ffff (0x82c8 bytes) 000402 1778614953.254818 write 0x00acea thru 0x00fafb (0x4e12 bytes) 000403 1778614953.290687 trunc from 0x040000 to 0x03f884 (0x77b bytes) 000404 1778614953.333945 write 0x002180 thru 0x003c61 (0x1ae2 bytes) 000405 1778614953.344502 write 0x0053ea thru 0x007346 (0x1f5d bytes) 000406 1778614953.356317 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f885): Mirror 6: 0xb214aaef (stale) Mirror 7: 0xb214aaef (stale) Mirror 9: 0xb214aaef (stale) Mirror 10: 0xb214aaef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f885): Mirror 6: 0xb214aaef (stale) Mirror 7: 0xb214aaef (stale) Mirror 9: 0xb214aaef (stale) Mirror 10: 0xb214aaef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f885: skipped 000407 1778614953.985172 mapwrite 0x03ee4d thru 0x03ffff (0x11b3 bytes) 000408 1778614954.076699 mapread 0x03191c thru 0x03ffff (0xe6e4 bytes) 000409 1778614954.103919 fallocate 0x00572a thru 0x013f1e (0xe7f5 bytes) 000410 1778614954.115886 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x69f9be1d (stale) Mirror 9: 0x69f9be1d (stale) Mirror 10: 0x69f9be1d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x69f9be1d (stale) Mirror 9: 0x69f9be1d (stale) Mirror 10: 0x69f9be1d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000412 1778614954.696601 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x69f9be1d (stale) Mirror 9: 0x69f9be1d (stale) Mirror 10: 0x69f9be1d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x917396e2 Mirror 11: 0x917396e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000413 1778614956.292474 fallocate 0x001c82 thru 0x00bc3e (0x9fbd bytes) 000414 1778614956.355929 read 0x02e3bb thru 0x0368dd (0x8523 bytes) 000415 1778614956.375051 write 0x03fc73 thru 0x03ffff (0x38d bytes) 000416 1778614956.398014 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x69f9be1d (stale) Mirror 10: 0x69f9be1d (stale) Mirror 11: 0x917396e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x69f9be1d (stale) Mirror 10: 0x69f9be1d (stale) Mirror 11: 0x917396e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000417 1778614956.896247 trunc from 0x02594c to 0x029150 (0x3805 bytes) 000418 1778614956.903327 mapwrite 0x03620d thru 0x03ffff (0x9df3 bytes) 000419 1778614957.010499 write 0x038076 thru 0x03ad7d (0x2d08 bytes) 000420 1778614957.049340 mapread 0x027896 thru 0x02d9dd (0x6148 bytes) 000421 1778614957.085319 read 0x020364 thru 0x0213c8 (0x1065 bytes) 000422 1778614957.095405 MIRROR_RESYNC 4 mirrors 000423 1778614958.976938 trunc from 0x0387da to 0x03ffff (0x7826 bytes) 000424 1778614959.040821 trunc from 0x040000 to 0x03fcee (0x311 bytes) 000425 1778614959.088242 mapread 0x012e89 thru 0x0130bb (0x233 bytes) 000426 1778614959.112975 mapwrite 0x0157d3 thru 0x01d116 (0x7944 bytes) 000427 1778614959.169038 mapwrite 0x019bce thru 0x02957b (0xf9ae bytes) 000428 1778614959.251659 MIRROR_RESYNC 4 mirrors 000429 1778614960.896028 mapread 0x00e008 thru 0x01de31 (0xfe2a bytes) 000430 1778614960.943422 fallocate 0x01f8c2 thru 0x02bf90 (0xc6cf bytes) 000431 1778614960.995942 trunc from 0x03fcef to 0x014ef1 (0x2adfd bytes) 000432 1778614961.014175 punch from 0x003668 to 0x00842f (0x4dc8 bytes) 000433 1778614961.019695 read 0x00c8bf thru 0x0138f8 (0x703a bytes) 000434 1778614961.027506 fallocate 0x0212fc thru 0x02c8a3 (0xb5a8 bytes) 000435 1778614961.031677 mapread 0x00d785 thru 0x016715 (0x8f91 bytes) 000436 1778614961.045639 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c8a4): Mirror 9: 0x68fa93a8 (stale) Mirror 10: 0x68fa93a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c8a4): Mirror 9: 0x68fa93a8 (stale) Mirror 10: 0x68fa93a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c8a4: skipped 000437 1778614961.539713 read 0x015882 thru 0x01b716 (0x5e95 bytes) 000438 1778614961.557481 write 0x00a777 thru 0x017f59 (0xd7e3 bytes) 000440 1778614961.579795 trunc from 0x02c8a4 to 0x019562 (0x13341 bytes) 000441 1778614961.614657 punch from 0x012b0e to 0x012d84 (0x277 bytes) 000442 1778614961.643044 fallocate 0x02a835 thru 0x02dda6 (0x3572 bytes) 000443 1778614961.650026 trunc from 0x021a51 to 0x02757e (0x5b2e bytes) 000445 1778614961.661743 trunc from 0x021c4e to 0x02a294 (0x8647 bytes) 000446 1778614961.668713 mapwrite 0x038b40 thru 0x03ffff (0x74c0 bytes) 000447 1778614961.767419 trunc from 0x01b160 to 0x02894f (0xd7f0 bytes) 000448 1778614961.779901 fallocate 0x03cd75 thru 0x03ffff (0x328b bytes) 000449 1778614961.787461 write 0x03b177 thru 0x03ffff (0x4e89 bytes) 000450 1778614961.789576 punch from 0x03481e to 0x038eb0 (0x4693 bytes) 000451 1778614961.808286 fallocate 0x032919 thru 0x03d110 (0xa7f8 bytes) 000452 1778614961.832301 punch from 0x02a9cd to 0x0303b9 (0x59ed bytes) 000453 1778614961.840608 punch from 0x0079cd to 0x009df9 (0x242d bytes) 000454 1778614961.855791 read 0x03358d thru 0x03ffff (0xca73 bytes) 000455 1778614961.902694 trunc from 0x00a612 to 0x012df8 (0x87e7 bytes) 000456 1778614961.913122 mapwrite 0x02e918 thru 0x035168 (0x6851 bytes) 000457 1778614961.958426 write 0x03d0c1 thru 0x03ffff (0x2f3f bytes) 000458 1778614961.959554 punch from 0x03f072 to 0x03ffff (0xf8e bytes) 000459 1778614961.973282 read 0x025525 thru 0x0306ad (0xb189 bytes) 000460 1778614961.974252 read 0x02a756 thru 0x02ed30 (0x45db bytes) 000461 1778614961.975819 punch from 0x03e6de to 0x03ffff (0x1922 bytes) 000462 1778614961.987496 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xa6d0e204 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xa6d0e204 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000463 1778614962.322702 trunc from 0x040000 to 0x0136fb (0x2c904 bytes) 000464 1778614962.347206 read 0x011472 thru 0x0136fb (0x228a bytes) 000465 1778614962.359193 fallocate 0x00d913 thru 0x0168b1 (0x8f9f bytes) 000466 1778614962.364027 trunc from 0x0168b2 to 0x03d94e (0x2709d bytes) 000467 1778614962.385583 mapread 0x027827 thru 0x036e45 (0xf61f bytes) 000469 1778614962.446279 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) 000472 1778614962.637757 write 0x00bd3f thru 0x015223 (0x94e5 bytes) 000473 1778614962.663063 punch from 0x0167ab to 0x022d30 (0xc586 bytes) 000474 1778614962.677616 MIRROR_EXTEND to 2 mirrors 000475 1778614963.841486 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) 000476 1778614964.019771 fallocate 0x00dd80 thru 0x012729 (0x49aa bytes) 000477 1778614964.034750 read 0x02e362 thru 0x031b92 (0x3831 bytes) 000478 1778614964.044793 punch from 0x022f10 to 0x02fe18 (0xcf09 bytes) 000480 1778614964.052700 trunc from 0x03d94f to 0x02eb60 (0xedee bytes) 000481 1778614964.077215 read 0x0262f5 thru 0x02def8 (0x7c04 bytes) 000482 1778614964.097985 trunc from 0x02eb61 to 0x00bb85 (0x22fdb bytes) 000483 1778614964.125395 mapwrite 0x01e7f7 thru 0x0219fc (0x3206 bytes) 000484 1778614964.231348 mapwrite 0x0085a0 thru 0x00cefa (0x495b bytes) 000486 1778614964.270219 mapwrite 0x00ef3b thru 0x0192ed (0xa3b3 bytes) 000487 1778614964.441136 write 0x005b6d thru 0x0147db (0xec6f bytes) 000488 1778614964.451428 trunc from 0x0219fd to 0x02eddf (0xd3e3 bytes) 000489 1778614964.476256 mapread 0x000ec0 thru 0x002484 (0x15c5 bytes) 000490 1778614964.477454 mapwrite 0x03a660 thru 0x03ffff (0x59a0 bytes) 000491 1778614964.568503 mapwrite 0x003dac thru 0x0065bc (0x2811 bytes) 000492 1778614964.585061 punch from 0x0026f9 to 0x00c236 (0x9b3e bytes) 000493 1778614964.591576 read 0x036939 thru 0x03ffff (0x96c7 bytes) 000494 1778614964.592346 trunc from 0x03528f to 0x03ffff (0xad71 bytes) 000495 1778614964.604799 MIRROR_EXTEND to 2 mirrors 000496 1778614965.695243 punch from 0x02a1a9 to 0x02fd8b (0x5be3 bytes) 000497 1778614965.738797 trunc from 0x034ade to 0x03f0bf (0xa5e2 bytes) 000498 1778614965.744459 trunc from 0x03251e to 0x03cb44 (0xa627 bytes) 000499 1778614965.750446 write 0x01425e thru 0x01e872 (0xa615 bytes) 000500 1778614965.766932 punch from 0x025361 to 0x0330ce (0xdd6e bytes) 000501 1778614965.772140 fallocate 0x00f2db thru 0x018bfa (0x9920 bytes) 000502 1778614965.777420 trunc from 0x040000 to 0x03519a (0xae65 bytes) 000503 1778614965.799658 fallocate 0x01c8e8 thru 0x0277f9 (0xaf12 bytes) 000504 1778614965.805939 trunc from 0x01abf1 to 0x020755 (0x5b65 bytes) 000505 1778614965.827476 write 0x024a58 thru 0x028b37 (0x40e0 bytes) 000506 1778614965.836451 mapread 0x000beb thru 0x00759c (0x69b2 bytes) 000507 1778614965.847244 write 0x014663 thru 0x0237f7 (0xf195 bytes) 000508 1778614965.849838 fallocate 0x012100 thru 0x01d038 (0xaf39 bytes) 000509 1778614965.855833 fallocate 0x034ecd thru 0x03ffff (0xb133 bytes) 000510 1778614965.886137 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2b1002d7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x25f42fbe Mirror 3: 0x25f42fbe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000511 1778614967.095074 read 0x029917 thru 0x032e9f (0x9589 bytes) 000512 1778614967.114805 trunc from 0x0319e2 to 0x03d05a (0xb679 bytes) 000513 1778614967.184515 fallocate 0x0267b8 thru 0x02d55e (0x6da7 bytes) 000515 1778614967.200987 read 0x016c2b thru 0x0214a5 (0xa87b bytes) 000516 1778614967.220725 mapread 0x00efd1 thru 0x01e014 (0xf044 bytes) 000517 1778614967.222218 punch from 0x038a0f to 0x038d76 (0x368 bytes) 000520 1778614967.245231 write 0x011623 thru 0x015b00 (0x44de bytes) 000521 1778614967.268435 read 0x0185f0 thru 0x02516a (0xcb7b bytes) 000522 1778614967.272242 mapwrite 0x02b784 thru 0x03064f (0x4ecc bytes) 000523 1778614967.300339 mapwrite 0x004e77 thru 0x00ea7f (0x9c09 bytes) 000524 1778614967.358220 trunc from 0x02e3f0 to 0x03a49b (0xc0ac bytes) 000526 1778614967.386307 write 0x027273 thru 0x02957b (0x2309 bytes) 000528 1778614967.392519 punch from 0x021607 to 0x021a55 (0x44f bytes) 000530 1778614967.401969 mapwrite 0x017abd thru 0x02482c (0xcd70 bytes) 000531 1778614967.482310 write 0x029140 thru 0x0299cd (0x88e bytes) 000532 1778614967.487899 trunc from 0x01b28b to 0x022af0 (0x7866 bytes) 000533 1778614967.499546 trunc from 0x0157d5 to 0x020b13 (0xb33f bytes) 000534 1778614967.520311 write 0x034901 thru 0x03c3fd (0x7afd bytes) 000535 1778614967.524514 punch from 0x026108 to 0x028c5f (0x2b58 bytes) 000536 1778614967.547641 write 0x025a70 thru 0x0295e4 (0x3b75 bytes) 000537 1778614967.553765 punch from 0x001ce3 to 0x004f91 (0x32af bytes) 000539 1778614967.570191 mapread 0x01fe23 thru 0x02c03b (0xc219 bytes) 000540 1778614967.612766 mapread 0x00fa4b thru 0x01f397 (0xf94d bytes) 000541 1778614967.724004 mapwrite 0x001ce9 thru 0x0067cb (0x4ae3 bytes) 000542 1778614967.800112 mapwrite 0x030a4c thru 0x038a24 (0x7fd9 bytes) 000544 1778614967.842600 mapwrite 0x036ad9 thru 0x03b095 (0x45bd bytes) 000545 1778614967.868307 fallocate 0x009a27 thru 0x012bca (0x91a4 bytes) 000546 1778614967.873929 mapwrite 0x0169b6 thru 0x020da3 (0xa3ee bytes) 000547 1778614967.903966 mapread 0x0286bb thru 0x0373a5 (0xeceb bytes) 000548 1778614967.905344 write 0x024600 thru 0x027b80 (0x3581 bytes) 000549 1778614967.910616 mapread 0x03f4c4 thru 0x03ffff (0xb3c bytes) 000550 1778614967.912371 fallocate 0x02f92c thru 0x03ab80 (0xb255 bytes) 000551 1778614967.918231 mapread 0x0221f6 thru 0x027675 (0x5480 bytes) 000552 1778614967.919373 mapwrite 0x011538 thru 0x01f9b3 (0xe47c bytes) 000553 1778614967.965006 trunc from 0x040000 to 0x00c89f (0x33760 bytes) 000554 1778614968.020075 fallocate 0x025b1f thru 0x030b08 (0xafea bytes) 000555 1778614968.030235 mapwrite 0x0197ac thru 0x0244a8 (0xacfd bytes) 000556 1778614968.163342 fallocate 0x03a7de thru 0x03ad73 (0x596 bytes) 000558 1778614968.172443 trunc from 0x0244a9 to 0x019b85 (0xa923 bytes) 000559 1778614968.189212 fallocate 0x0279fa thru 0x0303f2 (0x89f9 bytes) 000560 1778614968.197357 fallocate 0x02e3e5 thru 0x03db42 (0xf75e bytes) 000561 1778614968.205471 mapwrite 0x028ec7 thru 0x02b9c7 (0x2b01 bytes) 000562 1778614968.252817 trunc from 0x00393c to 0x00ffe1 (0xc6a6 bytes) 000563 1778614968.260186 fallocate 0x0075a8 thru 0x00e21f (0x6c78 bytes) 000564 1778614968.267304 write 0x009d4c thru 0x0103d4 (0x6689 bytes) 000565 1778614968.282852 mapwrite 0x03bb98 thru 0x03ffff (0x4468 bytes) 000566 1778614968.390849 mapread 0x035ca4 thru 0x03e4f1 (0x884e bytes) 000567 1778614968.454961 punch from 0x01cb88 to 0x0272b5 (0xa72e bytes) 000568 1778614968.464631 trunc from 0x018559 to 0x01ede9 (0x6891 bytes) 000569 1778614968.482316 read 0x037bd8 thru 0x03ffff (0x8428 bytes) 000570 1778614968.483258 read 0x03b8dc thru 0x03ffff (0x4724 bytes) 000571 1778614968.497443 punch from 0x0373e1 to 0x03ffff (0x8c1f bytes) 000572 1778614968.526729 trunc from 0x040000 to 0x010f95 (0x2f06a bytes) 000573 1778614968.559371 mapwrite 0x01375d thru 0x01d56b (0x9e0f bytes) 000574 1778614968.678124 mapread 0x00055f thru 0x007c39 (0x76db bytes) 000576 1778614968.685587 punch from 0x000cbe to 0x0063ef (0x5732 bytes) 000578 1778614968.698190 punch from 0x0182a5 to 0x01d56b (0x52c7 bytes) 000579 1778614968.712747 trunc from 0x01d56c to 0x019395 (0x41d6 bytes) 000580 1778614968.774037 mapwrite 0x03ccca thru 0x03ffff (0x3336 bytes) 000581 1778614968.831923 write 0x00d50a thru 0x012de2 (0x58d9 bytes) 000582 1778614968.833172 mapwrite 0x00e9a8 thru 0x016930 (0x7f89 bytes) 000583 1778614968.854614 punch from 0x03abf0 to 0x03ffff (0x5410 bytes) 000584 1778614968.860139 trunc from 0x03b923 to 0x03dd17 (0x23f5 bytes) 000585 1778614968.865192 write 0x021b5e thru 0x02f862 (0xdd05 bytes) 000586 1778614968.867862 trunc from 0x019620 to 0x01c560 (0x2f41 bytes) 000587 1778614968.873862 mapwrite 0x00d258 thru 0x00fa26 (0x27cf bytes) 000588 1778614968.893819 mapread 0x02ad84 thru 0x03530a (0xa587 bytes) 000589 1778614968.895787 trunc from 0x040000 to 0x021636 (0x1e9c9 bytes) 000590 1778614968.910832 trunc from 0x021637 to 0x015a19 (0xbc1d bytes) 000591 1778614968.932468 write 0x03d3e5 thru 0x03ffff (0x2c1b bytes) 000593 1778614968.936599 write 0x01cf3d thru 0x02a994 (0xda58 bytes) 000595 1778614968.948360 write 0x00a789 thru 0x0162e9 (0xbb61 bytes) 000596 1778614968.963320 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2b1002d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2b1002d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000597 1778614969.316015 mapwrite 0x01d139 thru 0x0276b2 (0xa57a bytes) 000598 1778614969.367027 read 0x018fac thru 0x027223 (0xe278 bytes) 000599 1778614969.368754 write 0x02c49b thru 0x03b6ce (0xf234 bytes) 000600 1778614969.390526 trunc from 0x040000 to 0x037f49 (0x80b6 bytes) 000601 1778614969.415919 trunc from 0x0068e6 to 0x015f06 (0xf621 bytes) 000602 1778614969.429512 trunc from 0x037f4a to 0x011657 (0x268f2 bytes) 000603 1778614969.453760 fallocate 0x03c426 thru 0x03ffff (0x3bda bytes) 000604 1778614969.465749 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) 000605 1778614969.638708 write 0x034bd5 thru 0x03ffff (0xb42b bytes) 000606 1778614969.645751 fallocate 0x00f971 thru 0x014748 (0x4dd8 bytes) 000607 1778614969.653888 trunc from 0x040000 to 0x039fb5 (0x604a bytes) 000608 1778614969.678535 mapread 0x01b7de thru 0x02b64b (0xfe6e bytes) 000609 1778614969.719538 trunc from 0x039fb6 to 0x033b93 (0x6422 bytes) 000610 1778614969.744400 mapread 0x00a9b6 thru 0x01812b (0xd776 bytes) 000611 1778614969.774021 trunc from 0x033b94 to 0x00380a (0x30389 bytes) 000612 1778614969.790864 trunc from 0x00380b to 0x03511e (0x31914 bytes) 000613 1778614969.812623 read 0x019438 thru 0x024c46 (0xb80f bytes) 000614 1778614969.822270 mapread 0x00d086 thru 0x01739b (0xa316 bytes) 000615 1778614969.840999 fallocate 0x03dfc9 thru 0x03e7fc (0x834 bytes) 000616 1778614969.847601 mapwrite 0x024f24 thru 0x028f19 (0x3ff6 bytes) 000619 1778614969.880532 read 0x004861 thru 0x00f452 (0xabf2 bytes) 000620 1778614969.894489 mapread 0x02f140 thru 0x0348c7 (0x5788 bytes) 000621 1778614969.899709 trunc from 0x027984 to 0x0320f7 (0xa774 bytes) 000622 1778614969.906844 trunc from 0x02c58e to 0x0308ab (0x431e bytes) 000623 1778614969.912538 mapread 0x02cdfb thru 0x02e43d (0x1643 bytes) 000624 1778614969.930962 trunc from 0x03511f to 0x02bb2a (0x95f4 bytes) 000625 1778614969.961704 mapwrite 0x01f8e6 thru 0x02352e (0x3c49 bytes) 000626 1778614969.988597 trunc from 0x0267cf to 0x02bb2a (0x535c bytes) 000627 1778614969.996321 punch from 0x02b3e9 to 0x02bb2a (0x742 bytes) 000628 1778614970.003361 mapwrite 0x009f69 thru 0x01341f (0x94b7 bytes) 000629 1778614970.028560 fallocate 0x02744b thru 0x029542 (0x20f8 bytes) 000630 1778614970.036348 mapread 0x029d24 thru 0x02bb2a (0x1e07 bytes) 000631 1778614970.055995 fallocate 0x028cf8 thru 0x0304fc (0x7805 bytes) 000632 1778614970.063597 write 0x0205a3 thru 0x02b892 (0xb2f0 bytes) 000634 1778614970.067960 fallocate 0x00a9fb thru 0x01a9cf (0xffd5 bytes) 000635 1778614970.074576 write 0x0251d2 thru 0x0298cb (0x46fa bytes) 000636 1778614970.076970 fallocate 0x0087ca thru 0x010878 (0x80af bytes) 000637 1778614970.083546 write 0x03c055 thru 0x03ffff (0x3fab bytes) 000638 1778614970.085908 trunc from 0x040000 to 0x009d35 (0x362ca bytes) 000639 1778614970.105782 read 0x006760 thru 0x009d35 (0x35d6 bytes) 000640 1778614970.107568 read 0x001a97 thru 0x004521 (0x2a8b bytes) 000641 1778614970.109589 read 0x009c4b thru 0x009d35 (0xeb bytes) 000643 1778614970.112773 read 0x000e79 thru 0x004a88 (0x3c10 bytes) 000644 1778614970.114492 punch from 0x0069fc to 0x009d35 (0x333a bytes) 000645 1778614970.119334 mapwrite 0x015a07 thru 0x017d35 (0x232f bytes) 000646 1778614970.163762 read 0x0135b5 thru 0x01672e (0x317a bytes) 000647 1778614970.165444 write 0x017e28 thru 0x01f8d9 (0x7ab2 bytes) 000648 1778614970.168861 write 0x00399e thru 0x00df6b (0xa5ce bytes) 000649 1778614970.172321 punch from 0x01a79a to 0x01a8b8 (0x11f bytes) 000650 1778614970.189869 trunc from 0x01f8da to 0x0264a7 (0x6bce bytes) 000651 1778614970.206435 punch from 0x017ba3 to 0x01e883 (0x6ce1 bytes) 000652 1778614970.212607 trunc from 0x00e432 to 0x017898 (0x9467 bytes) 000653 1778614970.220382 mapwrite 0x005c0c thru 0x00b4eb (0x58e0 bytes) 000654 1778614970.239851 write 0x010732 thru 0x01e974 (0xe243 bytes) 000655 1778614970.252393 trunc from 0x018daa to 0x0264a7 (0xd6fe bytes) 000656 1778614970.277131 fallocate 0x03cac7 thru 0x03ffff (0x3539 bytes) 000657 1778614970.284617 fallocate 0x020a93 thru 0x02cc66 (0xc1d4 bytes) 000658 1778614970.291564 write 0x0082f3 thru 0x00a369 (0x2077 bytes) 000659 1778614970.294538 read 0x03f56a thru 0x03ffff (0xa96 bytes) 000660 1778614970.312022 mapread 0x026ca7 thru 0x028a26 (0x1d80 bytes) 000661 1778614970.315175 mapread 0x00fbc4 thru 0x01da3f (0xde7c bytes) 000662 1778614970.319149 mapwrite 0x035cf1 thru 0x039c7f (0x3f8f bytes) 000665 1778614970.362660 punch from 0x023de7 to 0x02df05 (0xa11f bytes) 000666 1778614970.371449 mapread 0x014499 thru 0x01d7fe (0x9366 bytes) 000667 1778614970.376703 mapread 0x01c97a thru 0x01ee5f (0x24e6 bytes) 000668 1778614970.384908 fallocate 0x015ce1 thru 0x0172e0 (0x1600 bytes) 000669 1778614970.411893 read 0x019f31 thru 0x02470f (0xa7df bytes) 000670 1778614970.460831 read 0x018941 thru 0x01cc88 (0x4348 bytes) 000671 1778614970.463410 trunc from 0x040000 to 0x033a4a (0xc5b5 bytes) 000672 1778614970.509546 write 0x034869 thru 0x03e23b (0x99d3 bytes) 000673 1778614970.519527 write 0x03f465 thru 0x03ffff (0xb9b bytes) 000674 1778614970.525935 mapwrite 0x01a8cf thru 0x01e6ae (0x3de0 bytes) 000675 1778614970.576615 MIRROR_EXTEND to 2 mirrors 000676 1778614971.566667 write 0x00a642 thru 0x01198c (0x734b bytes) 000677 1778614971.648877 mapread 0x01c611 thru 0x02c3f2 (0xfde2 bytes) 000678 1778614971.675336 write 0x016fe3 thru 0x01bd8b (0x4da9 bytes) 000679 1778614971.686527 punch from 0x00eacc to 0x016d0e (0x8243 bytes) 000680 1778614971.715509 punch from 0x02bcfd to 0x033ca3 (0x7fa7 bytes) 000681 1778614971.725214 write 0x02beec thru 0x034b13 (0x8c28 bytes) 000683 1778614971.756999 write 0x00f679 thru 0x01ccf0 (0xd678 bytes) 000684 1778614971.775952 trunc from 0x01c454 to 0x01c73f (0x2ec bytes) 000685 1778614971.797280 punch from 0x0209d2 to 0x02a84e (0x9e7d bytes) 000686 1778614971.810997 mapwrite 0x02df64 thru 0x03243b (0x44d8 bytes) 000687 1778614971.836390 read 0x01b83f thru 0x01dd60 (0x2522 bytes) 000688 1778614971.843853 trunc from 0x040000 to 0x02639b (0x19c64 bytes) 000689 1778614971.900490 write 0x0006a7 thru 0x003943 (0x329d bytes) 000690 1778614971.943488 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2639c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2639c): Mirror 2: 0xa76a3960 (stale) CRC-32 checksum value for chunk [0, 0x2639c): Mirror 1: 0x6d03647 Mirror 3: 0x6d03647 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2639c: skipped 000691 1778614973.418748 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2639c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2639c): Mirror 2: 0xa76a3960 (stale) CRC-32 checksum value for chunk [0, 0x2639c): Mirror 1: 0x6d03647 Mirror 3: 0x6d03647 Mirror 4: 0x6d03647 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2639c: skipped 000692 1778614974.457745 write 0x01aace thru 0x02073f (0x5c72 bytes) 000693 1778614974.517179 punch from 0x0253fb to 0x02639b (0xfa1 bytes) 000694 1778614974.523139 mapwrite 0x00fb28 thru 0x0102ab (0x784 bytes) 000695 1778614974.565064 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2639c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2639c): Mirror 2: 0xa76a3960 (stale) Mirror 3: 0x6d03647 (stale) Mirror 4: 0x6d03647 (stale) CRC-32 checksum value for chunk [0, 0x2639c): Mirror 1: 0xcb8196be Mirror 5: 0xcb8196be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2639c: skipped 000696 1778614975.462723 punch from 0x00c478 to 0x010613 (0x419c bytes) 000697 1778614975.506198 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2639c): Mirror 2: 0xa76a3960 (stale) Mirror 3: 0x6d03647 (stale) Mirror 4: 0x6d03647 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2639c): Mirror 2: 0xa76a3960 (stale) Mirror 3: 0x6d03647 (stale) Mirror 4: 0x6d03647 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2639c: skipped 000698 1778614975.958499 mapread 0x013211 thru 0x01d3da (0xa1ca bytes) 000699 1778614976.009463 trunc from 0x018ed9 to 0x01a708 (0x1830 bytes) 000700 1778614976.023742 fallocate 0x01a1e8 thru 0x0292be (0xf0d7 bytes) 000701 1778614976.030375 trunc from 0x0260ee to 0x0292be (0x31d1 bytes) 000703 1778614976.043706 read 0x026913 thru 0x0292be (0x29ac bytes) 000704 1778614976.061396 trunc from 0x0292bf to 0x0368ff (0xd641 bytes) 000705 1778614976.129543 write 0x037778 thru 0x039eb2 (0x273b bytes) 000706 1778614976.149725 MIRROR_RESYNC 4 mirrors 000707 1778614977.833620 punch from 0x027772 to 0x02f25e (0x7aed bytes) 000708 1778614977.869729 read 0x033d00 thru 0x039eb2 (0x61b3 bytes) 000709 1778614977.881964 trunc from 0x039eb3 to 0x00309e (0x36e14 bytes) 000710 1778614977.909580 trunc from 0x00309f to 0x0324d8 (0x2f43a bytes) 000711 1778614977.928234 fallocate 0x00de40 thru 0x01bbf4 (0xddb5 bytes) 000712 1778614977.934881 mapwrite 0x0166f8 thru 0x024073 (0xd97c bytes) 000713 1778614978.001192 write 0x025b28 thru 0x0320ed (0xc5c6 bytes) 000714 1778614978.010866 mapwrite 0x013ee7 thru 0x022b79 (0xec93 bytes) 000715 1778614978.064289 MIRROR_RESYNC 4 mirrors 000716 1778614978.931398 write 0x02a72f thru 0x02b874 (0x1146 bytes) 000717 1778614978.984278 punch from 0x024052 to 0x02aeee (0x6e9d bytes) 000718 1778614978.994044 mapread 0x01c159 thru 0x0269fa (0xa8a2 bytes) 000719 1778614979.060793 fallocate 0x002cb4 thru 0x0067ac (0x3af9 bytes) 000720 1778614979.083225 write 0x024d67 thru 0x02c323 (0x75bd bytes) 000722 1778614979.095736 write 0x005be2 thru 0x015773 (0xfb92 bytes) 000723 1778614979.142461 read 0x0225a4 thru 0x02723d (0x4c9a bytes) 000724 1778614979.143126 write 0x003085 thru 0x0059a8 (0x2924 bytes) 000725 1778614979.144158 mapread 0x01e6d9 thru 0x029d98 (0xb6c0 bytes) 000726 1778614979.149390 read 0x01c39f thru 0x027be4 (0xb846 bytes) 000728 1778614979.154244 fallocate 0x037c2d thru 0x03ffff (0x83d3 bytes) 000729 1778614979.168238 trunc from 0x0324d9 to 0x00680d (0x2bccb bytes) 000730 1778614979.272066 trunc from 0x0040d0 to 0x00680d (0x273e bytes) 000731 1778614979.297989 fallocate 0x02e596 thru 0x03b334 (0xcd9f bytes) 000732 1778614979.306650 punch from 0x001909 to 0x00680d (0x4f05 bytes) 000733 1778614979.313716 mapwrite 0x03f8da thru 0x03ffff (0x726 bytes) 000734 1778614979.373605 mapread 0x00379b thru 0x010eb4 (0xd71a bytes) 000735 1778614979.375000 write 0x033900 thru 0x03a5d9 (0x6cda bytes) 000736 1778614979.378808 read 0x027df3 thru 0x028682 (0x890 bytes) 000738 1778614979.387597 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa9a9700a (stale) Mirror 4: 0xa9a9700a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa9a9700a (stale) Mirror 4: 0xa9a9700a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000739 1778614979.895532 write 0x033d90 thru 0x03ffff (0xc270 bytes) 000740 1778614979.915917 mapread 0x018fd8 thru 0x019c10 (0xc39 bytes) 000741 1778614979.941295 mapwrite 0x000d73 thru 0x001fe6 (0x1274 bytes) 000742 1778614979.981703 punch from 0x01784c to 0x0216bc (0x9e71 bytes) 000743 1778614979.991059 punch from 0x01f9ab to 0x02d7d8 (0xde2e bytes) 000744 1778614979.997758 mapread 0x033235 thru 0x03ffff (0xcdcb bytes) 000745 1778614980.003736 mapread 0x03a88c thru 0x03ffff (0x5774 bytes) 000746 1778614980.010322 fallocate 0x02693d thru 0x027068 (0x72c bytes) 000747 1778614980.022325 write 0x007f83 thru 0x00fea0 (0x7f1e bytes) 000748 1778614980.023586 write 0x0336cf thru 0x0382cf (0x4c01 bytes) 000749 1778614980.027177 read 0x015b97 thru 0x016f6a (0x13d4 bytes) 000750 1778614980.028501 read 0x029354 thru 0x03125d (0x7f0a bytes) 000751 1778614980.049469 write 0x00f31b thru 0x01adf3 (0xbad9 bytes) 000752 1778614980.054117 write 0x016a88 thru 0x019275 (0x27ee bytes) 000753 1778614980.076406 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa9a9700a (stale) Mirror 4: 0xa9a9700a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb9fe82d6 Mirror 5: 0xb9fe82d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000755 1778614981.281259 trunc from 0x00dc87 to 0x01a22c (0xc5a6 bytes) 000756 1778614981.332363 write 0x00a357 thru 0x00ed1b (0x49c5 bytes) 000757 1778614981.353722 write 0x02cbbe thru 0x02f202 (0x2645 bytes) 000758 1778614981.365524 trunc from 0x023cad to 0x028e9c (0x51f0 bytes) 000759 1778614981.371912 mapread 0x01ff04 thru 0x02767c (0x7779 bytes) 000760 1778614981.390525 punch from 0x01a13e to 0x01f397 (0x525a bytes) 000761 1778614981.397641 punch from 0x03ab39 to 0x03ffff (0x54c7 bytes) 000762 1778614981.412946 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa9a9700a (stale) Mirror 5: 0xb9fe82d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa9a9700a (stale) Mirror 5: 0xb9fe82d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000764 1778614982.044002 mapwrite 0x014dbb thru 0x01f4a3 (0xa6e9 bytes) 000765 1778614982.154951 mapwrite 0x035b62 thru 0x03bb51 (0x5ff0 bytes) 000766 1778614982.241349 mapread 0x03e176 thru 0x03ffff (0x1e8a bytes) 000767 1778614982.248244 trunc from 0x00f872 to 0x01ed94 (0xf523 bytes) 000768 1778614982.266872 trunc from 0x040000 to 0x0083bf (0x37c40 bytes) 000769 1778614982.337253 punch from 0x00128b to 0x001b55 (0x8cb bytes) 000770 1778614982.354366 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x83c0): Mirror 4: 0xc22174f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x83c0): Mirror 4: 0xc22174f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x83c0: skipped 000771 1778614982.845629 mapread 0x003284 thru 0x0083bf (0x513c bytes) 000772 1778614982.901810 trunc from 0x0083c0 to 0x024360 (0x1bfa1 bytes) 000773 1778614982.963295 fallocate 0x021c40 thru 0x02351e (0x18df bytes) 000774 1778614982.971789 fallocate 0x0083be thru 0x01560d (0xd250 bytes) 000775 1778614982.989801 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24361): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24361): Mirror 4: 0x8c07388e (stale) CRC-32 checksum value for chunk [0, 0x24361): Mirror 1: 0x167f062d Mirror 5: 0x167f062d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24361: skipped 000776 1778614983.981982 punch from 0x0160f5 to 0x01711a (0x1026 bytes) 000777 1778614984.033420 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24361): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24361): Mirror 4: 0x8c07388e (stale) Mirror 5: 0x167f062d (stale) CRC-32 checksum value for chunk [0, 0x24361): Mirror 1: 0x167f062d Mirror 6: 0x167f062d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24361: skipped 000778 1778614985.364636 trunc from 0x01cf30 to 0x02427f (0x7350 bytes) 000779 1778614985.417510 fallocate 0x036e6c thru 0x03c7fb (0x5990 bytes) 000780 1778614985.423475 mapread 0x011957 thru 0x01991a (0x7fc4 bytes) 000781 1778614985.445854 mapwrite 0x039ffe thru 0x03ffff (0x6002 bytes) 000782 1778614985.557936 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa9a9700a (stale) Mirror 5: 0x4eeb4c76 (stale) Mirror 6: 0x4eeb4c76 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x598dd70d Mirror 7: 0x598dd70d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000783 1778614987.143905 fallocate 0x0227a1 thru 0x0290d8 (0x6938 bytes) 000784 1778614987.208659 read 0x0308bc thru 0x035903 (0x5048 bytes) 000785 1778614987.227246 read 0x00c360 thru 0x011cb5 (0x5956 bytes) 000786 1778614987.244822 trunc from 0x013403 to 0x01cd0b (0x9909 bytes) 000787 1778614987.251887 punch from 0x028e24 to 0x033517 (0xa6f4 bytes) 000788 1778614987.259603 trunc from 0x040000 to 0x02497d (0x1b682 bytes) 000789 1778614987.319939 MIRROR_RESYNC 5 mirrors 000790 1778614988.867621 mapread 0x00f4f9 thru 0x014d1c (0x5824 bytes) 000791 1778614988.931699 trunc from 0x009acb to 0x00ad79 (0x12af bytes) 000792 1778614988.990147 write 0x00ceca thru 0x017e94 (0xafcb bytes) 000793 1778614989.015976 trunc from 0x02342c to 0x02497d (0x1552 bytes) 000794 1778614989.029818 trunc from 0x016b26 to 0x01a418 (0x38f3 bytes) 000795 1778614989.062955 write 0x012c8f thru 0x01b2c6 (0x8638 bytes) 000796 1778614989.077737 punch from 0x012494 to 0x016938 (0x44a5 bytes) 000797 1778614989.119600 mapread 0x015c00 thru 0x01d5b0 (0x79b1 bytes) 000798 1778614989.144519 MIRROR_RESYNC 5 mirrors 000799 1778614990.956963 fallocate 0x02c8bb thru 0x030258 (0x399e bytes) 000800 1778614991.028676 trunc from 0x01ce5a to 0x023f00 (0x70a7 bytes) 000801 1778614991.044906 punch from 0x00e351 to 0x01788d (0x953d bytes) 000802 1778614991.055169 fallocate 0x0384b9 thru 0x039366 (0xeae bytes) 000803 1778614991.068039 read 0x0296c3 thru 0x02cc53 (0x3591 bytes) 000804 1778614991.095232 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39367): Mirror 4: 0x3af05c18 (stale) Mirror 5: 0x3af05c18 (stale) Mirror 7: 0x3af05c18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39367): Mirror 4: 0x3af05c18 (stale) Mirror 5: 0x3af05c18 (stale) Mirror 7: 0x3af05c18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39367: skipped 000805 1778614991.685066 trunc from 0x00cc23 to 0x00e16b (0x1549 bytes) 000806 1778614991.693401 write 0x0133bf thru 0x01fbf8 (0xc83a bytes) 000807 1778614991.722601 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39367): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39367): Mirror 4: 0x3af05c18 (stale) Mirror 5: 0x3af05c18 (stale) Mirror 7: 0x3af05c18 (stale) CRC-32 checksum value for chunk [0, 0x39367): Mirror 1: 0x9517290a Mirror 8: 0x9517290a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39367: skipped 000809 1778614993.109684 punch from 0x01dab9 to 0x02c3fa (0xe942 bytes) 000810 1778614993.160064 mapread 0x011f51 thru 0x013e34 (0x1ee4 bytes) 000811 1778614993.179663 trunc from 0x039367 to 0x00a8ec (0x2ea7a bytes) 000812 1778614993.215959 MIRROR_RESYNC 5 mirrors 000813 1778614994.411600 trunc from 0x00a8ed to 0x0337e0 (0x28ef4 bytes) 000814 1778614994.456995 fallocate 0x027af9 thru 0x02fd8d (0x8295 bytes) 000815 1778614994.462419 punch from 0x018f76 to 0x01bfa8 (0x3033 bytes) 000816 1778614994.466795 write 0x03d89e thru 0x03ffff (0x2762 bytes) 000817 1778614994.468777 fallocate 0x02c27d thru 0x037b72 (0xb8f6 bytes) 000818 1778614994.475895 trunc from 0x03c2aa to 0x03ffff (0x3d56 bytes) 000819 1778614994.487485 fallocate 0x02db84 thru 0x03179e (0x3c1b bytes) 000820 1778614994.492600 trunc from 0x040000 to 0x02b73f (0x148c0 bytes) 000821 1778614994.515526 trunc from 0x0257f0 to 0x02b73f (0x5f50 bytes) 000824 1778614994.522591 mapread 0x0294fb thru 0x02b73f (0x2245 bytes) 000825 1778614994.548526 write 0x0237fa thru 0x030a41 (0xd248 bytes) 000826 1778614994.565449 fallocate 0x017bb4 thru 0x01c313 (0x4760 bytes) 000827 1778614994.571048 read 0x028c20 thru 0x02b1c2 (0x25a3 bytes) 000828 1778614994.579068 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30a42): Mirror 4: 0xc2a720df (stale) Mirror 5: 0xc2a720df (stale) Mirror 7: 0xc2a720df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30a42): Mirror 4: 0xc2a720df (stale) Mirror 5: 0xc2a720df (stale) Mirror 7: 0xc2a720df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30a42: skipped 000829 1778614994.998649 fallocate 0x01e8c8 thru 0x02af78 (0xc6b1 bytes) 000830 1778614995.004055 mapwrite 0x0137c5 thru 0x01d2f6 (0x9b32 bytes) 000831 1778614995.047678 trunc from 0x030a42 to 0x025cf1 (0xad50 bytes) 000833 1778614995.071429 fallocate 0x02c45d thru 0x0327cb (0x636f bytes) 000834 1778614995.086514 mapwrite 0x004606 thru 0x01317e (0xeb79 bytes) 000836 1778614995.158897 read 0x023b8e thru 0x025cf1 (0x2164 bytes) 000837 1778614995.184175 mapwrite 0x032a15 thru 0x036956 (0x3f42 bytes) 000838 1778614995.322346 trunc from 0x036957 to 0x006356 (0x30600 bytes) 000839 1778614995.378944 mapwrite 0x03a5e8 thru 0x03ffff (0x5a18 bytes) 000840 1778614995.486228 trunc from 0x0124e6 to 0x012eab (0x9c6 bytes) 000842 1778614995.497997 trunc from 0x040000 to 0x00bcf6 (0x34309 bytes) 000843 1778614995.535383 trunc from 0x005772 to 0x00bcf6 (0x6585 bytes) 000844 1778614995.548918 fallocate 0x00b3c9 thru 0x01b319 (0xff51 bytes) 000846 1778614995.557006 punch from 0x0152c0 to 0x01b319 (0x605a bytes) 000847 1778614995.564067 mapread 0x00573d thru 0x005c26 (0x4ea bytes) 000848 1778614995.576544 trunc from 0x005b18 to 0x00f045 (0x952e bytes) 000849 1778614995.584394 mapwrite 0x026b9a thru 0x02a187 (0x35ee bytes) 000851 1778614995.640375 punch from 0x029705 to 0x02a187 (0xa83 bytes) 000852 1778614995.650126 mapread 0x00599c thru 0x00833f (0x29a4 bytes) 000853 1778614995.664066 punch from 0x00b746 to 0x014ac1 (0x937c bytes) 000854 1778614995.674384 trunc from 0x02a188 to 0x030f73 (0x6dec bytes) 000855 1778614995.692918 mapread 0x01b4cd thru 0x01c9c8 (0x14fc bytes) 000856 1778614995.719061 trunc from 0x030f74 to 0x01f979 (0x115fa bytes) 000857 1778614995.746592 fallocate 0x00a8f6 thru 0x01533c (0xaa47 bytes) 000858 1778614995.754665 trunc from 0x001d59 to 0x005c71 (0x3f19 bytes) 000859 1778614995.759874 mapwrite 0x002500 thru 0x01163b (0xf13c bytes) 000860 1778614995.842771 read 0x01ab9c thru 0x01f979 (0x4dde bytes) 000862 1778614995.845524 punch from 0x012cb8 to 0x01ab74 (0x7ebd bytes) 000863 1778614995.852680 fallocate 0x0276dd thru 0x02dc6b (0x658f bytes) 000864 1778614995.864660 write 0x01644c thru 0x023587 (0xd13c bytes) 000865 1778614995.884662 punch from 0x01eefe to 0x02816d (0x9270 bytes) 000868 1778614995.911673 trunc from 0x02dc6c to 0x02e9d2 (0xd67 bytes) 000869 1778614995.942418 punch from 0x023515 to 0x027943 (0x442f bytes) 000870 1778614995.947678 fallocate 0x00e062 thru 0x016a78 (0x8a17 bytes) 000871 1778614995.954232 fallocate 0x0374b7 thru 0x03ffff (0x8b49 bytes) 000872 1778614995.960046 read 0x001480 thru 0x002dfb (0x197c bytes) 000873 1778614995.960720 trunc from 0x006213 to 0x00e4e8 (0x82d6 bytes) 000874 1778614995.973749 trunc from 0x02fc55 to 0x031336 (0x16e2 bytes) 000875 1778614995.981021 trunc from 0x040000 to 0x001cda (0x3e325 bytes) 000876 1778614996.016840 trunc from 0x001cdb to 0x02eac4 (0x2cdea bytes) 000877 1778614996.056930 write 0x01df58 thru 0x028e46 (0xaeef bytes) 000878 1778614996.083063 mapwrite 0x0112eb thru 0x01ac3f (0x9955 bytes) 000879 1778614996.161163 trunc from 0x0240dd to 0x02a5cc (0x64f0 bytes) 000880 1778614996.167715 write 0x00d098 thru 0x0103eb (0x3354 bytes) 000881 1778614996.180705 mapwrite 0x028bde thru 0x0299cc (0xdef bytes) 000882 1778614996.251015 trunc from 0x014d0b to 0x022949 (0xdc3f bytes) 000883 1778614996.270782 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2eac5): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eac5): Mirror 4: 0xcd86cd8c (stale) Mirror 5: 0xcd86cd8c (stale) Mirror 7: 0xcd86cd8c (stale) CRC-32 checksum value for chunk [0, 0x2eac5): Mirror 1: 0x4dd78739 Mirror 8: 0x4dd78739 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eac5: skipped 000884 1778614997.750673 trunc from 0x02eac5 to 0x03865a (0x9b96 bytes) 000885 1778614997.863517 punch from 0x00448d to 0x0073cd (0x2f41 bytes) 000886 1778614997.874685 trunc from 0x03865b to 0x020d98 (0x178c2 bytes) 000887 1778614997.928462 fallocate 0x039c86 thru 0x03eaaa (0x4e25 bytes) 000888 1778614997.938233 write 0x02fa3a thru 0x032c7c (0x3243 bytes) 000889 1778614997.959511 write 0x0006df thru 0x003544 (0x2e66 bytes) 000890 1778614998.008728 write 0x0006c5 thru 0x00446b (0x3da7 bytes) 000891 1778614998.010557 mapread 0x037fcc thru 0x03eaaa (0x6adf bytes) 000892 1778614998.037710 punch from 0x002aa8 to 0x00a588 (0x7ae1 bytes) 000893 1778614998.068013 trunc from 0x03eaab to 0x009806 (0x352a4 bytes) 000894 1778614998.092148 read 0x001b12 thru 0x009806 (0x7cf5 bytes) 000896 1778614998.122502 MIRROR_RESYNC 5 mirrors 000897 1778614999.281571 write 0x0186b6 thru 0x023957 (0xb2a2 bytes) 000898 1778614999.337969 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x23958): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23958): Mirror 4: 0x3ceadd33 (stale) Mirror 5: 0x3ceadd33 (stale) Mirror 7: 0x3ceadd33 (stale) Mirror 8: 0x3ceadd33 (stale) CRC-32 checksum value for chunk [0, 0x23958): Mirror 1: 0x2916a9d4 Mirror 9: 0x2916a9d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23958: skipped 000899 1778615000.743302 trunc from 0x023958 to 0x013d46 (0xfc11 bytes) 000900 1778615000.830648 read 0x004d19 thru 0x009fe2 (0x52ca bytes) 000901 1778615000.860202 trunc from 0x013d47 to 0x03cff7 (0x292b1 bytes) 000902 1778615000.929588 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cff8): Mirror 4: 0xdc7bdfbd (stale) Mirror 7: 0xdc7bdfbd (stale) Mirror 8: 0xdc7bdfbd (stale) Mirror 9: 0x9f4f1fc9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cff8): Mirror 4: 0xdc7bdfbd (stale) Mirror 7: 0xdc7bdfbd (stale) Mirror 8: 0xdc7bdfbd (stale) Mirror 9: 0x9f4f1fc9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cff8: skipped 000903 1778615001.537917 mapread 0x004021 thru 0x00e835 (0xa815 bytes) 000904 1778615001.576570 trunc from 0x03cff8 to 0x00388e (0x39769 bytes) 000906 1778615001.607239 trunc from 0x00388f to 0x03aa94 (0x37206 bytes) 000907 1778615001.631215 mapread 0x00f324 thru 0x014e25 (0x5b02 bytes) 000908 1778615001.651012 fallocate 0x0019c8 thru 0x007547 (0x5b80 bytes) 000909 1778615001.669302 mapwrite 0x02c0c4 thru 0x039099 (0xcfd6 bytes) 000910 1778615001.760030 mapread 0x004513 thru 0x0066f8 (0x21e6 bytes) 000911 1778615001.782100 read 0x031e41 thru 0x03aa94 (0x8c54 bytes) 000912 1778615001.785501 mapwrite 0x01f4ef thru 0x02e00b (0xeb1d bytes) 000914 1778615001.830273 read 0x030b83 thru 0x039498 (0x8916 bytes) 000915 1778615001.830959 trunc from 0x005d6d to 0x00e50e (0x87a2 bytes) 000916 1778615001.836786 trunc from 0x0135a9 to 0x021d74 (0xe7cc bytes) 000917 1778615001.841713 trunc from 0x03aa95 to 0x027b23 (0x12f71 bytes) 000918 1778615001.851712 trunc from 0x004d86 to 0x0095bd (0x4838 bytes) 000921 1778615001.858425 write 0x02fdc3 thru 0x03e759 (0xe997 bytes) 000922 1778615001.864449 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e75a): Mirror 4: 0x452752fe (stale) Mirror 7: 0x452752fe (stale) Mirror 9: 0x33d068e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e75a): Mirror 4: 0x452752fe (stale) Mirror 7: 0x452752fe (stale) Mirror 9: 0x33d068e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e75a: skipped 000923 1778615002.291533 write 0x02bd34 thru 0x037c95 (0xbf62 bytes) 000924 1778615002.317199 write 0x03129f thru 0x03ffff (0xed61 bytes) 000925 1778615002.321208 fallocate 0x023a40 thru 0x0276d2 (0x3c93 bytes) 000926 1778615002.328401 write 0x02e6d2 thru 0x03788a (0x91b9 bytes) 000927 1778615002.330846 read 0x03a26d thru 0x03ffff (0x5d93 bytes) 000928 1778615002.331854 mapread 0x0137e2 thru 0x01c5b7 (0x8dd6 bytes) 000929 1778615002.361008 mapread 0x021d41 thru 0x0308bc (0xeb7c bytes) 000930 1778615002.404857 fallocate 0x03bd03 thru 0x03deeb (0x21e9 bytes) 000931 1778615002.422911 write 0x036e0f thru 0x03c557 (0x5749 bytes) 000932 1778615002.431518 punch from 0x025d05 to 0x02ae55 (0x5151 bytes) 000933 1778615002.460064 MIRROR_RESYNC 4 mirrors 000934 1778615004.289620 MIRROR_RESYNC 4 mirrors 000935 1778615004.830426 MIRROR_EXTEND to 5 mirrors 000937 1778615005.714520 fallocate 0x021743 thru 0x02b10b (0x99c9 bytes) 000938 1778615005.786460 fallocate 0x03d6b5 thru 0x03ffff (0x294b bytes) 000939 1778615005.808720 trunc from 0x040000 to 0x03ac61 (0x539e bytes) 000940 1778615005.862768 fallocate 0x00dddb thru 0x010ddb (0x3001 bytes) 000941 1778615005.880688 read 0x008eb8 thru 0x018849 (0xf992 bytes) 000942 1778615005.901076 punch from 0x017451 to 0x01b0cd (0x3c7d bytes) 000943 1778615005.905794 mapread 0x022513 thru 0x0236ad (0x119b bytes) 000944 1778615005.916501 fallocate 0x019670 thru 0x01c94a (0x32db bytes) 000945 1778615005.924026 trunc from 0x03ac62 to 0x00b506 (0x2f75b bytes) 000946 1778615005.949072 trunc from 0x003a11 to 0x00b506 (0x7af6 bytes) 000947 1778615005.956421 fallocate 0x038cbc thru 0x03aa5c (0x1da1 bytes) 000948 1778615005.962358 trunc from 0x03aa5d to 0x0300f6 (0xa966 bytes) 000949 1778615005.978244 fallocate 0x01d7a2 thru 0x01dcf7 (0x556 bytes) 000950 1778615005.982883 punch from 0x025afc to 0x02d14a (0x764f bytes) 000951 1778615005.987220 trunc from 0x00a283 to 0x0160e2 (0xbe60 bytes) 000952 1778615005.991411 write 0x02c648 thru 0x032518 (0x5ed1 bytes) 000955 1778615006.003415 trunc from 0x015f50 to 0x0175a4 (0x1655 bytes) 000956 1778615006.007286 punch from 0x000e70 to 0x00d2c0 (0xc451 bytes) 000957 1778615006.011472 write 0x032528 thru 0x03e7b0 (0xc289 bytes) 000958 1778615006.013775 mapread 0x0387a6 thru 0x03acd5 (0x2530 bytes) 000960 1778615006.015727 mapwrite 0x014941 thru 0x023e6f (0xf52f bytes) 000961 1778615006.043898 mapwrite 0x01c34d thru 0x024f08 (0x8bbc bytes) 000962 1778615006.057117 punch from 0x029f1c to 0x02f637 (0x571c bytes) 000963 1778615006.060743 punch from 0x03a4d5 to 0x03e7b0 (0x42dc bytes) 000964 1778615006.065465 fallocate 0x000a28 thru 0x0079a2 (0x6f7b bytes) 000965 1778615006.068923 mapwrite 0x01067a thru 0x018cc4 (0x864b bytes) 000966 1778615006.084022 mapwrite 0x03f100 thru 0x03ffff (0xf00 bytes) 000968 1778615006.112292 mapread 0x0028b1 thru 0x0086cc (0x5e1c bytes) 000969 1778615006.120747 write 0x01cacf thru 0x02c201 (0xf733 bytes) 000970 1778615006.126306 mapwrite 0x02cf31 thru 0x02f1ee (0x22be bytes) 000971 1778615006.148028 punch from 0x012926 to 0x01367d (0xd58 bytes) 000972 1778615006.152343 mapwrite 0x0141be thru 0x01d680 (0x94c3 bytes) 000973 1778615006.168894 read 0x02d37b thru 0x034c92 (0x7918 bytes) 000975 1778615006.170218 mapwrite 0x032986 thru 0x037c09 (0x5284 bytes) 000976 1778615006.188872 read 0x02cec3 thru 0x02f3ca (0x2508 bytes) 000977 1778615006.194263 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb8d2e115 (stale) Mirror 7: 0xb8d2e115 (stale) Mirror 9: 0xb8d2e115 (stale) Mirror 10: 0xb8d2e115 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x87817744 Mirror 11: 0x87817744 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000978 1778615007.477333 mapwrite 0x02039f thru 0x0278b7 (0x7519 bytes) 000979 1778615007.702521 trunc from 0x040000 to 0x014b7d (0x2b482 bytes) 000980 1778615007.840028 write 0x01da84 thru 0x020bd3 (0x3150 bytes) 000981 1778615007.847340 write 0x02c6ac thru 0x036c15 (0xa56a bytes) 000982 1778615007.857819 mapwrite 0x0158ce thru 0x020aca (0xb1fd bytes) 000983 1778615007.947700 punch from 0x015e62 to 0x01e930 (0x8acf bytes) 000984 1778615007.957592 mapread 0x01f8ef thru 0x027a02 (0x8114 bytes) 000985 1778615007.976613 mapwrite 0x002879 thru 0x003eba (0x1642 bytes) 000986 1778615008.017736 mapwrite 0x0122e8 thru 0x01d063 (0xad7c bytes) 000987 1778615008.078267 read 0x008841 thru 0x008ada (0x29a bytes) 000988 1778615008.078963 trunc from 0x034261 to 0x036c15 (0x29b5 bytes) 000989 1778615008.086906 fallocate 0x019b59 thru 0x025e46 (0xc2ee bytes) 000990 1778615008.092784 punch from 0x0295db to 0x0330ba (0x9ae0 bytes) 000991 1778615008.100854 fallocate 0x0236f6 thru 0x02e408 (0xad13 bytes) 000992 1778615008.108359 write 0x03ea54 thru 0x03ffff (0x15ac bytes) 000993 1778615008.109528 mapwrite 0x02f47d thru 0x0309e3 (0x1567 bytes) 000994 1778615008.133863 write 0x01ecb3 thru 0x02b18b (0xc4d9 bytes) 000995 1778615008.135433 punch from 0x009ace to 0x010333 (0x6866 bytes) 000996 1778615008.146763 punch from 0x01f1a1 to 0x02b2ab (0xc10b bytes) 000997 1778615008.183342 trunc from 0x03143f to 0x03b343 (0x9f05 bytes) 000998 1778615008.198831 read 0x015968 thru 0x016451 (0xaea bytes) 000999 1778615008.199492 punch from 0x01e55c to 0x023944 (0x53e9 bytes) 001000 1778615008.206488 fallocate 0x008b03 thru 0x00af57 (0x2455 bytes) All operations completed A-OK! PASS 70a (107s) == sanity-flr test 71: check mirror extend parent fid ==== 15:43:35 (1778615015) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x1a6:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (14s) == sanity-flr test 72: check append on FLR file ========== 15:43:49 (1778615029) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.361636 s, 5.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160999 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 15:44:00 (1778615040) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3254464 .. Extending file size to 5990368 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5885312 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6783200 .. Extending file size to 8928352 .. Extending file size to 7890976 .. Extending file size to 8260480 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8570016 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 60894 60895 60896 60898 60899 done 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) 192.168.204.131@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:45:23 (1778615123) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2110560 .. Extending file size to 5287360 .. done Extending file size to 7637600 .. Extending file size to 4222656 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7895520 .. Extending file size to 7017408 .. Extending file size to 8821792 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8969600 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9021984 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9266976 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62228 62229 62231 62232 62235 done 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) 192.168.204.131@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:46:47 (1778615207) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2144 1269160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25840 3581180 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17940 3589080 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43780 7170260 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2144 1269160 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25840 3581180 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17940 3589080 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43780 7170260 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165696 s, 12.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.285903 s, 7.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xde1954d3 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:47:08 (1778615228) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113256 s, 18.5 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 15:47:16 (1778615236) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:47:25 (1778615245) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108431 s, 19.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x204:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x203:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x204:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:47:34 (1778615254) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109632 s, 19.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x205:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x206:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x205:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x205:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x206:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x205:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x206:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:47:43 (1778615263) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104053 s, 20.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:48:17 (1778615297) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg431-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109114 s, 9.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x207:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x208:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x207:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x207:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x208:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x207:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x208:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:49:09 (1778615349) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg431-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0920471 s, 22.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:50:03 (1778615403) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg431-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg431-server: Pool lustre.test_204f created Adding targets to pool oleg431-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg431-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0941291 s, 22.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg431-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg431-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg431-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:51:22 (1778615482) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:51:30 (1778615490) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:51:40 (1778615500) oleg431-server: Pool lustre.test_206 created Adding targets to pool oleg431-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg431-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg431-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg431-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg431-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:52:09 (1778615529) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111097 s, 9.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965203 s, 10.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x31d19fcc Mirror 3: 0x31d19fcc PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:52:21 (1778615541) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:52:25 (1778615545) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:52:29 (1778615549) Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0650318 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xe50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x16d0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x16d0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x16d0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2200000 Read this (4 bytes): BBBB 192.168.204.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg431-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:53:10 (1778615590) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0383581 s, 214 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:53:18 (1778615598) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568366 s, 18.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611265 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-12 15:53:19.000000000 -0400 Modify: 2026-05-12 15:53:20.000000000 -0400 Change: 2026-05-12 15:53:21.000000000 -0400 Birth: 2026-05-12 15:53:20.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:53:27 (1778615607) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344647 s, 30.4 MB/s oleg431-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg431-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg431-server: O/280000401/d26/13a: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:53:49 (1778615629) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.2724 s, 150 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0257309 s, 159 kB/s size after second write -rw-r--r-- 1 root root 4096 May 12 15:53 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x86:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 12 15:53 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:53:58 (1778615638) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:54:06 (1778615646) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1941 sec =========== 15:54:15 (1778615655) === sanity-flr: start cleanup 15:54:17 (1778615657) === === sanity-flr: finish cleanup 15:54:27 (1778615667) ===