-----============= acceptance-small: sanity-flr ============----- Tue Feb 27 12:09:39 EST 2024 excepting tests: 6 201 44c === sanity-flr: start setup 12:09:41 (1709053781) === oleg211-client.virtnet: executing check_config_client /mnt/lustre oleg211-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg211-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5cea800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5cea800.idle_timeout=debug disable quota as required oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 12:09:48 (1709053788) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7310] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 12:09:49 (1709053789) PASS 0a (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:09:54 (1709053794) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:09:57 (1709053797) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 12:09:59 (1709053799) PASS 0d (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:10:05 (1709053805) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:10:07 (1709053807) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 12:10:09 (1709053809) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b5cea800.stats=clear osc.lustre-OST0001-osc-ffff8800b5cea800.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:10:13 (1709053813) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:10:16 (1709053816) PASS 0j (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:10:19 (1709053819) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 12:10:21 (1709053821) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 12:10:24 (1709053824) PASS 2 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:10:27 (1709053827) PASS 3 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:10:30 (1709053830) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:10:33 (1709053833) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 12:10:37 (1709053837) PASS 7 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:10:40 (1709053840) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.165228 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0240766 s, 43.6 MB/s PASS 21 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:10:43 (1709053843) 13+0 records in 13+0 records out 13631488 bytes (14 MB) copied, 0.0836557 s, 163 MB/s PASS 22 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:10:46 (1709053846) Stopping /mnt/lustre-ost1 (opts:) on oleg211-server oleg211-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5cea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5cea800.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg211-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg211-server oleg211-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5cea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5cea800.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg211-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (20s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:11:08 (1709053868) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0371663 s, 84.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg211-server oleg211-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5cea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5cea800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg211-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:11:17 (1709053877) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg211-server oleg211-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5cea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5cea800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg211-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg211-server oleg211-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5cea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5cea800.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg211-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:11:37 (1709053897) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.704291 s, 74.4 MB/s -rw-r--r-- 1 root root 52428800 Feb 27 12:11 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.220s user 0m0.000s sys 0m0.130s Stopping /mnt/lustre-ost1 (opts:) on oleg211-server oleg211-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5cea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5cea800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.224s user 0m0.001s sys 0m0.107s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg211-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg211-server oleg211-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5cea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5cea800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.351s user 0m0.000s sys 0m0.124s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg211-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg211-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5cea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5cea800.ost_server_uuid in FULL state after 0 sec PASS 33b (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:12:00 (1709053920) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:12:07 (1709053927) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:12:10 (1709053930) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 12:12:12 (1709053932) PASS 35 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 12:12:15 (1709053935) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.104493 s, 120 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:12:19 (1709053939) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00894778 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00848225 s, 124 MB/s PASS 36b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:12:22 (1709053942) PASS 36c (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:12:26 (1709053946) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000523772 s, 2.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000525545 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000535505 s, 1.9 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00056512 s, 1.8 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 12:12:29 (1709053949) create test files with size 15739663 .. 06bdd5399a9733c14a466a4c0c4cdde4 - 83b08db1065c75c87426f5fe8b32ee72 - 70e6568c3f64f0145704b064e15188d2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg211-client.virtnet /mnt/lustre (opts:) Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre PASS 37 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 12:12:35 (1709053955) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:12:39 (1709053959) PASS 39 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:12:42 (1709053962) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110593 s, 94.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00910516 s, 115 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 12:12:45 (1709053965) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0155234 s, 135 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0571043 s, 73.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0271151 s, 77.3 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 12:12:49 (1709053969) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 12:12:51 (1709053971) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:12:53 (1709053973) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100369 s, 104 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 12:12:56 (1709053976) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0746027 s, 42.2 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 44a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:13:00 (1709053980) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x47:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:13:03 (1709053983) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0582901 s, 180 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg211-client.virtnet /mnt/lustre (opts:) Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:13:07 (1709053987) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a965c800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0578789 s, 181 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg211-client.virtnet /mnt/lustre (opts:) Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800b5d4f000.xattr_cache=1 PASS 44e (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:13:12 (1709053992) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 12:13:19 (1709053999) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 12:13:27 (1709054007) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 12:13:30 (1709054010) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0863665 s, 36.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0452061 s, 69.6 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: f50c8d0b020bf6a2bd4aefc5b1cd64b6 - mirror 1 checksum: ef7e7a0bc8e5200fa10d714a7d897016 - mirror 2 checksum: f50c8d0b020bf6a2bd4aefc5b1cd64b6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Feb 27 12:13:31 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Feb 27 12:13:31 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f50c8d0b020bf6a2bd4aefc5b1cd64b6 - mirror 1 checksum: ef7e7a0bc8e5200fa10d714a7d897016 - mirror 2 checksum: ef7e7a0bc8e5200fa10d714a7d897016 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Feb 27 12:13:31 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Feb 27 12:13:31 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } PASS 48 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 49a: FIEMAP upon FLR file ============= 12:13:34 (1709054014) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100971 s, 104 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 527360.. 528383: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:13:38 (1709054018) Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server 12:13:45 (1709054025) shut down Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 12:13:58 (1709054038) targets are mounted 12:13:58 (1709054038) facet_failover done oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102948 s, 102 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 50A (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:14:07 (1709054047) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0071317 s, 2.9 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0034527 s, 5.9 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 12:14:10 (1709054050) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00627128 s, 3.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00187932 s, 10.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:14:14 (1709054054) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00145692 s, 11.2 MB/s PASS 50c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 12:14:17 (1709054057) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00665333 s, 3.1 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:14:20 (1709054060) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00129729 s, 3.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00700236 s, 585 kB/s PASS 60a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:14:22 (1709054062) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:14:27 (1709054067) create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] PASS 61a (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:14:45 (1709054085) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:15:09 (1709054109) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:15:29 (1709054129) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 12:15:35 (1709054135) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 30083 mirror split pid 30085 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4637: 30083 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4637: 30085 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 12:16:38 (1709054198) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 4199 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000001 1709054199.451886 mapwrite 0x02b995 thru 0x039231 (0xd89d bytes) 000004 1709054199.499914 mapread 0x002471 thru 0x00ffc7 (0xdb57 bytes) 000005 1709054199.506015 mapread 0x00878c thru 0x013a54 (0xb2c9 bytes) truncating to largest ever: 0x3c058 000007 1709054199.511521 trunc from 0x039232 to 0x03c057 (0x2e26 bytes) 000008 1709054199.517290 mapread 0x00f492 thru 0x011716 (0x2285 bytes) 000009 1709054199.517709 write 0x01cdda thru 0x026996 (0x9bbd bytes) 000010 1709054199.520537 trunc from 0x03c058 to 0x01002e (0x2c029 bytes) 000011 1709054199.526995 mapread 0x00acbb thru 0x01002e (0x5374 bytes) 000012 1709054199.527598 read 0x0040d2 thru 0x00ae01 (0x6d30 bytes) 000014 1709054199.528264 read 0x00a26e thru 0x00fa9c (0x582f bytes) 000015 1709054199.528719 read 0x00168d thru 0x00b167 (0x9adb bytes) 000016 1709054199.529075 mapread 0x008d90 thru 0x009163 (0x3d4 bytes) 000017 1709054199.529593 mapread 0x0029f4 thru 0x004e3d (0x244a bytes) 000018 1709054199.530394 write 0x005ac5 thru 0x008245 (0x2781 bytes) 000022 1709054199.532265 punch from 0x00863c to 0x00a7b1 (0x2176 bytes) 000023 1709054199.536464 trunc from 0x01002f to 0x0052ce (0xad60 bytes) 000024 1709054199.542909 write 0x006edf thru 0x00b5d5 (0x46f7 bytes) 000025 1709054199.543710 fallocate 0x00ab4c thru 0x010f1a (0x63cf bytes) 000026 1709054199.546547 mapread 0x00f153 thru 0x010d39 (0x1be7 bytes) 000027 1709054199.550844 fallocate 0x01c14f thru 0x021039 (0x4eeb bytes) 000028 1709054199.554103 punch from 0x008fda to 0x00abb9 (0x1be0 bytes) 000029 1709054199.566434 punch from 0x0005a2 to 0x008dc7 (0x8826 bytes) 000030 1709054199.569517 trunc from 0x010f1b to 0x004610 (0xc90a bytes) 000031 1709054199.578031 write 0x012585 thru 0x01a97c (0x83f8 bytes) 000033 1709054199.579169 fallocate 0x0000d6 thru 0x00f99f (0xf8ca bytes) 000034 1709054199.582540 punch from 0x00e986 to 0x01a97c (0xbff7 bytes) 000035 1709054199.593905 MIRROR_EXTEND to 2 mirrors 000036 1709054199.781004 fallocate 0x02074a thru 0x02faeb (0xf3a2 bytes) 000038 1709054199.794541 read 0x0049b4 thru 0x009360 (0x49ad bytes) fallocating to largest ever: 0x3d24b 000039 1709054199.799591 fallocate 0x036382 thru 0x03d24a (0x6ec9 bytes) 000040 1709054199.803827 MIRROR_EXTEND to 3 mirrors 000041 1709054199.963562 write 0x00230c thru 0x006efb (0x4bf0 bytes) 000042 1709054199.985783 mapwrite 0x03a3c3 thru 0x03ffff (0x5c3d bytes) 000043 1709054200.017688 fallocate 0x018aaa thru 0x019da6 (0x12fd bytes) 000044 1709054200.020175 fallocate 0x00aa42 thru 0x015ed4 (0xb493 bytes) fallocating to largest ever: 0x40000 000045 1709054200.022289 fallocate 0x03b3ba thru 0x03ffff (0x4c46 bytes) 000046 1709054200.024938 mapwrite 0x0242df thru 0x032f10 (0xec32 bytes) 000047 1709054200.044034 trunc from 0x040000 to 0x0313de (0xec21 bytes) 000048 1709054200.049806 trunc from 0x0313df to 0x0285ad (0x8e31 bytes) 000050 1709054200.055530 punch from 0x01d213 to 0x01f1da (0x1fc8 bytes) 000051 1709054200.059577 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000053 1709054200.139456 trunc from 0x0285ae to 0x008782 (0x1fe2b bytes) 000055 1709054200.147094 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000057 1709054200.182976 read 0x003cc0 thru 0x008782 (0x4ac3 bytes) 000058 1709054200.186987 mapwrite 0x025381 thru 0x030144 (0xadc4 bytes) 000059 1709054200.216285 mapread 0x01b500 thru 0x02a1ea (0xeceb bytes) 000060 1709054200.220177 write 0x01b57a thru 0x01dc43 (0x26ca bytes) 000062 1709054200.221128 write 0x038da7 thru 0x03ffff (0x7259 bytes) 000065 1709054200.222723 punch from 0x023e2d to 0x02d0aa (0x927e bytes) 000066 1709054200.225055 mapwrite 0x0054db thru 0x009537 (0x405d bytes) 000067 1709054200.234950 mapread 0x030de1 thru 0x0388d3 (0x7af3 bytes) 000069 1709054200.238756 mapread 0x011cd5 thru 0x01ab46 (0x8e72 bytes) 000070 1709054200.242626 fallocate 0x02434b thru 0x032795 (0xe44b bytes) 000071 1709054200.244928 write 0x002616 thru 0x00f25b (0xcc46 bytes) 000072 1709054200.245915 fallocate 0x01a4f9 thru 0x026370 (0xbe78 bytes) 000073 1709054200.248274 mapwrite 0x017b05 thru 0x018ae6 (0xfe2 bytes) 000075 1709054200.257586 mapread 0x02ffc3 thru 0x03e704 (0xe742 bytes) 000078 1709054200.258511 mapread 0x036731 thru 0x03b64d (0x4f1d bytes) 000081 1709054200.259466 fallocate 0x01a1fd thru 0x01bb1b (0x191f bytes) 000084 1709054200.263636 mapwrite 0x010574 thru 0x01d2e6 (0xcd73 bytes) 000085 1709054200.274266 fallocate 0x0168c3 thru 0x021337 (0xaa75 bytes) 000086 1709054200.277274 write 0x024c5b thru 0x030f6e (0xc314 bytes) 000087 1709054200.282475 mapread 0x03f093 thru 0x03ffff (0xf6d bytes) 000088 1709054200.283062 punch from 0x00428e to 0x00745b (0x31ce bytes) 000089 1709054200.285815 fallocate 0x03a35a thru 0x03ffff (0x5ca6 bytes) 000091 1709054200.289691 mapwrite 0x001a9c thru 0x00c9f7 (0xaf5c bytes) 000092 1709054200.310038 punch from 0x02917f to 0x02a3f8 (0x127a bytes) 000094 1709054200.312869 trunc from 0x040000 to 0x00794d (0x386b2 bytes) 000096 1709054200.322692 mapread 0x0015ce thru 0x0035dd (0x2010 bytes) 000098 1709054200.323362 fallocate 0x01184a thru 0x014283 (0x2a3a bytes) 000099 1709054200.326838 trunc from 0x00794e to 0x03b6e7 (0x33d9a bytes) 000100 1709054200.332968 fallocate 0x004f56 thru 0x00c658 (0x7703 bytes) 000101 1709054200.335020 write 0x0285b4 thru 0x028c0a (0x657 bytes) 000103 1709054200.339299 punch from 0x001645 to 0x00fe99 (0xe855 bytes) 000105 1709054200.342235 mapwrite 0x028528 thru 0x029a78 (0x1551 bytes) 000106 1709054200.348773 fallocate 0x037454 thru 0x03c298 (0x4e45 bytes) 000107 1709054200.354141 trunc from 0x03c299 to 0x017a33 (0x24865 bytes) 000108 1709054200.362515 punch from 0x00ab5e to 0x00fc36 (0x50d9 bytes) 000109 1709054200.366299 mapwrite 0x01f3f7 thru 0x01fbca (0x7d4 bytes) 000110 1709054200.390098 mapread 0x00df5c thru 0x01bff2 (0xe097 bytes) 000112 1709054200.399997 mapread 0x01ef9a thru 0x01fbca (0xc31 bytes) 000114 1709054200.400879 mapread 0x0177e1 thru 0x01ac1c (0x343c bytes) 000116 1709054200.401652 fallocate 0x03ca14 thru 0x03ffff (0x35ec bytes) 000117 1709054200.405595 fallocate 0x0042f2 thru 0x00cacf (0x87de bytes) 000118 1709054200.407674 write 0x0399cc thru 0x03ffff (0x6634 bytes) 000119 1709054200.408464 read 0x03dc56 thru 0x03f21a (0x15c5 bytes) 000120 1709054200.408784 write 0x03a5f8 thru 0x03ffff (0x5a08 bytes) 000121 1709054200.409340 write 0x016a3d thru 0x01e243 (0x7807 bytes) 000124 1709054200.412369 MIRROR_EXTEND to 2 mirrors 000125 1709054200.597921 punch from 0x017a91 to 0x026dcd (0xf33d bytes) 000126 1709054200.611304 mapwrite 0x019d5b thru 0x02950e (0xf7b4 bytes) 000128 1709054200.628916 mapwrite 0x021fc4 thru 0x02462d (0x266a bytes) 000129 1709054200.636487 trunc from 0x040000 to 0x02a0eb (0x15f14 bytes) 000130 1709054200.652052 mapread 0x00800b thru 0x00ba0c (0x3a02 bytes) 000131 1709054200.660161 MIRROR_RESYNC 2 mirrors 000132 1709054200.832412 mapwrite 0x01deab thru 0x02a8e0 (0xca36 bytes) 000133 1709054200.886149 read 0x007c1f thru 0x00e5be (0x69a0 bytes) 000134 1709054200.892395 MIRROR_RESYNC 2 mirrors 000135 1709054201.070233 mapread 0x015440 thru 0x024fd5 (0xfb96 bytes) 000136 1709054201.086728 mapread 0x002be5 thru 0x00da32 (0xae4e bytes) 000137 1709054201.090819 trunc from 0x02a8e1 to 0x0308c0 (0x5fe0 bytes) 000138 1709054201.118905 write 0x01e675 thru 0x022e4e (0x47da bytes) 000139 1709054201.124122 trunc from 0x0308c1 to 0x02cea0 (0x3a20 bytes) 000140 1709054201.139414 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000142 1709054201.187227 read 0x00c684 thru 0x019925 (0xd2a2 bytes) 000143 1709054201.194665 write 0x009f2a thru 0x010ee6 (0x6fbd bytes) 000145 1709054201.199472 write 0x035351 thru 0x03798d (0x263d bytes) 000146 1709054201.200253 write 0x037962 thru 0x03eb67 (0x7206 bytes) 000147 1709054201.201614 punch from 0x038ada to 0x03eb67 (0x608e bytes) 000149 1709054201.207906 fallocate 0x02de56 thru 0x032897 (0x4a42 bytes) 000151 1709054201.210232 write 0x00560e thru 0x00b673 (0x6066 bytes) 000152 1709054201.212823 mapwrite 0x016d28 thru 0x01a21e (0x34f7 bytes) 000153 1709054201.230287 punch from 0x0211b6 to 0x02869c (0x74e7 bytes) 000154 1709054201.234107 mapwrite 0x02ec8d thru 0x02f4e4 (0x858 bytes) 000156 1709054201.242623 fallocate 0x03fbf5 thru 0x03fef7 (0x303 bytes) 000157 1709054201.244724 fallocate 0x02b396 thru 0x02bcb7 (0x922 bytes) 000159 1709054201.247088 punch from 0x02d3ae to 0x034235 (0x6e88 bytes) 000160 1709054201.249577 mapwrite 0x03a2b6 thru 0x03ab74 (0x8bf bytes) 000162 1709054201.258850 fallocate 0x0005f4 thru 0x001ead (0x18ba bytes) 000163 1709054201.260693 write 0x01872d thru 0x01ed50 (0x6624 bytes) 000164 1709054201.261233 mapread 0x023589 thru 0x024cea (0x1762 bytes) 000165 1709054201.264633 fallocate 0x0213c7 thru 0x0281e6 (0x6e20 bytes) 000166 1709054201.268800 punch from 0x01b304 to 0x021966 (0x6663 bytes) 000167 1709054201.278368 MIRROR_EXTEND to 2 mirrors 000168 1709054201.562153 read 0x00e3ab thru 0x01b9a8 (0xd5fe bytes) 000169 1709054201.565509 mapwrite 0x00d8c5 thru 0x011fda (0x4716 bytes) 000170 1709054201.618738 punch from 0x00f016 to 0x01e128 (0xf113 bytes) 000171 1709054201.624588 punch from 0x03af4a to 0x03eb67 (0x3c1e bytes) 000172 1709054201.627166 read 0x010839 thru 0x01bd8f (0xb557 bytes) 000174 1709054201.633156 read 0x00e638 thru 0x01b699 (0xd062 bytes) 000175 1709054201.635738 punch from 0x001360 to 0x005972 (0x4613 bytes) 000176 1709054201.653331 read 0x037b43 thru 0x03eb67 (0x7025 bytes) 000178 1709054201.654028 trunc from 0x03eb68 to 0x03c543 (0x2624 bytes) 000179 1709054201.678691 trunc from 0x03c544 to 0x0018d3 (0x3ac70 bytes) 000180 1709054201.689008 MIRROR_EXTEND to 3 mirrors 000181 1709054201.861271 mapread 0x00109d thru 0x0018d3 (0x837 bytes) 000182 1709054201.868150 mapwrite 0x01d7f9 thru 0x01edb7 (0x15bf bytes) 000183 1709054201.904692 fallocate 0x00ec71 thru 0x018fba (0xa34a bytes) 000184 1709054201.908151 MIRROR_RESYNC 3 mirrors 000185 1709054202.153374 mapwrite 0x01dedd thru 0x02208e (0x41b2 bytes) 000186 1709054202.180063 MIRROR_EXTEND to 4 mirrors 000187 1709054202.335046 fallocate 0x01c7e6 thru 0x02bbff (0xf41a bytes) 000188 1709054202.354661 punch from 0x01f9db to 0x02208e (0x26b4 bytes) 000189 1709054202.375238 mapwrite 0x019611 thru 0x028644 (0xf034 bytes) 000190 1709054202.423668 write 0x0291a8 thru 0x0361e2 (0xd03b bytes) 000191 1709054202.424753 mapwrite 0x032cef thru 0x03ecf0 (0xc002 bytes) 000193 1709054202.443066 mapread 0x033031 thru 0x03c45b (0x942b bytes) 000194 1709054202.443693 mapread 0x033cc1 thru 0x03cdea (0x912a bytes) 000195 1709054202.444233 write 0x008304 thru 0x00d74a (0x5447 bytes) 000196 1709054202.449146 trunc from 0x03ecf1 to 0x03b29e (0x3a52 bytes) 000197 1709054202.456260 read 0x032a5d thru 0x03aea4 (0x8448 bytes) 000198 1709054202.456679 mapread 0x01d3e3 thru 0x024736 (0x7354 bytes) 000199 1709054202.457230 write 0x037f72 thru 0x03ffff (0x808e bytes) 000200 1709054202.457983 read 0x006588 thru 0x0115bd (0xb036 bytes) 000201 1709054202.460689 trunc from 0x040000 to 0x0348b0 (0xb74f bytes) 000202 1709054202.468251 trunc from 0x0348b1 to 0x01cffb (0x178b5 bytes) 000203 1709054202.476028 fallocate 0x0114a9 thru 0x015320 (0x3e78 bytes) 000205 1709054202.478411 trunc from 0x01cffc to 0x012740 (0xa8bb bytes) 000207 1709054202.482967 write 0x00a60f thru 0x019e67 (0xf859 bytes) 000208 1709054202.486021 MIRROR_RESYNC 4 mirrors 000209 1709054202.676771 read 0x00f862 thru 0x019e67 (0xa606 bytes) 000210 1709054202.681639 trunc from 0x019e68 to 0x02dd84 (0x13f1d bytes) 000211 1709054202.702992 mapread 0x00c62e thru 0x014698 (0x806b bytes) 000212 1709054202.710271 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000214 1709054202.774245 punch from 0x009288 to 0x015e84 (0xcbfd bytes) 000215 1709054202.778171 mapwrite 0x01d98e thru 0x026392 (0x8a05 bytes) 000216 1709054202.797019 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000217 1709054202.867624 trunc from 0x02dd85 to 0x01bbb8 (0x121cc bytes) 000222 1709054202.876297 write 0x029aa0 thru 0x02af25 (0x1486 bytes) 000223 1709054202.876954 read 0x006708 thru 0x00cafe (0x63f7 bytes) 000224 1709054202.880856 mapread 0x019717 thru 0x020f0f (0x77f9 bytes) 000225 1709054202.888687 write 0x005000 thru 0x007a3b (0x2a3c bytes) 000226 1709054202.891019 MIRROR_RESYNC 2 mirrors 000227 1709054203.081137 MIRROR_RESYNC 2 mirrors 000228 1709054203.128791 punch from 0x012b9c to 0x020f15 (0xe37a bytes) 000229 1709054203.140827 punch from 0x00b867 to 0x01b15e (0xf8f8 bytes) 000230 1709054203.143006 punch from 0x01aa31 to 0x01f4c8 (0x4a98 bytes) 000231 1709054203.146572 mapwrite 0x036b99 thru 0x03706e (0x4d6 bytes) 000234 1709054203.164286 fallocate 0x00a8a7 thru 0x0104cd (0x5c27 bytes) 000235 1709054203.166880 write 0x012ab2 thru 0x01a688 (0x7bd7 bytes) 000236 1709054203.173016 punch from 0x02b801 to 0x032d92 (0x7592 bytes) 000237 1709054203.175437 punch from 0x031812 to 0x032339 (0xb28 bytes) 000238 1709054203.177435 mapread 0x00d84a thru 0x014122 (0x68d9 bytes) 000239 1709054203.177987 mapread 0x019f4b thru 0x01c8f2 (0x29a8 bytes) 000240 1709054203.182278 fallocate 0x036f62 thru 0x03a168 (0x3207 bytes) 000241 1709054203.184821 mapwrite 0x017bf7 thru 0x02535d (0xd767 bytes) 000242 1709054203.197788 fallocate 0x032ab5 thru 0x03ffff (0xd54b bytes) 000243 1709054203.201005 fallocate 0x01785b thru 0x02067a (0x8e20 bytes) 000244 1709054203.202953 fallocate 0x005734 thru 0x00ed87 (0x9654 bytes) 000245 1709054203.204804 mapwrite 0x00344b thru 0x00453e (0x10f4 bytes) 000246 1709054203.218684 trunc from 0x040000 to 0x0301d6 (0xfe29 bytes) 000248 1709054203.227390 mapread 0x02b64c thru 0x0301d6 (0x4b8b bytes) 000249 1709054203.236420 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000250 1709054203.295484 trunc from 0x0301d7 to 0x0117c0 (0x1ea16 bytes) 000251 1709054203.303338 read 0x0066ff thru 0x0117c0 (0xb0c2 bytes) 000252 1709054203.306623 trunc from 0x0117c1 to 0x0235a7 (0x11de7 bytes) 000253 1709054203.311433 mapwrite 0x02dbd5 thru 0x034651 (0x6a7d bytes) 000255 1709054203.340446 write 0x02ce8e thru 0x034b06 (0x7c79 bytes) 000257 1709054203.341530 fallocate 0x0141b2 thru 0x01bc61 (0x7ab0 bytes) 000258 1709054203.344967 trunc from 0x034b07 to 0x033b69 (0xf9d bytes) 000259 1709054203.362949 trunc from 0x033b6a to 0x02180b (0x1235e bytes) 000261 1709054203.371912 write 0x02bd63 thru 0x035a47 (0x9ce5 bytes) 000262 1709054203.372854 fallocate 0x011ba4 thru 0x013291 (0x16ee bytes) 000263 1709054203.375850 punch from 0x00268d to 0x0102e9 (0xdc5d bytes) 000264 1709054203.378750 read 0x015e11 thru 0x01e472 (0x8662 bytes) 000266 1709054203.387989 punch from 0x0107d2 to 0x01bd9c (0xb5cb bytes) 000267 1709054203.394482 read 0x0356b6 thru 0x035a47 (0x392 bytes) 000270 1709054203.395414 mapread 0x023906 thru 0x0256d7 (0x1dd2 bytes) 000271 1709054203.396030 fallocate 0x01e169 thru 0x02b10a (0xcfa2 bytes) 000272 1709054203.399484 mapread 0x00c6cc thru 0x00d273 (0xba8 bytes) 000273 1709054203.406441 fallocate 0x02e9da thru 0x039525 (0xab4c bytes) 000274 1709054203.409711 read 0x01fef6 thru 0x029234 (0x933f bytes) 000275 1709054203.410096 mapread 0x02ccf8 thru 0x035a47 (0x8d50 bytes) 000276 1709054203.410664 mapread 0x0154eb thru 0x01a1af (0x4cc5 bytes) 000278 1709054203.425433 trunc from 0x035a48 to 0x0166af (0x1f398 bytes) 000280 1709054203.442654 mapread 0x001c8e thru 0x010b39 (0xeeac bytes) 000281 1709054203.474609 write 0x019e2a thru 0x0250e2 (0xb2b9 bytes) 000283 1709054203.476022 read 0x00e745 thru 0x01b974 (0xd230 bytes) 000285 1709054203.480478 mapread 0x01fec6 thru 0x024e0d (0x4f48 bytes) 000286 1709054203.482366 write 0x032a7c thru 0x0354c0 (0x2a45 bytes) 000287 1709054203.483296 mapread 0x00e80f thru 0x01aa4d (0xc23f bytes) 000288 1709054203.483912 mapread 0x0231a9 thru 0x0312e3 (0xe13b bytes) 000289 1709054203.490308 read 0x00d8bf thru 0x014a89 (0x71cb bytes) 000290 1709054203.490661 trunc from 0x0354c1 to 0x00c164 (0x2935c bytes) 000291 1709054203.500800 write 0x02e054 thru 0x033752 (0x56ff bytes) 000292 1709054203.501828 fallocate 0x00ac52 thru 0x010e4f (0x61fe bytes) 000293 1709054203.505362 mapread 0x005bf4 thru 0x007a20 (0x1e2d bytes) 000295 1709054203.509429 mapwrite 0x02ba6c thru 0x02e355 (0x28ea bytes) 000296 1709054203.535831 read 0x01b7e3 thru 0x01f9f7 (0x4215 bytes) 000299 1709054203.536768 read 0x021a29 thru 0x02c15e (0xa736 bytes) 000300 1709054203.537077 trunc from 0x033753 to 0x02bfa6 (0x77ac bytes) 000301 1709054203.544621 mapwrite 0x0117c6 thru 0x018c33 (0x746e bytes) 000302 1709054203.557441 read 0x014c08 thru 0x0153ad (0x7a6 bytes) 000304 1709054203.558266 punch from 0x004a65 to 0x006703 (0x1c9f bytes) 000305 1709054203.560079 mapread 0x00e312 thru 0x00fc34 (0x1923 bytes) 000306 1709054203.560752 mapwrite 0x00f50e thru 0x010849 (0x133c bytes) 000309 1709054203.569358 trunc from 0x02bfa7 to 0x02522a (0x6d7c bytes) 000310 1709054203.578599 fallocate 0x01c99a thru 0x02428f (0x78f6 bytes) 000311 1709054203.581008 read 0x020f2d thru 0x02522a (0x42fe bytes) 000312 1709054203.581385 read 0x00334a thru 0x0051b8 (0x1e6f bytes) 000313 1709054203.585924 trunc from 0x02522b to 0x020bd0 (0x465a bytes) 000314 1709054203.593961 mapread 0x00dbc8 thru 0x00e034 (0x46d bytes) 000316 1709054203.595085 mapwrite 0x03910c thru 0x03b2c3 (0x21b8 bytes) 000317 1709054203.620520 fallocate 0x030a27 thru 0x03ffff (0xf5d9 bytes) 000318 1709054203.623349 read 0x022884 thru 0x02799c (0x5119 bytes) 000319 1709054203.628774 read 0x01ad5d thru 0x01ed2c (0x3fd0 bytes) 000320 1709054203.629361 write 0x037cf9 thru 0x03b0fd (0x3405 bytes) 000321 1709054203.629996 write 0x01e05a thru 0x024442 (0x63e9 bytes) 000323 1709054203.630953 mapwrite 0x02d9d7 thru 0x03a88a (0xceb4 bytes) 000324 1709054203.646975 mapread 0x02804a thru 0x02d03f (0x4ff6 bytes) 000325 1709054203.648142 trunc from 0x03b2c4 to 0x003f9d (0x37326 bytes) 000327 1709054203.660270 fallocate 0x02e613 thru 0x037f91 (0x997f bytes) 000328 1709054203.666144 fallocate 0x02d96e thru 0x02f4c1 (0x1b54 bytes) 000329 1709054203.669097 trunc from 0x02f4c2 to 0x02f11c (0x3a5 bytes) 000330 1709054203.684886 mapread 0x0117dc thru 0x01a349 (0x8b6e bytes) 000332 1709054203.724034 mapwrite 0x032a25 thru 0x03e516 (0xbaf2 bytes) 000333 1709054203.760371 punch from 0x02ce4a to 0x03617a (0x9331 bytes) 000334 1709054203.766001 mapwrite 0x03f542 thru 0x03ffff (0xabe bytes) 000335 1709054203.789898 mapread 0x02c6f6 thru 0x02ff45 (0x3850 bytes) 000336 1709054203.807818 MIRROR_EXTEND to 2 mirrors 000337 1709054204.028425 read 0x001094 thru 0x006654 (0x55c1 bytes) 000338 1709054204.031614 read 0x018994 thru 0x020ac1 (0x812e bytes) 000339 1709054204.037100 fallocate 0x00ab02 thru 0x01a4d6 (0xf9d5 bytes) 000340 1709054204.054559 MIRROR_EXTEND to 3 mirrors 000342 1709054204.306298 trunc from 0x040000 to 0x03bc58 (0x43a7 bytes) 000343 1709054204.328772 mapread 0x03a4ff thru 0x03b200 (0xd02 bytes) 000344 1709054204.334227 read 0x02957a thru 0x030f73 (0x79fa bytes) 000345 1709054204.337798 punch from 0x00e2d8 to 0x01d1c5 (0xeeee bytes) 000347 1709054204.341430 read 0x00eabe thru 0x017ecb (0x940e bytes) 000348 1709054204.346491 mapread 0x0316b1 thru 0x038337 (0x6c87 bytes) 000349 1709054204.346959 read 0x004e33 thru 0x0125a1 (0xd76f bytes) 000353 1709054204.352899 mapread 0x0195a1 thru 0x0287de (0xf23e bytes) 000354 1709054204.353944 mapread 0x01e7a2 thru 0x021f03 (0x3762 bytes) 000355 1709054204.355870 write 0x02997b thru 0x02ca4a (0x30d0 bytes) 000356 1709054204.356574 mapread 0x00f2a9 thru 0x016bd1 (0x7929 bytes) 000357 1709054204.357016 mapwrite 0x002038 thru 0x00fa74 (0xda3d bytes) 000358 1709054204.380027 MIRROR_EXTEND to 4 mirrors 000359 1709054204.564248 write 0x0125f9 thru 0x01e521 (0xbf29 bytes) 000360 1709054204.587085 fallocate 0x039f0c thru 0x03c7a3 (0x2898 bytes) 000361 1709054204.589580 mapwrite 0x02d187 thru 0x03444b (0x72c5 bytes) 000362 1709054204.612009 fallocate 0x006896 thru 0x00db4e (0x72b9 bytes) 000363 1709054204.616932 MIRROR_EXTEND to 5 mirrors 000364 1709054204.852770 mapwrite 0x032184 thru 0x03cd29 (0xaba6 bytes) 000365 1709054204.904992 trunc from 0x03cd2a to 0x026f85 (0x15da4 bytes) 000366 1709054204.912172 mapread 0x01e3a0 thru 0x026f85 (0x8be6 bytes) 000367 1709054204.916586 fallocate 0x014c3b thru 0x022eb3 (0xe279 bytes) 000368 1709054204.919353 write 0x003b1e thru 0x009172 (0x5655 bytes) 000369 1709054204.923884 mapread 0x00bea4 thru 0x00c355 (0x4b2 bytes) 000370 1709054204.925416 trunc from 0x026f86 to 0x01c9ba (0xa5cb bytes) 000371 1709054204.934909 mapwrite 0x02bae9 thru 0x039fb4 (0xe4cc bytes) 000372 1709054204.966416 punch from 0x014b23 to 0x01cdd3 (0x82b1 bytes) 000373 1709054204.971027 MIRROR_RESYNC 5 mirrors 000374 1709054205.322172 read 0x01544f thru 0x0163bf (0xf71 bytes) 000375 1709054205.323903 write 0x00035d thru 0x00c5da (0xc27e bytes) 000376 1709054205.336080 trunc from 0x039fb5 to 0x02d4b6 (0xcafe bytes) 000378 1709054205.342421 write 0x009f2c thru 0x019850 (0xf925 bytes) 000379 1709054205.346080 MIRROR_EXTEND to 6 mirrors 000380 1709054205.561783 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000382 1709054205.691494 mapwrite 0x00c95c thru 0x019df4 (0xd499 bytes) 000383 1709054205.737663 mapwrite 0x03749f thru 0x0377ac (0x30e bytes) 000384 1709054205.758625 fallocate 0x036fca thru 0x03ffff (0x9036 bytes) 000385 1709054205.760416 write 0x038b64 thru 0x03ffff (0x749c bytes) 000386 1709054205.763456 trunc from 0x040000 to 0x02b743 (0x148bc bytes) 000387 1709054205.767168 mapread 0x00c610 thru 0x01636e (0x9d5f bytes) 000389 1709054205.768104 mapwrite 0x035718 thru 0x03ffff (0xa8e8 bytes) 000390 1709054205.785814 trunc from 0x040000 to 0x03e513 (0x1aec bytes) 000391 1709054205.791508 punch from 0x0234fb to 0x027d45 (0x484b bytes) 000392 1709054205.795404 MIRROR_EXTEND to 6 mirrors 000393 1709054205.970449 punch from 0x037ce3 to 0x03e513 (0x6831 bytes) 000394 1709054205.979087 fallocate 0x00ca5b thru 0x0141e0 (0x7786 bytes) 000395 1709054205.980268 punch from 0x00ac00 to 0x00fd3b (0x513c bytes) 000396 1709054205.981560 punch from 0x02f88f to 0x0378a3 (0x8015 bytes) 000397 1709054205.982792 write 0x02bac7 thru 0x032a97 (0x6fd1 bytes) 000398 1709054205.986535 fallocate 0x01748a thru 0x01d14b (0x5cc2 bytes) 000399 1709054205.989008 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000401 1709054206.038792 fallocate 0x032ed6 thru 0x036010 (0x313b bytes) 000402 1709054206.040647 mapread 0x03d1be thru 0x03e513 (0x1356 bytes) 000404 1709054206.043851 mapwrite 0x0369c5 thru 0x03ffff (0x963b bytes) 000405 1709054206.062909 read 0x0387a5 thru 0x03f51c (0x6d78 bytes) 000406 1709054206.063354 punch from 0x02a80d to 0x037e93 (0xd687 bytes) 000409 1709054206.067525 punch from 0x02acd0 to 0x02c7bd (0x1aee bytes) 000410 1709054206.070295 read 0x00e193 thru 0x01d151 (0xefbf bytes) 000411 1709054206.075085 read 0x0025b7 thru 0x00513d (0x2b87 bytes) 000412 1709054206.079056 trunc from 0x040000 to 0x00bff7 (0x34008 bytes) 000413 1709054206.087164 MIRROR_EXTEND to 6 mirrors 000415 1709054206.216508 write 0x01dff8 thru 0x02115e (0x3167 bytes) 000416 1709054206.226663 write 0x025bb6 thru 0x03111a (0xb565 bytes) 000418 1709054206.227641 read 0x01b9e6 thru 0x021e3c (0x6457 bytes) 000419 1709054206.229200 mapread 0x030c13 thru 0x03111a (0x508 bytes) 000420 1709054206.229759 mapwrite 0x016ebe thru 0x025537 (0xe67a bytes) 000421 1709054206.241702 trunc from 0x03111b to 0x004ad4 (0x2c646 bytes) 000422 1709054206.250206 punch from 0x0022ed to 0x004ad4 (0x27e8 bytes) 000423 1709054206.251476 punch from 0x001418 to 0x002ebf (0x1aa8 bytes) 000424 1709054206.252707 trunc from 0x004ad5 to 0x002d0c (0x1dc8 bytes) 000426 1709054206.256177 read 0x0000dc thru 0x000603 (0x528 bytes) 000427 1709054206.257587 trunc from 0x002d0d to 0x027b58 (0x24e4c bytes) 000428 1709054206.260681 write 0x02af7c thru 0x03a667 (0xf6ec bytes) 000429 1709054206.261545 write 0x025a15 thru 0x025bae (0x19a bytes) 000430 1709054206.264898 mapread 0x000078 thru 0x001f72 (0x1efb bytes) 000431 1709054206.265839 punch from 0x0301ae to 0x03200e (0x1e61 bytes) 000432 1709054206.269873 read 0x0356de thru 0x03a667 (0x4f8a bytes) 000433 1709054206.270323 trunc from 0x03a668 to 0x02dd98 (0xc8cf bytes) 000434 1709054206.274588 punch from 0x00cf85 to 0x015408 (0x8484 bytes) 000436 1709054206.276698 punch from 0x01b22d to 0x01bd22 (0xaf6 bytes) 000437 1709054206.278346 write 0x01e48c thru 0x01ef8c (0xb01 bytes) 000438 1709054206.278865 fallocate 0x00766a thru 0x00a0ba (0x2a51 bytes) 000439 1709054206.280269 write 0x01d8ce thru 0x01ed52 (0x1485 bytes) 000440 1709054206.280795 write 0x01e176 thru 0x021e0b (0x3c96 bytes) 000441 1709054206.281407 fallocate 0x004d67 thru 0x00a1ca (0x5464 bytes) 000443 1709054206.283091 mapread 0x01dfab thru 0x022f7a (0x4fd0 bytes) 000444 1709054206.283685 write 0x031578 thru 0x0394f7 (0x7f80 bytes) 000445 1709054206.284411 read 0x015e77 thru 0x01ae38 (0x4fc2 bytes) 000446 1709054206.287440 trunc from 0x0394f8 to 0x033e04 (0x56f3 bytes) 000447 1709054206.291803 punch from 0x00486e to 0x009f8a (0x571d bytes) 000448 1709054206.293551 read 0x0046c2 thru 0x00ad50 (0x668f bytes) 000449 1709054206.296298 fallocate 0x010c34 thru 0x01bfda (0xb3a7 bytes) 000450 1709054206.297820 write 0x001ac3 thru 0x010365 (0xe8a3 bytes) 000451 1709054206.298576 punch from 0x029758 to 0x0316c4 (0x7f6d bytes) 000452 1709054206.304829 punch from 0x02539d to 0x02f3ed (0xa051 bytes) 000453 1709054206.306829 read 0x016227 thru 0x020cdb (0xaab5 bytes) 000454 1709054206.307612 trunc from 0x033e05 to 0x03e594 (0xa790 bytes) 000455 1709054206.313813 fallocate 0x02145c thru 0x02fb07 (0xe6ac bytes) 000456 1709054206.317278 fallocate 0x014bdb thru 0x01c1a1 (0x75c7 bytes) 000457 1709054206.319257 trunc from 0x03e595 to 0x0084ec (0x360a8 bytes) 000458 1709054206.327095 punch from 0x00745a to 0x0084ec (0x1093 bytes) 000459 1709054206.328955 mapread 0x0072a9 thru 0x0084ec (0x1244 bytes) 000460 1709054206.334226 write 0x004986 thru 0x0099e2 (0x505d bytes) 000461 1709054206.334921 punch from 0x0031bd to 0x0091f4 (0x6038 bytes) 000463 1709054206.341281 trunc from 0x0099e3 to 0x00ade4 (0x1402 bytes) 000464 1709054206.346209 trunc from 0x00ade5 to 0x00b80f (0xa2b bytes) 000466 1709054206.353806 MIRROR_EXTEND to 7 mirrors 000467 1709054206.518488 mapwrite 0x030d0e thru 0x03ffff (0xf2f2 bytes) 000469 1709054206.547969 mapwrite 0x0290b1 thru 0x035e3e (0xcd8e bytes) 000471 1709054206.556767 mapwrite 0x0373b8 thru 0x03a8ed (0x3536 bytes) 000472 1709054206.560925 mapwrite 0x00b089 thru 0x01548f (0xa407 bytes) 000473 1709054206.576699 mapwrite 0x0271ae thru 0x0362ff (0xf152 bytes) 000474 1709054206.584459 fallocate 0x003abd thru 0x006ef4 (0x3438 bytes) 000475 1709054206.586027 mapwrite 0x02d2a4 thru 0x03ad1c (0xda79 bytes) 000476 1709054206.597011 mapwrite 0x02538b thru 0x02ba80 (0x66f6 bytes) 000477 1709054206.603442 mapwrite 0x025970 thru 0x02b067 (0x56f8 bytes) 000478 1709054206.609442 fallocate 0x035337 thru 0x03ffff (0xacc9 bytes) 000479 1709054206.611363 mapread 0x01e77c thru 0x02bf2c (0xd7b1 bytes) 000480 1709054206.615006 read 0x00e927 thru 0x019f71 (0xb64b bytes) 000482 1709054206.615777 mapread 0x0180f4 thru 0x01ae91 (0x2d9e bytes) 000483 1709054206.616296 read 0x008f85 thru 0x016f72 (0xdfee bytes) 000484 1709054206.616610 read 0x03ba06 thru 0x03ffff (0x45fa bytes) 000485 1709054206.616935 mapread 0x016156 thru 0x0164ce (0x379 bytes) 000487 1709054206.618611 write 0x01611e thru 0x019f2a (0x3e0d bytes) 000488 1709054206.619248 mapread 0x005605 thru 0x00bb8a (0x6586 bytes) 000490 1709054206.620023 fallocate 0x00d1d7 thru 0x019f35 (0xcd5f bytes) 000491 1709054206.622207 write 0x004cfd thru 0x012327 (0xd62b bytes) 000492 1709054206.623008 punch from 0x021a02 to 0x0293a8 (0x79a7 bytes) 000493 1709054206.624927 read 0x023f5d thru 0x02a298 (0x633c bytes) 000494 1709054206.628700 write 0x02f7be thru 0x035074 (0x58b7 bytes) 000495 1709054206.629402 punch from 0x01260b to 0x020336 (0xdd2c bytes) 000496 1709054206.634008 punch from 0x00df65 to 0x010b5e (0x2bfa bytes) 000497 1709054206.635843 punch from 0x0092ed to 0x012603 (0x9317 bytes) 000498 1709054206.637377 punch from 0x01ffff to 0x02fe94 (0xfe96 bytes) 000501 1709054206.639832 mapwrite 0x01ef45 thru 0x0271b9 (0x8275 bytes) 000502 1709054206.664406 punch from 0x00a72a to 0x00ee34 (0x470b bytes) 000504 1709054206.666949 mapwrite 0x00e45e thru 0x01a34b (0xbeee bytes) 000505 1709054206.692054 mapwrite 0x01649c thru 0x0210fb (0xac60 bytes) 000507 1709054206.707178 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000509 1709054206.761989 read 0x0068da thru 0x0092e5 (0x2a0c bytes) 000510 1709054206.763942 mapwrite 0x01a972 thru 0x0222c4 (0x7953 bytes) 000511 1709054206.778933 fallocate 0x03f246 thru 0x03ffff (0xdba bytes) 000512 1709054206.782098 MIRROR_EXTEND to 7 mirrors 000513 1709054206.935612 mapread 0x003d40 thru 0x00f4af (0xb770 bytes) 000514 1709054206.944781 punch from 0x0371c9 to 0x03ffff (0x8e37 bytes) 000515 1709054206.955564 mapwrite 0x002fc6 thru 0x007043 (0x407e bytes) 000516 1709054206.964562 read 0x018ae5 thru 0x01d09b (0x45b7 bytes) 000517 1709054206.967350 punch from 0x03c89a to 0x03ffff (0x3766 bytes) 000518 1709054206.969081 write 0x02bf59 thru 0x039b35 (0xdbdd bytes) 000519 1709054206.974973 write 0x01a22a thru 0x024001 (0x9dd8 bytes) 000520 1709054206.975888 trunc from 0x040000 to 0x02c198 (0x13e67 bytes) 000522 1709054206.990063 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000524 1709054207.062086 write 0x029979 thru 0x02c683 (0x2d0b bytes) 000525 1709054207.067224 MIRROR_EXTEND to 7 mirrors 000526 1709054207.222392 fallocate 0x037bc9 thru 0x03f81d (0x7c55 bytes) 000527 1709054207.231141 trunc from 0x02c684 to 0x020c62 (0xba21 bytes) 000528 1709054207.236490 fallocate 0x036702 thru 0x03ffff (0x98fe bytes) 000529 1709054207.237822 mapwrite 0x02b025 thru 0x035f73 (0xaf4f bytes) 000530 1709054207.249674 write 0x02931e thru 0x037846 (0xe529 bytes) 000532 1709054207.250954 fallocate 0x0112b1 thru 0x020e7c (0xfbcc bytes) 000533 1709054207.252241 trunc from 0x037847 to 0x0366f2 (0x1154 bytes) 000534 1709054207.255418 fallocate 0x0349cd thru 0x03a1c5 (0x57f9 bytes) 000535 1709054207.256592 fallocate 0x03a417 thru 0x03ffff (0x5be9 bytes) 000537 1709054207.257748 punch from 0x01d0bb to 0x02b32a (0xe270 bytes) 000538 1709054207.260748 mapwrite 0x019c0e thru 0x01b851 (0x1c44 bytes) 000539 1709054207.267035 trunc from 0x0366f3 to 0x0278b0 (0xee42 bytes) 000540 1709054207.272273 trunc from 0x0278b1 to 0x03b353 (0x13aa3 bytes) 000542 1709054207.277232 write 0x00f9ac thru 0x010f7e (0x15d3 bytes) 000543 1709054207.281172 punch from 0x007ef2 to 0x01538a (0xd499 bytes) 000544 1709054207.285782 mapread 0x002d1d thru 0x002fab (0x28f bytes) 000545 1709054207.289966 write 0x00279d thru 0x00c652 (0x9eb6 bytes) 000546 1709054207.290856 read 0x036ccb thru 0x03b353 (0x4689 bytes) 000547 1709054207.293583 punch from 0x031ac0 to 0x034330 (0x2871 bytes) 000548 1709054207.296620 MIRROR_RESYNC 7 mirrors 000549 1709054207.571484 fallocate 0x037835 thru 0x0395ad (0x1d79 bytes) 000550 1709054207.582286 write 0x026dae thru 0x02a612 (0x3865 bytes) 000551 1709054207.584884 mapread 0x03b079 thru 0x03b353 (0x2db bytes) 000553 1709054207.587569 mapwrite 0x029fbc thru 0x02b1e2 (0x1227 bytes) 000554 1709054207.595741 read 0x01a225 thru 0x01c69e (0x247a bytes) 000555 1709054207.597434 trunc from 0x03b354 to 0x01d9bd (0x1d996 bytes) 000556 1709054207.605920 mapread 0x00ee07 thru 0x018640 (0x983a bytes) 000557 1709054207.611128 trunc from 0x01d9be to 0x018fec (0x49d1 bytes) 000558 1709054207.616242 mapread 0x018fb3 thru 0x018fec (0x3a bytes) 000559 1709054207.616782 read 0x013171 thru 0x018fec (0x5e7c bytes) 000560 1709054207.617053 punch from 0x00d8d9 to 0x017def (0xa517 bytes) 000561 1709054207.620060 MIRROR_EXTEND to 8 mirrors 000562 1709054207.782921 punch from 0x0013a7 to 0x00206b (0xcc5 bytes) 000563 1709054207.799810 MIRROR_EXTEND to 9 mirrors 000566 1709054207.941985 write 0x0192f5 thru 0x02101d (0x7d29 bytes) 000567 1709054207.956834 punch from 0x00782d to 0x014ba0 (0xd374 bytes) 000568 1709054207.959020 fallocate 0x012fcb thru 0x01ceaa (0x9ee0 bytes) 000569 1709054207.960950 write 0x03bcd0 thru 0x03ffff (0x4330 bytes) 000570 1709054207.961780 punch from 0x002ff0 to 0x012e48 (0xfe59 bytes) 000571 1709054207.964378 fallocate 0x03f042 thru 0x03fe3e (0xdfd bytes) 000575 1709054207.966861 trunc from 0x040000 to 0x02826c (0x17d93 bytes) 000576 1709054207.982792 MIRROR_RESYNC 9 mirrors 000577 1709054208.487277 trunc from 0x02826d to 0x002157 (0x26115 bytes) 000579 1709054208.499489 read 0x000c91 thru 0x002157 (0x14c7 bytes) 000580 1709054208.501494 trunc from 0x002158 to 0x0042bf (0x2168 bytes) 000581 1709054208.506701 punch from 0x0018f7 to 0x0042bf (0x29c9 bytes) 000582 1709054208.508396 punch from 0x00376b to 0x0042bf (0xb55 bytes) 000583 1709054208.509763 mapwrite 0x016ba7 thru 0x018a5b (0x1eb5 bytes) 000585 1709054208.526054 mapwrite 0x032058 thru 0x039d96 (0x7d3f bytes) 000589 1709054208.543179 mapwrite 0x0053bd thru 0x00d3ee (0x8032 bytes) 000591 1709054208.557602 fallocate 0x0325b4 thru 0x03ebd9 (0xc626 bytes) 000593 1709054208.559841 read 0x022e51 thru 0x02810c (0x52bc bytes) 000594 1709054208.564099 write 0x03e77b thru 0x03ffff (0x1885 bytes) 000595 1709054208.564771 mapwrite 0x031c36 thru 0x03405f (0x242a bytes) 000597 1709054208.571941 read 0x014ea8 thru 0x01e5cf (0x9728 bytes) 000598 1709054208.575965 mapwrite 0x00cfca thru 0x0163e9 (0x9420 bytes) 000599 1709054208.583084 mapread 0x02b4b8 thru 0x02f607 (0x4150 bytes) 000600 1709054208.585582 MIRROR_EXTEND to 10 mirrors 000601 1709054208.706765 punch from 0x02593e to 0x035346 (0xfa09 bytes) 000602 1709054208.719741 MIRROR_RESYNC 10 mirrors 000603 1709054209.055692 mapread 0x002d77 thru 0x006daa (0x4034 bytes) 000605 1709054209.062683 read 0x01362b thru 0x014a62 (0x1438 bytes) 000608 1709054209.065106 write 0x010561 thru 0x0148cd (0x436d bytes) 000610 1709054209.079907 read 0x0010f4 thru 0x006309 (0x5216 bytes) 000611 1709054209.082531 mapread 0x03f2fd thru 0x03ffff (0xd03 bytes) 000612 1709054209.083076 punch from 0x01c600 to 0x02c41c (0xfe1d bytes) 000613 1709054209.084806 mapwrite 0x031f05 thru 0x03ef0f (0xd00b bytes) 000614 1709054209.094896 MIRROR_RESYNC 10 mirrors 000615 1709054209.443317 fallocate 0x02ffc5 thru 0x03fcf6 (0xfd32 bytes) 000617 1709054209.453002 mapread 0x03c559 thru 0x03ffff (0x3aa7 bytes) 000618 1709054209.458071 read 0x0095f1 thru 0x0166c0 (0xd0d0 bytes) 000620 1709054209.462319 MIRROR_RESYNC 10 mirrors 000621 1709054209.820662 trunc from 0x040000 to 0x013543 (0x2cabc bytes) 000622 1709054209.845851 MIRROR_EXTEND to 11 mirrors 000623 1709054209.971257 MIRROR_RESYNC 11 mirrors 000624 1709054210.348857 fallocate 0x01bd8e thru 0x02969e (0xd911 bytes) 000625 1709054210.368718 MIRROR_EXTEND to 12 mirrors 000626 1709054210.524156 MIRROR_EXTEND to 13 mirrors 000629 1709054210.674959 mapread 0x007493 thru 0x007562 (0xd0 bytes) 000631 1709054210.676963 trunc from 0x02969f to 0x038349 (0xecab bytes) 000632 1709054210.693044 mapread 0x020553 thru 0x026e5c (0x690a bytes) 000633 1709054210.699225 MIRROR_EXTEND to 14 mirrors 000634 1709054210.841886 read 0x016891 thru 0x01f8d1 (0x9041 bytes) 000635 1709054210.846085 read 0x00bc7b thru 0x00cdc7 (0x114d bytes) 000636 1709054210.852073 mapread 0x00e9fc thru 0x012dd2 (0x43d7 bytes) 000637 1709054210.856330 read 0x011f84 thru 0x01f89f (0xd91c bytes) 000639 1709054210.861549 mapread 0x01f2a4 thru 0x020ff7 (0x1d54 bytes) 000640 1709054210.870855 MIRROR_RESYNC 14 mirrors 000641 1709054211.342280 mapwrite 0x024554 thru 0x02af8b (0x6a38 bytes) 000642 1709054211.377501 write 0x017d00 thru 0x020b39 (0x8e3a bytes) 000645 1709054211.380853 mapread 0x038034 thru 0x038349 (0x316 bytes) 000646 1709054211.383068 mapread 0x036d5e thru 0x038349 (0x15ec bytes) 000647 1709054211.383491 write 0x03697f thru 0x03f0dc (0x875e bytes) 000649 1709054211.384232 mapwrite 0x001f0c thru 0x010a5c (0xeb51 bytes) 000650 1709054211.394486 read 0x001356 thru 0x00a229 (0x8ed4 bytes) 000651 1709054211.394912 trunc from 0x03f0dd to 0x02b5e9 (0x13af3 bytes) 000653 1709054211.405774 write 0x0230a6 thru 0x02fd4e (0xcca9 bytes) 000654 1709054211.406786 mapread 0x002f9a thru 0x00dd28 (0xad8f bytes) 000656 1709054211.407686 punch from 0x023580 to 0x0298d1 (0x6352 bytes) 000658 1709054211.414025 write 0x01f941 thru 0x022ea3 (0x3563 bytes) 000660 1709054211.414890 read 0x008f01 thru 0x014350 (0xb450 bytes) 000662 1709054211.415538 mapread 0x0120d5 thru 0x021bc8 (0xfaf4 bytes) 000663 1709054211.416164 punch from 0x01b218 to 0x026e9b (0xbc84 bytes) 000664 1709054211.420232 write 0x03823e thru 0x03eba3 (0x6966 bytes) 000665 1709054211.421035 trunc from 0x03eba4 to 0x021974 (0x1d22f bytes) 000666 1709054211.427532 mapwrite 0x02186b thru 0x02c742 (0xaed8 bytes) 000667 1709054211.444769 read 0x01c789 thru 0x01f1e9 (0x2a61 bytes) 000668 1709054211.445206 trunc from 0x02c743 to 0x03c9b2 (0x10270 bytes) 000669 1709054211.449157 read 0x0307ac thru 0x033e80 (0x36d5 bytes) 000670 1709054211.451817 fallocate 0x02a2c4 thru 0x02cb08 (0x2845 bytes) 000671 1709054211.454608 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000672 1709054211.505469 write 0x0395e2 thru 0x03ffff (0x6a1e bytes) 000673 1709054211.508955 read 0x02259a thru 0x0249d7 (0x243e bytes) 000674 1709054211.510992 read 0x02ef7a thru 0x0372f4 (0x837b bytes) 000675 1709054211.513561 fallocate 0x004395 thru 0x00966b (0x52d7 bytes) 000676 1709054211.515257 read 0x03591b thru 0x03ffff (0xa6e5 bytes) 000677 1709054211.515667 punch from 0x03f827 to 0x03ffff (0x7d9 bytes) 000678 1709054211.520903 mapread 0x035072 thru 0x03ffff (0xaf8e bytes) 000680 1709054211.523157 punch from 0x03639a to 0x036ff1 (0xc58 bytes) 000681 1709054211.526784 MIRROR_EXTEND to 14 mirrors 000682 1709054211.691024 read 0x000f2f thru 0x007724 (0x67f6 bytes) 000684 1709054211.699209 read 0x00e4ee thru 0x018892 (0xa3a5 bytes) 000686 1709054211.706111 trunc from 0x040000 to 0x03d436 (0x2bc9 bytes) 000687 1709054211.726491 mapread 0x02f250 thru 0x03b854 (0xc605 bytes) 000688 1709054211.743273 MIRROR_EXTEND to 15 mirrors 000689 1709054211.898248 MIRROR_EXTEND to 16 mirrors 000691 1709054212.057084 mapread 0x009694 thru 0x018219 (0xeb86 bytes) 000692 1709054212.063885 read 0x02ac0d thru 0x02c6df (0x1ad3 bytes) 000693 1709054212.065219 read 0x003cc3 thru 0x005d3a (0x2078 bytes) 000694 1709054212.067598 read 0x027a3a thru 0x036b16 (0xf0dd bytes) 000695 1709054212.067846 mapwrite 0x000263 thru 0x00598f (0x572d bytes) 000697 1709054212.111085 punch from 0x025b57 to 0x026898 (0xd42 bytes) 000699 1709054212.114030 mapread 0x00d72f thru 0x018b8e (0xb460 bytes) 000700 1709054212.131653 trunc from 0x03d437 to 0x0303d7 (0xd05f bytes) 000701 1709054212.145176 write 0x0192b6 thru 0x01972a (0x475 bytes) 000702 1709054212.147068 fallocate 0x03f3c5 thru 0x03ffff (0xc3b bytes) 000704 1709054212.148915 fallocate 0x02436b thru 0x026658 (0x22ee bytes) 000705 1709054212.150404 read 0x00946c thru 0x00c9d7 (0x356c bytes) 000706 1709054212.152993 fallocate 0x0064ae thru 0x0082e4 (0x1e37 bytes) 000707 1709054212.154495 trunc from 0x0303d8 to 0x03ce2e (0xca57 bytes) 000708 1709054212.158697 read 0x0105c8 thru 0x01e1ce (0xdc07 bytes) 000709 1709054212.160292 MIRROR_RESYNC 16 mirrors 000710 1709054212.768652 mapwrite 0x02a0ab thru 0x038ae0 (0xea36 bytes) 000711 1709054212.807915 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000713 1709054212.855224 fallocate 0x005461 thru 0x0093c4 (0x3f64 bytes) 000714 1709054212.856625 read 0x02e8e5 thru 0x038209 (0x9925 bytes) 000715 1709054212.858668 trunc from 0x03ce2f to 0x0367bf (0x666f bytes) 000716 1709054212.865860 punch from 0x022ec9 to 0x026d59 (0x3e91 bytes) 000717 1709054212.867037 fallocate 0x026b2b thru 0x02bad9 (0x4faf bytes) 000718 1709054212.868228 fallocate 0x0031b0 thru 0x00608f (0x2ee0 bytes) 000719 1709054212.869450 punch from 0x01d374 to 0x01e322 (0xfaf bytes) 000720 1709054212.870626 write 0x017352 thru 0x01b611 (0x42c0 bytes) 000721 1709054212.872998 mapread 0x025a1b thru 0x027e3c (0x2422 bytes) 000722 1709054212.875410 mapread 0x02a009 thru 0x02dbcc (0x3bc4 bytes) 000723 1709054212.875907 mapwrite 0x02bc99 thru 0x03b0bf (0xf427 bytes) 000725 1709054212.891621 write 0x03e28e thru 0x03ffff (0x1d72 bytes) 000726 1709054212.892148 write 0x037b80 thru 0x03ffff (0x8480 bytes) 000727 1709054212.892640 write 0x035bc5 thru 0x03ab07 (0x4f43 bytes) 000728 1709054212.893155 fallocate 0x011a20 thru 0x01295e (0xf3f bytes) 000729 1709054212.896399 MIRROR_EXTEND to 16 mirrors 000730 1709054213.029168 MIRROR_SPLIT mirror 12 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000732 1709054213.101299 mapwrite 0x034349 thru 0x03ffff (0xbcb7 bytes) 000733 1709054213.135758 punch from 0x004ec5 to 0x0102a2 (0xb3de bytes) 000735 1709054213.137686 punch from 0x03e2df to 0x03ffff (0x1d21 bytes) 000736 1709054213.139162 fallocate 0x0089a2 thru 0x00afe5 (0x2644 bytes) 000738 1709054213.142308 read 0x02471b thru 0x02ae15 (0x66fb bytes) 000739 1709054213.145637 punch from 0x0082ff to 0x0144d6 (0xc1d8 bytes) 000740 1709054213.147143 read 0x008952 thru 0x00dc80 (0x532f bytes) 000741 1709054213.150739 trunc from 0x040000 to 0x00214c (0x3deb3 bytes) 000742 1709054213.164345 trunc from 0x00214d to 0x018955 (0x16809 bytes) 000744 1709054213.169831 trunc from 0x018956 to 0x01d24b (0x48f6 bytes) 000745 1709054213.176883 MIRROR_EXTEND to 16 mirrors 000746 1709054213.300352 mapread 0x0043a8 thru 0x00ea79 (0xa6d2 bytes) 000747 1709054213.309330 trunc from 0x01d24c to 0x0359f9 (0x187ae bytes) 000749 1709054213.331439 punch from 0x01a2a0 to 0x022dce (0x8b2f bytes) 000750 1709054213.334362 MIRROR_SPLIT mirror 15 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000754 1709054213.387555 mapread 0x00905f thru 0x011fee (0x8f90 bytes) 000755 1709054213.394927 punch from 0x01d3a5 to 0x02856a (0xb1c6 bytes) 000756 1709054213.397061 fallocate 0x028408 thru 0x036990 (0xe589 bytes) 000757 1709054213.399277 punch from 0x002033 to 0x00ee47 (0xce15 bytes) 000759 1709054213.401206 write 0x0188e5 thru 0x021f85 (0x96a1 bytes) 000760 1709054213.406627 trunc from 0x0359fa to 0x02395d (0x1209c bytes) 000761 1709054213.413240 MIRROR_RESYNC 15 mirrors 000763 1709054214.083440 punch from 0x022d82 to 0x02395d (0xbdc bytes) 000764 1709054214.094035 mapwrite 0x00edf0 thru 0x013bca (0x4ddb bytes) 000765 1709054214.110013 trunc from 0x02395e to 0x006371 (0x1d5ec bytes) 000766 1709054214.125880 trunc from 0x006372 to 0x02603f (0x1fcce bytes) 000769 1709054214.134380 write 0x037f4e thru 0x03ffff (0x80b2 bytes) 000770 1709054214.135472 mapread 0x0379b9 thru 0x03ee7d (0x74c5 bytes) 000771 1709054214.136122 mapread 0x023e30 thru 0x0255e3 (0x17b4 bytes) 000772 1709054214.142213 mapwrite 0x00281c thru 0x00693e (0x4123 bytes) 000773 1709054214.155753 MIRROR_EXTEND to 16 mirrors 000774 1709054214.345864 MIRROR_RESYNC 16 mirrors 000775 1709054214.769368 read 0x005847 thru 0x00e84e (0x9008 bytes) 000777 1709054214.772328 mapread 0x02432f thru 0x027f30 (0x3c02 bytes) 000778 1709054214.779098 MIRROR_SPLIT mirror 9 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000779 1709054214.892878 read 0x0175e3 thru 0x023e73 (0xc891 bytes) 000782 1709054214.896334 punch from 0x0282e6 to 0x037150 (0xee6b bytes) 000785 1709054214.910896 read 0x03a732 thru 0x03ffff (0x58ce bytes) 000786 1709054214.913330 punch from 0x00da60 to 0x015691 (0x7c32 bytes) 000787 1709054214.914971 fallocate 0x03d59a thru 0x03f23e (0x1ca5 bytes) 000788 1709054214.916419 punch from 0x020d61 to 0x021dc3 (0x1063 bytes) 000790 1709054214.918109 mapwrite 0x024efe thru 0x0267b3 (0x18b6 bytes) 000792 1709054214.928712 read 0x025853 thru 0x02dbb0 (0x835e bytes) 000793 1709054214.931992 mapread 0x02f33d thru 0x036a64 (0x7728 bytes) 000794 1709054214.934567 MIRROR_SPLIT mirror 11 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000796 1709054215.002604 trunc from 0x040000 to 0x02fdb2 (0x1024d bytes) 000797 1709054215.022447 read 0x0194c8 thru 0x01ebff (0x5738 bytes) 000799 1709054215.027779 mapwrite 0x006deb thru 0x016aae (0xfcc4 bytes) 000801 1709054215.061684 read 0x017c30 thru 0x01ed72 (0x7143 bytes) 000802 1709054215.063687 read 0x022fad thru 0x02eb3e (0xbb92 bytes) 000803 1709054215.069279 trunc from 0x02fdb3 to 0x029e13 (0x5f9f bytes) 000804 1709054215.079765 trunc from 0x029e14 to 0x023784 (0x668f bytes) 000807 1709054215.087890 fallocate 0x011465 thru 0x011b24 (0x6c0 bytes) 000808 1709054215.091335 mapwrite 0x022164 thru 0x02e81c (0xc6b9 bytes) 000809 1709054215.153626 write 0x017aa5 thru 0x025c2a (0xe186 bytes) 000812 1709054215.155630 trunc from 0x02e81d to 0x02dd7e (0xa9e bytes) 000813 1709054215.171567 fallocate 0x00a7ef thru 0x01051d (0x5d2f bytes) 000814 1709054215.179625 write 0x008b9a thru 0x017feb (0xf452 bytes) 000815 1709054215.180661 trunc from 0x02dd7f to 0x011ea2 (0x1bedc bytes) 000817 1709054215.192372 punch from 0x006a9c to 0x00aa59 (0x3fbe bytes) 000818 1709054215.208932 fallocate 0x012e6c thru 0x019272 (0x6407 bytes) 000819 1709054215.214636 trunc from 0x019273 to 0x027341 (0xe0cf bytes) 000820 1709054215.225102 punch from 0x01e22f to 0x025a46 (0x7818 bytes) 000821 1709054215.227943 mapwrite 0x00b663 thru 0x00defa (0x2898 bytes) 000822 1709054215.241107 read 0x018e6a thru 0x026eb1 (0xe048 bytes) 000823 1709054215.246300 punch from 0x00fba3 to 0x011c16 (0x2074 bytes) 000824 1709054215.250227 write 0x035f29 thru 0x03ffff (0xa0d7 bytes) 000825 1709054215.251799 fallocate 0x0398da thru 0x03ffff (0x6726 bytes) 000826 1709054215.254934 punch from 0x00178f to 0x0072c0 (0x5b32 bytes) 000829 1709054215.257300 punch from 0x0318be to 0x03762a (0x5d6d bytes) 000830 1709054215.263226 mapread 0x0078db thru 0x015790 (0xdeb6 bytes) 000833 1709054215.288827 trunc from 0x040000 to 0x02bb75 (0x1448a bytes) 000834 1709054215.294856 punch from 0x000786 to 0x0034c2 (0x2d3d bytes) 000835 1709054215.297271 write 0x019bda thru 0x02087a (0x6ca1 bytes) 000836 1709054215.298108 mapwrite 0x001426 thru 0x00a088 (0x8c63 bytes) 000837 1709054215.339420 punch from 0x02b526 to 0x02bb75 (0x650 bytes) 000838 1709054215.344477 mapread 0x006322 thru 0x0158f2 (0xf5d1 bytes) 000839 1709054215.345199 trunc from 0x02bb76 to 0x00aa2e (0x21147 bytes) 000840 1709054215.356636 fallocate 0x03d848 thru 0x03ffff (0x27b8 bytes) 000841 1709054215.360423 mapwrite 0x027ec2 thru 0x028a63 (0xba2 bytes) 000842 1709054215.390020 read 0x0275ce thru 0x028a63 (0x1496 bytes) 000843 1709054215.390403 read 0x00d811 thru 0x01a20a (0xc9fa bytes) 000844 1709054215.394847 mapread 0x0283fb thru 0x028a63 (0x669 bytes) 000845 1709054215.395473 fallocate 0x0047be thru 0x004909 (0x14c bytes) 000846 1709054215.399358 MIRROR_RESYNC 14 mirrors 000847 1709054216.141345 write 0x03de81 thru 0x03ffff (0x217f bytes) 000848 1709054216.170330 write 0x028581 thru 0x0311a6 (0x8c26 bytes) 000849 1709054216.180129 trunc from 0x040000 to 0x02365e (0x1c9a1 bytes) 000850 1709054216.202960 trunc from 0x02365f to 0x01cb54 (0x6b0a bytes) 000851 1709054216.209276 fallocate 0x030f49 thru 0x03622b (0x52e3 bytes) 000853 1709054216.211703 punch from 0x001448 to 0x0044be (0x3077 bytes) 000854 1709054216.214829 fallocate 0x02e2e7 thru 0x03a57f (0xc299 bytes) 000855 1709054216.217506 write 0x019e8a thru 0x02895f (0xead6 bytes) 000856 1709054216.221814 trunc from 0x028960 to 0x01812f (0x10830 bytes) 000857 1709054216.230162 trunc from 0x018130 to 0x022d5a (0xac2b bytes) 000858 1709054216.237736 mapwrite 0x010f4f thru 0x016ba5 (0x5c57 bytes) 000860 1709054216.248934 mapwrite 0x006f1d thru 0x01019f (0x9283 bytes) 000862 1709054216.264722 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000863 1709054216.349599 punch from 0x016f4f to 0x0228f8 (0xb9aa bytes) 000864 1709054216.351907 mapwrite 0x0102d8 thru 0x019218 (0x8f41 bytes) 000865 1709054216.374533 trunc from 0x022d5b to 0x032714 (0xf9ba bytes) 000866 1709054216.388029 mapread 0x023297 thru 0x02796c (0x46d6 bytes) 000869 1709054216.396452 read 0x019463 thru 0x026732 (0xd2d0 bytes) 000870 1709054216.399846 punch from 0x0107fb to 0x0185f7 (0x7dfd bytes) 000871 1709054216.403345 trunc from 0x032715 to 0x0232a3 (0xf471 bytes) 000872 1709054216.417974 MIRROR_RESYNC 13 mirrors 000873 1709054217.655383 write 0x00f9e4 thru 0x0177b0 (0x7dcd bytes) 000875 1709054217.688753 mapread 0x02287c thru 0x0232a3 (0xa28 bytes) 000877 1709054217.695886 punch from 0x010115 to 0x01b5da (0xb4c6 bytes) 000878 1709054217.704330 punch from 0x00951d to 0x0174e1 (0xdfc5 bytes) 000879 1709054217.707731 trunc from 0x0232a4 to 0x0146bb (0xebe8 bytes) 000880 1709054217.717260 MIRROR_EXTEND to 14 mirrors 000881 1709054217.981414 fallocate 0x033890 thru 0x03ffff (0xc770 bytes) 000882 1709054218.009346 read 0x01c5b7 thru 0x01fc0a (0x3654 bytes) 000883 1709054218.013970 read 0x039ece thru 0x03bae5 (0x1c18 bytes) 000884 1709054218.018170 mapread 0x00757d thru 0x0096c2 (0x2146 bytes) 000885 1709054218.025113 read 0x033cc8 thru 0x03796c (0x3ca5 bytes) 000886 1709054218.030778 mapread 0x0331ba thru 0x03ffff (0xce46 bytes) 000887 1709054218.031727 mapread 0x00a971 thru 0x00d3ad (0x2a3d bytes) 000888 1709054218.044102 MIRROR_EXTEND to 15 mirrors 000889 1709054218.329455 fallocate 0x032ef3 thru 0x039a93 (0x6ba1 bytes) 000890 1709054218.351054 MIRROR_RESYNC 15 mirrors 000891 1709054219.294764 mapread 0x014a47 thru 0x01a34a (0x5904 bytes) 000892 1709054219.305852 MIRROR_RESYNC 15 mirrors 000893 1709054219.484782 mapread 0x038c81 thru 0x03ffff (0x737f bytes) 000894 1709054219.497609 punch from 0x018bb6 to 0x01d482 (0x48cd bytes) 000895 1709054219.513792 fallocate 0x027c0e thru 0x02bb06 (0x3ef9 bytes) 000896 1709054219.516643 fallocate 0x0087b8 thru 0x017513 (0xed5c bytes) 000897 1709054219.520514 write 0x034424 thru 0x03cbc5 (0x87a2 bytes) 000899 1709054219.526902 write 0x032421 thru 0x03c8bc (0xa49c bytes) 000900 1709054219.527743 read 0x0060dd thru 0x00e773 (0x8697 bytes) 000901 1709054219.531216 mapread 0x03c833 thru 0x03ffff (0x37cd bytes) 000902 1709054219.534995 mapread 0x019893 thru 0x026f58 (0xd6c6 bytes) 000903 1709054219.544352 mapwrite 0x00a868 thru 0x01923a (0xe9d3 bytes) 000904 1709054219.563587 fallocate 0x03f3fb thru 0x03ffff (0xc05 bytes) 000905 1709054219.566761 trunc from 0x040000 to 0x01e43a (0x21bc5 bytes) 000906 1709054219.580887 trunc from 0x01e43b to 0x005458 (0x18fe2 bytes) 000909 1709054219.587685 fallocate 0x03d716 thru 0x03ffff (0x28ea bytes) 000910 1709054219.589897 read 0x0052f0 thru 0x005458 (0x169 bytes) 000911 1709054219.592523 trunc from 0x005459 to 0x003c71 (0x17e7 bytes) 000912 1709054219.598902 write 0x02b9a1 thru 0x02e5fd (0x2c5d bytes) 000913 1709054219.600666 write 0x004036 thru 0x01300d (0xefd8 bytes) 000914 1709054219.608032 trunc from 0x02e5fe to 0x0135ee (0x1b00f bytes) 000915 1709054219.614598 fallocate 0x03bb85 thru 0x03ffff (0x447b bytes) 000917 1709054219.619579 write 0x034c24 thru 0x03ebb9 (0x9f96 bytes) 000919 1709054219.620711 punch from 0x004026 to 0x0136c2 (0xf69d bytes) 000920 1709054219.627043 write 0x01760c thru 0x01da88 (0x647d bytes) 000921 1709054219.636554 MIRROR_EXTEND to 16 mirrors 000922 1709054219.882544 mapwrite 0x039476 thru 0x03e073 (0x4bfe bytes) 000923 1709054219.943797 MIRROR_SPLIT mirror 14 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000924 1709054220.018369 write 0x0128fb thru 0x01a6ef (0x7df5 bytes) 000926 1709054220.025247 mapwrite 0x036703 thru 0x03b29d (0x4b9b bytes) 000928 1709054220.037269 fallocate 0x00e3a0 thru 0x01419f (0x5e00 bytes) 000929 1709054220.040778 punch from 0x03b2a0 to 0x03cf11 (0x1c72 bytes) 000931 1709054220.042957 punch from 0x0288a3 to 0x02b6ec (0x2e4a bytes) 000932 1709054220.044783 fallocate 0x00838a thru 0x00cd9a (0x4a11 bytes) 000933 1709054220.046616 punch from 0x01e9f8 to 0x0223cb (0x39d4 bytes) 000934 1709054220.052109 MIRROR_EXTEND to 16 mirrors 000935 1709054220.291320 mapread 0x0398c9 thru 0x03ebb9 (0x52f1 bytes) 000936 1709054220.305517 MIRROR_SPLIT mirror 11 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000937 1709054220.402057 read 0x025f71 thru 0x0342ae (0xe33e bytes) 000939 1709054220.415619 write 0x027260 thru 0x0277ed (0x58e bytes) 000940 1709054220.435498 mapread 0x019fe6 thru 0x025ea0 (0xbebb bytes) 000941 1709054220.445232 fallocate 0x002bcf thru 0x010b28 (0xdf5a bytes) 000943 1709054220.447985 fallocate 0x03ff9c thru 0x03ffff (0x64 bytes) 000944 1709054220.450637 mapwrite 0x0370a4 thru 0x03788e (0x7eb bytes) 000945 1709054220.466845 MIRROR_EXTEND to 16 mirrors 000946 1709054220.700214 mapwrite 0x02f509 thru 0x03c9ba (0xd4b2 bytes) 000947 1709054220.746684 read 0x0184ff thru 0x022e14 (0xa916 bytes) 000948 1709054220.750497 MIRROR_RESYNC 16 mirrors 000949 1709054221.916949 trunc from 0x040000 to 0x032506 (0xdaf9 bytes) 000950 1709054221.942361 mapread 0x0248b9 thru 0x0259c1 (0x1109 bytes) 000951 1709054221.947527 mapwrite 0x0338cf thru 0x034df8 (0x152a bytes) 000952 1709054221.979933 punch from 0x03001e to 0x033003 (0x2fe6 bytes) 000953 1709054221.983769 write 0x0370b4 thru 0x03ffff (0x8f4c bytes) 000956 1709054221.986848 read 0x02eb78 thru 0x0388cc (0x9d55 bytes) 000957 1709054221.993176 trunc from 0x040000 to 0x03bfcb (0x4034 bytes) 000959 1709054221.997980 write 0x019680 thru 0x022874 (0x91f5 bytes) 000961 1709054222.002302 read 0x02cf70 thru 0x036fa4 (0xa035 bytes) 000962 1709054222.002744 punch from 0x024897 to 0x028cc7 (0x4431 bytes) 000963 1709054222.005901 MIRROR_RESYNC 16 mirrors 000964 1709054223.645138 punch from 0x0278b7 to 0x0305b6 (0x8d00 bytes) 000965 1709054223.659115 trunc from 0x03bfcc to 0x02f3b5 (0xcc16 bytes) 000967 1709054223.666642 mapread 0x0069c6 thru 0x00b593 (0x4bce bytes) 000968 1709054223.673371 mapwrite 0x01e1c4 thru 0x028a7d (0xa8ba bytes) 000969 1709054223.692634 read 0x022551 thru 0x02d636 (0xb0e6 bytes) 000971 1709054223.693749 mapread 0x006169 thru 0x010bde (0xaa76 bytes) 000972 1709054223.694382 write 0x03250a thru 0x03c450 (0x9f47 bytes) 000973 1709054223.695302 trunc from 0x03c451 to 0x036b80 (0x58d0 bytes) 000974 1709054223.701769 mapread 0x01f0f9 thru 0x02819a (0x90a2 bytes) 000976 1709054223.702688 fallocate 0x00960b thru 0x015a98 (0xc48e bytes) 000977 1709054223.704798 mapwrite 0x0086c1 thru 0x009e9f (0x17df bytes) 000978 1709054223.714491 punch from 0x01c7b3 to 0x02a510 (0xdd5e bytes) 000979 1709054223.716578 fallocate 0x0200aa thru 0x0235ce (0x3525 bytes) 000980 1709054223.718333 mapwrite 0x015a6b thru 0x022755 (0xcceb bytes) 000981 1709054223.735162 write 0x01341a thru 0x0194f3 (0x60da bytes) 000982 1709054223.735900 read 0x0362c5 thru 0x036b80 (0x8bc bytes) 000983 1709054223.736254 write 0x0196bc thru 0x0239f8 (0xa33d bytes) 000985 1709054223.737420 write 0x03d7de thru 0x03ffff (0x2822 bytes) 000986 1709054223.742538 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000987 1709054223.830436 write 0x00a793 thru 0x00efd7 (0x4845 bytes) 000988 1709054223.838151 trunc from 0x040000 to 0x011a90 (0x2e56f bytes) 000989 1709054223.846640 write 0x00d395 thru 0x01432c (0x6f98 bytes) 000991 1709054223.850278 MIRROR_SPLIT mirror 5 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000992 1709054223.940228 mapread 0x004541 thru 0x013d07 (0xf7c7 bytes) 000993 1709054223.950050 mapread 0x00aca9 thru 0x01432c (0x9684 bytes) 000994 1709054223.950652 punch from 0x00d108 to 0x01432c (0x7225 bytes) 000995 1709054223.952499 punch from 0x00fee8 to 0x01432c (0x4445 bytes) 000996 1709054223.953814 trunc from 0x01432d to 0x01425b (0xd1 bytes) 000997 1709054223.960404 write 0x03d0d7 thru 0x03ffff (0x2f29 bytes) 000998 1709054223.961030 punch from 0x01fea4 to 0x02c26c (0xc3c9 bytes) 000999 1709054223.967058 MIRROR_RESYNC 14 mirrors 001000 1709054224.803290 trunc from 0x040000 to 0x02b0b7 (0x14f48 bytes) All operations completed A-OK! PASS 70a (28s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 12:17:09 (1709054229) lfs mirror mirror: cannot get READ lease, ext 0: Operation not supported (95) error: lfs mirror extend: /mnt/lustre/f200a.sanity-flr: cannot get lease: Operation not supported sanity-flr test_200a: @@@@@@ FAIL: merging /mnt/lustre/f200a.sanity-flr-2 into /mnt/lustre/f200a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6947:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3432:test_200a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7287:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7347:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7173:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3493:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200a.*.1709054243.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200a.debug_log.oleg211-server.1709054243.log.kTPbQy" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200a.dmesg.oleg211-server.1709054243.log.MdUwab" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 200a (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:17:30 (1709054250) Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre2 Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6066976 .. Extending file size to 6445920 .. failed Extending file size to 6696768 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9067808 .. Extending file size to 7526944 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8340704 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8436256 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9139040 .. Extending file size to 8578976 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8859936 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 5090 5091 5093 5094 5096 192.168.202.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg211-client.virtnet /mnt/lustre2 (opts:) 192.168.202.111@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg211-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:18:40 (1709054320) Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2756 1284932 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2256 1285432 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21556 3585464 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13736 3593284 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 35292 7178748 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2756 1284932 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2256 1285432 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 21556 3585464 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 13736 3593284 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 35292 7178748 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.045369 s, 46.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0588882 s, 35.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg211-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:18:49 (1709054329) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0519453 s, 40.4 MB/s PASS 202 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 12:18:54 (1709054334) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x67a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x67a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x67b:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x677:0x0] } PASS 203 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:19:00 (1709054340) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0446251 s, 47.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x58:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x58:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:19:06 (1709054346) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0389423 s, 53.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x59:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x59:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x59:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x59:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5a:0x0] } PASS 204b (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:19:12 (1709054352) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0250381 s, 83.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5c:0x0] } PASS 204c (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:19:38 (1709054378) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg211-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0708951 s, 14.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5d:0x0] } PASS 204d (37s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:20:18 (1709054418) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg211-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x678:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x679:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0332585 s, 63.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x678:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x679:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x67c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x678:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x679:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x67c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (37s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:20:58 (1709054458) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg211-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool oleg211-server: Pool lustre.test_204f created Adding targets to pool oleg211-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg211-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0400898 s, 52.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x67a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x67d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x67a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x67d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg211-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg211-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg211-server: Pool lustre.test_204f destroyed PASS 204f (55s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer flag ========================================================== 12:21:55 (1709054515) PASS 205 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:22:00 (1709054520) oleg211-server: Pool lustre.test_206 created Adding targets to pool oleg211-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg211-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg211-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg211-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg211-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (15s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:22:18 (1709054538) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0295853 s, 35.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107735 s, 97.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x7286c4ee Mirror 3: 0x7286c4ee PASS 207 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:22:25 (1709054545) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:22:29 (1709054549) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:22:32 (1709054552) Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0240186 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1bb0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xe10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xe10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xe10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2300000 Read this (4 bytes): BBBB 192.168.202.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg211-client.virtnet /mnt/lustre2 (opts:) PASS 209a (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:22:51 (1709054571) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00514632 s, 1.6 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 12:22:56 (1709054576) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159258 s, 65.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x392 lmm_fid: [0x200000405:0x392:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x392 lmm_fid: [0x200000405:0x392:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x683:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339612050 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-02-27 12:22:56.000000000 -0500 Modify: 2024-02-27 12:22:56.000000000 -0500 Change: 2024-02-27 12:22:57.000000000 -0500 Birth: - PASS 210a (122s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 920 sec ============ 12:24:59 (1709054699) sanity-flr: FAIL: test_200a merging /mnt/lustre/f200a.sanity-flr-2 into /mnt/lustre/f200a.sanity-flr failed === sanity-flr: start cleanup 12:25:00 (1709054700) === === sanity-flr: finish cleanup 12:25:01 (1709054701) ===