-----============= acceptance-small: sanity-flr ============----- Thu Apr 4 20:05:41 EDT 2024 excepting tests: 6 201 44c === sanity-flr: start setup 20:05:43 (1712275543) === oleg420-client.virtnet: executing check_config_client /mnt/lustre oleg420-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg420-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6ee7000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6ee7000.idle_timeout=debug disable quota as required oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 20:05:50 (1712275550) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7340] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 20:05:51 (1712275551) PASS 0a (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:05:57 (1712275557) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:05:59 (1712275559) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 20:06:01 (1712275561) PASS 0d (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:06:06 (1712275566) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:06:09 (1712275569) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 20:06:11 (1712275571) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6ee7000.stats=clear osc.lustre-OST0001-osc-ffff8800b6ee7000.stats=clear PASS 0g (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:06:14 (1712275574) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:06:17 (1712275577) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:06:20 (1712275580) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 20:06:22 (1712275582) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 20:06:26 (1712275586) PASS 2 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:06:29 (1712275589) PASS 3 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:06:32 (1712275592) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:06:35 (1712275595) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 20:06:39 (1712275599) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:06:42 (1712275602) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0332475 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.02917 s, 35.9 MB/s PASS 21 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:06:46 (1712275606) 18+0 records in 18+0 records out 18874368 bytes (19 MB) copied, 0.133635 s, 141 MB/s PASS 22 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:06:50 (1712275610) Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6ee7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6ee7000.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6ee7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6ee7000.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:07:10 (1712275630) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0826793 s, 63.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6ee7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6ee7000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (8s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:07:20 (1712275640) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6ee7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6ee7000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6ee7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6ee7000.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:07:40 (1712275660) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.774562 s, 67.7 MB/s -rw-r--r-- 1 root root 52428800 Apr 4 20:07 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.204s user 0m0.000s sys 0m0.098s Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6ee7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6ee7000.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost2)... real 0m0.212s user 0m0.000s sys 0m0.100s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6ee7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6ee7000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.232s user 0m0.000s sys 0m0.109s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg420-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6ee7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6ee7000.ost_server_uuid in FULL state after 0 sec PASS 33b (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:08:05 (1712275685) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:08:07 (1712275687) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:08:09 (1712275689) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 20:08:12 (1712275692) PASS 35 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 20:08:15 (1712275695) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.113003 s, 111 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:08:19 (1712275699) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00767926 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00618794 s, 169 MB/s PASS 36b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:08:22 (1712275702) PASS 36c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:08:25 (1712275705) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000846968 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000597733 s, 1.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000579645 s, 1.8 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0010002 s, 1.0 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 20:08:29 (1712275709) create test files with size 15730122 .. d5ea8a94771d04c793ed5fed2f8effce - ce12328eba30263320709c0dc1f11ec2 - 953f4d967e991533ac7efb45e7ad2689 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre PASS 37 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 20:08:35 (1712275715) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:08:39 (1712275719) PASS 39 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:08:41 (1712275721) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0155389 s, 67.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x38:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00739912 s, 142 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 20:08:45 (1712275725) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0168378 s, 125 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0703638 s, 59.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.028202 s, 74.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 20:08:48 (1712275728) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 20:08:50 (1712275730) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:08:52 (1712275732) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00980252 s, 107 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 20:08:55 (1712275735) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0590433 s, 53.3 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 44a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:08:59 (1712275739) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x45:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x44:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (1s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:09:02 (1712275742) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0597717 s, 175 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:09:06 (1712275746) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff880130ec4000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0645078 s, 163 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88012d98a000.xattr_cache=1 PASS 44e (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:09:10 (1712275750) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 20:09:18 (1712275758) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 20:09:24 (1712275764) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 20:09:26 (1712275766) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0753451 s, 41.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0632057 s, 49.8 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: dbaaeed3cc8d00c80d6c8e0b23850f67 - mirror 1 checksum: b1cfeedf1d54826adec981003f5d3257 - mirror 2 checksum: dbaaeed3cc8d00c80d6c8e0b23850f67 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 4 20:09:27 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 4 20:09:27 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: dbaaeed3cc8d00c80d6c8e0b23850f67 - mirror 1 checksum: b1cfeedf1d54826adec981003f5d3257 - mirror 2 checksum: b1cfeedf1d54826adec981003f5d3257 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 4 20:09:27 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 4 20:09:27 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 48 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 49a: FIEMAP upon FLR file ============= 20:09:30 (1712275770) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116389 s, 90.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:09:33 (1712275773) Failing mds1 on oleg420-server Stopping /mnt/lustre-mds1 (opts:) on oleg420-server 20:09:41 (1712275781) shut down Failover mds1 to oleg420-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 20:09:53 (1712275793) targets are mounted 20:09:53 (1712275793) facet_failover done oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00811234 s, 129 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 50A (28s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:10:02 (1712275802) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00648156 s, 3.2 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00399746 s, 5.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 20:10:06 (1712275806) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0077189 s, 2.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00307786 s, 6.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:10:09 (1712275809) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00168047 s, 9.7 MB/s PASS 50c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 20:10:12 (1712275812) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00787575 s, 2.6 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:10:16 (1712275816) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00141042 s, 2.9 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00618935 s, 662 kB/s PASS 60a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:10:19 (1712275819) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:10:24 (1712275824) create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] PASS 61a (16s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:10:42 (1712275842) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (16s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:11:00 (1712275860) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:11:18 (1712275878) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 20:11:21 (1712275881) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 30135 mirror split pid 30136 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4661: 30135 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4661: 30136 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 20:12:25 (1712275945) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 5945 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000001 1712275945.610911 write 0x02c61a thru 0x02eab8 (0x249f bytes) fallocating to largest ever: 0x2e12e 000002 1712275945.614021 fallocate 0x02c944 thru 0x02e12d (0x17ea bytes) 000003 1712275945.615341 read 0x008aef thru 0x0163c6 (0xd8d8 bytes) 000005 1712275945.618099 write 0x00301b thru 0x00a58f (0x7575 bytes) 000006 1712275945.620087 punch from 0x014e78 to 0x01a5c4 (0x574d bytes) 000008 1712275945.622002 fallocate 0x009b29 thru 0x012750 (0x8c28 bytes) 000011 1712275945.624310 mapread 0x02d76b thru 0x02e4ae (0xd44 bytes) 000012 1712275945.625039 mapwrite 0x02b5fe thru 0x02eef2 (0x38f5 bytes) 000013 1712275945.636233 mapread 0x01a5ab thru 0x025c06 (0xb65c bytes) 000014 1712275945.640976 punch from 0x0288a2 to 0x02eef2 (0x6651 bytes) 000015 1712275945.642752 read 0x001b39 thru 0x004b9e (0x3066 bytes) 000016 1712275945.646331 MIRROR_EXTEND to 2 mirrors 000017 1712275945.749368 mapread 0x01d72f thru 0x024cc9 (0x759b bytes) 000018 1712275945.756693 fallocate 0x01b29f thru 0x01cfa4 (0x1d06 bytes) 000019 1712275945.766192 mapwrite 0x00dbaa thru 0x01911c (0xb573 bytes) 000020 1712275945.777017 read 0x026e61 thru 0x02eef2 (0x8092 bytes) 000021 1712275945.779588 mapread 0x013306 thru 0x01817f (0x4e7a bytes) 000022 1712275945.780225 fallocate 0x008e1a thru 0x0158a9 (0xca90 bytes) 000023 1712275945.782068 punch from 0x00b6b4 to 0x013bdb (0x8528 bytes) 000024 1712275945.784109 read 0x011642 thru 0x01e076 (0xca35 bytes) 000025 1712275945.788165 punch from 0x013b35 to 0x01d529 (0x99f5 bytes) 000026 1712275945.791460 MIRROR_EXTEND to 3 mirrors 000027 1712275945.906642 mapwrite 0x01b6b8 thru 0x01c42b (0xd74 bytes) 000028 1712275945.938038 fallocate 0x020699 thru 0x024416 (0x3d7e bytes) 000029 1712275945.939675 trunc from 0x02eef3 to 0x020e8c (0xe066 bytes) 000031 1712275945.950235 read 0x0005ba thru 0x003842 (0x3289 bytes) 000032 1712275945.952900 MIRROR_EXTEND to 4 mirrors 000033 1712275946.070585 punch from 0x0046f8 to 0x01078d (0xc096 bytes) 000034 1712275946.078784 write 0x01d548 thru 0x027c43 (0xa6fc bytes) 000036 1712275946.081591 mapwrite 0x024772 thru 0x02d0fc (0x898b bytes) 000038 1712275946.094594 write 0x03f851 thru 0x03ffff (0x7af bytes) 000039 1712275946.096499 MIRROR_EXTEND to 5 mirrors 000040 1712275946.224789 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000041 1712275946.297097 mapread 0x00c11a thru 0x01834c (0xc233 bytes) 000042 1712275946.308252 mapwrite 0x00e013 thru 0x00ed8a (0xd78 bytes) fallocating to largest ever: 0x3e84c 000043 1712275946.334710 fallocate 0x03847c thru 0x03e84b (0x63d0 bytes) 000044 1712275946.336043 read 0x00b489 thru 0x01232b (0x6ea3 bytes) 000045 1712275946.336349 read 0x02c15f thru 0x0337f0 (0x7692 bytes) 000046 1712275946.338810 write 0x02e857 thru 0x038a7e (0xa228 bytes) 000047 1712275946.340701 mapwrite 0x028cb2 thru 0x02b6bb (0x2a0a bytes) 000048 1712275946.345747 mapwrite 0x02b637 thru 0x03a174 (0xeb3e bytes) 000049 1712275946.351257 trunc from 0x040000 to 0x01d2ec (0x22d13 bytes) 000050 1712275946.356037 mapread 0x00ea81 thru 0x0159a8 (0x6f28 bytes) 000051 1712275946.356600 mapread 0x003c61 thru 0x01181f (0xdbbf bytes) 000052 1712275946.358310 MIRROR_EXTEND to 5 mirrors 000053 1712275946.466990 trunc from 0x01d2ed to 0x010d96 (0xc556 bytes) 000055 1712275946.482051 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000056 1712275946.526635 fallocate 0x015ac0 thru 0x016c6c (0x11ad bytes) 000057 1712275946.528125 mapwrite 0x02538c thru 0x02e7e6 (0x945b bytes) 000059 1712275946.545697 mapwrite 0x00e380 thru 0x0183f1 (0xa072 bytes) 000060 1712275946.556522 trunc from 0x02e7e7 to 0x02e760 (0x86 bytes) 000062 1712275946.561064 write 0x028680 thru 0x030575 (0x7ef6 bytes) 000063 1712275946.561688 mapread 0x012c5e thru 0x01ec65 (0xc008 bytes) 000064 1712275946.562210 mapread 0x012c9a thru 0x01713e (0x44a5 bytes) 000065 1712275946.562620 punch from 0x01cecc to 0x02c50a (0xf63f bytes) 000066 1712275946.568265 MIRROR_RESYNC 4 mirrors 000067 1712275946.739124 mapread 0x0070ac thru 0x007c07 (0xb5c bytes) 000068 1712275946.740751 mapread 0x0107b4 thru 0x011486 (0xcd3 bytes) 000069 1712275946.744129 punch from 0x001fea to 0x00a6ea (0x8701 bytes) 000070 1712275946.752948 punch from 0x003e4a to 0x012906 (0xeabd bytes) 000071 1712275946.754769 read 0x0245b9 thru 0x027646 (0x308e bytes) 000072 1712275946.756738 punch from 0x005dcd to 0x0069cc (0xc00 bytes) 000073 1712275946.758232 fallocate 0x00c553 thru 0x00f1cc (0x2c7a bytes) 000075 1712275946.760150 punch from 0x009488 to 0x011acd (0x8646 bytes) 000078 1712275946.762331 write 0x027957 thru 0x0347ca (0xce74 bytes) 000079 1712275946.763566 punch from 0x00b5c5 to 0x0188a1 (0xd2dd bytes) 000080 1712275946.766173 MIRROR_RESYNC 4 mirrors 000083 1712275946.927259 mapread 0x0034b6 thru 0x00c49e (0x8fe9 bytes) 000085 1712275946.932134 mapread 0x00f8a2 thru 0x011c7e (0x23dd bytes) 000086 1712275946.933502 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000089 1712275946.987671 fallocate 0x01142b thru 0x01fceb (0xe8c1 bytes) 000090 1712275946.996146 punch from 0x01db78 to 0x022839 (0x4cc2 bytes) 000091 1712275946.997734 write 0x00a086 thru 0x00b347 (0x12c2 bytes) 000092 1712275947.000949 write 0x0140a9 thru 0x021953 (0xd8ab bytes) 000093 1712275947.004590 write 0x02596a thru 0x02a52e (0x4bc5 bytes) 000094 1712275947.007079 read 0x007377 thru 0x0101fc (0x8e86 bytes) 000095 1712275947.009529 mapread 0x02ec92 thru 0x0347ca (0x5b39 bytes) 000096 1712275947.012090 write 0x01453d thru 0x01c1fb (0x7cbf bytes) 000097 1712275947.012978 punch from 0x01dd86 to 0x02c2c3 (0xe53e bytes) 000098 1712275947.016651 write 0x01e397 thru 0x02a707 (0xc371 bytes) 000099 1712275947.019734 read 0x019e7c thru 0x01b8fd (0x1a82 bytes) 000100 1712275947.020051 write 0x037bbd thru 0x03ffff (0x8443 bytes) 000101 1712275947.020698 read 0x01414f thru 0x0195b8 (0x546a bytes) 000102 1712275947.022202 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000104 1712275947.068139 trunc from 0x040000 to 0x00d677 (0x32988 bytes) 000105 1712275947.077080 punch from 0x008dc4 to 0x00d677 (0x48b4 bytes) 000106 1712275947.078878 read 0x00761e thru 0x00d677 (0x605a bytes) 000107 1712275947.081100 read 0x007fcc thru 0x00d677 (0x56ac bytes) 000108 1712275947.081521 write 0x021b6e thru 0x022dcf (0x1262 bytes) 000109 1712275947.082203 mapwrite 0x01d552 thru 0x01fdb2 (0x2861 bytes) 000110 1712275947.089458 read 0x0223ef thru 0x022dcf (0x9e1 bytes) 000111 1712275947.089851 trunc from 0x022dd0 to 0x02fd86 (0xcfb7 bytes) 000113 1712275947.093763 fallocate 0x02dd21 thru 0x03d54d (0xf82d bytes) 000114 1712275947.095140 fallocate 0x004ca7 thru 0x0128ac (0xdc06 bytes) 000115 1712275947.096613 mapwrite 0x00f71e thru 0x01b009 (0xb8ec bytes) 000116 1712275947.103405 trunc from 0x03d54e to 0x014493 (0x290ba bytes) 000117 1712275947.107146 write 0x02081b thru 0x02c667 (0xbe4d bytes) 000118 1712275947.108063 read 0x00f500 thru 0x01c3f3 (0xcef4 bytes) 000119 1712275947.110427 fallocate 0x00485a thru 0x012cef (0xe496 bytes) 000120 1712275947.111854 punch from 0x00dd19 to 0x01b711 (0xd9f9 bytes) 000121 1712275947.113330 fallocate 0x001aba thru 0x002903 (0xe4a bytes) fallocating to largest ever: 0x40000 000122 1712275947.114620 fallocate 0x03de4e thru 0x03ffff (0x21b2 bytes) 000123 1712275947.115971 trunc from 0x040000 to 0x00b86c (0x34793 bytes) 000124 1712275947.119533 write 0x00406f thru 0x00d1e8 (0x917a bytes) 000125 1712275947.121323 MIRROR_RESYNC 2 mirrors 000126 1712275947.203580 mapread 0x009672 thru 0x00d1e8 (0x3b77 bytes) 000127 1712275947.208205 write 0x0193ce thru 0x01e1a3 (0x4dd6 bytes) 000129 1712275947.216516 punch from 0x00bdb4 to 0x013046 (0x7293 bytes) 000130 1712275947.218117 mapread 0x005352 thru 0x014727 (0xf3d6 bytes) 000131 1712275947.222348 read 0x014c68 thru 0x01e1a3 (0x953c bytes) 000133 1712275947.222877 punch from 0x004d0e to 0x01094e (0xbc41 bytes) 000134 1712275947.224527 read 0x0038a7 thru 0x006985 (0x30df bytes) 000135 1712275947.226929 punch from 0x010643 to 0x01e1a3 (0xdb61 bytes) 000136 1712275947.232871 write 0x01dbf0 thru 0x024ee3 (0x72f4 bytes) 000137 1712275947.234753 read 0x01962c thru 0x024ee3 (0xb8b8 bytes) 000138 1712275947.237256 trunc from 0x024ee4 to 0x008279 (0x1cc6a bytes) 000139 1712275947.245896 write 0x0129c7 thru 0x022280 (0xf8ba bytes) 000140 1712275947.247194 mapwrite 0x029c89 thru 0x02de5d (0x41d5 bytes) 000141 1712275947.260200 MIRROR_EXTEND to 3 mirrors 000143 1712275947.370248 mapwrite 0x0223c0 thru 0x02c76b (0xa3ac bytes) 000144 1712275947.404461 fallocate 0x0040e4 thru 0x011c9a (0xdbb7 bytes) 000145 1712275947.406160 write 0x003ca7 thru 0x0083d2 (0x472c bytes) 000146 1712275947.410745 read 0x01f55d thru 0x027eee (0x8992 bytes) 000147 1712275947.411308 fallocate 0x002c2d thru 0x00ab7f (0x7f53 bytes) 000149 1712275947.413311 mapwrite 0x01112f thru 0x01ca47 (0xb919 bytes) 000150 1712275947.421438 mapread 0x0078b9 thru 0x009273 (0x19bb bytes) 000151 1712275947.421903 fallocate 0x03a681 thru 0x03ffff (0x597f bytes) 000152 1712275947.424580 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000154 1712275947.470434 fallocate 0x0157a4 thru 0x025785 (0xffe2 bytes) 000155 1712275947.472021 mapwrite 0x007753 thru 0x00fe2d (0x86db bytes) 000156 1712275947.480831 mapwrite 0x03a3cb thru 0x03ffff (0x5c35 bytes) 000159 1712275947.489015 MIRROR_EXTEND to 3 mirrors 000160 1712275947.596877 read 0x004b66 thru 0x00bcc8 (0x7163 bytes) 000161 1712275947.600418 punch from 0x02d261 to 0x034a90 (0x7830 bytes) 000162 1712275947.610951 fallocate 0x018627 thru 0x01c26d (0x3c47 bytes) 000163 1712275947.612644 mapread 0x007fee thru 0x01613b (0xe14e bytes) 000164 1712275947.620780 mapread 0x00e0b8 thru 0x01a0cf (0xc018 bytes) 000166 1712275947.621874 fallocate 0x0310ea thru 0x033e69 (0x2d80 bytes) 000167 1712275947.625771 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000168 1712275947.674588 mapwrite 0x024018 thru 0x0284b0 (0x4499 bytes) 000169 1712275947.685454 MIRROR_RESYNC 2 mirrors 000171 1712275947.772870 trunc from 0x040000 to 0x0194b7 (0x26b48 bytes) 000172 1712275947.786073 trunc from 0x0194b8 to 0x00c404 (0xd0b3 bytes) 000173 1712275947.790680 trunc from 0x00c405 to 0x028a46 (0x1c642 bytes) 000175 1712275947.795607 MIRROR_EXTEND to 3 mirrors 000176 1712275947.893494 fallocate 0x013d2f thru 0x01b678 (0x794a bytes) 000177 1712275947.901087 fallocate 0x01f4c9 thru 0x024d4a (0x5882 bytes) 000179 1712275947.902639 fallocate 0x009e66 thru 0x00bcfc (0x1e97 bytes) 000180 1712275947.903978 mapwrite 0x034154 thru 0x03aaa6 (0x6953 bytes) 000181 1712275947.922264 fallocate 0x00869b thru 0x017345 (0xecab bytes) 000182 1712275947.925420 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000184 1712275947.971568 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000185 1712275947.997223 fallocate 0x0280d5 thru 0x029265 (0x1191 bytes) 000186 1712275947.998889 trunc from 0x03aaa7 to 0x01924b (0x2185b bytes) 000187 1712275948.003057 fallocate 0x03e151 thru 0x03ffff (0x1eaf bytes) 000188 1712275948.004524 read 0x03c751 thru 0x03ffff (0x38af bytes) 000189 1712275948.006150 fallocate 0x00a4ea thru 0x00e2ca (0x3de1 bytes) 000190 1712275948.007445 write 0x033f6a thru 0x038484 (0x451b bytes) 000191 1712275948.009978 fallocate 0x022926 thru 0x029035 (0x6710 bytes) 000192 1712275948.011438 trunc from 0x040000 to 0x00cc0f (0x333f0 bytes) 000193 1712275948.015303 trunc from 0x00cc10 to 0x03be15 (0x2f206 bytes) 000194 1712275948.019007 punch from 0x01f3b2 to 0x029aa6 (0xa6f5 bytes) 000195 1712275948.020416 fallocate 0x02b873 thru 0x0333de (0x7b6c bytes) 000196 1712275948.021691 mapwrite 0x031afc thru 0x033405 (0x190a bytes) 000197 1712275948.028639 mapread 0x038bac thru 0x03be15 (0x326a bytes) 000200 1712275948.031400 mapread 0x025fd1 thru 0x0300cb (0xa0fb bytes) 000202 1712275948.034557 mapread 0x00c937 thru 0x00df30 (0x15fa bytes) 000203 1712275948.036781 read 0x02daf7 thru 0x03b7be (0xdcc8 bytes) 000204 1712275948.037090 mapwrite 0x006383 thru 0x011a8a (0xb708 bytes) 000205 1712275948.042075 write 0x0269a1 thru 0x031d59 (0xb3b9 bytes) 000206 1712275948.042680 fallocate 0x02f23c thru 0x0343fe (0x51c3 bytes) 000207 1712275948.043946 punch from 0x02bbfd to 0x036350 (0xa754 bytes) 000208 1712275948.046978 punch from 0x019432 to 0x02127f (0x7e4e bytes) 000209 1712275948.048411 mapwrite 0x007c67 thru 0x010694 (0x8a2e bytes) 000210 1712275948.053176 fallocate 0x017369 thru 0x023058 (0xbcf0 bytes) 000211 1712275948.054438 mapread 0x00fd33 thru 0x01d613 (0xd8e1 bytes) 000212 1712275948.056908 mapread 0x010c3c thru 0x01cbba (0xbf7f bytes) 000213 1712275948.057300 punch from 0x0244ce to 0x032b38 (0xe66b bytes) 000217 1712275948.059127 read 0x00e5d4 thru 0x012f6a (0x4997 bytes) 000218 1712275948.059396 fallocate 0x01fc06 thru 0x02a596 (0xa991 bytes) 000219 1712275948.060706 trunc from 0x03be16 to 0x02dddd (0xe038 bytes) 000220 1712275948.064825 read 0x02bea7 thru 0x02dddd (0x1f37 bytes) 000222 1712275948.067501 punch from 0x00af98 to 0x00cb4b (0x1bb4 bytes) 000223 1712275948.068976 write 0x039c64 thru 0x039e36 (0x1d3 bytes) 000224 1712275948.069529 mapwrite 0x007a06 thru 0x016bcc (0xf1c7 bytes) 000227 1712275948.082408 trunc from 0x039e37 to 0x03246a (0x79cc bytes) 000228 1712275948.086976 punch from 0x0251c3 to 0x02f38b (0xa1c9 bytes) 000229 1712275948.088752 fallocate 0x021b5f thru 0x02466b (0x2b0d bytes) 000231 1712275948.090904 fallocate 0x0055f1 thru 0x007e09 (0x2819 bytes) 000232 1712275948.092625 punch from 0x001fc9 to 0x0042b5 (0x22ed bytes) 000234 1712275948.094419 mapwrite 0x032f11 thru 0x036cf8 (0x3de8 bytes) 000235 1712275948.109182 mapread 0x005199 thru 0x006909 (0x1771 bytes) 000236 1712275948.109711 mapread 0x03315b thru 0x036cf8 (0x3b9e bytes) 000237 1712275948.110117 mapread 0x031fe8 thru 0x036cf8 (0x4d11 bytes) 000239 1712275948.110751 fallocate 0x0061ed thru 0x010073 (0x9e87 bytes) 000240 1712275948.112706 trunc from 0x036cf9 to 0x037ede (0x11e6 bytes) 000242 1712275948.116828 read 0x00dc9d thru 0x0133fc (0x5760 bytes) 000244 1712275948.117391 write 0x029812 thru 0x038a42 (0xf231 bytes) 000245 1712275948.120432 punch from 0x01ae7a to 0x0223e9 (0x7570 bytes) 000246 1712275948.121950 write 0x0240b6 thru 0x02674a (0x2695 bytes) 000247 1712275948.122389 trunc from 0x038a43 to 0x0187fb (0x20247 bytes) 000248 1712275948.126367 write 0x0108fd thru 0x01d7d1 (0xced5 bytes) 000249 1712275948.127305 mapwrite 0x00a698 thru 0x0154bf (0xae28 bytes) 000250 1712275948.134125 trunc from 0x01d7d2 to 0x029658 (0xbe87 bytes) 000253 1712275948.138735 mapread 0x00388a thru 0x0118f2 (0xe069 bytes) 000256 1712275948.139890 fallocate 0x00d45a thru 0x017005 (0x9bac bytes) 000258 1712275948.142313 fallocate 0x03733a thru 0x039486 (0x214d bytes) 000259 1712275948.144237 read 0x01b5b9 thru 0x029658 (0xe0a0 bytes) 000260 1712275948.146636 mapread 0x01d375 thru 0x027647 (0xa2d3 bytes) 000261 1712275948.147244 trunc from 0x029659 to 0x030089 (0x6a31 bytes) 000263 1712275948.151855 read 0x0111e6 thru 0x020d63 (0xfb7e bytes) 000264 1712275948.152323 mapread 0x005023 thru 0x0123b4 (0xd392 bytes) 000266 1712275948.153240 punch from 0x010ba1 to 0x0131e6 (0x2646 bytes) 000267 1712275948.154747 mapread 0x01105b thru 0x017638 (0x65de bytes) 000268 1712275948.156911 mapread 0x018d98 thru 0x0209b0 (0x7c19 bytes) 000269 1712275948.157288 write 0x03e96a thru 0x03ffff (0x1696 bytes) 000270 1712275948.157702 read 0x01991d thru 0x01a6e1 (0xdc5 bytes) 000271 1712275948.157952 write 0x01c02f thru 0x01f004 (0x2fd6 bytes) 000272 1712275948.158356 mapwrite 0x029b1d thru 0x031dc6 (0x82aa bytes) 000273 1712275948.166335 fallocate 0x01b9af thru 0x025d01 (0xa353 bytes) 000276 1712275948.169951 MIRROR_EXTEND to 2 mirrors 000277 1712275948.292857 fallocate 0x02137b thru 0x0266cd (0x5353 bytes) 000278 1712275948.300841 mapread 0x0017d4 thru 0x00603a (0x4867 bytes) 000279 1712275948.306348 MIRROR_EXTEND to 3 mirrors 000280 1712275948.430820 mapread 0x02981a thru 0x02eb11 (0x52f8 bytes) 000282 1712275948.440694 trunc from 0x040000 to 0x0124fa (0x2db05 bytes) 000283 1712275948.455446 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000285 1712275948.499823 write 0x037e0a thru 0x037f01 (0xf8 bytes) 000286 1712275948.502992 MIRROR_EXTEND to 3 mirrors 000287 1712275948.605622 read 0x02e71d thru 0x037f01 (0x97e5 bytes) 000288 1712275948.610480 MIRROR_RESYNC 3 mirrors 000289 1712275948.710717 fallocate 0x00892f thru 0x00a6fa (0x1dcc bytes) 000290 1712275948.717713 trunc from 0x037f02 to 0x02d38a (0xab77 bytes) 000291 1712275948.724687 read 0x0159fc thru 0x01d674 (0x7c79 bytes) 000292 1712275948.726838 trunc from 0x02d38b to 0x021aba (0xb8d0 bytes) 000293 1712275948.730401 mapwrite 0x00cc7b thru 0x01c2ca (0xf650 bytes) 000294 1712275948.740726 mapread 0x01795f thru 0x0180e3 (0x785 bytes) 000296 1712275948.741369 mapread 0x00d43b thru 0x01999c (0xc562 bytes) 000297 1712275948.742963 MIRROR_RESYNC 3 mirrors 000298 1712275948.863612 mapwrite 0x01e1a4 thru 0x020e7d (0x2cda bytes) 000299 1712275948.893545 mapwrite 0x02f1d4 thru 0x0363e5 (0x7212 bytes) 000300 1712275948.910196 mapread 0x00ce0d thru 0x01b7a0 (0xe994 bytes) 000301 1712275948.914030 read 0x0192bf thru 0x01e339 (0x507b bytes) 000302 1712275948.915565 MIRROR_RESYNC 3 mirrors 000303 1712275949.064944 mapwrite 0x028382 thru 0x03237e (0x9ffd bytes) 000305 1712275949.098530 read 0x02f6af thru 0x031421 (0x1d73 bytes) 000306 1712275949.098962 trunc from 0x0363e6 to 0x0226a2 (0x13d43 bytes) 000308 1712275949.103557 write 0x01be6c thru 0x026eca (0xb05f bytes) 000309 1712275949.106005 mapread 0x003834 thru 0x00753e (0x3d0b bytes) 000311 1712275949.109627 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000312 1712275949.155439 trunc from 0x026ecb to 0x020923 (0x65a7 bytes) 000314 1712275949.159959 read 0x00269e thru 0x0054a3 (0x2e06 bytes) 000315 1712275949.161729 mapread 0x01f975 thru 0x020923 (0xfaf bytes) 000316 1712275949.164453 punch from 0x010c82 to 0x019ae8 (0x8e67 bytes) 000317 1712275949.165918 write 0x029374 thru 0x02e87f (0x550c bytes) 000318 1712275949.166480 mapwrite 0x03b581 thru 0x03ff27 (0x49a7 bytes) 000320 1712275949.178074 punch from 0x0013b0 to 0x009737 (0x8388 bytes) 000321 1712275949.179713 punch from 0x016162 to 0x01a7ce (0x466d bytes) 000322 1712275949.182231 MIRROR_EXTEND to 3 mirrors 000323 1712275949.296682 fallocate 0x007b83 thru 0x011a4a (0x9ec8 bytes) 000324 1712275949.304613 read 0x004f89 thru 0x014a53 (0xfacb bytes) 000325 1712275949.307133 write 0x018ac6 thru 0x01c84b (0x3d86 bytes) 000326 1712275949.310610 punch from 0x02c35f to 0x02e2f5 (0x1f97 bytes) 000328 1712275949.312486 mapwrite 0x0255eb thru 0x028d44 (0x375a bytes) 000329 1712275949.320533 read 0x025bbf thru 0x027e07 (0x2249 bytes) 000330 1712275949.320994 read 0x00317b thru 0x01087b (0xd701 bytes) 000331 1712275949.323534 read 0x01c769 thru 0x02a1a6 (0xda3e bytes) 000334 1712275949.324219 mapread 0x001d2c thru 0x00bf65 (0xa23a bytes) 000335 1712275949.324626 mapread 0x004613 thru 0x01445b (0xfe49 bytes) 000336 1712275949.325137 punch from 0x034318 to 0x03aa17 (0x6700 bytes) 000337 1712275949.326483 trunc from 0x03ff28 to 0x01ea81 (0x214a6 bytes) 000340 1712275949.335226 write 0x03b040 thru 0x03ffff (0x4fc0 bytes) 000341 1712275949.335908 punch from 0x00e972 to 0x0104e7 (0x1b76 bytes) 000344 1712275949.338019 punch from 0x017d1e to 0x022672 (0xa955 bytes) 000345 1712275949.339691 mapread 0x0372a9 thru 0x03ffff (0x8d57 bytes) 000346 1712275949.343916 MIRROR_RESYNC 3 mirrors 000347 1712275949.482882 read 0x039b99 thru 0x03c1e1 (0x2649 bytes) 000348 1712275949.484366 read 0x03e7cf thru 0x03ffff (0x1831 bytes) 000350 1712275949.487066 mapwrite 0x037c47 thru 0x03ffff (0x83b9 bytes) 000351 1712275949.521160 trunc from 0x040000 to 0x002bbe (0x3d441 bytes) 000352 1712275949.531206 fallocate 0x03b412 thru 0x03ffff (0x4bee bytes) 000353 1712275949.532848 punch from 0x0005d2 to 0x002bbe (0x25ed bytes) 000354 1712275949.533972 fallocate 0x038051 thru 0x03ffff (0x7faf bytes) 000355 1712275949.536186 MIRROR_RESYNC 3 mirrors 000356 1712275949.623319 mapread 0x0014ca thru 0x0027b2 (0x12e9 bytes) 000357 1712275949.625074 fallocate 0x004422 thru 0x00c49f (0x807e bytes) 000358 1712275949.634824 read 0x0009b2 thru 0x002bbe (0x220d bytes) 000360 1712275949.636930 read 0x002009 thru 0x002bbe (0xbb6 bytes) 000361 1712275949.637396 punch from 0x001902 to 0x002bbe (0x12bd bytes) 000362 1712275949.638822 fallocate 0x0389c9 thru 0x03939b (0x9d3 bytes) 000363 1712275949.641974 MIRROR_EXTEND to 4 mirrors 000364 1712275949.746954 mapread 0x001f7f thru 0x002bbe (0xc40 bytes) 000365 1712275949.752589 fallocate 0x010658 thru 0x01891e (0x82c7 bytes) 000366 1712275949.762095 mapread 0x000f5f thru 0x001525 (0x5c7 bytes) 000367 1712275949.767353 MIRROR_RESYNC 4 mirrors 000368 1712275949.876412 read 0x00191c thru 0x002bbe (0x12a3 bytes) 000369 1712275949.877956 mapread 0x00258f thru 0x002bbe (0x630 bytes) 000370 1712275949.878396 trunc from 0x002bbf to 0x01bed8 (0x1931a bytes) 000371 1712275949.891208 punch from 0x00d1ff to 0x00e6a2 (0x14a4 bytes) 000372 1712275949.892551 punch from 0x0069c6 to 0x010939 (0x9f74 bytes) 000373 1712275949.893847 mapread 0x005cef thru 0x01426b (0xe57d bytes) 000374 1712275949.899914 read 0x003555 thru 0x009df3 (0x689f bytes) 000376 1712275949.900448 fallocate 0x011735 thru 0x01db0e (0xc3da bytes) 000377 1712275949.902062 mapread 0x00614c thru 0x008869 (0x271e bytes) 000378 1712275949.902578 mapwrite 0x02ea72 thru 0x0395be (0xab4d bytes) 000381 1712275949.921163 mapwrite 0x0382b1 thru 0x03ce65 (0x4bb5 bytes) 000382 1712275949.933719 trunc from 0x03ce66 to 0x00be0a (0x3105b bytes) 000383 1712275949.938829 write 0x0049a7 thru 0x00e5df (0x9c39 bytes) 000384 1712275949.939639 mapread 0x0018f1 thru 0x0077e9 (0x5ef9 bytes) 000386 1712275949.940359 fallocate 0x0197a0 thru 0x01a960 (0x11c1 bytes) 000387 1712275949.942408 mapread 0x009d77 thru 0x00e5df (0x4869 bytes) 000388 1712275949.943026 fallocate 0x001989 thru 0x004ca8 (0x3320 bytes) 000389 1712275949.944891 punch from 0x00e32e to 0x00e5df (0x2b2 bytes) 000390 1712275949.948546 write 0x03bc18 thru 0x03ffff (0x43e8 bytes) 000391 1712275949.949317 mapread 0x016db1 thru 0x019671 (0x28c1 bytes) 000392 1712275949.954178 trunc from 0x040000 to 0x03b866 (0x4799 bytes) 000395 1712275949.958650 trunc from 0x03b867 to 0x020346 (0x1b520 bytes) 000396 1712275949.961900 mapwrite 0x035103 thru 0x03ffff (0xaefd bytes) 000397 1712275949.983543 punch from 0x01ed59 to 0x026fef (0x8297 bytes) 000398 1712275949.985037 mapwrite 0x005abb thru 0x01280f (0xcd55 bytes) 000399 1712275949.997472 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000400 1712275950.044928 punch from 0x0065fe to 0x00b7c0 (0x51c3 bytes) 000401 1712275950.046456 read 0x01c443 thru 0x01ce3b (0x9f9 bytes) 000403 1712275950.048489 mapwrite 0x01d268 thru 0x01e307 (0x10a0 bytes) 000404 1712275950.054526 fallocate 0x00c56e thru 0x011a17 (0x54aa bytes) 000405 1712275950.056165 mapread 0x00c48d thru 0x018c06 (0xc77a bytes) 000406 1712275950.060389 trunc from 0x040000 to 0x018a30 (0x275cf bytes) 000407 1712275950.065774 MIRROR_RESYNC 3 mirrors 000408 1712275950.189579 mapwrite 0x030871 thru 0x03dac4 (0xd254 bytes) 000409 1712275950.214913 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000411 1712275950.259698 mapwrite 0x0371d7 thru 0x03c56f (0x5399 bytes) 000412 1712275950.268776 punch from 0x02917e to 0x02d947 (0x47ca bytes) 000413 1712275950.270254 read 0x000bd2 thru 0x00a22f (0x965e bytes) 000414 1712275950.272591 mapwrite 0x01c4e9 thru 0x022746 (0x625e bytes) 000415 1712275950.279400 fallocate 0x0133ee thru 0x01fb14 (0xc727 bytes) 000416 1712275950.281045 write 0x03252e thru 0x0387f4 (0x62c7 bytes) 000418 1712275950.283013 punch from 0x03367e to 0x03b94a (0x82cd bytes) 000419 1712275950.286029 trunc from 0x03dac5 to 0x0114fa (0x2c5ca bytes) 000420 1712275950.289684 fallocate 0x015b8e thru 0x0258d9 (0xfd4c bytes) 000421 1712275950.291131 mapread 0x009785 thru 0x0114fa (0x7d76 bytes) 000422 1712275950.294070 read 0x0032e4 thru 0x00bf7d (0x8c9a bytes) 000424 1712275950.295211 punch from 0x00e574 to 0x0114fa (0x2f87 bytes) 000426 1712275950.298947 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000427 1712275950.326303 punch from 0x00880e to 0x00b912 (0x3105 bytes) 000429 1712275950.328300 read 0x00f94a thru 0x0103e7 (0xa9e bytes) 000430 1712275950.330075 trunc from 0x0114fb to 0x00195f (0xfb9b bytes) 000431 1712275950.335665 read 0x000add thru 0x00195f (0xe83 bytes) 000432 1712275950.337526 fallocate 0x00a08c thru 0x01748e (0xd403 bytes) 000434 1712275950.339926 punch from 0x013328 to 0x01748e (0x4167 bytes) 000436 1712275950.341973 punch from 0x007ca5 to 0x009671 (0x19cd bytes) 000438 1712275950.343320 fallocate 0x0323a7 thru 0x03a453 (0x80ad bytes) 000439 1712275950.344692 mapread 0x012f35 thru 0x018a92 (0x5b5e bytes) 000441 1712275950.347994 write 0x01e135 thru 0x02c1ab (0xe077 bytes) 000442 1712275950.348658 read 0x02d4b9 thru 0x02dd38 (0x880 bytes) 000443 1712275950.348907 fallocate 0x0095d8 thru 0x012653 (0x907c bytes) 000444 1712275950.350796 mapwrite 0x01ab08 thru 0x024c7f (0xa178 bytes) 000445 1712275950.357548 trunc from 0x03a454 to 0x00f160 (0x2b2f3 bytes) 000446 1712275950.362008 trunc from 0x00f161 to 0x0072c9 (0x7e97 bytes) 000448 1712275950.366026 write 0x00546e thru 0x00cb6f (0x7702 bytes) 000449 1712275950.366607 mapwrite 0x03e7bd thru 0x03ffff (0x1843 bytes) 000451 1712275950.377529 punch from 0x032087 to 0x03fe69 (0xdde3 bytes) 000452 1712275950.379242 mapread 0x01199c thru 0x01f955 (0xdfba bytes) 000453 1712275950.380973 MIRROR_EXTEND to 2 mirrors 000454 1712275950.492153 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000455 1712275950.520492 fallocate 0x000ce1 thru 0x00c0fd (0xb41d bytes) 000456 1712275950.523927 read 0x025f27 thru 0x028cf3 (0x2dcd bytes) 000459 1712275950.526387 write 0x02aa5e thru 0x02f964 (0x4f07 bytes) 000463 1712275950.531718 punch from 0x002122 to 0x00ff88 (0xde67 bytes) 000465 1712275950.533765 mapwrite 0x000c91 thru 0x005fb7 (0x5327 bytes) 000466 1712275950.542531 read 0x03799c thru 0x03ffff (0x8664 bytes) 000468 1712275950.546506 trunc from 0x040000 to 0x02fef7 (0x10108 bytes) 000469 1712275950.555591 fallocate 0x03f5aa thru 0x03ffff (0xa56 bytes) 000470 1712275950.557447 mapread 0x009a38 thru 0x00e3fb (0x49c4 bytes) 000472 1712275950.558127 write 0x00784f thru 0x01255c (0xad0e bytes) 000473 1712275950.558802 fallocate 0x014313 thru 0x01546c (0x115a bytes) 000474 1712275950.560477 punch from 0x01263f to 0x0217ca (0xf18c bytes) 000475 1712275950.564600 trunc from 0x040000 to 0x024ccd (0x1b332 bytes) 000476 1712275950.568803 fallocate 0x0246bc thru 0x026d4e (0x2693 bytes) 000477 1712275950.570482 mapwrite 0x03b385 thru 0x03ffff (0x4c7b bytes) 000478 1712275950.585823 punch from 0x006a7d to 0x006db1 (0x335 bytes) 000479 1712275950.587255 punch from 0x0103db to 0x01d507 (0xd12d bytes) 000480 1712275950.588695 write 0x00224d thru 0x00bd91 (0x9b45 bytes) 000486 1712275950.591516 mapwrite 0x011108 thru 0x01eaed (0xd9e6 bytes) 000487 1712275950.614461 mapwrite 0x002967 thru 0x00f7c7 (0xce61 bytes) 000488 1712275950.620656 punch from 0x024815 to 0x032ff3 (0xe7df bytes) 000489 1712275950.622126 read 0x009a9e thru 0x01986c (0xfdcf bytes) 000491 1712275950.625093 mapread 0x025262 thru 0x02f58c (0xa32b bytes) 000492 1712275950.625789 read 0x03c0d6 thru 0x03ffff (0x3f2a bytes) 000493 1712275950.626200 write 0x03b59e thru 0x03ffff (0x4a62 bytes) 000494 1712275950.626929 mapread 0x011e82 thru 0x017820 (0x599f bytes) 000495 1712275950.627537 mapwrite 0x02f6f8 thru 0x03d98a (0xe293 bytes) 000496 1712275950.634846 punch from 0x018bd3 to 0x028193 (0xf5c1 bytes) 000499 1712275950.637199 write 0x034896 thru 0x034c22 (0x38d bytes) 000500 1712275950.639505 MIRROR_EXTEND to 2 mirrors 000501 1712275950.761199 mapread 0x035fef thru 0x038b8c (0x2b9e bytes) 000502 1712275950.769497 write 0x0386e7 thru 0x03ffff (0x7919 bytes) 000503 1712275950.780102 mapwrite 0x02cbf9 thru 0x035222 (0x862a bytes) 000504 1712275950.789587 read 0x023069 thru 0x031d5e (0xecf6 bytes) 000505 1712275950.791702 mapwrite 0x0036c4 thru 0x00e919 (0xb256 bytes) 000506 1712275950.801557 trunc from 0x040000 to 0x0180af (0x27f50 bytes) 000507 1712275950.811846 fallocate 0x012068 thru 0x0127d2 (0x76b bytes) 000508 1712275950.813547 punch from 0x0085ef to 0x00c93e (0x4350 bytes) 000510 1712275950.815171 mapread 0x008660 thru 0x0123c5 (0x9d66 bytes) 000511 1712275950.819385 mapread 0x005cb0 thru 0x00ccda (0x702b bytes) 000513 1712275950.820121 mapwrite 0x0209d0 thru 0x02a691 (0x9cc2 bytes) 000514 1712275950.839521 trunc from 0x02a692 to 0x00675d (0x23f34 bytes) 000515 1712275950.844102 read 0x0054e7 thru 0x00675d (0x1277 bytes) 000516 1712275950.845897 MIRROR_EXTEND to 3 mirrors 000517 1712275950.933045 fallocate 0x0366ad thru 0x03ffff (0x9953 bytes) 000518 1712275950.943036 punch from 0x033368 to 0x035b95 (0x282e bytes) 000519 1712275950.944495 punch from 0x007cd2 to 0x01460f (0xc93e bytes) 000520 1712275950.947285 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000521 1712275950.997317 write 0x01c08b thru 0x023b01 (0x7a77 bytes) 000522 1712275951.001516 mapwrite 0x032295 thru 0x035252 (0x2fbe bytes) 000523 1712275951.010181 trunc from 0x040000 to 0x01fde0 (0x2021f bytes) 000525 1712275951.018874 fallocate 0x0380a7 thru 0x03ffff (0x7f59 bytes) 000527 1712275951.020789 mapread 0x00b0fa thru 0x019744 (0xe64b bytes) 000528 1712275951.025974 write 0x005644 thru 0x00ee18 (0x97d5 bytes) 000529 1712275951.029122 write 0x02cabb thru 0x02d89e (0xde4 bytes) 000530 1712275951.029882 punch from 0x00f169 to 0x00fcd9 (0xb71 bytes) 000531 1712275951.033450 MIRROR_RESYNC 2 mirrors 000533 1712275951.148692 mapwrite 0x03c907 thru 0x03ffff (0x36f9 bytes) 000535 1712275951.175447 write 0x035ea3 thru 0x03fd67 (0x9ec5 bytes) 000536 1712275951.178246 punch from 0x0003e2 to 0x009dee (0x9a0d bytes) 000537 1712275951.180238 trunc from 0x040000 to 0x0039e0 (0x3c61f bytes) 000538 1712275951.185758 punch from 0x001b8c to 0x0039e0 (0x1e55 bytes) 000539 1712275951.187760 fallocate 0x0231e6 thru 0x02cb50 (0x996b bytes) 000540 1712275951.189458 read 0x0036ea thru 0x0039e0 (0x2f7 bytes) 000541 1712275951.191238 fallocate 0x00e863 thru 0x01a307 (0xbaa5 bytes) 000542 1712275951.192949 read 0x014ea4 thru 0x0191c3 (0x4320 bytes) 000543 1712275951.195831 trunc from 0x01a308 to 0x02afcc (0x10cc5 bytes) 000546 1712275951.201914 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000547 1712275951.233158 mapread 0x00394f thru 0x011174 (0xd826 bytes) 000548 1712275951.240206 trunc from 0x02afcd to 0x012879 (0x18753 bytes) 000549 1712275951.245335 trunc from 0x01287a to 0x02427a (0x11a01 bytes) 000550 1712275951.248937 write 0x01f58c thru 0x02c690 (0xd105 bytes) 000551 1712275951.251925 read 0x02ac31 thru 0x02b887 (0xc57 bytes) 000553 1712275951.252527 fallocate 0x0398a2 thru 0x03ffff (0x675e bytes) 000554 1712275951.254093 fallocate 0x006555 thru 0x007356 (0xe02 bytes) 000555 1712275951.255349 fallocate 0x0321b7 thru 0x033e23 (0x1c6d bytes) 000557 1712275951.257650 mapread 0x02eaec thru 0x031740 (0x2c55 bytes) 000559 1712275951.260776 fallocate 0x015747 thru 0x021e8f (0xc749 bytes) 000561 1712275951.263704 MIRROR_EXTEND to 2 mirrors 000562 1712275951.383413 read 0x0212f8 thru 0x02c1fd (0xaf06 bytes) 000563 1712275951.387225 trunc from 0x033e24 to 0x01365a (0x207c9 bytes) 000565 1712275951.401703 read 0x00e1c2 thru 0x01365a (0x5499 bytes) 000566 1712275951.405025 mapwrite 0x014537 thru 0x02062b (0xc0f5 bytes) 000567 1712275951.420903 fallocate 0x00cd6f thru 0x014279 (0x750b bytes) 000568 1712275951.422784 trunc from 0x02062c to 0x036dc7 (0x1679c bytes) 000569 1712275951.427099 punch from 0x005e8d to 0x0107e6 (0xa95a bytes) 000570 1712275951.428727 write 0x0299d5 thru 0x02c25f (0x288b bytes) 000572 1712275951.431851 mapread 0x019279 thru 0x01cf4d (0x3cd5 bytes) 000573 1712275951.432317 mapread 0x00c347 thru 0x01bf10 (0xfbca bytes) 000575 1712275951.439653 fallocate 0x01c9f0 thru 0x02bd75 (0xf386 bytes) 000576 1712275951.441849 trunc from 0x036dc8 to 0x01338d (0x23a3a bytes) 000578 1712275951.448388 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000579 1712275951.481078 mapwrite 0x00a760 thru 0x012deb (0x868c bytes) 000583 1712275951.494745 mapread 0x003690 thru 0x00e241 (0xabb2 bytes) 000584 1712275951.497091 fallocate 0x0170e0 thru 0x0238bd (0xc7de bytes) 000586 1712275951.499555 mapwrite 0x03d8ad thru 0x03ffff (0x2753 bytes) 000587 1712275951.512911 mapwrite 0x016ac4 thru 0x021e47 (0xb384 bytes) 000588 1712275951.523575 mapwrite 0x0333d8 thru 0x03ffff (0xcc28 bytes) 000589 1712275951.532052 write 0x005a28 thru 0x00d1c2 (0x779b bytes) 000590 1712275951.532958 trunc from 0x040000 to 0x02ef07 (0x110f8 bytes) 000591 1712275951.538725 fallocate 0x0393a0 thru 0x03e996 (0x55f7 bytes) 000592 1712275951.540687 fallocate 0x0150c2 thru 0x022689 (0xd5c8 bytes) 000593 1712275951.542615 write 0x00a397 thru 0x01a2da (0xff44 bytes) 000594 1712275951.543863 fallocate 0x02385b thru 0x02ff38 (0xc6de bytes) 000596 1712275951.546051 mapwrite 0x01c892 thru 0x027f4d (0xb6bc bytes) 000597 1712275951.555586 mapwrite 0x03bd95 thru 0x03f1f1 (0x345d bytes) 000598 1712275951.569045 punch from 0x035b3c to 0x039ae8 (0x3fad bytes) 000599 1712275951.570758 write 0x023d26 thru 0x02d0b8 (0x9393 bytes) 000600 1712275951.571641 read 0x000cd8 thru 0x009027 (0x8350 bytes) 000601 1712275951.574638 MIRROR_EXTEND to 2 mirrors 000602 1712275951.703509 read 0x025d71 thru 0x032783 (0xca13 bytes) 000603 1712275951.708589 mapwrite 0x0198d8 thru 0x022317 (0x8a40 bytes) 000606 1712275951.744003 mapread 0x017c98 thru 0x026805 (0xeb6e bytes) 000607 1712275951.747035 punch from 0x011751 to 0x018451 (0x6d01 bytes) 000608 1712275951.749499 trunc from 0x03f1f2 to 0x01d614 (0x21bdd bytes) 000610 1712275951.760584 read 0x0036cf thru 0x012d1b (0xf64d bytes) 000611 1712275951.763861 punch from 0x0118f0 to 0x0165ba (0x4ccb bytes) 000612 1712275951.765497 read 0x016da0 thru 0x018fb2 (0x2213 bytes) 000613 1712275951.768145 fallocate 0x031488 thru 0x03e904 (0xd47d bytes) 000614 1712275951.769790 write 0x038aa3 thru 0x03ffff (0x755d bytes) 000615 1712275951.773424 punch from 0x00e8e9 to 0x01c8a6 (0xdfbe bytes) 000616 1712275951.775464 punch from 0x03637e to 0x03ffff (0x9c82 bytes) 000617 1712275951.779351 write 0x003d2f thru 0x009c8a (0x5f5c bytes) 000619 1712275951.780317 trunc from 0x040000 to 0x00dcb6 (0x32349 bytes) 000621 1712275951.785524 fallocate 0x02a186 thru 0x039c0c (0xfa87 bytes) 000622 1712275951.789031 MIRROR_RESYNC 2 mirrors 000623 1712275951.893246 trunc from 0x039c0d to 0x0189fd (0x2120f bytes) 000624 1712275951.909030 write 0x02fba5 thru 0x03c311 (0xc76d bytes) 000625 1712275951.910066 trunc from 0x03c312 to 0x0159f8 (0x26919 bytes) 000626 1712275951.915068 read 0x00bef3 thru 0x011414 (0x5522 bytes) 000628 1712275951.917454 fallocate 0x03415b thru 0x03ded4 (0x9d7a bytes) 000629 1712275951.919051 trunc from 0x0159f9 to 0x025382 (0xf98a bytes) 000631 1712275951.923315 mapread 0x015755 thru 0x01c279 (0x6b25 bytes) 000632 1712275951.928437 read 0x00db80 thru 0x015c89 (0x810a bytes) 000633 1712275951.930955 fallocate 0x02fba9 thru 0x03e9db (0xee33 bytes) 000634 1712275951.932507 trunc from 0x025383 to 0x005bd6 (0x1f7ac bytes) 000635 1712275951.937592 trunc from 0x005bd7 to 0x011b72 (0xbf9c bytes) 000636 1712275951.941860 trunc from 0x011b73 to 0x01591e (0x3dac bytes) 000637 1712275951.946564 read 0x000cff thru 0x004c9f (0x3fa1 bytes) 000639 1712275951.947099 write 0x00db4b thru 0x00e60d (0xac3 bytes) 000640 1712275951.950524 read 0x011f3d thru 0x01591e (0x39e2 bytes) 000641 1712275951.951020 mapread 0x008419 thru 0x011043 (0x8c2b bytes) 000642 1712275951.953456 MIRROR_RESYNC 2 mirrors 000643 1712275952.072559 mapwrite 0x025b3c thru 0x02973b (0x3c00 bytes) 000644 1712275952.098177 MIRROR_EXTEND to 3 mirrors 000647 1712275952.220029 fallocate 0x03af7a thru 0x03cec2 (0x1f49 bytes) 000648 1712275952.231020 read 0x0131a5 thru 0x018f03 (0x5d5f bytes) 000649 1712275952.234412 mapread 0x000c0e thru 0x008c49 (0x803c bytes) 000650 1712275952.242001 trunc from 0x02973c to 0x038ca6 (0xf56b bytes) 000651 1712275952.254010 write 0x00465e thru 0x005b7e (0x1521 bytes) 000654 1712275952.255497 read 0x019c4f thru 0x01aeb2 (0x1264 bytes) 000655 1712275952.260591 MIRROR_RESYNC 3 mirrors 000656 1712275952.469108 punch from 0x00341f to 0x00d5c4 (0xa1a6 bytes) 000657 1712275952.480920 write 0x00201d thru 0x00664c (0x4630 bytes) 000658 1712275952.485049 write 0x00e74a thru 0x018a19 (0xa2d0 bytes) 000659 1712275952.490538 mapread 0x0261d0 thru 0x029e1c (0x3c4d bytes) 000660 1712275952.497222 MIRROR_RESYNC 3 mirrors 000661 1712275952.693454 punch from 0x009fa1 to 0x00b68a (0x16ea bytes) 000665 1712275952.705811 MIRROR_RESYNC 3 mirrors 000666 1712275952.884140 trunc from 0x038ca7 to 0x01f280 (0x19a26 bytes) 000667 1712275952.896569 mapread 0x0089cb thru 0x015f8c (0xd5c2 bytes) 000668 1712275952.904528 MIRROR_RESYNC 3 mirrors 000669 1712275953.052660 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000671 1712275953.112511 mapwrite 0x03f859 thru 0x03ffff (0x7a7 bytes) 000672 1712275953.130436 MIRROR_RESYNC 2 mirrors 000674 1712275953.245579 fallocate 0x000ec8 thru 0x00c236 (0xb36f bytes) 000675 1712275953.258765 MIRROR_EXTEND to 3 mirrors 000676 1712275953.388743 write 0x00025b thru 0x007b49 (0x78ef bytes) 000677 1712275953.399919 read 0x01e17a thru 0x01e1a6 (0x2d bytes) 000678 1712275953.403290 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000679 1712275953.466391 read 0x02ee54 thru 0x03d8e1 (0xea8e bytes) 000680 1712275953.469760 mapread 0x0121b9 thru 0x016ff3 (0x4e3b bytes) 000681 1712275953.475861 read 0x011dfb thru 0x017f1c (0x6122 bytes) 000682 1712275953.476344 write 0x030a7b thru 0x0330fb (0x2681 bytes) 000683 1712275953.478530 fallocate 0x03c2d4 thru 0x03e01c (0x1d49 bytes) 000685 1712275953.480900 trunc from 0x040000 to 0x03eeb2 (0x114d bytes) 000686 1712275953.488877 read 0x019b6d thru 0x021945 (0x7dd9 bytes) 000687 1712275953.492177 mapwrite 0x013886 thru 0x018e06 (0x5581 bytes) 000688 1712275953.499192 punch from 0x03e1e4 to 0x03eeb2 (0xccf bytes) 000689 1712275953.500584 read 0x019c2a thru 0x029bfb (0xffd2 bytes) 000691 1712275953.501365 write 0x004f1c thru 0x005d31 (0xe16 bytes) 000692 1712275953.502016 write 0x0145e1 thru 0x015a54 (0x1474 bytes) 000694 1712275953.504680 MIRROR_RESYNC 2 mirrors 000695 1712275953.596191 trunc from 0x03eeb3 to 0x0183bf (0x26af3 bytes) 000696 1712275953.610741 punch from 0x00530c to 0x00aa60 (0x5755 bytes) 000698 1712275953.613412 MIRROR_RESYNC 2 mirrors 000699 1712275953.691630 punch from 0x002cce to 0x0050a2 (0x23d5 bytes) 000700 1712275953.701576 MIRROR_RESYNC 2 mirrors 000701 1712275953.796426 read 0x0126b3 thru 0x0183bf (0x5d0d bytes) 000702 1712275953.798566 write 0x007272 thru 0x010360 (0x90ef bytes) 000703 1712275953.807603 write 0x02593a thru 0x0318df (0xbfa6 bytes) 000705 1712275953.810305 MIRROR_RESYNC 2 mirrors 000706 1712275953.920924 fallocate 0x01a6cd thru 0x026385 (0xbcb9 bytes) 000707 1712275953.929982 trunc from 0x0318e0 to 0x03e259 (0xc97a bytes) 000708 1712275953.938263 punch from 0x00040b to 0x0086a0 (0x8296 bytes) 000710 1712275953.941849 MIRROR_EXTEND to 3 mirrors 000711 1712275954.076837 write 0x01407e thru 0x020e7a (0xcdfd bytes) 000712 1712275954.090029 read 0x0132a9 thru 0x014f8f (0x1ce7 bytes) 000713 1712275954.092005 trunc from 0x03e25a to 0x0319c6 (0xc893 bytes) 000715 1712275954.099024 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000716 1712275954.145625 MIRROR_EXTEND to 3 mirrors 000717 1712275954.255306 MIRROR_EXTEND to 4 mirrors 000718 1712275954.369401 MIRROR_EXTEND to 5 mirrors 000720 1712275954.486863 punch from 0x002803 to 0x01105c (0xe85a bytes) 000721 1712275954.496895 mapread 0x030f2c thru 0x0319c6 (0xa9b bytes) 000723 1712275954.501254 MIRROR_RESYNC 5 mirrors 000725 1712275954.665928 write 0x0161a9 thru 0x021074 (0xaecc bytes) 000726 1712275954.678216 fallocate 0x00280b thru 0x008678 (0x5e6e bytes) 000728 1712275954.680353 mapwrite 0x0012e0 thru 0x00a602 (0x9323 bytes) 000729 1712275954.688616 write 0x00a28e thru 0x010f80 (0x6cf3 bytes) 000730 1712275954.691124 mapwrite 0x03e17b thru 0x03ffff (0x1e85 bytes) 000731 1712275954.708079 read 0x0338fc thru 0x03f23b (0xb940 bytes) 000732 1712275954.710431 write 0x006c02 thru 0x00982b (0x2c2a bytes) 000733 1712275954.710932 mapwrite 0x03911c thru 0x03ffff (0x6ee4 bytes) 000734 1712275954.716402 trunc from 0x040000 to 0x02e041 (0x11fbe bytes) 000735 1712275954.720522 fallocate 0x03ab32 thru 0x03ffff (0x54ce bytes) 000736 1712275954.722311 trunc from 0x02e042 to 0x03b649 (0xd608 bytes) 000737 1712275954.727854 MIRROR_EXTEND to 6 mirrors 000739 1712275954.850620 MIRROR_EXTEND to 7 mirrors 000740 1712275954.952784 mapread 0x011c10 thru 0x01892f (0x6d20 bytes) 000741 1712275954.958161 MIRROR_EXTEND to 8 mirrors 000742 1712275955.071567 punch from 0x0096c3 to 0x0168cb (0xd209 bytes) 000743 1712275955.082308 read 0x026957 thru 0x02fb7b (0x9225 bytes) 000744 1712275955.085146 mapwrite 0x0389da thru 0x03dacd (0x50f4 bytes) 000746 1712275955.107290 trunc from 0x03dace to 0x02e08b (0xfa42 bytes) 000747 1712275955.112233 fallocate 0x02ef6b thru 0x038c74 (0x9d0a bytes) 000748 1712275955.113909 punch from 0x00751b to 0x00916a (0x1c50 bytes) 000749 1712275955.115212 fallocate 0x03fd57 thru 0x03ffff (0x2a9 bytes) 000750 1712275955.116593 mapread 0x0204ee thru 0x02e08b (0xdb9e bytes) 000751 1712275955.118751 mapwrite 0x016021 thru 0x0168ea (0x8ca bytes) 000753 1712275955.125162 punch from 0x015394 to 0x024ffa (0xfc67 bytes) 000754 1712275955.126729 fallocate 0x00c581 thru 0x01ae43 (0xe8c3 bytes) 000755 1712275955.128236 fallocate 0x02fbdd thru 0x03ca4b (0xce6f bytes) 000756 1712275955.129757 punch from 0x00c162 to 0x019d17 (0xdbb6 bytes) 000757 1712275955.132586 MIRROR_RESYNC 8 mirrors 000758 1712275955.492471 trunc from 0x02e08c to 0x0307ec (0x2761 bytes) 000759 1712275955.505254 trunc from 0x0307ed to 0x030989 (0x19d bytes) 000760 1712275955.509940 fallocate 0x034b2b thru 0x037037 (0x250d bytes) 000762 1712275955.511780 write 0x0265c1 thru 0x0334a0 (0xcee0 bytes) 000764 1712275955.514384 fallocate 0x00319f thru 0x00ec2a (0xba8c bytes) 000765 1712275955.515998 read 0x01d949 thru 0x02a601 (0xccb9 bytes) 000766 1712275955.518360 read 0x01e3e3 thru 0x02738f (0x8fad bytes) 000768 1712275955.518854 write 0x00575b thru 0x012c1f (0xd4c5 bytes) 000769 1712275955.522878 trunc from 0x0334a1 to 0x031b88 (0x1918 bytes) 000771 1712275955.527619 mapwrite 0x012aa3 thru 0x01c69c (0x9bfa bytes) 000772 1712275955.535145 read 0x000ca0 thru 0x004db9 (0x411a bytes) 000773 1712275955.535516 mapread 0x02d187 thru 0x02d8ef (0x769 bytes) 000776 1712275955.536338 mapread 0x02caa3 thru 0x031b88 (0x50e6 bytes) 000777 1712275955.538199 MIRROR_EXTEND to 9 mirrors 000778 1712275955.646070 mapread 0x020346 thru 0x026d16 (0x69d1 bytes) 000779 1712275955.655939 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000780 1712275955.723071 mapread 0x0000d9 thru 0x007584 (0x74ac bytes) 000781 1712275955.737448 punch from 0x01eb32 to 0x023070 (0x453f bytes) 000782 1712275955.749263 mapread 0x02ea6e thru 0x031b88 (0x311b bytes) 000783 1712275955.752168 punch from 0x02976d to 0x031b88 (0x841c bytes) 000784 1712275955.754376 fallocate 0x000204 thru 0x00b9ff (0xb7fc bytes) 000785 1712275955.756108 punch from 0x011151 to 0x01fa38 (0xe8e8 bytes) 000787 1712275955.758155 read 0x0012c0 thru 0x00f794 (0xe4d5 bytes) 000788 1712275955.761679 trunc from 0x031b89 to 0x025dfe (0xbd8a bytes) 000789 1712275955.774131 trunc from 0x025dff to 0x00820d (0x1dbf1 bytes) 000790 1712275955.779453 MIRROR_RESYNC 8 mirrors 000791 1712275955.931933 trunc from 0x00820e to 0x01dea9 (0x15c9c bytes) 000792 1712275955.944789 read 0x010984 thru 0x01dea9 (0xd526 bytes) 000794 1712275955.947710 trunc from 0x01deaa to 0x00dc7f (0x1022a bytes) 000795 1712275955.952932 write 0x02dd64 thru 0x02e78d (0xa2a bytes) 000796 1712275955.953770 mapwrite 0x00ee53 thru 0x0184f2 (0x96a0 bytes) 000797 1712275955.964452 punch from 0x013e4d to 0x015246 (0x13fa bytes) 000798 1712275955.965816 mapwrite 0x024641 thru 0x02dec2 (0x9882 bytes) 000799 1712275955.973073 write 0x03abe9 thru 0x03b349 (0x761 bytes) 000800 1712275955.973672 mapread 0x00f8e8 thru 0x01f396 (0xfaaf bytes) 000801 1712275955.978548 MIRROR_RESYNC 8 mirrors 000803 1712275956.384103 fallocate 0x02fefe thru 0x0384a0 (0x85a3 bytes) 000804 1712275956.393360 mapwrite 0x03e5d7 thru 0x03ffff (0x1a29 bytes) 000807 1712275956.406356 read 0x0251a2 thru 0x02db8f (0x89ee bytes) 000810 1712275956.409750 read 0x01b9f2 thru 0x02638e (0xa99d bytes) 000812 1712275956.414096 trunc from 0x040000 to 0x03213c (0xdec3 bytes) 000814 1712275956.418286 trunc from 0x03213d to 0x01fd6c (0x123d0 bytes) 000815 1712275956.421594 write 0x030b2a thru 0x03a41c (0x98f3 bytes) 000816 1712275956.422302 mapwrite 0x00b1e1 thru 0x00f6bc (0x44dc bytes) 000817 1712275956.427954 write 0x010204 thru 0x01d007 (0xce04 bytes) 000818 1712275956.428905 punch from 0x005fed to 0x006e8a (0xe9e bytes) 000819 1712275956.430067 mapwrite 0x01761d thru 0x020cec (0x96d0 bytes) 000820 1712275956.436000 fallocate 0x03d7d4 thru 0x03ffff (0x282c bytes) 000821 1712275956.437440 punch from 0x02f961 to 0x03ec1b (0xf2bb bytes) 000822 1712275956.440297 MIRROR_EXTEND to 9 mirrors 000823 1712275956.560097 punch from 0x0215a4 to 0x026b16 (0x5573 bytes) 000825 1712275956.569044 mapwrite 0x0317a5 thru 0x03ff71 (0xe7cd bytes) 000826 1712275956.581915 punch from 0x03ef3e to 0x03ffff (0x10c2 bytes) 000827 1712275956.583573 mapwrite 0x009f8d thru 0x0155a6 (0xb61a bytes) 000828 1712275956.594920 trunc from 0x040000 to 0x002d97 (0x3d268 bytes) 000829 1712275956.606444 mapread 0x00207a thru 0x002d97 (0xd1e bytes) 000831 1712275956.607513 write 0x03a9bd thru 0x03b689 (0xccd bytes) 000832 1712275956.608242 fallocate 0x014673 thru 0x01d763 (0x90f1 bytes) 000833 1712275956.610259 fallocate 0x0043d5 thru 0x004df0 (0xa1c bytes) 000834 1712275956.612132 fallocate 0x03b08a thru 0x03ffff (0x4f76 bytes) 000835 1712275956.613765 write 0x0288fc thru 0x034fdd (0xc6e2 bytes) 000836 1712275956.618867 punch from 0x028878 to 0x02e3c9 (0x5b52 bytes) 000837 1712275956.623211 read 0x022535 thru 0x0297f1 (0x72bd bytes) 000838 1712275956.625095 read 0x000437 thru 0x00d7ca (0xd394 bytes) 000839 1712275956.628784 mapread 0x02aab9 thru 0x03672f (0xbc77 bytes) 000840 1712275956.629431 read 0x0330bd thru 0x03ea7d (0xb9c1 bytes) 000841 1712275956.629873 read 0x029eee thru 0x033641 (0x9754 bytes) 000842 1712275956.630167 read 0x023acc thru 0x031ba7 (0xe0dc bytes) 000843 1712275956.630504 mapwrite 0x024222 thru 0x030c3c (0xca1b bytes) 000845 1712275956.638183 write 0x00652a thru 0x00bc22 (0x56f9 bytes) 000847 1712275956.639342 mapwrite 0x00414d thru 0x004585 (0x439 bytes) 000848 1712275956.643612 trunc from 0x040000 to 0x011b00 (0x2e4ff bytes) 000849 1712275956.647695 mapwrite 0x0271a7 thru 0x033695 (0xc4ef bytes) 000850 1712275956.661099 read 0x002a02 thru 0x00ffdd (0xd5dc bytes) 000851 1712275956.661448 read 0x02efaf thru 0x033695 (0x46e7 bytes) 000852 1712275956.661712 mapread 0x00dd04 thru 0x01ad46 (0xd043 bytes) 000853 1712275956.662137 fallocate 0x02173b thru 0x02f648 (0xdf0e bytes) 000854 1712275956.663588 trunc from 0x033696 to 0x02238c (0x11309 bytes) 000855 1712275956.667445 read 0x01e8eb thru 0x020b16 (0x222c bytes) 000856 1712275956.667748 trunc from 0x02238d to 0x022cfe (0x972 bytes) 000857 1712275956.671085 fallocate 0x02d5e8 thru 0x03bf32 (0xe94b bytes) 000858 1712275956.672675 fallocate 0x025204 thru 0x02d234 (0x8031 bytes) 000859 1712275956.674222 mapread 0x0084ed thru 0x018357 (0xfe6b bytes) 000860 1712275956.674980 read 0x00a1a8 thru 0x015e87 (0xbce0 bytes) 000861 1712275956.675437 fallocate 0x0053cd thru 0x00c61d (0x7251 bytes) 000862 1712275956.677115 mapwrite 0x005f5d thru 0x0147c6 (0xe86a bytes) 000863 1712275956.685141 write 0x03c961 thru 0x03fa7b (0x311b bytes) 000864 1712275956.685712 punch from 0x01b669 to 0x01f922 (0x42ba bytes) 000865 1712275956.686952 mapread 0x025257 thru 0x027a71 (0x281b bytes) 000866 1712275956.689613 mapwrite 0x01df9b thru 0x02309e (0x5104 bytes) 000868 1712275956.695794 mapwrite 0x02cb84 thru 0x030c3c (0x40b9 bytes) 000870 1712275956.701499 mapread 0x020696 thru 0x025405 (0x4d70 bytes) 000871 1712275956.702038 fallocate 0x015341 thru 0x0189fe (0x36be bytes) 000872 1712275956.703712 punch from 0x03bb2d to 0x03fa7b (0x3f4f bytes) 000873 1712275956.705119 write 0x03f8cc thru 0x03ffff (0x734 bytes) 000875 1712275956.706735 punch from 0x031c47 to 0x03288f (0xc49 bytes) 000876 1712275956.707769 write 0x00bf70 thru 0x015778 (0x9809 bytes) 000877 1712275956.708326 read 0x014b91 thru 0x01c829 (0x7c99 bytes) 000878 1712275956.708595 mapwrite 0x0084ef thru 0x0094c9 (0xfdb bytes) 000879 1712275956.715906 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000882 1712275956.768541 write 0x009ebe thru 0x00eaf4 (0x4c37 bytes) 000883 1712275956.772447 mapwrite 0x03f852 thru 0x03ffff (0x7ae bytes) 000884 1712275956.779171 read 0x01c965 thru 0x02b9ae (0xf04a bytes) 000885 1712275956.781681 mapwrite 0x034285 thru 0x03bfa1 (0x7d1d bytes) 000886 1712275956.788263 mapwrite 0x00621e thru 0x011337 (0xb11a bytes) 000887 1712275956.797832 fallocate 0x027e8e thru 0x0335ba (0xb72d bytes) 000889 1712275956.799654 mapread 0x019850 thru 0x01cf40 (0x36f1 bytes) 000890 1712275956.801452 mapwrite 0x016b08 thru 0x01f658 (0x8b51 bytes) 000891 1712275956.807326 punch from 0x03a501 to 0x03ffff (0x5aff bytes) 000893 1712275956.808928 punch from 0x00462b to 0x008111 (0x3ae7 bytes) 000894 1712275956.810370 read 0x016642 thru 0x019a14 (0x33d3 bytes) 000896 1712275956.812471 MIRROR_EXTEND to 9 mirrors 000898 1712275956.943624 mapread 0x02b37e thru 0x02d221 (0x1ea4 bytes) 000899 1712275956.950972 fallocate 0x0100d9 thru 0x019cce (0x9bf6 bytes) 000900 1712275956.959707 mapwrite 0x03f5a6 thru 0x03ffff (0xa5a bytes) 000901 1712275956.965461 trunc from 0x040000 to 0x005774 (0x3a88b bytes) 000902 1712275956.978524 MIRROR_EXTEND to 10 mirrors 000903 1712275957.094973 punch from 0x001ae5 to 0x005774 (0x3c90 bytes) 000905 1712275957.106108 mapwrite 0x037618 thru 0x03ffff (0x89e8 bytes) 000906 1712275957.123411 trunc from 0x040000 to 0x009da7 (0x36258 bytes) 000907 1712275957.127843 trunc from 0x009da8 to 0x000ab4 (0x92f3 bytes) 000909 1712275957.132316 read 0x0007d5 thru 0x000ab4 (0x2e0 bytes) 000910 1712275957.134935 MIRROR_RESYNC 10 mirrors 000912 1712275957.282863 read 0x000342 thru 0x000ab4 (0x773 bytes) 000913 1712275957.283073 trunc from 0x000ab5 to 0x0119e6 (0x10f32 bytes) 000914 1712275957.296350 mapwrite 0x000b65 thru 0x0089b8 (0x7e54 bytes) 000915 1712275957.308586 mapread 0x00a275 thru 0x00c5d4 (0x2360 bytes) 000916 1712275957.309238 punch from 0x0013fc to 0x0068a8 (0x54ad bytes) 000917 1712275957.311066 fallocate 0x00a091 thru 0x00abda (0xb4a bytes) 000919 1712275957.313065 punch from 0x001db1 to 0x003a98 (0x1ce8 bytes) 000920 1712275957.316178 MIRROR_EXTEND to 11 mirrors 000921 1712275957.428589 punch from 0x006b12 to 0x008f8b (0x247a bytes) 000922 1712275957.440754 mapread 0x007624 thru 0x0117ec (0xa1c9 bytes) 000923 1712275957.448243 punch from 0x007646 to 0x0119e6 (0xa3a1 bytes) 000928 1712275957.451381 mapread 0x00b604 thru 0x0119e6 (0x63e3 bytes) 000929 1712275957.455785 MIRROR_EXTEND to 12 mirrors 000930 1712275957.566509 trunc from 0x0119e7 to 0x031963 (0x1ff7d bytes) 000931 1712275957.585973 fallocate 0x01f49d thru 0x02ecc5 (0xf829 bytes) 000932 1712275957.589036 MIRROR_EXTEND to 13 mirrors 000933 1712275957.706312 fallocate 0x016a4c thru 0x0234de (0xca93 bytes) 000934 1712275957.716470 mapread 0x030334 thru 0x031963 (0x1630 bytes) 000935 1712275957.719889 read 0x00e2c8 thru 0x011c46 (0x397f bytes) 000936 1712275957.721725 read 0x01f87e thru 0x023fb5 (0x4738 bytes) 000937 1712275957.725919 MIRROR_EXTEND to 14 mirrors 000938 1712275957.840771 mapwrite 0x01b6ed thru 0x01d943 (0x2257 bytes) 000939 1712275957.873591 write 0x010e7f thru 0x01ffe4 (0xf166 bytes) 000941 1712275957.878685 MIRROR_EXTEND to 15 mirrors 000942 1712275958.002167 fallocate 0x01ce9e thru 0x025774 (0x88d7 bytes) 000943 1712275958.011802 write 0x03077d thru 0x035977 (0x51fb bytes) 000944 1712275958.014386 read 0x00d0b4 thru 0x00ef85 (0x1ed2 bytes) 000946 1712275958.017523 MIRROR_RESYNC 15 mirrors 000947 1712275958.632990 read 0x029dbe thru 0x035977 (0xbbba bytes) 000948 1712275958.636376 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000949 1712275958.730205 read 0x02f6ed thru 0x035977 (0x628b bytes) 000950 1712275958.732612 mapread 0x028b2e thru 0x029ece (0x13a1 bytes) 000951 1712275958.735201 fallocate 0x009353 thru 0x017000 (0xdcae bytes) 000952 1712275958.748260 MIRROR_SPLIT mirror 8 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000954 1712275958.798511 fallocate 0x02ff65 thru 0x03324e (0x32ea bytes) 000955 1712275958.800198 read 0x013e6b thru 0x01553a (0x16d0 bytes) 000958 1712275958.802560 fallocate 0x02218d thru 0x02ea14 (0xc888 bytes) 000959 1712275958.804363 read 0x004805 thru 0x00f470 (0xac6c bytes) 000960 1712275958.807861 trunc from 0x035978 to 0x030cee (0x4c89 bytes) 000961 1712275958.817209 mapread 0x01b511 thru 0x01c1a1 (0xc91 bytes) 000962 1712275958.817657 write 0x028d77 thru 0x035165 (0xc3ef bytes) 000963 1712275958.819781 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000964 1712275958.872490 mapwrite 0x026aed thru 0x0336ca (0xcbde bytes) 000965 1712275958.887631 mapread 0x01ccb5 thru 0x022dd9 (0x6125 bytes) 000966 1712275958.890159 punch from 0x00da0b to 0x01c17d (0xe773 bytes) 000967 1712275958.891895 mapwrite 0x039055 thru 0x03f5cb (0x6577 bytes) 000968 1712275958.904845 write 0x00b860 thru 0x00e620 (0x2dc1 bytes) 000969 1712275958.907592 mapread 0x029736 thru 0x03683d (0xd108 bytes) 000970 1712275958.908345 read 0x0087f0 thru 0x01520f (0xca20 bytes) 000971 1712275958.909903 trunc from 0x03f5cc to 0x01aebe (0x2470d bytes) 000973 1712275958.913925 read 0x017c43 thru 0x018693 (0xa51 bytes) 000974 1712275958.916191 read 0x01ac31 thru 0x01aebe (0x28e bytes) 000975 1712275958.916583 read 0x011bf0 thru 0x018310 (0x6721 bytes) 000977 1712275958.917303 read 0x00bc72 thru 0x01299d (0x6d2c bytes) 000978 1712275958.917728 mapwrite 0x01a09c thru 0x01e2ce (0x4233 bytes) 000980 1712275958.928656 fallocate 0x00862b thru 0x00b47d (0x2e53 bytes) 000981 1712275958.929993 write 0x00dcb3 thru 0x011f61 (0x42af bytes) 000984 1712275958.930921 punch from 0x018c0d to 0x01e2ce (0x56c2 bytes) 000985 1712275958.932191 punch from 0x01cee9 to 0x01e2ce (0x13e6 bytes) 000986 1712275958.933681 read 0x01139c thru 0x01cbad (0xb812 bytes) 000987 1712275958.935717 trunc from 0x01e2cf to 0x00bbec (0x126e2 bytes) 000988 1712275958.939718 fallocate 0x02f523 thru 0x03751b (0x7ff9 bytes) 000989 1712275958.942325 MIRROR_RESYNC 12 mirrors 000990 1712275959.289746 punch from 0x0092e4 to 0x00bbec (0x2909 bytes) 000991 1712275959.300200 mapread 0x001fa6 thru 0x0083ca (0x6425 bytes) 000992 1712275959.306023 mapread 0x005b09 thru 0x00bbec (0x60e4 bytes) 000994 1712275959.306933 fallocate 0x00a003 thru 0x01428c (0xa28a bytes) 000997 1712275959.309548 punch from 0x0107bf to 0x01428c (0x3ace bytes) 000998 1712275959.310941 trunc from 0x01428d to 0x035355 (0x210c9 bytes) 000999 1712275959.316291 fallocate 0x00270b thru 0x0075e8 (0x4ede bytes) 001000 1712275959.317935 write 0x00aa8c thru 0x00b91a (0xe8f bytes) All operations completed A-OK! PASS 70a (15s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 20:12:42 (1712275962) Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre2 Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6180640 .. Extending file size to 1711136 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9038880 .. Extending file size to 3630464 .. Extending file size to 8046528 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8808608 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8933376 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9176832 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9088384 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9191008 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9252512 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 12426 12427 12428 12430 12433 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) 192.168.204.120@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg420-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:13:47 (1712276027) Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre2 Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3054208 .. Extending file size to 9008704 .. done Extending file size to 8593024 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8727200 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8948320 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9116576 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9165408 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 15863 15864 15866 15867 15870 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) 192.168.204.120@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg420-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:14:52 (1712276092) Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2580 1285108 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3360 1284328 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21672 3585348 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15768 3591252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37440 7176600 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2580 1285108 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3360 1284328 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 21672 3585348 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 15768 3591252 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37440 7176600 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0417995 s, 50.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0414677 s, 50.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:14:58 (1712276098) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0243005 s, 86.3 MB/s PASS 202 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 20:15:01 (1712276101) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } PASS 203 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:15:05 (1712276105) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0199268 s, 105 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x102e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x102e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x102f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:15:08 (1712276108) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0246391 s, 85.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } PASS 204b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:15:11 (1712276111) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0202338 s, 104 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } PASS 204c (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:15:36 (1712276136) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134343 s, 78.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x102f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1031:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1030:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x102f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1031:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1030:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1030:0x0] } PASS 204d (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:16:10 (1712276170) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0195808 s, 107 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:16:45 (1712276205) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool oleg420-server: Pool lustre.test_204f created Adding targets to pool oleg420-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg420-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0200947 s, 104 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg420-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg420-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg420-server: Pool lustre.test_204f destroyed PASS 204f (41s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer flag ========================================================== 20:17:28 (1712276248) PASS 205 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:17:31 (1712276251) oleg420-server: Pool lustre.test_206 created Adding targets to pool oleg420-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg420-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg420-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg420-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg420-server: Pool lustre.test_206 destroyed PASS 206 (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:17:44 (1712276264) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0194634 s, 53.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00936626 s, 112 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x3b799224 Mirror 3: 0x3b799224 PASS 207 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:17:48 (1712276268) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:17:50 (1712276270) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:17:52 (1712276272) Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00663437 s, 0.6 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x14a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1f10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1f10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1f10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2430000 Read this (4 bytes): BBBB 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:18:07 (1712276287) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00239449 s, 3.4 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 20:18:10 (1712276290) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107871 s, 97.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb9:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-04 20:18:11.000000000 -0400 Modify: 2024-04-04 20:18:11.000000000 -0400 Change: 2024-04-04 20:18:11.000000000 -0400 Birth: - PASS 210a (122s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 871 sec ============ 20:20:13 (1712276413) === sanity-flr: start cleanup 20:20:14 (1712276414) === === sanity-flr: finish cleanup 20:20:15 (1712276415) ===