== conf-sanity test 69: replace an OST with the same index ========================================================== 17:56:19 (1788213379) start mds service on oleg332-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1755536 Aug 31 10:08 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: kmemleak disabled oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: MODOPTS_LIBCFS= oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg332-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg332-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg332-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg332-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 695 (time 1788213489.26 total 10.00 last 69.50) - open/close 1374 (time 1788213499.27 total 20.01 last 67.81) - open/close 2101 (time 1788213509.29 total 30.03 last 72.61) - open/close 2797 (time 1788213519.30 total 40.04 last 69.51) - open/close 3424 (time 1788213529.30 total 50.04 last 62.68) - open/close 4034 (time 1788213539.32 total 60.06 last 60.89) - open/close 4737 (time 1788213549.33 total 70.07 last 70.25) - open/close 5397 (time 1788213559.33 total 80.07 last 65.96) - open/close 6069 (time 1788213569.36 total 90.10 last 67.01) - open/close 6815 (time 1788213579.37 total 100.11 last 74.55) - open/close 7505 (time 1788213589.37 total 110.11 last 68.98) - open/close 8202 (time 1788213599.38 total 120.12 last 69.66) - open/close 8903 (time 1788213609.39 total 130.13 last 70.04) - open/close 9597 (time 1788213619.39 total 140.13 last 69.38) total: 10000 open/close in 145.75 seconds: 68.61 ops/second - unlinked 0 (time 1788213628 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 563 (time 1788213736.81 total 10.00 last 56.30) - open/close 1291 (time 1788213746.82 total 20.01 last 72.73) - open/close 1933 (time 1788213756.83 total 30.02 last 64.11) - open/close 2535 (time 1788213766.83 total 40.03 last 60.18) - open/close 3184 (time 1788213776.84 total 50.03 last 64.84) - open/close 3798 (time 1788213786.86 total 60.05 last 61.27) - open/close 4389 (time 1788213796.88 total 70.08 last 58.97) - open/close 4909 (time 1788213806.90 total 80.09 last 51.92) - open/close 5451 (time 1788213816.90 total 90.09 last 54.20) - open/close 6163 (time 1788213826.90 total 100.09 last 71.20) - open/close 6877 (time 1788213836.91 total 110.10 last 71.35) - open/close 7552 (time 1788213846.91 total 120.11 last 67.45) - open/close 8201 (time 1788213856.92 total 130.11 last 64.88) - open/close 8916 (time 1788213866.92 total 140.11 last 71.50) - open/close 9556 (time 1788213876.93 total 150.12 last 63.92) total: 10000 open/close in 157.88 seconds: 63.34 ops/second - unlinked 0 (time 1788213888 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 713 (time 1788213993.39 total 10.00 last 71.26) - open/close 1406 (time 1788214003.40 total 20.01 last 69.26) - open/close 1980 (time 1788214013.41 total 30.03 last 57.30) - open/close 2546 (time 1788214023.43 total 40.04 last 56.53) - open/close 3217 (time 1788214033.43 total 50.05 last 67.07) - open/close 3953 (time 1788214043.46 total 60.07 last 73.41) - open/close 4635 (time 1788214053.47 total 70.08 last 68.13) - open/close 5319 (time 1788214063.48 total 80.09 last 68.34) - open/close 6045 (time 1788214073.49 total 90.10 last 72.52) - open/close 6732 (time 1788214083.50 total 100.11 last 68.62) - open/close 7358 (time 1788214093.51 total 110.12 last 62.55) - open/close 8112 (time 1788214103.51 total 120.13 last 75.37) - open/close 8737 (time 1788214113.52 total 130.13 last 62.46) - open/close 9414 (time 1788214123.52 total 140.13 last 67.68) total: 10000 open/close in 148.94 seconds: 67.14 ops/second - unlinked 0 (time 1788214136 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 608 (time 1788214240.49 total 10.01 last 60.75) - open/close 1310 (time 1788214250.49 total 20.01 last 70.17) - open/close 2045 (time 1788214260.49 total 30.02 last 73.47) - open/close 2701 (time 1788214270.51 total 40.03 last 65.48) - open/close 3349 (time 1788214280.52 total 50.04 last 64.75) - open/close 4014 (time 1788214290.55 total 60.07 last 66.33) - open/close 4641 (time 1788214300.55 total 70.07 last 62.68) - open/close 5332 (time 1788214310.55 total 80.08 last 69.06) - open/close 6036 (time 1788214320.57 total 90.10 last 70.26) - open/close 6717 (time 1788214330.57 total 100.10 last 68.10) - open/close 7374 (time 1788214340.59 total 110.11 last 65.59) - open/close 8108 (time 1788214350.59 total 120.11 last 73.39) - open/close 8769 (time 1788214360.59 total 130.12 last 66.09) - open/close 9380 (time 1788214370.61 total 140.13 last 61.03) total: 10000 open/close in 149.59 seconds: 66.85 ops/second - unlinked 0 (time 1788214384 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 591 (time 1788214493.89 total 10.01 last 59.06) - open/close 1333 (time 1788214503.91 total 20.03 last 74.05) - open/close 1929 (time 1788214513.92 total 30.03 last 59.55) - open/close 2509 (time 1788214523.93 total 40.04 last 57.97) - open/close 2890 (time 1788214533.93 total 50.04 last 38.09) - open/close 3303 (time 1788214543.94 total 60.05 last 41.25) - open/close 3670 (time 1788214553.96 total 70.07 last 36.64) - open/close 4168 (time 1788214563.96 total 80.08 last 49.77) - open/close 4826 (time 1788214573.96 total 90.08 last 65.80) - open/close 5475 (time 1788214583.97 total 100.08 last 64.88) - open/close 6148 (time 1788214593.98 total 110.09 last 67.23) - open/close 6785 (time 1788214603.99 total 120.10 last 63.61) - open/close 7462 (time 1788214614.00 total 130.11 last 67.64) - open/close 8084 (time 1788214624.00 total 140.11 last 62.20) - open/close 8750 (time 1788214634.04 total 150.15 last 66.36) - open/close 9479 (time 1788214644.04 total 160.15 last 72.89) total: 10000 open/close in 167.07 seconds: 59.86 ops/second - unlinked 0 (time 1788214654 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 630 (time 1788214755.92 total 10.01 last 62.95) - open/close 1317 (time 1788214765.93 total 20.02 last 68.63) - open/close 1973 (time 1788214775.94 total 30.02 last 65.56) - open/close 2643 (time 1788214786.02 total 40.10 last 66.48) - open/close 3290 (time 1788214796.02 total 50.11 last 64.68) - open/close 3951 (time 1788214806.03 total 60.12 last 66.03) - open/close 4645 (time 1788214816.04 total 70.12 last 69.36) - open/close 5291 (time 1788214826.05 total 80.14 last 64.49) - open/close 5911 (time 1788214836.06 total 90.14 last 61.97) - open/close 6565 (time 1788214846.07 total 100.16 last 65.29) - open/close 7140 (time 1788214856.11 total 110.19 last 57.30) - open/close 7812 (time 1788214866.14 total 120.22 last 67.01) - open/close 8484 (time 1788214876.15 total 130.23 last 67.13) - open/close 9161 (time 1788214886.15 total 140.24 last 67.67) - open/close 9802 (time 1788214896.16 total 150.25 last 64.03) total: 10000 open/close in 153.42 seconds: 65.18 ops/second - unlinked 0 (time 1788214904 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 688 (time 1788215012.59 total 10.02 last 68.65) - open/close 1400 (time 1788215022.59 total 20.03 last 71.18) - open/close 2072 (time 1788215032.59 total 30.03 last 67.20) - open/close 2728 (time 1788215042.60 total 40.03 last 65.55) - open/close 3460 (time 1788215052.60 total 50.04 last 73.17) - open/close 4101 (time 1788215062.61 total 60.04 last 64.05) - open/close 4850 (time 1788215072.62 total 70.05 last 74.83) - open/close 5577 (time 1788215082.63 total 80.06 last 72.66) - open/close 6305 (time 1788215092.63 total 90.06 last 72.80) - open/close 7003 (time 1788215102.64 total 100.07 last 69.73) - open/close 7617 (time 1788215112.65 total 110.08 last 61.32) - open/close 8262 (time 1788215122.65 total 120.09 last 64.49) - open/close 8872 (time 1788215132.67 total 130.10 last 60.89) - open/close 9387 (time 1788215142.68 total 140.11 last 51.47) - open/close 9838 (time 1788215152.68 total 150.11 last 45.10) total: 10000 open/close in 152.90 seconds: 65.40 ops/second - unlinked 0 (time 1788215161 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 712 (time 1788215265.30 total 10.01 last 71.13) - open/close 1400 (time 1788215275.30 total 20.02 last 68.74) - open/close 2077 (time 1788215285.31 total 30.02 last 67.69) - open/close 2687 (time 1788215295.31 total 40.03 last 60.97) - open/close 3410 (time 1788215305.32 total 50.03 last 72.25) - open/close 4055 (time 1788215315.32 total 60.04 last 64.45) - open/close 4691 (time 1788215325.33 total 70.04 last 63.57) - open/close 5463 (time 1788215335.33 total 80.05 last 77.18) - open/close 6131 (time 1788215345.33 total 90.05 last 66.80) - open/close 6828 (time 1788215355.37 total 100.09 last 69.42) - open/close 7422 (time 1788215365.38 total 110.09 last 59.36) - open/close 8102 (time 1788215375.38 total 120.09 last 68.00) - open/close 8755 (time 1788215385.38 total 130.09 last 65.30) - open/close 9478 (time 1788215395.40 total 140.11 last 72.17) total: 10000 open/close in 147.60 seconds: 67.75 ops/second - unlinked 0 (time 1788215406 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 477 (time 1788215506.53 total 10.03 last 47.57) - open/close 929 (time 1788215516.54 total 20.03 last 45.18) - open/close 1517 (time 1788215526.55 total 30.04 last 58.75) - open/close 2157 (time 1788215536.56 total 40.05 last 63.94) - open/close 2772 (time 1788215546.56 total 50.06 last 61.45) - open/close 3408 (time 1788215556.57 total 60.06 last 63.58) - open/close 4118 (time 1788215566.57 total 70.07 last 70.96) - open/close 4811 (time 1788215576.58 total 80.07 last 69.27) - open/close 5499 (time 1788215586.59 total 90.08 last 68.74) - open/close 6183 (time 1788215596.59 total 100.09 last 68.36) - open/close 6907 (time 1788215606.61 total 110.10 last 72.30) - open/close 7652 (time 1788215616.61 total 120.11 last 74.44) - open/close 8369 (time 1788215626.62 total 130.12 last 71.64) - open/close 9107 (time 1788215636.63 total 140.12 last 73.78) - open/close 9837 (time 1788215646.63 total 150.13 last 72.94) total: 10000 open/close in 152.29 seconds: 65.66 ops/second - unlinked 0 (time 1788215652 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 686 (time 1788215757.48 total 10.00 last 68.59) - open/close 1458 (time 1788215767.49 total 20.01 last 77.16) - open/close 2185 (time 1788215777.50 total 30.02 last 72.60) - open/close 2912 (time 1788215787.50 total 40.02 last 72.68) - open/close 3638 (time 1788215797.52 total 50.04 last 72.52) - open/close 4281 (time 1788215807.55 total 60.07 last 64.09) - open/close 4989 (time 1788215817.55 total 70.07 last 70.76) - open/close 5652 (time 1788215827.56 total 80.08 last 66.25) - open/close 6223 (time 1788215837.57 total 90.09 last 57.05) - open/close 6773 (time 1788215847.59 total 100.11 last 54.92) - open/close 7425 (time 1788215857.59 total 110.11 last 65.18) - open/close 8114 (time 1788215867.61 total 120.13 last 68.77) - open/close 8736 (time 1788215877.62 total 130.14 last 62.14) - open/close 9391 (time 1788215887.63 total 140.15 last 65.45) total: 10000 open/close in 148.45 seconds: 67.36 ops/second - unlinked 0 (time 1788215899 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.132@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.132@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg332-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg332-server: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg332-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg332-server: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg332-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre On OST0, 20520 used inodes rc=0 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.132@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.132@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.132@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.132@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.203.132@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg332-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg332-server: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg332-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg332-server: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg332-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg332-server unloading modules via unload_modules_local on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local oleg332-server: kmemleak disabled modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2