== conf-sanity test 69: replace an OST with the same index ========================================================== 16:02:14 (1710100934) start mds service on oleg141-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg141-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg141-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock oleg141-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11840 inodes available. Want 99936. rc=0 - open/close 1237 (time 1710100979.22 total 10.00 last 123.65) - open/close 2202 (time 1710100989.24 total 20.02 last 96.31) - open/close 3153 (time 1710100999.24 total 30.03 last 95.08) - open/close 4016 (time 1710101009.25 total 40.03 last 86.24) - open/close 4965 (time 1710101019.26 total 50.04 last 94.83) - open/close 5764 (time 1710101029.28 total 60.06 last 79.71) - open/close 6565 (time 1710101039.29 total 70.07 last 80.05) - open/close 7352 (time 1710101049.30 total 80.08 last 78.64) - open/close 8091 (time 1710101059.31 total 90.09 last 73.81) - open/close 8190 (time 1710101071.52 total 102.30 last 8.11) - open/close 8888 (time 1710101081.54 total 112.32 last 69.69) - open/close 9564 (time 1710101091.56 total 122.35 last 67.41) total: 10000 open/close in 127.95 seconds: 78.16 ops/second - unlinked 0 (time 1710101098 ; total 0 ; last 0) total: 10000 unlinks in 121 seconds: 82.644630 unlinks/second - open/close 926 (time 1710101231.47 total 10.00 last 92.59) - open/close 1796 (time 1710101241.50 total 20.03 last 86.75) - open/close 2572 (time 1710101251.51 total 30.04 last 77.53) - open/close 3348 (time 1710101261.51 total 40.04 last 77.60) - open/close 4030 (time 1710101271.51 total 50.04 last 68.19) - open/close 4544 (time 1710101290.09 total 68.62 last 27.66) - open/close 5258 (time 1710101300.12 total 78.66 last 71.16) - open/close 5887 (time 1710101310.13 total 88.66 last 62.84) - open/close 6480 (time 1710101320.15 total 98.68 last 59.23) - open/close 7072 (time 1710101330.15 total 108.68 last 59.18) - open/close 7652 (time 1710101340.16 total 118.69 last 57.96) - open/close 8185 (time 1710101350.16 total 128.69 last 53.29) - open/close 8712 (time 1710101360.16 total 138.69 last 52.67) - open/close 9239 (time 1710101370.19 total 148.72 last 52.55) - open/close 9767 (time 1710101380.19 total 158.73 last 52.78) total: 10000 open/close in 162.57 seconds: 61.51 ops/second - unlinked 0 (time 1710101386 ; total 0 ; last 0) total: 10000 unlinks in 123 seconds: 81.300812 unlinks/second - open/close 999 (time 1710101522.17 total 10.01 last 99.82) - open/close 1968 (time 1710101532.17 total 20.01 last 96.85) - open/close 2755 (time 1710101542.19 total 30.03 last 78.59) - open/close 3571 (time 1710101552.19 total 40.03 last 81.55) - open/close 4283 (time 1710101562.20 total 50.04 last 71.13) - open/close 5060 (time 1710101572.20 total 60.04 last 77.69) - open/close 5684 (time 1710101582.21 total 70.06 last 62.33) - open/close 6258 (time 1710101592.22 total 80.06 last 57.35) - open/close 6864 (time 1710101602.23 total 90.07 last 60.55) - open/close 7440 (time 1710101612.24 total 100.08 last 57.56) - open/close 7988 (time 1710101622.24 total 110.09 last 54.76) - open/close 8524 (time 1710101632.25 total 120.09 last 53.56) - open/close 9108 (time 1710101642.26 total 130.10 last 58.35) - open/close 9624 (time 1710101652.28 total 140.12 last 51.52) total: 10000 open/close in 147.44 seconds: 67.82 ops/second - unlinked 0 (time 1710101662 ; total 0 ; last 0) total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second - open/close 1022 (time 1710101781.58 total 10.00 last 102.18) - open/close 1943 (time 1710101791.62 total 20.04 last 91.74) - open/close 2405 (time 1710101807.98 total 36.41 last 28.23) - open/close 3162 (time 1710101817.99 total 46.41 last 75.63) - open/close 3842 (time 1710101827.99 total 56.42 last 67.99) - open/close 4551 (time 1710101837.99 total 66.42 last 70.89) - open/close 5257 (time 1710101848.01 total 76.43 last 70.52) - open/close 5868 (time 1710101868.48 total 96.90 last 29.84) - open/close 6463 (time 1710101878.49 total 106.92 last 59.42) - open/close 6670 (time 1710101893.03 total 121.45 last 14.24) - open/close 7324 (time 1710101903.03 total 131.46 last 65.38) - open/close 7900 (time 1710101913.06 total 141.48 last 57.45) - open/close 8447 (time 1710101923.07 total 151.49 last 54.63) - open/close 8961 (time 1710101933.08 total 161.50 last 51.36) - open/close 9507 (time 1710101943.08 total 171.50 last 54.59) total: 10000 open/close in 180.97 seconds: 55.26 ops/second - unlinked 0 (time 1710101954 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 947 (time 1710102073.65 total 10.01 last 94.62) - open/close 1779 (time 1710102083.66 total 20.01 last 83.16) - open/close 2651 (time 1710102093.67 total 30.03 last 87.06) - open/close 3382 (time 1710102103.68 total 40.04 last 73.03) - open/close 4076 (time 1710102113.69 total 50.05 last 69.35) - open/close 4765 (time 1710102123.69 total 60.05 last 68.89) - open/close 5490 (time 1710102133.70 total 70.06 last 72.42) - open/close 6122 (time 1710102143.72 total 80.07 last 63.10) - open/close 6738 (time 1710102153.72 total 90.07 last 61.60) - open/close 7278 (time 1710102163.76 total 100.12 last 53.77) - open/close 7791 (time 1710102173.76 total 110.12 last 51.28) - open/close 8353 (time 1710102183.78 total 120.14 last 56.11) - open/close 8970 (time 1710102193.78 total 130.14 last 61.68) - open/close 9559 (time 1710102203.79 total 140.15 last 58.86) total: 10000 open/close in 148.77 seconds: 67.22 ops/second - unlinked 0 (time 1710102215 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 916 (time 1710102323.38 total 10.00 last 91.60) - open/close 1792 (time 1710102333.39 total 20.01 last 87.54) - open/close 2581 (time 1710102343.40 total 30.02 last 78.78) - open/close 3412 (time 1710102353.41 total 40.03 last 83.04) - open/close 4173 (time 1710102363.42 total 50.04 last 76.05) - open/close 4970 (time 1710102373.43 total 60.05 last 79.61) - open/close 5647 (time 1710102383.43 total 70.05 last 67.66) - open/close 6258 (time 1710102393.45 total 80.07 last 61.02) - open/close 6841 (time 1710102403.46 total 90.08 last 58.25) - open/close 7408 (time 1710102413.46 total 100.08 last 56.66) - open/close 8057 (time 1710102423.48 total 110.10 last 64.79) - open/close 8654 (time 1710102444.02 total 130.64 last 29.07) - open/close 9187 (time 1710102454.02 total 140.64 last 53.29) - open/close 9728 (time 1710102464.04 total 150.66 last 54.00) total: 10000 open/close in 155.63 seconds: 64.25 ops/second - unlinked 0 (time 1710102471 ; total 0 ; last 0) total: 10000 unlinks in 110 seconds: 90.909088 unlinks/second - open/close 949 (time 1710102593.34 total 10.01 last 94.84) - open/close 1816 (time 1710102603.35 total 20.02 last 86.62) - open/close 2644 (time 1710102613.35 total 30.02 last 82.73) - open/close 3348 (time 1710102623.36 total 40.03 last 70.39) - open/close 4065 (time 1710102633.36 total 50.03 last 71.66) - open/close 4813 (time 1710102643.37 total 60.04 last 74.73) - open/close 5504 (time 1710102653.37 total 70.04 last 69.08) - open/close 6137 (time 1710102663.40 total 80.06 last 63.16) - open/close 6743 (time 1710102673.40 total 90.07 last 60.56) - open/close 7318 (time 1710102683.41 total 100.08 last 57.45) - open/close 7855 (time 1710102693.42 total 110.09 last 53.66) - open/close 8368 (time 1710102703.43 total 120.09 last 51.26) - open/close 8937 (time 1710102713.44 total 130.11 last 56.80) - open/close 9445 (time 1710102723.48 total 140.15 last 50.61) - open/close 9917 (time 1710102733.48 total 150.15 last 47.18) total: 10000 open/close in 151.91 seconds: 65.83 ops/second - unlinked 0 (time 1710102737 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 876 (time 1710102854.53 total 10.00 last 87.59) - open/close 1785 (time 1710102864.54 total 20.01 last 90.85) - open/close 2559 (time 1710102874.55 total 30.02 last 77.27) - open/close 3269 (time 1710102884.56 total 40.03 last 70.95) - open/close 3309 (time 1710102896.28 total 51.75 last 3.41) - open/close 3335 (time 1710102907.64 total 63.10 last 2.29) - open/close 4094 (time 1710102917.64 total 73.11 last 75.85) - open/close 4344 (time 1710102932.12 total 87.59 last 17.27) - open/close 5066 (time 1710102942.12 total 97.59 last 72.19) - open/close 5767 (time 1710102952.13 total 107.60 last 70.02) - open/close 6412 (time 1710102962.15 total 117.62 last 64.40) - open/close 7003 (time 1710102972.15 total 127.62 last 59.10) - open/close 7581 (time 1710102982.16 total 137.63 last 57.73) - open/close 8145 (time 1710102992.19 total 147.66 last 56.20) - open/close 8655 (time 1710103002.22 total 157.68 last 50.90) - open/close 9224 (time 1710103012.23 total 167.69 last 56.84) - open/close 9742 (time 1710103022.26 total 177.73 last 51.61) total: 10000 open/close in 182.54 seconds: 54.78 ops/second - unlinked 0 (time 1710103029 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 1009 (time 1710103135.38 total 10.00 last 100.87) - open/close 1844 (time 1710103145.40 total 20.02 last 83.34) - open/close 2725 (time 1710103155.43 total 30.04 last 87.89) - open/close 3529 (time 1710103165.48 total 40.10 last 79.99) - open/close 4345 (time 1710103175.48 total 50.10 last 81.58) - open/close 5221 (time 1710103185.53 total 60.15 last 87.17) - open/close 5981 (time 1710103195.53 total 70.15 last 75.98) - open/close 6715 (time 1710103205.54 total 80.16 last 73.34) - open/close 7385 (time 1710103215.54 total 90.16 last 67.00) - open/close 8085 (time 1710103225.56 total 100.18 last 69.89) - open/close 8661 (time 1710103235.61 total 110.23 last 57.28) - open/close 9254 (time 1710103245.61 total 120.23 last 59.29) - open/close 9851 (time 1710103255.62 total 130.24 last 59.64) total: 10000 open/close in 133.16 seconds: 75.10 ops/second - unlinked 0 (time 1710103260 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 1209 (time 1710103370.63 total 10.01 last 120.83) - open/close 2299 (time 1710103380.63 total 20.01 last 108.94) - open/close 3301 (time 1710103390.64 total 30.02 last 100.13) - open/close 4338 (time 1710103400.65 total 40.03 last 103.61) - open/close 5144 (time 1710103410.65 total 50.03 last 80.55) - open/close 6071 (time 1710103420.66 total 60.04 last 92.61) - open/close 6903 (time 1710103430.67 total 70.05 last 83.16) - open/close 7759 (time 1710103440.67 total 80.05 last 85.58) - open/close 8582 (time 1710103450.67 total 90.05 last 82.29) - open/close 9365 (time 1710103460.68 total 100.06 last 78.23) total: 10000 open/close in 107.64 seconds: 92.90 ops/second - unlinked 0 (time 1710103469 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.141@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.141@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg141-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg141-server: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg141-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg141-server: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg141-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock oleg141-server@tcp:/lustre /mnt/lustre On OST0, 9463 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg141-server unloading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2