== conf-sanity test 69: replace an OST with the same index ========================================================== 16:05:46 (1710101146) start mds service on oleg429-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg429-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg429-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg429-client.virtnet: -o user_xattr,flock oleg429-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11840 inodes available. Want 99936. rc=0 - open/close 841 (time 1710101195.00 total 10.00 last 84.09) - open/close 1861 (time 1710101205.00 total 20.00 last 101.98) - open/close 2755 (time 1710101215.00 total 30.00 last 89.39) - open/close 3507 (time 1710101225.02 total 40.02 last 75.09) - open/close 4246 (time 1710101235.02 total 50.02 last 73.89) - open/close 4928 (time 1710101245.02 total 60.03 last 68.16) - open/close 5630 (time 1710101255.03 total 70.04 last 70.12) - open/close 6255 (time 1710101265.04 total 80.04 last 62.45) - open/close 6903 (time 1710101275.05 total 90.05 last 64.75) - open/close 7520 (time 1710101285.05 total 100.06 last 61.66) - open/close 8129 (time 1710101295.06 total 110.06 last 60.88) - open/close 8710 (time 1710101305.06 total 120.07 last 58.07) - open/close 9320 (time 1710101315.07 total 130.07 last 60.97) - open/close 9840 (time 1710101325.08 total 140.08 last 51.94) total: 10000 open/close in 142.98 seconds: 69.94 ops/second - unlinked 0 (time 1710101329 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 1037 (time 1710101433.38 total 10.00 last 103.67) - open/close 2014 (time 1710101443.39 total 20.01 last 97.65) - open/close 2905 (time 1710101453.40 total 30.02 last 89.03) - open/close 3715 (time 1710101463.40 total 40.02 last 80.98) - open/close 4459 (time 1710101483.73 total 60.35 last 36.59) - open/close 5203 (time 1710101493.74 total 70.36 last 74.37) - open/close 5919 (time 1710101503.75 total 80.37 last 71.54) - open/close 6589 (time 1710101513.75 total 90.37 last 66.98) - open/close 7235 (time 1710101523.77 total 100.39 last 64.45) - open/close 7883 (time 1710101533.78 total 110.40 last 64.73) - open/close 8438 (time 1710101543.81 total 120.43 last 55.35) - open/close 9023 (time 1710101553.82 total 130.43 last 58.46) - open/close 9609 (time 1710101563.84 total 140.46 last 58.46) total: 10000 open/close in 147.38 seconds: 67.85 ops/second - unlinked 0 (time 1710101572 ; total 0 ; last 0) total: 10000 unlinks in 112 seconds: 89.285713 unlinks/second - open/close 1042 (time 1710101696.56 total 10.00 last 104.16) - open/close 1945 (time 1710101706.57 total 20.01 last 90.25) - open/close 2892 (time 1710101716.57 total 30.02 last 94.64) - open/close 3670 (time 1710101726.58 total 40.02 last 77.76) - open/close 4385 (time 1710101736.59 total 50.03 last 71.46) - open/close 5173 (time 1710101746.59 total 60.03 last 78.79) - open/close 5882 (time 1710101756.60 total 70.04 last 70.82) - open/close 6565 (time 1710101766.61 total 80.05 last 68.25) - open/close 7271 (time 1710101776.61 total 90.05 last 70.60) - open/close 7910 (time 1710101786.61 total 100.05 last 63.89) - open/close 8506 (time 1710101796.62 total 110.06 last 59.53) - open/close 9053 (time 1710101806.62 total 120.06 last 54.69) - open/close 9642 (time 1710101816.63 total 130.07 last 58.83) total: 10000 open/close in 136.06 seconds: 73.50 ops/second - unlinked 0 (time 1710101824 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 1034 (time 1710101929.99 total 10.01 last 103.29) - open/close 1947 (time 1710101940.00 total 20.02 last 91.24) - open/close 2801 (time 1710101950.00 total 30.03 last 85.33) - open/close 3619 (time 1710101960.00 total 40.03 last 81.80) - open/close 4454 (time 1710101970.01 total 50.03 last 83.45) - open/close 5219 (time 1710101980.02 total 60.04 last 76.45) - open/close 5949 (time 1710101990.04 total 70.06 last 72.85) - open/close 6688 (time 1710102000.05 total 80.07 last 73.81) - open/close 7395 (time 1710102010.06 total 90.08 last 70.62) - open/close 8019 (time 1710102020.06 total 100.09 last 62.38) - open/close 8688 (time 1710102030.08 total 110.10 last 66.78) - open/close 9346 (time 1710102040.09 total 120.11 last 65.76) - open/close 9926 (time 1710102050.10 total 130.12 last 57.93) total: 10000 open/close in 131.54 seconds: 76.02 ops/second - unlinked 0 (time 1710102053 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 1062 (time 1710102155.53 total 10.00 last 106.19) - open/close 2059 (time 1710102165.54 total 20.00 last 99.67) - open/close 3076 (time 1710102175.54 total 30.01 last 101.61) - open/close 3894 (time 1710102185.55 total 40.02 last 81.75) - open/close 4702 (time 1710102195.56 total 50.03 last 80.75) - open/close 5430 (time 1710102205.60 total 60.07 last 72.48) - open/close 6183 (time 1710102215.61 total 70.07 last 75.26) - open/close 6895 (time 1710102225.61 total 80.08 last 71.16) - open/close 7550 (time 1710102235.62 total 90.09 last 65.47) - open/close 8202 (time 1710102245.62 total 100.09 last 65.20) - open/close 8882 (time 1710102255.63 total 110.10 last 67.90) - open/close 9517 (time 1710102265.65 total 120.12 last 63.40) total: 10000 open/close in 128.79 seconds: 77.65 ops/second - unlinked 0 (time 1710102276 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 1058 (time 1710102377.33 total 10.00 last 105.77) - open/close 2098 (time 1710102387.33 total 20.01 last 103.94) - open/close 2274 (time 1710102400.55 total 33.23 last 13.31) - open/close 3056 (time 1710102410.55 total 43.23 last 78.17) - open/close 3889 (time 1710102420.56 total 53.24 last 83.24) - open/close 4660 (time 1710102430.57 total 63.25 last 77.03) - open/close 5419 (time 1710102440.57 total 73.25 last 75.87) - open/close 6147 (time 1710102450.59 total 83.26 last 72.71) - open/close 6849 (time 1710102460.59 total 93.27 last 70.16) - open/close 7578 (time 1710102470.63 total 103.31 last 72.61) - open/close 8264 (time 1710102480.64 total 113.32 last 68.54) - open/close 8925 (time 1710102490.65 total 123.33 last 66.01) - open/close 9508 (time 1710102500.66 total 133.34 last 58.28) total: 10000 open/close in 141.26 seconds: 70.79 ops/second - unlinked 0 (time 1710102510 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 1072 (time 1710102619.18 total 10.01 last 107.08) - open/close 2071 (time 1710102629.20 total 20.02 last 99.78) - open/close 3032 (time 1710102639.21 total 30.04 last 95.99) - open/close 3937 (time 1710102649.21 total 40.04 last 90.46) - open/close 4797 (time 1710102659.22 total 50.05 last 85.91) - open/close 5580 (time 1710102669.23 total 60.06 last 78.25) - open/close 6396 (time 1710102679.23 total 70.06 last 81.54) - open/close 7130 (time 1710102689.24 total 80.07 last 73.36) - open/close 7880 (time 1710102699.25 total 90.07 last 74.95) - open/close 8613 (time 1710102709.25 total 100.08 last 73.29) - open/close 9262 (time 1710102719.26 total 110.09 last 64.79) - open/close 9920 (time 1710102729.27 total 120.10 last 65.77) total: 10000 open/close in 121.21 seconds: 82.50 ops/second - unlinked 0 (time 1710102732 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 1291 (time 1710102819.84 total 10.01 last 129.00) - open/close 2453 (time 1710102829.85 total 20.02 last 116.11) - open/close 3549 (time 1710102839.85 total 30.02 last 109.52) - open/close 4535 (time 1710102849.85 total 40.02 last 98.59) - open/close 5413 (time 1710102859.86 total 50.03 last 87.78) - open/close 6293 (time 1710102869.86 total 60.03 last 87.95) - open/close 7221 (time 1710102879.87 total 70.04 last 92.76) - open/close 8068 (time 1710102889.87 total 80.04 last 84.67) - open/close 9009 (time 1710102899.87 total 90.04 last 94.07) - open/close 9948 (time 1710102909.87 total 100.04 last 93.89) total: 10000 open/close in 100.60 seconds: 99.41 ops/second - unlinked 0 (time 1710102911 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1877 (time 1710102978.44 total 10.00 last 187.68) - open/close 3426 (time 1710102988.45 total 20.01 last 154.79) - open/close 4941 (time 1710102998.46 total 30.01 last 151.41) - open/close 6280 (time 1710103008.46 total 40.02 last 133.88) - open/close 7508 (time 1710103018.46 total 50.02 last 122.72) - open/close 8772 (time 1710103028.47 total 60.03 last 126.35) - open/close 9980 (time 1710103038.47 total 70.03 last 120.73) total: 10000 open/close in 70.20 seconds: 142.45 ops/second - unlinked 0 (time 1710103039 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 3140 (time 1710103090.59 total 10.00 last 313.98) - open/close 5713 (time 1710103100.59 total 20.00 last 257.25) - open/close 7695 (time 1710103110.60 total 30.00 last 198.19) - open/close 9530 (time 1710103120.60 total 40.00 last 183.48) total: 10000 open/close in 42.63 seconds: 234.56 ops/second - unlinked 0 (time 1710103124 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg429-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.129@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.129@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg429-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg429-server: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg429-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec oleg429-server: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg429-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg429-client.virtnet: -o user_xattr,flock oleg429-server@tcp:/lustre /mnt/lustre On OST0, 20455 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg429-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg429-server unloading modules on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2