== conf-sanity test 69: replace an OST with the same index ========================================================== 15:58:04 (1710100684) start mds service on oleg119-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg119-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 3001 (time 1710100710.43 total 10.00 last 300.04) - open/close 5540 (time 1710100720.43 total 20.00 last 253.87) - open/close 7942 (time 1710100730.43 total 30.00 last 240.18) - open/close 9790 (time 1710100740.44 total 40.01 last 184.72) total: 10000 open/close in 41.05 seconds: 243.62 ops/second - unlinked 0 (time 1710100742 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 3255 (time 1710100795.50 total 10.00 last 325.37) - open/close 5506 (time 1710100805.51 total 20.01 last 224.98) - open/close 6759 (time 1710100815.51 total 30.01 last 125.30) - open/close 7777 (time 1710100825.52 total 40.02 last 101.74) - open/close 8706 (time 1710100835.53 total 50.03 last 92.80) - open/close 9571 (time 1710100845.53 total 60.03 last 86.43) total: 10000 open/close in 65.62 seconds: 152.40 ops/second - unlinked 0 (time 1710100852 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 1237 (time 1710100937.43 total 10.00 last 123.68) - open/close 2394 (time 1710100947.43 total 20.00 last 115.68) - open/close 3539 (time 1710100957.43 total 30.00 last 114.48) - open/close 4433 (time 1710100967.44 total 40.01 last 89.37) - open/close 5357 (time 1710100977.44 total 50.01 last 92.35) - open/close 6114 (time 1710100987.46 total 60.03 last 75.59) - open/close 6820 (time 1710100997.46 total 70.04 last 70.55) - open/close 7518 (time 1710101007.47 total 80.04 last 69.76) - open/close 8330 (time 1710101017.48 total 90.05 last 81.14) - open/close 9010 (time 1710101027.49 total 100.06 last 67.94) - open/close 9660 (time 1710101037.50 total 110.07 last 64.94) total: 10000 open/close in 114.64 seconds: 87.23 ops/second - unlinked 0 (time 1710101043 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 959 (time 1710101129.73 total 10.01 last 95.84) - open/close 1898 (time 1710101139.74 total 20.01 last 93.85) - open/close 2801 (time 1710101149.74 total 30.02 last 90.23) - open/close 3713 (time 1710101159.75 total 40.02 last 91.16) - open/close 4484 (time 1710101169.76 total 50.03 last 77.01) - open/close 5136 (time 1710101179.77 total 60.05 last 65.11) - open/close 5855 (time 1710101189.78 total 70.05 last 71.87) - open/close 6511 (time 1710101199.78 total 80.06 last 65.57) - open/close 7076 (time 1710101209.79 total 90.07 last 56.45) - open/close 7636 (time 1710101219.80 total 100.07 last 55.97) - open/close 8178 (time 1710101229.81 total 110.08 last 54.14) - open/close 8424 (time 1710101245.14 total 125.41 last 16.05) - open/close 8785 (time 1710101255.15 total 135.42 last 36.05) - open/close 9283 (time 1710101265.17 total 145.44 last 49.73) - open/close 9773 (time 1710101275.18 total 155.45 last 48.95) - open/close 9967 (time 1710101290.43 total 170.70 last 12.72) total: 10000 open/close in 171.34 seconds: 58.36 ops/second - unlinked 0 (time 1710101293 ; total 0 ; last 0) total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second - open/close 971 (time 1710101417.01 total 10.00 last 97.10) - open/close 1834 (time 1710101427.02 total 20.00 last 86.29) - open/close 2674 (time 1710101437.02 total 30.01 last 83.92) - open/close 3449 (time 1710101447.03 total 40.02 last 77.43) - open/close 4233 (time 1710101457.04 total 50.02 last 78.37) - open/close 4722 (time 1710101467.05 total 60.04 last 48.83) - open/close 5285 (time 1710101477.05 total 70.04 last 56.28) - open/close 5881 (time 1710101487.06 total 80.04 last 59.58) - open/close 6523 (time 1710101497.07 total 90.05 last 64.14) - open/close 7065 (time 1710101507.10 total 100.08 last 54.02) - open/close 7636 (time 1710101517.10 total 110.09 last 57.10) - open/close 8204 (time 1710101527.13 total 120.11 last 56.65) - open/close 8760 (time 1710101537.13 total 130.11 last 55.59) - open/close 9292 (time 1710101547.13 total 140.12 last 53.16) - open/close 9830 (time 1710101557.15 total 150.14 last 53.69) total: 10000 open/close in 153.81 seconds: 65.01 ops/second - unlinked 0 (time 1710101563 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 1001 (time 1710101675.23 total 10.01 last 100.05) - open/close 1954 (time 1710101685.24 total 20.01 last 95.25) - open/close 2786 (time 1710101695.25 total 30.02 last 83.13) - open/close 3209 (time 1710101711.63 total 46.40 last 25.82) - open/close 3667 (time 1710101721.65 total 56.42 last 45.72) - open/close 4335 (time 1710101731.70 total 66.47 last 66.47) - open/close 4953 (time 1710101741.70 total 76.47 last 61.79) - open/close 5609 (time 1710101751.70 total 86.47 last 65.59) - open/close 6229 (time 1710101761.77 total 96.54 last 61.57) - open/close 6779 (time 1710101771.77 total 106.54 last 54.98) - open/close 6952 (time 1710101785.72 total 120.49 last 12.40) - open/close 7504 (time 1710101795.77 total 130.54 last 54.96) - open/close 8045 (time 1710101805.78 total 140.55 last 54.05) - open/close 8520 (time 1710101815.79 total 150.56 last 47.45) - open/close 8989 (time 1710101825.79 total 160.56 last 46.89) - open/close 9433 (time 1710101835.80 total 170.58 last 44.34) - open/close 9464 (time 1710101847.60 total 182.37 last 2.63) - open/close 9981 (time 1710101857.61 total 192.38 last 51.63) total: 10000 open/close in 192.77 seconds: 51.87 ops/second - unlinked 0 (time 1710101860 ; total 0 ; last 0) total: 10000 unlinks in 119 seconds: 84.033615 unlinks/second - open/close 954 (time 1710101991.59 total 10.00 last 95.37) - open/close 1776 (time 1710102001.60 total 20.01 last 82.18) - open/close 2367 (time 1710102011.60 total 30.01 last 59.07) - open/close 3094 (time 1710102021.62 total 40.03 last 72.56) - open/close 3771 (time 1710102031.62 total 50.03 last 67.69) - open/close 4491 (time 1710102041.62 total 60.03 last 71.99) - open/close 5213 (time 1710102051.63 total 70.04 last 72.16) - open/close 5834 (time 1710102061.64 total 80.05 last 62.06) - open/close 6470 (time 1710102071.64 total 90.05 last 63.55) - open/close 7021 (time 1710102081.65 total 100.06 last 55.09) - open/close 7490 (time 1710102091.66 total 110.07 last 46.83) - open/close 8042 (time 1710102101.67 total 120.08 last 55.14) - open/close 8571 (time 1710102111.67 total 130.08 last 52.89) - open/close 9057 (time 1710102121.70 total 140.11 last 48.45) - open/close 9532 (time 1710102131.71 total 150.12 last 47.49) total: 10000 open/close in 159.52 seconds: 62.69 ops/second - unlinked 0 (time 1710102143 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 781 (time 1710102252.93 total 10.01 last 78.05) - open/close 1694 (time 1710102262.94 total 20.01 last 91.24) - open/close 2514 (time 1710102272.96 total 30.04 last 81.81) - open/close 3233 (time 1710102282.97 total 40.05 last 71.83) - open/close 3916 (time 1710102292.97 total 50.05 last 68.30) - open/close 4571 (time 1710102302.97 total 60.05 last 65.50) - open/close 5242 (time 1710102312.97 total 70.05 last 67.08) - open/close 5827 (time 1710102322.98 total 80.05 last 58.47) - open/close 6465 (time 1710102332.98 total 90.06 last 63.78) - open/close 7057 (time 1710102342.99 total 100.06 last 59.16) - open/close 7643 (time 1710102352.99 total 110.07 last 58.59) - open/close 8223 (time 1710102363.02 total 120.10 last 57.83) - open/close 8502 (time 1710102378.48 total 135.56 last 18.04) - open/close 8971 (time 1710102388.49 total 145.57 last 46.86) - open/close 9451 (time 1710102398.51 total 155.59 last 47.92) - open/close 9955 (time 1710102408.52 total 165.60 last 50.33) total: 10000 open/close in 166.94 seconds: 59.90 ops/second - unlinked 0 (time 1710102413 ; total 0 ; last 0) total: 10000 unlinks in 144 seconds: 69.444443 unlinks/second - open/close 1003 (time 1710102570.24 total 10.00 last 100.29) - open/close 1889 (time 1710102580.25 total 20.01 last 88.52) - open/close 2664 (time 1710102590.26 total 30.02 last 77.43) - open/close 3379 (time 1710102600.26 total 40.02 last 71.48) - open/close 4095 (time 1710102610.27 total 50.03 last 71.51) - open/close 4671 (time 1710102620.28 total 60.04 last 57.56) - open/close 5282 (time 1710102630.29 total 70.05 last 61.06) - open/close 5949 (time 1710102640.29 total 80.05 last 66.68) - open/close 6516 (time 1710102650.29 total 90.05 last 56.69) - open/close 7046 (time 1710102660.30 total 100.06 last 52.97) - open/close 7601 (time 1710102670.30 total 110.06 last 55.47) - open/close 8070 (time 1710102680.31 total 120.08 last 46.85) - open/close 8587 (time 1710102690.33 total 130.09 last 51.60) - open/close 9075 (time 1710102700.34 total 140.10 last 48.79) - open/close 9523 (time 1710102710.35 total 150.11 last 44.72) - open/close 9982 (time 1710102720.36 total 160.12 last 45.87) total: 10000 open/close in 160.59 seconds: 62.27 ops/second - unlinked 0 (time 1710102723 ; total 0 ; last 0) total: 10000 unlinks in 112 seconds: 89.285713 unlinks/second - open/close 892 (time 1710102848.41 total 10.01 last 89.10) - open/close 1800 (time 1710102858.41 total 20.01 last 90.77) - open/close 2577 (time 1710102868.45 total 30.05 last 77.45) - open/close 3281 (time 1710102878.45 total 40.05 last 70.39) - open/close 3958 (time 1710102888.47 total 50.06 last 67.59) - open/close 4599 (time 1710102898.49 total 60.09 last 63.96) - open/close 5246 (time 1710102908.49 total 70.09 last 64.66) - open/close 5863 (time 1710102918.51 total 80.11 last 61.62) - open/close 6468 (time 1710102928.52 total 90.11 last 60.44) - open/close 7049 (time 1710102938.52 total 100.12 last 58.04) - open/close 7558 (time 1710102948.54 total 110.13 last 50.85) - open/close 8120 (time 1710102958.54 total 120.14 last 56.16) - open/close 8637 (time 1710102968.54 total 130.14 last 51.69) - open/close 9187 (time 1710102978.55 total 140.15 last 54.98) - open/close 9719 (time 1710102988.55 total 150.15 last 53.17) total: 10000 open/close in 155.94 seconds: 64.13 ops/second - unlinked 0 (time 1710102997 ; total 0 ; last 0) total: 10000 unlinks in 125 seconds: 80.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.119@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.119@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg119-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg119-server: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg119-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre On OST0, 17685 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server unloading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2