== conf-sanity test 69: replace an OST with the same index ========================================================== 16:05:44 (1710101144) start mds service on oleg145-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg145-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg145-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock oleg145-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49472 inodes available. Want 99872. rc=0 - open/close 982 (time 1710101195.42 total 10.00 last 98.18) - open/close 1697 (time 1710101205.42 total 20.01 last 71.47) - open/close 1806 (time 1710101218.07 total 32.66 last 8.62) - open/close 2512 (time 1710101228.08 total 42.67 last 70.53) - open/close 2888 (time 1710101244.77 total 59.35 last 22.54) - open/close 3533 (time 1710101254.77 total 69.35 last 64.50) - open/close 4205 (time 1710101264.78 total 79.36 last 67.11) - open/close 4766 (time 1710101274.79 total 89.38 last 56.02) - open/close 5334 (time 1710101284.80 total 99.39 last 56.74) - open/close 5950 (time 1710101294.81 total 109.39 last 61.56) - open/close 6467 (time 1710101304.82 total 119.41 last 51.63) - open/close 6934 (time 1710101314.84 total 129.42 last 46.62) - open/close 7431 (time 1710101324.86 total 139.45 last 49.59) - open/close 7930 (time 1710101334.87 total 149.45 last 49.88) - open/close 8404 (time 1710101344.89 total 159.47 last 47.30) - open/close 8811 (time 1710101354.90 total 169.48 last 40.67) - open/close 9007 (time 1710101370.24 total 184.82 last 12.77) - open/close 9442 (time 1710101380.25 total 194.83 last 43.46) - open/close 9860 (time 1710101390.26 total 204.85 last 41.74) total: 10000 open/close in 208.19 seconds: 48.03 ops/second - unlinked 0 (time 1710101396 ; total 0 ; last 0) total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second - open/close 444 (time 1710101527.85 total 15.97 last 27.81) - open/close 1263 (time 1710101537.86 total 25.98 last 81.82) - open/close 2099 (time 1710101547.88 total 35.99 last 83.51) - open/close 2802 (time 1710101557.89 total 46.00 last 70.20) - open/close 3553 (time 1710101567.90 total 56.01 last 75.02) - open/close 4197 (time 1710101577.90 total 66.01 last 64.39) - open/close 4324 (time 1710101591.10 total 79.21 last 9.62) - open/close 4915 (time 1710101601.10 total 89.22 last 59.08) - open/close 5486 (time 1710101611.11 total 99.22 last 57.05) - open/close 6018 (time 1710101621.12 total 109.23 last 53.15) - open/close 6554 (time 1710101631.13 total 119.24 last 53.55) - open/close 7077 (time 1710101641.14 total 129.25 last 52.25) - open/close 7535 (time 1710101651.15 total 139.27 last 45.74) - open/close 8001 (time 1710101661.16 total 149.27 last 46.56) - open/close 8470 (time 1710101681.86 total 169.97 last 22.66) - open/close 9035 (time 1710101691.87 total 179.99 last 56.42) - open/close 9497 (time 1710101701.90 total 190.01 last 46.08) - open/close 9962 (time 1710101711.93 total 200.04 last 46.36) total: 10000 open/close in 201.08 seconds: 49.73 ops/second - unlinked 0 (time 1710101715 ; total 0 ; last 0) total: 10000 unlinks in 123 seconds: 81.300812 unlinks/second - open/close 910 (time 1710101850.79 total 10.01 last 90.87) - open/close 1734 (time 1710101860.80 total 20.02 last 82.34) - open/close 2515 (time 1710101870.84 total 30.07 last 77.72) - open/close 3190 (time 1710101880.86 total 40.08 last 67.40) - open/close 3904 (time 1710101890.87 total 50.10 last 71.32) - open/close 4583 (time 1710101900.88 total 60.10 last 67.86) - open/close 5217 (time 1710101910.89 total 70.11 last 63.33) - open/close 5773 (time 1710101920.90 total 80.12 last 55.54) - open/close 6307 (time 1710101930.99 total 90.22 last 52.89) - open/close 6854 (time 1710101940.99 total 100.22 last 54.69) - open/close 7368 (time 1710101951.05 total 110.28 last 51.09) - open/close 7946 (time 1710101961.06 total 120.28 last 57.78) - open/close 8444 (time 1710101971.08 total 130.30 last 49.71) - open/close 8950 (time 1710101981.08 total 140.31 last 50.56) - open/close 9405 (time 1710101991.09 total 150.31 last 45.50) - open/close 9827 (time 1710102011.38 total 170.61 last 20.79) - open/close 9906 (time 1710102024.11 total 183.34 last 6.21) total: 10000 open/close in 185.48 seconds: 53.91 ops/second - unlinked 0 (time 1710102028 ; total 0 ; last 0) total: 10000 unlinks in 123 seconds: 81.300812 unlinks/second - open/close 801 (time 1710102163.87 total 10.06 last 79.64) - open/close 1517 (time 1710102173.88 total 20.06 last 71.55) - open/close 1559 (time 1710102185.35 total 31.54 last 3.66) - open/close 1626 (time 1710102197.28 total 43.47 last 5.62) - open/close 2363 (time 1710102207.29 total 53.47 last 73.65) - open/close 3079 (time 1710102217.30 total 63.48 last 71.54) - open/close 3730 (time 1710102227.30 total 73.49 last 65.08) - open/close 4321 (time 1710102237.30 total 83.49 last 59.08) - open/close 4930 (time 1710102247.31 total 93.49 last 60.88) - open/close 5423 (time 1710102257.34 total 103.53 last 49.13) - open/close 5957 (time 1710102277.86 total 124.05 last 26.02) - open/close 6515 (time 1710102287.87 total 134.06 last 55.73) - open/close 7034 (time 1710102297.88 total 144.07 last 51.87) - open/close 7378 (time 1710102315.38 total 161.56 last 19.66) - open/close 7810 (time 1710102335.40 total 181.59 last 21.58) - open/close 8311 (time 1710102345.40 total 191.59 last 50.09) - open/close 8796 (time 1710102355.41 total 201.60 last 48.44) - open/close 9298 (time 1710102365.42 total 211.61 last 50.15) - open/close 9792 (time 1710102375.44 total 221.62 last 49.33) total: 10000 open/close in 226.72 seconds: 44.11 ops/second - unlinked 0 (time 1710102383 ; total 0 ; last 0) total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second - open/close 880 (time 1710102505.56 total 10.01 last 87.94) - open/close 1690 (time 1710102515.56 total 20.01 last 80.99) - open/close 2391 (time 1710102525.57 total 30.01 last 70.05) - open/close 3113 (time 1710102535.57 total 40.02 last 72.18) - open/close 3774 (time 1710102545.58 total 50.03 last 66.03) - open/close 4391 (time 1710102555.59 total 60.03 last 61.66) - open/close 4970 (time 1710102565.60 total 70.05 last 57.81) - open/close 5539 (time 1710102575.62 total 80.06 last 56.82) - open/close 6157 (time 1710102585.62 total 90.07 last 61.74) - open/close 6653 (time 1710102595.65 total 100.10 last 49.47) - open/close 7187 (time 1710102605.65 total 110.10 last 53.38) - open/close 7664 (time 1710102615.65 total 120.10 last 47.70) - open/close 8114 (time 1710102625.67 total 130.11 last 44.95) - open/close 8569 (time 1710102635.68 total 140.13 last 45.45) - open/close 9058 (time 1710102645.68 total 150.13 last 48.89) - open/close 9497 (time 1710102655.69 total 160.14 last 43.85) - open/close 9953 (time 1710102665.70 total 170.15 last 45.55) total: 10000 open/close in 171.20 seconds: 58.41 ops/second - unlinked 0 (time 1710102669 ; total 0 ; last 0) total: 10000 unlinks in 122 seconds: 81.967216 unlinks/second - open/close 961 (time 1710102803.73 total 10.01 last 95.98) - open/close 1776 (time 1710102813.75 total 20.03 last 81.35) - open/close 2542 (time 1710102823.76 total 30.04 last 76.54) - open/close 3180 (time 1710102833.78 total 40.06 last 63.66) - open/close 3794 (time 1710102843.79 total 50.07 last 61.34) - open/close 4447 (time 1710102853.79 total 60.07 last 65.28) - open/close 5042 (time 1710102863.87 total 70.15 last 59.03) - open/close 5372 (time 1710102880.88 total 87.16 last 19.40) - open/close 5914 (time 1710102890.89 total 97.17 last 54.13) - open/close 6454 (time 1710102900.89 total 107.17 last 54.00) - open/close 6970 (time 1710102910.89 total 117.18 last 51.59) - open/close 7527 (time 1710102920.91 total 127.19 last 55.62) - open/close 7599 (time 1710102933.22 total 139.50 last 5.85) - open/close 8075 (time 1710102943.23 total 149.51 last 47.53) - open/close 8491 (time 1710102953.23 total 159.52 last 41.59) - open/close 8963 (time 1710102963.25 total 169.53 last 47.14) - open/close 9416 (time 1710102973.26 total 179.54 last 45.25) - open/close 9839 (time 1710102983.26 total 189.54 last 42.30) total: 10000 open/close in 193.18 seconds: 51.76 ops/second - unlinked 0 (time 1710102989 ; total 0 ; last 0) total: 10000 unlinks in 140 seconds: 71.428574 unlinks/second - open/close 899 (time 1710103141.87 total 10.00 last 89.87) - open/close 1726 (time 1710103162.41 total 30.54 last 40.27) - open/close 2441 (time 1710103172.41 total 40.55 last 71.45) - open/close 3271 (time 1710103182.42 total 50.55 last 82.94) - open/close 3302 (time 1710103193.78 total 61.91 last 2.73) - open/close 4029 (time 1710103203.79 total 71.92 last 72.65) - open/close 4720 (time 1710103213.80 total 81.93 last 69.04) - open/close 5407 (time 1710103223.80 total 91.94 last 68.65) - open/close 5970 (time 1710103233.83 total 101.96 last 56.17) - open/close 6575 (time 1710103243.84 total 111.97 last 60.45) - open/close 7221 (time 1710103253.84 total 121.97 last 64.57) - open/close 7767 (time 1710103263.84 total 131.98 last 54.58) - open/close 8387 (time 1710103273.85 total 141.98 last 61.97) - open/close 8955 (time 1710103283.87 total 152.00 last 56.69) - open/close 9468 (time 1710103293.88 total 162.01 last 51.24) total: 10000 open/close in 171.56 seconds: 58.29 ops/second - unlinked 0 (time 1710103305 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 1127 (time 1710103403.58 total 10.00 last 112.68) - open/close 2075 (time 1710103413.60 total 20.01 last 94.69) - open/close 3071 (time 1710103423.60 total 30.02 last 99.57) - open/close 3991 (time 1710103433.61 total 40.03 last 91.92) - open/close 4831 (time 1710103443.62 total 50.03 last 83.92) - open/close 5625 (time 1710103453.63 total 60.05 last 79.31) - open/close 6424 (time 1710103463.64 total 70.05 last 79.83) - open/close 7238 (time 1710103473.64 total 80.06 last 81.36) - open/close 7980 (time 1710103483.64 total 90.06 last 74.20) - open/close 8755 (time 1710103493.65 total 100.07 last 77.43) - open/close 9566 (time 1710103503.66 total 110.08 last 81.00) total: 10000 open/close in 115.99 seconds: 86.22 ops/second - unlinked 0 (time 1710103510 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 1726 (time 1710103580.69 total 10.00 last 172.55) - open/close 3375 (time 1710103590.70 total 20.01 last 164.79) - open/close 4903 (time 1710103600.70 total 30.01 last 152.79) - open/close 6305 (time 1710103610.70 total 40.01 last 140.20) - open/close 7682 (time 1710103620.70 total 50.02 last 137.63) - open/close 8931 (time 1710103630.71 total 60.02 last 124.86) total: 10000 open/close in 68.85 seconds: 145.24 ops/second - unlinked 0 (time 1710103640 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 3387 (time 1710103689.18 total 10.00 last 338.61) - open/close 5831 (time 1710103699.18 total 20.00 last 244.39) - open/close 7853 (time 1710103709.18 total 30.01 last 202.15) - open/close 9822 (time 1710103719.18 total 40.01 last 196.86) total: 10000 open/close in 41.01 seconds: 243.87 ops/second - unlinked 0 (time 1710103721 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg145-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock oleg145-server@tcp:/lustre /mnt/lustre On OST0, 20397 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg145-server unloading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2