== conf-sanity test 69: replace an OST with the same index ========================================================== 15:56:36 (1710100596) start mds service on oleg126-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49602 inodes available. Want 99872. rc=0 - open/close 1837 (time 1710100619.52 total 10.00 last 183.66) - open/close 2833 (time 1710100629.52 total 20.01 last 99.56) - open/close 3732 (time 1710100639.53 total 30.01 last 89.85) - open/close 4549 (time 1710100649.54 total 40.02 last 81.65) - open/close 5483 (time 1710100659.54 total 50.02 last 93.38) - open/close 6945 (time 1710100669.54 total 60.02 last 146.12) - open/close 8545 (time 1710100679.54 total 70.03 last 159.98) - open/close 9982 (time 1710100689.55 total 80.03 last 143.66) total: 10000 open/close in 80.18 seconds: 124.72 ops/second - unlinked 0 (time 1710100690 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 3101 (time 1710100734.43 total 10.00 last 310.03) - open/close 5642 (time 1710100744.44 total 20.00 last 254.06) - open/close 6728 (time 1710100754.45 total 30.01 last 108.50) - open/close 7671 (time 1710100764.45 total 40.01 last 94.29) - open/close 8653 (time 1710100774.45 total 50.02 last 98.13) total: 10000 open/close in 59.40 seconds: 168.36 ops/second - unlinked 0 (time 1710100784 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1313 (time 1710100843.91 total 10.01 last 131.23) - open/close 2420 (time 1710100853.91 total 20.01 last 110.69) - open/close 3440 (time 1710100863.93 total 30.02 last 101.89) - open/close 4219 (time 1710100873.93 total 40.02 last 77.89) - open/close 5040 (time 1710100883.93 total 50.02 last 82.09) - open/close 5881 (time 1710100893.93 total 60.02 last 84.09) - open/close 6619 (time 1710100903.93 total 70.02 last 73.78) - open/close 7291 (time 1710100913.93 total 80.02 last 67.20) - open/close 7988 (time 1710100923.93 total 90.02 last 69.69) - open/close 8621 (time 1710100933.94 total 100.03 last 63.25) - open/close 9273 (time 1710100943.95 total 110.04 last 65.14) - open/close 9981 (time 1710100953.96 total 120.05 last 70.74) total: 10000 open/close in 120.32 seconds: 83.11 ops/second - unlinked 0 (time 1710100956 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 1106 (time 1710101052.94 total 10.00 last 110.57) - open/close 2171 (time 1710101062.95 total 20.01 last 106.43) - open/close 3179 (time 1710101072.96 total 30.03 last 100.63) - open/close 4023 (time 1710101082.97 total 40.03 last 84.34) - open/close 4865 (time 1710101092.98 total 50.04 last 84.16) - open/close 5696 (time 1710101102.99 total 60.05 last 83.03) - open/close 6392 (time 1710101112.99 total 70.05 last 69.55) - open/close 7011 (time 1710101123.00 total 80.06 last 61.85) - open/close 7549 (time 1710101133.01 total 90.07 last 53.76) - open/close 8058 (time 1710101143.08 total 100.14 last 50.54) - open/close 8627 (time 1710101153.09 total 110.15 last 56.83) - open/close 9154 (time 1710101163.11 total 120.17 last 52.62) - open/close 9657 (time 1710101173.11 total 130.17 last 50.28) total: 10000 open/close in 137.84 seconds: 72.55 ops/second - unlinked 0 (time 1710101183 ; total 0 ; last 0) total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second - open/close 808 (time 1710101310.60 total 10.00 last 80.76) - open/close 1622 (time 1710101320.60 total 20.01 last 81.39) - open/close 2347 (time 1710101330.61 total 30.02 last 72.38) - open/close 3091 (time 1710101340.63 total 40.04 last 74.29) - open/close 3716 (time 1710101350.64 total 50.05 last 62.43) - open/close 4360 (time 1710101360.65 total 60.06 last 64.33) - open/close 4940 (time 1710101370.66 total 70.07 last 57.94) - open/close 5524 (time 1710101380.68 total 80.09 last 58.28) - open/close 6123 (time 1710101390.70 total 90.11 last 59.79) - open/close 6663 (time 1710101400.70 total 100.11 last 53.98) - open/close 7169 (time 1710101410.71 total 110.12 last 50.55) - open/close 7654 (time 1710101420.72 total 120.13 last 48.44) - open/close 8175 (time 1710101430.73 total 130.14 last 52.06) - open/close 8656 (time 1710101440.76 total 140.17 last 47.96) - open/close 9200 (time 1710101450.77 total 150.18 last 54.34) - open/close 9334 (time 1710101464.61 total 164.02 last 9.68) - open/close 9756 (time 1710101474.62 total 174.03 last 42.14) total: 10000 open/close in 179.46 seconds: 55.72 ops/second - unlinked 0 (time 1710101482 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 893 (time 1710101598.10 total 10.00 last 89.28) - open/close 1701 (time 1710101608.14 total 20.04 last 80.48) - open/close 2463 (time 1710101618.14 total 30.04 last 76.19) - open/close 3147 (time 1710101628.15 total 40.05 last 68.35) - open/close 3835 (time 1710101638.16 total 50.06 last 68.71) - open/close 4466 (time 1710101648.16 total 60.07 last 63.08) - open/close 4989 (time 1710101658.18 total 70.09 last 52.20) - open/close 5603 (time 1710101668.19 total 80.09 last 61.39) - open/close 6196 (time 1710101678.20 total 90.11 last 59.20) - open/close 6818 (time 1710101688.22 total 100.12 last 62.11) - open/close 7351 (time 1710101698.22 total 110.12 last 53.29) - open/close 7910 (time 1710101708.22 total 120.13 last 55.87) - open/close 8404 (time 1710101718.23 total 130.13 last 49.36) - open/close 8853 (time 1710101728.24 total 140.14 last 44.88) - open/close 9277 (time 1710101738.25 total 150.15 last 42.35) - open/close 9715 (time 1710101748.26 total 160.16 last 43.76) total: 10000 open/close in 166.42 seconds: 60.09 ops/second - unlinked 0 (time 1710101756 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 959 (time 1710101875.85 total 10.01 last 95.81) - open/close 1773 (time 1710101885.86 total 20.02 last 81.34) - open/close 2609 (time 1710101895.87 total 30.03 last 83.51) - open/close 3394 (time 1710101905.88 total 40.04 last 78.43) - open/close 3824 (time 1710101923.33 total 57.48 last 24.65) - open/close 4469 (time 1710101933.33 total 67.49 last 64.47) - open/close 5090 (time 1710101943.34 total 77.50 last 62.05) - open/close 5538 (time 1710101962.79 total 96.95 last 23.03) - open/close 6098 (time 1710101972.81 total 106.96 last 55.94) - open/close 6125 (time 1710101984.57 total 118.73 last 2.29) - open/close 6679 (time 1710101994.58 total 128.74 last 55.34) - open/close 7205 (time 1710102004.60 total 138.75 last 52.53) - open/close 7718 (time 1710102014.60 total 148.75 last 51.29) - open/close 8185 (time 1710102024.60 total 158.76 last 46.68) - open/close 8650 (time 1710102034.61 total 168.76 last 46.47) - open/close 9196 (time 1710102044.62 total 178.77 last 54.55) - open/close 9632 (time 1710102054.63 total 188.79 last 43.55) total: 10000 open/close in 197.45 seconds: 50.65 ops/second - unlinked 0 (time 1710102065 ; total 0 ; last 0) total: 10000 unlinks in 119 seconds: 84.033615 unlinks/second - open/close 950 (time 1710102197.40 total 10.00 last 94.97) - open/close 1767 (time 1710102207.40 total 20.01 last 81.62) - open/close 2567 (time 1710102217.41 total 30.02 last 79.96) - open/close 3261 (time 1710102227.42 total 40.02 last 69.35) - open/close 3909 (time 1710102237.43 total 50.03 last 64.74) - open/close 4549 (time 1710102247.43 total 60.03 last 63.99) - open/close 5156 (time 1710102257.45 total 70.05 last 60.59) - open/close 5749 (time 1710102267.45 total 80.06 last 59.25) - open/close 6299 (time 1710102277.50 total 90.11 last 54.75) - open/close 6861 (time 1710102287.51 total 100.12 last 56.13) - open/close 7431 (time 1710102297.52 total 110.13 last 56.96) - open/close 7961 (time 1710102307.53 total 120.14 last 52.94) - open/close 8475 (time 1710102317.55 total 130.16 last 51.29) - open/close 8928 (time 1710102327.56 total 140.17 last 45.25) - open/close 9366 (time 1710102337.57 total 150.18 last 43.79) - open/close 9872 (time 1710102347.57 total 160.18 last 50.58) total: 10000 open/close in 162.78 seconds: 61.43 ops/second - unlinked 0 (time 1710102352 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 877 (time 1710102472.06 total 10.03 last 87.41) - open/close 1704 (time 1710102482.06 total 20.04 last 82.66) - open/close 2489 (time 1710102492.06 total 30.04 last 78.46) - open/close 3176 (time 1710102502.07 total 40.05 last 68.66) - open/close 3832 (time 1710102512.07 total 50.05 last 65.57) - open/close 4473 (time 1710102522.08 total 60.06 last 64.04) - open/close 5152 (time 1710102532.09 total 70.07 last 67.84) - open/close 5735 (time 1710102542.10 total 80.08 last 58.27) - open/close 6269 (time 1710102552.10 total 90.08 last 53.38) - open/close 6854 (time 1710102562.11 total 100.09 last 58.45) - open/close 7353 (time 1710102572.11 total 110.09 last 49.88) - open/close 7930 (time 1710102582.12 total 120.10 last 57.64) - open/close 8393 (time 1710102592.13 total 130.10 last 46.29) - open/close 8875 (time 1710102602.15 total 140.13 last 48.06) - open/close 9312 (time 1710102612.18 total 150.16 last 43.57) - open/close 9706 (time 1710102622.20 total 160.18 last 39.34) total: 10000 open/close in 166.87 seconds: 59.93 ops/second - unlinked 0 (time 1710102631 ; total 0 ; last 0) total: 10000 unlinks in 124 seconds: 80.645164 unlinks/second - open/close 868 (time 1710102767.61 total 10.00 last 86.78) - open/close 1654 (time 1710102777.62 total 20.01 last 78.52) - open/close 2433 (time 1710102787.67 total 30.06 last 77.52) - open/close 3250 (time 1710102797.68 total 40.07 last 81.64) - open/close 3916 (time 1710102807.70 total 50.09 last 66.48) - open/close 4533 (time 1710102817.71 total 60.10 last 61.64) - open/close 5056 (time 1710102837.16 total 79.55 last 26.89) - open/close 5613 (time 1710102847.16 total 89.55 last 55.67) - open/close 6174 (time 1710102857.18 total 99.57 last 55.99) - open/close 6759 (time 1710102867.19 total 109.57 last 58.47) - open/close 7270 (time 1710102877.20 total 119.59 last 51.03) - open/close 7736 (time 1710102887.21 total 129.60 last 46.56) - open/close 8224 (time 1710102897.21 total 139.60 last 48.76) - open/close 8751 (time 1710102907.23 total 149.62 last 52.63) - open/close 9289 (time 1710102917.24 total 159.63 last 53.73) - open/close 9748 (time 1710102927.24 total 169.63 last 45.89) total: 10000 open/close in 175.68 seconds: 56.92 ops/second - unlinked 0 (time 1710102935 ; total 0 ; last 0) total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.126@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg126-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg126-server: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg126-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock oleg126-server@tcp:/lustre /mnt/lustre On OST0, 20235 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server unloading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-client: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-client: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2