== conf-sanity test 69: replace an OST with the same index ========================================================== 16:02:56 (1710100976) start mds service on oleg359-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg359-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg359-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 1262 (time 1710101019.20 total 10.00 last 126.19) - open/close 2398 (time 1710101029.21 total 20.01 last 113.51) - open/close 3499 (time 1710101039.22 total 30.01 last 110.03) - open/close 4539 (time 1710101049.22 total 40.02 last 103.94) - open/close 5423 (time 1710101059.23 total 50.02 last 88.36) - open/close 6316 (time 1710101069.25 total 60.05 last 89.08) - open/close 7178 (time 1710101079.26 total 70.06 last 86.13) - open/close 8028 (time 1710101089.27 total 80.06 last 84.96) - open/close 8718 (time 1710101099.27 total 90.06 last 69.00) - open/close 9402 (time 1710101109.28 total 100.08 last 68.30) total: 10000 open/close in 108.95 seconds: 91.79 ops/second - unlinked 0 (time 1710101119 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 1123 (time 1710101209.14 total 10.00 last 112.27) - open/close 2244 (time 1710101219.14 total 20.00 last 112.09) - open/close 3289 (time 1710101229.15 total 30.01 last 104.45) - open/close 4275 (time 1710101239.15 total 40.02 last 98.52) - open/close 5194 (time 1710101249.16 total 50.02 last 91.85) - open/close 6113 (time 1710101259.17 total 60.03 last 91.85) - open/close 6930 (time 1710101269.17 total 70.04 last 81.64) - open/close 7572 (time 1710101288.44 total 89.30 last 33.33) - open/close 8358 (time 1710101298.45 total 99.31 last 78.52) - open/close 9151 (time 1710101308.45 total 109.31 last 79.23) - open/close 9884 (time 1710101318.46 total 119.32 last 73.27) total: 10000 open/close in 120.85 seconds: 82.74 ops/second - unlinked 0 (time 1710101321 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 1213 (time 1710101406.88 total 10.00 last 121.27) - open/close 2328 (time 1710101416.89 total 20.01 last 111.41) - open/close 3365 (time 1710101426.89 total 30.01 last 103.69) - open/close 4344 (time 1710101436.90 total 40.02 last 97.84) - open/close 5253 (time 1710101446.91 total 50.02 last 90.84) - open/close 6116 (time 1710101456.92 total 60.03 last 86.22) - open/close 6949 (time 1710101466.92 total 70.04 last 83.23) - open/close 7740 (time 1710101476.94 total 80.06 last 78.99) - open/close 8460 (time 1710101486.95 total 90.07 last 71.93) - open/close 9181 (time 1710101496.96 total 100.07 last 72.04) - open/close 9847 (time 1710101506.96 total 110.08 last 66.58) total: 10000 open/close in 112.33 seconds: 89.03 ops/second - unlinked 0 (time 1710101510 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 1196 (time 1710101595.31 total 10.00 last 119.58) - open/close 2297 (time 1710101605.32 total 20.01 last 110.03) - open/close 3349 (time 1710101615.32 total 30.01 last 105.15) - open/close 4313 (time 1710101625.32 total 40.02 last 96.37) - open/close 5254 (time 1710101635.33 total 50.02 last 94.03) - open/close 6138 (time 1710101645.34 total 60.03 last 88.33) - open/close 6930 (time 1710101655.34 total 70.04 last 79.15) - open/close 7717 (time 1710101665.35 total 80.04 last 78.64) - open/close 8481 (time 1710101675.36 total 90.06 last 76.32) - open/close 9245 (time 1710101685.37 total 100.07 last 76.31) - open/close 9930 (time 1710101695.37 total 110.07 last 68.50) total: 10000 open/close in 111.06 seconds: 90.04 ops/second - unlinked 0 (time 1710101698 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 1258 (time 1710101782.27 total 10.01 last 125.72) - open/close 2344 (time 1710101792.28 total 20.01 last 108.53) - open/close 3394 (time 1710101802.28 total 30.02 last 104.97) - open/close 4397 (time 1710101812.29 total 40.02 last 100.22) - open/close 5307 (time 1710101822.29 total 50.03 last 90.94) - open/close 6209 (time 1710101832.29 total 60.03 last 90.18) - open/close 7093 (time 1710101842.30 total 70.04 last 88.34) - open/close 7883 (time 1710101852.31 total 80.04 last 78.95) - open/close 8645 (time 1710101862.31 total 90.05 last 76.15) - open/close 9323 (time 1710101872.34 total 100.08 last 67.64) - open/close 9974 (time 1710101882.36 total 110.10 last 64.97) total: 10000 open/close in 110.46 seconds: 90.53 ops/second - unlinked 0 (time 1710101884 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1217 (time 1710101973.28 total 10.01 last 121.57) - open/close 2383 (time 1710101983.29 total 20.02 last 116.55) - open/close 3475 (time 1710101993.29 total 30.02 last 109.15) - open/close 4495 (time 1710102003.30 total 40.03 last 101.92) - open/close 5482 (time 1710102013.31 total 50.04 last 98.61) - open/close 6298 (time 1710102033.42 total 70.15 last 40.57) - open/close 7134 (time 1710102043.42 total 80.15 last 83.57) - open/close 7952 (time 1710102053.43 total 90.16 last 81.73) - open/close 8731 (time 1710102063.44 total 100.17 last 77.84) - open/close 9496 (time 1710102073.45 total 110.18 last 76.41) total: 10000 open/close in 116.95 seconds: 85.51 ops/second - unlinked 0 (time 1710102081 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 1335 (time 1710102174.31 total 10.01 last 133.40) - open/close 2580 (time 1710102184.32 total 20.01 last 124.45) - open/close 3761 (time 1710102194.32 total 30.02 last 118.03) - open/close 4857 (time 1710102204.33 total 40.03 last 109.50) - open/close 5862 (time 1710102214.34 total 50.03 last 100.41) - open/close 6819 (time 1710102224.34 total 60.04 last 95.67) - open/close 7720 (time 1710102234.34 total 70.04 last 90.08) - open/close 8592 (time 1710102244.35 total 80.04 last 87.17) - open/close 9395 (time 1710102254.36 total 90.06 last 80.17) total: 10000 open/close in 98.04 seconds: 102.00 ops/second - unlinked 0 (time 1710102263 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 1456 (time 1710102338.83 total 10.01 last 145.50) - open/close 2785 (time 1710102348.83 total 20.01 last 132.90) - open/close 3978 (time 1710102358.83 total 30.01 last 119.25) - open/close 5137 (time 1710102368.84 total 40.02 last 115.81) - open/close 6195 (time 1710102378.84 total 50.02 last 105.78) - open/close 7222 (time 1710102388.84 total 60.02 last 102.67) - open/close 8156 (time 1710102398.85 total 70.03 last 93.37) - open/close 9033 (time 1710102408.86 total 80.04 last 87.60) - open/close 9863 (time 1710102418.87 total 90.05 last 82.90) total: 10000 open/close in 91.71 seconds: 109.04 ops/second - unlinked 0 (time 1710102421 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 1718 (time 1710102493.97 total 10.00 last 171.76) - open/close 3250 (time 1710102503.97 total 20.01 last 153.10) - open/close 4686 (time 1710102513.98 total 30.01 last 143.55) - open/close 5994 (time 1710102523.98 total 40.01 last 130.78) - open/close 7208 (time 1710102533.98 total 50.02 last 121.33) - open/close 8342 (time 1710102543.98 total 60.02 last 113.39) - open/close 9475 (time 1710102553.98 total 70.02 last 113.30) total: 10000 open/close in 74.84 seconds: 133.62 ops/second - unlinked 0 (time 1710102559 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 3079 (time 1710102606.85 total 10.00 last 307.81) - open/close 5489 (time 1710102616.85 total 20.01 last 240.93) - open/close 7390 (time 1710102626.85 total 30.01 last 190.03) - open/close 9288 (time 1710102636.86 total 40.01 last 189.77) total: 10000 open/close in 43.58 seconds: 229.46 ops/second - unlinked 0 (time 1710102641 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.159@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg359-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg359-server: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg359-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg359-server: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg359-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock oleg359-server@tcp:/lustre /mnt/lustre On OST0, 14665 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg359-server unloading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2