== conf-sanity test 69: replace an OST with the same index ========================================================== 15:59:03 (1710100743) start mds service on oleg314-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49602 inodes available. Want 99872. rc=0 - open/close 1831 (time 1710100767.60 total 10.01 last 182.97) - open/close 3466 (time 1710100777.61 total 20.01 last 163.44) - open/close 5086 (time 1710100787.61 total 30.02 last 161.92) - open/close 6956 (time 1710100797.61 total 40.02 last 186.97) - open/close 8767 (time 1710100807.62 total 50.02 last 181.05) total: 10000 open/close in 58.21 seconds: 171.80 ops/second - unlinked 0 (time 1710100816 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1887 (time 1710100873.02 total 10.00 last 188.69) - open/close 3708 (time 1710100883.02 total 20.01 last 182.01) - open/close 5275 (time 1710100893.03 total 30.01 last 156.67) - open/close 6688 (time 1710100903.03 total 40.01 last 141.26) - open/close 8273 (time 1710100913.03 total 50.01 last 158.49) - open/close 9631 (time 1710100923.03 total 60.02 last 135.73) total: 10000 open/close in 63.53 seconds: 157.41 ops/second - unlinked 0 (time 1710100927 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1433 (time 1710100987.07 total 10.00 last 143.25) - open/close 2676 (time 1710100997.07 total 20.01 last 124.27) - open/close 4052 (time 1710101007.08 total 30.01 last 137.53) - open/close 5174 (time 1710101017.08 total 40.02 last 112.14) - open/close 6214 (time 1710101027.09 total 50.02 last 103.94) - open/close 7205 (time 1710101037.10 total 60.04 last 98.97) - open/close 8203 (time 1710101047.11 total 70.05 last 99.70) - open/close 9063 (time 1710101057.11 total 80.05 last 85.99) - open/close 9913 (time 1710101067.12 total 90.05 last 84.93) total: 10000 open/close in 91.09 seconds: 109.79 ops/second - unlinked 0 (time 1710101069 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1317 (time 1710101146.63 total 10.01 last 131.62) - open/close 2582 (time 1710101156.63 total 20.01 last 126.49) - open/close 3695 (time 1710101166.63 total 30.01 last 111.23) - open/close 4729 (time 1710101176.64 total 40.02 last 103.37) - open/close 5674 (time 1710101186.64 total 50.02 last 94.47) - open/close 6642 (time 1710101196.65 total 60.03 last 96.73) - open/close 7495 (time 1710101206.66 total 70.04 last 85.22) - open/close 8420 (time 1710101216.67 total 80.04 last 92.42) - open/close 9300 (time 1710101226.68 total 90.06 last 87.88) total: 10000 open/close in 98.40 seconds: 101.62 ops/second - unlinked 0 (time 1710101236 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1410 (time 1710101309.90 total 10.00 last 140.95) - open/close 2689 (time 1710101319.91 total 20.01 last 127.83) - open/close 3875 (time 1710101329.91 total 30.01 last 118.56) - open/close 4882 (time 1710101339.92 total 40.02 last 100.63) - open/close 5931 (time 1710101349.92 total 50.02 last 104.87) - open/close 6878 (time 1710101359.92 total 60.02 last 94.69) - open/close 7789 (time 1710101369.92 total 70.03 last 91.08) - open/close 8682 (time 1710101379.93 total 80.03 last 89.25) - open/close 9535 (time 1710101389.93 total 90.03 last 85.30) total: 10000 open/close in 96.06 seconds: 104.10 ops/second - unlinked 0 (time 1710101397 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1383 (time 1710101471.01 total 10.00 last 138.25) - open/close 2596 (time 1710101481.03 total 20.02 last 121.13) - open/close 3701 (time 1710101491.03 total 30.02 last 110.50) - open/close 4781 (time 1710101501.03 total 40.02 last 107.97) - open/close 5781 (time 1710101511.03 total 50.02 last 99.97) - open/close 6717 (time 1710101521.04 total 60.02 last 93.59) - open/close 7667 (time 1710101531.04 total 70.03 last 94.94) - open/close 8539 (time 1710101541.05 total 80.04 last 87.13) - open/close 9312 (time 1710101551.05 total 90.04 last 77.27) total: 10000 open/close in 98.85 seconds: 101.16 ops/second - unlinked 0 (time 1710101561 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1428 (time 1710101634.86 total 10.00 last 142.78) - open/close 2693 (time 1710101644.86 total 20.01 last 126.41) - open/close 3793 (time 1710101654.87 total 30.01 last 109.97) - open/close 4836 (time 1710101664.87 total 40.02 last 104.22) - open/close 5070 (time 1710101678.16 total 53.30 last 17.61) - open/close 6057 (time 1710101688.17 total 63.31 last 98.61) - open/close 7013 (time 1710101698.17 total 73.32 last 95.57) - open/close 7899 (time 1710101708.18 total 83.32 last 88.54) - open/close 8788 (time 1710101718.18 total 93.32 last 88.87) - open/close 9667 (time 1710101728.18 total 103.33 last 87.89) total: 10000 open/close in 107.27 seconds: 93.22 ops/second - unlinked 0 (time 1710101733 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 1436 (time 1710101805.30 total 10.00 last 143.55) - open/close 2757 (time 1710101815.30 total 20.01 last 132.00) - open/close 3934 (time 1710101825.31 total 30.01 last 117.67) - open/close 5021 (time 1710101835.31 total 40.02 last 108.64) - open/close 6033 (time 1710101845.32 total 50.02 last 101.14) - open/close 7003 (time 1710101855.32 total 60.03 last 96.98) - open/close 7933 (time 1710101865.33 total 70.03 last 92.93) - open/close 8782 (time 1710101875.33 total 80.03 last 84.90) - open/close 9566 (time 1710101885.33 total 90.04 last 78.39) total: 10000 open/close in 95.45 seconds: 104.77 ops/second - unlinked 0 (time 1710101892 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1432 (time 1710101966.76 total 10.01 last 143.09) - open/close 2718 (time 1710101976.77 total 20.01 last 128.54) - open/close 3886 (time 1710101986.77 total 30.01 last 116.78) - open/close 5038 (time 1710101996.78 total 40.03 last 115.06) - open/close 6108 (time 1710102006.78 total 50.03 last 106.98) - open/close 7128 (time 1710102016.79 total 60.04 last 101.88) - open/close 8054 (time 1710102026.79 total 70.04 last 92.59) - open/close 8965 (time 1710102036.80 total 80.05 last 91.05) - open/close 9808 (time 1710102046.80 total 90.05 last 84.25) total: 10000 open/close in 92.24 seconds: 108.41 ops/second - unlinked 0 (time 1710102050 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 1438 (time 1710102125.97 total 10.00 last 143.76) - open/close 2719 (time 1710102135.97 total 20.00 last 128.09) - open/close 3937 (time 1710102145.97 total 30.00 last 121.79) - open/close 5088 (time 1710102155.98 total 40.02 last 114.96) - open/close 6212 (time 1710102165.99 total 50.02 last 112.32) - open/close 7294 (time 1710102175.99 total 60.03 last 108.16) - open/close 8337 (time 1710102186.00 total 70.03 last 104.26) - open/close 9345 (time 1710102196.00 total 80.04 last 100.75) total: 10000 open/close in 87.11 seconds: 114.80 ops/second - unlinked 0 (time 1710102204 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 19 sec mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre On OST0, 20039 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server unloading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2