== conf-sanity test 69: replace an OST with the same index ========================================================== 16:00:50 (1710100850) start mds service on oleg157-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg157-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg157-client.virtnet: -o user_xattr,flock oleg157-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49602 inodes available. Want 99872. rc=0 - open/close 1104 (time 1710100880.77 total 10.00 last 110.38) - open/close 2235 (time 1710100890.77 total 20.00 last 113.08) - open/close 3179 (time 1710100900.78 total 30.01 last 94.35) - open/close 4023 (time 1710100910.79 total 40.02 last 84.33) - open/close 4816 (time 1710100920.83 total 50.06 last 78.99) - open/close 5606 (time 1710100930.83 total 60.06 last 79.00) - open/close 6362 (time 1710100940.83 total 70.06 last 75.56) - open/close 7134 (time 1710100950.84 total 80.07 last 77.16) - open/close 7889 (time 1710100960.85 total 90.08 last 75.41) - open/close 8554 (time 1710100970.85 total 100.08 last 66.47) - open/close 9197 (time 1710100980.87 total 110.10 last 64.17) - open/close 9747 (time 1710100990.87 total 120.11 last 54.99) total: 10000 open/close in 125.01 seconds: 79.99 ops/second - unlinked 0 (time 1710100997 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 1090 (time 1710101089.52 total 10.00 last 109.00) - open/close 2187 (time 1710101099.53 total 20.01 last 109.62) - open/close 2724 (time 1710101116.07 total 36.55 last 32.47) - open/close 3523 (time 1710101126.07 total 46.55 last 79.85) - open/close 4214 (time 1710101136.07 total 56.56 last 69.09) - open/close 4816 (time 1710101146.08 total 66.56 last 60.19) - open/close 5397 (time 1710101165.19 total 85.67 last 30.39) - open/close 6028 (time 1710101175.19 total 95.68 last 63.08) - open/close 6596 (time 1710101185.20 total 105.68 last 56.75) - open/close 7172 (time 1710101195.21 total 115.70 last 57.53) - open/close 7646 (time 1710101205.22 total 125.70 last 47.36) - open/close 8183 (time 1710101215.23 total 135.72 last 53.64) - open/close 8713 (time 1710101225.24 total 145.72 last 52.99) - open/close 9187 (time 1710101235.25 total 155.73 last 47.33) - open/close 9694 (time 1710101245.28 total 165.76 last 50.56) total: 10000 open/close in 174.08 seconds: 57.45 ops/second - unlinked 0 (time 1710101256 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 795 (time 1710101374.45 total 10.01 last 79.42) - open/close 1680 (time 1710101384.47 total 20.02 last 88.38) - open/close 2441 (time 1710101394.49 total 30.04 last 75.95) - open/close 3047 (time 1710101404.49 total 40.05 last 60.58) - open/close 3722 (time 1710101414.50 total 50.05 last 67.45) - open/close 4302 (time 1710101424.50 total 60.06 last 57.96) - open/close 4953 (time 1710101434.51 total 70.06 last 65.07) - open/close 5528 (time 1710101444.52 total 80.08 last 57.43) - open/close 6145 (time 1710101454.56 total 90.12 last 61.46) - open/close 6685 (time 1710101464.59 total 100.14 last 53.87) - open/close 7192 (time 1710101474.60 total 110.15 last 50.65) - open/close 7688 (time 1710101484.62 total 120.17 last 49.49) - open/close 8209 (time 1710101494.63 total 130.18 last 52.06) - open/close 8671 (time 1710101504.63 total 140.19 last 46.17) - open/close 9104 (time 1710101514.63 total 150.19 last 43.29) - open/close 9622 (time 1710101524.64 total 160.19 last 51.79) total: 10000 open/close in 168.41 seconds: 59.38 ops/second - unlinked 0 (time 1710101535 ; total 0 ; last 0) total: 10000 unlinks in 127 seconds: 78.740158 unlinks/second - open/close 921 (time 1710101674.99 total 10.00 last 92.06) - open/close 1801 (time 1710101685.00 total 20.01 last 87.91) - open/close 2618 (time 1710101695.01 total 30.03 last 81.60) - open/close 3353 (time 1710101705.02 total 40.04 last 73.42) - open/close 4043 (time 1710101715.04 total 50.06 last 68.85) - open/close 4615 (time 1710101725.05 total 60.07 last 57.15) - open/close 5188 (time 1710101735.06 total 70.08 last 57.26) - open/close 5783 (time 1710101745.06 total 80.08 last 59.50) - open/close 6373 (time 1710101755.07 total 90.08 last 58.95) - open/close 6928 (time 1710101765.08 total 100.10 last 55.44) - open/close 7398 (time 1710101775.08 total 110.10 last 47.00) - open/close 7923 (time 1710101785.08 total 120.10 last 52.48) - open/close 8392 (time 1710101795.09 total 130.10 last 46.89) - open/close 8895 (time 1710101805.09 total 140.11 last 50.25) - open/close 9357 (time 1710101815.10 total 150.12 last 46.17) - open/close 9771 (time 1710101825.11 total 160.12 last 41.38) total: 10000 open/close in 165.60 seconds: 60.39 ops/second - unlinked 0 (time 1710101833 ; total 0 ; last 0) total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second - open/close 950 (time 1710101960.03 total 10.01 last 94.93) - open/close 1760 (time 1710101970.04 total 20.02 last 80.89) - open/close 2461 (time 1710101980.05 total 30.03 last 70.06) - open/close 3205 (time 1710101990.07 total 40.05 last 74.26) - open/close 3903 (time 1710102000.12 total 50.10 last 69.44) - open/close 4514 (time 1710102010.12 total 60.10 last 61.08) - open/close 5074 (time 1710102020.15 total 70.13 last 55.84) - open/close 5635 (time 1710102030.16 total 80.14 last 56.04) - open/close 6209 (time 1710102040.17 total 90.15 last 57.35) - open/close 6784 (time 1710102050.18 total 100.16 last 57.44) - open/close 7299 (time 1710102060.19 total 110.17 last 51.46) - open/close 7345 (time 1710102072.21 total 122.19 last 3.83) - open/close 7781 (time 1710102082.22 total 132.20 last 43.56) - open/close 8234 (time 1710102092.24 total 142.22 last 45.21) - open/close 8399 (time 1710102106.62 total 156.60 last 11.47) - open/close 8853 (time 1710102116.64 total 166.61 last 45.35) - open/close 9267 (time 1710102126.64 total 176.62 last 41.38) - open/close 9731 (time 1710102136.64 total 186.62 last 46.38) total: 10000 open/close in 192.37 seconds: 51.98 ops/second - unlinked 0 (time 1710102144 ; total 0 ; last 0) total: 10000 unlinks in 140 seconds: 71.428574 unlinks/second - open/close 856 (time 1710102296.64 total 10.00 last 85.59) - open/close 1723 (time 1710102306.64 total 20.01 last 86.64) - open/close 2522 (time 1710102316.65 total 30.01 last 79.89) - open/close 3178 (time 1710102326.65 total 40.01 last 65.58) - open/close 3812 (time 1710102336.66 total 50.02 last 63.33) - open/close 4462 (time 1710102346.69 total 60.06 last 64.78) - open/close 5089 (time 1710102356.70 total 70.06 last 62.66) - open/close 5692 (time 1710102366.71 total 80.07 last 60.25) - open/close 5693 (time 1710102377.74 total 91.10 last 0.09) - open/close 6216 (time 1710102387.75 total 101.11 last 52.25) - open/close 6693 (time 1710102397.76 total 111.12 last 47.63) - open/close 7170 (time 1710102407.78 total 121.14 last 47.62) - open/close 7670 (time 1710102417.80 total 131.16 last 49.92) - open/close 8197 (time 1710102427.81 total 141.18 last 52.61) - open/close 8291 (time 1710102440.87 total 154.23 last 7.20) - open/close 8746 (time 1710102450.87 total 164.24 last 45.49) - open/close 9210 (time 1710102460.89 total 174.26 last 46.30) - open/close 9337 (time 1710102474.74 total 188.10 last 9.17) - open/close 9789 (time 1710102484.74 total 198.11 last 45.17) total: 10000 open/close in 202.17 seconds: 49.46 ops/second - unlinked 0 (time 1710102491 ; total 0 ; last 0) total: 10000 unlinks in 129 seconds: 77.519379 unlinks/second - open/close 903 (time 1710102633.25 total 10.01 last 90.20) - open/close 1747 (time 1710102643.26 total 20.02 last 84.35) - open/close 2461 (time 1710102653.27 total 30.03 last 71.33) - open/close 3147 (time 1710102663.27 total 40.03 last 68.58) - open/close 3852 (time 1710102673.29 total 50.05 last 70.38) - open/close 4433 (time 1710102683.29 total 60.05 last 58.07) - open/close 5036 (time 1710102693.30 total 70.06 last 60.28) - open/close 5581 (time 1710102703.30 total 80.06 last 54.49) - open/close 6085 (time 1710102713.31 total 90.07 last 50.34) - open/close 6622 (time 1710102723.32 total 100.08 last 53.63) - open/close 7160 (time 1710102733.32 total 110.09 last 53.79) - open/close 7688 (time 1710102743.34 total 120.10 last 52.74) - open/close 8203 (time 1710102764.41 total 141.17 last 24.44) - open/close 8656 (time 1710102774.47 total 151.23 last 45.03) - open/close 9125 (time 1710102784.54 total 161.30 last 46.57) - open/close 9551 (time 1710102794.54 total 171.31 last 42.59) total: 10000 open/close in 181.10 seconds: 55.22 ops/second - unlinked 0 (time 1710102807 ; total 0 ; last 0) total: 10000 unlinks in 132 seconds: 75.757576 unlinks/second - open/close 835 (time 1710102952.38 total 10.01 last 83.40) - open/close 1663 (time 1710102962.38 total 20.02 last 82.75) - open/close 2378 (time 1710102972.39 total 30.03 last 71.40) - open/close 3110 (time 1710102982.41 total 40.05 last 73.07) - open/close 3782 (time 1710102992.43 total 50.07 last 67.07) - open/close 4417 (time 1710103002.44 total 60.08 last 63.46) - open/close 4955 (time 1710103012.47 total 70.11 last 53.61) - open/close 5516 (time 1710103022.48 total 80.12 last 56.07) - open/close 6098 (time 1710103032.54 total 90.17 last 57.87) - open/close 6602 (time 1710103042.57 total 100.21 last 50.24) - open/close 7127 (time 1710103052.57 total 110.21 last 52.48) - open/close 7644 (time 1710103062.57 total 120.21 last 51.69) - open/close 8179 (time 1710103072.58 total 130.22 last 53.48) - open/close 8677 (time 1710103082.58 total 140.22 last 49.77) - open/close 8997 (time 1710103099.91 total 157.54 last 18.47) - open/close 9443 (time 1710103109.93 total 167.56 last 44.51) - open/close 9925 (time 1710103119.93 total 177.57 last 48.17) total: 10000 open/close in 179.08 seconds: 55.84 ops/second - unlinked 0 (time 1710103123 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 935 (time 1710103232.27 total 10.02 last 93.32) - open/close 1781 (time 1710103242.27 total 20.02 last 84.58) - open/close 2663 (time 1710103252.28 total 30.03 last 88.10) - open/close 3423 (time 1710103262.29 total 40.04 last 75.96) - open/close 4209 (time 1710103272.29 total 50.05 last 78.54) - open/close 4994 (time 1710103282.30 total 60.06 last 78.42) - open/close 5665 (time 1710103292.31 total 70.06 last 67.06) - open/close 6399 (time 1710103302.32 total 80.07 last 73.37) - open/close 7048 (time 1710103312.32 total 90.08 last 64.84) - open/close 7745 (time 1710103322.34 total 100.10 last 69.56) - open/close 8365 (time 1710103332.35 total 110.11 last 61.94) - open/close 8974 (time 1710103342.36 total 120.12 last 60.85) - open/close 9565 (time 1710103352.37 total 130.13 last 59.04) total: 10000 open/close in 138.28 seconds: 72.32 ops/second - unlinked 0 (time 1710103362 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 1243 (time 1710103452.83 total 10.01 last 124.13) - open/close 2348 (time 1710103462.83 total 20.02 last 110.47) - open/close 3408 (time 1710103472.84 total 30.02 last 105.94) - open/close 4360 (time 1710103482.84 total 40.02 last 95.18) - open/close 5343 (time 1710103492.85 total 50.03 last 98.20) - open/close 6392 (time 1710103502.85 total 60.04 last 104.86) - open/close 7344 (time 1710103512.86 total 70.04 last 95.15) - open/close 8320 (time 1710103522.86 total 80.04 last 97.58) - open/close 9245 (time 1710103532.87 total 90.05 last 92.42) total: 10000 open/close in 98.86 seconds: 101.15 ops/second - unlinked 0 (time 1710103543 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg157-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg157-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg157-server: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg157-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec mount lustre on /mnt/lustre..... Starting client: oleg157-client.virtnet: -o user_xattr,flock oleg157-server@tcp:/lustre /mnt/lustre On OST0, 10571 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg157-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server stop mds service on oleg157-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg157-server unloading modules on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2