== conf-sanity test 69: replace an OST with the same index ========================================================== 16:05:43 (1710101143) start mds service on oleg139-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg139-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg139-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock oleg139-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 1065 (time 1710101191.47 total 10.01 last 106.44) - open/close 1979 (time 1710101201.47 total 20.01 last 91.36) - open/close 2834 (time 1710101211.48 total 30.02 last 85.45) - open/close 3558 (time 1710101221.49 total 40.03 last 72.33) - open/close 4310 (time 1710101231.49 total 50.03 last 75.20) - open/close 5068 (time 1710101241.51 total 60.04 last 75.67) - open/close 5706 (time 1710101251.52 total 70.05 last 63.73) - open/close 6362 (time 1710101261.54 total 80.08 last 65.45) - open/close 6938 (time 1710101271.55 total 90.09 last 57.52) - open/close 7497 (time 1710101281.56 total 100.10 last 55.86) - open/close 8134 (time 1710101291.56 total 110.10 last 63.68) - open/close 8701 (time 1710101301.57 total 120.11 last 56.66) - open/close 9200 (time 1710101311.58 total 130.12 last 49.83) - open/close 9706 (time 1710101321.60 total 140.13 last 50.54) total: 10000 open/close in 147.51 seconds: 67.79 ops/second - unlinked 0 (time 1710101331 ; total 0 ; last 0) total: 10000 unlinks in 133 seconds: 75.187973 unlinks/second - open/close 949 (time 1710101477.62 total 10.00 last 94.87) - open/close 1674 (time 1710101487.62 total 20.00 last 72.50) - open/close 2450 (time 1710101497.69 total 30.08 last 77.04) - open/close 3147 (time 1710101507.72 total 40.10 last 69.50) - open/close 3868 (time 1710101517.73 total 50.11 last 72.08) - open/close 4538 (time 1710101527.74 total 60.12 last 66.91) - open/close 5079 (time 1710101537.75 total 70.13 last 54.05) - open/close 5728 (time 1710101547.77 total 80.15 last 64.78) - open/close 6386 (time 1710101557.77 total 90.15 last 65.76) - open/close 7006 (time 1710101567.78 total 100.16 last 61.98) - open/close 7539 (time 1710101577.80 total 110.18 last 53.17) - open/close 8057 (time 1710101587.81 total 120.20 last 51.72) - open/close 8578 (time 1710101597.83 total 130.21 last 52.02) - open/close 8802 (time 1710101613.37 total 145.75 last 14.41) - open/close 9261 (time 1710101623.38 total 155.76 last 45.87) - open/close 9275 (time 1710101634.81 total 167.19 last 1.22) - open/close 9787 (time 1710101644.83 total 177.21 last 51.12) total: 10000 open/close in 181.72 seconds: 55.03 ops/second - unlinked 0 (time 1710101652 ; total 0 ; last 0) total: 10000 unlinks in 120 seconds: 83.333336 unlinks/second - open/close 973 (time 1710101785.07 total 10.00 last 97.29) - open/close 1912 (time 1710101795.07 total 20.00 last 93.88) - open/close 2765 (time 1710101805.11 total 30.05 last 84.93) - open/close 3490 (time 1710101815.11 total 40.05 last 72.49) - open/close 4153 (time 1710101825.11 total 50.05 last 66.30) - open/close 4798 (time 1710101835.12 total 60.06 last 64.44) - open/close 4871 (time 1710101847.20 total 72.14 last 6.04) - open/close 5536 (time 1710101857.21 total 82.15 last 66.44) - open/close 6220 (time 1710101867.22 total 92.15 last 68.37) - open/close 6884 (time 1710101877.23 total 102.16 last 66.32) - open/close 7340 (time 1710101887.23 total 112.17 last 45.57) - open/close 7955 (time 1710101897.24 total 122.17 last 61.49) - open/close 8548 (time 1710101907.24 total 132.18 last 59.26) - open/close 9077 (time 1710101917.25 total 142.18 last 52.86) - open/close 9610 (time 1710101927.26 total 152.19 last 53.26) total: 10000 open/close in 160.22 seconds: 62.41 ops/second - unlinked 0 (time 1710101937 ; total 0 ; last 0) total: 10000 unlinks in 118 seconds: 84.745766 unlinks/second - open/close 260 (time 1710102071.63 total 13.27 last 19.59) - open/close 1166 (time 1710102081.63 total 23.27 last 90.59) - open/close 2032 (time 1710102091.63 total 33.28 last 86.54) - open/close 2865 (time 1710102101.64 total 43.28 last 83.27) - open/close 3560 (time 1710102111.64 total 53.28 last 69.49) - open/close 4307 (time 1710102121.64 total 63.29 last 74.67) - open/close 5032 (time 1710102131.65 total 73.29 last 72.45) - open/close 5690 (time 1710102141.65 total 83.30 last 65.79) - open/close 6321 (time 1710102151.66 total 93.31 last 63.01) - open/close 6881 (time 1710102161.66 total 103.31 last 56.00) - open/close 7439 (time 1710102171.67 total 113.31 last 55.78) - open/close 7958 (time 1710102191.59 total 133.24 last 26.05) - open/close 8561 (time 1710102201.59 total 143.24 last 60.29) - open/close 9112 (time 1710102211.59 total 153.24 last 55.10) - open/close 9515 (time 1710102221.60 total 163.25 last 40.27) total: 10000 open/close in 172.80 seconds: 57.87 ops/second - unlinked 0 (time 1710102234 ; total 0 ; last 0) total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second - open/close 1038 (time 1710102352.85 total 10.00 last 103.76) - open/close 1948 (time 1710102362.86 total 20.01 last 90.93) - open/close 2890 (time 1710102372.88 total 30.03 last 94.01) - open/close 3666 (time 1710102382.91 total 40.07 last 77.31) - open/close 4382 (time 1710102392.92 total 50.08 last 71.54) - open/close 5039 (time 1710102402.94 total 60.09 last 65.61) - open/close 5654 (time 1710102412.96 total 70.12 last 61.34) - open/close 6309 (time 1710102422.98 total 80.13 last 65.41) - open/close 6986 (time 1710102432.99 total 90.14 last 67.63) - open/close 7625 (time 1710102442.99 total 100.14 last 63.90) - open/close 8162 (time 1710102452.99 total 110.14 last 53.69) - open/close 8716 (time 1710102463.01 total 120.17 last 55.27) - open/close 9300 (time 1710102473.02 total 130.18 last 58.35) - open/close 9796 (time 1710102483.03 total 140.18 last 49.56) total: 10000 open/close in 143.79 seconds: 69.54 ops/second - unlinked 0 (time 1710102489 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 827 (time 1710102595.51 total 10.00 last 82.70) - open/close 1581 (time 1710102605.51 total 20.00 last 75.37) - open/close 2400 (time 1710102615.52 total 30.01 last 81.81) - open/close 3225 (time 1710102625.52 total 40.01 last 82.49) - open/close 3518 (time 1710102640.02 total 54.51 last 20.21) - open/close 4238 (time 1710102650.04 total 64.53 last 71.89) - open/close 4861 (time 1710102660.04 total 74.54 last 62.27) - open/close 5510 (time 1710102670.04 total 84.54 last 64.90) - open/close 5644 (time 1710102683.16 total 97.66 last 10.21) - open/close 6263 (time 1710102693.17 total 107.67 last 61.84) - open/close 6828 (time 1710102703.18 total 117.67 last 56.49) - open/close 7360 (time 1710102723.98 total 138.47 last 25.57) - open/close 7983 (time 1710102734.00 total 148.49 last 62.21) - open/close 8443 (time 1710102744.00 total 158.50 last 45.97) - open/close 8984 (time 1710102754.01 total 168.51 last 54.04) - open/close 9553 (time 1710102764.02 total 178.51 last 56.87) total: 10000 open/close in 187.38 seconds: 53.37 ops/second - unlinked 0 (time 1710102775 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 1000 (time 1710102880.76 total 10.02 last 99.81) - open/close 1706 (time 1710102890.77 total 20.03 last 70.50) - open/close 2537 (time 1710102900.78 total 30.04 last 83.08) - open/close 3114 (time 1710102919.26 total 48.52 last 31.22) - open/close 3798 (time 1710102929.26 total 58.52 last 68.36) - open/close 4551 (time 1710102939.27 total 68.53 last 75.27) - open/close 5235 (time 1710102949.27 total 78.53 last 68.39) - open/close 5894 (time 1710102959.31 total 88.57 last 65.63) - open/close 6516 (time 1710102969.32 total 98.58 last 62.16) - open/close 7113 (time 1710102979.33 total 108.59 last 59.64) - open/close 7676 (time 1710102989.34 total 118.60 last 56.23) - open/close 8167 (time 1710102999.35 total 128.61 last 49.06) - open/close 8706 (time 1710103009.38 total 138.64 last 53.75) - open/close 9213 (time 1710103019.38 total 148.64 last 50.66) - open/close 9749 (time 1710103029.39 total 158.65 last 53.59) total: 10000 open/close in 163.24 seconds: 61.26 ops/second - unlinked 0 (time 1710103036 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 981 (time 1710103140.71 total 10.01 last 98.01) - open/close 1917 (time 1710103150.72 total 20.01 last 93.58) - open/close 2483 (time 1710103167.96 total 37.26 last 32.82) - open/close 3202 (time 1710103188.05 total 57.35 last 35.79) - open/close 3954 (time 1710103198.06 total 67.35 last 75.19) - open/close 4728 (time 1710103208.06 total 77.36 last 77.34) - open/close 5483 (time 1710103218.08 total 87.37 last 75.39) - open/close 6192 (time 1710103228.09 total 97.39 last 70.80) - open/close 6808 (time 1710103238.10 total 107.40 last 61.56) - open/close 7531 (time 1710103248.11 total 117.41 last 72.22) - open/close 8005 (time 1710103267.22 total 136.51 last 24.81) - open/close 8700 (time 1710103277.23 total 146.53 last 69.38) - open/close 9315 (time 1710103287.25 total 156.55 last 61.38) - open/close 9876 (time 1710103297.25 total 166.55 last 56.09) total: 10000 open/close in 168.55 seconds: 59.33 ops/second - unlinked 0 (time 1710103301 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1283 (time 1710103390.28 total 10.00 last 128.26) - open/close 2469 (time 1710103400.29 total 20.01 last 118.58) - open/close 3303 (time 1710103420.20 total 39.92 last 41.88) - open/close 4293 (time 1710103430.21 total 49.92 last 98.95) - open/close 5238 (time 1710103440.21 total 59.93 last 94.47) - open/close 6168 (time 1710103450.21 total 69.93 last 92.98) - open/close 7018 (time 1710103460.22 total 79.94 last 84.92) - open/close 7878 (time 1710103470.23 total 89.95 last 85.93) - open/close 8695 (time 1710103480.23 total 99.95 last 81.70) - open/close 9549 (time 1710103490.23 total 109.95 last 85.39) total: 10000 open/close in 115.11 seconds: 86.87 ops/second - unlinked 0 (time 1710103496 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1682 (time 1710103562.91 total 10.00 last 168.18) - open/close 3318 (time 1710103572.91 total 20.01 last 163.51) - open/close 4857 (time 1710103582.92 total 30.01 last 153.80) - open/close 6332 (time 1710103592.92 total 40.02 last 147.42) - open/close 7720 (time 1710103602.93 total 50.02 last 138.75) - open/close 9031 (time 1710103612.93 total 60.03 last 131.00) total: 10000 open/close in 67.52 seconds: 148.10 ops/second - unlinked 0 (time 1710103621 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.139@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg139-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg139-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 17 sec oleg139-server: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg139-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock oleg139-server@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg139-server unloading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2