== conf-sanity test 69: replace an OST with the same index ========================================================== 16:05:52 (1710101152) start mds service on oleg115-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg115-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg115-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-OST0000 oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg115-client.virtnet: -o user_xattr,flock oleg115-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 234 (time 1710101203.22 total 13.21 last 17.71) - open/close 738 (time 1710101220.39 total 30.38 last 29.36) - open/close 1573 (time 1710101230.40 total 40.39 last 83.45) - open/close 2315 (time 1710101240.40 total 50.39 last 74.17) - open/close 2983 (time 1710101250.40 total 60.40 last 66.77) - open/close 3653 (time 1710101260.41 total 70.40 last 66.97) - open/close 4229 (time 1710101270.44 total 80.43 last 57.42) - open/close 4781 (time 1710101280.44 total 90.44 last 55.18) - open/close 5439 (time 1710101290.47 total 100.46 last 65.64) - open/close 6033 (time 1710101300.47 total 110.46 last 59.37) - open/close 6564 (time 1710101310.49 total 120.49 last 52.99) - open/close 7071 (time 1710101320.51 total 130.50 last 50.63) - open/close 7564 (time 1710101330.52 total 140.51 last 49.24) - open/close 8092 (time 1710101340.53 total 150.52 last 52.76) - open/close 8527 (time 1710101350.56 total 160.55 last 43.37) - open/close 9014 (time 1710101360.58 total 170.57 last 48.60) - open/close 9457 (time 1710101370.58 total 180.58 last 44.28) - open/close 9530 (time 1710101383.26 total 193.25 last 5.76) - open/close 9981 (time 1710101393.27 total 203.26 last 45.05) total: 10000 open/close in 203.67 seconds: 49.10 ops/second - unlinked 0 (time 1710101396 ; total 0 ; last 0) total: 10000 unlinks in 123 seconds: 81.300812 unlinks/second - open/close 929 (time 1710101531.30 total 10.01 last 92.82) - open/close 1709 (time 1710101541.30 total 20.01 last 77.99) - open/close 2483 (time 1710101551.31 total 30.02 last 77.32) - open/close 3173 (time 1710101561.31 total 40.02 last 68.98) - open/close 3890 (time 1710101571.37 total 50.08 last 71.27) - open/close 4432 (time 1710101581.38 total 60.09 last 54.15) - open/close 4998 (time 1710101591.38 total 70.09 last 56.59) - open/close 5566 (time 1710101601.40 total 80.11 last 56.71) - open/close 5988 (time 1710101620.12 total 98.83 last 22.54) - open/close 6509 (time 1710101630.13 total 108.84 last 52.08) - open/close 7036 (time 1710101640.14 total 118.85 last 52.64) - open/close 7537 (time 1710101650.14 total 128.85 last 50.06) - open/close 8007 (time 1710101660.15 total 138.86 last 46.99) - open/close 8496 (time 1710101670.15 total 148.86 last 48.89) - open/close 8977 (time 1710101680.16 total 158.87 last 48.03) - open/close 9477 (time 1710101690.18 total 168.89 last 49.91) - open/close 9912 (time 1710101700.18 total 178.89 last 43.49) total: 10000 open/close in 180.42 seconds: 55.43 ops/second - unlinked 0 (time 1710101704 ; total 0 ; last 0) total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second - open/close 746 (time 1710101826.00 total 10.01 last 74.55) - open/close 1500 (time 1710101836.01 total 20.01 last 75.35) - open/close 2296 (time 1710101846.02 total 30.03 last 79.48) - open/close 3068 (time 1710101856.04 total 40.05 last 77.06) - open/close 3733 (time 1710101866.05 total 50.05 last 66.45) - open/close 4363 (time 1710101876.06 total 60.07 last 62.92) - open/close 4945 (time 1710101886.08 total 70.09 last 58.08) - open/close 5586 (time 1710101896.09 total 80.10 last 64.04) - open/close 6175 (time 1710101906.10 total 90.10 last 58.88) - open/close 6716 (time 1710101916.10 total 100.11 last 54.06) - open/close 7248 (time 1710101926.11 total 110.11 last 53.18) - open/close 7727 (time 1710101936.11 total 120.12 last 47.87) - open/close 8150 (time 1710101955.49 total 139.49 last 21.83) - open/close 8623 (time 1710101965.52 total 149.52 last 47.16) - open/close 9064 (time 1710101975.53 total 159.53 last 44.06) - open/close 9254 (time 1710101990.85 total 174.86 last 12.40) - open/close 9726 (time 1710102000.86 total 184.87 last 47.14) total: 10000 open/close in 191.14 seconds: 52.32 ops/second - unlinked 0 (time 1710102009 ; total 0 ; last 0) total: 10000 unlinks in 123 seconds: 81.300812 unlinks/second - open/close 879 (time 1710102145.71 total 10.00 last 87.87) - open/close 1691 (time 1710102155.72 total 20.01 last 81.15) - open/close 2354 (time 1710102165.73 total 30.03 last 66.18) - open/close 3017 (time 1710102175.74 total 40.03 last 66.27) - open/close 3712 (time 1710102185.75 total 50.05 last 69.38) - open/close 4402 (time 1710102195.76 total 60.05 last 68.98) - open/close 5013 (time 1710102205.78 total 70.08 last 60.95) - open/close 5583 (time 1710102215.79 total 80.08 last 56.97) - open/close 6148 (time 1710102225.80 total 90.10 last 56.42) - open/close 6688 (time 1710102235.80 total 100.10 last 53.98) - open/close 7156 (time 1710102245.81 total 110.10 last 46.78) - open/close 7636 (time 1710102255.81 total 120.11 last 47.98) - open/close 8117 (time 1710102265.84 total 130.13 last 47.97) - open/close 8423 (time 1710102283.07 total 147.36 last 17.76) - open/close 8932 (time 1710102293.07 total 157.37 last 50.88) - open/close 9449 (time 1710102303.09 total 167.39 last 51.58) - open/close 9966 (time 1710102313.11 total 177.40 last 51.64) total: 10000 open/close in 177.97 seconds: 56.19 ops/second - unlinked 0 (time 1710102316 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 984 (time 1710102435.65 total 10.00 last 98.40) - open/close 1802 (time 1710102445.65 total 20.00 last 81.79) - open/close 2550 (time 1710102455.66 total 30.01 last 74.74) - open/close 3238 (time 1710102465.67 total 40.02 last 68.73) - open/close 3462 (time 1710102479.43 total 53.77 last 16.29) - open/close 4150 (time 1710102489.46 total 63.81 last 68.58) - open/close 4366 (time 1710102504.30 total 78.65 last 14.55) - open/close 4925 (time 1710102514.31 total 88.65 last 55.88) - open/close 5501 (time 1710102524.31 total 98.66 last 57.60) - open/close 6096 (time 1710102534.33 total 108.68 last 59.38) - open/close 6576 (time 1710102544.33 total 118.68 last 48.00) - open/close 7046 (time 1710102554.34 total 128.69 last 46.93) - open/close 7588 (time 1710102564.34 total 138.69 last 54.20) - open/close 8062 (time 1710102574.34 total 148.69 last 47.39) - open/close 8596 (time 1710102584.35 total 158.70 last 53.36) - open/close 9099 (time 1710102594.35 total 168.70 last 50.30) - open/close 9577 (time 1710102604.36 total 178.71 last 47.74) total: 10000 open/close in 188.18 seconds: 53.14 ops/second - unlinked 0 (time 1710102616 ; total 0 ; last 0) total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second - open/close 838 (time 1710102742.58 total 10.01 last 83.71) - open/close 1652 (time 1710102752.64 total 20.07 last 80.92) - open/close 2451 (time 1710102762.64 total 30.08 last 79.86) - open/close 3178 (time 1710102772.65 total 40.08 last 72.67) - open/close 3839 (time 1710102782.65 total 50.09 last 66.07) - open/close 4503 (time 1710102792.67 total 60.11 last 66.27) - open/close 5123 (time 1710102802.68 total 70.11 last 61.95) - open/close 5690 (time 1710102812.69 total 80.12 last 56.66) - open/close 6238 (time 1710102822.69 total 90.12 last 54.78) - open/close 6784 (time 1710102832.69 total 100.13 last 54.59) - open/close 7288 (time 1710102842.71 total 110.14 last 50.34) - open/close 7781 (time 1710102852.74 total 120.17 last 49.13) - open/close 8221 (time 1710102872.97 total 140.40 last 21.75) - open/close 8708 (time 1710102882.98 total 150.41 last 48.64) - open/close 9148 (time 1710102892.98 total 160.42 last 43.98) - open/close 9613 (time 1710102902.99 total 170.42 last 46.48) total: 10000 open/close in 179.08 seconds: 55.84 ops/second - unlinked 0 (time 1710102914 ; total 0 ; last 0) total: 10000 unlinks in 120 seconds: 83.333336 unlinks/second - open/close 844 (time 1710103047.05 total 10.01 last 84.33) - open/close 1654 (time 1710103057.06 total 20.02 last 80.94) - open/close 2394 (time 1710103067.06 total 30.02 last 73.99) - open/close 3108 (time 1710103077.08 total 40.03 last 71.27) - open/close 3810 (time 1710103087.09 total 50.05 last 70.10) - open/close 4511 (time 1710103097.10 total 60.05 last 70.07) - open/close 5125 (time 1710103107.10 total 70.06 last 61.36) - open/close 5749 (time 1710103117.11 total 80.07 last 62.35) - open/close 5751 (time 1710103128.14 total 91.10 last 0.18) - open/close 6292 (time 1710103138.15 total 101.11 last 54.04) - open/close 6834 (time 1710103148.16 total 111.11 last 54.18) - open/close 7206 (time 1710103165.35 total 128.31 last 21.63) - open/close 7395 (time 1710103180.43 total 143.39 last 12.54) - open/close 8038 (time 1710103190.46 total 153.41 last 64.14) - open/close 8596 (time 1710103200.46 total 163.42 last 55.77) - open/close 9115 (time 1710103210.46 total 173.42 last 51.88) - open/close 9683 (time 1710103220.48 total 183.44 last 56.71) total: 10000 open/close in 189.94 seconds: 52.65 ops/second - unlinked 0 (time 1710103229 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 1106 (time 1710103333.67 total 10.01 last 110.51) - open/close 2039 (time 1710103343.68 total 20.01 last 93.25) - open/close 2970 (time 1710103353.68 total 30.02 last 93.08) - open/close 3778 (time 1710103363.69 total 40.03 last 80.70) - open/close 4630 (time 1710103373.69 total 50.03 last 85.19) - open/close 5437 (time 1710103383.70 total 60.03 last 80.67) - open/close 6251 (time 1710103393.70 total 70.04 last 81.35) - open/close 7005 (time 1710103403.70 total 80.04 last 75.40) - open/close 7707 (time 1710103413.71 total 90.04 last 70.18) - open/close 8455 (time 1710103423.71 total 100.05 last 74.74) - open/close 9159 (time 1710103433.71 total 110.05 last 70.39) - open/close 9827 (time 1710103443.73 total 120.06 last 66.72) total: 10000 open/close in 122.93 seconds: 81.35 ops/second - unlinked 0 (time 1710103448 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 1459 (time 1710103530.29 total 10.00 last 145.85) - open/close 2795 (time 1710103540.30 total 20.01 last 133.47) - open/close 4005 (time 1710103550.30 total 30.01 last 121.00) - open/close 5184 (time 1710103560.31 total 40.02 last 117.86) - open/close 6379 (time 1710103570.31 total 50.02 last 119.44) - open/close 7523 (time 1710103580.32 total 60.03 last 114.33) - open/close 8663 (time 1710103590.32 total 70.03 last 113.95) - open/close 9794 (time 1710103600.32 total 80.03 last 113.08) total: 10000 open/close in 81.86 seconds: 122.16 ops/second - unlinked 0 (time 1710103603 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 2492 (time 1710103657.08 total 10.00 last 249.15) - open/close 4555 (time 1710103667.08 total 20.01 last 206.21) - open/close 6506 (time 1710103677.09 total 30.01 last 195.07) - open/close 8379 (time 1710103687.09 total 40.01 last 187.23) total: 10000 open/close in 49.93 seconds: 200.29 ops/second - unlinked 0 (time 1710103697 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg115-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg115-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.115@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg115-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-OST0000 oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg115-server: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg115-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec oleg115-server: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg115-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg115-client.virtnet: -o user_xattr,flock oleg115-server@tcp:/lustre /mnt/lustre On OST0, 20269 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg115-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg115-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server stop mds service on oleg115-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg115-server stop mds service on oleg115-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg115-server unloading modules on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2