== conf-sanity test 69: replace an OST with the same index ========================================================== 16:03:10 (1710100990) start mds service on oleg437-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg437-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg437-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock oleg437-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 1221 (time 1710101029.12 total 10.01 last 122.03) - open/close 2327 (time 1710101039.13 total 20.01 last 110.54) - open/close 3272 (time 1710101049.14 total 30.02 last 94.39) - open/close 4124 (time 1710101059.14 total 40.03 last 85.17) - open/close 5064 (time 1710101069.15 total 50.03 last 93.94) - open/close 5856 (time 1710101079.15 total 60.04 last 79.17) - open/close 6635 (time 1710101089.19 total 70.07 last 77.63) - open/close 6921 (time 1710101104.25 total 85.13 last 18.99) - open/close 7667 (time 1710101114.26 total 95.14 last 74.49) - open/close 8393 (time 1710101124.27 total 105.16 last 72.52) - open/close 9105 (time 1710101134.38 total 115.27 last 70.40) - open/close 9784 (time 1710101144.40 total 125.28 last 67.81) total: 10000 open/close in 128.09 seconds: 78.07 ops/second - unlinked 0 (time 1710101149 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 1141 (time 1710101247.04 total 10.01 last 114.04) - open/close 2167 (time 1710101257.06 total 20.02 last 102.43) - open/close 3060 (time 1710101267.07 total 30.03 last 89.19) - open/close 3939 (time 1710101277.08 total 40.04 last 87.85) - open/close 4757 (time 1710101287.08 total 50.05 last 81.75) - open/close 5586 (time 1710101297.09 total 60.06 last 82.80) - open/close 6370 (time 1710101307.10 total 70.06 last 78.34) - open/close 7099 (time 1710101317.11 total 80.07 last 72.84) - open/close 7810 (time 1710101327.12 total 90.08 last 71.06) - open/close 8545 (time 1710101337.12 total 100.09 last 73.45) - open/close 9196 (time 1710101347.13 total 110.10 last 65.02) - open/close 9862 (time 1710101357.14 total 120.10 last 66.56) total: 10000 open/close in 122.09 seconds: 81.90 ops/second - unlinked 0 (time 1710101361 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 1253 (time 1710101459.41 total 10.01 last 125.19) - open/close 2332 (time 1710101469.41 total 20.02 last 107.82) - open/close 3333 (time 1710101479.42 total 30.02 last 100.06) - open/close 4243 (time 1710101489.42 total 40.02 last 91.00) - open/close 5077 (time 1710101499.42 total 50.02 last 83.39) - open/close 5877 (time 1710101509.44 total 60.04 last 79.85) - open/close 6659 (time 1710101519.46 total 70.06 last 78.06) - open/close 7388 (time 1710101529.46 total 80.06 last 72.86) - open/close 8083 (time 1710101539.47 total 90.08 last 69.41) - open/close 8720 (time 1710101549.49 total 100.09 last 63.63) - open/close 9400 (time 1710101559.49 total 110.09 last 67.99) total: 10000 open/close in 119.25 seconds: 83.86 ops/second - unlinked 0 (time 1710101570 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 1207 (time 1710101657.93 total 10.02 last 120.44) - open/close 2281 (time 1710101667.94 total 20.02 last 107.36) - open/close 3327 (time 1710101677.94 total 30.03 last 104.52) - open/close 4245 (time 1710101687.95 total 40.04 last 91.78) - open/close 5189 (time 1710101697.96 total 50.04 last 94.31) - open/close 5966 (time 1710101707.96 total 60.05 last 77.66) - open/close 6845 (time 1710101717.98 total 70.07 last 87.70) - open/close 7544 (time 1710101727.99 total 80.08 last 69.83) - open/close 8219 (time 1710101738.00 total 90.09 last 67.47) - open/close 8940 (time 1710101748.01 total 100.10 last 72.01) - open/close 9606 (time 1710101758.03 total 110.11 last 66.50) total: 10000 open/close in 116.54 seconds: 85.81 ops/second - unlinked 0 (time 1710101766 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 1215 (time 1710101863.96 total 10.01 last 121.42) - open/close 2431 (time 1710101873.97 total 20.01 last 121.52) - open/close 3463 (time 1710101883.98 total 30.02 last 103.08) - open/close 4352 (time 1710101893.99 total 40.04 last 88.81) - open/close 5208 (time 1710101904.00 total 50.04 last 85.52) - open/close 5987 (time 1710101914.00 total 60.05 last 77.85) - open/close 6735 (time 1710101924.01 total 70.06 last 74.74) - open/close 7510 (time 1710101934.02 total 80.06 last 77.45) - open/close 8213 (time 1710101944.03 total 90.08 last 70.22) - open/close 8897 (time 1710101954.04 total 100.08 last 68.36) - open/close 9580 (time 1710101964.08 total 110.13 last 67.99) total: 10000 open/close in 115.85 seconds: 86.32 ops/second - unlinked 0 (time 1710101971 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 1214 (time 1710102058.14 total 10.01 last 121.26) - open/close 1560 (time 1710102072.31 total 24.18 last 24.41) - open/close 2596 (time 1710102082.32 total 34.20 last 103.47) - open/close 3592 (time 1710102092.33 total 44.20 last 99.56) - open/close 4528 (time 1710102102.33 total 54.21 last 93.54) - open/close 5386 (time 1710102112.34 total 64.21 last 85.76) - open/close 6278 (time 1710102122.34 total 74.21 last 89.19) - open/close 7129 (time 1710102132.35 total 84.22 last 85.05) - open/close 7884 (time 1710102142.36 total 94.23 last 75.41) - open/close 8648 (time 1710102152.36 total 104.23 last 76.38) - open/close 9298 (time 1710102162.37 total 114.24 last 64.93) total: 10000 open/close in 124.01 seconds: 80.64 ops/second - unlinked 0 (time 1710102173 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 1253 (time 1710102260.50 total 10.00 last 125.25) - open/close 1679 (time 1710102275.82 total 25.33 last 27.80) - open/close 2737 (time 1710102285.82 total 35.33 last 105.77) - open/close 3632 (time 1710102295.83 total 45.33 last 89.45) - open/close 4534 (time 1710102305.84 total 55.35 last 90.06) - open/close 5380 (time 1710102315.85 total 65.36 last 84.54) - open/close 6194 (time 1710102325.85 total 75.36 last 81.39) - open/close 6996 (time 1710102335.86 total 85.37 last 80.11) - open/close 7777 (time 1710102345.87 total 95.38 last 78.02) - open/close 8503 (time 1710102355.87 total 105.38 last 72.57) - open/close 8746 (time 1710102370.24 total 119.75 last 16.92) - open/close 9440 (time 1710102380.24 total 129.75 last 69.40) total: 10000 open/close in 137.55 seconds: 72.70 ops/second - unlinked 0 (time 1710102390 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1277 (time 1710102479.07 total 10.00 last 127.65) - open/close 2124 (time 1710102497.32 total 28.25 last 46.41) - open/close 3155 (time 1710102507.32 total 38.26 last 103.08) - open/close 4168 (time 1710102517.33 total 48.27 last 101.22) - open/close 5085 (time 1710102527.34 total 58.28 last 91.58) - open/close 5871 (time 1710102537.35 total 68.28 last 78.56) - open/close 6664 (time 1710102547.35 total 78.29 last 79.28) - open/close 7436 (time 1710102557.36 total 88.30 last 77.10) - open/close 8266 (time 1710102567.37 total 98.30 last 82.97) - open/close 9044 (time 1710102577.38 total 108.32 last 77.68) - open/close 9805 (time 1710102587.39 total 118.33 last 76.02) total: 10000 open/close in 121.36 seconds: 82.40 ops/second - unlinked 0 (time 1710102592 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 1288 (time 1710102674.80 total 10.00 last 128.76) - open/close 2535 (time 1710102684.81 total 20.01 last 124.58) - open/close 3518 (time 1710102694.82 total 30.02 last 98.22) - open/close 4581 (time 1710102704.83 total 40.03 last 106.22) - open/close 5546 (time 1710102714.83 total 50.03 last 96.49) - open/close 6391 (time 1710102724.83 total 60.04 last 84.45) - open/close 7286 (time 1710102734.84 total 70.04 last 89.43) - open/close 7332 (time 1710102746.37 total 81.57 last 3.99) - open/close 7688 (time 1710102761.73 total 96.93 last 23.18) - open/close 8593 (time 1710102771.73 total 106.93 last 90.49) - open/close 9467 (time 1710102781.74 total 116.95 last 87.26) total: 10000 open/close in 123.17 seconds: 81.19 ops/second - unlinked 0 (time 1710102789 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 1356 (time 1710102860.57 total 10.00 last 135.59) - open/close 2631 (time 1710102870.57 total 20.00 last 127.48) - open/close 3911 (time 1710102880.57 total 30.00 last 127.97) - open/close 5068 (time 1710102890.58 total 40.01 last 115.60) - open/close 6334 (time 1710102900.58 total 50.02 last 126.55) - open/close 7554 (time 1710102910.59 total 60.02 last 121.95) - open/close 8692 (time 1710102920.59 total 70.03 last 113.74) - open/close 9789 (time 1710102930.59 total 80.03 last 109.69) total: 10000 open/close in 82.07 seconds: 121.85 ops/second - unlinked 0 (time 1710102933 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.137@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg437-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg437-server: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg437-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg437-server: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg437-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock oleg437-server@tcp:/lustre /mnt/lustre On OST0, 20269 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg437-server unloading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2