== conf-sanity test 69: replace an OST with the same index ========================================================== 15:58:52 (1710100732) start mds service on oleg358-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg358-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg358-client.virtnet: -o user_xattr,flock oleg358-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49602 inodes available. Want 99872. rc=0 - open/close 2258 (time 1710100753.28 total 10.00 last 225.70) - open/close 3673 (time 1710100763.28 total 20.00 last 141.50) - open/close 5010 (time 1710100773.28 total 30.01 last 133.69) - open/close 6202 (time 1710100783.29 total 40.01 last 119.10) - open/close 7518 (time 1710100793.29 total 50.02 last 131.52) - open/close 9011 (time 1710100803.29 total 60.02 last 149.28) total: 10000 open/close in 67.66 seconds: 147.79 ops/second - unlinked 0 (time 1710100811 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1701 (time 1710100875.99 total 10.00 last 170.07) - open/close 3248 (time 1710100885.99 total 20.01 last 154.60) - open/close 4631 (time 1710100896.00 total 30.01 last 138.29) - open/close 5925 (time 1710100906.00 total 40.01 last 129.36) - open/close 7442 (time 1710100916.00 total 50.01 last 151.66) - open/close 8607 (time 1710100926.00 total 60.02 last 116.49) - open/close 9537 (time 1710100936.01 total 70.02 last 92.95) total: 10000 open/close in 75.38 seconds: 132.66 ops/second - unlinked 0 (time 1710100942 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1267 (time 1710101016.40 total 10.00 last 126.69) - open/close 2413 (time 1710101026.40 total 20.00 last 114.55) - open/close 3514 (time 1710101036.41 total 30.02 last 109.97) - open/close 4600 (time 1710101046.42 total 40.02 last 108.55) - open/close 5534 (time 1710101056.42 total 50.02 last 93.38) - open/close 6403 (time 1710101066.42 total 60.03 last 86.86) - open/close 7227 (time 1710101076.42 total 70.03 last 82.39) - open/close 8091 (time 1710101086.43 total 80.04 last 86.34) - open/close 8797 (time 1710101096.43 total 90.04 last 70.58) - open/close 9472 (time 1710101106.44 total 100.05 last 67.44) total: 10000 open/close in 107.70 seconds: 92.85 ops/second - unlinked 0 (time 1710101115 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1130 (time 1710101204.42 total 10.04 last 112.53) - open/close 2240 (time 1710101214.42 total 20.04 last 110.98) - open/close 3306 (time 1710101224.42 total 30.05 last 106.55) - open/close 3726 (time 1710101239.69 total 45.32 last 27.51) - open/close 4679 (time 1710101249.70 total 55.32 last 95.23) - open/close 5619 (time 1710101259.70 total 65.33 last 93.96) - open/close 6444 (time 1710101269.71 total 75.33 last 82.46) - open/close 7230 (time 1710101279.71 total 85.34 last 78.57) - open/close 8011 (time 1710101289.71 total 95.34 last 78.10) - open/close 8777 (time 1710101299.72 total 105.34 last 76.56) - open/close 9521 (time 1710101309.72 total 115.34 last 74.40) total: 10000 open/close in 122.21 seconds: 81.82 ops/second - unlinked 0 (time 1710101318 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 1191 (time 1710101403.23 total 10.00 last 119.05) - open/close 2285 (time 1710101413.24 total 20.01 last 109.32) - open/close 3347 (time 1710101423.25 total 30.02 last 106.12) - open/close 4327 (time 1710101433.26 total 40.03 last 97.93) - open/close 5243 (time 1710101443.26 total 50.03 last 91.55) - open/close 6141 (time 1710101453.27 total 60.04 last 89.77) - open/close 6988 (time 1710101463.27 total 70.04 last 84.66) - open/close 7783 (time 1710101473.28 total 80.05 last 79.45) - open/close 8539 (time 1710101483.28 total 90.05 last 75.58) - open/close 9237 (time 1710101493.29 total 100.06 last 69.72) - open/close 9917 (time 1710101503.30 total 110.07 last 67.96) total: 10000 open/close in 111.25 seconds: 89.89 ops/second - unlinked 0 (time 1710101506 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 1241 (time 1710101591.27 total 10.01 last 124.04) - open/close 2397 (time 1710101601.27 total 20.01 last 115.56) - open/close 3426 (time 1710101611.27 total 30.01 last 102.89) - open/close 4395 (time 1710101621.28 total 40.01 last 96.85) - open/close 5345 (time 1710101631.29 total 50.02 last 94.93) - open/close 6262 (time 1710101641.30 total 60.03 last 91.62) - open/close 7044 (time 1710101651.30 total 70.04 last 78.17) - open/close 7821 (time 1710101661.33 total 80.06 last 77.49) - open/close 8563 (time 1710101671.34 total 90.07 last 74.13) - open/close 9328 (time 1710101681.35 total 100.08 last 76.43) total: 10000 open/close in 110.02 seconds: 90.89 ops/second - unlinked 0 (time 1710101692 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 1275 (time 1710101776.99 total 10.00 last 127.48) - open/close 2414 (time 1710101786.99 total 20.01 last 113.84) - open/close 3458 (time 1710101796.99 total 30.01 last 104.36) - open/close 4468 (time 1710101807.00 total 40.01 last 100.96) - open/close 5438 (time 1710101817.00 total 50.02 last 96.94) - open/close 6311 (time 1710101827.01 total 60.02 last 87.29) - open/close 7135 (time 1710101837.02 total 70.04 last 82.26) - open/close 7938 (time 1710101847.03 total 80.05 last 80.23) - open/close 8154 (time 1710101860.81 total 93.83 last 15.67) - open/close 8852 (time 1710101870.82 total 103.83 last 69.76) - open/close 9534 (time 1710101880.83 total 113.85 last 68.10) total: 10000 open/close in 121.34 seconds: 82.41 ops/second - unlinked 0 (time 1710101890 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 1267 (time 1710101988.35 total 10.01 last 126.62) - open/close 2438 (time 1710101998.36 total 20.01 last 117.06) - open/close 3516 (time 1710102008.36 total 30.02 last 107.74) - open/close 4521 (time 1710102018.37 total 40.02 last 100.45) - open/close 5438 (time 1710102028.37 total 50.02 last 91.68) - open/close 6315 (time 1710102038.37 total 60.03 last 87.67) - open/close 7170 (time 1710102048.37 total 70.03 last 85.49) - open/close 7954 (time 1710102058.39 total 80.04 last 78.30) - open/close 8684 (time 1710102068.39 total 90.05 last 72.96) - open/close 9361 (time 1710102078.40 total 100.05 last 67.67) total: 10000 open/close in 109.57 seconds: 91.26 ops/second - unlinked 0 (time 1710102089 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 1336 (time 1710102172.79 total 10.00 last 133.58) - open/close 2599 (time 1710102182.80 total 20.01 last 126.23) - open/close 3813 (time 1710102192.80 total 30.01 last 121.33) - open/close 4932 (time 1710102202.80 total 40.01 last 111.89) - open/close 5945 (time 1710102212.81 total 50.02 last 101.20) - open/close 6921 (time 1710102222.81 total 60.02 last 97.58) - open/close 7819 (time 1710102232.82 total 70.03 last 89.71) - open/close 8687 (time 1710102242.83 total 80.04 last 86.75) - open/close 9473 (time 1710102252.84 total 90.05 last 78.51) total: 10000 open/close in 97.28 seconds: 102.80 ops/second - unlinked 0 (time 1710102261 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1522 (time 1710102337.93 total 10.00 last 152.13) - open/close 2883 (time 1710102347.93 total 20.01 last 136.10) - open/close 4100 (time 1710102357.94 total 30.01 last 121.62) - open/close 5260 (time 1710102367.94 total 40.02 last 115.95) - open/close 6335 (time 1710102377.95 total 50.02 last 107.43) - open/close 7365 (time 1710102387.95 total 60.03 last 102.96) - open/close 8300 (time 1710102397.95 total 70.03 last 93.50) - open/close 9164 (time 1710102407.96 total 80.04 last 86.31) - open/close 9978 (time 1710102417.97 total 90.04 last 81.37) total: 10000 open/close in 90.32 seconds: 110.71 ops/second - unlinked 0 (time 1710102419 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg358-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.158@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg358-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg358-server: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg358-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg358-client.virtnet: -o user_xattr,flock oleg358-server@tcp:/lustre /mnt/lustre On OST0, 20235 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg358-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server unloading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2