== conf-sanity test 69: replace an OST with the same index ========================================================== 13:08:43 (1726852123) start mds service on oleg438-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg438-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11840 inodes available. Want 99872. rc=0 - open/close 1470 (time 1726852183.08 total 10.01 last 146.89) - open/close 2798 (time 1726852193.09 total 20.01 last 132.73) - open/close 4035 (time 1726852203.10 total 30.02 last 123.62) - open/close 5174 (time 1726852213.10 total 40.02 last 113.86) - open/close 6421 (time 1726852223.11 total 50.03 last 124.59) - open/close 7486 (time 1726852233.11 total 60.04 last 106.44) - open/close 8588 (time 1726852243.11 total 70.04 last 110.18) - open/close 9869 (time 1726852253.12 total 80.04 last 128.03) total: 10000 open/close in 81.12 seconds: 123.28 ops/second - unlinked 0 (time 1726852255 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1303 (time 1726852311.22 total 10.01 last 130.21) - open/close 2745 (time 1726852321.22 total 20.01 last 144.12) - open/close 4196 (time 1726852331.23 total 30.02 last 144.99) - open/close 5484 (time 1726852341.23 total 40.02 last 128.77) - open/close 6802 (time 1726852351.24 total 50.02 last 131.77) - open/close 8165 (time 1726852361.24 total 60.03 last 136.24) - open/close 9430 (time 1726852371.24 total 70.03 last 126.45) total: 10000 open/close in 74.63 seconds: 133.99 ops/second - unlinked 0 (time 1726852377 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1313 (time 1726852432.21 total 10.00 last 131.24) - open/close 2671 (time 1726852442.21 total 20.01 last 135.79) - open/close 3965 (time 1726852452.21 total 30.01 last 129.38) - open/close 5202 (time 1726852462.21 total 40.01 last 123.69) - open/close 6512 (time 1726852472.21 total 50.01 last 130.94) - open/close 7717 (time 1726852482.22 total 60.02 last 120.42) - open/close 8887 (time 1726852492.23 total 70.02 last 116.92) total: 10000 open/close in 78.99 seconds: 126.60 ops/second - unlinked 0 (time 1726852503 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1312 (time 1726852560.59 total 10.01 last 131.08) - open/close 2717 (time 1726852570.59 total 20.01 last 140.49) - open/close 4001 (time 1726852580.60 total 30.01 last 128.36) - open/close 5346 (time 1726852590.60 total 40.02 last 134.42) - open/close 6728 (time 1726852600.60 total 50.02 last 138.16) - open/close 7970 (time 1726852610.61 total 60.03 last 124.12) - open/close 9179 (time 1726852620.62 total 70.03 last 120.82) total: 10000 open/close in 76.07 seconds: 131.46 ops/second - unlinked 0 (time 1726852628 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1502 (time 1726852681.73 total 10.00 last 150.14) - open/close 2727 (time 1726852691.73 total 20.01 last 122.46) - open/close 4140 (time 1726852701.73 total 30.01 last 141.26) - open/close 5666 (time 1726852711.74 total 40.02 last 152.49) - open/close 7081 (time 1726852721.74 total 50.02 last 141.48) - open/close 8581 (time 1726852731.74 total 60.02 last 149.93) - open/close 9785 (time 1726852741.75 total 70.03 last 120.36) total: 10000 open/close in 71.69 seconds: 139.50 ops/second - unlinked 0 (time 1726852744 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1270 (time 1726852797.71 total 10.01 last 126.93) - open/close 2653 (time 1726852807.71 total 20.01 last 138.26) - open/close 4114 (time 1726852817.72 total 30.02 last 146.00) - open/close 5428 (time 1726852827.73 total 40.03 last 131.24) - open/close 6782 (time 1726852837.73 total 50.03 last 135.37) - open/close 8125 (time 1726852847.73 total 60.03 last 134.26) - open/close 9394 (time 1726852857.75 total 70.04 last 126.77) total: 10000 open/close in 74.56 seconds: 134.11 ops/second - unlinked 0 (time 1726852863 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1479 (time 1726852918.00 total 10.01 last 147.80) - open/close 2844 (time 1726852928.00 total 20.01 last 136.49) - open/close 3975 (time 1726852938.01 total 30.01 last 113.05) - open/close 5217 (time 1726852948.01 total 40.01 last 124.19) - open/close 6486 (time 1726852958.01 total 50.01 last 126.88) - open/close 7654 (time 1726852968.01 total 60.02 last 116.73) - open/close 8783 (time 1726852978.01 total 70.02 last 112.89) total: 10000 open/close in 80.01 seconds: 124.98 ops/second - unlinked 0 (time 1726852989 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1405 (time 1726853045.77 total 10.00 last 140.47) - open/close 2596 (time 1726853055.77 total 20.01 last 119.02) - open/close 3889 (time 1726853065.78 total 30.01 last 129.27) - open/close 5237 (time 1726853075.78 total 40.01 last 134.77) - open/close 6390 (time 1726853085.78 total 50.02 last 115.23) - open/close 7578 (time 1726853095.79 total 60.02 last 118.79) - open/close 8796 (time 1726853105.79 total 70.03 last 121.73) - open/close 9982 (time 1726853115.80 total 80.03 last 118.52) total: 10000 open/close in 80.16 seconds: 124.76 ops/second - unlinked 0 (time 1726853117 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1306 (time 1726853174.76 total 10.00 last 130.55) - open/close 2642 (time 1726853184.77 total 20.01 last 133.47) - open/close 3984 (time 1726853194.77 total 30.01 last 134.20) - open/close 5320 (time 1726853204.78 total 40.03 last 133.41) - open/close 6548 (time 1726853214.79 total 50.03 last 122.76) - open/close 7878 (time 1726853224.79 total 60.04 last 132.95) - open/close 9277 (time 1726853234.80 total 70.04 last 139.85) total: 10000 open/close in 75.77 seconds: 131.97 ops/second - unlinked 0 (time 1726853242 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1407 (time 1726853297.19 total 10.00 last 140.68) - open/close 2698 (time 1726853307.19 total 20.01 last 129.05) - open/close 4036 (time 1726853317.20 total 30.01 last 133.78) - open/close 5473 (time 1726853327.20 total 40.01 last 143.68) - open/close 6806 (time 1726853337.20 total 50.01 last 133.23) - open/close 8116 (time 1726853347.21 total 60.02 last 130.96) - open/close 9500 (time 1726853357.21 total 70.02 last 138.35) total: 10000 open/close in 74.12 seconds: 134.92 ops/second - unlinked 0 (time 1726853363 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.138@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.138@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg438-server: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg438-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg438-server: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg438-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre On OST0, 9143 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg438-server unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2