== conf-sanity test 69: replace an OST with the same index ========================================================== 18:23:28 (1746138208) start mds service on oleg250-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing load_modules_local oleg250-server: Loading modules from /home/green/git/lustre-release/lustre oleg250-server: detected 4 online CPUs by sysfs oleg250-server: MODOPTS_LIBCFS= oleg250-server: Force libcfs to create 2 CPU partitions oleg250-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg250-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg250-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg250-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg250-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1045 (time 1746138255.35 total 10.01 last 104.41) - open/close 2178 (time 1746138265.35 total 20.01 last 113.29) - open/close 3223 (time 1746138275.38 total 30.03 last 104.24) - open/close 4292 (time 1746138285.39 total 40.04 last 106.83) - open/close 5350 (time 1746138295.40 total 50.05 last 105.68) - open/close 6619 (time 1746138305.40 total 60.06 last 126.83) - open/close 7771 (time 1746138315.41 total 70.07 last 115.09) - open/close 8853 (time 1746138325.43 total 80.08 last 108.06) - open/close 9947 (time 1746138335.43 total 90.08 last 109.38) total: 10000 open/close in 90.62 seconds: 110.35 ops/second - unlinked 0 (time 1746138337 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1464 (time 1746138396.96 total 10.00 last 146.37) - open/close 2613 (time 1746138406.96 total 20.00 last 114.88) - open/close 3691 (time 1746138416.96 total 30.01 last 107.73) - open/close 4804 (time 1746138426.97 total 40.01 last 111.26) - open/close 5800 (time 1746138436.97 total 50.02 last 99.58) - open/close 6941 (time 1746138446.97 total 60.02 last 114.10) - open/close 8143 (time 1746138456.98 total 70.02 last 120.12) - open/close 9279 (time 1746138466.99 total 80.03 last 113.49) total: 10000 open/close in 86.26 seconds: 115.94 ops/second - unlinked 0 (time 1746138474 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1167 (time 1746138532.83 total 10.01 last 116.63) - open/close 2481 (time 1746138542.84 total 20.01 last 131.32) - open/close 3823 (time 1746138552.84 total 30.01 last 134.20) - open/close 5311 (time 1746138562.84 total 40.02 last 148.69) - open/close 6570 (time 1746138572.85 total 50.02 last 125.87) - open/close 8106 (time 1746138582.85 total 60.02 last 153.58) - open/close 9347 (time 1746138592.85 total 70.03 last 124.03) total: 10000 open/close in 75.10 seconds: 133.15 ops/second - unlinked 0 (time 1746138599 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1068 (time 1746138653.09 total 10.00 last 106.75) - open/close 2396 (time 1746138663.09 total 20.01 last 132.72) - open/close 3549 (time 1746138673.10 total 30.02 last 115.23) - open/close 4786 (time 1746138683.10 total 40.02 last 123.70) - open/close 6273 (time 1746138693.10 total 50.02 last 148.64) - open/close 8106 (time 1746138703.10 total 60.02 last 183.28) - open/close 9781 (time 1746138713.11 total 70.03 last 167.45) total: 10000 open/close in 71.95 seconds: 138.98 ops/second - unlinked 0 (time 1746138716 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1129 (time 1746138777.96 total 10.00 last 112.89) - open/close 2576 (time 1746138787.96 total 20.00 last 144.64) - open/close 3969 (time 1746138797.96 total 30.01 last 139.26) - open/close 5210 (time 1746138807.97 total 40.01 last 124.06) - open/close 6524 (time 1746138817.98 total 50.02 last 131.26) - open/close 7609 (time 1746138827.98 total 60.02 last 108.49) - open/close 8682 (time 1746138837.99 total 70.03 last 107.18) - open/close 9935 (time 1746138848.00 total 80.04 last 125.24) total: 10000 open/close in 80.55 seconds: 124.15 ops/second - unlinked 0 (time 1746138850 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1473 (time 1746138907.08 total 10.00 last 147.23) - open/close 3108 (time 1746138917.08 total 20.01 last 163.44) - open/close 4527 (time 1746138927.08 total 30.01 last 141.90) - open/close 5810 (time 1746138937.08 total 40.01 last 128.27) - open/close 6930 (time 1746138947.08 total 50.01 last 111.97) - open/close 8151 (time 1746138957.09 total 60.02 last 122.08) - open/close 9290 (time 1746138967.09 total 70.02 last 113.83) total: 10000 open/close in 75.93 seconds: 131.69 ops/second - unlinked 0 (time 1746138974 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1267 (time 1746139032.44 total 10.00 last 126.67) - open/close 2482 (time 1746139042.46 total 20.01 last 121.37) - open/close 3830 (time 1746139052.46 total 30.02 last 134.75) - open/close 5716 (time 1746139062.46 total 40.02 last 188.57) - open/close 7522 (time 1746139072.46 total 50.02 last 180.57) - open/close 9250 (time 1746139082.46 total 60.02 last 172.77) total: 10000 open/close in 64.37 seconds: 155.36 ops/second - unlinked 0 (time 1746139088 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 2071 (time 1746139140.35 total 10.00 last 207.09) - open/close 3840 (time 1746139150.35 total 20.00 last 176.89) - open/close 5618 (time 1746139160.35 total 30.00 last 177.75) - open/close 7393 (time 1746139170.35 total 40.00 last 177.48) - open/close 8914 (time 1746139180.35 total 50.01 last 152.08) total: 10000 open/close in 56.69 seconds: 176.41 ops/second - unlinked 0 (time 1746139188 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1445 (time 1746139236.09 total 10.00 last 144.45) - open/close 3073 (time 1746139246.09 total 20.01 last 162.76) - open/close 4356 (time 1746139256.09 total 30.01 last 128.28) - open/close 5845 (time 1746139266.10 total 40.01 last 148.87) - open/close 7518 (time 1746139276.10 total 50.01 last 167.22) - open/close 8812 (time 1746139286.10 total 60.02 last 129.40) total: 10000 open/close in 67.33 seconds: 148.53 ops/second - unlinked 0 (time 1746139294 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1441 (time 1746139343.54 total 10.00 last 144.05) - open/close 2900 (time 1746139353.54 total 20.01 last 145.86) - open/close 4253 (time 1746139363.55 total 30.01 last 135.23) - open/close 5600 (time 1746139373.56 total 40.02 last 134.57) - open/close 6755 (time 1746139383.56 total 50.02 last 115.46) - open/close 8089 (time 1746139393.56 total 60.03 last 133.38) - open/close 9624 (time 1746139403.57 total 70.03 last 153.44) total: 10000 open/close in 72.98 seconds: 137.03 ops/second - unlinked 0 (time 1746139408 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg250-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.150@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.150@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg250-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg250-server: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg250-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre On OST0, 18389 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg250-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server unloading modules via unload_modules_local on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2