== conf-sanity test 69: replace an OST with the same index ========================================================== 13:52:30 (1776189150) start mds service on oleg321-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing load_modules_local oleg321-server: Loading modules from /home/green/git/lustre-release/lustre oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg321-server: detected 4 online CPUs by sysfs oleg321-server: Force libcfs to create 2 CPU partitions oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg321-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock oleg321-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 687 (time 1776189233.16 total 10.01 last 68.63) - open/close 1368 (time 1776189243.18 total 20.03 last 68.00) - open/close 2020 (time 1776189253.19 total 30.04 last 65.10) - open/close 2689 (time 1776189263.19 total 40.04 last 66.90) - open/close 3405 (time 1776189273.21 total 50.06 last 71.49) - open/close 4178 (time 1776189283.22 total 60.07 last 77.23) - open/close 4935 (time 1776189293.22 total 70.07 last 75.69) - open/close 5735 (time 1776189303.23 total 80.08 last 79.90) - open/close 6401 (time 1776189313.24 total 90.09 last 66.54) - open/close 7074 (time 1776189323.24 total 100.09 last 67.26) - open/close 7768 (time 1776189333.25 total 110.10 last 69.36) - open/close 8433 (time 1776189343.25 total 120.10 last 66.48) - open/close 9159 (time 1776189353.27 total 130.12 last 72.50) - open/close 9835 (time 1776189363.27 total 140.12 last 67.58) total: 10000 open/close in 142.59 seconds: 70.13 ops/second - unlinked 0 (time 1776189369 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 720 (time 1776189465.15 total 10.01 last 71.95) - open/close 1376 (time 1776189475.15 total 20.01 last 65.59) - open/close 2054 (time 1776189485.16 total 30.02 last 67.70) - open/close 2733 (time 1776189495.18 total 40.04 last 67.77) - open/close 3418 (time 1776189505.20 total 50.06 last 68.40) - open/close 4106 (time 1776189515.21 total 60.07 last 68.72) - open/close 4746 (time 1776189525.22 total 70.08 last 63.93) - open/close 5380 (time 1776189535.22 total 80.08 last 63.40) - open/close 6051 (time 1776189545.22 total 90.08 last 67.07) - open/close 6719 (time 1776189555.23 total 100.09 last 66.78) - open/close 7354 (time 1776189565.23 total 110.09 last 63.46) - open/close 7989 (time 1776189575.24 total 120.10 last 63.44) - open/close 8675 (time 1776189585.25 total 130.11 last 68.57) - open/close 9283 (time 1776189595.25 total 140.11 last 60.77) - open/close 9899 (time 1776189605.26 total 150.12 last 61.54) total: 10000 open/close in 151.56 seconds: 65.98 ops/second - unlinked 0 (time 1776189610 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 631 (time 1776189716.46 total 10.03 last 62.94) - open/close 1289 (time 1776189726.47 total 20.03 last 65.74) - open/close 1838 (time 1776189736.47 total 30.04 last 54.87) - open/close 2479 (time 1776189746.48 total 40.05 last 64.06) - open/close 3035 (time 1776189756.50 total 50.07 last 55.49) - open/close 3733 (time 1776189766.50 total 60.07 last 69.77) - open/close 4371 (time 1776189776.52 total 70.09 last 63.66) - open/close 4998 (time 1776189786.54 total 80.11 last 62.62) - open/close 5605 (time 1776189796.54 total 90.11 last 60.65) - open/close 6260 (time 1776189806.57 total 100.14 last 65.34) - open/close 6895 (time 1776189816.58 total 110.15 last 63.44) - open/close 7483 (time 1776189826.60 total 120.17 last 58.68) - open/close 8108 (time 1776189836.62 total 130.19 last 62.38) - open/close 8734 (time 1776189846.63 total 140.19 last 62.55) - open/close 9374 (time 1776189856.63 total 150.20 last 63.99) total: 10000 open/close in 159.51 seconds: 62.69 ops/second - unlinked 0 (time 1776189869 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 603 (time 1776189975.09 total 10.00 last 60.28) - open/close 1281 (time 1776189985.09 total 20.01 last 67.79) - open/close 1935 (time 1776189995.10 total 30.02 last 65.33) - open/close 2689 (time 1776190005.11 total 40.02 last 75.33) - open/close 3323 (time 1776190015.11 total 50.03 last 63.38) - open/close 3913 (time 1776190025.12 total 60.04 last 58.94) - open/close 4593 (time 1776190035.13 total 70.04 last 67.96) - open/close 5288 (time 1776190045.13 total 80.05 last 69.45) - open/close 5975 (time 1776190055.14 total 90.06 last 68.62) - open/close 6661 (time 1776190065.15 total 100.06 last 68.59) - open/close 7383 (time 1776190075.16 total 110.07 last 72.13) - open/close 8104 (time 1776190085.17 total 120.09 last 71.99) - open/close 8801 (time 1776190095.19 total 130.10 last 69.60) - open/close 9489 (time 1776190105.19 total 140.11 last 68.78) total: 10000 open/close in 147.83 seconds: 67.64 ops/second - unlinked 0 (time 1776190116 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 765 (time 1776190210.38 total 10.01 last 76.42) - open/close 1563 (time 1776190220.38 total 20.01 last 79.78) - open/close 2394 (time 1776190230.39 total 30.02 last 83.01) - open/close 3206 (time 1776190240.39 total 40.03 last 81.18) - open/close 4028 (time 1776190250.40 total 50.03 last 82.15) - open/close 4909 (time 1776190260.41 total 60.04 last 88.05) - open/close 5631 (time 1776190270.42 total 70.05 last 72.12) - open/close 6307 (time 1776190280.42 total 80.06 last 67.55) - open/close 6890 (time 1776190290.45 total 90.08 last 58.15) - open/close 7527 (time 1776190300.45 total 100.08 last 63.70) - open/close 8245 (time 1776190310.45 total 110.08 last 71.78) - open/close 8929 (time 1776190320.45 total 120.09 last 68.40) - open/close 9586 (time 1776190330.47 total 130.10 last 65.61) total: 10000 open/close in 135.84 seconds: 73.61 ops/second - unlinked 0 (time 1776190339 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 575 (time 1776190423.78 total 10.01 last 57.47) - open/close 1186 (time 1776190433.80 total 20.02 last 60.99) - open/close 1742 (time 1776190443.81 total 30.04 last 55.53) - open/close 2394 (time 1776190453.82 total 40.05 last 65.14) - open/close 3024 (time 1776190463.83 total 50.06 last 62.90) - open/close 3682 (time 1776190473.84 total 60.06 last 65.79) - open/close 4353 (time 1776190483.84 total 70.07 last 67.08) - open/close 5024 (time 1776190493.84 total 80.07 last 67.07) - open/close 5763 (time 1776190503.85 total 90.08 last 73.86) - open/close 6467 (time 1776190513.85 total 100.08 last 70.35) - open/close 7128 (time 1776190523.87 total 110.10 last 66.00) - open/close 7865 (time 1776190533.88 total 120.10 last 73.66) - open/close 8507 (time 1776190543.88 total 130.11 last 64.17) - open/close 9094 (time 1776190553.88 total 140.11 last 58.70) - open/close 9836 (time 1776190563.89 total 150.12 last 74.14) total: 10000 open/close in 152.41 seconds: 65.61 ops/second - unlinked 0 (time 1776190569 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 642 (time 1776190663.57 total 10.02 last 64.06) - open/close 1380 (time 1776190673.58 total 20.03 last 73.77) - open/close 2130 (time 1776190683.58 total 30.03 last 74.95) - open/close 2897 (time 1776190693.59 total 40.04 last 76.65) - open/close 3605 (time 1776190703.60 total 50.05 last 70.74) - open/close 4297 (time 1776190713.60 total 60.05 last 69.17) - open/close 4996 (time 1776190723.61 total 70.06 last 69.85) - open/close 5695 (time 1776190733.62 total 80.07 last 69.82) - open/close 6421 (time 1776190743.63 total 90.08 last 72.53) - open/close 7045 (time 1776190753.64 total 100.08 last 62.36) - open/close 7671 (time 1776190763.64 total 110.09 last 62.58) - open/close 8339 (time 1776190773.65 total 120.09 last 66.77) - open/close 9026 (time 1776190783.66 total 130.11 last 68.58) - open/close 9692 (time 1776190793.66 total 140.11 last 66.59) total: 10000 open/close in 144.43 seconds: 69.24 ops/second - unlinked 0 (time 1776190801 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 671 (time 1776190896.35 total 10.01 last 67.06) - open/close 1390 (time 1776190906.35 total 20.01 last 71.86) - open/close 2080 (time 1776190916.35 total 30.02 last 68.97) - open/close 2767 (time 1776190926.37 total 40.03 last 68.63) - open/close 3433 (time 1776190936.39 total 50.05 last 66.44) - open/close 4077 (time 1776190946.40 total 60.07 last 64.30) - open/close 4783 (time 1776190956.41 total 70.07 last 70.59) - open/close 5509 (time 1776190966.41 total 80.07 last 72.55) - open/close 6212 (time 1776190976.42 total 90.08 last 70.27) - open/close 6824 (time 1776190986.42 total 100.09 last 61.17) - open/close 7457 (time 1776190996.44 total 110.10 last 63.22) - open/close 8098 (time 1776191006.44 total 120.10 last 64.08) - open/close 8715 (time 1776191016.44 total 130.11 last 61.67) - open/close 9374 (time 1776191026.46 total 140.12 last 65.81) total: 10000 open/close in 149.03 seconds: 67.10 ops/second - unlinked 0 (time 1776191038 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 717 (time 1776191122.76 total 10.00 last 71.68) - open/close 1381 (time 1776191132.77 total 20.01 last 66.33) - open/close 2168 (time 1776191142.78 total 30.02 last 78.65) - open/close 2996 (time 1776191152.79 total 40.03 last 82.69) - open/close 3723 (time 1776191162.79 total 50.04 last 72.67) - open/close 4456 (time 1776191172.80 total 60.04 last 73.28) - open/close 5251 (time 1776191182.82 total 70.06 last 79.35) - open/close 5948 (time 1776191192.84 total 80.08 last 69.52) - open/close 6486 (time 1776191202.86 total 90.10 last 53.72) - open/close 7101 (time 1776191212.86 total 100.10 last 61.49) - open/close 7763 (time 1776191222.88 total 110.12 last 66.07) - open/close 8458 (time 1776191232.88 total 120.12 last 69.47) - open/close 9142 (time 1776191242.89 total 130.13 last 68.37) - open/close 9764 (time 1776191252.89 total 140.13 last 62.17) total: 10000 open/close in 143.26 seconds: 69.80 ops/second - unlinked 0 (time 1776191260 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 686 (time 1776191358.59 total 10.01 last 68.54) - open/close 1417 (time 1776191368.60 total 20.02 last 73.03) - open/close 2122 (time 1776191378.61 total 30.02 last 70.48) - open/close 2856 (time 1776191388.63 total 40.04 last 73.26) - open/close 3532 (time 1776191398.64 total 50.05 last 67.53) - open/close 4250 (time 1776191408.64 total 60.05 last 71.79) - open/close 5001 (time 1776191418.65 total 70.06 last 75.04) - open/close 5706 (time 1776191428.65 total 80.07 last 70.45) - open/close 6378 (time 1776191438.66 total 90.07 last 67.16) - open/close 7077 (time 1776191448.66 total 100.08 last 69.88) - open/close 7751 (time 1776191458.67 total 110.08 last 67.35) - open/close 8431 (time 1776191468.68 total 120.10 last 67.93) - open/close 9121 (time 1776191478.69 total 130.11 last 68.93) - open/close 9873 (time 1776191488.70 total 140.11 last 75.15) total: 10000 open/close in 141.61 seconds: 70.61 ops/second - unlinked 0 (time 1776191494 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.121@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg321-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg321-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock oleg321-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing unload_modules_local oleg321-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg321-server: LNET ready to unload oleg321-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2