== conf-sanity test 69: replace an OST with the same index ========================================================== 12:58:25 (1761238705) start mds service on oleg423-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: Force libcfs to create 2 CPU partitions oleg423-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg423-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg423-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg423-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock oleg423-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 769 (time 1761238804.82 total 10.00 last 76.89) - open/close 1499 (time 1761238814.83 total 20.01 last 72.92) - open/close 2287 (time 1761238824.84 total 30.02 last 78.75) - open/close 3066 (time 1761238834.84 total 40.02 last 77.88) - open/close 3845 (time 1761238844.84 total 50.02 last 77.87) - open/close 4632 (time 1761238854.85 total 60.03 last 78.66) - open/close 5418 (time 1761238864.85 total 70.03 last 78.56) - open/close 6228 (time 1761238874.86 total 80.04 last 80.97) - open/close 6978 (time 1761238884.86 total 90.04 last 74.97) - open/close 7801 (time 1761238894.87 total 100.05 last 82.26) - open/close 8669 (time 1761238904.87 total 110.05 last 86.75) - open/close 9493 (time 1761238914.88 total 120.06 last 82.35) total: 10000 open/close in 127.02 seconds: 78.73 ops/second - unlinked 0 (time 1761238925 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 788 (time 1761239009.52 total 10.00 last 78.78) - open/close 1648 (time 1761239019.52 total 20.01 last 85.95) - open/close 2491 (time 1761239029.53 total 30.01 last 84.25) - open/close 3326 (time 1761239039.53 total 40.02 last 83.48) - open/close 4271 (time 1761239049.55 total 50.03 last 94.36) - open/close 5160 (time 1761239059.55 total 60.03 last 88.89) - open/close 6012 (time 1761239069.55 total 70.03 last 85.19) - open/close 6864 (time 1761239079.56 total 80.05 last 85.08) - open/close 7789 (time 1761239089.56 total 90.05 last 92.48) - open/close 8788 (time 1761239099.57 total 100.05 last 99.88) - open/close 9747 (time 1761239109.58 total 110.07 last 95.74) total: 10000 open/close in 112.93 seconds: 88.55 ops/second - unlinked 0 (time 1761239115 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 911 (time 1761239184.17 total 10.01 last 91.04) - open/close 1863 (time 1761239194.17 total 20.01 last 95.20) - open/close 2783 (time 1761239204.19 total 30.02 last 91.84) - open/close 3654 (time 1761239214.19 total 40.03 last 87.08) - open/close 4685 (time 1761239224.19 total 50.03 last 103.10) - open/close 5600 (time 1761239234.20 total 60.03 last 91.47) - open/close 6543 (time 1761239244.20 total 70.03 last 94.27) - open/close 7490 (time 1761239254.20 total 80.04 last 94.69) - open/close 8486 (time 1761239264.20 total 90.04 last 99.60) - open/close 9368 (time 1761239274.22 total 100.06 last 88.03) total: 10000 open/close in 106.95 seconds: 93.50 ops/second - unlinked 0 (time 1761239283 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 822 (time 1761239361.97 total 10.00 last 82.17) - open/close 1867 (time 1761239371.97 total 20.00 last 104.48) - open/close 3066 (time 1761239381.97 total 30.01 last 119.87) - open/close 4117 (time 1761239391.98 total 40.01 last 105.09) - open/close 4951 (time 1761239401.98 total 50.01 last 83.38) - open/close 5931 (time 1761239411.98 total 60.01 last 97.97) - open/close 7214 (time 1761239421.98 total 70.02 last 128.28) - open/close 8464 (time 1761239431.99 total 80.02 last 124.93) - open/close 9598 (time 1761239441.99 total 90.02 last 113.38) total: 10000 open/close in 94.05 seconds: 106.33 ops/second - unlinked 0 (time 1761239448 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 919 (time 1761239506.73 total 10.00 last 91.90) - open/close 1851 (time 1761239516.74 total 20.01 last 93.12) - open/close 2711 (time 1761239526.75 total 30.02 last 85.92) - open/close 3568 (time 1761239536.76 total 40.03 last 85.58) - open/close 4413 (time 1761239546.77 total 50.04 last 84.42) - open/close 5365 (time 1761239556.78 total 60.04 last 95.19) - open/close 6389 (time 1761239566.78 total 70.05 last 102.37) - open/close 7360 (time 1761239576.78 total 80.05 last 97.08) - open/close 8405 (time 1761239586.78 total 90.05 last 104.47) - open/close 9395 (time 1761239596.78 total 100.05 last 98.99) total: 10000 open/close in 106.84 seconds: 93.60 ops/second - unlinked 0 (time 1761239606 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 870 (time 1761239681.66 total 10.01 last 86.95) - open/close 1755 (time 1761239691.66 total 20.01 last 88.43) - open/close 2651 (time 1761239701.67 total 30.02 last 89.55) - open/close 3533 (time 1761239711.67 total 40.02 last 88.16) - open/close 4510 (time 1761239721.68 total 50.03 last 97.66) - open/close 5427 (time 1761239731.68 total 60.03 last 91.67) - open/close 6327 (time 1761239741.69 total 70.04 last 89.95) - open/close 7206 (time 1761239751.69 total 80.04 last 87.89) - open/close 8082 (time 1761239761.70 total 90.05 last 87.50) - open/close 8944 (time 1761239771.70 total 100.05 last 86.15) - open/close 9930 (time 1761239781.71 total 110.06 last 98.54) total: 10000 open/close in 110.66 seconds: 90.37 ops/second - unlinked 0 (time 1761239784 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1043 (time 1761239855.38 total 10.00 last 104.30) - open/close 2204 (time 1761239865.38 total 20.00 last 116.07) - open/close 3249 (time 1761239875.38 total 30.01 last 104.48) - open/close 4408 (time 1761239885.39 total 40.01 last 115.82) - open/close 5684 (time 1761239895.39 total 50.01 last 127.60) - open/close 6830 (time 1761239905.39 total 60.01 last 114.58) - open/close 8272 (time 1761239915.40 total 70.02 last 144.15) - open/close 9737 (time 1761239925.40 total 80.02 last 146.48) total: 10000 open/close in 81.97 seconds: 121.99 ops/second - unlinked 0 (time 1761239929 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 931 (time 1761240002.09 total 10.00 last 93.08) - open/close 1869 (time 1761240012.10 total 20.01 last 93.71) - open/close 3065 (time 1761240022.10 total 30.01 last 119.58) - open/close 3943 (time 1761240032.11 total 40.02 last 87.71) - open/close 4711 (time 1761240042.11 total 50.03 last 76.79) - open/close 5512 (time 1761240052.13 total 60.04 last 79.96) - open/close 6283 (time 1761240062.13 total 70.05 last 77.08) - open/close 7133 (time 1761240072.13 total 80.05 last 84.96) - open/close 7963 (time 1761240082.13 total 90.05 last 82.99) - open/close 8789 (time 1761240092.14 total 100.06 last 82.54) - open/close 9694 (time 1761240102.15 total 110.07 last 90.42) total: 10000 open/close in 114.02 seconds: 87.70 ops/second - unlinked 0 (time 1761240109 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 936 (time 1761240194.16 total 10.01 last 93.53) - open/close 1908 (time 1761240204.16 total 20.01 last 97.17) - open/close 2819 (time 1761240214.17 total 30.02 last 91.03) - open/close 3574 (time 1761240224.18 total 40.02 last 75.45) - open/close 4395 (time 1761240234.18 total 50.03 last 82.05) - open/close 5168 (time 1761240244.19 total 60.03 last 77.27) - open/close 5996 (time 1761240254.20 total 70.04 last 82.71) - open/close 6801 (time 1761240264.20 total 80.05 last 80.47) - open/close 7617 (time 1761240274.21 total 90.05 last 81.57) - open/close 8470 (time 1761240284.21 total 100.06 last 85.24) - open/close 9315 (time 1761240294.21 total 110.06 last 84.49) total: 10000 open/close in 117.38 seconds: 85.19 ops/second - unlinked 0 (time 1761240304 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 903 (time 1761240380.46 total 10.00 last 90.30) - open/close 1764 (time 1761240390.47 total 20.01 last 86.03) - open/close 2705 (time 1761240400.48 total 30.01 last 94.07) - open/close 3584 (time 1761240410.48 total 40.02 last 87.84) - open/close 4469 (time 1761240420.48 total 50.02 last 88.48) - open/close 5222 (time 1761240430.50 total 60.03 last 75.22) - open/close 5961 (time 1761240440.50 total 70.03 last 73.90) - open/close 6780 (time 1761240450.51 total 80.04 last 81.82) - open/close 7543 (time 1761240460.51 total 90.04 last 76.29) - open/close 8273 (time 1761240470.52 total 100.05 last 72.94) - open/close 8912 (time 1761240480.52 total 110.06 last 63.86) - open/close 9632 (time 1761240490.52 total 120.06 last 71.99) total: 10000 open/close in 125.04 seconds: 79.97 ops/second - unlinked 0 (time 1761240498 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.123@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.123@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg423-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg423-server: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg423-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg423-server: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg423-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock oleg423-server@tcp:/lustre /mnt/lustre On OST0, 2959 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg423-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing unload_modules_local oleg423-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg423-server: LNET ready to unload oleg423-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2