== conf-sanity test 69: replace an OST with the same index ========================================================== 09:16:26 (1761311786) start mds service on oleg233-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing load_modules_local oleg233-server: Loading modules from /home/green/git/lustre-release/lustre oleg233-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg233-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg233-server: detected 4 online CPUs by sysfs oleg233-server: Force libcfs to create 2 CPU partitions oleg233-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg233-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg233-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg233-client.virtnet: -o user_xattr,flock oleg233-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 713 (time 1761311848.60 total 10.01 last 71.25) - open/close 1423 (time 1761311858.61 total 20.02 last 70.93) - open/close 2160 (time 1761311868.63 total 30.03 last 73.60) - open/close 2894 (time 1761311878.63 total 40.04 last 73.35) - open/close 3645 (time 1761311888.63 total 50.04 last 75.09) - open/close 4445 (time 1761311898.64 total 60.04 last 79.97) - open/close 5256 (time 1761311908.64 total 70.05 last 81.06) - open/close 6062 (time 1761311918.67 total 80.07 last 80.42) - open/close 6854 (time 1761311928.67 total 90.08 last 79.16) - open/close 7697 (time 1761311938.68 total 100.08 last 84.25) - open/close 8425 (time 1761311948.68 total 110.09 last 72.76) - open/close 9176 (time 1761311958.69 total 120.10 last 75.05) - open/close 9936 (time 1761311968.69 total 130.10 last 75.98) total: 10000 open/close in 130.93 seconds: 76.38 ops/second - unlinked 0 (time 1761311973 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 833 (time 1761312062.07 total 10.01 last 83.23) - open/close 1653 (time 1761312072.07 total 20.02 last 81.93) - open/close 2447 (time 1761312082.08 total 30.02 last 79.37) - open/close 3295 (time 1761312092.09 total 40.03 last 84.71) - open/close 4124 (time 1761312102.10 total 50.05 last 82.76) - open/close 5020 (time 1761312112.11 total 60.05 last 89.59) - open/close 5849 (time 1761312122.11 total 70.05 last 82.87) - open/close 6630 (time 1761312132.11 total 80.05 last 78.09) - open/close 7425 (time 1761312142.11 total 90.05 last 79.48) - open/close 8217 (time 1761312152.12 total 100.06 last 79.17) - open/close 8963 (time 1761312162.12 total 110.06 last 74.59) - open/close 9699 (time 1761312172.12 total 120.06 last 73.56) total: 10000 open/close in 123.43 seconds: 81.02 ops/second - unlinked 0 (time 1761312177 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 766 (time 1761312261.72 total 10.01 last 76.52) - open/close 1564 (time 1761312271.72 total 20.02 last 79.76) - open/close 2455 (time 1761312281.74 total 30.03 last 88.97) - open/close 3337 (time 1761312291.76 total 40.05 last 88.05) - open/close 4128 (time 1761312301.77 total 50.06 last 78.99) - open/close 4916 (time 1761312311.78 total 60.07 last 78.73) - open/close 5702 (time 1761312321.79 total 70.08 last 78.50) - open/close 6405 (time 1761312331.80 total 80.09 last 70.23) - open/close 7185 (time 1761312341.80 total 90.10 last 77.97) - open/close 8107 (time 1761312351.81 total 100.10 last 92.15) - open/close 9024 (time 1761312361.82 total 110.11 last 91.62) total: 10000 open/close in 120.00 seconds: 83.33 ops/second - unlinked 0 (time 1761312374 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 808 (time 1761312453.27 total 10.00 last 80.77) - open/close 1729 (time 1761312463.27 total 20.01 last 92.08) - open/close 2584 (time 1761312473.27 total 30.01 last 85.49) - open/close 3385 (time 1761312483.29 total 40.02 last 80.00) - open/close 4223 (time 1761312493.29 total 50.02 last 83.76) - open/close 5056 (time 1761312503.30 total 60.03 last 83.26) - open/close 5936 (time 1761312513.30 total 70.04 last 87.95) - open/close 6774 (time 1761312523.31 total 80.04 last 83.78) - open/close 7605 (time 1761312533.31 total 90.04 last 83.09) - open/close 8460 (time 1761312543.31 total 100.04 last 85.46) - open/close 9304 (time 1761312553.32 total 110.05 last 84.34) total: 10000 open/close in 117.94 seconds: 84.79 ops/second - unlinked 0 (time 1761312564 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 731 (time 1761312647.90 total 10.00 last 73.09) - open/close 1596 (time 1761312657.91 total 20.01 last 86.42) - open/close 2379 (time 1761312667.93 total 30.02 last 78.19) - open/close 3209 (time 1761312677.93 total 40.03 last 82.99) - open/close 4063 (time 1761312687.93 total 50.03 last 85.39) - open/close 4843 (time 1761312697.93 total 60.03 last 77.95) - open/close 5629 (time 1761312707.94 total 70.04 last 78.56) - open/close 6500 (time 1761312717.94 total 80.04 last 87.06) - open/close 7341 (time 1761312727.96 total 90.05 last 84.01) - open/close 8117 (time 1761312737.96 total 100.06 last 77.57) - open/close 8945 (time 1761312747.97 total 110.06 last 82.75) - open/close 9762 (time 1761312757.98 total 120.07 last 81.61) total: 10000 open/close in 122.92 seconds: 81.35 ops/second - unlinked 0 (time 1761312764 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 846 (time 1761312840.40 total 10.00 last 84.59) - open/close 1660 (time 1761312850.40 total 20.00 last 81.39) - open/close 2527 (time 1761312860.41 total 30.00 last 86.69) - open/close 3361 (time 1761312870.41 total 40.01 last 83.36) - open/close 4207 (time 1761312880.42 total 50.02 last 84.50) - open/close 5042 (time 1761312890.43 total 60.03 last 83.40) - open/close 5921 (time 1761312900.44 total 70.03 last 87.88) - open/close 6823 (time 1761312910.44 total 80.04 last 90.14) - open/close 7886 (time 1761312920.45 total 90.04 last 106.26) - open/close 8936 (time 1761312930.45 total 100.05 last 104.98) - open/close 9973 (time 1761312940.46 total 110.05 last 103.61) total: 10000 open/close in 110.32 seconds: 90.65 ops/second - unlinked 0 (time 1761312943 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 913 (time 1761313019.35 total 10.00 last 91.26) - open/close 1845 (time 1761313029.36 total 20.01 last 93.14) - open/close 2986 (time 1761313039.36 total 30.01 last 114.08) - open/close 3928 (time 1761313049.36 total 40.01 last 94.20) - open/close 4800 (time 1761313059.37 total 50.02 last 87.13) - open/close 5768 (time 1761313069.38 total 60.03 last 96.73) - open/close 6692 (time 1761313079.38 total 70.03 last 92.36) - open/close 7627 (time 1761313089.39 total 80.04 last 93.40) - open/close 8578 (time 1761313099.40 total 90.05 last 95.04) - open/close 9603 (time 1761313109.40 total 100.06 last 102.43) total: 10000 open/close in 104.34 seconds: 95.84 ops/second - unlinked 0 (time 1761313116 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 903 (time 1761313204.24 total 10.00 last 90.27) - open/close 1709 (time 1761313214.24 total 20.00 last 80.59) - open/close 2550 (time 1761313224.24 total 30.01 last 84.04) - open/close 3422 (time 1761313234.25 total 40.01 last 87.19) - open/close 4256 (time 1761313244.25 total 50.02 last 83.33) - open/close 5062 (time 1761313254.27 total 60.04 last 80.46) - open/close 5880 (time 1761313264.28 total 70.04 last 81.76) - open/close 6688 (time 1761313274.28 total 80.04 last 80.79) - open/close 7576 (time 1761313284.28 total 90.05 last 88.77) - open/close 8398 (time 1761313294.29 total 100.06 last 82.12) - open/close 9182 (time 1761313304.30 total 110.06 last 78.36) total: 10000 open/close in 119.64 seconds: 83.58 ops/second - unlinked 0 (time 1761313316 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 792 (time 1761313400.87 total 10.00 last 79.17) - open/close 1618 (time 1761313410.87 total 20.00 last 82.60) - open/close 2431 (time 1761313420.88 total 30.01 last 81.24) - open/close 3255 (time 1761313430.88 total 40.01 last 82.39) - open/close 4077 (time 1761313440.89 total 50.02 last 82.12) - open/close 4864 (time 1761313450.89 total 60.02 last 78.70) - open/close 5650 (time 1761313460.90 total 70.03 last 78.55) - open/close 6476 (time 1761313470.91 total 80.04 last 82.48) - open/close 7314 (time 1761313480.92 total 90.05 last 83.72) - open/close 8074 (time 1761313490.92 total 100.06 last 75.98) - open/close 8689 (time 1761313500.93 total 110.06 last 61.46) - open/close 9368 (time 1761313510.94 total 120.07 last 67.83) total: 10000 open/close in 128.06 seconds: 78.09 ops/second - unlinked 0 (time 1761313522 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 840 (time 1761313606.41 total 10.01 last 83.92) - open/close 1598 (time 1761313616.43 total 20.03 last 75.66) - open/close 2403 (time 1761313626.44 total 30.04 last 80.38) - open/close 3205 (time 1761313636.44 total 40.04 last 80.19) - open/close 3988 (time 1761313646.46 total 50.07 last 78.12) - open/close 4690 (time 1761313656.49 total 60.09 last 70.04) - open/close 5378 (time 1761313666.49 total 70.09 last 68.78) - open/close 6144 (time 1761313676.52 total 80.12 last 76.39) - open/close 6946 (time 1761313686.52 total 90.13 last 80.15) - open/close 7750 (time 1761313696.53 total 100.13 last 80.38) - open/close 8598 (time 1761313706.53 total 110.13 last 84.78) - open/close 9597 (time 1761313716.53 total 120.14 last 99.83) total: 10000 open/close in 124.76 seconds: 80.15 ops/second - unlinked 0 (time 1761313723 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg233-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg233-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.133@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.133@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg233-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg233-server: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg233-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg233-client.virtnet: -o user_xattr,flock oleg233-server@tcp:/lustre /mnt/lustre On OST0, 557 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg233-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg233-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server stop mds service on oleg233-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg233-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing unload_modules_local oleg233-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg233-server: LNET ready to unload oleg233-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2