== conf-sanity test 69: replace an OST with the same index ========================================================== 09:49:48 (1761313788) start mds service on oleg445-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg445-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg445-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock oleg445-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 820 (time 1761313846.80 total 10.00 last 82.00) - open/close 1661 (time 1761313856.82 total 20.01 last 83.99) - open/close 2500 (time 1761313866.83 total 30.02 last 83.83) - open/close 3265 (time 1761313876.83 total 40.03 last 76.48) - open/close 4023 (time 1761313886.83 total 50.03 last 75.79) - open/close 4789 (time 1761313896.83 total 60.03 last 76.58) - open/close 5595 (time 1761313906.84 total 70.04 last 80.53) - open/close 6380 (time 1761313916.84 total 80.04 last 78.47) - open/close 7155 (time 1761313926.86 total 90.05 last 77.40) - open/close 7924 (time 1761313936.87 total 100.07 last 76.80) - open/close 8690 (time 1761313946.89 total 110.09 last 76.45) - open/close 9415 (time 1761313956.89 total 120.09 last 72.48) total: 10000 open/close in 128.82 seconds: 77.63 ops/second - unlinked 0 (time 1761313969 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 813 (time 1761314057.88 total 10.01 last 81.25) - open/close 1681 (time 1761314067.89 total 20.02 last 86.71) - open/close 2494 (time 1761314077.90 total 30.03 last 81.21) - open/close 3271 (time 1761314087.91 total 40.04 last 77.63) - open/close 4063 (time 1761314097.92 total 50.04 last 79.16) - open/close 4930 (time 1761314107.93 total 60.06 last 86.55) - open/close 5709 (time 1761314117.94 total 70.06 last 77.86) - open/close 6425 (time 1761314127.95 total 80.07 last 71.54) - open/close 7130 (time 1761314137.95 total 90.08 last 70.45) - open/close 7920 (time 1761314147.96 total 100.09 last 78.91) - open/close 8675 (time 1761314157.97 total 110.10 last 75.44) - open/close 9475 (time 1761314167.98 total 120.11 last 79.91) total: 10000 open/close in 127.11 seconds: 78.67 ops/second - unlinked 0 (time 1761314178 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 798 (time 1761314261.72 total 10.01 last 79.71) - open/close 1593 (time 1761314271.72 total 20.01 last 79.47) - open/close 2585 (time 1761314281.73 total 30.02 last 99.19) - open/close 3482 (time 1761314291.73 total 40.02 last 89.67) - open/close 4278 (time 1761314301.74 total 50.03 last 79.54) - open/close 5106 (time 1761314311.75 total 60.04 last 82.73) - open/close 5932 (time 1761314321.75 total 70.04 last 82.59) - open/close 6874 (time 1761314331.75 total 80.04 last 94.20) - open/close 7848 (time 1761314341.75 total 90.04 last 97.37) - open/close 8667 (time 1761314351.75 total 100.04 last 81.87) - open/close 9493 (time 1761314361.75 total 110.04 last 82.59) total: 10000 open/close in 116.40 seconds: 85.91 ops/second - unlinked 0 (time 1761314371 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 805 (time 1761314448.23 total 10.00 last 80.49) - open/close 1637 (time 1761314458.25 total 20.02 last 83.04) - open/close 2353 (time 1761314468.25 total 30.02 last 71.60) - open/close 3141 (time 1761314478.26 total 40.03 last 78.69) - open/close 4006 (time 1761314488.26 total 50.04 last 86.46) - open/close 4843 (time 1761314498.27 total 60.04 last 83.69) - open/close 5696 (time 1761314508.27 total 70.05 last 85.26) - open/close 6460 (time 1761314518.29 total 80.07 last 76.22) - open/close 7192 (time 1761314528.29 total 90.07 last 73.20) - open/close 8005 (time 1761314538.30 total 100.07 last 81.27) - open/close 8768 (time 1761314548.30 total 110.08 last 76.26) - open/close 9467 (time 1761314558.31 total 120.08 last 69.86) total: 10000 open/close in 127.62 seconds: 78.36 ops/second - unlinked 0 (time 1761314569 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 819 (time 1761314655.34 total 10.00 last 81.87) - open/close 1597 (time 1761314665.35 total 20.01 last 77.71) - open/close 2397 (time 1761314675.35 total 30.02 last 79.97) - open/close 3352 (time 1761314685.35 total 40.02 last 95.49) - open/close 4227 (time 1761314695.35 total 50.02 last 87.50) - open/close 4973 (time 1761314705.36 total 60.03 last 74.53) - open/close 5709 (time 1761314715.37 total 70.04 last 73.54) - open/close 6485 (time 1761314725.37 total 80.04 last 77.58) - open/close 7201 (time 1761314735.38 total 90.05 last 71.57) - open/close 7884 (time 1761314745.38 total 100.05 last 68.29) - open/close 8647 (time 1761314755.40 total 110.06 last 76.17) - open/close 9372 (time 1761314765.41 total 120.08 last 72.38) total: 10000 open/close in 128.52 seconds: 77.81 ops/second - unlinked 0 (time 1761314777 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 773 (time 1761314867.39 total 10.00 last 77.30) - open/close 1510 (time 1761314877.40 total 20.01 last 73.65) - open/close 2337 (time 1761314887.40 total 30.01 last 82.68) - open/close 3129 (time 1761314897.40 total 40.01 last 79.19) - open/close 3869 (time 1761314907.41 total 50.02 last 73.94) - open/close 4583 (time 1761314917.43 total 60.03 last 71.29) - open/close 5250 (time 1761314927.43 total 70.04 last 66.66) - open/close 5972 (time 1761314937.44 total 80.05 last 72.14) - open/close 6628 (time 1761314947.45 total 90.05 last 65.56) - open/close 7356 (time 1761314957.45 total 100.06 last 72.75) - open/close 8158 (time 1761314967.46 total 110.07 last 80.12) - open/close 8971 (time 1761314977.47 total 120.07 last 81.28) - open/close 9671 (time 1761314987.47 total 130.08 last 69.98) total: 10000 open/close in 134.68 seconds: 74.25 ops/second - unlinked 0 (time 1761314997 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 779 (time 1761315092.63 total 10.01 last 77.81) - open/close 1490 (time 1761315102.64 total 20.02 last 71.07) - open/close 2293 (time 1761315112.64 total 30.02 last 80.28) - open/close 3107 (time 1761315122.64 total 40.02 last 81.40) - open/close 3876 (time 1761315132.64 total 50.02 last 76.86) - open/close 4547 (time 1761315142.65 total 60.03 last 67.06) - open/close 5209 (time 1761315152.66 total 70.04 last 66.15) - open/close 5855 (time 1761315162.67 total 80.05 last 64.52) - open/close 6584 (time 1761315172.67 total 90.05 last 72.87) - open/close 7310 (time 1761315182.69 total 100.07 last 72.46) - open/close 8064 (time 1761315192.71 total 110.09 last 75.28) - open/close 8826 (time 1761315202.71 total 120.09 last 76.19) - open/close 9532 (time 1761315212.71 total 130.09 last 70.60) total: 10000 open/close in 136.83 seconds: 73.08 ops/second - unlinked 0 (time 1761315223 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 755 (time 1761315312.82 total 10.01 last 75.43) - open/close 1510 (time 1761315322.83 total 20.02 last 75.44) - open/close 2229 (time 1761315332.83 total 30.02 last 71.90) - open/close 2938 (time 1761315342.84 total 40.02 last 70.86) - open/close 3705 (time 1761315352.84 total 50.02 last 76.69) - open/close 4443 (time 1761315362.86 total 60.04 last 73.67) - open/close 5090 (time 1761315372.86 total 70.05 last 64.66) - open/close 5803 (time 1761315382.87 total 80.05 last 71.26) - open/close 6488 (time 1761315392.87 total 90.06 last 68.48) - open/close 7201 (time 1761315402.89 total 100.07 last 71.18) - open/close 7928 (time 1761315412.89 total 110.08 last 72.69) - open/close 8643 (time 1761315422.89 total 120.08 last 71.48) - open/close 9275 (time 1761315432.89 total 130.08 last 63.20) - open/close 9937 (time 1761315442.90 total 140.08 last 66.19) total: 10000 open/close in 141.05 seconds: 70.89 ops/second - unlinked 0 (time 1761315448 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 640 (time 1761315549.91 total 10.00 last 63.98) - open/close 1314 (time 1761315559.92 total 20.01 last 67.37) - open/close 2030 (time 1761315569.92 total 30.01 last 71.58) - open/close 2665 (time 1761315579.93 total 40.02 last 63.46) - open/close 3295 (time 1761315589.93 total 50.02 last 62.98) - open/close 3914 (time 1761315599.95 total 60.04 last 61.77) - open/close 4608 (time 1761315609.96 total 70.05 last 69.34) - open/close 5335 (time 1761315619.97 total 80.06 last 72.62) - open/close 6041 (time 1761315629.97 total 90.06 last 70.57) - open/close 6724 (time 1761315639.98 total 100.08 last 68.23) - open/close 7400 (time 1761315649.99 total 110.08 last 67.54) - open/close 8132 (time 1761315659.99 total 120.08 last 73.19) - open/close 8817 (time 1761315670.00 total 130.09 last 68.48) - open/close 9400 (time 1761315680.01 total 140.10 last 58.21) total: 10000 open/close in 148.35 seconds: 67.41 ops/second - unlinked 0 (time 1761315692 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 718 (time 1761315794.37 total 10.01 last 71.75) - open/close 1467 (time 1761315804.38 total 20.02 last 74.84) - open/close 2145 (time 1761315814.39 total 30.02 last 67.75) - open/close 2832 (time 1761315824.39 total 40.03 last 68.69) - open/close 3506 (time 1761315834.41 total 50.04 last 67.30) - open/close 4195 (time 1761315844.44 total 60.08 last 68.66) - open/close 4896 (time 1761315854.45 total 70.09 last 70.02) - open/close 5629 (time 1761315864.45 total 80.09 last 73.29) - open/close 6359 (time 1761315874.46 total 90.10 last 72.92) - open/close 6986 (time 1761315884.47 total 100.10 last 62.66) - open/close 7528 (time 1761315894.48 total 110.11 last 54.15) - open/close 8065 (time 1761315904.48 total 120.11 last 53.70) - open/close 8688 (time 1761315914.48 total 130.11 last 62.30) - open/close 9384 (time 1761315924.49 total 140.12 last 69.54) total: 10000 open/close in 149.86 seconds: 66.73 ops/second - unlinked 0 (time 1761315938 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.145@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.145@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg445-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg445-server: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg445-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock oleg445-server@tcp:/lustre /mnt/lustre On OST0, 1397 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local oleg445-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg445-server: LNET ready to unload oleg445-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2