== conf-sanity test 69: replace an OST with the same index ========================================================== 08:55:45 (1761310545) start mds service on oleg346-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: Force libcfs to create 2 CPU partitions oleg346-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg346-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 970 (time 1761310599.40 total 10.00 last 96.96) - open/close 2026 (time 1761310609.41 total 20.01 last 105.52) - open/close 3130 (time 1761310619.42 total 30.02 last 110.32) - open/close 4725 (time 1761310629.42 total 40.02 last 159.41) - open/close 6127 (time 1761310639.43 total 50.03 last 140.14) - open/close 7430 (time 1761310649.43 total 60.03 last 130.25) - open/close 9257 (time 1761310659.43 total 70.03 last 182.67) total: 10000 open/close in 72.81 seconds: 137.34 ops/second - unlinked 0 (time 1761310663 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2217 (time 1761310698.22 total 10.00 last 221.63) - open/close 4329 (time 1761310708.23 total 20.01 last 211.10) - open/close 6573 (time 1761310718.23 total 30.01 last 224.34) - open/close 8159 (time 1761310728.23 total 40.01 last 158.60) - open/close 9898 (time 1761310738.23 total 50.01 last 173.86) total: 10000 open/close in 50.46 seconds: 198.17 ops/second - unlinked 0 (time 1761310740 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2155 (time 1761310776.02 total 10.00 last 215.50) - open/close 4183 (time 1761310786.02 total 20.00 last 202.76) - open/close 6063 (time 1761310796.02 total 30.00 last 188.00) - open/close 6844 (time 1761310806.04 total 40.02 last 77.96) - open/close 7507 (time 1761310816.05 total 50.02 last 66.28) - open/close 8175 (time 1761310826.05 total 60.03 last 66.77) - open/close 8822 (time 1761310836.06 total 70.04 last 64.64) - open/close 9523 (time 1761310846.06 total 80.04 last 70.08) total: 10000 open/close in 87.90 seconds: 113.77 ops/second - unlinked 0 (time 1761310857 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 889 (time 1761310936.64 total 10.00 last 88.88) - open/close 1761 (time 1761310946.64 total 20.01 last 87.16) - open/close 2810 (time 1761310956.65 total 30.01 last 104.88) - open/close 3796 (time 1761310966.65 total 40.01 last 98.57) - open/close 4640 (time 1761310976.65 total 50.02 last 84.36) - open/close 5356 (time 1761310986.66 total 60.03 last 71.53) - open/close 6035 (time 1761310996.67 total 70.03 last 67.88) - open/close 6778 (time 1761311006.67 total 80.03 last 74.29) - open/close 7597 (time 1761311016.68 total 90.04 last 81.82) - open/close 8397 (time 1761311026.68 total 100.04 last 79.96) - open/close 9190 (time 1761311036.68 total 110.05 last 79.28) total: 10000 open/close in 119.58 seconds: 83.63 ops/second - unlinked 0 (time 1761311048 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1329 (time 1761311114.15 total 10.00 last 132.86) - open/close 2812 (time 1761311124.15 total 20.01 last 148.23) - open/close 4439 (time 1761311134.15 total 30.01 last 162.63) - open/close 6157 (time 1761311144.15 total 40.01 last 171.79) - open/close 7673 (time 1761311154.16 total 50.02 last 151.52) - open/close 9178 (time 1761311164.16 total 60.02 last 150.49) total: 10000 open/close in 66.62 seconds: 150.10 ops/second - unlinked 0 (time 1761311172 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1479 (time 1761311220.35 total 10.01 last 147.73) - open/close 2699 (time 1761311230.35 total 20.02 last 121.93) - open/close 3856 (time 1761311240.36 total 30.02 last 115.64) - open/close 4911 (time 1761311250.36 total 40.03 last 105.46) - open/close 6092 (time 1761311260.36 total 50.03 last 118.08) - open/close 7480 (time 1761311270.37 total 60.03 last 138.71) - open/close 9001 (time 1761311280.38 total 70.04 last 152.03) total: 10000 open/close in 80.00 seconds: 124.99 ops/second - unlinked 0 (time 1761311295 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 877 (time 1761311377.46 total 10.01 last 87.63) - open/close 1785 (time 1761311387.46 total 20.01 last 90.78) - open/close 2655 (time 1761311397.47 total 30.02 last 86.95) - open/close 3199 (time 1761311407.48 total 40.03 last 54.35) - open/close 3756 (time 1761311417.48 total 50.03 last 55.68) - open/close 4304 (time 1761311427.50 total 60.05 last 54.71) - open/close 4930 (time 1761311437.50 total 70.05 last 62.56) - open/close 5555 (time 1761311447.51 total 80.06 last 62.44) - open/close 6307 (time 1761311457.52 total 90.07 last 75.15) - open/close 7031 (time 1761311467.53 total 100.08 last 72.29) - open/close 7822 (time 1761311477.54 total 110.09 last 79.03) - open/close 8654 (time 1761311487.55 total 120.10 last 83.14) - open/close 9421 (time 1761311497.55 total 130.11 last 76.67) total: 10000 open/close in 136.83 seconds: 73.08 ops/second - unlinked 0 (time 1761311508 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 788 (time 1761311596.29 total 10.01 last 78.74) - open/close 1575 (time 1761311606.30 total 20.01 last 78.67) - open/close 2388 (time 1761311616.32 total 30.03 last 81.14) - open/close 3263 (time 1761311626.33 total 40.04 last 87.41) - open/close 4102 (time 1761311636.34 total 50.06 last 83.79) - open/close 4953 (time 1761311646.34 total 60.06 last 85.09) - open/close 5788 (time 1761311656.35 total 70.06 last 83.47) - open/close 6595 (time 1761311666.36 total 80.07 last 80.61) - open/close 7423 (time 1761311676.37 total 90.08 last 82.72) - open/close 8232 (time 1761311686.39 total 100.10 last 80.75) - open/close 8972 (time 1761311696.39 total 110.10 last 73.97) - open/close 9733 (time 1761311706.40 total 120.11 last 76.04) total: 10000 open/close in 123.88 seconds: 80.73 ops/second - unlinked 0 (time 1761311713 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 895 (time 1761311803.50 total 10.00 last 89.47) - open/close 1735 (time 1761311813.51 total 20.01 last 83.94) - open/close 2599 (time 1761311823.51 total 30.01 last 86.40) - open/close 3465 (time 1761311833.51 total 40.01 last 86.58) - open/close 4338 (time 1761311843.52 total 50.02 last 87.27) - open/close 5209 (time 1761311853.53 total 60.03 last 87.02) - open/close 6053 (time 1761311863.53 total 70.03 last 84.34) - open/close 6743 (time 1761311873.54 total 80.04 last 68.93) - open/close 7632 (time 1761311883.55 total 90.04 last 88.90) - open/close 8490 (time 1761311893.56 total 100.06 last 85.70) - open/close 9353 (time 1761311903.56 total 110.06 last 86.25) total: 10000 open/close in 117.53 seconds: 85.09 ops/second - unlinked 0 (time 1761311914 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 871 (time 1761311997.88 total 10.01 last 87.05) - open/close 1755 (time 1761312007.88 total 20.01 last 88.38) - open/close 2628 (time 1761312017.89 total 30.01 last 87.24) - open/close 3447 (time 1761312027.89 total 40.02 last 81.89) - open/close 4297 (time 1761312037.90 total 50.02 last 84.92) - open/close 5170 (time 1761312047.91 total 60.04 last 87.17) - open/close 6000 (time 1761312057.92 total 70.05 last 82.96) - open/close 6853 (time 1761312067.93 total 80.05 last 85.22) - open/close 7720 (time 1761312077.93 total 90.06 last 86.66) - open/close 8606 (time 1761312087.94 total 100.07 last 88.54) - open/close 9473 (time 1761312097.94 total 110.07 last 86.69) total: 10000 open/close in 116.09 seconds: 86.14 ops/second - unlinked 0 (time 1761312106 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.146@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.146@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg346-server: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg346-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local oleg346-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg346-server: LNET ready to unload oleg346-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2