== conf-sanity test 69: replace an OST with the same index ========================================================== 10:00:02 (1761228002) start mds service on oleg314-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing load_modules_local oleg314-server: Loading modules from /home/green/git/lustre-release/lustre oleg314-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg314-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg314-server: detected 4 online CPUs by sysfs oleg314-server: Force libcfs to create 2 CPU partitions oleg314-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg314-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 635 (time 1761228074.42 total 10.00 last 63.48) - open/close 1327 (time 1761228084.42 total 20.00 last 69.19) - open/close 1992 (time 1761228094.43 total 30.01 last 66.46) - open/close 2770 (time 1761228104.43 total 40.01 last 77.79) - open/close 3500 (time 1761228114.44 total 50.02 last 72.93) - open/close 4253 (time 1761228124.44 total 60.03 last 75.24) - open/close 5002 (time 1761228134.45 total 70.04 last 74.84) - open/close 5773 (time 1761228144.45 total 80.04 last 77.08) - open/close 6524 (time 1761228154.47 total 90.05 last 75.01) - open/close 7290 (time 1761228164.48 total 100.06 last 76.53) - open/close 8018 (time 1761228174.49 total 110.07 last 72.71) - open/close 8766 (time 1761228184.53 total 120.12 last 74.48) - open/close 9480 (time 1761228194.54 total 130.12 last 71.37) total: 10000 open/close in 137.19 seconds: 72.89 ops/second - unlinked 0 (time 1761228205 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 731 (time 1761228299.61 total 10.01 last 73.06) - open/close 1540 (time 1761228309.62 total 20.01 last 80.85) - open/close 2289 (time 1761228319.62 total 30.02 last 74.87) - open/close 3010 (time 1761228329.63 total 40.03 last 72.02) - open/close 3788 (time 1761228339.65 total 50.04 last 77.68) - open/close 4549 (time 1761228349.66 total 60.06 last 76.00) - open/close 5257 (time 1761228359.67 total 70.06 last 70.76) - open/close 5928 (time 1761228369.68 total 80.08 last 66.98) - open/close 6595 (time 1761228379.69 total 90.08 last 66.68) - open/close 7339 (time 1761228389.69 total 100.08 last 74.39) - open/close 8089 (time 1761228399.69 total 110.09 last 74.98) - open/close 8837 (time 1761228409.70 total 120.10 last 74.71) - open/close 9612 (time 1761228419.71 total 130.10 last 77.47) total: 10000 open/close in 135.48 seconds: 73.81 ops/second - unlinked 0 (time 1761228428 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 760 (time 1761228516.14 total 10.01 last 75.94) - open/close 1536 (time 1761228526.16 total 20.03 last 77.45) - open/close 2355 (time 1761228536.17 total 30.03 last 81.86) - open/close 3132 (time 1761228546.17 total 40.03 last 77.69) - open/close 3982 (time 1761228556.18 total 50.04 last 84.95) - open/close 4798 (time 1761228566.18 total 60.05 last 81.53) - open/close 5511 (time 1761228576.20 total 70.06 last 71.21) - open/close 6238 (time 1761228586.21 total 80.07 last 72.62) - open/close 7019 (time 1761228596.21 total 90.08 last 78.05) - open/close 7794 (time 1761228606.22 total 100.09 last 77.41) - open/close 8561 (time 1761228616.24 total 110.10 last 76.59) - open/close 9287 (time 1761228626.24 total 120.10 last 72.59) - open/close 9951 (time 1761228636.24 total 130.11 last 66.38) total: 10000 open/close in 130.78 seconds: 76.46 ops/second - unlinked 0 (time 1761228641 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 744 (time 1761228734.75 total 10.02 last 74.27) - open/close 1501 (time 1761228744.76 total 20.02 last 75.67) - open/close 2174 (time 1761228754.77 total 30.03 last 67.21) - open/close 2912 (time 1761228764.77 total 40.04 last 73.79) - open/close 3659 (time 1761228774.77 total 50.04 last 74.69) - open/close 4411 (time 1761228784.78 total 60.04 last 75.18) - open/close 5181 (time 1761228794.78 total 70.04 last 76.98) - open/close 5946 (time 1761228804.79 total 80.05 last 76.42) - open/close 6647 (time 1761228814.80 total 90.07 last 70.01) - open/close 7421 (time 1761228824.81 total 100.07 last 77.33) - open/close 8204 (time 1761228834.83 total 110.09 last 78.17) - open/close 8991 (time 1761228844.83 total 120.09 last 78.68) - open/close 9744 (time 1761228854.84 total 130.10 last 75.28) total: 10000 open/close in 133.54 seconds: 74.89 ops/second - unlinked 0 (time 1761228861 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 745 (time 1761228953.90 total 10.01 last 74.40) - open/close 1439 (time 1761228963.91 total 20.02 last 69.36) - open/close 2138 (time 1761228973.92 total 30.03 last 69.82) - open/close 2815 (time 1761228983.93 total 40.04 last 67.67) - open/close 3488 (time 1761228993.93 total 50.04 last 67.29) - open/close 4253 (time 1761229003.94 total 60.05 last 76.38) - open/close 4979 (time 1761229013.97 total 70.08 last 72.41) - open/close 5651 (time 1761229023.97 total 80.09 last 67.16) - open/close 6355 (time 1761229033.99 total 90.10 last 70.32) - open/close 7055 (time 1761229044.00 total 100.11 last 69.93) - open/close 7763 (time 1761229054.00 total 110.11 last 70.75) - open/close 8574 (time 1761229064.01 total 120.12 last 81.07) - open/close 9382 (time 1761229074.02 total 130.13 last 80.72) total: 10000 open/close in 138.55 seconds: 72.18 ops/second - unlinked 0 (time 1761229086 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 855 (time 1761229173.21 total 10.01 last 85.39) - open/close 1624 (time 1761229183.21 total 20.01 last 76.88) - open/close 2370 (time 1761229193.23 total 30.03 last 74.47) - open/close 3094 (time 1761229203.24 total 40.04 last 72.34) - open/close 3793 (time 1761229213.25 total 50.06 last 69.78) - open/close 4524 (time 1761229223.27 total 60.07 last 73.00) - open/close 5250 (time 1761229233.28 total 70.09 last 72.49) - open/close 5976 (time 1761229243.30 total 80.10 last 72.50) - open/close 6673 (time 1761229253.31 total 90.11 last 69.63) - open/close 7430 (time 1761229263.31 total 100.12 last 75.65) - open/close 8203 (time 1761229273.32 total 110.13 last 77.24) - open/close 8926 (time 1761229283.32 total 120.13 last 72.28) - open/close 9659 (time 1761229293.34 total 130.14 last 73.20) total: 10000 open/close in 135.09 seconds: 74.03 ops/second - unlinked 0 (time 1761229301 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 772 (time 1761229393.76 total 10.01 last 77.15) - open/close 1582 (time 1761229403.77 total 20.01 last 80.94) - open/close 2346 (time 1761229413.77 total 30.01 last 76.40) - open/close 3128 (time 1761229423.78 total 40.02 last 78.13) - open/close 3919 (time 1761229433.79 total 50.04 last 78.96) - open/close 4630 (time 1761229443.79 total 60.04 last 71.09) - open/close 5376 (time 1761229453.80 total 70.05 last 74.52) - open/close 6179 (time 1761229463.81 total 80.06 last 80.23) - open/close 6908 (time 1761229473.82 total 90.07 last 72.83) - open/close 7633 (time 1761229483.82 total 100.07 last 72.49) - open/close 8371 (time 1761229493.83 total 110.08 last 73.78) - open/close 9278 (time 1761229503.84 total 120.09 last 90.59) total: 10000 open/close in 129.02 seconds: 77.51 ops/second - unlinked 0 (time 1761229515 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 688 (time 1761229612.62 total 10.00 last 68.77) - open/close 1393 (time 1761229622.62 total 20.01 last 70.46) - open/close 2050 (time 1761229632.64 total 30.02 last 65.62) - open/close 2646 (time 1761229642.66 total 40.04 last 59.47) - open/close 3258 (time 1761229652.70 total 50.08 last 60.98) - open/close 3826 (time 1761229662.70 total 60.09 last 56.76) - open/close 4476 (time 1761229672.71 total 70.10 last 64.96) - open/close 5032 (time 1761229682.73 total 80.12 last 55.48) - open/close 5686 (time 1761229692.74 total 90.12 last 65.35) - open/close 6319 (time 1761229702.74 total 100.13 last 63.28) - open/close 6909 (time 1761229712.75 total 110.13 last 58.96) - open/close 7496 (time 1761229722.76 total 120.14 last 58.64) - open/close 8117 (time 1761229732.76 total 130.15 last 62.09) - open/close 8742 (time 1761229742.77 total 140.15 last 62.45) - open/close 9395 (time 1761229752.77 total 150.15 last 65.29) total: 10000 open/close in 159.46 seconds: 62.71 ops/second - unlinked 0 (time 1761229766 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 624 (time 1761229874.84 total 10.00 last 62.37) - open/close 1193 (time 1761229884.87 total 20.03 last 56.77) - open/close 1741 (time 1761229894.87 total 30.04 last 54.76) - open/close 2319 (time 1761229904.90 total 40.06 last 57.64) - open/close 2969 (time 1761229914.92 total 50.08 last 64.87) - open/close 3556 (time 1761229924.92 total 60.08 last 58.69) - open/close 4127 (time 1761229934.95 total 70.11 last 56.93) - open/close 4705 (time 1761229944.96 total 80.12 last 57.76) - open/close 5370 (time 1761229954.99 total 90.15 last 66.32) - open/close 5960 (time 1761229965.00 total 100.16 last 58.91) - open/close 6569 (time 1761229975.01 total 110.17 last 60.87) - open/close 7191 (time 1761229985.03 total 120.19 last 62.05) - open/close 7861 (time 1761229995.04 total 130.20 last 66.92) - open/close 8509 (time 1761230005.06 total 140.22 last 64.71) - open/close 9150 (time 1761230015.06 total 150.22 last 64.07) - open/close 9777 (time 1761230025.08 total 160.24 last 62.60) total: 10000 open/close in 164.60 seconds: 60.75 ops/second - unlinked 0 (time 1761230034 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 674 (time 1761230150.78 total 10.01 last 67.34) - open/close 1370 (time 1761230160.78 total 20.01 last 69.59) - open/close 2022 (time 1761230170.80 total 30.02 last 65.13) - open/close 2676 (time 1761230180.80 total 40.02 last 65.39) - open/close 3340 (time 1761230190.81 total 50.04 last 66.31) - open/close 3995 (time 1761230200.84 total 60.06 last 65.34) - open/close 4642 (time 1761230210.86 total 70.08 last 64.55) - open/close 5216 (time 1761230220.89 total 80.12 last 57.21) - open/close 5703 (time 1761230230.89 total 90.12 last 48.69) - open/close 6263 (time 1761230240.89 total 100.12 last 55.99) - open/close 6828 (time 1761230250.90 total 110.13 last 56.46) - open/close 7405 (time 1761230260.91 total 120.13 last 57.66) - open/close 8030 (time 1761230270.91 total 130.14 last 62.49) - open/close 8725 (time 1761230280.91 total 140.14 last 69.47) - open/close 9433 (time 1761230290.93 total 150.15 last 70.71) total: 10000 open/close in 159.27 seconds: 62.79 ops/second - unlinked 0 (time 1761230304 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.114@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg314-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing unload_modules_local oleg314-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg314-server: LNET ready to unload oleg314-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2