== conf-sanity test 69: replace an OST with the same index ========================================================== 09:36:58 (1761313018) start mds service on oleg311-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing load_modules_local oleg311-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg311-server: Loading modules from /home/green/git/lustre-release/lustre oleg311-server: detected 4 online CPUs by sysfs oleg311-server: Force libcfs to create 2 CPU partitions oleg311-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg311-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg311-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg311-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock oleg311-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 723 (time 1761313080.28 total 10.00 last 72.29) - open/close 1599 (time 1761313090.28 total 20.01 last 87.56) - open/close 2461 (time 1761313100.28 total 30.01 last 86.19) - open/close 3338 (time 1761313110.29 total 40.02 last 87.62) - open/close 4212 (time 1761313120.30 total 50.03 last 87.32) - open/close 5026 (time 1761313130.30 total 60.03 last 81.39) - open/close 5865 (time 1761313140.31 total 70.04 last 83.81) - open/close 6685 (time 1761313150.33 total 80.06 last 81.83) - open/close 7445 (time 1761313160.34 total 90.07 last 75.93) - open/close 8228 (time 1761313170.35 total 100.07 last 78.26) - open/close 9041 (time 1761313180.36 total 110.09 last 81.18) - open/close 9908 (time 1761313190.37 total 120.09 last 86.66) total: 10000 open/close in 121.35 seconds: 82.41 ops/second - unlinked 0 (time 1761313194 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 898 (time 1761313279.76 total 10.00 last 89.78) - open/close 1945 (time 1761313289.77 total 20.01 last 104.62) - open/close 2781 (time 1761313299.78 total 30.02 last 83.52) - open/close 3598 (time 1761313309.79 total 40.03 last 81.64) - open/close 4489 (time 1761313319.79 total 50.03 last 89.05) - open/close 5271 (time 1761313329.80 total 60.04 last 78.14) - open/close 6025 (time 1761313339.81 total 70.05 last 75.33) - open/close 6791 (time 1761313349.81 total 80.05 last 76.60) - open/close 7594 (time 1761313359.82 total 90.06 last 80.25) - open/close 8390 (time 1761313369.82 total 100.06 last 79.56) - open/close 9167 (time 1761313379.82 total 110.06 last 77.68) - open/close 9940 (time 1761313389.82 total 120.06 last 77.29) total: 10000 open/close in 120.76 seconds: 82.81 ops/second - unlinked 0 (time 1761313393 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 781 (time 1761313482.86 total 10.01 last 78.06) - open/close 1556 (time 1761313492.86 total 20.01 last 77.50) - open/close 2278 (time 1761313502.87 total 30.01 last 72.14) - open/close 3054 (time 1761313512.88 total 40.02 last 77.57) - open/close 3850 (time 1761313522.88 total 50.03 last 79.54) - open/close 4591 (time 1761313532.88 total 60.03 last 74.10) - open/close 5373 (time 1761313542.89 total 70.03 last 78.15) - open/close 6137 (time 1761313552.90 total 80.05 last 76.30) - open/close 6824 (time 1761313562.90 total 90.05 last 68.70) - open/close 7592 (time 1761313572.92 total 100.06 last 76.70) - open/close 8314 (time 1761313582.92 total 110.06 last 72.18) - open/close 9041 (time 1761313592.92 total 120.06 last 72.70) - open/close 9807 (time 1761313602.93 total 130.07 last 76.55) total: 10000 open/close in 132.24 seconds: 75.62 ops/second - unlinked 0 (time 1761313608 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 1007 (time 1761313691.89 total 10.01 last 100.63) - open/close 1901 (time 1761313701.90 total 20.01 last 89.36) - open/close 2838 (time 1761313711.91 total 30.02 last 93.59) - open/close 3636 (time 1761313721.91 total 40.02 last 79.79) - open/close 4522 (time 1761313731.92 total 50.04 last 88.49) - open/close 5402 (time 1761313741.92 total 60.04 last 87.99) - open/close 6319 (time 1761313751.92 total 70.04 last 91.68) - open/close 7323 (time 1761313761.95 total 80.07 last 100.13) - open/close 8069 (time 1761313771.96 total 90.07 last 74.55) - open/close 8937 (time 1761313781.96 total 100.08 last 86.75) - open/close 9831 (time 1761313791.97 total 110.09 last 89.30) total: 10000 open/close in 112.12 seconds: 89.19 ops/second - unlinked 0 (time 1761313796 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 811 (time 1761313876.74 total 10.00 last 81.07) - open/close 1672 (time 1761313886.74 total 20.01 last 86.08) - open/close 2497 (time 1761313896.74 total 30.01 last 82.47) - open/close 3337 (time 1761313906.75 total 40.01 last 83.98) - open/close 4219 (time 1761313916.75 total 50.02 last 88.17) - open/close 5088 (time 1761313926.76 total 60.02 last 86.85) - open/close 5887 (time 1761313936.76 total 70.03 last 79.85) - open/close 6745 (time 1761313946.77 total 80.03 last 85.75) - open/close 7541 (time 1761313956.78 total 90.05 last 79.50) - open/close 8318 (time 1761313966.79 total 100.05 last 77.65) - open/close 9139 (time 1761313976.79 total 110.05 last 82.09) - open/close 9957 (time 1761313986.80 total 120.07 last 81.71) total: 10000 open/close in 120.66 seconds: 82.88 ops/second - unlinked 0 (time 1761313990 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 895 (time 1761314075.34 total 10.00 last 89.47) - open/close 1737 (time 1761314085.36 total 20.02 last 84.09) - open/close 2591 (time 1761314095.37 total 30.03 last 85.30) - open/close 3401 (time 1761314105.39 total 40.05 last 80.85) - open/close 4223 (time 1761314115.40 total 50.05 last 82.14) - open/close 5042 (time 1761314125.40 total 60.06 last 81.84) - open/close 5823 (time 1761314135.41 total 70.07 last 78.03) - open/close 6744 (time 1761314145.42 total 80.08 last 92.02) - open/close 7668 (time 1761314155.43 total 90.09 last 92.31) - open/close 8470 (time 1761314165.44 total 100.10 last 80.10) - open/close 9248 (time 1761314175.44 total 110.10 last 77.80) - open/close 9984 (time 1761314185.45 total 120.11 last 73.56) total: 10000 open/close in 120.27 seconds: 83.15 ops/second - unlinked 0 (time 1761314189 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 776 (time 1761314279.49 total 10.00 last 77.59) - open/close 1577 (time 1761314289.50 total 20.01 last 80.05) - open/close 2370 (time 1761314299.50 total 30.01 last 79.30) - open/close 3138 (time 1761314309.50 total 40.01 last 76.77) - open/close 3902 (time 1761314319.51 total 50.02 last 76.31) - open/close 4632 (time 1761314329.52 total 60.03 last 72.97) - open/close 5394 (time 1761314339.52 total 70.03 last 76.19) - open/close 6227 (time 1761314349.52 total 80.03 last 83.25) - open/close 7007 (time 1761314359.53 total 90.04 last 77.95) - open/close 7826 (time 1761314369.53 total 100.04 last 81.89) - open/close 8634 (time 1761314379.54 total 110.05 last 80.76) - open/close 9324 (time 1761314389.55 total 120.06 last 68.94) total: 10000 open/close in 129.29 seconds: 77.35 ops/second - unlinked 0 (time 1761314402 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 795 (time 1761314483.58 total 10.00 last 79.49) - open/close 1584 (time 1761314493.58 total 20.00 last 78.87) - open/close 2440 (time 1761314503.60 total 30.02 last 85.50) - open/close 3228 (time 1761314513.60 total 40.02 last 78.77) - open/close 3998 (time 1761314523.61 total 50.03 last 76.92) - open/close 4734 (time 1761314533.61 total 60.03 last 73.60) - open/close 5472 (time 1761314543.61 total 70.03 last 73.79) - open/close 6238 (time 1761314553.61 total 80.03 last 76.60) - open/close 6949 (time 1761314563.61 total 90.03 last 71.10) - open/close 7680 (time 1761314573.63 total 100.05 last 72.96) - open/close 8392 (time 1761314583.65 total 110.07 last 71.08) - open/close 9160 (time 1761314593.65 total 120.07 last 76.78) - open/close 9962 (time 1761314603.66 total 130.08 last 80.11) total: 10000 open/close in 130.52 seconds: 76.62 ops/second - unlinked 0 (time 1761314607 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 740 (time 1761314690.29 total 10.00 last 73.99) - open/close 1530 (time 1761314700.30 total 20.01 last 78.96) - open/close 2295 (time 1761314710.31 total 30.01 last 76.45) - open/close 3109 (time 1761314720.31 total 40.02 last 81.39) - open/close 3910 (time 1761314730.31 total 50.02 last 80.10) - open/close 4715 (time 1761314740.31 total 60.02 last 80.47) - open/close 5635 (time 1761314750.32 total 70.02 last 91.95) - open/close 6538 (time 1761314760.32 total 80.03 last 90.28) - open/close 7397 (time 1761314770.34 total 90.05 last 85.74) - open/close 8270 (time 1761314780.34 total 100.05 last 87.28) - open/close 9108 (time 1761314790.34 total 110.05 last 83.78) - open/close 9861 (time 1761314800.36 total 120.07 last 75.19) total: 10000 open/close in 122.13 seconds: 81.88 ops/second - unlinked 0 (time 1761314806 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 842 (time 1761314892.90 total 10.01 last 84.12) - open/close 1691 (time 1761314902.90 total 20.02 last 84.82) - open/close 2496 (time 1761314912.91 total 30.02 last 80.47) - open/close 3354 (time 1761314922.92 total 40.04 last 85.69) - open/close 4160 (time 1761314932.93 total 50.04 last 80.54) - open/close 4953 (time 1761314942.94 total 60.05 last 79.25) - open/close 5876 (time 1761314952.94 total 70.05 last 92.29) - open/close 6848 (time 1761314962.95 total 80.06 last 97.09) - open/close 7730 (time 1761314972.96 total 90.07 last 88.14) - open/close 8687 (time 1761314982.96 total 100.07 last 95.68) - open/close 9828 (time 1761314992.96 total 110.08 last 114.07) total: 10000 open/close in 111.89 seconds: 89.38 ops/second - unlinked 0 (time 1761314997 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.111@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.111@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg311-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg311-server: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg311-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock oleg311-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing unload_modules_local oleg311-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg311-server: LNET ready to unload oleg311-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2