== conf-sanity test 69: replace an OST with the same index ========================================================== 09:04:37 (1761311077) start mds service on oleg137-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg137-server: detected 4 online CPUs by sysfs oleg137-server: Force libcfs to create 2 CPU partitions oleg137-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg137-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg137-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock oleg137-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2001 (time 1761311122.82 total 10.00 last 200.07) - open/close 4540 (time 1761311132.82 total 20.00 last 253.90) - open/close 7738 (time 1761311142.82 total 30.00 last 319.75) - open/close 9691 (time 1761311152.83 total 40.01 last 195.25) total: 10000 open/close in 41.54 seconds: 240.72 ops/second - unlinked 0 (time 1761311156 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 1846 (time 1761311193.10 total 10.00 last 184.59) - open/close 4462 (time 1761311203.10 total 20.00 last 261.53) - open/close 6647 (time 1761311213.10 total 30.00 last 218.47) - open/close 8812 (time 1761311223.11 total 40.01 last 216.43) total: 10000 open/close in 44.48 seconds: 224.80 ops/second - unlinked 0 (time 1761311229 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2024 (time 1761311265.02 total 10.00 last 202.34) - open/close 4598 (time 1761311275.02 total 20.00 last 257.38) - open/close 7672 (time 1761311285.02 total 30.01 last 307.34) - open/close 8587 (time 1761311295.03 total 40.02 last 91.40) - open/close 9268 (time 1761311305.03 total 50.02 last 68.07) total: 10000 open/close in 59.57 seconds: 167.87 ops/second - unlinked 0 (time 1761311318 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 494 (time 1761311411.37 total 10.01 last 49.37) - open/close 1145 (time 1761311421.38 total 20.01 last 65.05) - open/close 1804 (time 1761311431.38 total 30.01 last 65.90) - open/close 2382 (time 1761311441.40 total 40.03 last 57.69) - open/close 3036 (time 1761311451.41 total 50.05 last 65.30) - open/close 3749 (time 1761311461.42 total 60.06 last 71.25) - open/close 4453 (time 1761311471.43 total 70.07 last 70.32) - open/close 5204 (time 1761311481.44 total 80.07 last 75.07) - open/close 5981 (time 1761311491.44 total 90.08 last 77.66) - open/close 6695 (time 1761311501.45 total 100.08 last 71.37) - open/close 7430 (time 1761311511.45 total 110.09 last 73.46) - open/close 8187 (time 1761311521.46 total 120.09 last 75.65) - open/close 8873 (time 1761311531.47 total 130.10 last 68.56) - open/close 9604 (time 1761311541.47 total 140.10 last 73.10) total: 10000 open/close in 145.45 seconds: 68.75 ops/second - unlinked 0 (time 1761311550 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 735 (time 1761311644.80 total 10.00 last 73.48) - open/close 1409 (time 1761311654.80 total 20.01 last 67.34) - open/close 2076 (time 1761311664.81 total 30.02 last 66.66) - open/close 2832 (time 1761311674.81 total 40.02 last 75.59) - open/close 3540 (time 1761311684.82 total 50.02 last 70.76) - open/close 4263 (time 1761311694.82 total 60.03 last 72.29) - open/close 4955 (time 1761311704.83 total 70.04 last 69.14) - open/close 5637 (time 1761311714.84 total 80.04 last 68.15) - open/close 6253 (time 1761311724.84 total 90.05 last 61.55) - open/close 6862 (time 1761311734.85 total 100.06 last 60.85) - open/close 7488 (time 1761311744.85 total 110.06 last 62.60) - open/close 8080 (time 1761311754.86 total 120.06 last 59.17) - open/close 8738 (time 1761311764.86 total 130.07 last 65.78) - open/close 9403 (time 1761311774.87 total 140.07 last 66.46) total: 10000 open/close in 148.76 seconds: 67.22 ops/second - unlinked 0 (time 1761311787 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 639 (time 1761311889.72 total 10.00 last 63.90) - open/close 1327 (time 1761311899.72 total 20.00 last 68.79) - open/close 1994 (time 1761311909.73 total 30.01 last 66.66) - open/close 2640 (time 1761311919.74 total 40.01 last 64.56) - open/close 3339 (time 1761311929.75 total 50.03 last 69.80) - open/close 4012 (time 1761311939.75 total 60.03 last 67.28) - open/close 4706 (time 1761311949.76 total 70.04 last 69.34) - open/close 5408 (time 1761311959.77 total 80.05 last 70.15) - open/close 6074 (time 1761311969.77 total 90.05 last 66.58) - open/close 6770 (time 1761311979.77 total 100.05 last 69.59) - open/close 7461 (time 1761311989.78 total 110.06 last 69.07) - open/close 8126 (time 1761311999.78 total 120.06 last 66.45) - open/close 8793 (time 1761312009.79 total 130.07 last 66.66) - open/close 9422 (time 1761312019.79 total 140.07 last 62.87) total: 10000 open/close in 148.64 seconds: 67.28 ops/second - unlinked 0 (time 1761312031 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 681 (time 1761312131.19 total 10.00 last 68.08) - open/close 1324 (time 1761312141.19 total 20.01 last 64.28) - open/close 1993 (time 1761312151.20 total 30.01 last 66.86) - open/close 2643 (time 1761312161.20 total 40.02 last 64.97) - open/close 3337 (time 1761312171.21 total 50.03 last 69.33) - open/close 4006 (time 1761312181.22 total 60.04 last 66.85) - open/close 4692 (time 1761312191.23 total 70.05 last 68.54) - open/close 5381 (time 1761312201.24 total 80.06 last 68.83) - open/close 6061 (time 1761312211.24 total 90.06 last 67.97) - open/close 6740 (time 1761312221.24 total 100.06 last 67.88) - open/close 7409 (time 1761312231.26 total 110.08 last 66.77) - open/close 8029 (time 1761312241.27 total 120.09 last 61.94) - open/close 8662 (time 1761312251.28 total 130.10 last 63.26) - open/close 9331 (time 1761312261.29 total 140.11 last 66.82) total: 10000 open/close in 150.03 seconds: 66.65 ops/second - unlinked 0 (time 1761312275 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 680 (time 1761312371.77 total 10.01 last 67.91) - open/close 1356 (time 1761312381.79 total 20.03 last 67.52) - open/close 2071 (time 1761312391.79 total 30.03 last 71.46) - open/close 2778 (time 1761312401.80 total 40.04 last 70.64) - open/close 3496 (time 1761312411.82 total 50.06 last 71.67) - open/close 4172 (time 1761312421.82 total 60.06 last 67.60) - open/close 4832 (time 1761312431.83 total 70.07 last 65.94) - open/close 5495 (time 1761312441.83 total 80.07 last 66.30) - open/close 6203 (time 1761312451.83 total 90.07 last 70.76) - open/close 6920 (time 1761312461.85 total 100.09 last 71.60) - open/close 7658 (time 1761312471.86 total 110.10 last 73.74) - open/close 8354 (time 1761312481.86 total 120.10 last 69.53) - open/close 9044 (time 1761312491.88 total 130.12 last 68.89) - open/close 9777 (time 1761312501.88 total 140.12 last 73.27) total: 10000 open/close in 143.20 seconds: 69.83 ops/second - unlinked 0 (time 1761312508 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 839 (time 1761312594.71 total 10.00 last 83.89) - open/close 1745 (time 1761312604.72 total 20.01 last 90.52) - open/close 2653 (time 1761312614.72 total 30.02 last 90.74) - open/close 3499 (time 1761312624.73 total 40.02 last 84.56) - open/close 4268 (time 1761312634.73 total 50.02 last 76.87) - open/close 5171 (time 1761312644.73 total 60.03 last 90.28) - open/close 6027 (time 1761312654.74 total 70.03 last 85.56) - open/close 6801 (time 1761312664.75 total 80.04 last 77.31) - open/close 7598 (time 1761312674.75 total 90.05 last 79.66) - open/close 8350 (time 1761312684.76 total 100.05 last 75.17) - open/close 9197 (time 1761312694.76 total 110.05 last 84.69) total: 10000 open/close in 119.15 seconds: 83.93 ops/second - unlinked 0 (time 1761312706 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 796 (time 1761312788.90 total 10.00 last 79.59) - open/close 1640 (time 1761312798.90 total 20.00 last 84.38) - open/close 2417 (time 1761312808.91 total 30.01 last 77.68) - open/close 3216 (time 1761312818.93 total 40.03 last 79.74) - open/close 3969 (time 1761312828.94 total 50.04 last 75.23) - open/close 4756 (time 1761312838.94 total 60.04 last 78.66) - open/close 5531 (time 1761312848.94 total 70.04 last 77.50) - open/close 6282 (time 1761312858.95 total 80.05 last 75.05) - open/close 7037 (time 1761312868.95 total 90.05 last 75.47) - open/close 7774 (time 1761312878.96 total 100.06 last 73.66) - open/close 8473 (time 1761312888.97 total 110.07 last 69.84) - open/close 9298 (time 1761312898.97 total 120.07 last 82.49) total: 10000 open/close in 129.64 seconds: 77.14 ops/second - unlinked 0 (time 1761312911 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.137@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.137@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg137-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg137-server: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg137-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock oleg137-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing unload_modules_local oleg137-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg137-server: LNET ready to unload oleg137-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2