== conf-sanity test 69: replace an OST with the same index ========================================================== 18:41:38 (1753742498) start mds service on oleg222-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 558 (time 1753742578.58 total 10.01 last 55.76) - open/close 1165 (time 1753742588.59 total 20.01 last 60.67) - open/close 1817 (time 1753742598.59 total 30.02 last 65.14) - open/close 2446 (time 1753742608.60 total 40.02 last 62.89) - open/close 3073 (time 1753742618.60 total 50.03 last 62.66) - open/close 3675 (time 1753742628.61 total 60.04 last 60.16) - open/close 4270 (time 1753742638.62 total 70.05 last 59.43) - open/close 4942 (time 1753742648.62 total 80.05 last 67.19) - open/close 5577 (time 1753742658.63 total 90.06 last 63.43) - open/close 6226 (time 1753742668.65 total 100.08 last 64.80) - open/close 6883 (time 1753742678.66 total 110.08 last 65.64) - open/close 7481 (time 1753742688.66 total 120.09 last 59.80) - open/close 8020 (time 1753742698.67 total 130.09 last 53.86) - open/close 8567 (time 1753742708.69 total 140.12 last 54.57) - open/close 9139 (time 1753742718.70 total 150.12 last 57.15) - open/close 9706 (time 1753742728.70 total 160.13 last 56.68) total: 10000 open/close in 165.57 seconds: 60.40 ops/second - unlinked 0 (time 1753742738 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 554 (time 1753742843.61 total 10.00 last 55.39) - open/close 1152 (time 1753742853.64 total 20.03 last 59.63) - open/close 1750 (time 1753742863.66 total 30.05 last 59.69) - open/close 2374 (time 1753742873.67 total 40.05 last 62.36) - open/close 3031 (time 1753742883.68 total 50.06 last 65.64) - open/close 3672 (time 1753742893.69 total 60.07 last 64.04) - open/close 4277 (time 1753742903.69 total 70.08 last 60.49) - open/close 4904 (time 1753742913.69 total 80.08 last 62.66) - open/close 5530 (time 1753742923.70 total 90.09 last 62.55) - open/close 6156 (time 1753742933.72 total 100.10 last 62.51) - open/close 6724 (time 1753742943.72 total 110.11 last 56.77) - open/close 7297 (time 1753742953.73 total 120.12 last 57.24) - open/close 7842 (time 1753742963.74 total 130.13 last 54.47) - open/close 8475 (time 1753742973.76 total 140.15 last 63.17) - open/close 9086 (time 1753742983.76 total 150.15 last 61.06) - open/close 9644 (time 1753742993.77 total 160.16 last 55.76) total: 10000 open/close in 166.05 seconds: 60.22 ops/second - unlinked 0 (time 1753743004 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 515 (time 1753743116.78 total 10.01 last 51.47) - open/close 1045 (time 1753743126.79 total 20.02 last 52.94) - open/close 1646 (time 1753743136.79 total 30.02 last 60.09) - open/close 2244 (time 1753743146.81 total 40.04 last 59.68) - open/close 2840 (time 1753743156.82 total 50.05 last 59.55) - open/close 3431 (time 1753743166.84 total 60.07 last 58.97) - open/close 4022 (time 1753743176.85 total 70.07 last 59.09) - open/close 4566 (time 1753743186.85 total 80.08 last 54.37) - open/close 5218 (time 1753743196.86 total 90.09 last 65.12) - open/close 5832 (time 1753743206.87 total 100.09 last 61.37) - open/close 6404 (time 1753743216.87 total 110.10 last 57.17) - open/close 7026 (time 1753743226.88 total 120.11 last 62.15) - open/close 7547 (time 1753743236.89 total 130.12 last 52.04) - open/close 7946 (time 1753743246.91 total 140.14 last 39.83) - open/close 8491 (time 1753743256.92 total 150.14 last 54.46) - open/close 9017 (time 1753743266.92 total 160.15 last 52.59) - open/close 9585 (time 1753743276.93 total 170.15 last 56.75) total: 10000 open/close in 177.68 seconds: 56.28 ops/second - unlinked 0 (time 1753743288 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 630 (time 1753743396.95 total 10.01 last 62.93) - open/close 1245 (time 1753743406.95 total 20.01 last 61.50) - open/close 1862 (time 1753743416.95 total 30.01 last 61.70) - open/close 2500 (time 1753743426.95 total 40.01 last 63.80) - open/close 3151 (time 1753743436.97 total 50.03 last 64.97) - open/close 3814 (time 1753743446.97 total 60.04 last 66.26) - open/close 4434 (time 1753743456.98 total 70.04 last 61.96) - open/close 5011 (time 1753743466.98 total 80.05 last 57.69) - open/close 5624 (time 1753743476.99 total 90.05 last 61.26) - open/close 6238 (time 1753743487.00 total 100.06 last 61.35) - open/close 6816 (time 1753743497.00 total 110.07 last 57.76) - open/close 7419 (time 1753743507.01 total 120.07 last 60.25) - open/close 8039 (time 1753743517.02 total 130.08 last 61.94) - open/close 8687 (time 1753743527.03 total 140.09 last 64.74) - open/close 9311 (time 1753743537.05 total 150.12 last 62.25) - open/close 9844 (time 1753743547.06 total 160.12 last 53.27) total: 10000 open/close in 163.00 seconds: 61.35 ops/second - unlinked 0 (time 1753743554 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 482 (time 1753743662.40 total 10.00 last 48.19) - open/close 905 (time 1753743672.45 total 20.06 last 42.08) - open/close 1559 (time 1753743682.47 total 30.07 last 65.31) - open/close 2219 (time 1753743692.48 total 40.08 last 65.94) - open/close 2878 (time 1753743702.49 total 50.09 last 65.82) - open/close 3513 (time 1753743712.51 total 60.11 last 63.39) - open/close 4203 (time 1753743722.51 total 70.11 last 68.98) - open/close 4837 (time 1753743732.52 total 80.12 last 63.34) - open/close 5454 (time 1753743742.53 total 90.13 last 61.63) - open/close 6083 (time 1753743752.56 total 100.16 last 62.74) - open/close 6681 (time 1753743762.57 total 110.17 last 59.74) - open/close 7330 (time 1753743772.57 total 120.17 last 64.88) - open/close 7920 (time 1753743782.58 total 130.18 last 58.96) - open/close 8481 (time 1753743792.58 total 140.19 last 56.07) - open/close 9120 (time 1753743802.59 total 150.20 last 63.84) - open/close 9682 (time 1753743812.59 total 160.20 last 56.20) total: 10000 open/close in 165.65 seconds: 60.37 ops/second - unlinked 0 (time 1753743822 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 560 (time 1753743932.83 total 10.01 last 55.97) - open/close 1159 (time 1753743942.84 total 20.02 last 59.82) - open/close 1804 (time 1753743952.85 total 30.03 last 64.43) - open/close 2433 (time 1753743962.86 total 40.04 last 62.86) - open/close 3107 (time 1753743972.87 total 50.05 last 67.31) - open/close 3750 (time 1753743982.88 total 60.05 last 64.29) - open/close 4380 (time 1753743992.88 total 70.06 last 62.96) - open/close 5027 (time 1753744002.90 total 80.08 last 64.57) - open/close 5667 (time 1753744012.93 total 90.10 last 63.84) - open/close 6241 (time 1753744022.94 total 100.11 last 57.32) - open/close 6831 (time 1753744032.95 total 110.12 last 58.95) - open/close 7490 (time 1753744042.95 total 120.13 last 65.86) - open/close 8154 (time 1753744052.97 total 130.14 last 66.32) - open/close 8839 (time 1753744062.98 total 140.15 last 68.44) - open/close 9534 (time 1753744072.98 total 150.16 last 69.44) total: 10000 open/close in 156.94 seconds: 63.72 ops/second - unlinked 0 (time 1753744084 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 661 (time 1753744187.71 total 10.01 last 66.03) - open/close 1302 (time 1753744197.71 total 20.02 last 64.05) - open/close 1907 (time 1753744207.71 total 30.02 last 60.49) - open/close 2534 (time 1753744217.72 total 40.02 last 62.69) - open/close 3178 (time 1753744227.72 total 50.03 last 64.37) - open/close 3807 (time 1753744237.72 total 60.03 last 62.89) - open/close 4437 (time 1753744247.72 total 70.03 last 63.00) - open/close 5087 (time 1753744257.72 total 80.03 last 65.00) - open/close 5733 (time 1753744267.74 total 90.04 last 64.50) - open/close 6367 (time 1753744277.75 total 100.05 last 63.34) - open/close 7075 (time 1753744287.76 total 110.06 last 70.75) - open/close 7796 (time 1753744297.77 total 120.07 last 72.01) - open/close 8511 (time 1753744307.77 total 130.08 last 71.47) - open/close 9213 (time 1753744317.77 total 140.08 last 70.18) - open/close 9920 (time 1753744327.78 total 150.09 last 70.65) total: 10000 open/close in 151.19 seconds: 66.14 ops/second - unlinked 0 (time 1753744333 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 674 (time 1753744426.48 total 10.01 last 67.33) - open/close 1429 (time 1753744436.48 total 20.01 last 75.48) - open/close 2155 (time 1753744446.48 total 30.02 last 72.58) - open/close 2894 (time 1753744456.49 total 40.03 last 73.82) - open/close 3663 (time 1753744466.50 total 50.04 last 76.83) - open/close 4371 (time 1753744476.52 total 60.05 last 70.70) - open/close 4960 (time 1753744486.52 total 70.06 last 58.87) - open/close 5608 (time 1753744496.52 total 80.06 last 64.79) - open/close 6306 (time 1753744506.54 total 90.07 last 69.72) - open/close 6933 (time 1753744516.55 total 100.08 last 62.64) - open/close 7593 (time 1753744526.55 total 110.08 last 65.97) - open/close 8268 (time 1753744536.57 total 120.10 last 67.40) - open/close 8903 (time 1753744546.58 total 130.12 last 63.39) - open/close 9576 (time 1753744556.59 total 140.13 last 67.23) total: 10000 open/close in 146.71 seconds: 68.16 ops/second - unlinked 0 (time 1753744566 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 697 (time 1753744666.01 total 10.01 last 69.62) - open/close 1403 (time 1753744676.01 total 20.01 last 70.59) - open/close 2212 (time 1753744686.01 total 30.01 last 80.90) - open/close 2972 (time 1753744696.02 total 40.02 last 75.92) - open/close 3789 (time 1753744706.04 total 50.04 last 81.58) - open/close 4567 (time 1753744716.05 total 60.06 last 77.66) - open/close 5311 (time 1753744726.06 total 70.06 last 74.38) - open/close 6073 (time 1753744736.06 total 80.06 last 76.19) - open/close 6808 (time 1753744746.08 total 90.08 last 73.39) - open/close 7557 (time 1753744756.08 total 100.08 last 74.87) - open/close 8253 (time 1753744766.09 total 110.09 last 69.53) - open/close 8995 (time 1753744776.09 total 120.09 last 74.17) - open/close 9859 (time 1753744786.09 total 130.10 last 86.39) total: 10000 open/close in 131.76 seconds: 75.89 ops/second - unlinked 0 (time 1753744790 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 727 (time 1753744879.32 total 10.00 last 72.67) - open/close 1446 (time 1753744889.33 total 20.01 last 71.87) - open/close 2130 (time 1753744899.34 total 30.02 last 68.31) - open/close 2863 (time 1753744909.35 total 40.04 last 73.18) - open/close 3621 (time 1753744919.37 total 50.05 last 75.70) - open/close 4386 (time 1753744929.37 total 60.05 last 76.48) - open/close 5096 (time 1753744939.38 total 70.07 last 70.92) - open/close 5783 (time 1753744949.39 total 80.08 last 68.62) - open/close 6510 (time 1753744959.40 total 90.09 last 72.64) - open/close 7252 (time 1753744969.41 total 100.10 last 74.12) - open/close 7973 (time 1753744979.43 total 110.12 last 71.96) - open/close 8672 (time 1753744989.43 total 120.12 last 69.90) - open/close 9310 (time 1753744999.46 total 130.14 last 63.64) - open/close 9994 (time 1753745009.47 total 140.16 last 68.30) total: 10000 open/close in 140.23 seconds: 71.31 ops/second - unlinked 0 (time 1753745012 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.122@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg222-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg222-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2