== conf-sanity test 69: replace an OST with the same index ========================================================== 18:45:59 (1786747559) start mds service on oleg429-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing load_modules_local oleg429-server: Loading modules from /home/green/git/lustre-release/lustre oleg429-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg429-server: detected 4 online CPUs by sysfs oleg429-server: MODOPTS_LIBCFS= oleg429-server: Force libcfs to create 2 CPU partitions oleg429-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg429-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg429-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg429-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 631 (time 1786747656.39 total 10.04 last 62.87) - open/close 1304 (time 1786747666.41 total 20.05 last 67.19) - open/close 1900 (time 1786747676.42 total 30.06 last 59.53) - open/close 2520 (time 1786747686.43 total 40.08 last 61.93) - open/close 3096 (time 1786747696.44 total 50.08 last 57.56) - open/close 3776 (time 1786747706.46 total 60.11 last 67.85) - open/close 4451 (time 1786747716.46 total 70.11 last 67.50) - open/close 5115 (time 1786747726.47 total 80.11 last 66.37) - open/close 5739 (time 1786747736.47 total 90.12 last 62.36) - open/close 6391 (time 1786747746.48 total 100.13 last 65.13) - open/close 7020 (time 1786747756.50 total 110.15 last 62.76) - open/close 7626 (time 1786747766.51 total 120.15 last 60.58) - open/close 8334 (time 1786747776.52 total 130.16 last 70.72) - open/close 8990 (time 1786747786.54 total 140.18 last 65.48) - open/close 9648 (time 1786747796.54 total 150.19 last 65.75) total: 10000 open/close in 155.50 seconds: 64.31 ops/second - unlinked 0 (time 1786747806 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 592 (time 1786747921.77 total 10.02 last 59.11) - open/close 1194 (time 1786747931.77 total 20.02 last 60.20) - open/close 1828 (time 1786747941.78 total 30.02 last 63.38) - open/close 2443 (time 1786747951.78 total 40.02 last 61.47) - open/close 3069 (time 1786747961.80 total 50.04 last 62.48) - open/close 3711 (time 1786747971.81 total 60.05 last 64.13) - open/close 4412 (time 1786747981.82 total 70.06 last 70.06) - open/close 5036 (time 1786747991.82 total 80.07 last 62.35) - open/close 5680 (time 1786748001.83 total 90.07 last 64.38) - open/close 6342 (time 1786748011.83 total 100.07 last 66.18) - open/close 6909 (time 1786748021.83 total 110.08 last 56.68) - open/close 7493 (time 1786748031.85 total 120.09 last 58.31) - open/close 8117 (time 1786748041.86 total 130.11 last 62.32) - open/close 8682 (time 1786748051.88 total 140.13 last 56.38) - open/close 9259 (time 1786748061.89 total 150.13 last 57.68) - open/close 9798 (time 1786748071.92 total 160.16 last 53.73) total: 10000 open/close in 164.99 seconds: 60.61 ops/second - unlinked 0 (time 1786748082 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 661 (time 1786748197.82 total 10.01 last 66.06) - open/close 1356 (time 1786748207.83 total 20.02 last 69.43) - open/close 1981 (time 1786748217.83 total 30.02 last 62.49) - open/close 2619 (time 1786748227.84 total 40.03 last 63.73) - open/close 3199 (time 1786748237.85 total 50.04 last 57.94) - open/close 3854 (time 1786748247.86 total 60.05 last 65.46) - open/close 4487 (time 1786748257.87 total 70.05 last 63.25) - open/close 5101 (time 1786748267.87 total 80.06 last 61.39) - open/close 5746 (time 1786748277.88 total 90.07 last 64.43) - open/close 6409 (time 1786748287.89 total 100.08 last 66.22) - open/close 7082 (time 1786748297.91 total 110.10 last 67.18) - open/close 7597 (time 1786748307.91 total 120.10 last 51.48) - open/close 8145 (time 1786748317.93 total 130.12 last 54.68) - open/close 8811 (time 1786748327.94 total 140.13 last 66.56) - open/close 9458 (time 1786748337.94 total 150.13 last 64.69) total: 10000 open/close in 158.89 seconds: 62.94 ops/second - unlinked 0 (time 1786748351 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 574 (time 1786748467.34 total 10.00 last 57.37) - open/close 1171 (time 1786748477.35 total 20.02 last 59.63) - open/close 1815 (time 1786748487.36 total 30.03 last 64.30) - open/close 2443 (time 1786748497.38 total 40.04 last 62.72) - open/close 3112 (time 1786748507.38 total 50.05 last 66.86) - open/close 3687 (time 1786748517.39 total 60.05 last 57.47) - open/close 4248 (time 1786748527.40 total 70.07 last 56.02) - open/close 4846 (time 1786748537.41 total 80.08 last 59.73) - open/close 5401 (time 1786748547.42 total 90.08 last 55.48) - open/close 6013 (time 1786748557.42 total 100.09 last 61.15) - open/close 6609 (time 1786748567.43 total 110.09 last 59.59) - open/close 7206 (time 1786748577.43 total 120.10 last 59.65) - open/close 7841 (time 1786748587.45 total 130.12 last 63.41) - open/close 8473 (time 1786748597.46 total 140.13 last 63.09) - open/close 9056 (time 1786748607.48 total 150.15 last 58.22) - open/close 9680 (time 1786748617.49 total 160.15 last 62.34) total: 10000 open/close in 165.08 seconds: 60.58 ops/second - unlinked 0 (time 1786748626 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 531 (time 1786748734.68 total 10.01 last 53.03) - open/close 1102 (time 1786748744.69 total 20.02 last 57.05) - open/close 1681 (time 1786748754.69 total 30.02 last 57.90) - open/close 2289 (time 1786748764.69 total 40.02 last 60.80) - open/close 2913 (time 1786748774.70 total 50.03 last 62.37) - open/close 3514 (time 1786748784.71 total 60.04 last 60.04) - open/close 4092 (time 1786748794.71 total 70.04 last 57.78) - open/close 4686 (time 1786748804.72 total 80.05 last 59.32) - open/close 5301 (time 1786748814.72 total 90.06 last 61.49) - open/close 5896 (time 1786748824.72 total 100.06 last 59.49) - open/close 6449 (time 1786748834.73 total 110.06 last 55.28) - open/close 7066 (time 1786748844.74 total 120.07 last 61.61) - open/close 7627 (time 1786748854.76 total 130.09 last 56.01) - open/close 8226 (time 1786748864.76 total 140.09 last 59.89) - open/close 8909 (time 1786748874.76 total 150.09 last 68.30) - open/close 9552 (time 1786748884.77 total 160.10 last 64.23) total: 10000 open/close in 166.81 seconds: 59.95 ops/second - unlinked 0 (time 1786748895 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 565 (time 1786749009.20 total 10.01 last 56.43) - open/close 1192 (time 1786749019.20 total 20.01 last 62.69) - open/close 1794 (time 1786749029.21 total 30.02 last 60.16) - open/close 2408 (time 1786749039.23 total 40.04 last 61.28) - open/close 3034 (time 1786749049.23 total 50.04 last 62.60) - open/close 3677 (time 1786749059.24 total 60.05 last 64.24) - open/close 4266 (time 1786749069.24 total 70.06 last 58.85) - open/close 4815 (time 1786749079.25 total 80.06 last 54.87) - open/close 5339 (time 1786749089.25 total 90.06 last 52.39) - open/close 5753 (time 1786749099.26 total 100.08 last 41.35) - open/close 6333 (time 1786749109.28 total 110.09 last 57.90) - open/close 6969 (time 1786749119.28 total 120.09 last 63.60) - open/close 7639 (time 1786749129.29 total 130.10 last 66.94) - open/close 8326 (time 1786749139.30 total 140.12 last 68.61) - open/close 8901 (time 1786749149.33 total 150.14 last 57.34) - open/close 9399 (time 1786749159.33 total 160.15 last 49.79) - open/close 9906 (time 1786749169.33 total 170.15 last 50.70) total: 10000 open/close in 171.60 seconds: 58.28 ops/second - unlinked 0 (time 1786749175 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 588 (time 1786749297.62 total 10.00 last 58.77) - open/close 1229 (time 1786749307.63 total 20.01 last 64.09) - open/close 1840 (time 1786749317.63 total 30.01 last 61.09) - open/close 2436 (time 1786749327.63 total 40.01 last 59.56) - open/close 3089 (time 1786749337.65 total 50.03 last 65.22) - open/close 3694 (time 1786749347.66 total 60.04 last 60.40) - open/close 4377 (time 1786749357.66 total 70.04 last 68.30) - open/close 5023 (time 1786749367.67 total 80.05 last 64.58) - open/close 5614 (time 1786749377.68 total 90.06 last 59.02) - open/close 6212 (time 1786749387.69 total 100.07 last 59.72) - open/close 6683 (time 1786749397.71 total 110.09 last 47.02) - open/close 7201 (time 1786749407.72 total 120.10 last 51.76) - open/close 7726 (time 1786749417.76 total 130.14 last 52.30) - open/close 8261 (time 1786749427.77 total 140.15 last 53.43) - open/close 8690 (time 1786749437.82 total 150.20 last 42.70) - open/close 9166 (time 1786749447.83 total 160.21 last 47.55) - open/close 9659 (time 1786749457.83 total 170.21 last 49.29) total: 10000 open/close in 177.15 seconds: 56.45 ops/second - unlinked 0 (time 1786749469 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 651 (time 1786749585.21 total 10.03 last 64.92) - open/close 1237 (time 1786749595.22 total 20.04 last 58.55) - open/close 1844 (time 1786749605.24 total 30.06 last 60.57) - open/close 2429 (time 1786749615.25 total 40.07 last 58.44) - open/close 3024 (time 1786749625.25 total 50.07 last 59.49) - open/close 3618 (time 1786749635.26 total 60.07 last 59.38) - open/close 4145 (time 1786749645.26 total 70.08 last 52.69) - open/close 4741 (time 1786749655.26 total 80.08 last 59.57) - open/close 5322 (time 1786749665.27 total 90.09 last 58.05) - open/close 5924 (time 1786749675.29 total 100.11 last 60.09) - open/close 6532 (time 1786749685.29 total 110.11 last 60.79) - open/close 7185 (time 1786749695.30 total 120.12 last 65.23) - open/close 7702 (time 1786749705.31 total 130.12 last 51.68) - open/close 8258 (time 1786749715.31 total 140.13 last 55.58) - open/close 8866 (time 1786749725.31 total 150.13 last 60.79) - open/close 9456 (time 1786749735.34 total 160.16 last 58.84) total: 10000 open/close in 169.63 seconds: 58.95 ops/second - unlinked 0 (time 1786749749 ; total 0 ; last 0) total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second - open/close 591 (time 1786749875.88 total 10.00 last 59.07) - open/close 1165 (time 1786749885.89 total 20.02 last 57.33) - open/close 1792 (time 1786749895.90 total 30.02 last 62.65) - open/close 2442 (time 1786749905.91 total 40.03 last 64.94) - open/close 3088 (time 1786749915.91 total 50.04 last 64.56) - open/close 3693 (time 1786749925.93 total 60.06 last 60.39) - open/close 4245 (time 1786749935.95 total 70.07 last 55.12) - open/close 4796 (time 1786749945.96 total 80.09 last 55.02) - open/close 5389 (time 1786749955.97 total 90.09 last 59.25) - open/close 5932 (time 1786749965.99 total 100.11 last 54.20) - open/close 6552 (time 1786749976.00 total 110.12 last 61.93) - open/close 7029 (time 1786749986.01 total 120.13 last 47.67) - open/close 7575 (time 1786749996.01 total 130.14 last 54.56) - open/close 8190 (time 1786750006.02 total 140.15 last 61.46) - open/close 8758 (time 1786750016.03 total 150.16 last 56.74) - open/close 9340 (time 1786750026.03 total 160.16 last 58.19) - open/close 9845 (time 1786750036.05 total 170.17 last 50.43) total: 10000 open/close in 173.61 seconds: 57.60 ops/second - unlinked 0 (time 1786750045 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 672 (time 1786750163.36 total 10.02 last 67.04) - open/close 1306 (time 1786750173.36 total 20.02 last 63.40) - open/close 1945 (time 1786750183.37 total 30.03 last 63.88) - open/close 2576 (time 1786750193.37 total 40.03 last 63.09) - open/close 3263 (time 1786750203.38 total 50.04 last 68.64) - open/close 3906 (time 1786750213.38 total 60.04 last 64.28) - open/close 4588 (time 1786750223.38 total 70.05 last 68.17) - open/close 5262 (time 1786750233.39 total 80.05 last 67.37) - open/close 5934 (time 1786750243.40 total 90.06 last 67.12) - open/close 6501 (time 1786750253.42 total 100.08 last 56.61) - open/close 7109 (time 1786750263.42 total 110.08 last 60.76) - open/close 7725 (time 1786750273.42 total 120.09 last 61.59) - open/close 8337 (time 1786750283.43 total 130.09 last 61.14) - open/close 8924 (time 1786750293.44 total 140.10 last 58.65) - open/close 9467 (time 1786750303.47 total 150.13 last 54.15) - open/close 9879 (time 1786750313.49 total 160.15 last 41.13) total: 10000 open/close in 162.35 seconds: 61.60 ops/second - unlinked 0 (time 1786750321 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg429-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.129@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.129@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg429-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg429-server: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg429-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg429-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.129@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.129@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.129@tcp sys.timeout=20 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.129@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.129@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg429-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg429-server: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg429-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg429-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg429-server unloading modules via unload_modules_local on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing unload_modules_local oleg429-server: modules unloaded. pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2