== conf-sanity test 69: replace an OST with the same index ========================================================== 01:25:32 (1785389132) start mds service on oleg423-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions oleg423-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg423-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg423-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 562 (time 1785389238.92 total 10.02 last 56.10) - open/close 1133 (time 1785389248.94 total 20.03 last 57.02) - open/close 1732 (time 1785389258.95 total 30.05 last 59.81) - open/close 2522 (time 1785389268.96 total 40.05 last 78.99) - open/close 3170 (time 1785389278.97 total 50.06 last 64.73) - open/close 3660 (time 1785389288.97 total 60.07 last 48.97) - open/close 4215 (time 1785389299.01 total 70.10 last 55.31) - open/close 4581 (time 1785389309.02 total 80.11 last 36.57) - open/close 4994 (time 1785389319.03 total 90.12 last 41.25) - open/close 5743 (time 1785389329.04 total 100.13 last 74.84) - open/close 6625 (time 1785389339.05 total 110.14 last 88.12) - open/close 7231 (time 1785389349.06 total 120.16 last 60.51) - open/close 8088 (time 1785389359.07 total 130.16 last 85.62) - open/close 8646 (time 1785389369.08 total 140.17 last 55.77) - open/close 9166 (time 1785389379.08 total 150.18 last 51.96) - open/close 9840 (time 1785389389.10 total 160.19 last 67.32) total: 10000 open/close in 163.59 seconds: 61.13 ops/second - unlinked 0 (time 1785389397 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 685 (time 1785389516.66 total 10.00 last 68.48) - open/close 1365 (time 1785389526.67 total 20.01 last 67.97) - open/close 1852 (time 1785389536.70 total 30.03 last 48.57) - open/close 2359 (time 1785389546.74 total 40.08 last 50.47) - open/close 2675 (time 1785389556.75 total 50.09 last 31.57) - open/close 3014 (time 1785389566.75 total 60.09 last 33.89) - open/close 3458 (time 1785389576.76 total 70.10 last 44.38) - open/close 4026 (time 1785389586.78 total 80.12 last 56.67) - open/close 4837 (time 1785389596.79 total 90.13 last 81.04) - open/close 5742 (time 1785389606.79 total 100.13 last 90.48) - open/close 6272 (time 1785389616.80 total 110.14 last 52.93) - open/close 6829 (time 1785389626.81 total 120.14 last 55.69) - open/close 7299 (time 1785389636.82 total 130.16 last 46.92) - open/close 7782 (time 1785389646.83 total 140.17 last 48.26) - open/close 8290 (time 1785389656.85 total 150.19 last 50.72) - open/close 8770 (time 1785389666.85 total 160.19 last 47.97) - open/close 9379 (time 1785389676.86 total 170.20 last 60.85) - open/close 9860 (time 1785389686.87 total 180.20 last 48.08) total: 10000 open/close in 182.74 seconds: 54.72 ops/second - unlinked 0 (time 1785389695 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 514 (time 1785389811.99 total 10.02 last 51.32) - open/close 971 (time 1785389821.99 total 20.02 last 45.68) - open/close 1747 (time 1785389832.00 total 30.03 last 77.55) - open/close 2546 (time 1785389842.01 total 40.04 last 79.82) - open/close 3428 (time 1785389852.01 total 50.04 last 88.15) - open/close 4197 (time 1785389862.02 total 60.05 last 76.87) - open/close 4678 (time 1785389872.04 total 70.07 last 48.01) - open/close 5325 (time 1785389882.04 total 80.07 last 64.68) - open/close 5788 (time 1785389892.04 total 90.07 last 46.28) - open/close 6404 (time 1785389902.05 total 100.08 last 61.58) - open/close 6996 (time 1785389912.05 total 110.08 last 59.20) - open/close 7390 (time 1785389922.05 total 120.08 last 39.40) - open/close 7838 (time 1785389932.05 total 130.08 last 44.79) - open/close 8359 (time 1785389942.06 total 140.08 last 52.07) - open/close 8897 (time 1785389952.06 total 150.09 last 53.76) - open/close 9432 (time 1785389962.08 total 160.11 last 53.40) total: 10000 open/close in 167.71 seconds: 59.63 ops/second - unlinked 0 (time 1785389975 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 507 (time 1785390084.01 total 10.01 last 50.63) - open/close 1395 (time 1785390094.02 total 20.03 last 88.69) - open/close 1811 (time 1785390104.02 total 30.03 last 41.59) - open/close 2169 (time 1785390114.03 total 40.03 last 35.78) - open/close 2501 (time 1785390124.06 total 50.06 last 33.10) - open/close 2916 (time 1785390134.06 total 60.07 last 41.47) - open/close 3709 (time 1785390144.07 total 70.07 last 79.26) - open/close 4571 (time 1785390154.07 total 80.08 last 86.15) - open/close 4972 (time 1785390164.09 total 90.10 last 40.04) - open/close 5615 (time 1785390174.10 total 100.11 last 64.20) - open/close 6268 (time 1785390184.11 total 110.11 last 65.28) - open/close 6961 (time 1785390194.12 total 120.13 last 69.21) - open/close 7401 (time 1785390204.13 total 130.14 last 43.94) - open/close 7728 (time 1785390214.13 total 140.14 last 32.69) - open/close 8339 (time 1785390224.15 total 150.15 last 61.02) - open/close 8880 (time 1785390234.17 total 160.17 last 53.99) - open/close 9358 (time 1785390244.17 total 170.17 last 47.80) - open/close 9773 (time 1785390254.18 total 180.18 last 41.47) total: 10000 open/close in 185.49 seconds: 53.91 ops/second - unlinked 0 (time 1785390265 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 478 (time 1785390387.26 total 10.01 last 47.76) - open/close 995 (time 1785390397.29 total 20.04 last 51.54) - open/close 1746 (time 1785390407.30 total 30.04 last 75.07) - open/close 2252 (time 1785390417.30 total 40.05 last 50.57) - open/close 2644 (time 1785390427.33 total 50.08 last 39.08) - open/close 3112 (time 1785390437.35 total 60.09 last 46.74) - open/close 3644 (time 1785390447.35 total 70.10 last 53.18) - open/close 4103 (time 1785390457.36 total 80.10 last 45.86) - open/close 4955 (time 1785390467.38 total 90.13 last 84.99) - open/close 5542 (time 1785390477.40 total 100.15 last 58.59) - open/close 6115 (time 1785390487.41 total 110.15 last 57.27) - open/close 6743 (time 1785390497.42 total 120.16 last 62.73) - open/close 7285 (time 1785390507.44 total 130.18 last 54.08) - open/close 7780 (time 1785390517.45 total 140.20 last 49.44) - open/close 8280 (time 1785390527.45 total 150.20 last 49.99) - open/close 9121 (time 1785390537.46 total 160.21 last 84.00) - open/close 9978 (time 1785390547.47 total 170.21 last 85.69) total: 10000 open/close in 170.70 seconds: 58.58 ops/second - unlinked 0 (time 1785390552 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 507 (time 1785390668.67 total 10.01 last 50.64) - open/close 1416 (time 1785390678.69 total 20.03 last 90.70) - open/close 2016 (time 1785390688.69 total 30.04 last 59.99) - open/close 2682 (time 1785390698.71 total 40.05 last 66.52) - open/close 3148 (time 1785390708.73 total 50.07 last 46.50) - open/close 3557 (time 1785390718.75 total 60.09 last 40.83) - open/close 3885 (time 1785390728.77 total 70.11 last 32.73) - open/close 4782 (time 1785390738.77 total 80.11 last 89.69) - open/close 5507 (time 1785390748.78 total 90.12 last 72.44) - open/close 6075 (time 1785390758.78 total 100.12 last 56.79) - open/close 6646 (time 1785390768.80 total 110.14 last 56.98) - open/close 7060 (time 1785390778.80 total 120.14 last 41.40) - open/close 7568 (time 1785390788.82 total 130.16 last 50.70) - open/close 8152 (time 1785390798.83 total 140.17 last 58.33) - open/close 8684 (time 1785390808.86 total 150.20 last 53.02) - open/close 9192 (time 1785390818.88 total 160.22 last 50.73) - open/close 9694 (time 1785390828.88 total 170.22 last 50.18) total: 10000 open/close in 174.46 seconds: 57.32 ops/second - unlinked 0 (time 1785390839 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 694 (time 1785390951.33 total 10.00 last 69.38) - open/close 1572 (time 1785390961.35 total 20.02 last 87.63) - open/close 2188 (time 1785390971.36 total 30.03 last 61.56) - open/close 2781 (time 1785390981.36 total 40.04 last 59.26) - open/close 3300 (time 1785390991.37 total 50.04 last 51.86) - open/close 3737 (time 1785391001.38 total 60.06 last 43.64) - open/close 4157 (time 1785391011.39 total 70.06 last 41.99) - open/close 4695 (time 1785391021.40 total 80.07 last 53.73) - open/close 5143 (time 1785391031.41 total 90.08 last 44.76) - open/close 5792 (time 1785391041.42 total 100.09 last 64.85) - open/close 6339 (time 1785391051.44 total 110.11 last 54.59) - open/close 6766 (time 1785391061.45 total 120.12 last 42.64) - open/close 7340 (time 1785391071.47 total 130.14 last 57.31) - open/close 7843 (time 1785391081.47 total 140.15 last 50.28) - open/close 8501 (time 1785391091.49 total 150.16 last 65.69) - open/close 9093 (time 1785391101.49 total 160.16 last 59.19) - open/close 9626 (time 1785391111.53 total 170.21 last 53.07) total: 10000 open/close in 177.13 seconds: 56.45 ops/second - unlinked 0 (time 1785391123 ; total 0 ; last 0) total: 10000 unlinks in 110 seconds: 90.909088 unlinks/second - open/close 523 (time 1785391251.28 total 10.01 last 52.27) - open/close 1127 (time 1785391261.28 total 20.01 last 60.39) - open/close 1575 (time 1785391271.29 total 30.02 last 44.76) - open/close 2178 (time 1785391281.29 total 40.02 last 60.29) - open/close 2784 (time 1785391291.30 total 50.03 last 60.54) - open/close 3396 (time 1785391301.31 total 60.04 last 61.13) - open/close 3983 (time 1785391311.31 total 70.04 last 58.69) - open/close 4579 (time 1785391321.32 total 80.05 last 59.56) - open/close 5087 (time 1785391331.36 total 90.08 last 50.62) - open/close 5549 (time 1785391341.38 total 100.11 last 46.07) - open/close 6295 (time 1785391351.40 total 110.13 last 74.48) - open/close 6664 (time 1785391361.40 total 120.13 last 36.88) - open/close 7107 (time 1785391371.43 total 130.15 last 44.20) - open/close 7815 (time 1785391381.43 total 140.16 last 70.79) - open/close 8475 (time 1785391391.44 total 150.17 last 65.94) - open/close 9294 (time 1785391401.45 total 160.17 last 81.83) total: 10000 open/close in 169.94 seconds: 58.85 ops/second - unlinked 0 (time 1785391416 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 449 (time 1785391531.77 total 10.02 last 44.81) - open/close 903 (time 1785391541.77 total 20.02 last 45.40) - open/close 1657 (time 1785391551.77 total 30.03 last 75.36) - open/close 2307 (time 1785391561.77 total 40.03 last 65.00) - open/close 3126 (time 1785391571.79 total 50.04 last 81.79) - open/close 3967 (time 1785391581.80 total 60.05 last 84.00) - open/close 4559 (time 1785391591.82 total 70.07 last 59.09) - open/close 5170 (time 1785391601.83 total 80.09 last 61.00) - open/close 5617 (time 1785391611.84 total 90.09 last 44.69) - open/close 6034 (time 1785391621.85 total 100.10 last 41.64) - open/close 6802 (time 1785391631.88 total 110.13 last 76.59) - open/close 7222 (time 1785391641.88 total 120.13 last 42.00) - open/close 7731 (time 1785391651.88 total 130.14 last 50.88) - open/close 8215 (time 1785391661.90 total 140.15 last 48.33) - open/close 8905 (time 1785391671.92 total 150.17 last 68.85) - open/close 9617 (time 1785391681.97 total 160.22 last 70.87) total: 10000 open/close in 166.00 seconds: 60.24 ops/second - unlinked 0 (time 1785391695 ; total 0 ; last 0) total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second - open/close 657 (time 1785391821.11 total 10.02 last 65.55) - open/close 1283 (time 1785391831.11 total 20.03 last 62.56) - open/close 1875 (time 1785391841.12 total 30.04 last 59.15) - open/close 2437 (time 1785391851.12 total 40.04 last 56.19) - open/close 3360 (time 1785391861.12 total 50.04 last 92.28) - open/close 4262 (time 1785391871.13 total 60.05 last 90.16) - open/close 5029 (time 1785391881.14 total 70.05 last 76.65) - open/close 5744 (time 1785391891.14 total 80.06 last 71.46) - open/close 6822 (time 1785391901.18 total 90.09 last 107.41) - open/close 7288 (time 1785391911.18 total 100.10 last 46.59) - open/close 7847 (time 1785391921.19 total 110.10 last 55.85) - open/close 8161 (time 1785391931.19 total 120.11 last 31.38) - open/close 8618 (time 1785391941.20 total 130.12 last 45.67) - open/close 9183 (time 1785391951.21 total 140.13 last 56.43) - open/close 9753 (time 1785391961.22 total 150.14 last 56.93) total: 10000 open/close in 154.73 seconds: 64.63 ops/second - unlinked 0 (time 1785391972 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.123@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.123@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg423-server: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg423-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.123@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.123@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.123@tcp autodegrade=on sys.timeout=20 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.123@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.123@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg423-server: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg423-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server unloading modules via unload_modules_local on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing unload_modules_local oleg423-server: modules unloaded. pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2