== conf-sanity test 69: replace an OST with the same index ========================================================== 20:40:10 (1786063210) start mds service on oleg428-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing load_modules_local oleg428-server: Loading modules from /home/green/git/lustre-release/lustre oleg428-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg428-server: detected 4 online CPUs by sysfs oleg428-server: MODOPTS_LIBCFS= oleg428-server: Force libcfs to create 2 CPU partitions oleg428-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg428-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg428-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg428-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 514 (time 1786063318.73 total 10.01 last 51.33) - open/close 1040 (time 1786063328.74 total 20.02 last 52.56) - open/close 1597 (time 1786063338.75 total 30.03 last 55.65) - open/close 2195 (time 1786063348.75 total 40.03 last 59.78) - open/close 2797 (time 1786063358.76 total 50.04 last 60.14) - open/close 3404 (time 1786063368.77 total 60.05 last 60.63) - open/close 3983 (time 1786063378.79 total 70.07 last 57.80) - open/close 4534 (time 1786063388.79 total 80.08 last 55.08) - open/close 5098 (time 1786063398.81 total 90.09 last 56.30) - open/close 5625 (time 1786063408.81 total 100.10 last 52.68) - open/close 6158 (time 1786063418.84 total 110.12 last 53.18) - open/close 6739 (time 1786063428.89 total 120.17 last 57.81) - open/close 7351 (time 1786063438.90 total 130.18 last 61.14) - open/close 7981 (time 1786063448.92 total 140.20 last 62.88) - open/close 8567 (time 1786063458.93 total 150.21 last 58.51) - open/close 9123 (time 1786063468.94 total 160.22 last 55.56) - open/close 9641 (time 1786063478.95 total 170.23 last 51.75) total: 10000 open/close in 176.58 seconds: 56.63 ops/second - unlinked 0 (time 1786063491 ; total 0 ; last 0) total: 10000 unlinks in 116 seconds: 86.206894 unlinks/second - open/close 606 (time 1786063624.41 total 10.01 last 60.51) - open/close 1223 (time 1786063634.41 total 20.02 last 61.68) - open/close 1856 (time 1786063644.42 total 30.02 last 63.28) - open/close 2531 (time 1786063654.43 total 40.04 last 67.41) - open/close 3123 (time 1786063664.45 total 50.06 last 59.06) - open/close 3588 (time 1786063674.46 total 60.07 last 46.45) - open/close 4156 (time 1786063684.47 total 70.08 last 56.74) - open/close 4775 (time 1786063694.49 total 80.10 last 61.80) - open/close 5201 (time 1786063704.52 total 90.13 last 42.48) - open/close 5574 (time 1786063714.52 total 100.13 last 37.28) - open/close 6136 (time 1786063724.54 total 110.14 last 56.13) - open/close 6720 (time 1786063734.55 total 120.16 last 58.30) - open/close 7280 (time 1786063744.56 total 130.17 last 55.95) - open/close 7937 (time 1786063754.56 total 140.17 last 65.69) - open/close 8528 (time 1786063764.58 total 150.19 last 59.00) - open/close 9096 (time 1786063774.59 total 160.20 last 56.74) - open/close 9689 (time 1786063784.60 total 170.21 last 59.22) total: 10000 open/close in 175.49 seconds: 56.98 ops/second - unlinked 0 (time 1786063796 ; total 0 ; last 0) total: 10000 unlinks in 119 seconds: 84.033615 unlinks/second - open/close 519 (time 1786063931.24 total 10.01 last 51.84) - open/close 1084 (time 1786063941.25 total 20.02 last 56.44) - open/close 1753 (time 1786063951.26 total 30.03 last 66.83) - open/close 2349 (time 1786063961.28 total 40.06 last 59.44) - open/close 2871 (time 1786063971.28 total 50.06 last 52.20) - open/close 3401 (time 1786063981.29 total 60.06 last 52.99) - open/close 3931 (time 1786063991.29 total 70.06 last 52.99) - open/close 4476 (time 1786064001.30 total 80.08 last 54.42) - open/close 5007 (time 1786064011.32 total 90.09 last 53.04) - open/close 5559 (time 1786064021.32 total 100.10 last 55.17) - open/close 6124 (time 1786064031.35 total 110.13 last 56.34) - open/close 6723 (time 1786064041.36 total 120.14 last 59.84) - open/close 7326 (time 1786064051.36 total 130.14 last 60.29) - open/close 7991 (time 1786064061.38 total 140.15 last 66.41) - open/close 8651 (time 1786064071.39 total 150.16 last 65.91) - open/close 9302 (time 1786064081.40 total 160.18 last 65.01) - open/close 9915 (time 1786064091.41 total 170.19 last 61.23) total: 10000 open/close in 171.69 seconds: 58.25 ops/second - unlinked 0 (time 1786064099 ; total 0 ; last 0) total: 10000 unlinks in 118 seconds: 84.745766 unlinks/second - open/close 658 (time 1786064233.99 total 10.01 last 65.73) - open/close 1259 (time 1786064244.00 total 20.01 last 60.08) - open/close 1883 (time 1786064254.01 total 30.03 last 62.33) - open/close 2514 (time 1786064264.01 total 40.03 last 63.09) - open/close 3194 (time 1786064274.02 total 50.03 last 67.95) - open/close 3852 (time 1786064284.02 total 60.04 last 65.77) - open/close 4530 (time 1786064294.05 total 70.07 last 67.60) - open/close 5184 (time 1786064304.06 total 80.08 last 65.35) - open/close 5804 (time 1786064314.07 total 90.08 last 61.97) - open/close 6446 (time 1786064324.07 total 100.09 last 64.15) - open/close 7096 (time 1786064334.08 total 110.10 last 64.93) - open/close 7805 (time 1786064344.09 total 120.11 last 70.84) - open/close 8443 (time 1786064354.10 total 130.12 last 63.76) - open/close 9121 (time 1786064364.12 total 140.13 last 67.68) - open/close 9822 (time 1786064374.12 total 150.14 last 70.07) total: 10000 open/close in 152.87 seconds: 65.41 ops/second - unlinked 0 (time 1786064382 ; total 0 ; last 0) total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second - open/close 650 (time 1786064510.48 total 10.01 last 64.96) - open/close 1254 (time 1786064520.50 total 20.02 last 60.31) - open/close 1858 (time 1786064530.51 total 30.04 last 60.31) - open/close 2526 (time 1786064540.52 total 40.05 last 66.73) - open/close 3174 (time 1786064550.53 total 50.05 last 64.78) - open/close 3801 (time 1786064560.53 total 60.05 last 62.69) - open/close 4426 (time 1786064570.53 total 70.05 last 62.49) - open/close 5066 (time 1786064580.53 total 80.06 last 63.98) - open/close 5691 (time 1786064590.55 total 90.07 last 62.44) - open/close 6397 (time 1786064600.55 total 100.07 last 70.57) - open/close 7114 (time 1786064610.56 total 110.08 last 71.66) - open/close 7807 (time 1786064620.56 total 120.08 last 69.26) - open/close 8494 (time 1786064630.57 total 130.09 last 68.63) - open/close 9230 (time 1786064640.58 total 140.10 last 73.53) - open/close 9871 (time 1786064650.59 total 150.11 last 64.02) total: 10000 open/close in 152.44 seconds: 65.60 ops/second - unlinked 0 (time 1786064658 ; total 0 ; last 0) total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second - open/close 675 (time 1786064785.31 total 10.01 last 67.43) - open/close 1347 (time 1786064795.32 total 20.02 last 67.14) - open/close 1978 (time 1786064805.32 total 30.02 last 63.08) - open/close 2620 (time 1786064815.33 total 40.03 last 64.15) - open/close 3200 (time 1786064825.34 total 50.04 last 57.92) - open/close 3812 (time 1786064835.35 total 60.05 last 61.15) - open/close 4471 (time 1786064845.36 total 70.06 last 65.87) - open/close 5110 (time 1786064855.36 total 80.06 last 63.88) - open/close 5691 (time 1786064865.36 total 90.06 last 58.08) - open/close 6260 (time 1786064875.37 total 100.07 last 56.85) - open/close 6863 (time 1786064885.38 total 110.08 last 60.24) - open/close 7476 (time 1786064895.38 total 120.08 last 61.29) - open/close 8105 (time 1786064905.39 total 130.09 last 62.87) - open/close 8744 (time 1786064915.40 total 140.10 last 63.83) - open/close 9401 (time 1786064925.41 total 150.11 last 65.62) total: 10000 open/close in 159.10 seconds: 62.85 ops/second - unlinked 0 (time 1786064939 ; total 0 ; last 0) total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second - open/close 650 (time 1786065063.18 total 10.05 last 64.68) - open/close 1328 (time 1786065073.20 total 20.06 last 67.71) - open/close 2013 (time 1786065083.20 total 30.07 last 68.46) - open/close 2705 (time 1786065093.21 total 40.08 last 69.13) - open/close 3359 (time 1786065103.22 total 50.09 last 65.35) - open/close 3988 (time 1786065113.23 total 60.10 last 62.84) - open/close 4681 (time 1786065123.23 total 70.10 last 69.28) - open/close 5354 (time 1786065133.23 total 80.10 last 67.29) - open/close 6034 (time 1786065143.25 total 90.11 last 67.92) - open/close 6691 (time 1786065153.25 total 100.12 last 65.64) - open/close 7354 (time 1786065163.26 total 110.12 last 66.30) - open/close 8004 (time 1786065173.26 total 120.13 last 64.95) - open/close 8653 (time 1786065183.26 total 130.13 last 64.90) - open/close 9297 (time 1786065193.27 total 140.14 last 64.35) - open/close 9937 (time 1786065203.29 total 150.15 last 63.91) total: 10000 open/close in 151.26 seconds: 66.11 ops/second - unlinked 0 (time 1786065211 ; total 0 ; last 0) total: 10000 unlinks in 112 seconds: 89.285713 unlinks/second - open/close 730 (time 1786065339.28 total 10.01 last 72.94) - open/close 1386 (time 1786065349.29 total 20.01 last 65.57) - open/close 2062 (time 1786065359.29 total 30.02 last 67.59) - open/close 2708 (time 1786065369.32 total 40.04 last 64.42) - open/close 3377 (time 1786065379.32 total 50.05 last 66.84) - open/close 4053 (time 1786065389.33 total 60.06 last 67.56) - open/close 4742 (time 1786065399.34 total 70.06 last 68.87) - open/close 5411 (time 1786065409.35 total 80.08 last 66.77) - open/close 5990 (time 1786065419.36 total 90.08 last 57.89) - open/close 6611 (time 1786065429.36 total 100.09 last 62.06) - open/close 7290 (time 1786065439.39 total 110.11 last 67.73) - open/close 7992 (time 1786065449.39 total 120.12 last 70.19) - open/close 8645 (time 1786065459.40 total 130.13 last 65.22) - open/close 9296 (time 1786065469.40 total 140.13 last 65.08) - open/close 9986 (time 1786065479.41 total 150.14 last 68.94) total: 10000 open/close in 150.31 seconds: 66.53 ops/second - unlinked 0 (time 1786065484 ; total 0 ; last 0) total: 10000 unlinks in 110 seconds: 90.909088 unlinks/second - open/close 525 (time 1786065608.71 total 10.01 last 52.45) - open/close 1142 (time 1786065618.72 total 20.01 last 61.69) - open/close 1785 (time 1786065628.72 total 30.02 last 64.27) - open/close 2415 (time 1786065638.74 total 40.04 last 62.87) - open/close 3088 (time 1786065648.76 total 50.05 last 67.18) - open/close 3731 (time 1786065658.77 total 60.06 last 64.24) - open/close 4393 (time 1786065668.77 total 70.07 last 66.16) - open/close 5070 (time 1786065678.78 total 80.07 last 67.68) - open/close 5708 (time 1786065688.79 total 90.08 last 63.73) - open/close 6381 (time 1786065698.80 total 100.09 last 67.26) - open/close 6998 (time 1786065708.80 total 110.09 last 61.68) - open/close 7673 (time 1786065718.80 total 120.10 last 67.46) - open/close 8350 (time 1786065728.83 total 130.12 last 67.56) - open/close 8972 (time 1786065738.84 total 140.14 last 62.10) - open/close 9569 (time 1786065748.86 total 150.16 last 59.58) total: 10000 open/close in 157.14 seconds: 63.64 ops/second - unlinked 0 (time 1786065760 ; total 0 ; last 0) total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second - open/close 624 (time 1786065890.79 total 10.01 last 62.36) - open/close 1269 (time 1786065900.80 total 20.02 last 64.43) - open/close 1918 (time 1786065910.80 total 30.02 last 64.89) - open/close 2539 (time 1786065920.81 total 40.03 last 62.03) - open/close 3187 (time 1786065930.81 total 50.03 last 64.80) - open/close 3838 (time 1786065940.83 total 60.04 last 65.01) - open/close 4438 (time 1786065950.85 total 70.06 last 59.88) - open/close 5121 (time 1786065960.85 total 80.07 last 68.28) - open/close 5797 (time 1786065970.86 total 90.08 last 67.53) - open/close 6459 (time 1786065980.86 total 100.08 last 66.17) - open/close 7115 (time 1786065990.88 total 110.09 last 65.52) - open/close 7715 (time 1786066000.90 total 120.12 last 59.85) - open/close 8317 (time 1786066010.91 total 130.12 last 60.17) - open/close 8947 (time 1786066020.91 total 140.13 last 62.95) - open/close 9572 (time 1786066030.94 total 150.15 last 62.37) total: 10000 open/close in 157.22 seconds: 63.60 ops/second - unlinked 0 (time 1786066042 ; total 0 ; last 0) total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg428-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.128@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.128@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg428-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg428-server: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg428-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg428-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.128@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.128@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.128@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.128@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.204.128@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg428-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg428-server: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg428-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg428-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg428-server unloading modules via unload_modules_local on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing unload_modules_local oleg428-server: modules unloaded. pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2