== conf-sanity test 69: replace an OST with the same index ========================================================== 02:27:56 (1786688876) start mds service on oleg451-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg451-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 596 (time 1786688960.71 total 10.01 last 59.52) - open/close 1173 (time 1786688970.72 total 20.02 last 57.67) - open/close 1691 (time 1786688980.73 total 30.03 last 51.75) - open/close 2321 (time 1786688990.74 total 40.04 last 62.92) - open/close 2941 (time 1786689000.74 total 50.04 last 61.98) - open/close 3537 (time 1786689010.78 total 60.08 last 59.38) - open/close 4100 (time 1786689020.80 total 70.10 last 56.19) - open/close 4636 (time 1786689030.81 total 80.11 last 53.56) - open/close 5246 (time 1786689040.82 total 90.12 last 60.94) - open/close 5911 (time 1786689050.82 total 100.12 last 66.47) - open/close 6458 (time 1786689060.83 total 110.13 last 54.65) - open/close 6999 (time 1786689070.84 total 120.14 last 54.03) - open/close 7597 (time 1786689080.85 total 130.15 last 59.77) - open/close 8128 (time 1786689090.87 total 140.17 last 52.96) - open/close 8682 (time 1786689100.88 total 150.18 last 55.36) - open/close 9260 (time 1786689110.89 total 160.19 last 57.76) - open/close 9835 (time 1786689120.90 total 170.20 last 57.41) total: 10000 open/close in 173.20 seconds: 57.74 ops/second - unlinked 0 (time 1786689129 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 597 (time 1786689246.38 total 10.00 last 59.69) - open/close 1254 (time 1786689256.39 total 20.01 last 65.67) - open/close 1897 (time 1786689266.39 total 30.01 last 64.27) - open/close 2576 (time 1786689276.40 total 40.01 last 67.89) - open/close 3263 (time 1786689286.40 total 50.02 last 68.67) - open/close 3868 (time 1786689296.42 total 60.04 last 60.38) - open/close 4483 (time 1786689306.44 total 70.05 last 61.40) - open/close 5115 (time 1786689316.45 total 80.07 last 63.12) - open/close 5714 (time 1786689326.46 total 90.08 last 59.80) - open/close 6190 (time 1786689336.47 total 100.09 last 47.58) - open/close 6749 (time 1786689346.49 total 110.10 last 55.80) - open/close 7390 (time 1786689356.49 total 120.11 last 64.04) - open/close 7990 (time 1786689366.50 total 130.11 last 59.99) - open/close 8640 (time 1786689376.50 total 140.12 last 65.00) - open/close 9255 (time 1786689386.50 total 150.12 last 61.49) - open/close 9879 (time 1786689396.50 total 160.12 last 62.39) total: 10000 open/close in 161.71 seconds: 61.84 ops/second - unlinked 0 (time 1786689402 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 637 (time 1786689520.02 total 10.01 last 63.64) - open/close 1286 (time 1786689530.03 total 20.01 last 64.88) - open/close 1895 (time 1786689540.04 total 30.02 last 60.84) - open/close 2546 (time 1786689550.05 total 40.03 last 65.05) - open/close 3143 (time 1786689560.05 total 50.04 last 59.67) - open/close 3697 (time 1786689570.06 total 60.05 last 55.34) - open/close 4363 (time 1786689580.06 total 70.05 last 66.59) - open/close 5003 (time 1786689590.09 total 80.07 last 63.85) - open/close 5630 (time 1786689600.09 total 90.08 last 62.66) - open/close 6228 (time 1786689610.10 total 100.08 last 59.78) - open/close 6901 (time 1786689620.10 total 110.09 last 67.26) - open/close 7484 (time 1786689630.11 total 120.10 last 58.25) - open/close 8037 (time 1786689640.12 total 130.10 last 55.27) - open/close 8543 (time 1786689650.13 total 140.11 last 50.55) - open/close 9158 (time 1786689660.14 total 150.12 last 61.44) - open/close 9757 (time 1786689670.14 total 160.13 last 59.86) total: 10000 open/close in 164.45 seconds: 60.81 ops/second - unlinked 0 (time 1786689678 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 578 (time 1786689792.83 total 10.01 last 57.77) - open/close 1180 (time 1786689802.85 total 20.02 last 60.12) - open/close 1767 (time 1786689812.87 total 30.04 last 58.57) - open/close 2374 (time 1786689822.87 total 40.04 last 60.70) - open/close 2971 (time 1786689832.88 total 50.05 last 59.66) - open/close 3519 (time 1786689842.90 total 60.07 last 54.68) - open/close 4016 (time 1786689852.91 total 70.08 last 49.66) - open/close 4687 (time 1786689862.91 total 80.09 last 67.04) - open/close 5276 (time 1786689872.92 total 90.09 last 58.86) - open/close 5911 (time 1786689882.93 total 100.10 last 63.43) - open/close 6554 (time 1786689892.94 total 110.11 last 64.24) - open/close 7149 (time 1786689902.94 total 120.12 last 59.50) - open/close 7820 (time 1786689912.95 total 130.12 last 67.07) - open/close 8477 (time 1786689922.95 total 140.12 last 65.67) - open/close 9067 (time 1786689932.96 total 150.14 last 58.92) - open/close 9688 (time 1786689942.98 total 160.16 last 61.98) total: 10000 open/close in 165.18 seconds: 60.54 ops/second - unlinked 0 (time 1786689952 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 614 (time 1786690075.00 total 10.00 last 61.38) - open/close 1218 (time 1786690085.01 total 20.01 last 60.38) - open/close 1857 (time 1786690095.02 total 30.02 last 63.81) - open/close 2401 (time 1786690105.04 total 40.04 last 54.30) - open/close 2943 (time 1786690115.05 total 50.05 last 54.12) - open/close 3559 (time 1786690125.06 total 60.06 last 61.55) - open/close 4129 (time 1786690135.06 total 70.06 last 56.98) - open/close 4763 (time 1786690145.08 total 80.08 last 63.29) - open/close 5330 (time 1786690155.08 total 90.08 last 56.69) - open/close 5958 (time 1786690165.08 total 100.08 last 62.80) - open/close 6544 (time 1786690175.10 total 110.10 last 58.49) - open/close 7128 (time 1786690185.10 total 120.10 last 58.39) - open/close 7710 (time 1786690195.11 total 130.11 last 58.14) - open/close 8275 (time 1786690205.12 total 140.12 last 56.45) - open/close 8867 (time 1786690215.12 total 150.12 last 59.20) - open/close 9490 (time 1786690225.12 total 160.12 last 62.28) total: 10000 open/close in 168.47 seconds: 59.36 ops/second - unlinked 0 (time 1786690238 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 598 (time 1786690361.29 total 10.01 last 59.74) - open/close 1207 (time 1786690371.30 total 20.01 last 60.88) - open/close 1807 (time 1786690381.32 total 30.03 last 59.89) - open/close 2343 (time 1786690391.32 total 40.04 last 53.56) - open/close 2957 (time 1786690401.33 total 50.05 last 61.36) - open/close 3500 (time 1786690411.35 total 60.07 last 54.17) - open/close 4058 (time 1786690421.36 total 70.07 last 55.79) - open/close 4600 (time 1786690431.37 total 80.09 last 54.13) - open/close 5155 (time 1786690441.39 total 90.10 last 55.41) - open/close 5748 (time 1786690451.41 total 100.12 last 59.19) - open/close 6353 (time 1786690461.42 total 110.14 last 60.39) - open/close 6970 (time 1786690471.44 total 120.16 last 61.57) - open/close 7614 (time 1786690481.45 total 130.16 last 64.39) - open/close 8190 (time 1786690491.45 total 140.16 last 57.59) - open/close 8703 (time 1786690501.47 total 150.19 last 51.19) - open/close 9145 (time 1786690511.48 total 160.19 last 44.17) - open/close 9696 (time 1786690521.48 total 170.20 last 55.06) total: 10000 open/close in 175.57 seconds: 56.96 ops/second - unlinked 0 (time 1786690531 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 653 (time 1786690653.43 total 10.01 last 65.23) - open/close 1237 (time 1786690663.45 total 20.03 last 58.30) - open/close 1793 (time 1786690673.45 total 30.04 last 55.56) - open/close 2346 (time 1786690683.46 total 40.04 last 55.26) - open/close 2935 (time 1786690693.47 total 50.05 last 58.84) - open/close 3499 (time 1786690703.48 total 60.06 last 56.34) - open/close 4102 (time 1786690713.50 total 70.08 last 60.21) - open/close 4708 (time 1786690723.51 total 80.09 last 60.54) - open/close 5287 (time 1786690733.51 total 90.10 last 57.85) - open/close 5877 (time 1786690743.52 total 100.10 last 58.99) - open/close 6439 (time 1786690753.53 total 110.12 last 56.10) - open/close 7043 (time 1786690763.54 total 120.12 last 60.37) - open/close 7658 (time 1786690773.54 total 130.13 last 61.48) - open/close 8219 (time 1786690783.55 total 140.13 last 56.08) - open/close 8830 (time 1786690793.57 total 150.16 last 60.93) - open/close 9402 (time 1786690803.59 total 160.18 last 57.10) total: 10000 open/close in 170.08 seconds: 58.79 ops/second - unlinked 0 (time 1786690818 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 650 (time 1786690936.65 total 10.00 last 64.98) - open/close 1249 (time 1786690946.66 total 20.01 last 59.83) - open/close 1878 (time 1786690956.67 total 30.02 last 62.88) - open/close 2323 (time 1786690966.69 total 40.04 last 44.39) - open/close 2870 (time 1786690976.70 total 50.05 last 54.65) - open/close 3444 (time 1786690986.71 total 60.06 last 57.33) - open/close 4046 (time 1786690996.74 total 70.09 last 60.05) - open/close 4637 (time 1786691006.74 total 80.09 last 59.09) - open/close 5260 (time 1786691016.75 total 90.10 last 62.25) - open/close 5879 (time 1786691026.75 total 100.11 last 61.86) - open/close 6539 (time 1786691036.78 total 110.13 last 65.81) - open/close 7169 (time 1786691046.80 total 120.15 last 62.92) - open/close 7787 (time 1786691056.80 total 130.16 last 61.74) - open/close 8360 (time 1786691066.81 total 140.16 last 57.28) - open/close 8985 (time 1786691076.81 total 150.16 last 62.48) - open/close 9637 (time 1786691086.82 total 160.17 last 65.17) total: 10000 open/close in 165.99 seconds: 60.25 ops/second - unlinked 0 (time 1786691096 ; total 0 ; last 0) total: 10000 unlinks in 112 seconds: 89.285713 unlinks/second - open/close 615 (time 1786691222.54 total 10.00 last 61.48) - open/close 1259 (time 1786691232.54 total 20.01 last 64.35) - open/close 1897 (time 1786691242.57 total 30.04 last 63.63) - open/close 2538 (time 1786691252.58 total 40.05 last 64.04) - open/close 3175 (time 1786691262.58 total 50.05 last 63.68) - open/close 3821 (time 1786691272.60 total 60.07 last 64.48) - open/close 4462 (time 1786691282.62 total 70.09 last 63.99) - open/close 5099 (time 1786691292.62 total 80.09 last 63.67) - open/close 5740 (time 1786691302.64 total 90.10 last 64.01) - open/close 6356 (time 1786691312.64 total 100.11 last 61.57) - open/close 6912 (time 1786691322.65 total 110.12 last 55.55) - open/close 7527 (time 1786691332.65 total 120.12 last 61.50) - open/close 8130 (time 1786691342.66 total 130.13 last 60.22) - open/close 8735 (time 1786691352.67 total 140.14 last 60.43) - open/close 9406 (time 1786691362.68 total 150.15 last 67.06) total: 10000 open/close in 160.03 seconds: 62.49 ops/second - unlinked 0 (time 1786691377 ; total 0 ; last 0) total: 10000 unlinks in 110 seconds: 90.909088 unlinks/second - open/close 612 (time 1786691503.45 total 10.00 last 61.18) - open/close 1164 (time 1786691513.45 total 20.00 last 55.20) - open/close 1778 (time 1786691523.45 total 30.00 last 61.40) - open/close 2305 (time 1786691533.45 total 40.01 last 52.66) - open/close 2781 (time 1786691543.46 total 50.02 last 47.58) - open/close 3370 (time 1786691553.47 total 60.03 last 58.83) - open/close 4010 (time 1786691563.48 total 70.04 last 63.95) - open/close 4628 (time 1786691573.49 total 80.05 last 61.74) - open/close 5099 (time 1786691583.52 total 90.08 last 46.95) - open/close 5692 (time 1786691593.52 total 100.08 last 59.30) - open/close 6324 (time 1786691603.53 total 110.09 last 63.15) - open/close 6980 (time 1786691613.54 total 120.10 last 65.52) - open/close 7572 (time 1786691623.55 total 130.10 last 59.19) - open/close 8215 (time 1786691633.55 total 140.11 last 64.26) - open/close 8847 (time 1786691643.56 total 150.11 last 63.18) - open/close 9485 (time 1786691653.57 total 160.13 last 63.71) total: 10000 open/close in 168.34 seconds: 59.40 ops/second - unlinked 0 (time 1786691666 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.151@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre On OST0, 16790 used inodes rc=0 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.151@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.151@tcp autodegrade=on sys.timeout=20 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.151@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.151@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server unloading modules via unload_modules_local on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local oleg451-server: modules unloaded. pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2