== conf-sanity test 69: replace an OST with the same index ========================================================== 12:47:04 (1786034824) start mds service on oleg301-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing load_modules_local oleg301-server: Loading modules from /home/green/git/lustre-release/lustre oleg301-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg301-server: detected 4 online CPUs by sysfs oleg301-server: MODOPTS_LIBCFS= oleg301-server: Force libcfs to create 2 CPU partitions oleg301-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg301-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg301-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 632 (time 1786034938.47 total 10.02 last 63.08) - open/close 1296 (time 1786034948.50 total 20.05 last 66.21) - open/close 1783 (time 1786034958.52 total 30.06 last 48.63) - open/close 2324 (time 1786034968.53 total 40.08 last 54.01) - open/close 2891 (time 1786034978.54 total 50.09 last 56.64) - open/close 3285 (time 1786034988.57 total 60.11 last 39.30) - open/close 3799 (time 1786034998.57 total 70.12 last 51.39) - open/close 4291 (time 1786035008.58 total 80.12 last 49.17) - open/close 4785 (time 1786035018.58 total 90.12 last 49.38) - open/close 5182 (time 1786035028.58 total 100.13 last 39.69) - open/close 5587 (time 1786035038.59 total 110.13 last 40.47) - open/close 6034 (time 1786035048.62 total 120.17 last 44.56) - open/close 6547 (time 1786035058.63 total 130.18 last 51.24) - open/close 7067 (time 1786035068.64 total 140.19 last 51.94) - open/close 7664 (time 1786035078.66 total 150.21 last 59.60) - open/close 8316 (time 1786035088.66 total 160.21 last 65.19) - open/close 8864 (time 1786035098.67 total 170.21 last 54.76) - open/close 9356 (time 1786035108.69 total 180.24 last 49.09) - open/close 9981 (time 1786035118.71 total 190.25 last 62.39) total: 10000 open/close in 190.58 seconds: 52.47 ops/second - unlinked 0 (time 1786035124 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 659 (time 1786035246.51 total 10.00 last 65.89) - open/close 1249 (time 1786035256.51 total 20.01 last 58.97) - open/close 1772 (time 1786035266.53 total 30.02 last 52.22) - open/close 2241 (time 1786035276.54 total 40.04 last 46.84) - open/close 2801 (time 1786035286.54 total 50.04 last 55.99) - open/close 3419 (time 1786035296.54 total 60.04 last 61.80) - open/close 4030 (time 1786035306.55 total 70.05 last 61.06) - open/close 4638 (time 1786035316.57 total 80.06 last 60.68) - open/close 5175 (time 1786035326.59 total 90.08 last 53.60) - open/close 5856 (time 1786035336.60 total 100.10 last 68.00) - open/close 6274 (time 1786035346.65 total 110.14 last 41.62) - open/close 6656 (time 1786035356.68 total 120.18 last 38.06) - open/close 7177 (time 1786035366.69 total 130.18 last 52.08) - open/close 7734 (time 1786035376.71 total 140.21 last 55.57) - open/close 8281 (time 1786035386.72 total 150.22 last 54.65) - open/close 8785 (time 1786035396.73 total 160.23 last 50.35) - open/close 9385 (time 1786035406.74 total 170.23 last 59.96) - open/close 9955 (time 1786035416.74 total 180.24 last 56.98) total: 10000 open/close in 180.95 seconds: 55.27 ops/second - unlinked 0 (time 1786035422 ; total 0 ; last 0) total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second - open/close 778 (time 1786035551.55 total 10.02 last 77.65) - open/close 1302 (time 1786035561.55 total 20.02 last 52.39) - open/close 1790 (time 1786035571.57 total 30.04 last 48.72) - open/close 2327 (time 1786035581.58 total 40.05 last 53.62) - open/close 2858 (time 1786035591.60 total 50.06 last 53.04) - open/close 3403 (time 1786035601.61 total 60.07 last 54.45) - open/close 3995 (time 1786035611.62 total 70.09 last 59.13) - open/close 4504 (time 1786035621.66 total 80.13 last 50.69) - open/close 4947 (time 1786035631.67 total 90.14 last 44.25) - open/close 5493 (time 1786035641.71 total 100.17 last 54.41) - open/close 6335 (time 1786035651.72 total 110.19 last 84.08) - open/close 6829 (time 1786035661.73 total 120.20 last 49.36) - open/close 7402 (time 1786035671.73 total 130.20 last 57.27) - open/close 8189 (time 1786035681.74 total 140.21 last 78.62) - open/close 8704 (time 1786035691.76 total 150.23 last 51.39) - open/close 9291 (time 1786035701.77 total 160.24 last 58.64) - open/close 9684 (time 1786035711.80 total 170.27 last 39.20) total: 10000 open/close in 174.30 seconds: 57.37 ops/second - unlinked 0 (time 1786035721 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 513 (time 1786035832.29 total 10.01 last 51.27) - open/close 999 (time 1786035842.29 total 20.01 last 48.59) - open/close 1471 (time 1786035852.30 total 30.03 last 47.12) - open/close 1935 (time 1786035862.32 total 40.04 last 46.35) - open/close 2412 (time 1786035872.32 total 50.04 last 47.66) - open/close 2879 (time 1786035882.34 total 60.06 last 46.63) - open/close 3275 (time 1786035892.36 total 70.08 last 39.50) - open/close 3699 (time 1786035902.38 total 80.10 last 42.34) - open/close 4247 (time 1786035912.39 total 90.11 last 54.74) - open/close 4715 (time 1786035922.42 total 100.14 last 46.65) - open/close 5166 (time 1786035932.42 total 110.15 last 45.09) - open/close 5763 (time 1786035942.43 total 120.16 last 59.64) - open/close 6321 (time 1786035952.44 total 130.16 last 55.80) - open/close 6988 (time 1786035962.44 total 140.16 last 66.64) - open/close 7526 (time 1786035972.46 total 150.18 last 53.72) - open/close 8069 (time 1786035982.48 total 160.20 last 54.17) - open/close 8529 (time 1786035992.50 total 170.22 last 45.94) - open/close 8981 (time 1786036002.51 total 180.23 last 45.13) - open/close 9491 (time 1786036012.51 total 190.24 last 50.99) total: 10000 open/close in 198.32 seconds: 50.42 ops/second - unlinked 0 (time 1786036025 ; total 0 ; last 0) total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second - open/close 609 (time 1786036152.17 total 10.00 last 60.88) - open/close 1106 (time 1786036162.17 total 20.01 last 49.66) - open/close 1649 (time 1786036172.18 total 30.02 last 54.27) - open/close 2188 (time 1786036182.19 total 40.03 last 53.83) - open/close 2764 (time 1786036192.22 total 50.05 last 57.45) - open/close 3282 (time 1786036202.24 total 60.07 last 51.70) - open/close 3895 (time 1786036212.24 total 70.08 last 61.26) - open/close 4470 (time 1786036222.25 total 80.09 last 57.47) - open/close 5061 (time 1786036232.25 total 90.09 last 59.09) - open/close 5676 (time 1786036242.26 total 100.10 last 61.45) - open/close 6397 (time 1786036252.27 total 110.10 last 72.04) - open/close 7164 (time 1786036262.27 total 120.11 last 76.69) - open/close 8003 (time 1786036272.27 total 130.11 last 83.85) - open/close 8568 (time 1786036282.28 total 140.12 last 56.48) - open/close 9085 (time 1786036292.29 total 150.12 last 51.65) - open/close 9633 (time 1786036302.29 total 160.13 last 54.76) total: 10000 open/close in 167.35 seconds: 59.76 ops/second - unlinked 0 (time 1786036313 ; total 0 ; last 0) total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second - open/close 701 (time 1786036438.17 total 10.02 last 69.96) - open/close 1327 (time 1786036448.19 total 20.04 last 62.48) - open/close 1917 (time 1786036458.19 total 30.04 last 58.99) - open/close 2410 (time 1786036468.21 total 40.06 last 49.20) - open/close 2947 (time 1786036478.24 total 50.09 last 53.54) - open/close 3498 (time 1786036488.26 total 60.11 last 54.98) - open/close 4193 (time 1786036498.28 total 70.13 last 69.37) - open/close 4818 (time 1786036508.28 total 80.13 last 62.49) - open/close 5354 (time 1786036518.29 total 90.14 last 53.57) - open/close 5826 (time 1786036528.29 total 100.14 last 47.20) - open/close 6608 (time 1786036538.29 total 110.14 last 78.16) - open/close 7194 (time 1786036548.31 total 120.16 last 58.47) - open/close 7842 (time 1786036558.33 total 130.18 last 64.71) - open/close 8412 (time 1786036568.33 total 140.18 last 56.99) - open/close 8947 (time 1786036578.35 total 150.20 last 53.40) - open/close 9563 (time 1786036588.35 total 160.20 last 61.59) total: 10000 open/close in 167.16 seconds: 59.82 ops/second - unlinked 0 (time 1786036600 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 638 (time 1786036718.82 total 10.01 last 63.76) - open/close 1189 (time 1786036728.82 total 20.01 last 55.07) - open/close 1745 (time 1786036738.83 total 30.02 last 55.58) - open/close 2171 (time 1786036748.85 total 40.04 last 42.49) - open/close 2697 (time 1786036758.85 total 50.04 last 52.59) - open/close 3230 (time 1786036768.86 total 60.05 last 53.25) - open/close 3930 (time 1786036778.86 total 70.05 last 70.00) - open/close 4559 (time 1786036788.92 total 80.11 last 62.56) - open/close 5003 (time 1786036798.93 total 90.12 last 44.35) - open/close 5478 (time 1786036808.95 total 100.13 last 47.42) - open/close 5960 (time 1786036818.98 total 110.16 last 48.06) - open/close 6418 (time 1786036828.99 total 120.17 last 45.75) - open/close 6829 (time 1786036839.00 total 130.19 last 41.05) - open/close 7321 (time 1786036849.02 total 140.21 last 49.07) - open/close 7766 (time 1786036859.05 total 150.23 last 44.41) - open/close 8551 (time 1786036869.05 total 160.23 last 78.50) - open/close 9274 (time 1786036879.05 total 170.24 last 72.26) - open/close 9821 (time 1786036889.08 total 180.27 last 54.53) total: 10000 open/close in 183.38 seconds: 54.53 ops/second - unlinked 0 (time 1786036897 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 574 (time 1786037017.64 total 10.01 last 57.33) - open/close 1095 (time 1786037027.64 total 20.01 last 52.10) - open/close 1771 (time 1786037037.66 total 30.03 last 67.49) - open/close 2358 (time 1786037047.66 total 40.03 last 58.69) - open/close 2985 (time 1786037057.67 total 50.03 last 62.68) - open/close 3620 (time 1786037067.68 total 60.04 last 63.44) - open/close 4091 (time 1786037077.69 total 70.06 last 47.05) - open/close 4654 (time 1786037087.69 total 80.06 last 56.28) - open/close 5225 (time 1786037097.69 total 90.06 last 57.09) - open/close 5783 (time 1786037107.70 total 100.07 last 55.77) - open/close 6373 (time 1786037117.70 total 110.07 last 59.00) - open/close 6997 (time 1786037127.71 total 120.08 last 62.31) - open/close 7429 (time 1786037137.72 total 130.09 last 43.16) - open/close 7957 (time 1786037147.75 total 140.12 last 52.67) - open/close 8544 (time 1786037157.77 total 150.14 last 58.57) - open/close 9063 (time 1786037167.79 total 160.16 last 51.80) - open/close 9572 (time 1786037177.79 total 170.16 last 50.87) total: 10000 open/close in 177.56 seconds: 56.32 ops/second - unlinked 0 (time 1786037190 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 542 (time 1786037305.86 total 10.01 last 54.14) - open/close 1385 (time 1786037315.89 total 20.04 last 84.09) - open/close 2050 (time 1786037325.93 total 30.08 last 66.22) - open/close 2536 (time 1786037335.96 total 40.11 last 48.47) - open/close 3231 (time 1786037345.98 total 50.13 last 69.36) - open/close 3969 (time 1786037356.01 total 60.16 last 73.56) - open/close 4683 (time 1786037366.02 total 70.17 last 71.30) - open/close 5304 (time 1786037376.02 total 80.17 last 62.10) - open/close 5889 (time 1786037386.05 total 90.20 last 58.35) - open/close 6462 (time 1786037396.06 total 100.21 last 57.27) - open/close 7025 (time 1786037406.07 total 110.22 last 56.23) - open/close 7712 (time 1786037416.08 total 120.22 last 68.65) - open/close 8217 (time 1786037426.08 total 130.23 last 50.46) - open/close 8614 (time 1786037436.09 total 140.24 last 39.68) - open/close 9062 (time 1786037446.11 total 150.26 last 44.72) - open/close 9609 (time 1786037456.15 total 160.29 last 54.49) total: 10000 open/close in 168.53 seconds: 59.34 ops/second - unlinked 0 (time 1786037469 ; total 0 ; last 0) total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second - open/close 525 (time 1786037593.57 total 10.02 last 52.38) - open/close 994 (time 1786037603.58 total 20.04 last 46.83) - open/close 1450 (time 1786037613.59 total 30.04 last 45.58) - open/close 1961 (time 1786037623.59 total 40.04 last 51.09) - open/close 2770 (time 1786037633.62 total 50.08 last 80.64) - open/close 3326 (time 1786037643.64 total 60.09 last 55.50) - open/close 3869 (time 1786037653.64 total 70.10 last 54.29) - open/close 4334 (time 1786037663.64 total 80.10 last 46.49) - open/close 4655 (time 1786037673.65 total 90.10 last 32.09) - open/close 5086 (time 1786037683.69 total 100.15 last 42.91) - open/close 5605 (time 1786037693.71 total 110.16 last 51.82) - open/close 6194 (time 1786037703.72 total 120.17 last 58.83) - open/close 6510 (time 1786037713.73 total 130.19 last 31.56) - open/close 6931 (time 1786037723.75 total 140.20 last 42.04) - open/close 7341 (time 1786037733.76 total 150.22 last 40.93) - open/close 7797 (time 1786037743.78 total 160.24 last 45.51) - open/close 8264 (time 1786037753.79 total 170.25 last 46.67) - open/close 8774 (time 1786037763.80 total 180.26 last 50.95) - open/close 9222 (time 1786037773.81 total 190.26 last 44.78) - open/close 9801 (time 1786037783.82 total 200.27 last 57.84) total: 10000 open/close in 204.03 seconds: 49.01 ops/second - unlinked 0 (time 1786037793 ; total 0 ; last 0) total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.101@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.101@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg301-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.101@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.101@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 mgsnode=192.168.203.101@tcp autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 mgsnode=192.168.203.101@tcp autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:mgsnode=192.168.203.101@tcp lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg301-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server unloading modules via unload_modules_local on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing unload_modules_local oleg301-server: modules unloaded. pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2