== conf-sanity test 69: replace an OST with the same index ========================================================== 22:52:35 (1787367155) start mds service on oleg257-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg257-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 599 (time 1787367249.86 total 10.03 last 59.72) - open/close 1193 (time 1787367259.86 total 20.03 last 59.37) - open/close 1871 (time 1787367269.87 total 30.04 last 67.75) - open/close 2521 (time 1787367279.88 total 40.05 last 64.92) - open/close 3000 (time 1787367289.89 total 50.06 last 47.88) - open/close 3581 (time 1787367299.89 total 60.06 last 58.08) - open/close 4323 (time 1787367309.89 total 70.06 last 74.19) - open/close 5026 (time 1787367319.90 total 80.07 last 70.24) - open/close 5591 (time 1787367329.90 total 90.07 last 56.48) - open/close 6185 (time 1787367339.91 total 100.08 last 59.35) - open/close 6849 (time 1787367349.91 total 110.08 last 66.39) - open/close 7413 (time 1787367359.93 total 120.10 last 56.32) - open/close 7948 (time 1787367369.93 total 130.10 last 53.49) - open/close 8598 (time 1787367379.96 total 140.13 last 64.77) - open/close 9228 (time 1787367389.97 total 150.14 last 62.98) - open/close 9790 (time 1787367399.97 total 160.14 last 56.17) total: 10000 open/close in 163.40 seconds: 61.20 ops/second - unlinked 0 (time 1787367408 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 602 (time 1787367518.42 total 10.00 last 60.18) - open/close 1248 (time 1787367528.42 total 20.01 last 64.58) - open/close 1922 (time 1787367538.43 total 30.01 last 67.38) - open/close 2629 (time 1787367548.43 total 40.02 last 70.65) - open/close 3265 (time 1787367558.45 total 50.04 last 63.47) - open/close 3863 (time 1787367568.46 total 60.04 last 59.79) - open/close 4544 (time 1787367578.46 total 70.04 last 68.09) - open/close 5237 (time 1787367588.47 total 80.05 last 69.23) - open/close 5922 (time 1787367598.47 total 90.06 last 68.46) - open/close 6536 (time 1787367608.49 total 100.07 last 61.32) - open/close 7181 (time 1787367618.51 total 110.10 last 64.32) - open/close 7867 (time 1787367628.54 total 120.12 last 68.45) - open/close 8518 (time 1787367638.54 total 130.13 last 65.04) - open/close 9105 (time 1787367648.56 total 140.14 last 58.61) - open/close 9670 (time 1787367658.58 total 150.17 last 56.36) total: 10000 open/close in 156.15 seconds: 64.04 ops/second - unlinked 0 (time 1787367670 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 709 (time 1787367776.83 total 10.00 last 70.89) - open/close 1340 (time 1787367786.83 total 20.00 last 63.08) - open/close 2034 (time 1787367796.84 total 30.01 last 69.35) - open/close 2802 (time 1787367806.85 total 40.02 last 76.73) - open/close 3595 (time 1787367816.85 total 50.03 last 79.24) - open/close 4229 (time 1787367826.86 total 60.03 last 63.38) - open/close 4906 (time 1787367836.89 total 70.06 last 67.51) - open/close 5571 (time 1787367846.89 total 80.06 last 66.46) - open/close 6214 (time 1787367856.90 total 90.07 last 64.26) - open/close 6943 (time 1787367866.91 total 100.08 last 72.83) - open/close 7678 (time 1787367876.91 total 110.09 last 73.45) - open/close 8200 (time 1787367886.92 total 120.09 last 52.17) - open/close 8785 (time 1787367896.93 total 130.11 last 58.40) - open/close 9427 (time 1787367906.95 total 140.13 last 64.07) total: 10000 open/close in 148.51 seconds: 67.34 ops/second - unlinked 0 (time 1787367919 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 587 (time 1787368022.02 total 10.00 last 58.69) - open/close 1332 (time 1787368032.04 total 20.02 last 74.35) - open/close 2015 (time 1787368042.04 total 30.02 last 68.28) - open/close 2727 (time 1787368052.05 total 40.03 last 71.13) - open/close 3388 (time 1787368062.07 total 50.06 last 65.93) - open/close 4019 (time 1787368072.08 total 60.06 last 63.09) - open/close 4634 (time 1787368082.08 total 70.07 last 61.45) - open/close 5387 (time 1787368092.09 total 80.08 last 75.23) - open/close 6128 (time 1787368102.10 total 90.08 last 74.08) - open/close 6705 (time 1787368112.11 total 100.09 last 57.64) - open/close 7281 (time 1787368122.13 total 110.12 last 57.45) - open/close 7814 (time 1787368132.14 total 120.13 last 53.26) - open/close 8510 (time 1787368142.14 total 130.13 last 69.57) - open/close 9201 (time 1787368152.15 total 140.13 last 69.07) - open/close 9825 (time 1787368162.15 total 150.14 last 62.36) total: 10000 open/close in 152.70 seconds: 65.49 ops/second - unlinked 0 (time 1787368169 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 648 (time 1787368283.67 total 10.01 last 64.72) - open/close 1305 (time 1787368293.70 total 20.04 last 65.51) - open/close 1915 (time 1787368303.70 total 30.04 last 60.99) - open/close 2486 (time 1787368313.70 total 40.04 last 57.09) - open/close 3136 (time 1787368323.71 total 50.05 last 64.94) - open/close 3785 (time 1787368333.71 total 60.06 last 64.88) - open/close 4467 (time 1787368343.72 total 70.07 last 68.11) - open/close 5164 (time 1787368353.73 total 80.08 last 69.66) - open/close 5862 (time 1787368363.74 total 90.09 last 69.71) - open/close 6427 (time 1787368373.75 total 100.10 last 56.44) - open/close 6983 (time 1787368383.80 total 110.14 last 55.37) - open/close 7526 (time 1787368393.81 total 120.15 last 54.23) - open/close 8086 (time 1787368403.81 total 130.16 last 55.99) - open/close 8687 (time 1787368413.82 total 140.17 last 60.02) - open/close 9366 (time 1787368423.82 total 150.17 last 67.89) total: 10000 open/close in 159.36 seconds: 62.75 ops/second - unlinked 0 (time 1787368437 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 657 (time 1787368552.34 total 10.01 last 65.64) - open/close 1312 (time 1787368562.36 total 20.03 last 65.37) - open/close 2048 (time 1787368572.36 total 30.03 last 73.56) - open/close 2749 (time 1787368582.36 total 40.04 last 70.09) - open/close 3476 (time 1787368592.37 total 50.04 last 72.68) - open/close 4198 (time 1787368602.37 total 60.05 last 72.15) - open/close 4849 (time 1787368612.37 total 70.05 last 65.09) - open/close 5481 (time 1787368622.39 total 80.07 last 63.08) - open/close 6110 (time 1787368632.42 total 90.09 last 62.73) - open/close 6717 (time 1787368642.44 total 100.11 last 60.60) - open/close 7395 (time 1787368652.44 total 110.11 last 67.79) - open/close 8026 (time 1787368662.44 total 120.11 last 63.10) - open/close 8743 (time 1787368672.49 total 130.16 last 71.35) - open/close 9467 (time 1787368682.50 total 140.17 last 72.33) total: 10000 open/close in 147.41 seconds: 67.84 ops/second - unlinked 0 (time 1787368694 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 629 (time 1787368799.68 total 10.00 last 62.88) - open/close 1370 (time 1787368809.74 total 20.06 last 73.69) - open/close 2056 (time 1787368819.74 total 30.06 last 68.56) - open/close 2688 (time 1787368829.76 total 40.08 last 63.08) - open/close 3270 (time 1787368839.77 total 50.09 last 58.15) - open/close 3867 (time 1787368849.77 total 60.09 last 59.69) - open/close 4629 (time 1787368859.77 total 70.09 last 76.20) - open/close 5265 (time 1787368869.78 total 80.10 last 63.54) - open/close 5799 (time 1787368879.82 total 90.14 last 53.18) - open/close 6426 (time 1787368889.83 total 100.15 last 62.68) - open/close 7039 (time 1787368899.84 total 110.16 last 61.23) - open/close 7826 (time 1787368909.84 total 120.16 last 78.69) - open/close 8515 (time 1787368919.84 total 130.16 last 68.88) - open/close 9255 (time 1787368929.84 total 140.17 last 73.97) - open/close 9953 (time 1787368939.85 total 150.17 last 69.76) total: 10000 open/close in 150.70 seconds: 66.36 ops/second - unlinked 0 (time 1787368944 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 661 (time 1787369045.75 total 10.00 last 66.10) - open/close 1381 (time 1787369055.77 total 20.02 last 71.87) - open/close 1930 (time 1787369065.78 total 30.03 last 54.83) - open/close 2560 (time 1787369075.79 total 40.04 last 62.95) - open/close 3252 (time 1787369085.79 total 50.04 last 69.17) - open/close 3933 (time 1787369095.80 total 60.04 last 68.10) - open/close 4591 (time 1787369105.80 total 70.05 last 65.78) - open/close 5232 (time 1787369115.80 total 80.05 last 64.10) - open/close 5882 (time 1787369125.80 total 90.05 last 64.97) - open/close 6568 (time 1787369135.81 total 100.06 last 68.54) - open/close 7282 (time 1787369145.82 total 110.07 last 71.35) - open/close 7951 (time 1787369155.84 total 120.08 last 66.77) - open/close 8651 (time 1787369165.84 total 130.09 last 69.95) - open/close 9403 (time 1787369175.85 total 140.10 last 75.14) total: 10000 open/close in 147.42 seconds: 67.83 ops/second - unlinked 0 (time 1787369187 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 842 (time 1787369296.25 total 10.00 last 84.18) - open/close 1627 (time 1787369306.26 total 20.01 last 78.45) - open/close 2295 (time 1787369316.27 total 30.02 last 66.73) - open/close 2965 (time 1787369326.27 total 40.02 last 66.99) - open/close 3603 (time 1787369336.27 total 50.02 last 63.80) - open/close 4197 (time 1787369346.27 total 60.02 last 59.38) - open/close 4636 (time 1787369356.28 total 70.03 last 43.87) - open/close 5160 (time 1787369366.29 total 80.04 last 52.36) - open/close 5767 (time 1787369376.31 total 90.07 last 60.53) - open/close 6410 (time 1787369386.32 total 100.07 last 64.24) - open/close 7060 (time 1787369396.33 total 110.08 last 64.97) - open/close 7870 (time 1787369406.33 total 120.08 last 80.95) - open/close 8651 (time 1787369416.33 total 130.09 last 78.09) - open/close 9415 (time 1787369426.34 total 140.09 last 76.36) total: 10000 open/close in 149.35 seconds: 66.96 ops/second - unlinked 0 (time 1787369440 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 651 (time 1787369545.62 total 10.00 last 65.08) - open/close 1368 (time 1787369555.63 total 20.01 last 71.63) - open/close 2121 (time 1787369565.65 total 30.03 last 75.15) - open/close 2772 (time 1787369575.65 total 40.03 last 65.09) - open/close 3432 (time 1787369585.66 total 50.04 last 65.94) - open/close 4152 (time 1787369595.66 total 60.04 last 71.98) - open/close 4684 (time 1787369605.67 total 70.06 last 53.15) - open/close 5342 (time 1787369615.67 total 80.06 last 65.78) - open/close 6007 (time 1787369625.68 total 90.07 last 66.42) - open/close 6667 (time 1787369635.69 total 100.08 last 65.94) - open/close 7382 (time 1787369645.70 total 110.09 last 71.42) - open/close 8065 (time 1787369655.71 total 120.10 last 68.26) - open/close 8776 (time 1787369665.75 total 130.13 last 70.84) - open/close 9480 (time 1787369675.76 total 140.14 last 70.32) total: 10000 open/close in 147.53 seconds: 67.78 ops/second - unlinked 0 (time 1787369687 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.157@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.157@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.157@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.157@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.202.157@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2