== conf-sanity test 69: replace an OST with the same index ========================================================== 01:38:09 (1775194689) start mds service on oleg649-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg649-server' oleg649-server: oleg649-server.virtnet: executing load_modules_local oleg649-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg649-server: Loading modules from /home/green/git/lustre-release/lustre oleg649-server: detected 4 online CPUs by sysfs oleg649-server: MODOPTS_LIBCFS= oleg649-server: Force libcfs to create 2 CPU partitions oleg649-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg649-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg649-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg649-server: oleg649-server.virtnet: executing set_default_debug -1 all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg649-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg649-server: oleg649-server.virtnet: executing set_default_debug -1 all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-OST0000 oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg649-client.virtnet: -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 563 (time 1775194769.91 total 10.00 last 56.30) - open/close 1105 (time 1775194779.93 total 20.01 last 54.13) - open/close 1626 (time 1775194789.94 total 30.03 last 52.01) - open/close 2249 (time 1775194799.95 total 40.04 last 62.27) - open/close 2911 (time 1775194809.96 total 50.04 last 66.16) - open/close 3582 (time 1775194819.96 total 60.05 last 67.06) - open/close 4250 (time 1775194829.97 total 70.06 last 66.73) - open/close 4925 (time 1775194839.98 total 80.06 last 67.49) - open/close 5582 (time 1775194849.99 total 90.07 last 65.62) - open/close 6258 (time 1775194859.99 total 100.08 last 67.59) - open/close 6969 (time 1775194869.99 total 110.08 last 71.10) - open/close 7576 (time 1775194880.00 total 120.08 last 60.66) - open/close 8130 (time 1775194890.00 total 130.09 last 55.36) - open/close 8800 (time 1775194900.01 total 140.09 last 66.98) - open/close 9462 (time 1775194910.01 total 150.10 last 66.17) - open/close 9982 (time 1775194920.02 total 160.10 last 51.97) total: 10000 open/close in 160.46 seconds: 62.32 ops/second - unlinked 0 (time 1775194924 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 652 (time 1775195029.85 total 10.01 last 65.15) - open/close 1324 (time 1775195039.86 total 20.02 last 67.14) - open/close 1999 (time 1775195049.87 total 30.02 last 67.49) - open/close 2659 (time 1775195059.87 total 40.02 last 65.97) - open/close 3378 (time 1775195069.88 total 50.03 last 71.82) - open/close 4073 (time 1775195079.90 total 60.05 last 69.37) - open/close 4767 (time 1775195089.91 total 70.07 last 69.31) - open/close 5444 (time 1775195099.91 total 80.07 last 67.69) - open/close 6131 (time 1775195109.92 total 90.07 last 68.67) - open/close 6881 (time 1775195119.92 total 100.07 last 74.99) - open/close 7574 (time 1775195129.92 total 110.07 last 69.29) - open/close 8281 (time 1775195139.92 total 120.08 last 70.69) - open/close 8950 (time 1775195149.94 total 130.09 last 66.81) - open/close 9632 (time 1775195159.94 total 140.10 last 68.16) total: 10000 open/close in 145.93 seconds: 68.53 ops/second - unlinked 0 (time 1775195169 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 698 (time 1775195278.70 total 10.00 last 69.77) - open/close 1476 (time 1775195288.72 total 20.02 last 77.69) - open/close 2186 (time 1775195298.72 total 30.02 last 70.97) - open/close 2877 (time 1775195308.73 total 40.03 last 69.05) - open/close 3662 (time 1775195318.73 total 50.03 last 78.49) - open/close 4362 (time 1775195328.73 total 60.03 last 69.99) - open/close 5075 (time 1775195338.74 total 70.04 last 71.27) - open/close 5748 (time 1775195348.74 total 80.05 last 67.24) - open/close 6453 (time 1775195358.75 total 90.05 last 70.44) - open/close 7141 (time 1775195368.76 total 100.06 last 68.77) - open/close 7799 (time 1775195378.77 total 110.07 last 65.75) - open/close 8466 (time 1775195388.77 total 120.07 last 66.67) - open/close 9160 (time 1775195398.79 total 130.09 last 69.29) - open/close 9804 (time 1775195408.80 total 140.10 last 64.32) total: 10000 open/close in 143.03 seconds: 69.91 ops/second - unlinked 0 (time 1775195416 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 728 (time 1775195520.67 total 10.00 last 72.77) - open/close 1447 (time 1775195530.70 total 20.03 last 71.71) - open/close 2150 (time 1775195540.71 total 30.05 last 70.19) - open/close 2863 (time 1775195550.73 total 40.07 last 71.14) - open/close 3528 (time 1775195560.74 total 50.08 last 66.45) - open/close 4208 (time 1775195570.75 total 60.08 last 67.95) - open/close 4893 (time 1775195580.78 total 70.11 last 68.31) - open/close 5566 (time 1775195590.79 total 80.13 last 67.19) - open/close 6294 (time 1775195600.79 total 90.13 last 72.79) - open/close 6994 (time 1775195610.80 total 100.13 last 69.97) - open/close 7719 (time 1775195620.80 total 110.14 last 72.49) - open/close 8393 (time 1775195630.81 total 120.15 last 67.31) - open/close 9070 (time 1775195640.82 total 130.15 last 67.66) - open/close 9709 (time 1775195650.83 total 140.16 last 63.84) total: 10000 open/close in 144.68 seconds: 69.12 ops/second - unlinked 0 (time 1775195660 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 671 (time 1775195769.38 total 10.01 last 67.03) - open/close 1347 (time 1775195779.38 total 20.02 last 67.55) - open/close 1991 (time 1775195789.41 total 30.04 last 64.25) - open/close 2562 (time 1775195799.41 total 40.04 last 57.09) - open/close 3223 (time 1775195809.41 total 50.04 last 66.09) - open/close 3863 (time 1775195819.42 total 60.06 last 63.91) - open/close 4527 (time 1775195829.43 total 70.07 last 66.33) - open/close 4993 (time 1775195839.44 total 80.07 last 46.59) - open/close 5634 (time 1775195849.44 total 90.08 last 64.05) - open/close 6308 (time 1775195859.45 total 100.09 last 67.33) - open/close 6943 (time 1775195869.46 total 110.10 last 63.43) - open/close 7673 (time 1775195879.49 total 120.12 last 72.84) - open/close 8338 (time 1775195889.49 total 130.12 last 66.48) - open/close 9014 (time 1775195899.51 total 140.14 last 67.47) - open/close 9634 (time 1775195909.53 total 150.16 last 61.89) total: 10000 open/close in 155.68 seconds: 64.23 ops/second - unlinked 0 (time 1775195920 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 653 (time 1775196038.77 total 10.00 last 65.27) - open/close 1356 (time 1775196048.78 total 20.01 last 70.26) - open/close 2031 (time 1775196058.78 total 30.01 last 67.49) - open/close 2730 (time 1775196068.79 total 40.02 last 69.83) - open/close 3513 (time 1775196078.80 total 50.03 last 78.22) - open/close 4252 (time 1775196088.81 total 60.05 last 73.80) - open/close 4875 (time 1775196098.84 total 70.07 last 62.14) - open/close 5569 (time 1775196108.88 total 80.11 last 69.13) - open/close 6230 (time 1775196118.88 total 90.12 last 66.07) - open/close 6927 (time 1775196128.90 total 100.13 last 69.62) - open/close 7582 (time 1775196138.90 total 110.13 last 65.46) - open/close 8216 (time 1775196148.92 total 120.15 last 63.29) - open/close 8887 (time 1775196158.93 total 130.16 last 67.02) - open/close 9513 (time 1775196168.94 total 140.17 last 62.54) total: 10000 open/close in 147.24 seconds: 67.92 ops/second - unlinked 0 (time 1775196180 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 682 (time 1775196291.26 total 10.00 last 68.18) - open/close 1203 (time 1775196301.27 total 20.01 last 52.06) - open/close 1872 (time 1775196311.27 total 30.01 last 66.90) - open/close 2609 (time 1775196321.27 total 40.01 last 73.69) - open/close 3283 (time 1775196331.27 total 50.02 last 67.38) - open/close 4000 (time 1775196341.27 total 60.02 last 71.69) - open/close 4707 (time 1775196351.29 total 70.03 last 70.61) - open/close 5487 (time 1775196361.30 total 80.04 last 77.90) - open/close 6167 (time 1775196371.31 total 90.05 last 67.94) - open/close 6903 (time 1775196381.33 total 100.07 last 73.45) - open/close 7496 (time 1775196391.34 total 110.08 last 59.24) - open/close 8149 (time 1775196401.34 total 120.09 last 65.28) - open/close 8825 (time 1775196411.35 total 130.10 last 67.52) - open/close 9430 (time 1775196421.36 total 140.10 last 60.46) total: 10000 open/close in 148.76 seconds: 67.22 ops/second - unlinked 0 (time 1775196434 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 366 (time 1775196542.46 total 10.00 last 36.60) - open/close 833 (time 1775196552.47 total 20.01 last 46.66) - open/close 1279 (time 1775196562.47 total 30.02 last 44.56) - open/close 1622 (time 1775196572.51 total 40.05 last 34.18) - open/close 2123 (time 1775196582.54 total 50.09 last 49.94) - open/close 2839 (time 1775196592.54 total 60.09 last 71.59) - open/close 3588 (time 1775196602.56 total 70.10 last 74.78) - open/close 4286 (time 1775196612.56 total 80.11 last 69.77) - open/close 4902 (time 1775196622.58 total 90.13 last 61.48) - open/close 5597 (time 1775196632.58 total 100.13 last 69.49) - open/close 6269 (time 1775196642.59 total 110.13 last 67.17) - open/close 6930 (time 1775196652.61 total 120.15 last 65.97) - open/close 7650 (time 1775196662.63 total 130.17 last 71.87) - open/close 8331 (time 1775196672.65 total 140.20 last 67.92) - open/close 8986 (time 1775196682.68 total 150.22 last 65.33) - open/close 9581 (time 1775196692.70 total 160.25 last 59.34) total: 10000 open/close in 166.48 seconds: 60.07 ops/second - unlinked 0 (time 1775196703 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 587 (time 1775196814.90 total 10.01 last 58.62) - open/close 1120 (time 1775196824.91 total 20.02 last 53.26) - open/close 1730 (time 1775196834.92 total 30.04 last 60.90) - open/close 2306 (time 1775196844.94 total 40.06 last 57.49) - open/close 2873 (time 1775196854.95 total 50.07 last 56.64) - open/close 3375 (time 1775196864.96 total 60.07 last 50.18) - open/close 3906 (time 1775196874.96 total 70.08 last 53.06) - open/close 4565 (time 1775196884.98 total 80.09 last 65.80) - open/close 5168 (time 1775196894.98 total 90.10 last 60.27) - open/close 5796 (time 1775196904.99 total 100.11 last 62.76) - open/close 6315 (time 1775196915.00 total 110.12 last 51.83) - open/close 6848 (time 1775196925.01 total 120.12 last 53.28) - open/close 7340 (time 1775196935.01 total 130.13 last 49.16) - open/close 7869 (time 1775196945.02 total 140.14 last 52.86) - open/close 8472 (time 1775196955.03 total 150.14 last 60.26) - open/close 9091 (time 1775196965.03 total 160.15 last 61.88) - open/close 9720 (time 1775196975.04 total 170.15 last 62.85) total: 10000 open/close in 175.14 seconds: 57.10 ops/second - unlinked 0 (time 1775196985 ; total 0 ; last 0) total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second - open/close 616 (time 1775197109.01 total 10.00 last 61.59) - open/close 1225 (time 1775197119.02 total 20.01 last 60.85) - open/close 1909 (time 1775197129.02 total 30.01 last 68.37) - open/close 2532 (time 1775197139.04 total 40.03 last 62.20) - open/close 3167 (time 1775197149.04 total 50.03 last 63.48) - open/close 3788 (time 1775197159.05 total 60.04 last 62.06) - open/close 4401 (time 1775197169.05 total 70.05 last 61.27) - open/close 5013 (time 1775197179.06 total 80.05 last 61.14) - open/close 5659 (time 1775197189.07 total 90.06 last 64.57) - open/close 6290 (time 1775197199.07 total 100.07 last 63.05) - open/close 6931 (time 1775197209.13 total 110.12 last 63.77) - open/close 7463 (time 1775197219.14 total 120.14 last 53.10) - open/close 8041 (time 1775197229.15 total 130.14 last 57.78) - open/close 8608 (time 1775197239.16 total 140.15 last 56.62) - open/close 9139 (time 1775197249.19 total 150.18 last 52.98) - open/close 9656 (time 1775197259.19 total 160.18 last 51.69) total: 10000 open/close in 167.44 seconds: 59.72 ops/second - unlinked 0 (time 1775197273 ; total 0 ; last 0) total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg649-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg649-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg649-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.206.149@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.149@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg649-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg649-server: oleg649-server.virtnet: executing set_default_debug -1 all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-OST0000 oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg649-server: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg649-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg649-client.virtnet: -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg649-client: oleg649-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg649-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg649-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg649-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.206.149@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.149@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.206.149@tcp sys.timeout=20 autodegrade=on Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.206.149@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.149@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg649-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg649-server: oleg649-server.virtnet: executing set_default_debug -1 all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-OST0000 oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg649-server: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg649-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg649-client.virtnet: -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg649-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg649-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg649-server stop mds service on oleg649-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg649-server unloading modules via unload_modules_local on: 'oleg649-server' oleg649-server: oleg649-server.virtnet: executing unload_modules_local oleg649-server: modules unloaded. pdsh@oleg649-client: oleg649-client: ssh exited with exit code 2 pdsh@oleg649-client: oleg649-server: ssh exited with exit code 2 pdsh@oleg649-client: oleg649-client: ssh exited with exit code 2 pdsh@oleg649-client: oleg649-server: ssh exited with exit code 2