== conf-sanity test 69: replace an OST with the same index ========================================================== 21:18:56 (1763345936) start mds service on oleg152-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg152-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 658 (time 1763346010.14 total 10.02 last 65.68) - open/close 1284 (time 1763346020.17 total 20.04 last 62.43) - open/close 1911 (time 1763346030.18 total 30.05 last 62.65) - open/close 2594 (time 1763346040.18 total 40.05 last 68.29) - open/close 3178 (time 1763346050.19 total 50.06 last 58.35) - open/close 3833 (time 1763346060.19 total 60.06 last 65.48) - open/close 4511 (time 1763346070.19 total 70.07 last 67.80) - open/close 5170 (time 1763346080.20 total 80.07 last 65.86) - open/close 5839 (time 1763346090.20 total 90.08 last 66.86) - open/close 6522 (time 1763346100.20 total 100.08 last 68.29) - open/close 7171 (time 1763346110.21 total 110.09 last 64.83) - open/close 7770 (time 1763346120.23 total 120.11 last 59.79) - open/close 8299 (time 1763346130.25 total 130.12 last 52.83) - open/close 8841 (time 1763346140.25 total 140.12 last 54.19) - open/close 9484 (time 1763346150.26 total 150.14 last 64.22) total: 10000 open/close in 157.80 seconds: 63.37 ops/second - unlinked 0 (time 1763346162 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 676 (time 1763346268.87 total 10.01 last 67.50) - open/close 1317 (time 1763346278.87 total 20.02 last 64.10) - open/close 1971 (time 1763346288.88 total 30.02 last 65.36) - open/close 2638 (time 1763346298.89 total 40.03 last 66.63) - open/close 3312 (time 1763346308.91 total 50.06 last 67.24) - open/close 3918 (time 1763346318.91 total 60.06 last 60.60) - open/close 4530 (time 1763346328.92 total 70.06 last 61.16) - open/close 5192 (time 1763346338.92 total 80.07 last 66.18) - open/close 5888 (time 1763346348.93 total 90.07 last 69.55) - open/close 6554 (time 1763346358.93 total 100.08 last 66.58) - open/close 7188 (time 1763346368.94 total 110.08 last 63.39) - open/close 7902 (time 1763346378.95 total 120.09 last 71.29) - open/close 8566 (time 1763346388.97 total 130.11 last 66.27) - open/close 9188 (time 1763346398.97 total 140.12 last 62.19) - open/close 9840 (time 1763346408.98 total 150.12 last 65.17) total: 10000 open/close in 152.37 seconds: 65.63 ops/second - unlinked 0 (time 1763346414 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 703 (time 1763346510.68 total 10.01 last 70.22) - open/close 1392 (time 1763346520.69 total 20.02 last 68.85) - open/close 2110 (time 1763346530.70 total 30.03 last 71.70) - open/close 2783 (time 1763346540.72 total 40.05 last 67.16) - open/close 3480 (time 1763346550.73 total 50.06 last 69.66) - open/close 4174 (time 1763346560.74 total 60.07 last 69.30) - open/close 4889 (time 1763346570.75 total 70.08 last 71.47) - open/close 5574 (time 1763346580.77 total 80.10 last 68.34) - open/close 6236 (time 1763346590.78 total 90.11 last 66.13) - open/close 6918 (time 1763346600.79 total 100.12 last 68.12) - open/close 7617 (time 1763346610.80 total 110.13 last 69.87) - open/close 8295 (time 1763346620.80 total 120.13 last 67.79) - open/close 8986 (time 1763346630.82 total 130.15 last 68.96) - open/close 9709 (time 1763346640.83 total 140.16 last 72.23) total: 10000 open/close in 143.73 seconds: 69.58 ops/second - unlinked 0 (time 1763346647 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 790 (time 1763346735.27 total 10.01 last 78.91) - open/close 1474 (time 1763346745.27 total 20.02 last 68.36) - open/close 2158 (time 1763346755.28 total 30.02 last 68.36) - open/close 2859 (time 1763346765.29 total 40.03 last 70.04) - open/close 3584 (time 1763346775.30 total 50.04 last 72.43) - open/close 4323 (time 1763346785.31 total 60.05 last 73.81) - open/close 5066 (time 1763346795.32 total 70.06 last 74.23) - open/close 5764 (time 1763346805.32 total 80.06 last 69.78) - open/close 6460 (time 1763346815.33 total 90.07 last 69.56) - open/close 7154 (time 1763346825.34 total 100.09 last 69.30) - open/close 7867 (time 1763346835.34 total 110.09 last 71.30) - open/close 8597 (time 1763346845.35 total 120.09 last 72.97) - open/close 9237 (time 1763346855.35 total 130.10 last 63.95) - open/close 9934 (time 1763346865.36 total 140.10 last 69.66) total: 10000 open/close in 140.96 seconds: 70.94 ops/second - unlinked 0 (time 1763346869 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 703 (time 1763346963.97 total 10.02 last 70.18) - open/close 1397 (time 1763346973.97 total 20.02 last 69.36) - open/close 2128 (time 1763346983.97 total 30.02 last 73.09) - open/close 2858 (time 1763346993.98 total 40.03 last 72.98) - open/close 3579 (time 1763347003.98 total 50.03 last 72.05) - open/close 4329 (time 1763347013.99 total 60.04 last 74.96) - open/close 5029 (time 1763347023.99 total 70.04 last 69.97) - open/close 5707 (time 1763347034.00 total 80.05 last 67.76) - open/close 6369 (time 1763347044.01 total 90.06 last 66.10) - open/close 7026 (time 1763347054.02 total 100.07 last 65.64) - open/close 7698 (time 1763347064.04 total 110.09 last 67.07) - open/close 8365 (time 1763347074.05 total 120.10 last 66.63) - open/close 9011 (time 1763347084.08 total 130.13 last 64.43) - open/close 9699 (time 1763347094.08 total 140.14 last 68.76) total: 10000 open/close in 145.55 seconds: 68.71 ops/second - unlinked 0 (time 1763347103 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 674 (time 1763347205.08 total 10.01 last 67.36) - open/close 1301 (time 1763347215.10 total 20.03 last 62.58) - open/close 1907 (time 1763347225.10 total 30.03 last 60.58) - open/close 2455 (time 1763347235.11 total 40.03 last 54.78) - open/close 2950 (time 1763347245.11 total 50.04 last 49.47) - open/close 3623 (time 1763347255.11 total 60.04 last 67.29) - open/close 4282 (time 1763347265.12 total 70.04 last 65.88) - open/close 5003 (time 1763347275.12 total 80.05 last 72.08) - open/close 5667 (time 1763347285.12 total 90.05 last 66.36) - open/close 6335 (time 1763347295.13 total 100.06 last 66.76) - open/close 7029 (time 1763347305.13 total 110.06 last 69.37) - open/close 7674 (time 1763347315.14 total 120.06 last 64.48) - open/close 8301 (time 1763347325.15 total 130.08 last 62.62) - open/close 8985 (time 1763347335.16 total 140.09 last 68.34) - open/close 9612 (time 1763347345.17 total 150.10 last 62.64) total: 10000 open/close in 155.75 seconds: 64.21 ops/second - unlinked 0 (time 1763347355 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 714 (time 1763347458.59 total 10.01 last 71.34) - open/close 1376 (time 1763347468.60 total 20.02 last 66.15) - open/close 2030 (time 1763347478.61 total 30.02 last 65.35) - open/close 2669 (time 1763347488.62 total 40.03 last 63.84) - open/close 3300 (time 1763347498.62 total 50.04 last 63.05) - open/close 3906 (time 1763347508.63 total 60.05 last 60.56) - open/close 4596 (time 1763347518.64 total 70.05 last 68.95) - open/close 5272 (time 1763347528.65 total 80.06 last 67.55) - open/close 5929 (time 1763347538.66 total 90.08 last 65.60) - open/close 6619 (time 1763347548.66 total 100.08 last 68.98) - open/close 7249 (time 1763347558.68 total 110.10 last 62.90) - open/close 7878 (time 1763347568.69 total 120.11 last 62.83) - open/close 8553 (time 1763347578.70 total 130.12 last 67.46) - open/close 9164 (time 1763347588.70 total 140.12 last 61.09) - open/close 9817 (time 1763347598.72 total 150.14 last 65.17) total: 10000 open/close in 152.58 seconds: 65.54 ops/second - unlinked 0 (time 1763347606 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 581 (time 1763347716.49 total 10.02 last 57.97) - open/close 1219 (time 1763347726.50 total 20.03 last 63.73) - open/close 1880 (time 1763347736.51 total 30.04 last 66.07) - open/close 2527 (time 1763347746.52 total 40.05 last 64.63) - open/close 3225 (time 1763347756.53 total 50.06 last 69.74) - open/close 3839 (time 1763347766.53 total 60.07 last 61.35) - open/close 4452 (time 1763347776.53 total 70.07 last 61.30) - open/close 5078 (time 1763347786.55 total 80.08 last 62.51) - open/close 5710 (time 1763347796.55 total 90.09 last 63.17) - open/close 6363 (time 1763347806.56 total 100.09 last 65.27) - open/close 7008 (time 1763347816.57 total 110.11 last 64.41) - open/close 7677 (time 1763347826.59 total 120.12 last 66.79) - open/close 8305 (time 1763347836.60 total 130.13 last 62.73) - open/close 8968 (time 1763347846.61 total 140.14 last 66.25) - open/close 9543 (time 1763347856.61 total 150.14 last 57.48) total: 10000 open/close in 156.78 seconds: 63.78 ops/second - unlinked 0 (time 1763347867 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 659 (time 1763347972.13 total 10.00 last 65.90) - open/close 1261 (time 1763347982.13 total 20.00 last 60.17) - open/close 1935 (time 1763347992.15 total 30.02 last 67.28) - open/close 2620 (time 1763348002.16 total 40.03 last 68.42) - open/close 3315 (time 1763348012.19 total 50.06 last 69.31) - open/close 4010 (time 1763348022.19 total 60.06 last 69.48) - open/close 4663 (time 1763348032.19 total 70.07 last 65.29) - open/close 5377 (time 1763348042.21 total 80.08 last 71.28) - open/close 6083 (time 1763348052.22 total 90.09 last 70.53) - open/close 6705 (time 1763348062.23 total 100.10 last 62.13) - open/close 7401 (time 1763348072.24 total 110.11 last 69.53) - open/close 8101 (time 1763348082.25 total 120.12 last 69.98) - open/close 8690 (time 1763348092.25 total 130.12 last 58.85) - open/close 9398 (time 1763348102.26 total 140.13 last 70.74) total: 10000 open/close in 149.25 seconds: 67.00 ops/second - unlinked 0 (time 1763348114 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 731 (time 1763348214.20 total 10.01 last 72.99) - open/close 1419 (time 1763348224.21 total 20.02 last 68.77) - open/close 2089 (time 1763348234.21 total 30.02 last 67.00) - open/close 2754 (time 1763348244.21 total 40.02 last 66.48) - open/close 3422 (time 1763348254.22 total 50.04 last 66.70) - open/close 4122 (time 1763348264.23 total 60.05 last 69.92) - open/close 4880 (time 1763348274.24 total 70.05 last 75.78) - open/close 5545 (time 1763348284.24 total 80.05 last 66.47) - open/close 6170 (time 1763348294.24 total 90.06 last 62.49) - open/close 6829 (time 1763348304.24 total 100.06 last 65.88) - open/close 7474 (time 1763348314.26 total 110.07 last 64.42) - open/close 8164 (time 1763348324.27 total 120.08 last 68.90) - open/close 8832 (time 1763348334.28 total 130.09 last 66.75) - open/close 9509 (time 1763348344.29 total 140.10 last 67.65) total: 10000 open/close in 147.80 seconds: 67.66 ops/second - unlinked 0 (time 1763348356 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.152@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.152@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg152-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.152@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.152@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.152@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.152@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.201.152@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg152-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local oleg152-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg152-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2