== conf-sanity test 69: replace an OST with the same index ========================================================== 02:13:14 (1778652794) start mds service on oleg306-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg306-server: detected 4 online CPUs by sysfs oleg306-server: MODOPTS_LIBCFS= oleg306-server: Force libcfs to create 2 CPU partitions oleg306-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg306-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg306-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg306-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg306-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 1012 (time 1778652862.94 total 10.01 last 101.14) - open/close 1994 (time 1778652872.94 total 20.01 last 98.19) - open/close 3043 (time 1778652882.95 total 30.01 last 104.86) - open/close 4014 (time 1778652892.96 total 40.02 last 97.00) - open/close 4920 (time 1778652902.96 total 50.02 last 90.58) - open/close 5879 (time 1778652912.97 total 60.03 last 95.84) - open/close 6942 (time 1778652922.98 total 70.04 last 106.18) - open/close 7973 (time 1778652932.98 total 80.05 last 103.04) - open/close 9038 (time 1778652942.99 total 90.05 last 106.44) total: 10000 open/close in 99.09 seconds: 100.91 ops/second - unlinked 0 (time 1778652954 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 461 (time 1778653070.17 total 10.01 last 46.05) - open/close 1043 (time 1778653080.18 total 20.03 last 58.12) - open/close 1561 (time 1778653090.18 total 30.03 last 51.78) - open/close 2229 (time 1778653100.19 total 40.03 last 66.77) - open/close 2880 (time 1778653110.20 total 50.05 last 64.99) - open/close 3475 (time 1778653120.21 total 60.06 last 59.46) - open/close 4018 (time 1778653130.25 total 70.09 last 54.11) - open/close 4504 (time 1778653140.25 total 80.09 last 48.59) - open/close 5140 (time 1778653150.25 total 90.10 last 63.57) - open/close 5718 (time 1778653160.27 total 100.12 last 57.68) - open/close 6156 (time 1778653170.29 total 110.14 last 43.71) - open/close 6677 (time 1778653180.30 total 120.15 last 52.06) - open/close 7243 (time 1778653190.31 total 130.15 last 56.58) - open/close 7803 (time 1778653200.33 total 140.18 last 55.88) - open/close 8383 (time 1778653210.33 total 150.18 last 57.98) - open/close 8999 (time 1778653220.34 total 160.18 last 61.57) - open/close 9606 (time 1778653230.34 total 170.19 last 60.68) total: 10000 open/close in 176.18 seconds: 56.76 ops/second - unlinked 0 (time 1778653240 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 725 (time 1778653340.79 total 10.01 last 72.43) - open/close 1441 (time 1778653350.79 total 20.01 last 71.57) - open/close 2143 (time 1778653360.80 total 30.02 last 70.18) - open/close 2838 (time 1778653370.81 total 40.03 last 69.41) - open/close 3456 (time 1778653380.82 total 50.04 last 61.75) - open/close 3995 (time 1778653390.82 total 60.05 last 53.86) - open/close 4496 (time 1778653400.84 total 70.06 last 50.03) - open/close 5052 (time 1778653410.84 total 80.06 last 55.59) - open/close 5837 (time 1778653420.85 total 90.07 last 78.44) - open/close 6611 (time 1778653430.87 total 100.09 last 77.22) - open/close 7368 (time 1778653440.88 total 110.10 last 75.65) - open/close 8075 (time 1778653450.88 total 120.10 last 70.66) - open/close 8750 (time 1778653460.88 total 130.10 last 67.49) - open/close 9527 (time 1778653470.89 total 140.11 last 77.67) total: 10000 open/close in 145.57 seconds: 68.70 ops/second - unlinked 0 (time 1778653479 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 894 (time 1778653555.24 total 10.01 last 89.35) - open/close 1817 (time 1778653565.24 total 20.01 last 92.30) - open/close 2802 (time 1778653575.25 total 30.01 last 98.42) - open/close 3838 (time 1778653585.25 total 40.02 last 103.57) - open/close 4873 (time 1778653595.27 total 50.03 last 103.34) - open/close 5811 (time 1778653605.27 total 60.04 last 93.75) - open/close 6635 (time 1778653615.28 total 70.04 last 82.34) - open/close 7512 (time 1778653625.30 total 80.06 last 87.58) - open/close 8419 (time 1778653635.30 total 90.06 last 90.67) - open/close 9364 (time 1778653645.31 total 100.07 last 94.40) total: 10000 open/close in 106.29 seconds: 94.08 ops/second - unlinked 0 (time 1778653654 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 930 (time 1778653727.33 total 10.01 last 92.95) - open/close 1814 (time 1778653737.34 total 20.02 last 88.28) - open/close 2785 (time 1778653747.35 total 30.02 last 97.04) - open/close 3699 (time 1778653757.35 total 40.03 last 91.39) - open/close 4659 (time 1778653767.36 total 50.04 last 95.91) - open/close 5617 (time 1778653777.37 total 60.04 last 95.74) - open/close 6591 (time 1778653787.38 total 70.05 last 97.28) - open/close 7573 (time 1778653797.38 total 80.06 last 98.17) - open/close 8503 (time 1778653807.39 total 90.07 last 92.92) - open/close 9496 (time 1778653817.40 total 100.07 last 99.24) total: 10000 open/close in 104.99 seconds: 95.24 ops/second - unlinked 0 (time 1778653824 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 998 (time 1778653887.31 total 10.00 last 99.77) - open/close 2124 (time 1778653897.31 total 20.00 last 112.60) - open/close 3142 (time 1778653907.32 total 30.01 last 101.74) - open/close 4221 (time 1778653917.32 total 40.01 last 107.86) - open/close 5369 (time 1778653927.33 total 50.02 last 114.76) - open/close 6563 (time 1778653937.33 total 60.02 last 119.36) - open/close 7811 (time 1778653947.34 total 70.03 last 124.71) - open/close 9040 (time 1778653957.34 total 80.03 last 122.89) total: 10000 open/close in 87.01 seconds: 114.92 ops/second - unlinked 0 (time 1778653966 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1164 (time 1778654018.39 total 10.01 last 116.31) - open/close 2341 (time 1778654028.39 total 20.01 last 117.69) - open/close 3600 (time 1778654038.39 total 30.01 last 125.84) - open/close 5089 (time 1778654048.40 total 40.02 last 148.84) - open/close 6582 (time 1778654058.40 total 50.03 last 149.19) - open/close 7978 (time 1778654068.41 total 60.03 last 139.57) - open/close 9444 (time 1778654078.41 total 70.03 last 146.49) total: 10000 open/close in 73.24 seconds: 136.54 ops/second - unlinked 0 (time 1778654083 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1489 (time 1778654127.35 total 10.00 last 148.83) - open/close 2314 (time 1778654137.44 total 20.09 last 81.79) - open/close 2803 (time 1778654147.47 total 30.12 last 48.77) - open/close 3266 (time 1778654157.48 total 40.13 last 46.25) - open/close 3772 (time 1778654167.50 total 50.15 last 50.51) - open/close 4229 (time 1778654177.51 total 60.17 last 45.62) - open/close 4615 (time 1778654187.55 total 70.20 last 38.47) - open/close 4933 (time 1778654197.55 total 80.20 last 31.79) - open/close 5325 (time 1778654207.57 total 90.22 last 39.15) - open/close 5835 (time 1778654217.57 total 100.22 last 50.99) - open/close 6436 (time 1778654227.57 total 110.23 last 60.07) - open/close 7067 (time 1778654237.58 total 120.23 last 63.06) - open/close 7623 (time 1778654247.59 total 130.24 last 55.54) - open/close 8158 (time 1778654257.60 total 140.25 last 53.46) - open/close 8716 (time 1778654267.61 total 150.26 last 55.74) - open/close 9443 (time 1778654277.61 total 160.27 last 72.68) total: 10000 open/close in 169.23 seconds: 59.09 ops/second - unlinked 0 (time 1778654290 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 723 (time 1778654392.70 total 10.01 last 72.22) - open/close 1466 (time 1778654402.70 total 20.02 last 74.25) - open/close 2121 (time 1778654412.71 total 30.03 last 65.45) - open/close 2792 (time 1778654422.72 total 40.03 last 67.04) - open/close 3399 (time 1778654432.73 total 50.04 last 60.64) - open/close 4092 (time 1778654442.73 total 60.05 last 69.28) - open/close 4803 (time 1778654452.75 total 70.06 last 71.01) - open/close 5535 (time 1778654462.75 total 80.06 last 73.17) - open/close 6324 (time 1778654472.76 total 90.07 last 78.86) - open/close 7032 (time 1778654482.76 total 100.07 last 70.76) - open/close 7755 (time 1778654492.77 total 110.09 last 72.22) - open/close 8456 (time 1778654502.78 total 120.09 last 70.05) - open/close 9094 (time 1778654512.79 total 130.11 last 63.72) - open/close 9825 (time 1778654522.81 total 140.12 last 73.00) total: 10000 open/close in 142.38 seconds: 70.23 ops/second - unlinked 0 (time 1778654529 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 649 (time 1778654632.17 total 10.01 last 64.81) - open/close 1315 (time 1778654642.18 total 20.02 last 66.54) - open/close 2017 (time 1778654652.19 total 30.03 last 70.12) - open/close 2752 (time 1778654662.20 total 40.05 last 73.40) - open/close 3497 (time 1778654672.20 total 50.05 last 74.48) - open/close 4185 (time 1778654682.23 total 60.08 last 68.63) - open/close 4841 (time 1778654692.23 total 70.08 last 65.60) - open/close 5523 (time 1778654702.26 total 80.11 last 67.97) - open/close 6279 (time 1778654712.27 total 90.12 last 75.52) - open/close 7004 (time 1778654722.28 total 100.13 last 72.46) - open/close 7709 (time 1778654732.29 total 110.14 last 70.44) - open/close 8400 (time 1778654742.30 total 120.14 last 69.04) - open/close 9107 (time 1778654752.30 total 130.15 last 70.66) - open/close 9790 (time 1778654762.31 total 140.15 last 68.26) total: 10000 open/close in 143.24 seconds: 69.81 ops/second - unlinked 0 (time 1778654769 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.106@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg306-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg306-server: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg306-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg306-server: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg306-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.106@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.106@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.106@tcp sys.timeout=20 start ost1 service on oleg306-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg306-server: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg306-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg306-server: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg306-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg306-server unloading modules via unload_modules_local on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing unload_modules_local oleg306-server: modules unloaded. pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2