== conf-sanity test 69: replace an OST with the same index ========================================================== 07:43:46 (1787831026) start mds service on oleg421-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg421-server: detected 4 online CPUs by sysfs oleg421-server: MODOPTS_LIBCFS= oleg421-server: Force libcfs to create 2 CPU partitions oleg421-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg421-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg421-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg421-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 768 (time 1787831136.92 total 10.00 last 76.76) - open/close 1476 (time 1787831146.92 total 20.01 last 70.80) - open/close 2157 (time 1787831156.92 total 30.01 last 68.09) - open/close 2918 (time 1787831166.93 total 40.02 last 76.02) - open/close 3692 (time 1787831176.94 total 50.02 last 77.35) - open/close 4356 (time 1787831186.94 total 60.02 last 66.39) - open/close 5103 (time 1787831196.95 total 70.03 last 74.67) - open/close 5762 (time 1787831206.95 total 80.03 last 65.90) - open/close 6425 (time 1787831216.97 total 90.05 last 66.17) - open/close 7123 (time 1787831226.97 total 100.05 last 69.79) - open/close 7800 (time 1787831236.97 total 110.06 last 67.67) - open/close 8487 (time 1787831246.97 total 120.06 last 68.68) - open/close 9217 (time 1787831256.99 total 130.07 last 72.90) - open/close 9931 (time 1787831267.00 total 140.08 last 71.32) total: 10000 open/close in 141.02 seconds: 70.91 ops/second - unlinked 0 (time 1787831272 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 639 (time 1787831371.66 total 10.00 last 63.90) - open/close 1257 (time 1787831381.66 total 20.00 last 61.79) - open/close 1935 (time 1787831391.67 total 30.01 last 67.74) - open/close 2642 (time 1787831401.67 total 40.02 last 70.65) - open/close 3342 (time 1787831411.68 total 50.02 last 69.99) - open/close 4068 (time 1787831421.68 total 60.02 last 72.58) - open/close 4811 (time 1787831431.68 total 70.02 last 74.29) - open/close 5532 (time 1787831441.69 total 80.03 last 72.03) - open/close 6234 (time 1787831451.69 total 90.03 last 70.18) - open/close 6889 (time 1787831461.71 total 100.05 last 65.37) - open/close 7661 (time 1787831471.72 total 110.07 last 77.12) - open/close 8370 (time 1787831481.73 total 120.07 last 70.86) - open/close 9036 (time 1787831491.73 total 130.08 last 66.56) - open/close 9779 (time 1787831501.74 total 140.09 last 74.23) total: 10000 open/close in 143.41 seconds: 69.73 ops/second - unlinked 0 (time 1787831509 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 617 (time 1787831610.05 total 10.01 last 61.66) - open/close 1278 (time 1787831620.06 total 20.01 last 66.06) - open/close 1836 (time 1787831630.06 total 30.02 last 55.75) - open/close 2483 (time 1787831640.07 total 40.03 last 64.66) - open/close 3233 (time 1787831650.07 total 50.03 last 74.97) - open/close 3801 (time 1787831660.09 total 60.05 last 56.71) - open/close 4351 (time 1787831670.10 total 70.06 last 54.92) - open/close 4935 (time 1787831680.13 total 80.09 last 58.22) - open/close 5512 (time 1787831690.17 total 90.13 last 57.48) - open/close 6149 (time 1787831700.19 total 100.15 last 63.57) - open/close 6793 (time 1787831710.20 total 110.16 last 64.33) - open/close 7517 (time 1787831720.22 total 120.18 last 72.26) - open/close 8228 (time 1787831730.23 total 130.18 last 71.07) - open/close 8903 (time 1787831740.24 total 140.19 last 67.44) - open/close 9588 (time 1787831750.24 total 150.20 last 68.46) total: 10000 open/close in 155.45 seconds: 64.33 ops/second - unlinked 0 (time 1787831759 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 695 (time 1787831858.32 total 10.00 last 69.47) - open/close 1407 (time 1787831868.33 total 20.01 last 71.15) - open/close 2156 (time 1787831878.34 total 30.02 last 74.84) - open/close 2818 (time 1787831888.34 total 40.02 last 66.17) - open/close 3522 (time 1787831898.35 total 50.03 last 70.33) - open/close 4053 (time 1787831908.36 total 60.04 last 53.05) - open/close 4701 (time 1787831918.36 total 70.04 last 64.79) - open/close 5372 (time 1787831928.37 total 80.05 last 67.03) - open/close 6028 (time 1787831938.37 total 90.05 last 65.59) - open/close 6695 (time 1787831948.38 total 100.06 last 66.66) - open/close 7426 (time 1787831958.38 total 110.06 last 73.10) - open/close 8121 (time 1787831968.39 total 120.07 last 69.41) - open/close 8879 (time 1787831978.40 total 130.08 last 75.77) - open/close 9571 (time 1787831988.41 total 140.09 last 69.13) total: 10000 open/close in 147.04 seconds: 68.01 ops/second - unlinked 0 (time 1787832000 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 700 (time 1787832097.53 total 10.03 last 69.80) - open/close 1304 (time 1787832107.54 total 20.04 last 60.34) - open/close 1892 (time 1787832117.54 total 30.04 last 58.79) - open/close 2552 (time 1787832127.55 total 40.05 last 65.97) - open/close 3179 (time 1787832137.56 total 50.06 last 62.61) - open/close 3856 (time 1787832147.56 total 60.06 last 67.69) - open/close 4517 (time 1787832157.56 total 70.06 last 66.09) - open/close 5028 (time 1787832167.57 total 80.07 last 51.08) - open/close 5534 (time 1787832177.60 total 90.09 last 50.46) - open/close 6174 (time 1787832187.60 total 100.10 last 63.95) - open/close 6735 (time 1787832197.63 total 110.12 last 55.97) - open/close 7333 (time 1787832207.66 total 120.16 last 59.59) - open/close 7875 (time 1787832217.66 total 130.16 last 54.20) - open/close 8450 (time 1787832227.68 total 140.18 last 57.39) - open/close 9026 (time 1787832237.71 total 150.20 last 57.45) - open/close 9697 (time 1787832247.71 total 160.21 last 67.06) total: 10000 open/close in 165.47 seconds: 60.44 ops/second - unlinked 0 (time 1787832258 ; total 1 ; last 1) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 656 (time 1787832361.86 total 10.00 last 65.58) - open/close 1325 (time 1787832371.87 total 20.01 last 66.84) - open/close 2075 (time 1787832381.93 total 30.07 last 74.58) - open/close 2601 (time 1787832391.93 total 40.07 last 52.59) - open/close 3140 (time 1787832401.94 total 50.08 last 53.84) - open/close 3758 (time 1787832411.95 total 60.08 last 61.77) - open/close 4403 (time 1787832421.95 total 70.09 last 64.48) - open/close 5106 (time 1787832431.96 total 80.09 last 70.25) - open/close 5843 (time 1787832441.96 total 90.10 last 73.70) - open/close 6595 (time 1787832451.97 total 100.11 last 75.12) - open/close 7242 (time 1787832461.98 total 110.12 last 64.59) - open/close 7950 (time 1787832472.00 total 120.14 last 70.69) - open/close 8580 (time 1787832482.01 total 130.15 last 62.95) - open/close 9213 (time 1787832492.01 total 140.15 last 63.27) - open/close 9825 (time 1787832502.02 total 150.16 last 61.16) total: 10000 open/close in 152.73 seconds: 65.47 ops/second - unlinked 0 (time 1787832509 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 390 (time 1787832617.11 total 10.01 last 38.95) - open/close 812 (time 1787832627.11 total 20.02 last 42.18) - open/close 1309 (time 1787832637.12 total 30.02 last 49.67) - open/close 1878 (time 1787832647.13 total 40.04 last 56.82) - open/close 2362 (time 1787832657.13 total 50.04 last 48.38) - open/close 3094 (time 1787832667.14 total 60.05 last 73.14) - open/close 3794 (time 1787832677.15 total 70.06 last 69.91) - open/close 4496 (time 1787832687.15 total 80.06 last 70.19) - open/close 5071 (time 1787832697.16 total 90.07 last 57.47) - open/close 5529 (time 1787832707.17 total 100.07 last 45.78) - open/close 6111 (time 1787832717.18 total 110.09 last 58.12) - open/close 6760 (time 1787832727.19 total 120.09 last 64.84) - open/close 7331 (time 1787832737.19 total 130.10 last 57.09) - open/close 8051 (time 1787832747.20 total 140.10 last 71.95) - open/close 8720 (time 1787832757.21 total 150.12 last 66.81) - open/close 9414 (time 1787832767.22 total 160.12 last 69.35) total: 10000 open/close in 169.24 seconds: 59.09 ops/second - unlinked 0 (time 1787832780 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 575 (time 1787832882.25 total 10.00 last 57.49) - open/close 1164 (time 1787832892.27 total 20.02 last 58.81) - open/close 1883 (time 1787832902.27 total 30.02 last 71.85) - open/close 2522 (time 1787832912.28 total 40.03 last 63.85) - open/close 3136 (time 1787832922.28 total 50.04 last 61.37) - open/close 3785 (time 1787832932.29 total 60.04 last 64.88) - open/close 4482 (time 1787832942.29 total 70.05 last 69.66) - open/close 5189 (time 1787832952.30 total 80.05 last 70.65) - open/close 5865 (time 1787832962.31 total 90.06 last 67.53) - open/close 6569 (time 1787832972.31 total 100.06 last 70.38) - open/close 7229 (time 1787832982.31 total 110.07 last 65.99) - open/close 7787 (time 1787832992.32 total 120.07 last 55.79) - open/close 8537 (time 1787833002.33 total 130.08 last 74.91) - open/close 9213 (time 1787833012.34 total 140.09 last 67.50) - open/close 9853 (time 1787833022.35 total 150.10 last 63.97) total: 10000 open/close in 152.59 seconds: 65.53 ops/second - unlinked 0 (time 1787833029 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 774 (time 1787833136.26 total 10.00 last 77.37) - open/close 1552 (time 1787833146.27 total 20.02 last 77.71) - open/close 2269 (time 1787833156.29 total 30.04 last 71.56) - open/close 3033 (time 1787833166.30 total 40.04 last 76.35) - open/close 3737 (time 1787833176.30 total 50.04 last 70.40) - open/close 4478 (time 1787833186.31 total 60.06 last 73.98) - open/close 5177 (time 1787833196.32 total 70.06 last 69.87) - open/close 5860 (time 1787833206.32 total 80.07 last 68.25) - open/close 6556 (time 1787833216.34 total 90.09 last 69.46) - open/close 7298 (time 1787833226.35 total 100.09 last 74.19) - open/close 7999 (time 1787833236.35 total 110.09 last 70.09) - open/close 8666 (time 1787833246.35 total 120.09 last 66.70) - open/close 9356 (time 1787833256.36 total 130.10 last 68.93) total: 10000 open/close in 139.42 seconds: 71.72 ops/second - unlinked 0 (time 1787833270 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 644 (time 1787833366.04 total 10.02 last 64.28) - open/close 1268 (time 1787833376.07 total 20.04 last 62.24) - open/close 1929 (time 1787833386.09 total 30.07 last 65.96) - open/close 2697 (time 1787833396.11 total 40.08 last 76.67) - open/close 3289 (time 1787833406.11 total 50.09 last 59.17) - open/close 3820 (time 1787833416.12 total 60.10 last 53.03) - open/close 4407 (time 1787833426.14 total 70.12 last 58.61) - open/close 4965 (time 1787833436.16 total 80.13 last 55.71) - open/close 5620 (time 1787833446.19 total 90.17 last 65.26) - open/close 6306 (time 1787833456.20 total 100.18 last 68.57) - open/close 6946 (time 1787833466.21 total 110.19 last 63.91) - open/close 7690 (time 1787833476.22 total 120.20 last 74.31) - open/close 8342 (time 1787833486.24 total 130.21 last 65.12) - open/close 8971 (time 1787833496.25 total 140.22 last 62.85) - open/close 9672 (time 1787833506.26 total 150.24 last 70.01) total: 10000 open/close in 154.50 seconds: 64.72 ops/second - unlinked 0 (time 1787833513 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.121@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg421-server: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg421-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg421-server: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg421-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.121@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.121@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.121@tcp sys.timeout=20 start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg421-server: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg421-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg421-server: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg421-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg421-server unloading modules via unload_modules_local on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing unload_modules_local oleg421-server: modules unloaded. pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2