== conf-sanity test 69: replace an OST with the same index ========================================================== 11:57:30 (1784044650) start mds service on oleg245-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg245-server: detected 4 online CPUs by sysfs oleg245-server: MODOPTS_LIBCFS= oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg245-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg245-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg245-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 718 (time 1784044778.36 total 10.01 last 71.72) - open/close 1288 (time 1784044788.37 total 20.02 last 56.93) - open/close 2010 (time 1784044798.39 total 30.04 last 72.05) - open/close 2819 (time 1784044808.40 total 40.06 last 80.79) - open/close 3449 (time 1784044818.41 total 50.06 last 62.99) - open/close 4216 (time 1784044828.41 total 60.07 last 76.64) - open/close 4965 (time 1784044838.43 total 70.09 last 74.76) - open/close 5651 (time 1784044848.44 total 80.10 last 68.53) - open/close 6490 (time 1784044858.45 total 90.10 last 83.88) - open/close 7245 (time 1784044868.45 total 100.10 last 75.47) - open/close 8004 (time 1784044878.45 total 110.10 last 75.89) - open/close 8811 (time 1784044888.45 total 120.11 last 80.69) - open/close 9623 (time 1784044898.45 total 130.11 last 81.19) total: 10000 open/close in 134.31 seconds: 74.46 ops/second - unlinked 0 (time 1784044907 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 869 (time 1784044999.14 total 10.00 last 86.90) - open/close 1603 (time 1784045009.14 total 20.00 last 73.40) - open/close 2368 (time 1784045019.15 total 30.01 last 76.39) - open/close 3143 (time 1784045029.16 total 40.02 last 77.44) - open/close 4003 (time 1784045039.17 total 50.03 last 85.96) - open/close 4776 (time 1784045049.18 total 60.04 last 77.22) - open/close 5545 (time 1784045059.18 total 70.04 last 76.89) - open/close 6262 (time 1784045069.18 total 80.04 last 71.69) - open/close 6961 (time 1784045079.20 total 90.06 last 69.75) - open/close 7455 (time 1784045089.23 total 100.09 last 49.28) - open/close 8057 (time 1784045099.23 total 110.09 last 60.20) - open/close 8687 (time 1784045109.24 total 120.10 last 62.94) - open/close 9267 (time 1784045119.25 total 130.11 last 57.94) - open/close 9854 (time 1784045129.26 total 140.12 last 58.60) total: 10000 open/close in 142.32 seconds: 70.26 ops/second - unlinked 0 (time 1784045136 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 532 (time 1784045232.41 total 10.00 last 53.19) - open/close 920 (time 1784045242.43 total 20.03 last 38.70) - open/close 1657 (time 1784045252.45 total 30.04 last 73.58) - open/close 2539 (time 1784045262.46 total 40.05 last 88.13) - open/close 3210 (time 1784045272.48 total 50.08 last 66.94) - open/close 3917 (time 1784045282.48 total 60.08 last 70.67) - open/close 4531 (time 1784045292.49 total 70.09 last 61.34) - open/close 5075 (time 1784045302.52 total 80.11 last 54.28) - open/close 5930 (time 1784045312.53 total 90.12 last 85.41) - open/close 6768 (time 1784045322.53 total 100.12 last 83.80) - open/close 7619 (time 1784045332.53 total 110.13 last 85.05) - open/close 8222 (time 1784045342.54 total 120.13 last 60.28) - open/close 8943 (time 1784045352.56 total 130.15 last 71.92) - open/close 9766 (time 1784045362.57 total 140.16 last 82.24) total: 10000 open/close in 142.78 seconds: 70.04 ops/second - unlinked 0 (time 1784045369 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 850 (time 1784045457.59 total 10.01 last 84.94) - open/close 1628 (time 1784045467.60 total 20.02 last 77.73) - open/close 2330 (time 1784045477.61 total 30.02 last 70.14) - open/close 3135 (time 1784045487.61 total 40.03 last 80.50) - open/close 3965 (time 1784045497.62 total 50.03 last 82.93) - open/close 4678 (time 1784045507.63 total 60.04 last 71.24) - open/close 5447 (time 1784045517.63 total 70.04 last 76.89) - open/close 6069 (time 1784045527.64 total 80.05 last 62.14) - open/close 6817 (time 1784045537.64 total 90.06 last 74.78) - open/close 7654 (time 1784045547.65 total 100.06 last 83.63) - open/close 8417 (time 1784045557.66 total 110.07 last 76.23) - open/close 9122 (time 1784045567.67 total 120.08 last 70.45) - open/close 9903 (time 1784045577.68 total 130.09 last 78.03) total: 10000 open/close in 131.21 seconds: 76.21 ops/second - unlinked 0 (time 1784045583 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 829 (time 1784045667.50 total 10.00 last 82.90) - open/close 1644 (time 1784045677.50 total 20.00 last 81.49) - open/close 2381 (time 1784045687.51 total 30.01 last 73.65) - open/close 3034 (time 1784045697.52 total 40.02 last 65.22) - open/close 4003 (time 1784045707.53 total 50.03 last 96.82) - open/close 4868 (time 1784045717.54 total 60.04 last 86.44) - open/close 5651 (time 1784045727.54 total 70.04 last 78.29) - open/close 6472 (time 1784045737.55 total 80.05 last 81.99) - open/close 7134 (time 1784045747.56 total 90.06 last 66.15) - open/close 7822 (time 1784045757.56 total 100.06 last 68.79) - open/close 8493 (time 1784045767.56 total 110.06 last 67.08) - open/close 9139 (time 1784045777.57 total 120.07 last 64.54) - open/close 9777 (time 1784045787.58 total 130.08 last 63.74) total: 10000 open/close in 134.09 seconds: 74.58 ops/second - unlinked 0 (time 1784045797 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 880 (time 1784045887.35 total 10.01 last 87.93) - open/close 1703 (time 1784045897.40 total 20.05 last 81.94) - open/close 2329 (time 1784045907.41 total 30.06 last 62.52) - open/close 2960 (time 1784045917.41 total 40.07 last 63.08) - open/close 3731 (time 1784045927.42 total 50.07 last 77.05) - open/close 4502 (time 1784045937.42 total 60.08 last 77.08) - open/close 5231 (time 1784045947.42 total 70.08 last 72.88) - open/close 5963 (time 1784045957.44 total 80.09 last 73.09) - open/close 6624 (time 1784045967.46 total 90.11 last 65.96) - open/close 7406 (time 1784045977.46 total 100.12 last 78.15) - open/close 8215 (time 1784045987.47 total 110.12 last 80.87) - open/close 8912 (time 1784045997.48 total 120.14 last 69.62) - open/close 9669 (time 1784046007.50 total 130.16 last 75.52) total: 10000 open/close in 134.82 seconds: 74.17 ops/second - unlinked 0 (time 1784046018 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 828 (time 1784046107.88 total 10.01 last 82.73) - open/close 1364 (time 1784046117.89 total 20.02 last 53.54) - open/close 2013 (time 1784046127.90 total 30.03 last 64.87) - open/close 2684 (time 1784046137.90 total 40.03 last 67.08) - open/close 3353 (time 1784046147.90 total 50.03 last 66.88) - open/close 4124 (time 1784046157.91 total 60.04 last 77.07) - open/close 5014 (time 1784046167.92 total 70.04 last 88.91) - open/close 5854 (time 1784046177.94 total 80.06 last 83.84) - open/close 6575 (time 1784046187.94 total 90.06 last 72.10) - open/close 7357 (time 1784046197.95 total 100.07 last 78.13) - open/close 8156 (time 1784046207.95 total 110.08 last 79.86) - open/close 8947 (time 1784046217.96 total 120.09 last 79.03) - open/close 9737 (time 1784046227.97 total 130.09 last 78.95) total: 10000 open/close in 133.40 seconds: 74.96 ops/second - unlinked 0 (time 1784046235 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 610 (time 1784046328.68 total 10.00 last 60.99) - open/close 1237 (time 1784046338.73 total 20.05 last 62.40) - open/close 2112 (time 1784046348.74 total 30.07 last 87.35) - open/close 2947 (time 1784046358.76 total 40.08 last 83.39) - open/close 3655 (time 1784046368.76 total 50.09 last 70.75) - open/close 4388 (time 1784046378.77 total 60.09 last 73.25) - open/close 5022 (time 1784046388.77 total 70.10 last 63.39) - open/close 5629 (time 1784046398.80 total 80.12 last 60.56) - open/close 6265 (time 1784046408.81 total 90.13 last 63.50) - open/close 6949 (time 1784046418.81 total 100.14 last 68.39) - open/close 7759 (time 1784046428.86 total 110.18 last 80.65) - open/close 8551 (time 1784046438.86 total 120.19 last 79.14) - open/close 9229 (time 1784046448.87 total 130.20 last 67.73) total: 10000 open/close in 140.13 seconds: 71.36 ops/second - unlinked 0 (time 1784046463 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 628 (time 1784046553.70 total 10.01 last 62.74) - open/close 1256 (time 1784046563.72 total 20.03 last 62.70) - open/close 1890 (time 1784046573.73 total 30.04 last 63.30) - open/close 2711 (time 1784046583.75 total 40.06 last 81.95) - open/close 3518 (time 1784046593.75 total 50.06 last 80.68) - open/close 4331 (time 1784046603.77 total 60.07 last 81.21) - open/close 5192 (time 1784046613.77 total 70.08 last 86.03) - open/close 5939 (time 1784046623.78 total 80.09 last 74.66) - open/close 6781 (time 1784046633.78 total 90.09 last 84.19) - open/close 7578 (time 1784046643.79 total 100.10 last 79.63) - open/close 8370 (time 1784046653.80 total 110.11 last 79.14) - open/close 9148 (time 1784046663.80 total 120.11 last 77.77) - open/close 9868 (time 1784046673.81 total 130.12 last 71.96) total: 10000 open/close in 132.27 seconds: 75.60 ops/second - unlinked 0 (time 1784046680 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 841 (time 1784046770.03 total 10.01 last 84.02) - open/close 1610 (time 1784046780.03 total 20.01 last 76.90) - open/close 2393 (time 1784046790.03 total 30.01 last 78.29) - open/close 3372 (time 1784046800.03 total 40.01 last 97.89) - open/close 4184 (time 1784046810.04 total 50.02 last 81.14) - open/close 5066 (time 1784046820.04 total 60.02 last 88.20) - open/close 5959 (time 1784046830.05 total 70.03 last 89.19) - open/close 6798 (time 1784046840.06 total 80.04 last 83.84) - open/close 7521 (time 1784046850.06 total 90.04 last 72.28) - open/close 8322 (time 1784046860.07 total 100.05 last 80.08) - open/close 9222 (time 1784046870.07 total 110.05 last 89.98) - open/close 9975 (time 1784046880.09 total 120.07 last 75.11) total: 10000 open/close in 120.47 seconds: 83.01 ops/second - unlinked 0 (time 1784046885 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.145@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg245-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.145@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.145@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.145@tcp sys.timeout=20 start ost1 service on oleg245-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg245-server: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg245-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local oleg245-server: modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2