== conf-sanity test 69: replace an OST with the same index ========================================================== 18:27:16 (1787869636) start mds service on oleg111-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions oleg111-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg111-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg111-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg111-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg111-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 1040 (time 1787869738.66 total 10.00 last 103.99) - open/close 2099 (time 1787869748.66 total 20.00 last 105.89) - open/close 3136 (time 1787869758.66 total 30.00 last 103.70) - open/close 4130 (time 1787869768.67 total 40.01 last 99.33) - open/close 5114 (time 1787869778.68 total 50.01 last 98.35) - open/close 6114 (time 1787869788.69 total 60.03 last 99.85) - open/close 7055 (time 1787869798.70 total 70.04 last 93.99) - open/close 7908 (time 1787869808.71 total 80.05 last 85.22) - open/close 8899 (time 1787869818.72 total 90.06 last 99.05) - open/close 9921 (time 1787869828.72 total 100.06 last 102.19) total: 10000 open/close in 100.80 seconds: 99.21 ops/second - unlinked 0 (time 1787869833 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 906 (time 1787869904.04 total 10.01 last 90.52) - open/close 1920 (time 1787869914.05 total 20.02 last 101.33) - open/close 2912 (time 1787869924.05 total 30.02 last 99.18) - open/close 3804 (time 1787869934.06 total 40.03 last 89.13) - open/close 4794 (time 1787869944.06 total 50.03 last 98.99) - open/close 5699 (time 1787869954.09 total 60.05 last 90.26) - open/close 6598 (time 1787869964.09 total 70.06 last 89.89) - open/close 7413 (time 1787869974.09 total 80.06 last 81.47) - open/close 8353 (time 1787869984.12 total 90.09 last 93.74) - open/close 9208 (time 1787869994.12 total 100.09 last 85.47) total: 10000 open/close in 109.51 seconds: 91.31 ops/second - unlinked 0 (time 1787870007 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 663 (time 1787870088.17 total 10.00 last 66.29) - open/close 1423 (time 1787870098.19 total 20.02 last 75.86) - open/close 2339 (time 1787870108.20 total 30.03 last 91.50) - open/close 3046 (time 1787870118.20 total 40.03 last 70.68) - open/close 3920 (time 1787870128.21 total 50.04 last 87.33) - open/close 4878 (time 1787870138.22 total 60.05 last 95.73) - open/close 5787 (time 1787870148.23 total 70.06 last 90.80) - open/close 6696 (time 1787870158.23 total 80.06 last 90.89) - open/close 7613 (time 1787870168.25 total 90.08 last 91.54) - open/close 8514 (time 1787870178.27 total 100.10 last 89.90) - open/close 9198 (time 1787870188.27 total 110.10 last 68.39) total: 10000 open/close in 119.92 seconds: 83.39 ops/second - unlinked 0 (time 1787870203 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 772 (time 1787870283.63 total 10.01 last 77.13) - open/close 1597 (time 1787870293.63 total 20.01 last 82.47) - open/close 2305 (time 1787870303.65 total 30.03 last 70.67) - open/close 3232 (time 1787870313.66 total 40.04 last 92.63) - open/close 4129 (time 1787870323.66 total 50.04 last 89.70) - open/close 5047 (time 1787870333.67 total 60.05 last 91.66) - open/close 5747 (time 1787870343.68 total 70.06 last 69.95) - open/close 6608 (time 1787870353.68 total 80.06 last 86.07) - open/close 7281 (time 1787870363.69 total 90.07 last 67.24) - open/close 8208 (time 1787870373.70 total 100.08 last 92.64) - open/close 9159 (time 1787870383.71 total 110.09 last 95.01) total: 10000 open/close in 119.28 seconds: 83.84 ops/second - unlinked 0 (time 1787870396 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 800 (time 1787870467.03 total 10.00 last 79.97) - open/close 1617 (time 1787870477.04 total 20.01 last 81.65) - open/close 2387 (time 1787870487.05 total 30.02 last 76.93) - open/close 3263 (time 1787870497.05 total 40.02 last 87.59) - open/close 4099 (time 1787870507.05 total 50.03 last 83.55) - open/close 5181 (time 1787870517.06 total 60.03 last 108.18) - open/close 6189 (time 1787870527.06 total 70.03 last 100.76) - open/close 7141 (time 1787870537.06 total 80.03 last 95.18) - open/close 8074 (time 1787870547.07 total 90.04 last 93.22) - open/close 8838 (time 1787870557.08 total 100.05 last 76.34) - open/close 9803 (time 1787870567.08 total 110.05 last 96.49) total: 10000 open/close in 111.95 seconds: 89.32 ops/second - unlinked 0 (time 1787870573 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 1068 (time 1787870652.44 total 10.00 last 106.79) - open/close 1775 (time 1787870662.44 total 20.01 last 70.66) - open/close 2765 (time 1787870672.45 total 30.01 last 98.98) - open/close 3724 (time 1787870682.46 total 40.02 last 95.81) - open/close 4627 (time 1787870692.46 total 50.02 last 90.29) - open/close 5458 (time 1787870702.46 total 60.02 last 83.06) - open/close 6446 (time 1787870712.47 total 70.03 last 98.73) - open/close 7373 (time 1787870722.48 total 80.05 last 92.56) - open/close 8234 (time 1787870732.49 total 90.06 last 86.00) - open/close 9275 (time 1787870742.50 total 100.06 last 104.08) total: 10000 open/close in 108.10 seconds: 92.51 ops/second - unlinked 0 (time 1787870754 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 901 (time 1787870830.91 total 10.03 last 89.84) - open/close 1867 (time 1787870840.92 total 20.03 last 96.56) - open/close 2819 (time 1787870850.92 total 30.04 last 95.16) - open/close 3888 (time 1787870860.93 total 40.04 last 106.82) - open/close 4902 (time 1787870870.94 total 50.05 last 101.35) - open/close 5827 (time 1787870880.94 total 60.05 last 92.46) - open/close 6784 (time 1787870890.95 total 70.06 last 95.63) - open/close 7707 (time 1787870900.95 total 80.06 last 92.28) - open/close 8371 (time 1787870910.95 total 90.07 last 66.38) - open/close 9069 (time 1787870920.96 total 100.07 last 69.76) total: 10000 open/close in 108.20 seconds: 92.42 ops/second - unlinked 0 (time 1787870934 ; total 1 ; last 1) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 873 (time 1787871007.91 total 10.01 last 87.24) - open/close 1700 (time 1787871017.92 total 20.01 last 82.64) - open/close 2692 (time 1787871027.92 total 30.02 last 99.16) - open/close 3611 (time 1787871037.93 total 40.03 last 91.84) - open/close 4597 (time 1787871047.93 total 50.03 last 98.55) - open/close 5623 (time 1787871057.94 total 60.04 last 102.50) - open/close 6535 (time 1787871067.95 total 70.05 last 91.15) - open/close 7441 (time 1787871077.95 total 80.05 last 90.56) - open/close 8480 (time 1787871087.96 total 90.06 last 103.83) - open/close 9461 (time 1787871097.97 total 100.06 last 98.03) total: 10000 open/close in 106.06 seconds: 94.29 ops/second - unlinked 0 (time 1787871108 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 721 (time 1787871182.92 total 10.00 last 72.09) - open/close 1667 (time 1787871192.94 total 20.02 last 94.38) - open/close 2739 (time 1787871202.94 total 30.03 last 107.15) - open/close 3719 (time 1787871212.95 total 40.03 last 97.99) - open/close 4666 (time 1787871222.95 total 50.04 last 94.65) - open/close 5618 (time 1787871232.96 total 60.05 last 95.09) - open/close 6612 (time 1787871242.97 total 70.05 last 99.35) - open/close 7649 (time 1787871252.97 total 80.06 last 103.67) - open/close 8802 (time 1787871262.98 total 90.06 last 115.25) - open/close 9854 (time 1787871272.98 total 100.07 last 105.13) total: 10000 open/close in 101.66 seconds: 98.37 ops/second - unlinked 0 (time 1787871278 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 994 (time 1787871352.52 total 10.01 last 99.35) - open/close 1850 (time 1787871362.52 total 20.01 last 85.57) - open/close 2747 (time 1787871372.53 total 30.02 last 89.61) - open/close 3731 (time 1787871382.54 total 40.03 last 98.30) - open/close 4641 (time 1787871392.55 total 50.04 last 90.88) - open/close 5508 (time 1787871402.56 total 60.05 last 86.67) - open/close 6346 (time 1787871412.56 total 70.05 last 83.78) - open/close 7392 (time 1787871422.56 total 80.05 last 104.54) - open/close 8349 (time 1787871432.57 total 90.06 last 95.67) - open/close 9354 (time 1787871442.57 total 100.06 last 100.44) total: 10000 open/close in 107.91 seconds: 92.67 ops/second - unlinked 0 (time 1787871454 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg111-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg111-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg111-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.111@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg111-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg111-server: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg111-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg111-server: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg111-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg111-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg111-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg111-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.111@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.111@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.111@tcp sys.timeout=20 start ost1 service on oleg111-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg111-server: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg111-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg111-server: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg111-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg111-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg111-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg111-server stop mds service on oleg111-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg111-server stop mds service on oleg111-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg111-server unloading modules via unload_modules_local on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing unload_modules_local oleg111-server: modules unloaded. pdsh@oleg111-client: oleg111-client: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-client: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2