== conf-sanity test 69: replace an OST with the same index ========================================================== 09:34:26 (1786800866) start mds service on oleg106-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing load_modules_local oleg106-server: Loading modules from /home/green/git/lustre-release/lustre oleg106-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg106-server: detected 4 online CPUs by sysfs oleg106-server: MODOPTS_LIBCFS= oleg106-server: Force libcfs to create 2 CPU partitions oleg106-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg106-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg106-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg106-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg106-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 592 (time 1786801000.01 total 10.00 last 59.17) - open/close 1149 (time 1786801010.03 total 20.02 last 55.59) - open/close 1900 (time 1786801020.03 total 30.03 last 75.05) - open/close 2658 (time 1786801030.03 total 40.03 last 75.80) - open/close 3368 (time 1786801040.04 total 50.03 last 70.98) - open/close 4035 (time 1786801050.04 total 60.04 last 66.68) - open/close 4817 (time 1786801060.05 total 70.05 last 78.12) - open/close 5625 (time 1786801070.05 total 80.05 last 80.78) - open/close 6372 (time 1786801080.06 total 90.05 last 74.68) - open/close 7091 (time 1786801090.07 total 100.07 last 71.78) - open/close 7896 (time 1786801100.07 total 110.07 last 80.50) - open/close 8717 (time 1786801110.08 total 120.08 last 82.06) - open/close 9410 (time 1786801120.09 total 130.08 last 69.24) total: 10000 open/close in 138.62 seconds: 72.14 ops/second - unlinked 0 (time 1786801133 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 766 (time 1786801222.90 total 10.02 last 76.45) - open/close 1558 (time 1786801232.91 total 20.03 last 79.10) - open/close 2304 (time 1786801242.91 total 30.03 last 74.59) - open/close 2898 (time 1786801252.93 total 40.05 last 59.30) - open/close 3554 (time 1786801262.94 total 50.07 last 65.50) - open/close 4100 (time 1786801272.96 total 60.08 last 54.51) - open/close 4837 (time 1786801282.97 total 70.10 last 73.60) - open/close 5344 (time 1786801292.98 total 80.10 last 50.67) - open/close 5875 (time 1786801302.99 total 90.11 last 53.06) - open/close 6476 (time 1786801312.99 total 100.12 last 60.06) - open/close 7245 (time 1786801323.01 total 110.13 last 76.81) - open/close 7904 (time 1786801333.02 total 120.14 last 65.82) - open/close 8565 (time 1786801343.03 total 130.15 last 66.01) - open/close 9294 (time 1786801353.05 total 140.18 last 72.75) total: 10000 open/close in 149.49 seconds: 66.89 ops/second - unlinked 0 (time 1786801367 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 812 (time 1786801458.95 total 10.01 last 81.12) - open/close 1552 (time 1786801468.96 total 20.02 last 73.90) - open/close 2372 (time 1786801478.97 total 30.03 last 81.97) - open/close 3157 (time 1786801488.98 total 40.04 last 78.37) - open/close 3979 (time 1786801498.98 total 50.04 last 82.19) - open/close 4740 (time 1786801509.00 total 60.06 last 75.97) - open/close 5352 (time 1786801519.01 total 70.07 last 61.15) - open/close 6066 (time 1786801529.02 total 80.07 last 71.37) - open/close 6930 (time 1786801539.02 total 90.08 last 86.36) - open/close 7692 (time 1786801549.02 total 100.08 last 76.16) - open/close 8380 (time 1786801559.03 total 110.09 last 68.78) - open/close 9042 (time 1786801569.03 total 120.09 last 66.20) - open/close 9737 (time 1786801579.03 total 130.09 last 69.47) total: 10000 open/close in 133.64 seconds: 74.83 ops/second - unlinked 0 (time 1786801587 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 709 (time 1786801693.44 total 10.00 last 70.87) - open/close 1386 (time 1786801703.44 total 20.00 last 67.70) - open/close 2142 (time 1786801713.44 total 30.01 last 75.59) - open/close 2943 (time 1786801723.44 total 40.01 last 80.08) - open/close 3663 (time 1786801733.45 total 50.02 last 71.93) - open/close 4125 (time 1786801743.46 total 60.02 last 46.17) - open/close 4747 (time 1786801753.47 total 70.03 last 62.16) - open/close 5447 (time 1786801763.47 total 80.03 last 69.98) - open/close 6153 (time 1786801773.48 total 90.04 last 70.54) - open/close 6746 (time 1786801783.49 total 100.05 last 59.23) - open/close 7383 (time 1786801793.52 total 110.09 last 63.50) - open/close 8079 (time 1786801803.52 total 120.09 last 69.60) - open/close 8748 (time 1786801813.54 total 130.11 last 66.74) - open/close 9410 (time 1786801823.55 total 140.11 last 66.17) - open/close 9995 (time 1786801833.56 total 150.12 last 58.44) total: 10000 open/close in 150.17 seconds: 66.59 ops/second - unlinked 0 (time 1786801839 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 559 (time 1786801945.26 total 10.00 last 55.89) - open/close 1033 (time 1786801955.26 total 20.00 last 47.38) - open/close 1604 (time 1786801965.27 total 30.02 last 57.03) - open/close 2213 (time 1786801975.28 total 40.02 last 60.87) - open/close 2892 (time 1786801985.28 total 50.03 last 67.86) - open/close 3598 (time 1786801995.29 total 60.04 last 70.52) - open/close 4368 (time 1786802005.31 total 70.05 last 76.90) - open/close 5119 (time 1786802015.32 total 80.07 last 75.00) - open/close 5836 (time 1786802025.35 total 90.10 last 71.48) - open/close 6720 (time 1786802035.38 total 100.12 last 88.17) - open/close 7383 (time 1786802045.38 total 110.12 last 66.29) - open/close 7912 (time 1786802055.40 total 120.14 last 52.80) - open/close 8505 (time 1786802065.41 total 130.16 last 59.22) - open/close 9017 (time 1786802075.41 total 140.16 last 51.20) - open/close 9617 (time 1786802085.42 total 150.16 last 59.95) total: 10000 open/close in 155.12 seconds: 64.47 ops/second - unlinked 0 (time 1786802095 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 548 (time 1786802200.26 total 10.05 last 54.51) - open/close 1243 (time 1786802210.27 total 20.06 last 69.42) - open/close 1851 (time 1786802220.30 total 30.09 last 60.67) - open/close 2471 (time 1786802230.30 total 40.09 last 61.96) - open/close 3250 (time 1786802240.31 total 50.10 last 77.82) - open/close 3972 (time 1786802250.32 total 60.11 last 72.13) - open/close 4748 (time 1786802260.33 total 70.12 last 77.52) - open/close 5335 (time 1786802270.34 total 80.13 last 58.65) - open/close 5990 (time 1786802280.34 total 90.13 last 65.50) - open/close 6785 (time 1786802290.35 total 100.14 last 79.45) - open/close 7566 (time 1786802300.35 total 110.14 last 78.04) - open/close 8340 (time 1786802310.36 total 120.15 last 77.38) - open/close 9014 (time 1786802320.36 total 130.15 last 67.40) - open/close 9619 (time 1786802330.36 total 140.15 last 60.48) total: 10000 open/close in 146.94 seconds: 68.05 ops/second - unlinked 0 (time 1786802342 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 521 (time 1786802448.24 total 10.00 last 52.08) - open/close 1130 (time 1786802458.25 total 20.01 last 60.84) - open/close 1689 (time 1786802468.27 total 30.03 last 55.78) - open/close 2296 (time 1786802478.28 total 40.04 last 60.64) - open/close 2953 (time 1786802488.30 total 50.06 last 65.57) - open/close 3673 (time 1786802498.31 total 60.07 last 71.95) - open/close 4346 (time 1786802508.31 total 70.08 last 67.26) - open/close 5134 (time 1786802518.33 total 80.09 last 78.71) - open/close 5702 (time 1786802528.33 total 90.10 last 56.77) - open/close 6515 (time 1786802538.34 total 100.10 last 81.27) - open/close 7325 (time 1786802548.34 total 110.10 last 80.98) - open/close 8068 (time 1786802558.34 total 120.10 last 74.30) - open/close 8781 (time 1786802568.34 total 130.11 last 71.28) - open/close 9503 (time 1786802578.35 total 140.11 last 72.17) total: 10000 open/close in 146.18 seconds: 68.41 ops/second - unlinked 0 (time 1786802590 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 603 (time 1786802690.71 total 10.02 last 60.16) - open/close 1183 (time 1786802700.72 total 20.03 last 57.94) - open/close 1798 (time 1786802710.74 total 30.05 last 61.38) - open/close 2369 (time 1786802720.74 total 40.06 last 57.07) - open/close 2926 (time 1786802730.76 total 50.07 last 55.63) - open/close 3617 (time 1786802740.78 total 60.09 last 68.96) - open/close 4082 (time 1786802750.78 total 70.09 last 46.49) - open/close 4760 (time 1786802760.78 total 80.10 last 67.77) - open/close 5502 (time 1786802770.81 total 90.13 last 73.97) - open/close 6280 (time 1786802780.82 total 100.13 last 77.76) - open/close 6942 (time 1786802790.83 total 110.14 last 66.13) - open/close 7749 (time 1786802800.83 total 120.15 last 80.68) - open/close 8437 (time 1786802810.84 total 130.15 last 68.78) - open/close 9233 (time 1786802820.85 total 140.16 last 79.51) - open/close 9897 (time 1786802830.85 total 150.16 last 66.39) total: 10000 open/close in 151.86 seconds: 65.85 ops/second - unlinked 0 (time 1786802837 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 826 (time 1786802933.54 total 10.02 last 82.42) - open/close 1611 (time 1786802943.55 total 20.03 last 78.44) - open/close 2377 (time 1786802953.55 total 30.04 last 76.55) - open/close 3071 (time 1786802963.56 total 40.04 last 69.33) - open/close 4015 (time 1786802973.57 total 50.05 last 94.32) - open/close 4765 (time 1786802983.59 total 60.07 last 74.86) - open/close 5565 (time 1786802993.60 total 70.08 last 79.94) - open/close 6313 (time 1786803003.60 total 80.08 last 74.77) - open/close 6981 (time 1786803013.62 total 90.10 last 66.67) - open/close 7825 (time 1786803023.62 total 100.10 last 84.39) - open/close 8633 (time 1786803033.63 total 110.11 last 80.75) - open/close 9389 (time 1786803043.63 total 120.11 last 75.59) total: 10000 open/close in 128.73 seconds: 77.68 ops/second - unlinked 0 (time 1786803057 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 676 (time 1786803150.70 total 10.02 last 67.45) - open/close 1453 (time 1786803160.71 total 20.03 last 77.66) - open/close 2087 (time 1786803170.74 total 30.06 last 63.18) - open/close 2680 (time 1786803180.75 total 40.07 last 59.26) - open/close 3596 (time 1786803190.75 total 50.08 last 91.54) - open/close 4452 (time 1786803200.77 total 60.09 last 85.50) - open/close 5236 (time 1786803210.78 total 70.11 last 78.27) - open/close 6013 (time 1786803220.78 total 80.11 last 77.69) - open/close 6902 (time 1786803230.80 total 90.12 last 88.78) - open/close 7738 (time 1786803240.80 total 100.12 last 83.57) - open/close 8512 (time 1786803250.81 total 110.13 last 77.34) - open/close 9382 (time 1786803260.82 total 120.14 last 86.94) total: 10000 open/close in 127.46 seconds: 78.45 ops/second - unlinked 0 (time 1786803272 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg106-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg106-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg106-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.106@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg106-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg106-server: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg106-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg106-server: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg106-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg106-client: oleg106-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg106-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg106-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg106-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.106@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.106@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.106@tcp sys.timeout=20 start ost1 service on oleg106-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg106-server: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg106-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg106-server: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg106-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg106-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg106-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg106-server stop mds service on oleg106-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg106-server stop mds service on oleg106-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg106-server unloading modules via unload_modules_local on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing unload_modules_local oleg106-server: modules unloaded. pdsh@oleg106-client: oleg106-server: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-client: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-server: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-client: ssh exited with exit code 2