== conf-sanity test 69: replace an OST with the same index ========================================================== 01:05:11 (1787115911) start mds service on oleg310-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing load_modules_local oleg310-server: Loading modules from /home/green/git/lustre-release/lustre oleg310-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg310-server: detected 4 online CPUs by sysfs oleg310-server: MODOPTS_LIBCFS= oleg310-server: Force libcfs to create 2 CPU partitions oleg310-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg310-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg310-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg310-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg310-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 979 (time 1787116010.07 total 10.01 last 97.77) - open/close 1924 (time 1787116020.08 total 20.02 last 94.39) - open/close 2804 (time 1787116030.09 total 30.03 last 87.97) - open/close 3690 (time 1787116040.09 total 40.03 last 88.54) - open/close 4645 (time 1787116050.09 total 50.03 last 95.50) - open/close 5598 (time 1787116060.10 total 60.04 last 95.28) - open/close 6524 (time 1787116070.10 total 70.05 last 92.52) - open/close 7457 (time 1787116080.11 total 80.05 last 93.29) - open/close 8354 (time 1787116090.12 total 90.06 last 89.59) - open/close 9280 (time 1787116100.12 total 100.06 last 92.55) total: 10000 open/close in 107.84 seconds: 92.73 ops/second - unlinked 0 (time 1787116111 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 859 (time 1787116187.93 total 10.00 last 85.87) - open/close 1790 (time 1787116197.94 total 20.01 last 93.01) - open/close 2739 (time 1787116207.94 total 30.02 last 94.87) - open/close 3632 (time 1787116217.94 total 40.02 last 89.29) - open/close 4524 (time 1787116227.95 total 50.03 last 89.13) - open/close 5362 (time 1787116237.96 total 60.04 last 83.71) - open/close 6287 (time 1787116247.97 total 70.05 last 92.42) - open/close 7187 (time 1787116257.98 total 80.05 last 89.92) - open/close 8088 (time 1787116267.99 total 90.06 last 90.03) - open/close 8913 (time 1787116277.99 total 100.07 last 82.44) - open/close 9676 (time 1787116288.01 total 110.09 last 76.14) total: 10000 open/close in 113.93 seconds: 87.77 ops/second - unlinked 0 (time 1787116295 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 949 (time 1787116375.21 total 10.00 last 94.90) - open/close 1806 (time 1787116385.21 total 20.00 last 85.68) - open/close 2717 (time 1787116395.22 total 30.01 last 91.02) - open/close 3610 (time 1787116405.23 total 40.03 last 89.17) - open/close 4365 (time 1787116415.24 total 50.03 last 75.49) - open/close 5187 (time 1787116425.25 total 60.04 last 82.06) - open/close 6027 (time 1787116435.26 total 70.05 last 83.94) - open/close 6977 (time 1787116445.27 total 80.06 last 94.93) - open/close 7988 (time 1787116455.27 total 90.06 last 101.08) - open/close 8928 (time 1787116465.28 total 100.07 last 93.90) - open/close 9760 (time 1787116475.28 total 110.07 last 83.18) total: 10000 open/close in 112.45 seconds: 88.93 ops/second - unlinked 0 (time 1787116481 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 944 (time 1787116558.83 total 10.00 last 94.36) - open/close 1878 (time 1787116568.84 total 20.01 last 93.34) - open/close 2864 (time 1787116578.84 total 30.01 last 98.59) - open/close 3723 (time 1787116588.86 total 40.03 last 85.73) - open/close 4532 (time 1787116598.86 total 50.04 last 80.84) - open/close 5436 (time 1787116608.87 total 60.04 last 90.38) - open/close 6340 (time 1787116618.87 total 70.05 last 90.36) - open/close 7284 (time 1787116628.88 total 80.06 last 94.29) - open/close 8134 (time 1787116638.89 total 90.07 last 84.91) - open/close 9059 (time 1787116648.91 total 100.08 last 92.36) - open/close 9959 (time 1787116658.92 total 110.09 last 89.93) total: 10000 open/close in 110.70 seconds: 90.34 ops/second - unlinked 0 (time 1787116663 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 884 (time 1787116741.74 total 10.00 last 88.39) - open/close 1769 (time 1787116751.75 total 20.01 last 88.45) - open/close 2546 (time 1787116761.76 total 30.02 last 77.63) - open/close 3448 (time 1787116771.76 total 40.02 last 90.18) - open/close 4308 (time 1787116781.77 total 50.03 last 85.91) - open/close 5151 (time 1787116791.77 total 60.03 last 84.27) - open/close 6007 (time 1787116801.79 total 70.05 last 85.45) - open/close 6791 (time 1787116811.79 total 80.05 last 78.37) - open/close 7689 (time 1787116821.80 total 90.06 last 89.78) - open/close 8639 (time 1787116831.81 total 100.07 last 94.86) - open/close 9558 (time 1787116841.82 total 110.08 last 91.86) total: 10000 open/close in 114.76 seconds: 87.14 ops/second - unlinked 0 (time 1787116850 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 849 (time 1787116930.84 total 10.00 last 84.88) - open/close 1575 (time 1787116940.86 total 20.02 last 72.45) - open/close 2470 (time 1787116950.86 total 30.03 last 89.45) - open/close 3351 (time 1787116960.88 total 40.05 last 87.93) - open/close 4098 (time 1787116970.89 total 50.05 last 74.66) - open/close 4677 (time 1787116980.90 total 60.06 last 57.84) - open/close 5270 (time 1787116990.90 total 70.07 last 59.29) - open/close 6062 (time 1787117000.92 total 80.08 last 79.05) - open/close 6933 (time 1787117010.92 total 90.09 last 87.06) - open/close 7897 (time 1787117020.93 total 100.10 last 96.33) - open/close 8800 (time 1787117030.95 total 110.12 last 90.12) - open/close 9752 (time 1787117040.97 total 120.13 last 95.06) total: 10000 open/close in 122.56 seconds: 81.60 ops/second - unlinked 0 (time 1787117047 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 851 (time 1787117129.08 total 10.00 last 85.08) - open/close 1703 (time 1787117139.08 total 20.00 last 85.19) - open/close 2570 (time 1787117149.09 total 30.01 last 86.63) - open/close 3397 (time 1787117159.09 total 40.01 last 82.68) - open/close 4184 (time 1787117169.11 total 50.04 last 78.52) - open/close 5002 (time 1787117179.12 total 60.04 last 81.75) - open/close 5855 (time 1787117189.13 total 70.05 last 85.25) - open/close 6738 (time 1787117199.13 total 80.05 last 88.26) - open/close 7715 (time 1787117209.13 total 90.06 last 97.66) - open/close 8603 (time 1787117219.15 total 100.07 last 88.66) - open/close 9517 (time 1787117229.15 total 110.07 last 91.40) total: 10000 open/close in 115.25 seconds: 86.77 ops/second - unlinked 0 (time 1787117238 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 843 (time 1787117320.30 total 10.01 last 84.24) - open/close 1699 (time 1787117330.30 total 20.01 last 85.59) - open/close 2568 (time 1787117340.31 total 30.01 last 86.85) - open/close 3417 (time 1787117350.31 total 40.02 last 84.87) - open/close 4334 (time 1787117360.31 total 50.02 last 91.70) - open/close 5157 (time 1787117370.31 total 60.02 last 82.26) - open/close 6077 (time 1787117380.32 total 70.03 last 91.95) - open/close 6898 (time 1787117390.33 total 80.04 last 82.02) - open/close 7720 (time 1787117400.34 total 90.05 last 82.14) - open/close 8443 (time 1787117410.34 total 100.05 last 72.27) - open/close 9261 (time 1787117420.35 total 110.06 last 81.76) total: 10000 open/close in 119.14 seconds: 83.93 ops/second - unlinked 0 (time 1787117433 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 892 (time 1787117514.01 total 10.01 last 89.12) - open/close 1670 (time 1787117524.02 total 20.02 last 77.70) - open/close 2500 (time 1787117534.03 total 30.03 last 82.96) - open/close 3463 (time 1787117544.03 total 40.03 last 96.30) - open/close 4407 (time 1787117554.06 total 50.06 last 94.08) - open/close 5241 (time 1787117564.06 total 60.06 last 83.39) - open/close 6146 (time 1787117574.09 total 70.09 last 90.26) - open/close 6994 (time 1787117584.11 total 80.11 last 84.62) - open/close 7831 (time 1787117594.12 total 90.11 last 83.66) - open/close 8711 (time 1787117604.14 total 100.13 last 87.84) - open/close 9563 (time 1787117614.14 total 110.13 last 85.19) total: 10000 open/close in 114.41 seconds: 87.41 ops/second - unlinked 0 (time 1787117622 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 850 (time 1787117707.37 total 10.01 last 84.88) - open/close 1642 (time 1787117717.38 total 20.02 last 79.16) - open/close 2440 (time 1787117727.38 total 30.02 last 79.77) - open/close 3275 (time 1787117737.39 total 40.03 last 83.41) - open/close 4203 (time 1787117747.40 total 50.04 last 92.75) - open/close 5060 (time 1787117757.41 total 60.05 last 85.63) - open/close 5907 (time 1787117767.43 total 70.07 last 84.51) - open/close 6806 (time 1787117777.45 total 80.09 last 89.76) - open/close 7677 (time 1787117787.45 total 90.09 last 87.04) - open/close 8613 (time 1787117797.46 total 100.10 last 93.55) - open/close 9521 (time 1787117807.46 total 110.10 last 90.78) total: 10000 open/close in 115.90 seconds: 86.28 ops/second - unlinked 0 (time 1787117817 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.110@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg310-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg310-server: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg310-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg310-server: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg310-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.110@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.110@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.110@tcp sys.timeout=20 start ost1 service on oleg310-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg310-server: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg310-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg310-server: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg310-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg310-server unloading modules via unload_modules_local on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing unload_modules_local oleg310-server: modules unloaded. pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2