== conf-sanity test 69: replace an OST with the same index ========================================================== 02:28:07 (1787120887) start mds service on oleg138-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions oleg138-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg138-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg138-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg138-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg138-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 783 (time 1787121000.11 total 10.00 last 78.27) - open/close 1583 (time 1787121010.11 total 20.00 last 80.00) - open/close 2244 (time 1787121020.11 total 30.01 last 66.08) - open/close 3021 (time 1787121030.12 total 40.02 last 77.59) - open/close 3845 (time 1787121040.12 total 50.02 last 82.39) - open/close 4571 (time 1787121050.13 total 60.03 last 72.55) - open/close 5332 (time 1787121060.14 total 70.04 last 76.04) - open/close 6076 (time 1787121070.14 total 80.04 last 74.38) - open/close 6768 (time 1787121080.14 total 90.04 last 69.20) - open/close 7599 (time 1787121090.14 total 100.04 last 83.09) - open/close 8518 (time 1787121100.14 total 110.04 last 91.89) - open/close 9364 (time 1787121110.15 total 120.04 last 84.57) total: 10000 open/close in 128.10 seconds: 78.06 ops/second - unlinked 0 (time 1787121122 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 767 (time 1787121216.10 total 10.00 last 76.67) - open/close 1459 (time 1787121226.11 total 20.01 last 69.14) - open/close 2250 (time 1787121236.12 total 30.02 last 79.03) - open/close 2979 (time 1787121246.13 total 40.03 last 72.82) - open/close 3717 (time 1787121256.16 total 50.06 last 73.57) - open/close 4392 (time 1787121266.16 total 60.07 last 67.47) - open/close 5125 (time 1787121276.17 total 70.08 last 73.23) - open/close 6003 (time 1787121286.18 total 80.08 last 87.75) - open/close 6792 (time 1787121296.19 total 90.09 last 78.83) - open/close 7610 (time 1787121306.19 total 100.10 last 81.76) - open/close 8328 (time 1787121316.19 total 110.10 last 71.79) - open/close 9106 (time 1787121326.21 total 120.11 last 77.67) - open/close 9831 (time 1787121336.23 total 130.13 last 72.39) total: 10000 open/close in 133.50 seconds: 74.90 ops/second - unlinked 0 (time 1787121344 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 699 (time 1787121438.11 total 10.02 last 69.78) - open/close 1525 (time 1787121448.12 total 20.02 last 82.55) - open/close 2271 (time 1787121458.12 total 30.02 last 74.60) - open/close 2909 (time 1787121468.13 total 40.03 last 63.75) - open/close 3499 (time 1787121478.13 total 50.03 last 58.98) - open/close 4193 (time 1787121488.14 total 60.04 last 69.34) - open/close 4856 (time 1787121498.14 total 70.04 last 66.30) - open/close 5615 (time 1787121508.15 total 80.05 last 75.83) - open/close 6298 (time 1787121518.16 total 90.06 last 68.25) - open/close 7071 (time 1787121528.20 total 100.10 last 76.97) - open/close 7726 (time 1787121538.21 total 110.12 last 65.41) - open/close 8496 (time 1787121548.22 total 120.12 last 76.94) - open/close 9178 (time 1787121558.23 total 130.13 last 68.14) - open/close 9938 (time 1787121568.26 total 140.17 last 75.74) total: 10000 open/close in 140.99 seconds: 70.93 ops/second - unlinked 0 (time 1787121573 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 656 (time 1787121665.41 total 10.02 last 65.49) - open/close 1425 (time 1787121675.42 total 20.02 last 76.84) - open/close 2231 (time 1787121685.42 total 30.03 last 80.56) - open/close 3123 (time 1787121695.43 total 40.03 last 89.17) - open/close 4033 (time 1787121705.43 total 50.04 last 90.96) - open/close 4693 (time 1787121715.43 total 60.04 last 65.98) - open/close 5522 (time 1787121725.46 total 70.06 last 82.69) - open/close 6384 (time 1787121735.50 total 80.10 last 85.86) - open/close 7102 (time 1787121745.51 total 90.12 last 71.70) - open/close 8024 (time 1787121755.53 total 100.14 last 92.00) - open/close 8652 (time 1787121765.54 total 110.14 last 62.79) - open/close 9260 (time 1787121775.57 total 120.18 last 60.60) - open/close 9954 (time 1787121785.58 total 130.18 last 69.36) total: 10000 open/close in 130.64 seconds: 76.55 ops/second - unlinked 0 (time 1787121791 ; total 1 ; last 1) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 729 (time 1787121885.46 total 10.02 last 72.78) - open/close 1537 (time 1787121895.46 total 20.02 last 80.79) - open/close 2387 (time 1787121905.47 total 30.02 last 84.94) - open/close 3345 (time 1787121915.49 total 40.04 last 95.64) - open/close 4146 (time 1787121925.49 total 50.05 last 80.07) - open/close 4904 (time 1787121935.49 total 60.05 last 75.79) - open/close 5742 (time 1787121945.50 total 70.06 last 83.73) - open/close 6519 (time 1787121955.52 total 80.07 last 77.58) - open/close 7237 (time 1787121965.53 total 90.08 last 71.75) - open/close 8062 (time 1787121975.53 total 100.09 last 82.44) - open/close 8805 (time 1787121985.54 total 110.09 last 74.29) - open/close 9567 (time 1787121995.55 total 120.10 last 76.12) total: 10000 open/close in 126.48 seconds: 79.06 ops/second - unlinked 0 (time 1787122006 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 733 (time 1787122096.54 total 10.00 last 73.27) - open/close 1394 (time 1787122106.54 total 20.00 last 66.10) - open/close 2260 (time 1787122116.55 total 30.01 last 86.55) - open/close 3088 (time 1787122126.56 total 40.02 last 82.69) - open/close 3843 (time 1787122136.57 total 50.03 last 75.47) - open/close 4749 (time 1787122146.58 total 60.04 last 90.46) - open/close 5536 (time 1787122156.59 total 70.05 last 78.65) - open/close 6310 (time 1787122166.60 total 80.06 last 77.32) - open/close 6879 (time 1787122176.61 total 90.07 last 56.85) - open/close 7497 (time 1787122186.62 total 100.08 last 61.74) - open/close 8278 (time 1787122196.62 total 110.08 last 78.08) - open/close 9184 (time 1787122206.63 total 120.10 last 90.47) total: 10000 open/close in 129.88 seconds: 76.99 ops/second - unlinked 0 (time 1787122221 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 711 (time 1787122313.44 total 10.00 last 71.07) - open/close 1533 (time 1787122323.44 total 20.01 last 82.17) - open/close 2482 (time 1787122333.44 total 30.01 last 94.88) - open/close 3268 (time 1787122343.45 total 40.02 last 78.55) - open/close 3988 (time 1787122353.45 total 50.02 last 71.97) - open/close 4613 (time 1787122363.45 total 60.02 last 62.49) - open/close 5341 (time 1787122373.46 total 70.03 last 72.78) - open/close 6010 (time 1787122383.47 total 80.04 last 66.82) - open/close 6680 (time 1787122393.48 total 90.05 last 66.93) - open/close 7266 (time 1787122403.49 total 100.06 last 58.52) - open/close 7937 (time 1787122413.50 total 110.07 last 67.05) - open/close 8600 (time 1787122423.51 total 120.07 last 66.25) - open/close 9259 (time 1787122433.51 total 130.08 last 65.87) - open/close 9975 (time 1787122443.51 total 140.08 last 71.59) total: 10000 open/close in 140.38 seconds: 71.24 ops/second - unlinked 0 (time 1787122448 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 761 (time 1787122549.31 total 10.01 last 76.01) - open/close 1367 (time 1787122559.32 total 20.02 last 60.53) - open/close 2155 (time 1787122569.33 total 30.03 last 78.77) - open/close 2928 (time 1787122579.33 total 40.03 last 77.28) - open/close 3710 (time 1787122589.34 total 50.04 last 78.14) - open/close 4442 (time 1787122599.34 total 60.04 last 73.19) - open/close 5263 (time 1787122609.34 total 70.04 last 82.07) - open/close 5953 (time 1787122619.35 total 80.05 last 68.96) - open/close 6642 (time 1787122629.36 total 90.06 last 68.84) - open/close 7332 (time 1787122639.36 total 100.07 last 68.95) - open/close 8143 (time 1787122649.37 total 110.07 last 81.08) - open/close 8980 (time 1787122659.39 total 120.09 last 83.50) - open/close 9929 (time 1787122669.39 total 130.10 last 94.85) total: 10000 open/close in 130.81 seconds: 76.45 ops/second - unlinked 0 (time 1787122674 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 933 (time 1787122770.65 total 10.01 last 93.21) - open/close 1711 (time 1787122780.65 total 20.01 last 77.79) - open/close 2387 (time 1787122790.69 total 30.04 last 67.39) - open/close 3164 (time 1787122800.69 total 40.05 last 77.68) - open/close 4033 (time 1787122810.69 total 50.05 last 86.90) - open/close 4807 (time 1787122820.70 total 60.05 last 77.33) - open/close 5532 (time 1787122830.70 total 70.06 last 72.46) - open/close 6260 (time 1787122840.71 total 80.06 last 72.77) - open/close 6973 (time 1787122850.72 total 90.07 last 71.23) - open/close 7709 (time 1787122860.73 total 100.08 last 73.52) - open/close 8315 (time 1787122870.75 total 110.10 last 60.48) - open/close 8870 (time 1787122880.75 total 120.11 last 55.47) - open/close 9473 (time 1787122890.76 total 130.11 last 60.27) total: 10000 open/close in 137.38 seconds: 72.79 ops/second - unlinked 0 (time 1787122903 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 660 (time 1787122995.50 total 10.01 last 65.93) - open/close 1355 (time 1787123005.51 total 20.02 last 69.46) - open/close 1984 (time 1787123015.52 total 30.03 last 62.82) - open/close 2600 (time 1787123025.52 total 40.03 last 61.59) - open/close 3463 (time 1787123035.53 total 50.04 last 86.24) - open/close 4219 (time 1787123045.53 total 60.04 last 75.57) - open/close 4761 (time 1787123055.56 total 70.07 last 54.06) - open/close 5467 (time 1787123065.57 total 80.08 last 70.49) - open/close 6014 (time 1787123075.61 total 90.11 last 54.53) - open/close 6527 (time 1787123085.61 total 100.12 last 51.28) - open/close 7260 (time 1787123095.62 total 110.13 last 73.22) - open/close 7921 (time 1787123105.62 total 120.13 last 66.07) - open/close 8530 (time 1787123115.64 total 130.15 last 60.80) - open/close 9214 (time 1787123125.65 total 140.16 last 68.35) - open/close 9915 (time 1787123135.66 total 150.16 last 70.06) total: 10000 open/close in 150.98 seconds: 66.23 ops/second - unlinked 0 (time 1787123142 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg138-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.138@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg138-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg138-server: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg138-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg138-server: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg138-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg138-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.138@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.138@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.138@tcp sys.timeout=20 start ost1 service on oleg138-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg138-server: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg138-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg138-server: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg138-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg138-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server stop mds service on oleg138-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg138-server stop mds service on oleg138-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg138-server unloading modules via unload_modules_local on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing unload_modules_local oleg138-server: modules unloaded. pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-client: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-client: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2