== conf-sanity test 69: replace an OST with the same index ========================================================== 03:42:30 (1784187750) start mds service on oleg423-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions oleg423-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg423-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg423-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg423-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 816 (time 1784187867.57 total 10.00 last 81.56) - open/close 1591 (time 1784187877.58 total 20.01 last 77.44) - open/close 2360 (time 1784187887.59 total 30.02 last 76.83) - open/close 3146 (time 1784187897.59 total 40.02 last 78.59) - open/close 3958 (time 1784187907.60 total 50.03 last 81.13) - open/close 4636 (time 1784187917.61 total 60.04 last 67.73) - open/close 5248 (time 1784187927.61 total 70.05 last 61.17) - open/close 6001 (time 1784187937.62 total 80.06 last 75.23) - open/close 6742 (time 1784187947.62 total 90.06 last 74.09) - open/close 7528 (time 1784187957.63 total 100.06 last 78.59) - open/close 8282 (time 1784187967.63 total 110.06 last 75.38) - open/close 9010 (time 1784187977.64 total 120.07 last 72.71) - open/close 9751 (time 1784187987.64 total 130.08 last 74.07) total: 10000 open/close in 133.27 seconds: 75.04 ops/second - unlinked 0 (time 1784187995 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 826 (time 1784188085.77 total 10.00 last 82.59) - open/close 1713 (time 1784188095.78 total 20.01 last 88.62) - open/close 2550 (time 1784188105.79 total 30.02 last 83.61) - open/close 3398 (time 1784188115.79 total 40.02 last 84.78) - open/close 4223 (time 1784188125.79 total 50.02 last 82.50) - open/close 4933 (time 1784188135.80 total 60.03 last 70.94) - open/close 5663 (time 1784188145.80 total 70.03 last 73.00) - open/close 6394 (time 1784188155.80 total 80.04 last 73.08) - open/close 7114 (time 1784188165.83 total 90.06 last 71.82) - open/close 7759 (time 1784188175.83 total 100.06 last 64.48) - open/close 8640 (time 1784188185.83 total 110.07 last 88.06) - open/close 9362 (time 1784188195.84 total 120.07 last 72.19) total: 10000 open/close in 128.02 seconds: 78.11 ops/second - unlinked 0 (time 1784188208 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 874 (time 1784188296.15 total 10.01 last 87.35) - open/close 1746 (time 1784188306.15 total 20.01 last 87.19) - open/close 2532 (time 1784188316.16 total 30.02 last 78.50) - open/close 3278 (time 1784188326.18 total 40.03 last 74.49) - open/close 4051 (time 1784188336.18 total 50.04 last 77.25) - open/close 4862 (time 1784188346.19 total 60.05 last 81.03) - open/close 5669 (time 1784188356.20 total 70.06 last 80.61) - open/close 6458 (time 1784188366.21 total 80.07 last 78.83) - open/close 7279 (time 1784188376.22 total 90.08 last 82.02) - open/close 8132 (time 1784188386.22 total 100.08 last 85.28) - open/close 8960 (time 1784188396.23 total 110.09 last 82.74) - open/close 9763 (time 1784188406.23 total 120.09 last 80.29) total: 10000 open/close in 122.65 seconds: 81.53 ops/second - unlinked 0 (time 1784188413 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 770 (time 1784188509.12 total 10.01 last 76.95) - open/close 1472 (time 1784188519.12 total 20.01 last 70.17) - open/close 2146 (time 1784188529.13 total 30.02 last 67.37) - open/close 2749 (time 1784188539.13 total 40.02 last 60.28) - open/close 3286 (time 1784188549.13 total 50.02 last 53.70) - open/close 4025 (time 1784188559.14 total 60.03 last 73.85) - open/close 4848 (time 1784188569.14 total 70.03 last 82.25) - open/close 5581 (time 1784188579.15 total 80.04 last 73.23) - open/close 6301 (time 1784188589.16 total 90.05 last 71.96) - open/close 7073 (time 1784188599.17 total 100.05 last 77.13) - open/close 7662 (time 1784188609.18 total 110.07 last 58.82) - open/close 8482 (time 1784188619.19 total 120.08 last 81.91) - open/close 9166 (time 1784188629.21 total 130.10 last 68.26) - open/close 9889 (time 1784188639.22 total 140.11 last 72.24) total: 10000 open/close in 141.88 seconds: 70.48 ops/second - unlinked 0 (time 1784188645 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 821 (time 1784188744.38 total 10.01 last 82.05) - open/close 1595 (time 1784188754.39 total 20.01 last 77.33) - open/close 2376 (time 1784188764.39 total 30.02 last 78.07) - open/close 3109 (time 1784188774.42 total 40.04 last 73.12) - open/close 3947 (time 1784188784.42 total 50.05 last 83.75) - open/close 4680 (time 1784188794.43 total 60.06 last 73.26) - open/close 5417 (time 1784188804.43 total 70.06 last 73.68) - open/close 6194 (time 1784188814.43 total 80.06 last 77.68) - open/close 6989 (time 1784188824.44 total 90.07 last 79.44) - open/close 7847 (time 1784188834.44 total 100.07 last 85.79) - open/close 8542 (time 1784188844.45 total 110.08 last 69.46) - open/close 9344 (time 1784188854.45 total 120.08 last 80.15) total: 10000 open/close in 127.91 seconds: 78.18 ops/second - unlinked 0 (time 1784188867 ; total 1 ; last 1) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 780 (time 1784188955.08 total 10.01 last 77.94) - open/close 1562 (time 1784188965.09 total 20.02 last 78.14) - open/close 2235 (time 1784188975.10 total 30.02 last 67.26) - open/close 2961 (time 1784188985.10 total 40.03 last 72.56) - open/close 3783 (time 1784188995.11 total 50.04 last 82.11) - open/close 4633 (time 1784189005.11 total 60.04 last 85.00) - open/close 5392 (time 1784189015.12 total 70.04 last 75.85) - open/close 6243 (time 1784189025.12 total 80.05 last 85.07) - open/close 7057 (time 1784189035.13 total 90.05 last 81.36) - open/close 7744 (time 1784189045.14 total 100.06 last 68.63) - open/close 8551 (time 1784189055.14 total 110.07 last 80.68) - open/close 9387 (time 1784189065.14 total 120.07 last 83.59) total: 10000 open/close in 127.34 seconds: 78.53 ops/second - unlinked 0 (time 1784189076 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 730 (time 1784189166.42 total 10.01 last 72.94) - open/close 1502 (time 1784189176.42 total 20.01 last 77.16) - open/close 2085 (time 1784189186.43 total 30.02 last 58.25) - open/close 2814 (time 1784189196.43 total 40.02 last 72.89) - open/close 3540 (time 1784189206.44 total 50.03 last 72.54) - open/close 4433 (time 1784189216.44 total 60.03 last 89.30) - open/close 5254 (time 1784189226.45 total 70.04 last 82.07) - open/close 5998 (time 1784189236.46 total 80.04 last 74.34) - open/close 6711 (time 1784189246.47 total 90.06 last 71.21) - open/close 7245 (time 1784189256.48 total 100.07 last 53.32) - open/close 7950 (time 1784189266.49 total 110.07 last 70.47) - open/close 8788 (time 1784189276.49 total 120.08 last 83.73) - open/close 9550 (time 1784189286.49 total 130.08 last 76.19) total: 10000 open/close in 135.10 seconds: 74.02 ops/second - unlinked 0 (time 1784189295 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 817 (time 1784189383.30 total 10.00 last 81.69) - open/close 1542 (time 1784189393.31 total 20.01 last 72.46) - open/close 2201 (time 1784189403.31 total 30.01 last 65.90) - open/close 2945 (time 1784189413.32 total 40.02 last 74.33) - open/close 3762 (time 1784189423.32 total 50.02 last 81.66) - open/close 4455 (time 1784189433.33 total 60.03 last 69.26) - open/close 5233 (time 1784189443.34 total 70.04 last 77.72) - open/close 6055 (time 1784189453.34 total 80.04 last 82.17) - open/close 6874 (time 1784189463.35 total 90.05 last 81.86) - open/close 7654 (time 1784189473.35 total 100.05 last 77.98) - open/close 8494 (time 1784189483.35 total 110.05 last 83.95) - open/close 9237 (time 1784189493.37 total 120.07 last 74.20) - open/close 9782 (time 1784189503.40 total 130.10 last 54.30) total: 10000 open/close in 132.95 seconds: 75.22 ops/second - unlinked 0 (time 1784189511 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 666 (time 1784189602.06 total 10.01 last 66.55) - open/close 1317 (time 1784189612.06 total 20.01 last 65.07) - open/close 2081 (time 1784189622.06 total 30.01 last 76.39) - open/close 2783 (time 1784189632.08 total 40.03 last 70.09) - open/close 3481 (time 1784189642.10 total 50.05 last 69.64) - open/close 4170 (time 1784189652.11 total 60.06 last 68.84) - open/close 4852 (time 1784189662.11 total 70.06 last 68.19) - open/close 5503 (time 1784189672.12 total 80.07 last 65.05) - open/close 6204 (time 1784189682.12 total 90.08 last 70.05) - open/close 6997 (time 1784189692.13 total 100.08 last 79.28) - open/close 7771 (time 1784189702.14 total 110.09 last 77.34) - open/close 8505 (time 1784189712.14 total 120.10 last 73.34) - open/close 9181 (time 1784189722.16 total 130.11 last 67.51) - open/close 9962 (time 1784189732.20 total 140.15 last 77.81) total: 10000 open/close in 141.01 seconds: 70.92 ops/second - unlinked 0 (time 1784189738 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 719 (time 1784189828.28 total 10.00 last 71.87) - open/close 1344 (time 1784189838.28 total 20.00 last 62.49) - open/close 1952 (time 1784189848.30 total 30.02 last 60.69) - open/close 2660 (time 1784189858.30 total 40.03 last 70.75) - open/close 3416 (time 1784189868.31 total 50.04 last 75.53) - open/close 4191 (time 1784189878.33 total 60.05 last 77.39) - open/close 4807 (time 1784189888.34 total 70.06 last 61.55) - open/close 5305 (time 1784189898.36 total 80.09 last 49.66) - open/close 5719 (time 1784189908.36 total 90.09 last 41.40) - open/close 6215 (time 1784189918.36 total 100.09 last 49.60) - open/close 6900 (time 1784189928.38 total 110.10 last 68.42) - open/close 7680 (time 1784189938.39 total 120.11 last 77.90) - open/close 8255 (time 1784189948.40 total 130.12 last 57.46) - open/close 8856 (time 1784189958.41 total 140.13 last 60.03) - open/close 9667 (time 1784189968.41 total 150.13 last 81.09) total: 10000 open/close in 154.10 seconds: 64.89 ops/second - unlinked 0 (time 1784189977 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg423-server: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg423-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg423-server: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg423-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.123@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.123@tcp sys.timeout=20 start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg423-server: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg423-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg423-server: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg423-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg423-server unloading modules via unload_modules_local on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing unload_modules_local oleg423-server: modules unloaded. pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2