== conf-sanity test 69: replace an OST with the same index ========================================================== 15:56:52 (1786305412) start mds service on oleg334-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions oleg334-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg334-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg334-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg334-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg334-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 1070 (time 1786305510.63 total 10.00 last 106.99) - open/close 2052 (time 1786305520.63 total 20.00 last 98.18) - open/close 3092 (time 1786305530.64 total 30.01 last 103.92) - open/close 4117 (time 1786305540.64 total 40.01 last 102.50) - open/close 5129 (time 1786305550.64 total 50.02 last 101.16) - open/close 6105 (time 1786305560.65 total 60.02 last 97.55) - open/close 7113 (time 1786305570.65 total 70.03 last 100.75) - open/close 8144 (time 1786305580.66 total 80.03 last 103.02) - open/close 9196 (time 1786305590.67 total 90.04 last 105.16) total: 10000 open/close in 98.00 seconds: 102.04 ops/second - unlinked 0 (time 1786305602 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 945 (time 1786305676.53 total 10.01 last 94.42) - open/close 1905 (time 1786305686.53 total 20.01 last 95.99) - open/close 2973 (time 1786305696.54 total 30.02 last 106.73) - open/close 3992 (time 1786305706.55 total 40.03 last 101.81) - open/close 4925 (time 1786305716.55 total 50.03 last 93.29) - open/close 5984 (time 1786305726.56 total 60.04 last 105.78) - open/close 6943 (time 1786305736.56 total 70.04 last 95.89) - open/close 7912 (time 1786305746.57 total 80.05 last 96.79) - open/close 8865 (time 1786305756.58 total 90.05 last 95.26) - open/close 9808 (time 1786305766.58 total 100.06 last 94.27) total: 10000 open/close in 102.22 seconds: 97.83 ops/second - unlinked 0 (time 1786305772 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 895 (time 1786305852.69 total 10.02 last 89.32) - open/close 1909 (time 1786305862.74 total 20.07 last 100.87) - open/close 2923 (time 1786305872.75 total 30.08 last 101.37) - open/close 3839 (time 1786305882.76 total 40.09 last 91.50) - open/close 4702 (time 1786305892.76 total 50.09 last 86.27) - open/close 5552 (time 1786305902.76 total 60.09 last 84.99) - open/close 6492 (time 1786305912.76 total 70.09 last 93.98) - open/close 7445 (time 1786305922.76 total 80.09 last 95.29) - open/close 8406 (time 1786305932.77 total 90.10 last 96.07) - open/close 9348 (time 1786305942.77 total 100.10 last 94.17) total: 10000 open/close in 106.27 seconds: 94.10 ops/second - unlinked 0 (time 1786305952 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 956 (time 1786306032.74 total 10.00 last 95.58) - open/close 1848 (time 1786306042.75 total 20.01 last 89.17) - open/close 2836 (time 1786306052.75 total 30.01 last 98.76) - open/close 3801 (time 1786306062.75 total 40.01 last 96.48) - open/close 4815 (time 1786306072.76 total 50.02 last 101.33) - open/close 5765 (time 1786306082.78 total 60.04 last 94.78) - open/close 6738 (time 1786306092.79 total 70.05 last 97.24) - open/close 7726 (time 1786306102.79 total 80.05 last 98.75) - open/close 8689 (time 1786306112.80 total 90.06 last 96.24) - open/close 9664 (time 1786306122.81 total 100.06 last 97.45) total: 10000 open/close in 103.57 seconds: 96.55 ops/second - unlinked 0 (time 1786306130 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 929 (time 1786306208.64 total 10.01 last 92.78) - open/close 1807 (time 1786306218.64 total 20.01 last 87.79) - open/close 2815 (time 1786306229.21 total 30.58 last 95.37) - open/close 3681 (time 1786306239.21 total 40.59 last 86.58) - open/close 4624 (time 1786306249.22 total 50.59 last 94.21) - open/close 5540 (time 1786306259.22 total 60.60 last 91.54) - open/close 6586 (time 1786306269.23 total 70.61 last 104.51) - open/close 7574 (time 1786306279.24 total 80.61 last 98.77) - open/close 8616 (time 1786306289.25 total 90.62 last 104.09) - open/close 9588 (time 1786306299.25 total 100.63 last 97.13) total: 10000 open/close in 105.56 seconds: 94.73 ops/second - unlinked 0 (time 1786306308 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 1083 (time 1786306388.67 total 10.01 last 108.23) - open/close 1889 (time 1786306398.76 total 20.09 last 79.93) - open/close 2727 (time 1786306408.77 total 30.10 last 83.73) - open/close 3768 (time 1786306418.78 total 40.11 last 103.99) - open/close 4746 (time 1786306428.78 total 50.11 last 97.78) - open/close 5730 (time 1786306438.78 total 60.11 last 98.38) - open/close 6743 (time 1786306448.78 total 70.12 last 101.25) - open/close 7699 (time 1786306458.79 total 80.12 last 95.56) - open/close 8480 (time 1786306468.80 total 90.13 last 78.00) - open/close 9353 (time 1786306478.80 total 100.14 last 87.26) total: 10000 open/close in 106.41 seconds: 93.97 ops/second - unlinked 0 (time 1786306489 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 997 (time 1786306563.74 total 10.00 last 99.65) - open/close 1974 (time 1786306573.75 total 20.01 last 97.60) - open/close 2969 (time 1786306583.76 total 30.03 last 99.38) - open/close 3985 (time 1786306593.77 total 40.04 last 101.46) - open/close 5028 (time 1786306603.82 total 50.09 last 103.78) - open/close 5916 (time 1786306613.83 total 60.09 last 88.77) - open/close 6957 (time 1786306623.84 total 70.10 last 104.01) - open/close 7981 (time 1786306633.84 total 80.10 last 102.39) - open/close 9043 (time 1786306643.84 total 90.11 last 106.15) - open/close 9980 (time 1786306653.85 total 100.11 last 93.67) total: 10000 open/close in 100.27 seconds: 99.73 ops/second - unlinked 0 (time 1786306657 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 923 (time 1786306735.99 total 10.00 last 92.29) - open/close 1904 (time 1786306745.99 total 20.00 last 98.07) - open/close 3002 (time 1786306755.99 total 30.01 last 109.76) - open/close 3937 (time 1786306766.00 total 40.01 last 93.46) - open/close 4944 (time 1786306776.00 total 50.02 last 100.66) - open/close 6038 (time 1786306786.01 total 60.02 last 109.32) - open/close 7125 (time 1786306796.03 total 70.05 last 108.44) - open/close 8078 (time 1786306806.04 total 80.06 last 95.20) - open/close 9071 (time 1786306816.06 total 90.08 last 99.11) total: 10000 open/close in 99.59 seconds: 100.41 ops/second - unlinked 0 (time 1786306830 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 949 (time 1786306911.61 total 10.00 last 94.88) - open/close 1803 (time 1786306921.61 total 20.00 last 85.39) - open/close 2799 (time 1786306931.61 total 30.01 last 99.55) - open/close 3708 (time 1786306941.62 total 40.01 last 90.86) - open/close 4706 (time 1786306951.63 total 50.02 last 99.68) - open/close 5739 (time 1786306961.75 total 60.14 last 102.11) - open/close 6535 (time 1786306971.80 total 70.19 last 79.18) - open/close 7448 (time 1786306981.80 total 80.20 last 91.25) - open/close 8380 (time 1786306991.81 total 90.20 last 93.17) - open/close 9391 (time 1786307001.81 total 100.21 last 101.05) total: 10000 open/close in 106.23 seconds: 94.13 ops/second - unlinked 0 (time 1786307011 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 1001 (time 1786307086.46 total 10.02 last 99.91) - open/close 1940 (time 1786307096.47 total 20.02 last 93.87) - open/close 2935 (time 1786307106.47 total 30.03 last 99.45) - open/close 3936 (time 1786307116.47 total 40.03 last 100.07) - open/close 4968 (time 1786307126.48 total 50.03 last 103.18) - open/close 6013 (time 1786307136.48 total 60.03 last 104.50) - open/close 7066 (time 1786307146.48 total 70.04 last 105.26) - open/close 8097 (time 1786307156.48 total 80.04 last 103.09) - open/close 9127 (time 1786307166.48 total 90.04 last 102.96) - open/close 9984 (time 1786307176.49 total 100.05 last 85.62) total: 10000 open/close in 100.21 seconds: 99.79 ops/second - unlinked 0 (time 1786307181 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.134@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg334-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg334-server: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg334-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg334-server: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg334-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.134@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.134@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.134@tcp sys.timeout=20 start ost1 service on oleg334-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg334-server: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg334-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg334-server: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg334-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg334-server unloading modules via unload_modules_local on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing unload_modules_local oleg334-server: modules unloaded. pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2