== conf-sanity test 69: replace an OST with the same index ========================================================== 16:04:43 (1710101083) start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 1140 (time 1710101128.64 total 10.01 last 113.90) - open/close 1190 (time 1710101140.18 total 21.55 last 4.33) - open/close 2242 (time 1710101150.18 total 31.55 last 105.19) - open/close 3111 (time 1710101160.19 total 41.56 last 86.82) - open/close 4026 (time 1710101170.19 total 51.56 last 91.49) - open/close 4175 (time 1710101182.96 total 64.33 last 11.67) - open/close 4939 (time 1710101192.97 total 74.34 last 76.32) - open/close 5701 (time 1710101202.97 total 84.34 last 76.20) - open/close 6483 (time 1710101212.98 total 94.35 last 78.15) - open/close 7184 (time 1710101222.99 total 104.36 last 70.04) - open/close 7314 (time 1710101236.04 total 117.41 last 9.96) - open/close 7935 (time 1710101246.05 total 127.42 last 62.04) - open/close 8582 (time 1710101266.44 total 147.81 last 31.73) - open/close 9205 (time 1710101276.45 total 157.82 last 62.22) - open/close 9842 (time 1710101286.47 total 167.84 last 63.63) total: 10000 open/close in 170.37 seconds: 58.70 ops/second - unlinked 0 (time 1710101290 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 1076 (time 1710101389.29 total 10.01 last 107.52) - open/close 2042 (time 1710101399.30 total 20.01 last 96.54) - open/close 2957 (time 1710101409.31 total 30.02 last 91.42) - open/close 3805 (time 1710101419.31 total 40.03 last 84.75) - open/close 4614 (time 1710101429.31 total 50.03 last 80.88) - open/close 5416 (time 1710101439.31 total 60.03 last 80.19) - open/close 6155 (time 1710101449.32 total 70.04 last 73.86) - open/close 6845 (time 1710101459.33 total 80.05 last 68.94) - open/close 7488 (time 1710101469.42 total 90.13 last 63.74) - open/close 8067 (time 1710101479.49 total 100.20 last 57.50) - open/close 8667 (time 1710101489.49 total 110.21 last 59.96) - open/close 9249 (time 1710101499.50 total 120.22 last 58.15) - open/close 9815 (time 1710101509.51 total 130.23 last 56.54) total: 10000 open/close in 133.79 seconds: 74.74 ops/second - unlinked 0 (time 1710101515 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 6 (time 1710101613.79 total 11.14 last 0.54) - open/close 1040 (time 1710101634.52 total 31.88 last 49.86) - open/close 1970 (time 1710101644.52 total 41.88 last 93.00) - open/close 2885 (time 1710101654.53 total 51.88 last 91.49) - open/close 3374 (time 1710101670.99 total 68.35 last 29.69) - open/close 4248 (time 1710101681.00 total 78.35 last 87.35) - open/close 5060 (time 1710101691.01 total 88.36 last 81.14) - open/close 5854 (time 1710101701.01 total 98.37 last 79.36) - open/close 6595 (time 1710101711.01 total 108.37 last 74.08) - open/close 7343 (time 1710101721.03 total 118.38 last 74.70) - open/close 7354 (time 1710101732.26 total 129.61 last 0.98) - open/close 8026 (time 1710101742.27 total 139.63 last 67.12) - open/close 8604 (time 1710101752.28 total 149.63 last 57.75) - open/close 9217 (time 1710101762.29 total 159.65 last 61.23) - open/close 9847 (time 1710101772.30 total 169.65 last 62.95) total: 10000 open/close in 171.82 seconds: 58.20 ops/second - unlinked 0 (time 1710101776 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 1093 (time 1710101869.92 total 10.00 last 109.27) - open/close 2033 (time 1710101879.93 total 20.01 last 93.94) - open/close 2927 (time 1710101889.94 total 30.02 last 89.33) - open/close 3715 (time 1710101909.69 total 49.77 last 39.89) - open/close 4528 (time 1710101919.70 total 59.78 last 81.23) - open/close 5317 (time 1710101929.70 total 69.78 last 78.90) - open/close 5997 (time 1710101939.71 total 79.79 last 67.91) - open/close 6617 (time 1710101960.00 total 100.08 last 30.56) - open/close 7263 (time 1710101970.01 total 110.09 last 64.55) - open/close 7968 (time 1710101980.01 total 120.09 last 70.50) - open/close 8641 (time 1710101990.02 total 130.10 last 67.26) - open/close 9287 (time 1710102000.04 total 140.11 last 64.48) - open/close 9907 (time 1710102010.04 total 150.12 last 61.98) total: 10000 open/close in 151.57 seconds: 65.98 ops/second - unlinked 0 (time 1710102013 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 1047 (time 1710102116.38 total 10.00 last 104.69) - open/close 2012 (time 1710102126.38 total 20.00 last 96.46) - open/close 2950 (time 1710102136.39 total 30.01 last 93.72) - open/close 3788 (time 1710102146.39 total 40.01 last 83.80) - open/close 4621 (time 1710102156.39 total 50.01 last 83.30) - open/close 5423 (time 1710102166.40 total 60.02 last 80.18) - open/close 6203 (time 1710102176.40 total 70.02 last 77.98) - open/close 6935 (time 1710102186.41 total 80.03 last 73.14) - open/close 7600 (time 1710102196.41 total 90.03 last 66.46) - open/close 8296 (time 1710102206.42 total 100.04 last 69.57) - open/close 8914 (time 1710102216.42 total 110.04 last 61.80) - open/close 9533 (time 1710102226.42 total 120.04 last 61.89) total: 10000 open/close in 127.09 seconds: 78.68 ops/second - unlinked 0 (time 1710102235 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 818 (time 1710102349.36 total 18.65 last 43.87) - open/close 1893 (time 1710102359.37 total 28.65 last 107.45) - open/close 2821 (time 1710102369.37 total 38.65 last 92.78) - open/close 3340 (time 1710102386.17 total 55.46 last 30.88) - open/close 4186 (time 1710102396.18 total 65.47 last 84.55) - open/close 5012 (time 1710102406.19 total 75.48 last 82.53) - open/close 5802 (time 1710102416.20 total 85.48 last 78.94) - open/close 6550 (time 1710102426.20 total 95.49 last 74.78) - open/close 7290 (time 1710102436.20 total 105.49 last 73.99) - open/close 7534 (time 1710102450.57 total 119.86 last 16.98) - open/close 8210 (time 1710102460.57 total 129.86 last 67.59) - open/close 8938 (time 1710102470.57 total 139.86 last 72.80) - open/close 9553 (time 1710102480.58 total 149.86 last 61.48) total: 10000 open/close in 156.60 seconds: 63.86 ops/second - unlinked 0 (time 1710102489 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1093 (time 1710102578.44 total 10.00 last 109.28) - open/close 2123 (time 1710102588.45 total 20.01 last 102.93) - open/close 3062 (time 1710102598.45 total 30.02 last 93.84) - open/close 4038 (time 1710102608.46 total 40.02 last 97.53) - open/close 4873 (time 1710102618.48 total 50.05 last 83.30) - open/close 5735 (time 1710102628.48 total 60.05 last 86.20) - open/close 6528 (time 1710102638.50 total 70.06 last 79.17) - open/close 7316 (time 1710102648.50 total 80.07 last 78.77) - open/close 8059 (time 1710102658.50 total 90.07 last 74.30) - open/close 8746 (time 1710102668.51 total 100.08 last 68.62) - open/close 9421 (time 1710102678.52 total 110.09 last 67.43) total: 10000 open/close in 118.36 seconds: 84.49 ops/second - unlinked 0 (time 1710102688 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 626 (time 1710102779.64 total 15.66 last 39.98) - open/close 1820 (time 1710102789.65 total 25.67 last 119.30) - open/close 2903 (time 1710102799.66 total 35.68 last 108.22) - open/close 3943 (time 1710102809.67 total 45.68 last 103.93) - open/close 4981 (time 1710102819.67 total 55.68 last 103.78) - open/close 5976 (time 1710102829.67 total 65.68 last 99.49) - open/close 6923 (time 1710102839.67 total 75.69 last 94.65) - open/close 7828 (time 1710102849.68 total 85.70 last 90.45) - open/close 8700 (time 1710102859.68 total 95.70 last 87.19) - open/close 9220 (time 1710102876.91 total 112.92 last 30.18) total: 10000 open/close in 122.15 seconds: 81.86 ops/second - unlinked 0 (time 1710102887 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1729 (time 1710102956.15 total 10.00 last 172.87) - open/close 3347 (time 1710102966.16 total 20.01 last 161.74) - open/close 4854 (time 1710102976.16 total 30.01 last 150.69) - open/close 6475 (time 1710102986.16 total 40.01 last 162.02) - open/close 8093 (time 1710102996.17 total 50.02 last 161.74) - open/close 9472 (time 1710103006.17 total 60.02 last 137.85) total: 10000 open/close in 64.10 seconds: 156.00 ops/second - unlinked 0 (time 1710103011 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 2963 (time 1710103061.97 total 10.00 last 296.21) - open/close 5200 (time 1710103071.97 total 20.00 last 223.68) - open/close 7129 (time 1710103081.97 total 30.01 last 192.81) - open/close 8951 (time 1710103091.97 total 40.01 last 182.19) total: 10000 open/close in 46.11 seconds: 216.87 ops/second - unlinked 0 (time 1710103098 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre On OST0, 17649 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server unloading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2