== conf-sanity test 69: replace an OST with the same index ========================================================== 12:13:39 (1709054019) start mds service on oleg143-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: detected 4 online CPUs by sysfs oleg143-server: Force libcfs to create 2 CPU partitions oleg143-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg143-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg143-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg143-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1425 (time 1709054076.19 total 10.00 last 142.50) - open/close 3090 (time 1709054086.19 total 20.00 last 166.44) - open/close 4802 (time 1709054096.19 total 30.00 last 171.19) - open/close 5449 (time 1709054113.88 total 47.69 last 36.59) - open/close 6163 (time 1709054123.88 total 57.69 last 71.37) - open/close 7260 (time 1709054133.89 total 67.70 last 109.59) - open/close 8365 (time 1709054143.90 total 77.71 last 110.46) - open/close 9175 (time 1709054153.91 total 87.72 last 80.90) - open/close 9770 (time 1709054163.91 total 97.72 last 59.47) total: 10000 open/close in 101.48 seconds: 98.54 ops/second - unlinked 0 (time 1709054169 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 943 (time 1709054245.83 total 10.01 last 94.24) - open/close 1970 (time 1709054255.84 total 20.02 last 102.57) - open/close 2885 (time 1709054265.84 total 30.02 last 91.46) - open/close 3697 (time 1709054275.84 total 40.03 last 81.19) - open/close 4367 (time 1709054285.86 total 50.04 last 66.88) - open/close 4961 (time 1709054295.87 total 60.05 last 59.36) - open/close 5640 (time 1709054305.89 total 70.07 last 67.74) - open/close 5801 (time 1709054319.73 total 83.91 last 11.63) - open/close 6456 (time 1709054329.74 total 93.93 last 65.41) - open/close 7082 (time 1709054339.75 total 103.93 last 62.59) - open/close 7669 (time 1709054349.75 total 113.93 last 58.69) - open/close 8265 (time 1709054359.77 total 123.95 last 59.49) - open/close 8833 (time 1709054369.78 total 133.96 last 56.75) - open/close 9444 (time 1709054379.80 total 143.98 last 60.94) total: 10000 open/close in 153.18 seconds: 65.28 ops/second - unlinked 0 (time 1709054391 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 1057 (time 1709054506.51 total 10.00 last 105.68) - open/close 2017 (time 1709054516.52 total 20.01 last 95.88) - open/close 2854 (time 1709054526.52 total 30.02 last 83.68) - open/close 3689 (time 1709054536.53 total 40.02 last 83.45) - open/close 4474 (time 1709054546.54 total 50.03 last 78.42) - open/close 5151 (time 1709054556.55 total 60.04 last 67.63) - open/close 5435 (time 1709054571.76 total 75.25 last 18.67) - open/close 6158 (time 1709054581.76 total 85.26 last 72.26) - open/close 6769 (time 1709054591.76 total 95.26 last 61.09) - open/close 7358 (time 1709054601.78 total 105.28 last 58.78) - open/close 7932 (time 1709054611.78 total 115.28 last 57.40) - open/close 8531 (time 1709054621.79 total 125.29 last 59.85) - open/close 9140 (time 1709054631.80 total 135.29 last 60.87) - open/close 9654 (time 1709054641.80 total 145.30 last 51.37) total: 10000 open/close in 151.99 seconds: 65.79 ops/second - unlinked 0 (time 1709054650 ; total 0 ; last 0) total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second - open/close 986 (time 1709054771.14 total 10.01 last 98.54) - open/close 2042 (time 1709054781.15 total 20.02 last 105.50) - open/close 3117 (time 1709054791.15 total 30.02 last 107.44) - open/close 4051 (time 1709054801.16 total 40.03 last 93.36) - open/close 4894 (time 1709054811.16 total 50.03 last 84.24) - open/close 5687 (time 1709054821.16 total 60.03 last 79.30) - open/close 6450 (time 1709054831.16 total 70.03 last 76.29) - open/close 7192 (time 1709054841.17 total 80.04 last 74.14) - open/close 7878 (time 1709054851.18 total 90.05 last 68.54) - open/close 8479 (time 1709054861.19 total 100.06 last 60.05) - open/close 9074 (time 1709054871.19 total 110.06 last 59.47) - open/close 9583 (time 1709054881.20 total 120.07 last 50.88) total: 10000 open/close in 126.70 seconds: 78.92 ops/second - unlinked 0 (time 1709054889 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 1410 (time 1709054973.91 total 10.00 last 141.00) - open/close 2429 (time 1709054983.92 total 20.01 last 101.77) - open/close 3296 (time 1709054993.93 total 30.02 last 86.64) - open/close 4065 (time 1709055003.93 total 40.02 last 76.86) - open/close 4810 (time 1709055013.94 total 50.03 last 74.43) - open/close 5528 (time 1709055023.95 total 60.05 last 71.70) - open/close 6215 (time 1709055033.98 total 70.07 last 68.51) - open/close 6904 (time 1709055043.98 total 80.08 last 68.89) - open/close 7491 (time 1709055053.99 total 90.08 last 58.67) - open/close 8108 (time 1709055063.99 total 100.08 last 61.68) - open/close 8795 (time 1709055074.00 total 110.09 last 68.67) - open/close 9394 (time 1709055084.01 total 120.10 last 59.82) - open/close 9954 (time 1709055094.01 total 130.10 last 55.99) total: 10000 open/close in 130.82 seconds: 76.44 ops/second - unlinked 0 (time 1709055096 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 1264 (time 1709055174.77 total 10.01 last 126.22) - open/close 2280 (time 1709055184.78 total 20.02 last 101.56) - open/close 3229 (time 1709055194.78 total 30.02 last 94.88) - open/close 4139 (time 1709055204.78 total 40.02 last 90.99) - open/close 4436 (time 1709055219.37 total 54.61 last 20.36) - open/close 5211 (time 1709055229.37 total 64.62 last 77.45) - open/close 6015 (time 1709055239.38 total 74.62 last 80.33) - open/close 6781 (time 1709055249.38 total 84.63 last 76.57) - open/close 7459 (time 1709055259.40 total 94.64 last 67.70) - open/close 8086 (time 1709055269.42 total 104.66 last 62.60) - open/close 8700 (time 1709055279.43 total 114.67 last 61.31) - open/close 9218 (time 1709055289.48 total 124.72 last 51.56) - open/close 9728 (time 1709055299.48 total 134.73 last 50.97) total: 10000 open/close in 140.04 seconds: 71.41 ops/second - unlinked 0 (time 1709055306 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 1070 (time 1709055422.98 total 10.00 last 106.98) - open/close 2095 (time 1709055432.99 total 20.01 last 102.39) - open/close 2964 (time 1709055442.99 total 30.02 last 86.84) - open/close 3937 (time 1709055453.00 total 40.03 last 97.23) - open/close 4756 (time 1709055463.01 total 50.03 last 81.84) - open/close 5008 (time 1709055477.75 total 64.77 last 17.10) - open/close 5845 (time 1709055487.75 total 74.77 last 83.68) - open/close 6753 (time 1709055497.75 total 84.77 last 90.78) - open/close 7557 (time 1709055507.75 total 94.78 last 80.37) - open/close 8380 (time 1709055517.75 total 104.78 last 82.29) - open/close 9237 (time 1709055527.76 total 114.79 last 85.63) - open/close 9971 (time 1709055537.77 total 124.79 last 73.35) total: 10000 open/close in 125.28 seconds: 79.82 ops/second - unlinked 0 (time 1709055539 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 1476 (time 1709055611.95 total 10.00 last 147.53) - open/close 2837 (time 1709055621.96 total 20.02 last 135.94) - open/close 3286 (time 1709055637.27 total 35.32 last 29.34) - open/close 4297 (time 1709055647.27 total 45.32 last 101.05) - open/close 5208 (time 1709055657.28 total 55.33 last 91.00) - open/close 6346 (time 1709055667.28 total 65.34 last 113.78) - open/close 7282 (time 1709055677.29 total 75.35 last 93.50) - open/close 8116 (time 1709055687.30 total 85.35 last 83.37) - open/close 8780 (time 1709055697.30 total 95.35 last 66.39) - open/close 9554 (time 1709055707.30 total 105.35 last 77.39) total: 10000 open/close in 109.22 seconds: 91.56 ops/second - unlinked 0 (time 1709055712 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1652 (time 1709055778.22 total 10.01 last 165.06) - open/close 2874 (time 1709055788.22 total 20.01 last 122.14) - open/close 3897 (time 1709055798.22 total 30.02 last 102.29) - open/close 4869 (time 1709055808.23 total 40.02 last 97.14) - open/close 5863 (time 1709055818.24 total 50.03 last 99.34) - open/close 6695 (time 1709055828.24 total 60.03 last 83.20) - open/close 7528 (time 1709055838.27 total 70.06 last 83.02) - open/close 8283 (time 1709055848.27 total 80.07 last 75.46) - open/close 8416 (time 1709055860.80 total 92.59 last 10.62) - open/close 9202 (time 1709055870.80 total 102.59 last 78.58) - open/close 9923 (time 1709055880.81 total 112.60 last 72.01) total: 10000 open/close in 113.73 seconds: 87.93 ops/second - unlinked 0 (time 1709055883 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 1310 (time 1709055974.73 total 10.00 last 130.98) - open/close 2451 (time 1709055984.73 total 20.01 last 114.03) - open/close 3510 (time 1709055994.74 total 30.01 last 105.86) - open/close 4576 (time 1709056004.74 total 40.02 last 106.52) - open/close 5546 (time 1709056014.76 total 50.03 last 96.90) - open/close 6492 (time 1709056024.76 total 60.03 last 94.59) - open/close 7354 (time 1709056034.76 total 70.04 last 86.15) - open/close 8313 (time 1709056044.77 total 80.04 last 95.83) - open/close 9098 (time 1709056054.77 total 90.05 last 78.47) - open/close 9890 (time 1709056064.78 total 100.05 last 79.16) total: 10000 open/close in 101.67 seconds: 98.36 ops/second - unlinked 0 (time 1709056067 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg143-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server unloading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2