== conf-sanity test 69: replace an OST with the same index ========================================================== 23:28:00 (1746156480) start mds service on oleg112-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: MODOPTS_LIBCFS= oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg112-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 881 (time 1746156550.75 total 10.00 last 88.09) - open/close 1794 (time 1746156560.76 total 20.01 last 91.22) - open/close 2730 (time 1746156570.76 total 30.01 last 93.57) - open/close 3667 (time 1746156580.77 total 40.03 last 93.58) - open/close 4639 (time 1746156590.78 total 50.03 last 97.13) - open/close 5483 (time 1746156600.79 total 60.04 last 84.31) - open/close 6363 (time 1746156610.79 total 70.05 last 87.97) - open/close 7148 (time 1746156620.80 total 80.05 last 78.45) - open/close 8001 (time 1746156630.81 total 90.06 last 85.19) - open/close 8919 (time 1746156640.82 total 100.08 last 91.69) - open/close 9755 (time 1746156650.82 total 110.08 last 83.59) total: 10000 open/close in 112.56 seconds: 88.84 ops/second - unlinked 0 (time 1746156655 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 779 (time 1746156737.10 total 10.01 last 77.82) - open/close 1619 (time 1746156747.10 total 20.01 last 83.96) - open/close 2468 (time 1746156757.10 total 30.02 last 84.87) - open/close 3358 (time 1746156767.11 total 40.03 last 88.93) - open/close 4256 (time 1746156777.12 total 50.04 last 89.71) - open/close 5115 (time 1746156787.14 total 60.05 last 85.76) - open/close 5913 (time 1746156797.15 total 70.06 last 79.74) - open/close 6741 (time 1746156807.15 total 80.07 last 82.76) - open/close 7536 (time 1746156817.16 total 90.07 last 79.45) - open/close 8197 (time 1746156827.16 total 100.08 last 66.06) - open/close 8992 (time 1746156837.18 total 110.10 last 79.34) - open/close 9814 (time 1746156847.18 total 120.10 last 82.20) total: 10000 open/close in 122.18 seconds: 81.84 ops/second - unlinked 0 (time 1746156852 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 883 (time 1746156937.82 total 10.01 last 88.25) - open/close 1765 (time 1746156947.83 total 20.01 last 88.19) - open/close 2664 (time 1746156957.83 total 30.01 last 89.89) - open/close 3537 (time 1746156967.83 total 40.01 last 87.25) - open/close 4398 (time 1746156977.83 total 50.01 last 86.10) - open/close 5146 (time 1746156987.83 total 60.02 last 74.78) - open/close 5966 (time 1746156997.84 total 70.02 last 81.98) - open/close 6887 (time 1746157007.84 total 80.02 last 92.10) - open/close 7754 (time 1746157017.85 total 90.03 last 86.63) - open/close 8596 (time 1746157027.85 total 100.03 last 84.15) - open/close 9456 (time 1746157037.86 total 110.04 last 85.96) total: 10000 open/close in 115.10 seconds: 86.88 ops/second - unlinked 0 (time 1746157045 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 848 (time 1746157130.72 total 10.00 last 84.78) - open/close 1643 (time 1746157140.73 total 20.01 last 79.47) - open/close 2537 (time 1746157150.73 total 30.01 last 89.35) - open/close 3446 (time 1746157160.74 total 40.01 last 90.88) - open/close 4308 (time 1746157170.74 total 50.02 last 86.15) - open/close 5119 (time 1746157180.74 total 60.02 last 81.08) - open/close 6005 (time 1746157190.76 total 70.03 last 88.48) - open/close 6932 (time 1746157200.77 total 80.05 last 92.58) - open/close 7815 (time 1746157210.78 total 90.06 last 88.17) - open/close 8769 (time 1746157220.79 total 100.07 last 95.36) - open/close 9583 (time 1746157230.79 total 110.07 last 81.38) total: 10000 open/close in 115.25 seconds: 86.77 ops/second - unlinked 0 (time 1746157238 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 930 (time 1746157319.65 total 10.00 last 92.99) - open/close 1947 (time 1746157329.65 total 20.00 last 101.69) - open/close 2840 (time 1746157339.65 total 30.00 last 89.28) - open/close 3692 (time 1746157349.66 total 40.01 last 85.14) - open/close 4562 (time 1746157359.66 total 50.01 last 86.97) - open/close 5486 (time 1746157369.67 total 60.02 last 92.32) - open/close 6389 (time 1746157379.67 total 70.03 last 90.27) - open/close 7205 (time 1746157389.67 total 80.03 last 81.59) - open/close 8100 (time 1746157399.68 total 90.03 last 89.45) - open/close 8887 (time 1746157409.70 total 100.05 last 78.57) - open/close 9612 (time 1746157419.70 total 110.06 last 72.45) total: 10000 open/close in 114.98 seconds: 86.97 ops/second - unlinked 0 (time 1746157427 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 891 (time 1746157506.82 total 10.01 last 89.02) - open/close 1778 (time 1746157516.83 total 20.02 last 88.61) - open/close 2575 (time 1746157526.84 total 30.02 last 79.67) - open/close 3360 (time 1746157536.85 total 40.04 last 78.36) - open/close 4229 (time 1746157546.85 total 50.04 last 86.89) - open/close 5158 (time 1746157556.86 total 60.05 last 92.85) - open/close 5998 (time 1746157566.86 total 70.05 last 83.96) - open/close 6867 (time 1746157576.87 total 80.06 last 86.84) - open/close 7808 (time 1746157586.88 total 90.07 last 94.01) - open/close 8721 (time 1746157596.89 total 100.08 last 91.20) - open/close 9645 (time 1746157606.92 total 110.10 last 92.17) total: 10000 open/close in 114.00 seconds: 87.72 ops/second - unlinked 0 (time 1746157612 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 1103 (time 1746157683.79 total 10.01 last 110.23) - open/close 2014 (time 1746157693.80 total 20.01 last 91.02) - open/close 2917 (time 1746157703.80 total 30.02 last 90.27) - open/close 3776 (time 1746157713.80 total 40.02 last 85.88) - open/close 4511 (time 1746157723.80 total 50.02 last 73.49) - open/close 5338 (time 1746157733.80 total 60.02 last 82.68) - open/close 6170 (time 1746157743.82 total 70.04 last 83.06) - open/close 7010 (time 1746157753.82 total 80.04 last 83.99) - open/close 7780 (time 1746157763.83 total 90.05 last 76.90) - open/close 8601 (time 1746157773.84 total 100.06 last 82.01) - open/close 9386 (time 1746157783.85 total 110.07 last 78.44) total: 10000 open/close in 117.42 seconds: 85.16 ops/second - unlinked 0 (time 1746157793 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 784 (time 1746157875.48 total 10.00 last 78.38) - open/close 1559 (time 1746157885.49 total 20.01 last 77.41) - open/close 2462 (time 1746157895.50 total 30.02 last 90.22) - open/close 3498 (time 1746157905.51 total 40.03 last 103.56) - open/close 4388 (time 1746157915.51 total 50.03 last 88.94) - open/close 5284 (time 1746157925.52 total 60.04 last 89.50) - open/close 6196 (time 1746157935.54 total 70.06 last 91.08) - open/close 7049 (time 1746157945.54 total 80.06 last 85.26) - open/close 7935 (time 1746157955.55 total 90.07 last 88.57) - open/close 8782 (time 1746157965.55 total 100.07 last 84.64) - open/close 9809 (time 1746157975.56 total 110.08 last 102.68) total: 10000 open/close in 111.86 seconds: 89.40 ops/second - unlinked 0 (time 1746157979 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 1089 (time 1746158050.64 total 10.00 last 108.89) - open/close 2068 (time 1746158060.65 total 20.01 last 97.82) - open/close 2983 (time 1746158070.66 total 30.02 last 91.40) - open/close 3983 (time 1746158080.66 total 40.02 last 99.97) - open/close 5050 (time 1746158090.67 total 50.03 last 106.62) - open/close 5991 (time 1746158100.67 total 60.03 last 94.08) - open/close 6847 (time 1746158110.68 total 70.04 last 85.56) - open/close 7746 (time 1746158120.68 total 80.04 last 89.85) - open/close 8707 (time 1746158130.69 total 90.05 last 96.04) - open/close 9594 (time 1746158140.70 total 100.06 last 88.63) total: 10000 open/close in 104.93 seconds: 95.31 ops/second - unlinked 0 (time 1746158148 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1045 (time 1746158223.94 total 10.00 last 104.47) - open/close 2162 (time 1746158233.94 total 20.00 last 111.69) - open/close 3228 (time 1746158243.94 total 30.00 last 106.59) - open/close 4407 (time 1746158253.95 total 40.01 last 117.82) - open/close 5467 (time 1746158263.95 total 50.02 last 105.94) - open/close 6580 (time 1746158273.96 total 60.02 last 111.23) - open/close 7578 (time 1746158283.98 total 70.05 last 99.58) - open/close 8532 (time 1746158293.98 total 80.05 last 95.39) - open/close 9664 (time 1746158303.99 total 90.05 last 113.12) total: 10000 open/close in 93.38 seconds: 107.09 ops/second - unlinked 0 (time 1746158309 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg112-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2