== conf-sanity test 69: replace an OST with the same index ========================================================== 15:56:32 (1746129392) start mds service on oleg146-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing load_modules_local oleg146-server: Loading modules from /home/green/git/lustre-release/lustre oleg146-server: detected 4 online CPUs by sysfs oleg146-server: MODOPTS_LIBCFS= oleg146-server: Force libcfs to create 2 CPU partitions oleg146-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg146-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg146-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg146-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg146-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 847 (time 1746129464.75 total 10.00 last 84.68) - open/close 1651 (time 1746129474.77 total 20.02 last 80.24) - open/close 2626 (time 1746129484.78 total 30.03 last 97.39) - open/close 3532 (time 1746129494.79 total 40.04 last 90.50) - open/close 4342 (time 1746129504.80 total 50.05 last 80.94) - open/close 5154 (time 1746129514.80 total 60.05 last 81.20) - open/close 6062 (time 1746129524.80 total 70.06 last 90.75) - open/close 6899 (time 1746129534.81 total 80.06 last 83.69) - open/close 7738 (time 1746129544.81 total 90.07 last 83.84) - open/close 8555 (time 1746129554.83 total 100.08 last 81.59) - open/close 9435 (time 1746129564.83 total 110.09 last 87.95) total: 10000 open/close in 116.75 seconds: 85.65 ops/second - unlinked 0 (time 1746129574 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 880 (time 1746129654.12 total 10.00 last 87.98) - open/close 1823 (time 1746129664.13 total 20.01 last 94.26) - open/close 2631 (time 1746129674.13 total 30.01 last 80.79) - open/close 3426 (time 1746129684.14 total 40.02 last 79.42) - open/close 4256 (time 1746129694.14 total 50.02 last 82.97) - open/close 5104 (time 1746129704.14 total 60.02 last 84.78) - open/close 5862 (time 1746129714.15 total 70.03 last 75.74) - open/close 6838 (time 1746129724.16 total 80.04 last 97.47) - open/close 7830 (time 1746129734.17 total 90.05 last 99.17) - open/close 8589 (time 1746129744.17 total 100.05 last 75.88) - open/close 9408 (time 1746129754.18 total 110.06 last 81.81) total: 10000 open/close in 116.49 seconds: 85.84 ops/second - unlinked 0 (time 1746129762 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 839 (time 1746129844.35 total 10.01 last 83.78) - open/close 1684 (time 1746129854.35 total 20.02 last 84.46) - open/close 2615 (time 1746129864.36 total 30.03 last 93.04) - open/close 3451 (time 1746129874.37 total 40.04 last 83.51) - open/close 4356 (time 1746129884.38 total 50.04 last 90.43) - open/close 5116 (time 1746129894.38 total 60.05 last 75.98) - open/close 5793 (time 1746129904.40 total 70.07 last 67.57) - open/close 6593 (time 1746129914.41 total 80.07 last 79.94) - open/close 7368 (time 1746129924.42 total 90.09 last 77.41) - open/close 8148 (time 1746129934.43 total 100.09 last 77.95) - open/close 9044 (time 1746129944.43 total 110.10 last 89.55) - open/close 9780 (time 1746129954.44 total 120.10 last 73.56) total: 10000 open/close in 122.74 seconds: 81.47 ops/second - unlinked 0 (time 1746129959 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 841 (time 1746130046.50 total 10.01 last 84.02) - open/close 1747 (time 1746130056.51 total 20.02 last 90.54) - open/close 2624 (time 1746130066.51 total 30.02 last 87.67) - open/close 3410 (time 1746130076.51 total 40.02 last 78.57) - open/close 4232 (time 1746130086.52 total 50.03 last 82.15) - open/close 5130 (time 1746130096.53 total 60.04 last 89.72) - open/close 5916 (time 1746130106.53 total 70.04 last 78.56) - open/close 6746 (time 1746130116.56 total 80.07 last 82.76) - open/close 7521 (time 1746130126.56 total 90.07 last 77.50) - open/close 8384 (time 1746130136.57 total 100.08 last 86.23) - open/close 9267 (time 1746130146.58 total 110.09 last 88.25) total: 10000 open/close in 118.25 seconds: 84.57 ops/second - unlinked 0 (time 1746130157 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 910 (time 1746130235.02 total 10.00 last 90.97) - open/close 1802 (time 1746130245.03 total 20.01 last 89.16) - open/close 2946 (time 1746130255.03 total 30.01 last 114.40) - open/close 3861 (time 1746130265.03 total 40.01 last 91.45) - open/close 4819 (time 1746130275.03 total 50.02 last 95.78) - open/close 5847 (time 1746130285.04 total 60.02 last 102.77) - open/close 6689 (time 1746130295.04 total 70.03 last 84.14) - open/close 7500 (time 1746130305.05 total 80.03 last 81.08) - open/close 8240 (time 1746130315.06 total 90.05 last 73.87) - open/close 9056 (time 1746130325.07 total 100.05 last 81.58) - open/close 9753 (time 1746130335.07 total 110.05 last 69.66) total: 10000 open/close in 112.72 seconds: 88.71 ops/second - unlinked 0 (time 1746130340 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 900 (time 1746130423.82 total 10.01 last 89.93) - open/close 1821 (time 1746130433.82 total 20.01 last 92.09) - open/close 2573 (time 1746130443.83 total 30.01 last 75.15) - open/close 3445 (time 1746130453.83 total 40.02 last 87.18) - open/close 4330 (time 1746130463.83 total 50.02 last 88.46) - open/close 5217 (time 1746130473.84 total 60.03 last 88.64) - open/close 6199 (time 1746130483.84 total 70.03 last 98.16) - open/close 7246 (time 1746130493.85 total 80.04 last 104.63) - open/close 8197 (time 1746130503.85 total 90.04 last 95.07) - open/close 9161 (time 1746130513.86 total 100.05 last 96.35) total: 10000 open/close in 109.85 seconds: 91.03 ops/second - unlinked 0 (time 1746130526 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 909 (time 1746130603.99 total 10.01 last 90.82) - open/close 1864 (time 1746130614.00 total 20.02 last 95.44) - open/close 2832 (time 1746130624.01 total 30.03 last 96.69) - open/close 3798 (time 1746130634.02 total 40.03 last 96.54) - open/close 4860 (time 1746130644.02 total 50.03 last 106.19) - open/close 5845 (time 1746130654.03 total 60.05 last 98.36) - open/close 6741 (time 1746130664.05 total 70.06 last 89.45) - open/close 7709 (time 1746130674.05 total 80.07 last 96.79) - open/close 8698 (time 1746130684.05 total 90.07 last 98.90) - open/close 9664 (time 1746130694.06 total 100.07 last 96.55) total: 10000 open/close in 103.33 seconds: 96.78 ops/second - unlinked 0 (time 1746130699 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1221 (time 1746130768.24 total 10.00 last 122.07) - open/close 2229 (time 1746130778.25 total 20.01 last 100.73) - open/close 3300 (time 1746130788.25 total 30.01 last 107.08) - open/close 4308 (time 1746130798.26 total 40.03 last 100.65) - open/close 5123 (time 1746130808.27 total 50.03 last 81.45) - open/close 6100 (time 1746130818.27 total 60.04 last 97.64) - open/close 7118 (time 1746130828.28 total 70.05 last 101.72) - open/close 8085 (time 1746130838.29 total 80.05 last 96.65) - open/close 9078 (time 1746130848.30 total 90.06 last 99.21) total: 10000 open/close in 99.29 seconds: 100.71 ops/second - unlinked 0 (time 1746130859 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 926 (time 1746130931.99 total 10.01 last 92.54) - open/close 1713 (time 1746130942.00 total 20.02 last 78.61) - open/close 2704 (time 1746130952.00 total 30.02 last 99.07) - open/close 3686 (time 1746130962.00 total 40.02 last 98.18) - open/close 4680 (time 1746130972.02 total 50.03 last 99.28) - open/close 5604 (time 1746130982.02 total 60.04 last 92.34) - open/close 6571 (time 1746130992.03 total 70.04 last 96.67) - open/close 7603 (time 1746131002.04 total 80.06 last 103.08) - open/close 8598 (time 1746131012.04 total 90.06 last 99.43) - open/close 9447 (time 1746131022.05 total 100.07 last 84.83) total: 10000 open/close in 106.29 seconds: 94.08 ops/second - unlinked 0 (time 1746131030 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1062 (time 1746131099.10 total 10.00 last 106.17) - open/close 2073 (time 1746131109.10 total 20.00 last 101.08) - open/close 3109 (time 1746131119.11 total 30.01 last 103.52) - open/close 4259 (time 1746131129.11 total 40.02 last 114.96) - open/close 5322 (time 1746131139.12 total 50.02 last 106.25) - open/close 6390 (time 1746131149.12 total 60.03 last 106.75) - open/close 7614 (time 1746131159.13 total 70.03 last 122.34) - open/close 8759 (time 1746131169.13 total 80.04 last 114.45) - open/close 9778 (time 1746131179.14 total 90.04 last 101.86) total: 10000 open/close in 92.03 seconds: 108.66 ops/second - unlinked 0 (time 1746131183 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg146-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.146@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg146-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg146-server: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg146-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec oleg146-server: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg146-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre On OST0, 20077 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg146-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg146-server unloading modules via unload_modules_local on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2