== conf-sanity test 69: replace an OST with the same index ========================================================== 23:05:21 (1746155121) start mds service on oleg127-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing load_modules_local oleg127-server: Loading modules from /home/green/git/lustre-release/lustre oleg127-server: detected 4 online CPUs by sysfs oleg127-server: MODOPTS_LIBCFS= oleg127-server: Force libcfs to create 2 CPU partitions oleg127-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg127-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg127-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg127-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg127-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 806 (time 1746155192.80 total 10.01 last 80.54) - open/close 1534 (time 1746155202.80 total 20.01 last 72.80) - open/close 2366 (time 1746155212.81 total 30.02 last 83.11) - open/close 3151 (time 1746155222.82 total 40.02 last 78.47) - open/close 3926 (time 1746155232.83 total 50.03 last 77.44) - open/close 4799 (time 1746155242.83 total 60.03 last 87.29) - open/close 5582 (time 1746155252.83 total 70.04 last 78.24) - open/close 6277 (time 1746155262.84 total 80.05 last 69.45) - open/close 6930 (time 1746155272.85 total 90.05 last 65.27) - open/close 7579 (time 1746155282.87 total 100.07 last 64.78) - open/close 8211 (time 1746155292.87 total 110.08 last 63.18) - open/close 8939 (time 1746155302.89 total 120.09 last 72.69) - open/close 9759 (time 1746155312.89 total 130.09 last 81.99) total: 10000 open/close in 133.03 seconds: 75.17 ops/second - unlinked 0 (time 1746155318 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 883 (time 1746155401.41 total 10.01 last 88.22) - open/close 1780 (time 1746155411.42 total 20.01 last 89.65) - open/close 2714 (time 1746155421.42 total 30.02 last 93.34) - open/close 3530 (time 1746155431.43 total 40.03 last 81.56) - open/close 4420 (time 1746155441.43 total 50.03 last 88.96) - open/close 5325 (time 1746155451.44 total 60.04 last 90.45) - open/close 6258 (time 1746155461.45 total 70.05 last 93.21) - open/close 7178 (time 1746155471.45 total 80.05 last 91.95) - open/close 8164 (time 1746155481.46 total 90.06 last 98.54) - open/close 8980 (time 1746155491.47 total 100.07 last 81.53) - open/close 9718 (time 1746155501.47 total 110.07 last 73.80) total: 10000 open/close in 113.00 seconds: 88.50 ops/second - unlinked 0 (time 1746155506 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 913 (time 1746155586.43 total 10.01 last 91.17) - open/close 1763 (time 1746155596.43 total 20.02 last 84.98) - open/close 2590 (time 1746155606.45 total 30.03 last 82.56) - open/close 3431 (time 1746155616.47 total 40.05 last 83.94) - open/close 4233 (time 1746155626.47 total 50.06 last 80.17) - open/close 5073 (time 1746155636.48 total 60.06 last 83.96) - open/close 5847 (time 1746155646.48 total 70.07 last 77.35) - open/close 6590 (time 1746155656.48 total 80.07 last 74.30) - open/close 7324 (time 1746155666.49 total 90.07 last 73.37) - open/close 8177 (time 1746155676.51 total 100.09 last 85.10) - open/close 8955 (time 1746155686.51 total 110.10 last 77.78) - open/close 9716 (time 1746155696.52 total 120.11 last 76.04) total: 10000 open/close in 123.89 seconds: 80.72 ops/second - unlinked 0 (time 1746155702 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 883 (time 1746155785.81 total 10.00 last 88.27) - open/close 1856 (time 1746155795.82 total 20.01 last 97.23) - open/close 2743 (time 1746155805.83 total 30.02 last 88.60) - open/close 3684 (time 1746155815.84 total 40.03 last 94.03) - open/close 4574 (time 1746155825.84 total 50.04 last 88.94) - open/close 5543 (time 1746155835.85 total 60.04 last 96.83) - open/close 6537 (time 1746155845.86 total 70.05 last 99.29) - open/close 7498 (time 1746155855.87 total 80.06 last 96.05) - open/close 8420 (time 1746155865.87 total 90.06 last 92.19) - open/close 9568 (time 1746155875.87 total 100.06 last 114.75) total: 10000 open/close in 104.01 seconds: 96.15 ops/second - unlinked 0 (time 1746155882 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 1001 (time 1746155962.82 total 10.00 last 100.07) - open/close 2046 (time 1746155972.82 total 20.01 last 104.45) - open/close 3011 (time 1746155982.85 total 30.03 last 96.26) - open/close 3851 (time 1746155992.86 total 40.04 last 83.94) - open/close 4739 (time 1746156002.86 total 50.04 last 88.77) - open/close 5612 (time 1746156012.86 total 60.04 last 87.28) - open/close 6569 (time 1746156022.86 total 70.05 last 95.67) - open/close 7555 (time 1746156032.87 total 80.05 last 98.55) - open/close 8524 (time 1746156042.88 total 90.06 last 96.78) - open/close 9582 (time 1746156052.89 total 100.07 last 105.73) total: 10000 open/close in 104.75 seconds: 95.46 ops/second - unlinked 0 (time 1746156060 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 890 (time 1746156137.51 total 10.00 last 88.96) - open/close 1879 (time 1746156147.52 total 20.02 last 98.78) - open/close 2898 (time 1746156157.52 total 30.02 last 101.89) - open/close 4005 (time 1746156167.53 total 40.03 last 110.58) - open/close 4843 (time 1746156177.53 total 50.03 last 83.78) - open/close 5971 (time 1746156187.53 total 60.03 last 112.80) - open/close 7213 (time 1746156197.55 total 70.04 last 124.04) - open/close 8264 (time 1746156207.55 total 80.05 last 105.03) - open/close 9270 (time 1746156217.55 total 90.05 last 100.59) total: 10000 open/close in 96.59 seconds: 103.53 ops/second - unlinked 0 (time 1746156226 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1114 (time 1746156290.54 total 10.00 last 111.35) - open/close 2356 (time 1746156300.55 total 20.02 last 123.99) - open/close 3588 (time 1746156310.55 total 30.02 last 123.19) - open/close 4792 (time 1746156320.56 total 40.03 last 120.36) - open/close 5756 (time 1746156330.57 total 50.04 last 96.28) - open/close 6779 (time 1746156340.57 total 60.04 last 102.26) - open/close 7903 (time 1746156350.58 total 70.05 last 112.36) - open/close 8989 (time 1746156360.58 total 80.05 last 108.60) total: 10000 open/close in 89.62 seconds: 111.58 ops/second - unlinked 0 (time 1746156372 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 936 (time 1746156444.90 total 10.01 last 93.55) - open/close 1870 (time 1746156454.90 total 20.01 last 93.38) - open/close 2745 (time 1746156464.90 total 30.01 last 87.48) - open/close 3685 (time 1746156474.90 total 40.01 last 93.99) - open/close 4601 (time 1746156484.91 total 50.02 last 91.56) - open/close 5532 (time 1746156494.91 total 60.02 last 93.03) - open/close 6473 (time 1746156504.92 total 70.02 last 94.08) - open/close 7432 (time 1746156514.92 total 80.03 last 95.87) - open/close 8380 (time 1746156524.92 total 90.03 last 94.75) - open/close 9306 (time 1746156534.93 total 100.04 last 92.57) total: 10000 open/close in 107.81 seconds: 92.75 ops/second - unlinked 0 (time 1746156545 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 794 (time 1746156621.46 total 10.01 last 79.34) - open/close 1715 (time 1746156631.48 total 20.02 last 91.97) - open/close 2623 (time 1746156641.48 total 30.02 last 90.78) - open/close 3501 (time 1746156651.50 total 40.04 last 87.62) - open/close 4502 (time 1746156661.53 total 50.07 last 99.82) - open/close 5507 (time 1746156671.53 total 60.07 last 100.47) - open/close 6490 (time 1746156681.54 total 70.08 last 98.23) - open/close 7280 (time 1746156691.55 total 80.10 last 78.89) - open/close 8165 (time 1746156701.56 total 90.10 last 88.45) - open/close 9082 (time 1746156711.57 total 100.11 last 91.62) - open/close 9996 (time 1746156721.57 total 110.11 last 91.39) total: 10000 open/close in 110.14 seconds: 90.79 ops/second - unlinked 0 (time 1746156724 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 864 (time 1746156810.14 total 10.01 last 86.35) - open/close 1809 (time 1746156820.15 total 20.01 last 94.44) - open/close 2566 (time 1746156830.15 total 30.01 last 75.69) - open/close 3453 (time 1746156840.15 total 40.02 last 88.68) - open/close 4255 (time 1746156850.16 total 50.03 last 80.11) - open/close 5088 (time 1746156860.17 total 60.03 last 83.23) - open/close 5882 (time 1746156870.18 total 70.04 last 79.32) - open/close 6691 (time 1746156880.18 total 80.05 last 80.87) - open/close 7517 (time 1746156890.20 total 90.06 last 82.48) - open/close 8404 (time 1746156900.21 total 100.08 last 88.56) - open/close 9288 (time 1746156910.21 total 110.08 last 88.40) total: 10000 open/close in 118.58 seconds: 84.33 ops/second - unlinked 0 (time 1746156921 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg127-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg127-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg127-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.127@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg127-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg127-server: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg127-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg127-server: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg127-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg127-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg127-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg127-server stop mds service on oleg127-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg127-server stop mds service on oleg127-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg127-server unloading modules via unload_modules_local on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg127-client: oleg127-client: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-client: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2