== conf-sanity test 69: replace an OST with the same index ========================================================== 19:39:49 (1746142789) start mds service on oleg314-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing load_modules_local oleg314-server: Loading modules from /home/green/git/lustre-release/lustre oleg314-server: detected 4 online CPUs by sysfs oleg314-server: MODOPTS_LIBCFS= oleg314-server: Force libcfs to create 2 CPU partitions oleg314-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg314-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg314-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg314-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1110 (time 1746142849.44 total 10.00 last 110.97) - open/close 2331 (time 1746142859.45 total 20.01 last 122.02) - open/close 3450 (time 1746142869.45 total 30.01 last 111.86) - open/close 4666 (time 1746142879.46 total 40.02 last 121.53) - open/close 5798 (time 1746142889.47 total 50.02 last 113.13) - open/close 7043 (time 1746142899.47 total 60.03 last 124.46) - open/close 8181 (time 1746142909.47 total 70.03 last 113.75) - open/close 9528 (time 1746142919.48 total 80.04 last 134.63) total: 10000 open/close in 84.16 seconds: 118.82 ops/second - unlinked 0 (time 1746142925 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1116 (time 1746142985.78 total 10.00 last 111.56) - open/close 2306 (time 1746142995.78 total 20.01 last 118.98) - open/close 3537 (time 1746143005.78 total 30.01 last 123.09) - open/close 4858 (time 1746143015.79 total 40.01 last 132.03) - open/close 6419 (time 1746143025.79 total 50.01 last 156.08) - open/close 7688 (time 1746143035.79 total 60.01 last 126.90) - open/close 9052 (time 1746143045.79 total 70.02 last 136.32) total: 10000 open/close in 77.05 seconds: 129.79 ops/second - unlinked 0 (time 1746143054 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1043 (time 1746143122.40 total 10.00 last 104.26) - open/close 2156 (time 1746143132.41 total 20.01 last 111.25) - open/close 3462 (time 1746143142.41 total 30.01 last 130.59) - open/close 4878 (time 1746143152.41 total 40.01 last 141.57) - open/close 6359 (time 1746143162.41 total 50.01 last 148.09) - open/close 7393 (time 1746143172.41 total 60.02 last 103.37) - open/close 8476 (time 1746143182.42 total 70.02 last 108.22) - open/close 9500 (time 1746143192.43 total 80.03 last 102.35) total: 10000 open/close in 84.64 seconds: 118.15 ops/second - unlinked 0 (time 1746143199 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1048 (time 1746143266.92 total 10.01 last 104.68) - open/close 2065 (time 1746143276.92 total 20.01 last 101.69) - open/close 3282 (time 1746143286.93 total 30.02 last 121.67) - open/close 4315 (time 1746143296.94 total 40.02 last 103.21) - open/close 5369 (time 1746143306.94 total 50.03 last 105.34) - open/close 6543 (time 1746143316.95 total 60.03 last 117.34) - open/close 7564 (time 1746143326.95 total 70.04 last 102.03) - open/close 8688 (time 1746143336.96 total 80.05 last 112.31) - open/close 9745 (time 1746143346.97 total 90.05 last 105.66) total: 10000 open/close in 92.09 seconds: 108.59 ops/second - unlinked 0 (time 1746143350 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1101 (time 1746143411.37 total 10.00 last 110.09) - open/close 2395 (time 1746143421.37 total 20.01 last 129.34) - open/close 3759 (time 1746143431.38 total 30.01 last 136.37) - open/close 4857 (time 1746143441.38 total 40.01 last 109.73) - open/close 6018 (time 1746143451.39 total 50.02 last 116.01) - open/close 7297 (time 1746143461.40 total 60.03 last 127.84) - open/close 8664 (time 1746143471.40 total 70.03 last 136.65) - open/close 9940 (time 1746143481.40 total 80.03 last 127.58) total: 10000 open/close in 80.52 seconds: 124.19 ops/second - unlinked 0 (time 1746143483 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1369 (time 1746143541.73 total 10.00 last 136.86) - open/close 2696 (time 1746143551.74 total 20.01 last 132.62) - open/close 3934 (time 1746143561.74 total 30.01 last 123.79) - open/close 5197 (time 1746143571.74 total 40.02 last 126.23) - open/close 6587 (time 1746143581.75 total 50.02 last 138.96) - open/close 7780 (time 1746143591.75 total 60.02 last 119.23) - open/close 9383 (time 1746143601.75 total 70.03 last 160.28) total: 10000 open/close in 74.16 seconds: 134.84 ops/second - unlinked 0 (time 1746143607 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1431 (time 1746143659.49 total 10.01 last 143.01) - open/close 2857 (time 1746143669.50 total 20.01 last 142.51) - open/close 4493 (time 1746143679.50 total 30.01 last 163.58) - open/close 6026 (time 1746143689.50 total 40.02 last 153.27) - open/close 7234 (time 1746143699.51 total 50.02 last 120.74) - open/close 8552 (time 1746143709.51 total 60.03 last 131.75) - open/close 9921 (time 1746143719.51 total 70.03 last 136.90) total: 10000 open/close in 70.50 seconds: 141.84 ops/second - unlinked 0 (time 1746143721 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1609 (time 1746143772.78 total 10.00 last 160.83) - open/close 3183 (time 1746143782.78 total 20.01 last 157.35) - open/close 5024 (time 1746143792.78 total 30.01 last 184.05) - open/close 6912 (time 1746143802.79 total 40.02 last 188.70) - open/close 8666 (time 1746143812.79 total 50.02 last 175.36) - open/close 9909 (time 1746143822.79 total 60.02 last 124.27) total: 10000 open/close in 60.82 seconds: 164.43 ops/second - unlinked 0 (time 1746143825 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1098 (time 1746143881.72 total 10.01 last 109.74) - open/close 2240 (time 1746143891.73 total 20.01 last 114.16) - open/close 3605 (time 1746143901.73 total 30.01 last 136.45) - open/close 4935 (time 1746143911.73 total 40.01 last 132.97) - open/close 6127 (time 1746143921.74 total 50.02 last 119.14) - open/close 7307 (time 1746143931.75 total 60.03 last 117.88) - open/close 8623 (time 1746143941.75 total 70.03 last 131.60) - open/close 9711 (time 1746143951.75 total 80.03 last 108.80) total: 10000 open/close in 82.70 seconds: 120.92 ops/second - unlinked 0 (time 1746143956 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1591 (time 1746144013.87 total 10.02 last 158.81) - open/close 3020 (time 1746144023.89 total 20.03 last 142.68) - open/close 4796 (time 1746144033.89 total 30.04 last 177.53) - open/close 6655 (time 1746144043.90 total 40.04 last 185.84) - open/close 9041 (time 1746144053.90 total 50.04 last 238.59) total: 10000 open/close in 53.54 seconds: 186.77 ops/second - unlinked 0 (time 1746144058 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg314-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre On OST0, 14797 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg314-server unloading modules via unload_modules_local on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2