== conf-sanity test 69: replace an OST with the same index ========================================================== 16:25:34 (1746131134) start mds service on oleg355-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing load_modules_local oleg355-server: Loading modules from /home/green/git/lustre-release/lustre oleg355-server: detected 4 online CPUs by sysfs oleg355-server: MODOPTS_LIBCFS= oleg355-server: Force libcfs to create 2 CPU partitions oleg355-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg355-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg355-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg355-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg355-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1047 (time 1746131199.69 total 10.01 last 104.58) - open/close 2298 (time 1746131209.69 total 20.01 last 125.09) - open/close 3437 (time 1746131219.70 total 30.02 last 113.83) - open/close 4610 (time 1746131229.70 total 40.03 last 117.21) - open/close 5646 (time 1746131239.71 total 50.03 last 103.52) - open/close 7002 (time 1746131249.71 total 60.04 last 135.56) - open/close 8000 (time 1746131259.72 total 70.04 last 99.79) - open/close 9011 (time 1746131269.72 total 80.04 last 101.07) total: 10000 open/close in 88.43 seconds: 113.08 ops/second - unlinked 0 (time 1746131280 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1203 (time 1746131344.38 total 10.01 last 120.22) - open/close 2372 (time 1746131354.38 total 20.01 last 116.87) - open/close 3521 (time 1746131364.38 total 30.01 last 114.88) - open/close 4717 (time 1746131374.39 total 40.01 last 119.56) - open/close 5859 (time 1746131384.39 total 50.02 last 114.16) - open/close 6878 (time 1746131394.40 total 60.03 last 101.81) - open/close 7934 (time 1746131404.40 total 70.03 last 105.53) - open/close 9065 (time 1746131414.41 total 80.04 last 113.06) total: 10000 open/close in 89.55 seconds: 111.67 ops/second - unlinked 0 (time 1746131426 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1177 (time 1746131488.76 total 10.01 last 117.57) - open/close 2491 (time 1746131498.76 total 20.01 last 131.35) - open/close 3666 (time 1746131508.76 total 30.02 last 117.45) - open/close 4777 (time 1746131518.78 total 40.03 last 110.97) - open/close 5899 (time 1746131528.78 total 50.03 last 112.18) - open/close 7074 (time 1746131538.79 total 60.05 last 117.32) - open/close 8275 (time 1746131548.80 total 70.05 last 120.02) - open/close 9435 (time 1746131558.80 total 80.06 last 115.96) total: 10000 open/close in 85.30 seconds: 117.23 ops/second - unlinked 0 (time 1746131566 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1154 (time 1746131629.22 total 10.01 last 115.32) - open/close 2089 (time 1746131639.23 total 20.02 last 93.38) - open/close 3185 (time 1746131649.26 total 30.05 last 109.26) - open/close 4276 (time 1746131659.26 total 40.05 last 109.10) - open/close 5351 (time 1746131669.27 total 50.06 last 107.44) - open/close 6501 (time 1746131679.28 total 60.07 last 114.87) - open/close 7464 (time 1746131689.29 total 70.08 last 96.14) - open/close 8552 (time 1746131699.30 total 80.09 last 108.78) - open/close 9759 (time 1746131709.30 total 90.09 last 120.64) total: 10000 open/close in 92.01 seconds: 108.69 ops/second - unlinked 0 (time 1746131713 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1160 (time 1746131775.07 total 10.00 last 115.95) - open/close 2370 (time 1746131785.08 total 20.02 last 120.86) - open/close 3630 (time 1746131795.08 total 30.02 last 125.92) - open/close 4883 (time 1746131805.08 total 40.02 last 125.30) - open/close 6146 (time 1746131815.09 total 50.02 last 126.27) - open/close 7142 (time 1746131825.10 total 60.04 last 99.44) - open/close 8185 (time 1746131835.12 total 70.06 last 104.12) - open/close 9426 (time 1746131845.13 total 80.06 last 124.02) total: 10000 open/close in 85.21 seconds: 117.36 ops/second - unlinked 0 (time 1746131852 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1314 (time 1746131911.18 total 10.00 last 131.37) - open/close 2721 (time 1746131921.19 total 20.01 last 140.56) - open/close 4018 (time 1746131931.19 total 30.02 last 129.58) - open/close 5257 (time 1746131941.20 total 40.02 last 123.86) - open/close 6386 (time 1746131951.21 total 50.03 last 112.79) - open/close 7592 (time 1746131961.21 total 60.04 last 120.58) - open/close 8855 (time 1746131971.21 total 70.04 last 126.27) - open/close 9984 (time 1746131981.21 total 80.04 last 112.89) total: 10000 open/close in 80.14 seconds: 124.78 ops/second - unlinked 0 (time 1746131983 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1197 (time 1746132044.69 total 10.02 last 119.42) - open/close 2331 (time 1746132054.70 total 20.03 last 113.29) - open/close 3443 (time 1746132064.70 total 30.04 last 111.11) - open/close 4457 (time 1746132074.71 total 40.04 last 101.38) - open/close 5725 (time 1746132084.72 total 50.05 last 126.68) - open/close 7032 (time 1746132094.72 total 60.06 last 130.66) - open/close 8204 (time 1746132104.72 total 70.06 last 117.19) - open/close 9344 (time 1746132114.72 total 80.06 last 113.96) total: 10000 open/close in 86.53 seconds: 115.57 ops/second - unlinked 0 (time 1746132122 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1244 (time 1746132184.89 total 10.00 last 124.35) - open/close 2536 (time 1746132194.89 total 20.01 last 129.18) - open/close 3745 (time 1746132204.89 total 30.01 last 120.88) - open/close 4983 (time 1746132214.90 total 40.01 last 123.74) - open/close 6262 (time 1746132224.90 total 50.02 last 127.85) - open/close 7586 (time 1746132234.91 total 60.02 last 132.28) - open/close 8809 (time 1746132244.92 total 70.03 last 122.22) - open/close 9948 (time 1746132254.92 total 80.03 last 113.88) total: 10000 open/close in 80.47 seconds: 124.27 ops/second - unlinked 0 (time 1746132257 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1271 (time 1746132310.42 total 10.00 last 127.06) - open/close 2707 (time 1746132320.42 total 20.00 last 143.59) - open/close 3994 (time 1746132330.43 total 30.01 last 128.63) - open/close 5312 (time 1746132340.43 total 40.01 last 131.74) - open/close 6630 (time 1746132350.43 total 50.01 last 131.79) - open/close 8019 (time 1746132360.44 total 60.02 last 138.76) - open/close 9184 (time 1746132370.44 total 70.02 last 116.49) total: 10000 open/close in 77.40 seconds: 129.21 ops/second - unlinked 0 (time 1746132379 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1142 (time 1746132438.25 total 10.01 last 114.10) - open/close 2150 (time 1746132448.26 total 20.01 last 100.77) - open/close 3212 (time 1746132458.26 total 30.01 last 106.17) - open/close 4350 (time 1746132468.26 total 40.02 last 113.75) - open/close 5235 (time 1746132478.27 total 50.03 last 88.42) - open/close 6177 (time 1746132488.27 total 60.03 last 94.19) - open/close 7323 (time 1746132498.28 total 70.04 last 114.51) - open/close 8790 (time 1746132508.29 total 80.04 last 146.63) - open/close 9962 (time 1746132518.30 total 90.05 last 117.07) total: 10000 open/close in 90.34 seconds: 110.69 ops/second - unlinked 0 (time 1746132520 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg355-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg355-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg355-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.155@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg355-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg355-server: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg355-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg355-server: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg355-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre On OST0, 17883 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg355-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg355-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg355-server unloading modules via unload_modules_local on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2