== conf-sanity test 69: replace an OST with the same index ========================================================== 18:29:47 (1745620187) start mds service on oleg333-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing load_modules_local oleg333-server: Loading modules from /home/green/git/lustre-release/lustre oleg333-server: detected 4 online CPUs by sysfs oleg333-server: MODOPTS_LIBCFS= oleg333-server: Force libcfs to create 2 CPU partitions oleg333-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg333-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg333-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49536 inodes available. Want 99872. rc=0 - open/close 914 (time 1745620251.83 total 10.01 last 91.35) - open/close 1789 (time 1745620261.84 total 20.01 last 87.46) - open/close 2665 (time 1745620271.84 total 30.02 last 87.55) - open/close 3978 (time 1745620281.85 total 40.02 last 131.23) - open/close 5013 (time 1745620291.85 total 50.02 last 103.49) - open/close 5866 (time 1745620301.86 total 60.03 last 85.22) - open/close 6723 (time 1745620311.86 total 70.04 last 85.66) - open/close 7512 (time 1745620321.86 total 80.04 last 78.88) - open/close 8552 (time 1745620331.86 total 90.04 last 104.00) - open/close 9327 (time 1745620341.87 total 100.04 last 77.49) total: 10000 open/close in 107.87 seconds: 92.71 ops/second - unlinked 0 (time 1745620351 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 1546 (time 1745620426.65 total 10.00 last 154.56) - open/close 2789 (time 1745620436.65 total 20.00 last 124.28) - open/close 4060 (time 1745620446.65 total 30.00 last 127.09) - open/close 5479 (time 1745620456.65 total 40.01 last 141.88) - open/close 7538 (time 1745620466.66 total 50.01 last 205.77) - open/close 8921 (time 1745620476.67 total 60.02 last 138.23) - open/close 9901 (time 1745620486.67 total 70.02 last 97.97) total: 10000 open/close in 71.03 seconds: 140.78 ops/second - unlinked 0 (time 1745620489 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1520 (time 1745620542.43 total 10.01 last 151.91) - open/close 2435 (time 1745620552.44 total 20.02 last 91.36) - open/close 3268 (time 1745620562.45 total 30.03 last 83.24) - open/close 4094 (time 1745620572.46 total 40.04 last 82.54) - open/close 5509 (time 1745620582.46 total 50.04 last 141.43) - open/close 6631 (time 1745620592.47 total 60.05 last 112.13) - open/close 7821 (time 1745620602.47 total 70.05 last 118.92) - open/close 9060 (time 1745620612.48 total 80.06 last 123.86) total: 10000 open/close in 88.31 seconds: 113.24 ops/second - unlinked 0 (time 1745620622 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1483 (time 1745620688.14 total 10.01 last 148.19) - open/close 2817 (time 1745620698.15 total 20.01 last 133.39) - open/close 4367 (time 1745620708.15 total 30.01 last 154.96) - open/close 5687 (time 1745620718.15 total 40.01 last 131.99) - open/close 7112 (time 1745620728.15 total 50.01 last 142.47) - open/close 8414 (time 1745620738.16 total 60.02 last 130.10) - open/close 9687 (time 1745620748.16 total 70.02 last 127.26) total: 10000 open/close in 72.41 seconds: 138.09 ops/second - unlinked 0 (time 1745620752 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 1512 (time 1745620801.17 total 10.00 last 151.19) - open/close 2818 (time 1745620811.17 total 20.00 last 130.59) - open/close 4208 (time 1745620821.18 total 30.00 last 138.98) - open/close 5561 (time 1745620831.18 total 40.01 last 135.27) - open/close 7092 (time 1745620841.18 total 50.01 last 153.03) - open/close 8501 (time 1745620851.19 total 60.02 last 140.81) - open/close 9813 (time 1745620861.19 total 70.02 last 131.15) total: 10000 open/close in 71.30 seconds: 140.26 ops/second - unlinked 0 (time 1745620863 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1654 (time 1745620915.70 total 10.00 last 165.33) - open/close 3382 (time 1745620925.71 total 20.01 last 172.73) - open/close 5182 (time 1745620935.71 total 30.01 last 179.98) - open/close 7028 (time 1745620945.71 total 40.01 last 184.52) - open/close 8375 (time 1745620955.72 total 50.02 last 134.64) - open/close 9567 (time 1745620965.72 total 60.02 last 119.14) total: 10000 open/close in 63.38 seconds: 157.78 ops/second - unlinked 0 (time 1745620970 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 2058 (time 1745621011.51 total 10.01 last 205.69) - open/close 4217 (time 1745621021.53 total 20.02 last 215.61) - open/close 6454 (time 1745621031.53 total 30.02 last 223.69) - open/close 8529 (time 1745621041.53 total 40.02 last 207.47) total: 10000 open/close in 46.44 seconds: 215.34 ops/second - unlinked 0 (time 1745621049 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2631 (time 1745621079.26 total 10.00 last 263.02) - open/close 4633 (time 1745621089.26 total 20.01 last 200.10) - open/close 6477 (time 1745621099.27 total 30.01 last 184.36) - open/close 8480 (time 1745621109.27 total 40.01 last 200.23) total: 10000 open/close in 48.20 seconds: 207.46 ops/second - unlinked 0 (time 1745621118 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2140 (time 1745621154.98 total 10.00 last 214.00) - open/close 4138 (time 1745621164.99 total 20.01 last 199.69) - open/close 5707 (time 1745621175.00 total 30.02 last 156.73) - open/close 7312 (time 1745621185.00 total 40.02 last 160.43) - open/close 9136 (time 1745621195.00 total 50.02 last 182.38) total: 10000 open/close in 55.20 seconds: 181.16 ops/second - unlinked 0 (time 1745621201 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1631 (time 1745621250.90 total 10.01 last 162.99) - open/close 3314 (time 1745621260.90 total 20.01 last 168.24) - open/close 4804 (time 1745621270.90 total 30.01 last 148.96) - open/close 6008 (time 1745621280.90 total 40.01 last 120.39) - open/close 7235 (time 1745621290.90 total 50.01 last 122.69) - open/close 8642 (time 1745621300.92 total 60.03 last 140.53) total: 10000 open/close in 69.93 seconds: 143.00 ops/second - unlinked 0 (time 1745621312 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.133@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg333-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg333-server: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg333-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 12 sec oleg333-server: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg333-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre On OST0, 15293 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server unloading modules via unload_modules_local on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2