== conf-sanity test 69: replace an OST with the same index ========================================================== 20:51:51 (1746147111) start mds service on oleg123-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing load_modules_local oleg123-server: Loading modules from /home/green/git/lustre-release/lustre oleg123-server: detected 4 online CPUs by sysfs oleg123-server: MODOPTS_LIBCFS= oleg123-server: Force libcfs to create 2 CPU partitions oleg123-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg123-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg123-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg123-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 879 (time 1746147180.22 total 10.00 last 87.87) - open/close 1612 (time 1746147190.23 total 20.01 last 73.24) - open/close 2451 (time 1746147200.24 total 30.02 last 83.84) - open/close 3280 (time 1746147210.24 total 40.02 last 82.85) - open/close 4198 (time 1746147220.25 total 50.03 last 91.73) - open/close 5105 (time 1746147230.26 total 60.04 last 90.67) - open/close 5958 (time 1746147240.26 total 70.04 last 85.28) - open/close 6753 (time 1746147250.27 total 80.05 last 79.44) - open/close 7589 (time 1746147260.27 total 90.05 last 83.54) - open/close 8391 (time 1746147270.28 total 100.06 last 80.16) - open/close 9151 (time 1746147280.28 total 110.06 last 75.95) - open/close 9931 (time 1746147290.29 total 120.07 last 77.96) total: 10000 open/close in 120.94 seconds: 82.69 ops/second - unlinked 0 (time 1746147294 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 848 (time 1746147382.62 total 10.00 last 84.80) - open/close 1785 (time 1746147392.63 total 20.01 last 93.64) - open/close 2732 (time 1746147402.65 total 30.03 last 94.44) - open/close 3619 (time 1746147412.67 total 40.05 last 88.56) - open/close 4525 (time 1746147422.67 total 50.05 last 90.57) - open/close 5320 (time 1746147432.68 total 60.06 last 79.45) - open/close 6138 (time 1746147442.68 total 70.06 last 81.80) - open/close 6954 (time 1746147452.69 total 80.07 last 81.55) - open/close 7769 (time 1746147462.70 total 90.08 last 81.36) - open/close 8508 (time 1746147472.72 total 100.10 last 73.80) - open/close 9336 (time 1746147482.73 total 110.11 last 82.71) total: 10000 open/close in 118.46 seconds: 84.41 ops/second - unlinked 0 (time 1746147493 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 842 (time 1746147574.98 total 10.01 last 84.11) - open/close 1754 (time 1746147584.99 total 20.02 last 91.14) - open/close 2706 (time 1746147594.99 total 30.02 last 95.20) - open/close 3663 (time 1746147605.00 total 40.03 last 95.57) - open/close 4512 (time 1746147615.00 total 50.03 last 84.89) - open/close 5487 (time 1746147625.01 total 60.04 last 97.47) - open/close 6499 (time 1746147635.01 total 70.04 last 101.18) - open/close 7507 (time 1746147645.01 total 80.04 last 100.78) - open/close 8564 (time 1746147655.02 total 90.05 last 105.55) - open/close 9675 (time 1746147665.03 total 100.06 last 110.99) total: 10000 open/close in 103.04 seconds: 97.05 ops/second - unlinked 0 (time 1746147670 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 962 (time 1746147744.58 total 10.01 last 96.11) - open/close 1897 (time 1746147754.59 total 20.01 last 93.49) - open/close 2875 (time 1746147764.60 total 30.02 last 97.69) - open/close 3814 (time 1746147774.61 total 40.03 last 93.80) - open/close 4835 (time 1746147784.61 total 50.04 last 102.05) - open/close 5781 (time 1746147794.62 total 60.04 last 94.56) - open/close 6758 (time 1746147804.63 total 70.05 last 97.59) - open/close 7772 (time 1746147814.63 total 80.06 last 101.35) - open/close 8754 (time 1746147824.64 total 90.06 last 98.15) - open/close 9626 (time 1746147834.64 total 100.06 last 87.20) total: 10000 open/close in 103.92 seconds: 96.23 ops/second - unlinked 0 (time 1746147841 ; total 1 ; last 1) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1043 (time 1746147918.16 total 10.01 last 104.21) - open/close 2027 (time 1746147928.16 total 20.01 last 98.39) - open/close 3115 (time 1746147938.18 total 30.02 last 108.67) - open/close 4049 (time 1746147948.19 total 40.03 last 93.28) - open/close 4890 (time 1746147958.20 total 50.04 last 84.04) - open/close 5717 (time 1746147968.21 total 60.06 last 82.56) - open/close 6556 (time 1746147978.21 total 70.06 last 83.89) - open/close 7506 (time 1746147988.21 total 80.06 last 94.99) - open/close 8379 (time 1746147998.22 total 90.07 last 87.26) - open/close 9172 (time 1746148008.23 total 100.07 last 79.24) total: 10000 open/close in 109.26 seconds: 91.53 ops/second - unlinked 0 (time 1746148019 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 902 (time 1746148094.64 total 10.01 last 90.08) - open/close 1721 (time 1746148104.64 total 20.01 last 81.89) - open/close 2525 (time 1746148114.65 total 30.02 last 80.37) - open/close 3339 (time 1746148124.65 total 40.02 last 81.36) - open/close 4249 (time 1746148134.65 total 50.02 last 90.99) - open/close 4993 (time 1746148144.65 total 60.03 last 74.39) - open/close 5743 (time 1746148154.66 total 70.03 last 74.95) - open/close 6459 (time 1746148164.66 total 80.04 last 71.58) - open/close 7333 (time 1746148174.67 total 90.04 last 87.35) - open/close 8299 (time 1746148184.68 total 100.05 last 96.47) - open/close 9197 (time 1746148194.70 total 110.07 last 89.63) - open/close 9984 (time 1746148204.71 total 120.09 last 78.61) total: 10000 open/close in 120.28 seconds: 83.14 ops/second - unlinked 0 (time 1746148207 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 995 (time 1746148291.17 total 10.00 last 99.45) - open/close 1971 (time 1746148301.17 total 20.01 last 97.56) - open/close 2827 (time 1746148311.17 total 30.01 last 85.59) - open/close 3641 (time 1746148321.17 total 40.01 last 81.38) - open/close 4542 (time 1746148331.18 total 50.02 last 90.05) - open/close 5569 (time 1746148341.18 total 60.02 last 102.65) - open/close 6440 (time 1746148351.19 total 70.03 last 87.02) - open/close 7387 (time 1746148361.19 total 80.03 last 94.68) - open/close 8259 (time 1746148371.20 total 90.04 last 87.15) - open/close 9241 (time 1746148381.20 total 100.04 last 98.20) total: 10000 open/close in 109.33 seconds: 91.47 ops/second - unlinked 0 (time 1746148393 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 928 (time 1746148478.32 total 10.00 last 92.78) - open/close 1923 (time 1746148488.32 total 20.00 last 99.49) - open/close 2817 (time 1746148498.33 total 30.01 last 89.33) - open/close 3724 (time 1746148508.33 total 40.01 last 90.69) - open/close 4486 (time 1746148518.34 total 50.02 last 76.14) - open/close 5357 (time 1746148528.34 total 60.03 last 87.05) - open/close 6294 (time 1746148538.35 total 70.03 last 93.62) - open/close 7006 (time 1746148548.35 total 80.04 last 71.17) - open/close 7770 (time 1746148558.36 total 90.04 last 76.38) - open/close 8555 (time 1746148568.36 total 100.04 last 78.47) - open/close 9411 (time 1746148578.37 total 110.05 last 85.53) total: 10000 open/close in 115.61 seconds: 86.50 ops/second - unlinked 0 (time 1746148585 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 775 (time 1746148668.76 total 10.01 last 77.43) - open/close 1666 (time 1746148678.76 total 20.01 last 89.06) - open/close 2557 (time 1746148688.77 total 30.02 last 89.06) - open/close 3477 (time 1746148698.77 total 40.02 last 91.94) - open/close 4290 (time 1746148708.78 total 50.03 last 81.29) - open/close 5129 (time 1746148718.80 total 60.05 last 83.73) - open/close 5940 (time 1746148728.80 total 70.05 last 81.04) - open/close 6869 (time 1746148738.81 total 80.06 last 92.80) - open/close 7545 (time 1746148748.81 total 90.06 last 67.60) - open/close 8253 (time 1746148758.82 total 100.07 last 70.74) - open/close 8960 (time 1746148768.82 total 110.07 last 70.69) - open/close 9716 (time 1746148778.83 total 120.08 last 75.56) total: 10000 open/close in 123.73 seconds: 80.82 ops/second - unlinked 0 (time 1746148785 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 902 (time 1746148869.22 total 10.00 last 90.17) - open/close 1648 (time 1746148879.23 total 20.01 last 74.54) - open/close 2308 (time 1746148889.23 total 30.01 last 65.98) - open/close 3104 (time 1746148899.25 total 40.04 last 79.41) - open/close 3919 (time 1746148909.26 total 50.04 last 81.48) - open/close 4876 (time 1746148919.27 total 60.05 last 95.60) - open/close 5735 (time 1746148929.28 total 70.06 last 85.81) - open/close 6541 (time 1746148939.28 total 80.07 last 80.56) - open/close 7358 (time 1746148949.28 total 90.07 last 81.68) - open/close 8174 (time 1746148959.30 total 100.08 last 81.51) - open/close 9047 (time 1746148969.30 total 110.09 last 87.23) - open/close 9968 (time 1746148979.31 total 120.09 last 92.05) total: 10000 open/close in 120.49 seconds: 82.99 ops/second - unlinked 0 (time 1746148982 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg123-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg123-server: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg123-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg123-server: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg123-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre On OST0, 13388 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg123-server unloading modules via unload_modules_local on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2