== conf-sanity test 69: replace an OST with the same index ========================================================== 15:57:33 (1710100653) start mds service on oleg308-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg308-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49602 inodes available. Want 99872. rc=0 - open/close 3967 (time 1710100679.97 total 10.00 last 396.68) - open/close 7090 (time 1710100689.97 total 20.00 last 312.29) - open/close 9726 (time 1710100699.97 total 30.00 last 263.56) total: 10000 open/close in 31.23 seconds: 320.22 ops/second - unlinked 0 (time 1710100702 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 4121 (time 1710100736.33 total 10.00 last 412.01) - open/close 6868 (time 1710100746.33 total 20.00 last 274.64) - open/close 8284 (time 1710100756.33 total 30.01 last 141.57) - open/close 9482 (time 1710100766.34 total 40.02 last 119.66) total: 10000 open/close in 44.63 seconds: 224.08 ops/second - unlinked 0 (time 1710100772 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 2548 (time 1710100818.85 total 10.00 last 254.78) - open/close 4224 (time 1710100828.86 total 20.01 last 167.51) - open/close 5729 (time 1710100838.87 total 30.01 last 150.38) - open/close 7202 (time 1710100848.87 total 40.02 last 147.28) - open/close 8345 (time 1710100858.87 total 50.02 last 114.30) - open/close 9380 (time 1710100868.88 total 60.02 last 103.43) total: 10000 open/close in 65.57 seconds: 152.51 ops/second - unlinked 0 (time 1710100875 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 2203 (time 1710100926.88 total 10.00 last 220.20) - open/close 3766 (time 1710100936.88 total 20.01 last 156.22) - open/close 5249 (time 1710100946.89 total 30.01 last 148.24) - open/close 6594 (time 1710100956.89 total 40.02 last 134.43) - open/close 7799 (time 1710100966.89 total 50.02 last 120.49) - open/close 8743 (time 1710100976.90 total 60.02 last 94.37) - open/close 9539 (time 1710100986.90 total 70.02 last 79.58) total: 10000 open/close in 75.74 seconds: 132.03 ops/second - unlinked 0 (time 1710100994 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 1415 (time 1710101064.69 total 10.00 last 141.48) - open/close 2726 (time 1710101074.69 total 20.00 last 131.08) - open/close 3977 (time 1710101084.69 total 30.01 last 125.04) - open/close 4990 (time 1710101094.70 total 40.01 last 101.29) - open/close 5956 (time 1710101104.70 total 50.01 last 96.60) - open/close 6867 (time 1710101114.71 total 60.02 last 91.01) - open/close 7778 (time 1710101124.71 total 70.02 last 91.10) - open/close 8556 (time 1710101134.74 total 80.05 last 77.56) - open/close 9329 (time 1710101144.74 total 90.05 last 77.30) total: 10000 open/close in 98.87 seconds: 101.14 ops/second - unlinked 0 (time 1710101155 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 1441 (time 1710101233.38 total 10.00 last 144.06) - open/close 2736 (time 1710101243.39 total 20.01 last 129.38) - open/close 3936 (time 1710101253.40 total 30.02 last 119.95) - open/close 5031 (time 1710101263.41 total 40.03 last 109.39) - open/close 6017 (time 1710101273.41 total 50.03 last 98.57) - open/close 6986 (time 1710101283.42 total 60.04 last 96.82) - open/close 7896 (time 1710101293.42 total 70.04 last 90.94) - open/close 8791 (time 1710101303.43 total 80.05 last 89.44) - open/close 9661 (time 1710101313.43 total 90.05 last 86.99) total: 10000 open/close in 94.30 seconds: 106.05 ops/second - unlinked 0 (time 1710101319 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1431 (time 1710101393.62 total 10.00 last 143.05) - open/close 2676 (time 1710101403.63 total 20.01 last 124.41) - open/close 3848 (time 1710101413.63 total 30.02 last 117.12) - open/close 4971 (time 1710101423.64 total 40.02 last 112.25) - open/close 6032 (time 1710101433.64 total 50.02 last 106.08) - open/close 7007 (time 1710101443.64 total 60.03 last 97.47) - open/close 7931 (time 1710101453.65 total 70.04 last 92.32) - open/close 8793 (time 1710101463.66 total 80.05 last 86.11) - open/close 9596 (time 1710101473.67 total 90.05 last 80.26) total: 10000 open/close in 95.79 seconds: 104.40 ops/second - unlinked 0 (time 1710101480 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 1376 (time 1710101557.90 total 10.00 last 137.55) - open/close 2659 (time 1710101567.91 total 20.01 last 128.24) - open/close 3865 (time 1710101577.91 total 30.01 last 120.54) - open/close 4979 (time 1710101587.92 total 40.02 last 111.35) - open/close 6022 (time 1710101597.92 total 50.02 last 104.26) - open/close 7014 (time 1710101607.96 total 60.06 last 98.83) - open/close 7949 (time 1710101617.96 total 70.06 last 93.45) - open/close 8863 (time 1710101627.97 total 80.07 last 91.33) - open/close 9754 (time 1710101637.97 total 90.07 last 89.09) total: 10000 open/close in 93.04 seconds: 107.48 ops/second - unlinked 0 (time 1710101642 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 1419 (time 1710101718.43 total 10.01 last 141.83) - open/close 2706 (time 1710101728.43 total 20.01 last 128.66) - open/close 3880 (time 1710101738.44 total 30.01 last 117.35) - open/close 4934 (time 1710101748.45 total 40.02 last 105.28) - open/close 6030 (time 1710101758.45 total 50.03 last 109.55) - open/close 7015 (time 1710101768.46 total 60.04 last 98.42) - open/close 7955 (time 1710101778.46 total 70.04 last 93.96) - open/close 8815 (time 1710101788.47 total 80.05 last 85.94) - open/close 9672 (time 1710101798.49 total 90.06 last 85.58) total: 10000 open/close in 93.95 seconds: 106.44 ops/second - unlinked 0 (time 1710101803 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1343 (time 1710101877.05 total 10.01 last 134.13) - open/close 2526 (time 1710101887.06 total 20.02 last 118.26) - open/close 3668 (time 1710101897.06 total 30.02 last 114.18) - open/close 4679 (time 1710101907.06 total 40.02 last 101.06) - open/close 5659 (time 1710101917.07 total 50.03 last 97.93) - open/close 6556 (time 1710101927.08 total 60.04 last 89.61) - open/close 7443 (time 1710101937.08 total 70.05 last 88.65) - open/close 7814 (time 1710101952.37 total 85.33 last 24.28) - open/close 8697 (time 1710101962.37 total 95.33 last 88.24) - open/close 9523 (time 1710101972.37 total 105.33 last 82.60) total: 10000 open/close in 111.15 seconds: 89.97 ops/second - unlinked 0 (time 1710101979 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg308-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg308-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre On OST0, 20225 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server unloading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2