== conf-sanity test 69: replace an OST with the same index ========================================================== 16:02:25 (1710100945) start mds service on oleg341-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg341-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg341-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock oleg341-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 1185 (time 1710100986.77 total 10.00 last 118.49) - open/close 2259 (time 1710100996.77 total 20.00 last 107.40) - open/close 3446 (time 1710101006.77 total 30.00 last 118.69) - open/close 4452 (time 1710101016.78 total 40.00 last 100.59) - open/close 5396 (time 1710101026.78 total 50.00 last 94.38) - open/close 6326 (time 1710101036.78 total 60.01 last 92.97) - open/close 7230 (time 1710101046.79 total 70.02 last 90.32) - open/close 7903 (time 1710101066.48 total 89.70 last 34.19) - open/close 8707 (time 1710101076.49 total 99.71 last 80.32) - open/close 9502 (time 1710101086.50 total 109.72 last 79.43) total: 10000 open/close in 117.39 seconds: 85.18 ops/second - unlinked 0 (time 1710101096 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 613 (time 1710101193.39 total 16.68 last 36.76) - open/close 1653 (time 1710101203.40 total 26.69 last 103.92) - open/close 2684 (time 1710101213.40 total 36.69 last 103.04) - open/close 3663 (time 1710101223.41 total 46.70 last 97.85) - open/close 4612 (time 1710101233.41 total 56.70 last 94.87) - open/close 5518 (time 1710101243.42 total 66.71 last 90.54) - open/close 6400 (time 1710101253.43 total 76.71 last 88.15) - open/close 7217 (time 1710101263.43 total 86.72 last 81.65) - open/close 7968 (time 1710101273.43 total 96.72 last 75.09) - open/close 8690 (time 1710101283.44 total 106.72 last 72.16) - open/close 8812 (time 1710101296.20 total 119.49 last 9.56) - open/close 9527 (time 1710101306.21 total 129.49 last 71.45) total: 10000 open/close in 136.44 seconds: 73.29 ops/second - unlinked 0 (time 1710101314 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 1197 (time 1710101401.49 total 10.00 last 119.69) - open/close 2306 (time 1710101411.50 total 20.01 last 110.83) - open/close 3341 (time 1710101421.51 total 30.02 last 103.42) - open/close 4335 (time 1710101431.51 total 40.02 last 99.35) - open/close 5250 (time 1710101441.52 total 50.03 last 91.43) - open/close 6150 (time 1710101451.53 total 60.03 last 89.94) - open/close 7008 (time 1710101461.54 total 70.05 last 85.68) - open/close 7794 (time 1710101471.55 total 80.06 last 78.53) - open/close 8528 (time 1710101481.56 total 90.07 last 73.31) - open/close 9219 (time 1710101491.56 total 100.07 last 69.08) - open/close 9891 (time 1710101501.57 total 110.08 last 67.13) total: 10000 open/close in 111.71 seconds: 89.52 ops/second - unlinked 0 (time 1710101504 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 1206 (time 1710101591.63 total 10.00 last 120.55) - open/close 2327 (time 1710101601.64 total 20.01 last 112.02) - open/close 3354 (time 1710101611.65 total 30.02 last 102.65) - open/close 4314 (time 1710101621.65 total 40.02 last 95.99) - open/close 5243 (time 1710101631.65 total 50.02 last 92.84) - open/close 6108 (time 1710101641.66 total 60.03 last 86.41) - open/close 6935 (time 1710101651.67 total 70.04 last 82.65) - open/close 7694 (time 1710101661.68 total 80.05 last 75.82) - open/close 8424 (time 1710101671.69 total 90.06 last 72.94) - open/close 9164 (time 1710101681.71 total 100.08 last 73.86) - open/close 9827 (time 1710101691.71 total 110.08 last 66.25) total: 10000 open/close in 112.62 seconds: 88.80 ops/second - unlinked 0 (time 1710101695 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 1277 (time 1710101781.63 total 10.03 last 127.35) - open/close 2394 (time 1710101791.64 total 20.03 last 111.69) - open/close 3483 (time 1710101801.64 total 30.03 last 108.85) - open/close 4484 (time 1710101811.65 total 40.04 last 100.01) - open/close 5381 (time 1710101821.66 total 50.05 last 89.62) - open/close 6252 (time 1710101831.66 total 60.06 last 87.06) - open/close 7106 (time 1710101841.66 total 70.06 last 85.40) - open/close 7885 (time 1710101851.67 total 80.07 last 77.82) - open/close 8655 (time 1710101861.69 total 90.08 last 76.90) - open/close 9359 (time 1710101871.69 total 100.09 last 70.36) - open/close 9983 (time 1710101881.70 total 110.09 last 62.35) total: 10000 open/close in 110.31 seconds: 90.65 ops/second - unlinked 0 (time 1710101884 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 1242 (time 1710101974.06 total 10.00 last 124.14) - open/close 2359 (time 1710101984.07 total 20.01 last 111.68) - open/close 3441 (time 1710101994.07 total 30.01 last 108.15) - open/close 4429 (time 1710102004.07 total 40.01 last 98.77) - open/close 5394 (time 1710102014.08 total 50.02 last 96.47) - open/close 6274 (time 1710102024.08 total 60.02 last 87.99) - open/close 7135 (time 1710102034.09 total 70.03 last 86.05) - open/close 7929 (time 1710102044.09 total 80.03 last 79.40) - open/close 8682 (time 1710102054.10 total 90.04 last 75.22) - open/close 9408 (time 1710102064.11 total 100.05 last 72.50) total: 10000 open/close in 108.99 seconds: 91.75 ops/second - unlinked 0 (time 1710102074 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 1319 (time 1710102160.64 total 10.00 last 131.87) - open/close 2521 (time 1710102170.65 total 20.01 last 120.14) - open/close 3671 (time 1710102180.65 total 30.01 last 114.98) - open/close 4776 (time 1710102190.66 total 40.01 last 110.44) - open/close 5833 (time 1710102200.66 total 50.02 last 105.64) - open/close 6786 (time 1710102210.67 total 60.03 last 95.20) - open/close 7697 (time 1710102220.68 total 70.04 last 91.01) - open/close 8546 (time 1710102230.69 total 80.05 last 84.82) - open/close 9368 (time 1710102240.70 total 90.05 last 82.18) total: 10000 open/close in 98.37 seconds: 101.65 ops/second - unlinked 0 (time 1710102250 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 1420 (time 1710102329.10 total 10.00 last 141.99) - open/close 2766 (time 1710102339.10 total 20.00 last 134.59) - open/close 3961 (time 1710102349.11 total 30.00 last 119.46) - open/close 5075 (time 1710102359.11 total 40.01 last 111.40) - open/close 6147 (time 1710102369.12 total 50.01 last 107.12) - open/close 7120 (time 1710102379.12 total 60.02 last 97.24) - open/close 8070 (time 1710102389.13 total 70.03 last 94.93) - open/close 8946 (time 1710102399.13 total 80.03 last 87.55) - open/close 9763 (time 1710102409.14 total 90.04 last 81.66) total: 10000 open/close in 92.84 seconds: 107.71 ops/second - unlinked 0 (time 1710102413 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1573 (time 1710102486.82 total 10.00 last 157.29) - open/close 3106 (time 1710102496.82 total 20.00 last 153.26) - open/close 4590 (time 1710102506.82 total 30.01 last 148.37) - open/close 5895 (time 1710102516.82 total 40.01 last 130.48) - open/close 7120 (time 1710102526.83 total 50.01 last 122.45) - open/close 8265 (time 1710102536.83 total 60.02 last 114.41) - open/close 9330 (time 1710102546.83 total 70.02 last 106.50) total: 10000 open/close in 75.98 seconds: 131.62 ops/second - unlinked 0 (time 1710102553 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 3179 (time 1710102602.49 total 10.00 last 317.78) - open/close 5606 (time 1710102612.49 total 20.00 last 242.68) - open/close 7580 (time 1710102622.49 total 30.01 last 197.38) - open/close 9294 (time 1710102632.50 total 40.01 last 171.32) total: 10000 open/close in 44.01 seconds: 227.21 ops/second - unlinked 0 (time 1710102637 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.141@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg341-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg341-server: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg341-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg341-server: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg341-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock oleg341-server@tcp:/lustre /mnt/lustre On OST0, 17287 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg341-server unloading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2