== conf-sanity test 69: replace an OST with the same index ========================================================== 16:02:58 (1710100978) start mds service on oleg339-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg339-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg339-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock oleg339-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 1314 (time 1710101014.78 total 10.01 last 131.29) - open/close 2422 (time 1710101024.78 total 20.01 last 110.79) - open/close 3500 (time 1710101034.79 total 30.02 last 107.74) - open/close 4537 (time 1710101044.79 total 40.02 last 103.63) - open/close 5460 (time 1710101054.81 total 50.04 last 92.18) - open/close 6319 (time 1710101064.81 total 60.04 last 85.87) - open/close 7203 (time 1710101074.81 total 70.04 last 88.37) - open/close 8057 (time 1710101084.83 total 80.06 last 85.27) - open/close 8794 (time 1710101094.84 total 90.07 last 73.61) - open/close 9488 (time 1710101104.85 total 100.08 last 69.34) total: 10000 open/close in 108.10 seconds: 92.51 ops/second - unlinked 0 (time 1710101114 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1088 (time 1710101203.45 total 10.00 last 108.80) - open/close 2123 (time 1710101213.52 total 20.07 last 102.79) - open/close 3132 (time 1710101223.52 total 30.07 last 100.89) - open/close 4139 (time 1710101233.52 total 40.08 last 100.63) - open/close 5094 (time 1710101243.53 total 50.09 last 95.42) - open/close 5979 (time 1710101253.54 total 60.10 last 88.40) - open/close 6803 (time 1710101263.55 total 70.10 last 82.36) - open/close 7555 (time 1710101273.55 total 80.10 last 75.20) - open/close 8292 (time 1710101283.56 total 90.11 last 73.63) - open/close 9008 (time 1710101293.57 total 100.12 last 71.52) - open/close 9748 (time 1710101303.58 total 110.13 last 73.94) total: 10000 open/close in 113.75 seconds: 87.91 ops/second - unlinked 0 (time 1710101308 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 1227 (time 1710101392.46 total 10.00 last 122.67) - open/close 2280 (time 1710101402.47 total 20.01 last 105.22) - open/close 3299 (time 1710101412.47 total 30.02 last 101.81) - open/close 4120 (time 1710101431.83 total 49.38 last 42.41) - open/close 5030 (time 1710101441.83 total 59.38 last 91.00) - open/close 5949 (time 1710101451.84 total 69.39 last 91.80) - open/close 6800 (time 1710101461.85 total 79.39 last 85.08) - open/close 7611 (time 1710101471.86 total 89.40 last 81.00) - open/close 8366 (time 1710101481.87 total 99.41 last 75.42) - open/close 9045 (time 1710101491.87 total 109.41 last 67.90) - open/close 9720 (time 1710101501.88 total 119.42 last 67.44) total: 10000 open/close in 123.48 seconds: 80.98 ops/second - unlinked 0 (time 1710101507 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 1184 (time 1710101594.23 total 10.00 last 118.39) - open/close 2337 (time 1710101604.24 total 20.01 last 115.21) - open/close 3364 (time 1710101614.24 total 30.01 last 102.67) - open/close 4337 (time 1710101624.25 total 40.02 last 97.25) - open/close 5296 (time 1710101634.25 total 50.02 last 95.88) - open/close 6189 (time 1710101644.27 total 60.04 last 89.13) - open/close 7015 (time 1710101654.28 total 70.05 last 82.52) - open/close 7773 (time 1710101664.28 total 80.05 last 75.77) - open/close 8525 (time 1710101674.30 total 90.07 last 75.08) - open/close 9279 (time 1710101684.31 total 100.08 last 75.29) - open/close 9437 (time 1710101697.75 total 113.52 last 11.75) total: 10000 open/close in 122.23 seconds: 81.82 ops/second - unlinked 0 (time 1710101708 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 1245 (time 1710101794.07 total 10.01 last 124.38) - open/close 2407 (time 1710101804.07 total 20.02 last 116.13) - open/close 3478 (time 1710101814.08 total 30.02 last 107.05) - open/close 4477 (time 1710101824.08 total 40.02 last 99.86) - open/close 5387 (time 1710101834.08 total 50.03 last 90.99) - open/close 6272 (time 1710101844.09 total 60.03 last 88.48) - open/close 7112 (time 1710101854.09 total 70.03 last 83.96) - open/close 7877 (time 1710101864.10 total 80.05 last 76.40) - open/close 8549 (time 1710101874.11 total 90.06 last 67.13) - open/close 9224 (time 1710101884.12 total 100.07 last 67.42) - open/close 9904 (time 1710101894.14 total 110.08 last 67.93) total: 10000 open/close in 111.55 seconds: 89.64 ops/second - unlinked 0 (time 1710101897 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 1200 (time 1710101985.26 total 10.00 last 119.99) - open/close 2332 (time 1710101995.26 total 20.01 last 113.14) - open/close 3397 (time 1710102005.27 total 30.01 last 106.43) - open/close 4425 (time 1710102015.27 total 40.02 last 102.73) - open/close 5351 (time 1710102025.27 total 50.02 last 92.59) - open/close 6224 (time 1710102035.28 total 60.02 last 87.29) - open/close 7069 (time 1710102045.28 total 70.03 last 84.47) - open/close 7866 (time 1710102055.28 total 80.03 last 79.67) - open/close 8610 (time 1710102065.29 total 90.03 last 74.37) - open/close 9336 (time 1710102075.29 total 100.03 last 72.60) total: 10000 open/close in 109.15 seconds: 91.61 ops/second - unlinked 0 (time 1710102086 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 1311 (time 1710102169.71 total 10.00 last 131.04) - open/close 2557 (time 1710102179.71 total 20.01 last 124.58) - open/close 3736 (time 1710102189.71 total 30.01 last 117.83) - open/close 4878 (time 1710102199.73 total 40.03 last 114.04) - open/close 5885 (time 1710102209.73 total 50.03 last 100.67) - open/close 6862 (time 1710102219.74 total 60.04 last 97.59) - open/close 7761 (time 1710102229.75 total 70.05 last 89.85) - open/close 8621 (time 1710102239.75 total 80.05 last 85.99) - open/close 9423 (time 1710102249.75 total 90.05 last 80.20) total: 10000 open/close in 97.50 seconds: 102.57 ops/second - unlinked 0 (time 1710102258 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 1429 (time 1710102335.67 total 10.00 last 142.85) - open/close 2767 (time 1710102345.67 total 20.00 last 133.80) - open/close 3985 (time 1710102355.67 total 30.01 last 121.78) - open/close 5122 (time 1710102365.67 total 40.01 last 113.64) - open/close 6182 (time 1710102375.68 total 50.02 last 105.90) - open/close 7185 (time 1710102385.69 total 60.03 last 100.25) - open/close 8125 (time 1710102395.69 total 70.03 last 93.95) - open/close 8974 (time 1710102405.69 total 80.03 last 84.90) - open/close 9807 (time 1710102415.69 total 90.03 last 83.30) total: 10000 open/close in 92.26 seconds: 108.39 ops/second - unlinked 0 (time 1710102419 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1709 (time 1710102493.20 total 10.00 last 170.85) - open/close 3223 (time 1710102503.20 total 20.01 last 151.37) - open/close 4661 (time 1710102513.20 total 30.01 last 143.77) - open/close 5979 (time 1710102523.20 total 40.01 last 131.77) - open/close 7193 (time 1710102533.21 total 50.01 last 121.40) - open/close 8324 (time 1710102543.21 total 60.01 last 113.08) - open/close 9424 (time 1710102553.22 total 70.02 last 109.90) total: 10000 open/close in 75.32 seconds: 132.76 ops/second - unlinked 0 (time 1710102559 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 3152 (time 1710102606.71 total 10.00 last 315.18) - open/close 5547 (time 1710102616.72 total 20.00 last 239.48) - open/close 7445 (time 1710102626.72 total 30.00 last 189.77) - open/close 9297 (time 1710102636.72 total 40.00 last 185.18) total: 10000 open/close in 43.88 seconds: 227.89 ops/second - unlinked 0 (time 1710102641 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg339-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 17 sec oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock oleg339-server@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server unloading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2