== conf-sanity test 69: replace an OST with the same index ========================================================== 16:00:05 (1710100805) start mds service on oleg123-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg123-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock oleg123-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49602 inodes available. Want 99872. rc=0 - open/close 1524 (time 1710100830.93 total 10.00 last 152.38) - open/close 2903 (time 1710100840.93 total 20.01 last 137.82) - open/close 4111 (time 1710100850.94 total 30.01 last 120.76) - open/close 5263 (time 1710100860.94 total 40.01 last 115.17) - open/close 6139 (time 1710100870.95 total 50.02 last 87.51) - open/close 6889 (time 1710100880.96 total 60.03 last 74.93) - open/close 7748 (time 1710100890.97 total 70.05 last 85.78) - open/close 8525 (time 1710100900.99 total 80.06 last 77.61) - open/close 9181 (time 1710100911.00 total 90.07 last 65.52) - open/close 9851 (time 1710100921.01 total 100.08 last 66.94) total: 10000 open/close in 101.90 seconds: 98.14 ops/second - unlinked 0 (time 1710100924 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 1248 (time 1710101004.85 total 10.01 last 124.65) - open/close 2477 (time 1710101014.90 total 20.06 last 122.34) - open/close 3580 (time 1710101024.91 total 30.06 last 110.23) - open/close 4425 (time 1710101034.92 total 40.07 last 84.42) - open/close 5376 (time 1710101044.92 total 50.08 last 95.05) - open/close 6172 (time 1710101054.92 total 60.08 last 79.60) - open/close 7085 (time 1710101064.93 total 70.09 last 91.24) - open/close 7957 (time 1710101074.93 total 80.09 last 87.18) - open/close 8657 (time 1710101084.94 total 90.09 last 69.96) - open/close 9427 (time 1710101094.94 total 100.10 last 76.96) total: 10000 open/close in 107.66 seconds: 92.89 ops/second - unlinked 0 (time 1710101104 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 1065 (time 1710101207.26 total 10.01 last 106.42) - open/close 2118 (time 1710101217.34 total 20.09 last 104.49) - open/close 2949 (time 1710101227.35 total 30.10 last 83.01) - open/close 3817 (time 1710101237.35 total 40.10 last 86.78) - open/close 4541 (time 1710101247.35 total 50.10 last 72.40) - open/close 5262 (time 1710101257.36 total 60.11 last 72.04) - open/close 5899 (time 1710101267.37 total 70.12 last 63.64) - open/close 6519 (time 1710101277.38 total 80.12 last 61.95) - open/close 7219 (time 1710101287.39 total 90.13 last 69.92) - open/close 7917 (time 1710101297.39 total 100.13 last 69.80) - open/close 8505 (time 1710101307.41 total 110.15 last 58.71) - open/close 9073 (time 1710101317.41 total 120.16 last 56.76) - open/close 9615 (time 1710101327.42 total 130.17 last 54.16) total: 10000 open/close in 136.67 seconds: 73.17 ops/second - unlinked 0 (time 1710101336 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 1083 (time 1710101438.19 total 10.01 last 108.22) - open/close 2022 (time 1710101448.20 total 20.01 last 93.86) - open/close 2877 (time 1710101458.22 total 30.03 last 85.31) - open/close 3653 (time 1710101468.23 total 40.04 last 77.55) - open/close 4324 (time 1710101478.26 total 50.07 last 66.88) - open/close 5004 (time 1710101488.34 total 60.16 last 67.41) - open/close 5711 (time 1710101498.39 total 70.20 last 70.41) - open/close 6162 (time 1710101516.54 total 88.36 last 24.84) - open/close 6923 (time 1710101526.55 total 98.36 last 76.06) - open/close 7531 (time 1710101536.57 total 108.38 last 60.70) - open/close 8164 (time 1710101546.58 total 118.40 last 63.20) - open/close 8784 (time 1710101556.59 total 128.41 last 61.94) - open/close 9400 (time 1710101566.59 total 138.41 last 61.60) total: 10000 open/close in 148.25 seconds: 67.45 ops/second - unlinked 0 (time 1710101579 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 1128 (time 1710101679.70 total 10.01 last 112.74) - open/close 2192 (time 1710101689.71 total 20.02 last 106.28) - open/close 3085 (time 1710101699.71 total 30.02 last 89.29) - open/close 3802 (time 1710101719.50 total 49.81 last 36.22) - open/close 4481 (time 1710101729.51 total 59.82 last 67.86) - open/close 5190 (time 1710101739.52 total 69.83 last 70.83) - open/close 5851 (time 1710101749.54 total 79.85 last 65.96) - open/close 6514 (time 1710101759.54 total 89.85 last 66.27) - open/close 7152 (time 1710101769.55 total 99.86 last 63.74) - open/close 7826 (time 1710101779.56 total 109.87 last 67.32) - open/close 8418 (time 1710101789.57 total 119.88 last 59.14) - open/close 9037 (time 1710101799.58 total 129.89 last 61.86) - open/close 9613 (time 1710101809.58 total 139.89 last 57.59) total: 10000 open/close in 147.61 seconds: 67.75 ops/second - unlinked 0 (time 1710101820 ; total 0 ; last 0) total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second - open/close 1126 (time 1710101938.72 total 10.00 last 112.55) - open/close 2134 (time 1710101948.73 total 20.01 last 100.77) - open/close 3052 (time 1710101958.74 total 30.02 last 91.72) - open/close 3824 (time 1710101968.75 total 40.03 last 77.12) - open/close 4589 (time 1710101978.75 total 50.03 last 76.45) - open/close 5329 (time 1710101988.76 total 60.04 last 73.98) - open/close 6053 (time 1710101998.77 total 70.05 last 72.28) - open/close 6663 (time 1710102008.77 total 80.05 last 60.99) - open/close 7299 (time 1710102018.79 total 90.07 last 63.52) - open/close 7897 (time 1710102028.80 total 100.08 last 59.74) - open/close 8473 (time 1710102038.80 total 110.08 last 57.57) - open/close 9162 (time 1710102048.82 total 120.10 last 68.78) - open/close 9740 (time 1710102058.82 total 130.10 last 57.79) total: 10000 open/close in 134.69 seconds: 74.25 ops/second - unlinked 0 (time 1710102065 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 1002 (time 1710102165.86 total 10.01 last 100.12) - open/close 1983 (time 1710102175.86 total 20.01 last 98.08) - open/close 2870 (time 1710102185.87 total 30.02 last 88.59) - open/close 3025 (time 1710102198.67 total 42.82 last 12.11) - open/close 3806 (time 1710102208.68 total 52.83 last 78.05) - open/close 4491 (time 1710102218.69 total 62.84 last 68.43) - open/close 5234 (time 1710102228.69 total 72.84 last 74.30) - open/close 5886 (time 1710102238.69 total 82.84 last 65.18) - open/close 6563 (time 1710102248.70 total 92.85 last 67.65) - open/close 7155 (time 1710102258.74 total 102.89 last 58.99) - open/close 7781 (time 1710102268.74 total 112.89 last 62.59) - open/close 8420 (time 1710102278.91 total 123.06 last 62.82) - open/close 9030 (time 1710102288.91 total 133.06 last 61.00) - open/close 9582 (time 1710102298.92 total 143.07 last 55.14) total: 10000 open/close in 150.15 seconds: 66.60 ops/second - unlinked 0 (time 1710102308 ; total 0 ; last 0) total: 10000 unlinks in 118 seconds: 84.745766 unlinks/second - open/close 1224 (time 1710102438.69 total 10.00 last 122.39) - open/close 2146 (time 1710102448.69 total 20.00 last 92.16) - open/close 3010 (time 1710102458.70 total 30.01 last 86.38) - open/close 3874 (time 1710102468.70 total 40.01 last 86.34) - open/close 4567 (time 1710102478.72 total 50.03 last 69.19) - open/close 5392 (time 1710102488.72 total 60.04 last 82.45) - open/close 6127 (time 1710102498.73 total 70.04 last 73.44) - open/close 6805 (time 1710102508.75 total 80.06 last 67.70) - open/close 7415 (time 1710102518.76 total 90.07 last 60.91) - open/close 8018 (time 1710102528.77 total 100.08 last 60.24) - open/close 8581 (time 1710102538.84 total 110.15 last 55.94) - open/close 9128 (time 1710102548.84 total 120.15 last 54.68) - open/close 9716 (time 1710102558.84 total 130.15 last 58.79) total: 10000 open/close in 134.67 seconds: 74.26 ops/second - unlinked 0 (time 1710102565 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 621 (time 1710102667.88 total 17.11 last 36.28) - open/close 1540 (time 1710102677.89 total 27.12 last 91.85) - open/close 2334 (time 1710102697.82 total 47.06 last 39.83) - open/close 3134 (time 1710102707.87 total 57.10 last 79.64) - open/close 3330 (time 1710102721.43 total 70.66 last 14.45) - open/close 3863 (time 1710102739.87 total 89.11 last 28.90) - open/close 4641 (time 1710102749.88 total 99.12 last 77.74) - open/close 4669 (time 1710102761.28 total 110.52 last 2.46) - open/close 5400 (time 1710102771.34 total 120.58 last 72.65) - open/close 6121 (time 1710102781.35 total 130.58 last 72.08) - open/close 6816 (time 1710102791.35 total 140.58 last 69.50) - open/close 6938 (time 1710102804.09 total 153.32 last 9.57) - open/close 7566 (time 1710102814.10 total 163.33 last 62.75) - open/close 8181 (time 1710102824.12 total 173.36 last 61.34) - open/close 8784 (time 1710102834.13 total 183.36 last 60.29) - open/close 9366 (time 1710102844.13 total 193.36 last 58.19) - open/close 9916 (time 1710102854.18 total 203.42 last 54.69) total: 10000 open/close in 205.00 seconds: 48.78 ops/second - unlinked 0 (time 1710102858 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 967 (time 1710102964.79 total 20.13 last 48.05) - open/close 1927 (time 1710102974.80 total 30.14 last 95.90) - open/close 2828 (time 1710102984.81 total 40.14 last 90.04) - open/close 3629 (time 1710102994.81 total 50.15 last 80.06) - open/close 4388 (time 1710103004.81 total 60.15 last 75.87) - open/close 5075 (time 1710103014.83 total 70.16 last 68.62) - open/close 5761 (time 1710103024.83 total 80.17 last 68.59) - open/close 6414 (time 1710103034.83 total 90.17 last 65.27) - open/close 6996 (time 1710103044.84 total 100.18 last 58.15) - open/close 7647 (time 1710103054.85 total 110.18 last 65.08) - open/close 8246 (time 1710103064.85 total 120.19 last 59.86) - open/close 8854 (time 1710103074.86 total 130.20 last 60.76) - open/close 9438 (time 1710103084.86 total 140.20 last 58.39) total: 10000 open/close in 149.20 seconds: 67.03 ops/second - unlinked 0 (time 1710103096 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg123-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg123-server: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg123-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock oleg123-server@tcp:/lustre /mnt/lustre On OST0, 20185 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server unloading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2