== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:23:34 (1746141814) Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) oleg403-server: oleg403-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg403-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1118 (time 1746141870.01 total 10.00 last 111.76) - open/close 2223 (time 1746141880.02 total 20.02 last 110.38) - open/close 3303 (time 1746141890.02 total 30.02 last 108.00) - open/close 4748 (time 1746141900.03 total 40.03 last 144.36) - open/close 5917 (time 1746141910.04 total 50.04 last 116.78) - open/close 6977 (time 1746141920.04 total 60.04 last 105.99) - open/close 7970 (time 1746141930.04 total 70.04 last 99.27) - open/close 9028 (time 1746141940.06 total 80.05 last 105.66) - open/close 10000 (time 1746141949.48 total 89.48 last 103.16) - open/close 10986 (time 1746141959.48 total 99.48 last 98.56) - open/close 12092 (time 1746141969.49 total 109.49 last 110.52) - open/close 13201 (time 1746141979.49 total 119.49 last 110.88) - open/close 14385 (time 1746141989.50 total 129.50 last 118.29) - open/close 15782 (time 1746141999.51 total 139.51 last 139.56) - open/close 17039 (time 1746142009.52 total 149.51 last 125.61) - open/close 18339 (time 1746142019.52 total 159.52 last 129.97) - open/close 19458 (time 1746142029.52 total 169.52 last 111.85) - open/close 20000 (time 1746142035.12 total 175.12 last 96.86) - open/close 21132 (time 1746142045.13 total 185.12 last 113.15) - open/close 22417 (time 1746142055.13 total 195.13 last 128.41) - open/close 23795 (time 1746142065.14 total 205.14 last 137.71) - open/close 24984 (time 1746142075.14 total 215.14 last 118.89) - open/close 26136 (time 1746142085.14 total 225.14 last 115.20) - open/close 27322 (time 1746142095.15 total 235.15 last 118.50) - open/close 28360 (time 1746142105.15 total 245.15 last 103.77) - open/close 29643 (time 1746142115.15 total 255.15 last 128.28) - open/close 30000 (time 1746142117.77 total 257.76 last 136.61) - open/close 31280 (time 1746142127.77 total 267.77 last 127.91) - open/close 32592 (time 1746142137.78 total 277.78 last 131.08) - open/close 33878 (time 1746142147.78 total 287.78 last 128.60) - open/close 35043 (time 1746142157.79 total 297.79 last 116.43) - open/close 36020 (time 1746142167.80 total 307.79 last 97.63) - open/close 37184 (time 1746142177.80 total 317.80 last 116.32) - open/close 38582 (time 1746142187.81 total 327.80 last 139.74) - open/close 39639 (time 1746142197.81 total 337.81 last 105.63) - open/close 40000 (time 1746142200.76 total 340.76 last 122.47) - open/close 41127 (time 1746142210.77 total 350.76 last 112.66) - open/close 42766 (time 1746142220.77 total 360.76 last 163.89) - open/close 44540 (time 1746142230.77 total 370.77 last 177.28) - open/close 45811 (time 1746142240.78 total 380.77 last 127.04) - open/close 47037 (time 1746142250.78 total 390.78 last 122.52) - open/close 48187 (time 1746142260.79 total 400.78 last 114.99) - open/close 49411 (time 1746142270.79 total 410.78 last 122.37) - open/close 50000 (time 1746142275.72 total 415.71 last 119.49) - open/close 51301 (time 1746142285.72 total 425.72 last 130.07) - open/close 52670 (time 1746142295.72 total 435.72 last 136.85) - open/close 54176 (time 1746142305.73 total 445.72 last 150.53) - open/close 55638 (time 1746142315.73 total 455.72 last 146.19) - open/close 56883 (time 1746142325.73 total 465.73 last 124.48) - open/close 58052 (time 1746142335.73 total 475.73 last 116.86) - open/close 59458 (time 1746142345.74 total 485.73 last 140.53) - open/close 60000 (time 1746142349.30 total 489.30 last 151.98) - open/close 61270 (time 1746142359.30 total 499.30 last 126.99) - open/close 62492 (time 1746142369.31 total 509.30 last 122.17) - open/close 63890 (time 1746142379.31 total 519.31 last 139.75) total: 64768 open/close in 526.87 seconds: 122.93 ops/second - /unlink 1818 (time 1746142398.51 total 10.00 last 181.74) - /unlink 3989 (time 1746142408.51 total 20.00 last 217.07) - /unlink 5949 (time 1746142418.51 total 30.01 last 195.97) - /unlink 8054 (time 1746142428.52 total 40.01 last 210.35) - /unlink 9749 (time 1746142438.52 total 50.02 last 169.46) - /unlink 11513 (time 1746142448.53 total 60.02 last 176.30) - /unlink 13746 (time 1746142458.53 total 70.03 last 223.21) - /unlink 16628 (time 1746142468.53 total 80.03 last 288.10) - /unlink 19474 (time 1746142478.54 total 90.04 last 284.34) - /unlink 20000 (time 1746142480.94 total 92.43 last 219.84) - /unlink 22324 (time 1746142490.94 total 102.43 last 232.36) - /unlink 24592 (time 1746142500.94 total 112.44 last 226.69) - /unlink 26490 (time 1746142510.94 total 122.44 last 189.78) - /unlink 28603 (time 1746142520.95 total 132.44 last 211.26) - /unlink 30000 (time 1746142527.24 total 138.73 last 222.04) - /unlink 31927 (time 1746142537.24 total 148.73 last 192.67) - /unlink 33655 (time 1746142547.24 total 158.74 last 172.71) - /unlink 35586 (time 1746142557.25 total 168.74 last 193.03) - /unlink 37285 (time 1746142567.25 total 178.74 last 169.89) - /unlink 38934 (time 1746142577.25 total 188.75 last 164.86) - /unlink 40000 (time 1746142583.31 total 194.81 last 175.81) - /unlink 41985 (time 1746142593.31 total 204.81 last 198.49) - /unlink 43842 (time 1746142603.32 total 214.81 last 185.64) - /unlink 45609 (time 1746142613.32 total 224.81 last 176.66) - /unlink 47460 (time 1746142623.32 total 234.82 last 185.09) - /unlink 49552 (time 1746142633.32 total 244.82 last 209.11) - /unlink 51850 (time 1746142643.33 total 254.83 last 229.67) - /unlink 53941 (time 1746142653.33 total 264.83 last 209.08) - /unlink 56348 (time 1746142663.34 total 274.83 last 240.51) - /unlink 59235 (time 1746142673.34 total 284.84 last 288.63) - /unlink 60000 (time 1746142676.36 total 287.86 last 253.44) - /unlink 62730 (time 1746142686.36 total 297.86 last 272.99) total: 64768 /unlink in 306.61 seconds: 211.24 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1309 (time 1746142725.47 total 10.00 last 130.89) - open/close 2448 (time 1746142735.48 total 20.01 last 113.76) - open/close 3481 (time 1746142745.50 total 30.03 last 103.11) - open/close 4566 (time 1746142755.51 total 40.04 last 108.42) - open/close 5708 (time 1746142765.51 total 50.04 last 114.17) - open/close 6921 (time 1746142775.51 total 60.04 last 121.28) - open/close 7984 (time 1746142785.52 total 70.05 last 106.26) - open/close 8935 (time 1746142795.52 total 80.06 last 95.02) - open/close 9956 (time 1746142805.53 total 90.06 last 102.08) - open/close 10918 (time 1746142815.54 total 100.07 last 96.10) - open/close 11920 (time 1746142825.55 total 110.08 last 100.10) - open/close 12965 (time 1746142835.56 total 120.09 last 104.38) - open/close 13987 (time 1746142845.56 total 130.09 last 102.19) - open/close 15069 (time 1746142855.56 total 140.09 last 108.20) - open/close 16208 (time 1746142865.56 total 150.09 last 113.86) - open/close 17296 (time 1746142875.57 total 160.10 last 108.75) - open/close 18459 (time 1746142885.58 total 170.12 last 116.10) - open/close 19799 (time 1746142895.59 total 180.12 last 133.93) - open/close 21006 (time 1746142905.59 total 190.12 last 120.65) - open/close 22151 (time 1746142915.60 total 200.14 last 114.37) - open/close 23158 (time 1746142925.61 total 210.14 last 100.63) - open/close 24225 (time 1746142935.61 total 220.15 last 106.67) - open/close 25336 (time 1746142945.62 total 230.15 last 111.06) - open/close 26530 (time 1746142955.62 total 240.15 last 119.36) - open/close 27947 (time 1746142965.63 total 250.16 last 141.61) - open/close 29327 (time 1746142975.64 total 260.17 last 137.86) - open/close 30000 (time 1746142981.28 total 265.81 last 119.33) - open/close 31516 (time 1746142991.28 total 275.81 last 151.58) - open/close 32758 (time 1746143001.28 total 285.81 last 124.20) - open/close 34069 (time 1746143011.28 total 295.81 last 131.08) - open/close 35432 (time 1746143021.29 total 305.82 last 136.18) - open/close 36654 (time 1746143031.29 total 315.82 last 122.19) - open/close 37791 (time 1746143041.29 total 325.82 last 113.70) - open/close 39050 (time 1746143051.29 total 335.82 last 125.89) - open/close 40000 (time 1746143059.04 total 343.57 last 122.60) - open/close 41284 (time 1746143069.04 total 353.57 last 128.38) - open/close 42426 (time 1746143079.05 total 363.58 last 114.10) - open/close 43573 (time 1746143089.06 total 373.59 last 114.61) - open/close 44889 (time 1746143099.06 total 383.59 last 131.59) - open/close 46428 (time 1746143109.06 total 393.60 last 153.84) - open/close 47905 (time 1746143119.07 total 403.60 last 147.68) - open/close 49333 (time 1746143129.07 total 413.60 last 142.79) - open/close 50000 (time 1746143134.22 total 418.76 last 129.33) - open/close 51491 (time 1746143144.22 total 428.76 last 149.09) - open/close 53411 (time 1746143154.23 total 438.76 last 191.94) - open/close 54604 (time 1746143164.23 total 448.76 last 119.29) - open/close 55730 (time 1746143174.23 total 458.76 last 112.55) - open/close 57057 (time 1746143184.24 total 468.77 last 132.63) - open/close 58469 (time 1746143194.24 total 478.77 last 141.17) - open/close 59787 (time 1746143204.25 total 488.78 last 131.73) - open/close 61118 (time 1746143214.25 total 498.78 last 133.02) - open/close 62484 (time 1746143224.25 total 508.78 last 136.59) - open/close 63698 (time 1746143234.26 total 518.79 last 121.30) total: 64768 open/close in 528.09 seconds: 122.65 ops/second - /unlink 2084 (time 1746143255.34 total 10.00 last 208.35) - /unlink 3803 (time 1746143265.35 total 20.00 last 171.87) - /unlink 5699 (time 1746143275.35 total 30.01 last 189.56) - /unlink 7702 (time 1746143285.35 total 40.01 last 200.19) - /unlink 9758 (time 1746143295.36 total 50.01 last 205.55) - /unlink 11898 (time 1746143305.36 total 60.02 last 213.96) - /unlink 14396 (time 1746143315.36 total 70.02 last 249.78) - /unlink 17395 (time 1746143325.36 total 80.02 last 299.83) - /unlink 20000 (time 1746143334.11 total 88.77 last 297.57) - /unlink 22390 (time 1746143344.12 total 98.78 last 238.89) - /unlink 25488 (time 1746143354.12 total 108.78 last 309.77) - /unlink 28007 (time 1746143364.12 total 118.78 last 251.85) - /unlink 30000 (time 1746143373.26 total 127.92 last 218.11) - /unlink 32015 (time 1746143383.26 total 137.92 last 201.48) - /unlink 34531 (time 1746143393.26 total 147.92 last 251.59) - /unlink 36594 (time 1746143403.26 total 157.92 last 206.25) - /unlink 38879 (time 1746143413.27 total 167.93 last 228.42) - /unlink 40000 (time 1746143417.69 total 172.35 last 253.21) - /unlink 43088 (time 1746143427.70 total 182.36 last 308.73) - /unlink 45905 (time 1746143437.70 total 192.36 last 281.61) - /unlink 49115 (time 1746143447.70 total 202.36 last 320.98) - /unlink 50000 (time 1746143450.00 total 204.66 last 385.31) - /unlink 53423 (time 1746143460.00 total 214.66 last 342.26) - /unlink 56270 (time 1746143470.00 total 224.66 last 284.59) - /unlink 59092 (time 1746143480.00 total 234.66 last 282.17) - /unlink 60000 (time 1746143483.40 total 238.06 last 267.51) - /unlink 62531 (time 1746143493.40 total 248.06 last 253.07) total: 64768 /unlink in 257.22 seconds: 251.80 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1195 (time 1746143532.12 total 10.01 last 119.44) - open/close 2284 (time 1746143542.12 total 20.01 last 108.88) - open/close 3494 (time 1746143552.14 total 30.03 last 120.74) - open/close 4565 (time 1746143562.15 total 40.04 last 106.98) - open/close 5653 (time 1746143572.16 total 50.05 last 108.70) - open/close 6696 (time 1746143582.17 total 60.06 last 104.23) - open/close 7880 (time 1746143592.17 total 70.06 last 118.36) - open/close 9122 (time 1746143602.17 total 80.06 last 124.17) - open/close 10000 (time 1746143609.96 total 87.85 last 112.79) - open/close 11354 (time 1746143619.97 total 97.85 last 135.28) - open/close 12840 (time 1746143629.97 total 107.86 last 148.53) - open/close 14236 (time 1746143639.98 total 117.87 last 139.46) - open/close 15535 (time 1746143649.99 total 127.87 last 129.84) - open/close 16820 (time 1746143659.99 total 137.88 last 128.47) - open/close 18241 (time 1746143670.00 total 147.88 last 142.01) - open/close 19444 (time 1746143680.00 total 157.89 last 120.26) - open/close 20000 (time 1746143683.81 total 161.70 last 145.92) - open/close 21474 (time 1746143693.82 total 171.70 last 147.31) - open/close 22684 (time 1746143703.82 total 181.71 last 120.96) - open/close 23969 (time 1746143713.82 total 191.71 last 128.47) - open/close 25213 (time 1746143723.82 total 201.71 last 124.36) - open/close 26348 (time 1746143733.84 total 211.73 last 113.31) - open/close 27223 (time 1746143743.85 total 221.73 last 87.44) - open/close 28293 (time 1746143753.85 total 231.74 last 106.97) - open/close 29548 (time 1746143763.86 total 241.74 last 125.41) - open/close 30000 (time 1746143767.67 total 245.56 last 118.51) - open/close 31202 (time 1746143777.67 total 255.56 last 120.18) - open/close 32437 (time 1746143787.67 total 265.56 last 123.49) - open/close 33870 (time 1746143797.67 total 275.56 last 143.28) - open/close 35208 (time 1746143807.68 total 285.56 last 133.76) - open/close 36620 (time 1746143817.68 total 295.57 last 141.11) - open/close 37858 (time 1746143827.69 total 305.58 last 123.75) - open/close 39128 (time 1746143837.69 total 315.58 last 126.94) - open/close 40000 (time 1746143845.12 total 323.01 last 117.36) - open/close 41157 (time 1746143855.13 total 333.01 last 115.66) - open/close 42190 (time 1746143865.14 total 343.03 last 103.15) - open/close 43273 (time 1746143875.14 total 353.03 last 108.27) - open/close 44218 (time 1746143885.14 total 363.03 last 94.49) - open/close 45283 (time 1746143895.15 total 373.04 last 106.40) - open/close 46440 (time 1746143905.17 total 383.06 last 115.53) - open/close 47521 (time 1746143915.17 total 393.06 last 108.09) - open/close 48653 (time 1746143925.18 total 403.07 last 113.05) - open/close 49879 (time 1746143935.19 total 413.08 last 122.51) - open/close 51019 (time 1746143945.19 total 423.08 last 113.99) - open/close 52369 (time 1746143955.19 total 433.08 last 134.99) - open/close 53966 (time 1746143965.20 total 443.08 last 159.63) - open/close 55559 (time 1746143975.20 total 453.08 last 159.28) - open/close 56885 (time 1746143985.20 total 463.09 last 132.54) - open/close 58217 (time 1746143995.21 total 473.09 last 133.13) - open/close 59638 (time 1746144005.21 total 483.10 last 142.08) - open/close 60000 (time 1746144007.83 total 485.72 last 137.93) - open/close 61435 (time 1746144017.84 total 495.72 last 143.46) - open/close 62864 (time 1746144027.84 total 505.72 last 142.87) - open/close 64641 (time 1746144037.84 total 515.73 last 177.64) total: 64768 open/close in 516.54 seconds: 125.39 ops/second - /unlink 2327 (time 1746144050.20 total 10.00 last 232.65) - /unlink 4822 (time 1746144060.20 total 20.00 last 249.45) - /unlink 7830 (time 1746144070.20 total 30.01 last 300.67) - /unlink 10000 (time 1746144078.29 total 38.10 last 268.22) - /unlink 12584 (time 1746144088.30 total 48.10 last 258.33) - /unlink 15036 (time 1746144098.30 total 58.11 last 245.08) - /unlink 17408 (time 1746144108.30 total 68.11 last 237.19) - /unlink 19545 (time 1746144118.30 total 78.11 last 213.67) - /unlink 20000 (time 1746144120.60 total 80.40 last 198.39) - /unlink 21941 (time 1746144130.60 total 90.41 last 194.01) - /unlink 24769 (time 1746144140.60 total 100.41 last 282.75) - /unlink 27595 (time 1746144150.60 total 110.41 last 282.51) - /unlink 29955 (time 1746144160.61 total 120.41 last 235.94) - /unlink 33144 (time 1746144170.61 total 130.42 last 318.81) - /unlink 35606 (time 1746144180.61 total 140.42 last 246.20) - /unlink 37780 (time 1746144190.61 total 150.42 last 217.34) - /unlink 40000 (time 1746144200.05 total 159.85 last 235.31) - /unlink 42228 (time 1746144210.05 total 169.86 last 222.72) - /unlink 44361 (time 1746144220.05 total 179.86 last 213.25) - /unlink 46458 (time 1746144230.05 total 189.86 last 209.67) - /unlink 48909 (time 1746144240.06 total 199.86 last 245.09) - /unlink 50000 (time 1746144244.89 total 204.70 last 225.73) - /unlink 51880 (time 1746144254.90 total 214.70 last 187.85) - /unlink 53753 (time 1746144264.90 total 224.71 last 187.22) - /unlink 55445 (time 1746144274.91 total 234.71 last 169.11) - /unlink 57239 (time 1746144284.91 total 244.72 last 179.30) - /unlink 59111 (time 1746144294.91 total 254.72 last 187.19) - /unlink 60000 (time 1746144299.02 total 258.83 last 216.45) - /unlink 61865 (time 1746144309.02 total 268.83 last 186.45) - /unlink 63888 (time 1746144319.03 total 278.83 last 202.23) total: 64768 /unlink in 282.95 seconds: 228.90 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1507 (time 1746144353.23 total 10.00 last 150.67) - open/close 2838 (time 1746144363.23 total 20.00 last 133.06) - open/close 4278 (time 1746144373.23 total 30.01 last 143.97) - open/close 5764 (time 1746144383.24 total 40.01 last 148.53) - open/close 7236 (time 1746144393.24 total 50.01 last 147.18) - open/close 8983 (time 1746144403.24 total 60.02 last 174.61) - open/close 10000 (time 1746144408.98 total 65.75 last 177.40) - open/close 11360 (time 1746144418.98 total 75.76 last 135.93) - open/close 12505 (time 1746144428.99 total 85.76 last 114.45) - open/close 13582 (time 1746144438.99 total 95.77 last 107.65) - open/close 14861 (time 1746144449.00 total 105.77 last 127.86) - open/close 15947 (time 1746144459.00 total 115.77 last 108.55) - open/close 16906 (time 1746144469.00 total 125.77 last 95.89) - open/close 18003 (time 1746144479.01 total 135.79 last 109.56) - open/close 19057 (time 1746144489.02 total 145.80 last 105.30) - open/close 20000 (time 1746144498.49 total 155.26 last 99.61) - open/close 21092 (time 1746144508.49 total 165.27 last 109.18) - open/close 22244 (time 1746144518.49 total 175.27 last 115.19) - open/close 23470 (time 1746144528.50 total 185.27 last 122.52) - open/close 24632 (time 1746144538.51 total 195.28 last 116.09) - open/close 25769 (time 1746144548.52 total 205.29 last 113.65) - open/close 26882 (time 1746144558.52 total 215.30 last 111.23) - open/close 27912 (time 1746144568.53 total 225.30 last 102.97) - open/close 29100 (time 1746144578.53 total 235.30 last 118.75) - open/close 30000 (time 1746144586.09 total 242.86 last 119.01) - open/close 31068 (time 1746144596.10 total 252.87 last 106.75) - open/close 32360 (time 1746144606.10 total 262.87 last 129.16) - open/close 33662 (time 1746144616.10 total 272.88 last 130.16) - open/close 35099 (time 1746144626.11 total 282.88 last 143.65) - open/close 36434 (time 1746144636.11 total 292.88 last 133.49) - open/close 37650 (time 1746144646.11 total 302.89 last 121.53) - open/close 38795 (time 1746144656.11 total 312.89 last 114.50) - open/close 39937 (time 1746144666.11 total 322.89 last 114.17) - open/close 41040 (time 1746144676.12 total 332.89 last 110.29) - open/close 42339 (time 1746144686.12 total 342.89 last 129.87) - open/close 43640 (time 1746144696.12 total 352.89 last 130.09) - open/close 44898 (time 1746144706.12 total 362.90 last 125.76) - open/close 46254 (time 1746144716.13 total 372.90 last 135.50) - open/close 47407 (time 1746144726.13 total 382.91 last 115.24) - open/close 48646 (time 1746144736.14 total 392.92 last 123.80) - open/close 49854 (time 1746144746.14 total 402.92 last 120.78) - open/close 51109 (time 1746144756.14 total 412.92 last 125.49) - open/close 52267 (time 1746144766.15 total 422.93 last 115.71) - open/close 53440 (time 1746144776.16 total 432.93 last 117.26) - open/close 54645 (time 1746144786.16 total 442.93 last 120.44) - open/close 55956 (time 1746144796.16 total 452.94 last 131.04) - open/close 57062 (time 1746144806.17 total 462.94 last 110.60) - open/close 58597 (time 1746144816.17 total 472.94 last 153.44) - open/close 60000 (time 1746144825.93 total 482.70 last 143.80) - open/close 61579 (time 1746144835.93 total 492.71 last 157.77) - open/close 62796 (time 1746144845.94 total 502.71 last 121.62) - open/close 63947 (time 1746144855.95 total 512.72 last 115.04) total: 64768 open/close in 519.41 seconds: 124.69 ops/second - /unlink 3029 (time 1746144874.20 total 10.00 last 302.84) - /unlink 5836 (time 1746144884.21 total 20.00 last 280.62) - /unlink 8364 (time 1746144894.21 total 30.01 last 252.72) - /unlink 10000 (time 1746144901.36 total 37.16 last 228.81) - /unlink 12779 (time 1746144911.36 total 47.16 last 277.86) - /unlink 15367 (time 1746144921.36 total 57.16 last 258.80) - /unlink 18229 (time 1746144931.36 total 67.16 last 286.14) - /unlink 20000 (time 1746144937.37 total 73.17 last 294.81) - /unlink 23388 (time 1746144947.37 total 83.17 last 338.74) - /unlink 27011 (time 1746144957.37 total 93.17 last 362.23) - /unlink 30000 (time 1746144965.48 total 101.28 last 368.69) - /unlink 32958 (time 1746144975.48 total 111.28 last 295.76) - /unlink 35846 (time 1746144985.49 total 121.29 last 288.65) - /unlink 37896 (time 1746144995.50 total 131.29 last 204.83) - /unlink 40000 (time 1746145003.77 total 139.57 last 254.28) - /unlink 43133 (time 1746145013.77 total 149.57 last 313.23) - /unlink 45503 (time 1746145023.77 total 159.57 last 236.97) - /unlink 47636 (time 1746145033.77 total 169.57 last 213.27) - /unlink 50000 (time 1746145041.76 total 177.56 last 295.99) - /unlink 52939 (time 1746145051.76 total 187.56 last 293.81) - /unlink 55667 (time 1746145061.77 total 197.57 last 272.73) - /unlink 59159 (time 1746145071.77 total 207.57 last 349.16) - /unlink 60000 (time 1746145074.06 total 209.86 last 366.87) - /unlink 62690 (time 1746145084.06 total 219.86 last 268.98) total: 64768 /unlink in 227.15 seconds: 285.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1868 (time 1746145124.50 total 10.00 last 186.78) - open/close 4254 (time 1746145134.51 total 20.01 last 238.36) - open/close 5790 (time 1746145144.52 total 30.01 last 153.56) - open/close 8337 (time 1746145154.52 total 40.01 last 254.68) - open/close 10000 (time 1746145158.96 total 44.46 last 374.04) - open/close 12568 (time 1746145168.97 total 54.46 last 256.74) - open/close 14432 (time 1746145178.97 total 64.46 last 186.38) - open/close 16527 (time 1746145188.97 total 74.46 last 209.48) - open/close 18438 (time 1746145198.97 total 84.47 last 191.08) - open/close 20000 (time 1746145206.39 total 91.88 last 210.63) - open/close 21947 (time 1746145216.39 total 101.88 last 194.66) - open/close 23666 (time 1746145226.39 total 111.89 last 171.79) - open/close 25473 (time 1746145236.40 total 121.89 last 180.67) - open/close 27644 (time 1746145246.40 total 131.89 last 217.05) - open/close 29616 (time 1746145256.41 total 141.90 last 197.06) - open/close 30000 (time 1746145258.85 total 144.35 last 157.06) - open/close 31826 (time 1746145268.85 total 154.35 last 182.56) - open/close 33538 (time 1746145278.86 total 164.35 last 171.12) - open/close 35000 (time 1746145288.87 total 174.36 last 146.05) - open/close 36565 (time 1746145298.88 total 184.37 last 156.36) - open/close 38328 (time 1746145308.88 total 194.38 last 176.16) - open/close 39831 (time 1746145318.89 total 204.38 last 150.26) - open/close 42049 (time 1746145328.89 total 214.39 last 221.75) - open/close 44400 (time 1746145338.89 total 224.39 last 235.03) - open/close 46290 (time 1746145348.89 total 234.39 last 188.98) - open/close 48030 (time 1746145358.90 total 244.40 last 173.83) - open/close 49954 (time 1746145368.91 total 254.40 last 192.34) - open/close 51448 (time 1746145378.91 total 264.40 last 149.36) - open/close 53229 (time 1746145388.91 total 274.41 last 178.06) - open/close 54671 (time 1746145398.92 total 284.41 last 144.13) - open/close 56053 (time 1746145408.92 total 294.41 last 138.17) - open/close 57808 (time 1746145418.92 total 304.42 last 175.46) - open/close 60000 (time 1746145427.30 total 312.80 last 261.51) - open/close 61817 (time 1746145437.30 total 322.80 last 181.65) - open/close 63839 (time 1746145447.31 total 332.80 last 202.14) total: 64768 open/close in 338.69 seconds: 191.23 ops/second - /unlink 3009 (time 1746145464.54 total 10.00 last 300.83) - /unlink 7453 (time 1746145474.54 total 20.00 last 444.31) - /unlink 10000 (time 1746145482.14 total 27.61 last 335.09) - /unlink 13783 (time 1746145492.14 total 37.61 last 378.30) - /unlink 15970 (time 1746145502.14 total 47.61 last 218.67) - /unlink 19074 (time 1746145512.14 total 57.61 last 310.35) - /unlink 20000 (time 1746145514.45 total 59.91 last 402.08) - /unlink 22737 (time 1746145524.45 total 69.92 last 273.56) - /unlink 25975 (time 1746145534.45 total 79.92 last 323.69) - /unlink 28814 (time 1746145544.46 total 89.92 last 283.81) - /unlink 30000 (time 1746145549.02 total 94.48 last 260.10) - /unlink 32145 (time 1746145559.02 total 104.48 last 214.47) - /unlink 33829 (time 1746145569.02 total 114.49 last 168.35) - /unlink 35432 (time 1746145579.03 total 124.49 last 160.21) - /unlink 37223 (time 1746145589.03 total 134.49 last 179.08) - /unlink 39489 (time 1746145599.03 total 144.49 last 226.59) - /unlink 40000 (time 1746145601.48 total 146.95 last 208.36) - /unlink 41916 (time 1746145611.48 total 156.95 last 191.58) - /unlink 44221 (time 1746145621.48 total 166.95 last 230.45) - /unlink 46398 (time 1746145631.49 total 176.95 last 217.67) - /unlink 48421 (time 1746145641.49 total 186.95 last 202.23) - /unlink 50000 (time 1746145649.06 total 194.53 last 208.54) - /unlink 52091 (time 1746145659.06 total 204.53 last 209.04) - /unlink 54268 (time 1746145669.07 total 214.53 last 217.66) - /unlink 56169 (time 1746145679.07 total 224.53 last 190.09) - /unlink 58457 (time 1746145689.07 total 234.54 last 228.72) - /unlink 60000 (time 1746145697.13 total 242.59 last 191.48) - /unlink 62173 (time 1746145707.13 total 252.60 last 217.18) total: 64768 /unlink in 261.99 seconds: 247.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server unloading modules via unload_modules_local on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2