== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:54:13 (1746154453) Stopping clients: oleg203-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg203-client.virtnet /mnt/lustre2 (opts:-f) oleg203-server: oleg203-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: MODOPTS_LIBCFS= oleg203-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg203-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1216 (time 1746154508.37 total 10.00 last 121.57) - open/close 2460 (time 1746154518.37 total 20.00 last 124.37) - open/close 3661 (time 1746154528.37 total 30.01 last 120.09) - open/close 4881 (time 1746154538.38 total 40.01 last 121.91) - open/close 6062 (time 1746154548.39 total 50.03 last 117.91) - open/close 7045 (time 1746154558.40 total 60.04 last 98.24) - open/close 8082 (time 1746154568.40 total 70.04 last 103.68) - open/close 9084 (time 1746154578.40 total 80.04 last 100.19) - open/close 10000 (time 1746154586.26 total 87.90 last 116.56) - open/close 11106 (time 1746154596.27 total 97.90 last 110.54) - open/close 12258 (time 1746154606.27 total 107.90 last 115.20) - open/close 13405 (time 1746154616.27 total 117.90 last 114.67) - open/close 14548 (time 1746154626.27 total 127.91 last 114.27) - open/close 15789 (time 1746154636.27 total 137.91 last 124.06) - open/close 17239 (time 1746154646.28 total 147.91 last 144.94) - open/close 18517 (time 1746154656.28 total 157.92 last 127.72) - open/close 19674 (time 1746154666.29 total 167.92 last 115.69) - open/close 20000 (time 1746154668.53 total 170.16 last 145.32) - open/close 21281 (time 1746154678.53 total 180.17 last 128.07) - open/close 22605 (time 1746154688.54 total 190.17 last 132.33) - open/close 23871 (time 1746154698.54 total 200.18 last 126.53) - open/close 25117 (time 1746154708.54 total 210.18 last 124.56) - open/close 26318 (time 1746154718.55 total 220.19 last 120.01) - open/close 27498 (time 1746154728.56 total 230.19 last 117.95) - open/close 28647 (time 1746154738.56 total 240.20 last 114.82) - open/close 29915 (time 1746154748.56 total 250.20 last 126.79) - open/close 31115 (time 1746154758.57 total 260.20 last 119.95) - open/close 32268 (time 1746154768.57 total 270.21 last 115.26) - open/close 33389 (time 1746154778.58 total 280.21 last 112.02) - open/close 34520 (time 1746154788.58 total 290.22 last 113.04) - open/close 35735 (time 1746154798.58 total 300.22 last 121.49) - open/close 37160 (time 1746154808.59 total 310.23 last 142.43) - open/close 38352 (time 1746154818.59 total 320.23 last 119.15) - open/close 39675 (time 1746154828.60 total 330.23 last 132.26) - open/close 40000 (time 1746154830.70 total 332.34 last 154.22) - open/close 41563 (time 1746154840.71 total 342.35 last 156.23) - open/close 42956 (time 1746154850.71 total 352.35 last 139.29) - open/close 44302 (time 1746154860.71 total 362.35 last 134.55) - open/close 45449 (time 1746154870.72 total 372.35 last 114.64) - open/close 46517 (time 1746154880.72 total 382.35 last 106.79) - open/close 47634 (time 1746154890.72 total 392.36 last 111.67) - open/close 48956 (time 1746154900.72 total 402.36 last 132.17) - open/close 50000 (time 1746154909.45 total 411.08 last 119.68) - open/close 51239 (time 1746154919.46 total 421.09 last 123.76) - open/close 52627 (time 1746154929.46 total 431.10 last 138.72) - open/close 53857 (time 1746154939.47 total 441.10 last 122.98) - open/close 55091 (time 1746154949.47 total 451.10 last 123.37) - open/close 56339 (time 1746154959.47 total 461.11 last 124.75) - open/close 57538 (time 1746154969.47 total 471.11 last 119.89) - open/close 58859 (time 1746154979.48 total 481.12 last 131.96) - open/close 60000 (time 1746154987.68 total 489.32 last 139.15) - open/close 61429 (time 1746154997.68 total 499.32 last 142.89) - open/close 62860 (time 1746155007.68 total 509.32 last 143.10) - open/close 64136 (time 1746155017.69 total 519.32 last 127.55) total: 64768 open/close in 524.49 seconds: 123.49 ops/second - /unlink 2163 (time 1746155034.82 total 10.00 last 216.22) - /unlink 4422 (time 1746155044.82 total 20.01 last 225.82) - /unlink 6585 (time 1746155054.82 total 30.01 last 216.26) - /unlink 8498 (time 1746155064.83 total 40.01 last 191.25) - /unlink 10000 (time 1746155072.19 total 47.38 last 203.87) - /unlink 12053 (time 1746155082.19 total 57.38 last 205.27) - /unlink 14070 (time 1746155092.20 total 67.38 last 201.68) - /unlink 15986 (time 1746155102.20 total 77.38 last 191.60) - /unlink 17936 (time 1746155112.20 total 87.38 last 195.00) - /unlink 20000 (time 1746155121.97 total 97.15 last 211.19) - /unlink 22064 (time 1746155131.97 total 107.16 last 206.36) - /unlink 23864 (time 1746155141.98 total 117.16 last 179.89) - /unlink 25505 (time 1746155151.98 total 127.16 last 164.10) - /unlink 27277 (time 1746155161.98 total 137.17 last 177.15) - /unlink 29204 (time 1746155171.98 total 147.17 last 192.64) - /unlink 30000 (time 1746155175.23 total 150.42 last 244.86) - /unlink 32490 (time 1746155185.24 total 160.42 last 248.90) - /unlink 34640 (time 1746155195.24 total 170.43 last 214.95) - /unlink 36734 (time 1746155205.24 total 180.43 last 209.39) - /unlink 38463 (time 1746155215.24 total 190.43 last 172.89) - /unlink 40000 (time 1746155223.41 total 198.60 last 188.14) - /unlink 41749 (time 1746155233.41 total 208.60 last 174.85) - /unlink 43510 (time 1746155243.42 total 218.60 last 176.05) - /unlink 45431 (time 1746155253.42 total 228.60 last 192.07) - /unlink 47472 (time 1746155263.42 total 238.60 last 204.09) - /unlink 49763 (time 1746155273.42 total 248.61 last 229.06) - /unlink 51815 (time 1746155283.43 total 258.61 last 205.08) - /unlink 53828 (time 1746155293.43 total 268.61 last 201.26) - /unlink 55793 (time 1746155303.43 total 278.62 last 196.42) - /unlink 57884 (time 1746155313.43 total 288.62 last 209.08) - /unlink 60000 (time 1746155323.40 total 298.59 last 212.29) - /unlink 62251 (time 1746155333.40 total 308.59 last 225.07) - /unlink 64564 (time 1746155343.41 total 318.60 last 231.11) total: 64768 /unlink in 319.46 seconds: 202.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1331 (time 1746155374.31 total 10.01 last 133.03) - open/close 2651 (time 1746155384.33 total 20.02 last 131.81) - open/close 3906 (time 1746155394.34 total 30.03 last 125.39) - open/close 5166 (time 1746155404.34 total 40.03 last 125.94) - open/close 6472 (time 1746155414.35 total 50.04 last 130.53) - open/close 7778 (time 1746155424.35 total 60.04 last 130.57) - open/close 9020 (time 1746155434.35 total 70.04 last 124.14) - open/close 10000 (time 1746155442.74 total 78.43 last 116.88) - open/close 11355 (time 1746155452.74 total 88.43 last 135.44) - open/close 12644 (time 1746155462.74 total 98.44 last 128.89) - open/close 13841 (time 1746155472.75 total 108.44 last 119.66) - open/close 15125 (time 1746155482.76 total 118.45 last 128.23) - open/close 16432 (time 1746155492.77 total 128.46 last 130.58) - open/close 17623 (time 1746155502.78 total 138.47 last 119.03) - open/close 18900 (time 1746155512.78 total 148.47 last 127.64) - open/close 20000 (time 1746155521.17 total 156.86 last 131.17) - open/close 21247 (time 1746155531.17 total 166.86 last 124.68) - open/close 22531 (time 1746155541.18 total 176.87 last 128.31) - open/close 23696 (time 1746155551.18 total 186.87 last 116.46) - open/close 24964 (time 1746155561.18 total 196.87 last 126.76) - open/close 26134 (time 1746155571.18 total 206.87 last 116.97) - open/close 27267 (time 1746155581.19 total 216.88 last 113.20) - open/close 28429 (time 1746155591.20 total 226.89 last 116.15) - open/close 29838 (time 1746155601.20 total 236.89 last 140.87) - open/close 31347 (time 1746155611.20 total 246.89 last 150.84) - open/close 32669 (time 1746155621.20 total 256.89 last 132.19) - open/close 34077 (time 1746155631.21 total 266.90 last 140.75) - open/close 35583 (time 1746155641.21 total 276.90 last 150.59) - open/close 37128 (time 1746155651.21 total 286.90 last 154.50) - open/close 38485 (time 1746155661.21 total 296.90 last 135.64) - open/close 39796 (time 1746155671.22 total 306.91 last 131.03) - open/close 41137 (time 1746155681.22 total 316.91 last 134.06) - open/close 42663 (time 1746155691.23 total 326.92 last 152.50) - open/close 43905 (time 1746155701.24 total 336.93 last 124.07) - open/close 45144 (time 1746155711.24 total 346.93 last 123.87) - open/close 46380 (time 1746155721.24 total 356.93 last 123.56) - open/close 47629 (time 1746155731.25 total 366.94 last 124.83) - open/close 48935 (time 1746155741.25 total 376.94 last 130.56) - open/close 50000 (time 1746155749.22 total 384.91 last 133.60) - open/close 51270 (time 1746155759.22 total 394.91 last 127.00) - open/close 52648 (time 1746155769.23 total 404.92 last 137.75) - open/close 53993 (time 1746155779.23 total 414.92 last 134.49) - open/close 55468 (time 1746155789.23 total 424.92 last 147.45) - open/close 57122 (time 1746155799.24 total 434.93 last 165.34) - open/close 58706 (time 1746155809.24 total 444.93 last 158.33) - open/close 60000 (time 1746155816.79 total 452.48 last 171.33) - open/close 61516 (time 1746155826.80 total 462.49 last 151.51) - open/close 62735 (time 1746155836.80 total 472.49 last 121.85) - open/close 64195 (time 1746155846.81 total 482.50 last 145.96) total: 64768 open/close in 487.38 seconds: 132.89 ops/second - /unlink 1840 (time 1746155863.76 total 10.01 last 183.89) - /unlink 3446 (time 1746155873.76 total 20.01 last 160.58) - /unlink 5133 (time 1746155883.76 total 30.01 last 168.68) - /unlink 7013 (time 1746155893.78 total 40.03 last 187.66) - /unlink 9062 (time 1746155903.79 total 50.03 last 204.83) - /unlink 10000 (time 1746155907.24 total 53.49 last 271.28) - /unlink 11958 (time 1746155917.25 total 63.49 last 195.69) - /unlink 13584 (time 1746155927.25 total 73.50 last 162.57) - /unlink 15228 (time 1746155937.25 total 83.50 last 164.37) - /unlink 16903 (time 1746155947.25 total 93.50 last 167.47) - /unlink 18549 (time 1746155957.26 total 103.50 last 164.53) - /unlink 20000 (time 1746155965.86 total 112.11 last 168.62) - /unlink 21623 (time 1746155975.87 total 122.11 last 162.25) - /unlink 23265 (time 1746155985.87 total 132.11 last 164.20) - /unlink 25041 (time 1746155995.87 total 142.12 last 177.53) - /unlink 26756 (time 1746156005.87 total 152.12 last 171.50) - /unlink 28489 (time 1746156015.87 total 162.12 last 173.26) - /unlink 30000 (time 1746156024.88 total 171.13 last 167.74) - /unlink 31821 (time 1746156034.89 total 181.13 last 182.01) - /unlink 33846 (time 1746156044.89 total 191.13 last 202.50) - /unlink 35915 (time 1746156054.89 total 201.14 last 206.74) - /unlink 38101 (time 1746156064.90 total 211.14 last 218.54) - /unlink 40000 (time 1746156073.86 total 220.11 last 211.78) - /unlink 42221 (time 1746156083.86 total 230.11 last 222.08) - /unlink 44416 (time 1746156093.87 total 240.11 last 219.43) - /unlink 47125 (time 1746156103.87 total 250.12 last 270.79) - /unlink 49219 (time 1746156113.87 total 260.12 last 209.35) - /unlink 50000 (time 1746156117.63 total 263.87 last 208.10) - /unlink 52756 (time 1746156127.63 total 273.87 last 275.53) - /unlink 55460 (time 1746156137.63 total 283.88 last 270.36) - /unlink 58075 (time 1746156147.63 total 293.88 last 261.44) - /unlink 60000 (time 1746156156.69 total 302.94 last 212.49) - /unlink 62075 (time 1746156166.70 total 312.95 last 207.31) - /unlink 64411 (time 1746156176.70 total 322.95 last 233.58) total: 64768 /unlink in 324.72 seconds: 199.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1251 (time 1746156206.67 total 10.01 last 125.01) - open/close 2616 (time 1746156216.67 total 20.01 last 136.45) - open/close 4012 (time 1746156226.68 total 30.02 last 139.47) - open/close 5330 (time 1746156236.69 total 40.02 last 131.75) - open/close 6652 (time 1746156246.69 total 50.03 last 132.15) - open/close 7899 (time 1746156256.69 total 60.03 last 124.69) - open/close 9239 (time 1746156266.70 total 70.03 last 133.92) - open/close 10000 (time 1746156272.47 total 75.80 last 131.90) - open/close 11224 (time 1746156282.47 total 85.81 last 122.33) - open/close 12517 (time 1746156292.47 total 95.81 last 129.27) - open/close 13841 (time 1746156302.48 total 105.82 last 132.29) - open/close 15154 (time 1746156312.49 total 115.82 last 131.26) - open/close 16476 (time 1746156322.49 total 125.83 last 132.16) - open/close 17719 (time 1746156332.49 total 135.83 last 124.27) - open/close 19064 (time 1746156342.49 total 145.83 last 134.49) - open/close 20000 (time 1746156350.03 total 153.37 last 124.11) - open/close 21251 (time 1746156360.04 total 163.38 last 125.01) - open/close 22639 (time 1746156370.04 total 173.38 last 138.77) - open/close 23898 (time 1746156380.04 total 183.38 last 125.86) - open/close 25106 (time 1746156390.05 total 193.38 last 120.77) - open/close 26304 (time 1746156400.05 total 203.39 last 119.80) - open/close 27438 (time 1746156410.05 total 213.39 last 113.33) - open/close 28616 (time 1746156420.06 total 223.39 last 117.77) - open/close 29877 (time 1746156430.06 total 233.39 last 126.09) - open/close 31189 (time 1746156440.06 total 243.40 last 131.13) - open/close 32355 (time 1746156450.07 total 253.41 last 116.48) - open/close 33691 (time 1746156460.08 total 263.41 last 133.55) - open/close 34902 (time 1746156470.08 total 273.42 last 121.01) - open/close 36144 (time 1746156480.09 total 283.43 last 124.07) - open/close 37408 (time 1746156490.10 total 293.44 last 126.31) - open/close 38762 (time 1746156500.11 total 303.44 last 135.33) - open/close 40000 (time 1746156510.06 total 313.40 last 124.35) - open/close 41341 (time 1746156520.07 total 323.40 last 134.02) - open/close 42574 (time 1746156530.07 total 333.41 last 123.25) - open/close 43710 (time 1746156540.07 total 343.41 last 113.57) - open/close 44974 (time 1746156550.08 total 353.42 last 126.34) - open/close 46134 (time 1746156560.09 total 363.43 last 115.89) - open/close 47334 (time 1746156570.09 total 373.43 last 119.99) - open/close 48658 (time 1746156580.09 total 383.43 last 132.37) - open/close 50000 (time 1746156589.49 total 392.82 last 142.84) - open/close 51603 (time 1746156599.49 total 402.83 last 160.26) - open/close 53477 (time 1746156609.49 total 412.83 last 187.33) - open/close 55123 (time 1746156619.49 total 422.83 last 164.57) - open/close 56661 (time 1746156629.49 total 432.83 last 153.80) - open/close 58439 (time 1746156639.50 total 442.84 last 177.71) - open/close 59903 (time 1746156649.51 total 452.84 last 146.28) - open/close 61438 (time 1746156659.51 total 462.85 last 153.48) - open/close 62892 (time 1746156669.51 total 472.85 last 145.38) - open/close 64313 (time 1746156679.51 total 482.85 last 142.04) total: 64768 open/close in 485.87 seconds: 133.30 ops/second - /unlink 2571 (time 1746156693.85 total 10.00 last 257.04) - /unlink 5581 (time 1746156703.85 total 20.01 last 300.91) - /unlink 8900 (time 1746156713.85 total 30.01 last 331.84) - /unlink 10000 (time 1746156718.42 total 34.57 last 240.97) - /unlink 12683 (time 1746156728.42 total 44.57 last 268.27) - /unlink 15202 (time 1746156738.42 total 54.57 last 251.89) - /unlink 17766 (time 1746156748.43 total 64.58 last 256.22) - /unlink 20000 (time 1746156757.52 total 73.68 last 245.61) - /unlink 22329 (time 1746156767.53 total 83.68 last 232.74) - /unlink 25021 (time 1746156777.53 total 93.68 last 269.17) - /unlink 28554 (time 1746156787.53 total 103.69 last 353.22) - /unlink 30000 (time 1746156791.22 total 107.37 last 392.41) - /unlink 33594 (time 1746156801.22 total 117.38 last 359.22) - /unlink 37820 (time 1746156811.22 total 127.38 last 422.51) - /unlink 40000 (time 1746156815.48 total 131.63 last 512.34) - /unlink 43657 (time 1746156825.48 total 141.63 last 365.69) - /unlink 48053 (time 1746156835.48 total 151.63 last 439.58) - /unlink 50000 (time 1746156840.30 total 156.46 last 403.45) - /unlink 53677 (time 1746156850.31 total 166.46 last 367.65) - /unlink 57806 (time 1746156860.31 total 176.46 last 412.89) - /unlink 60000 (time 1746156865.00 total 181.15 last 467.54) - /unlink 63600 (time 1746156875.00 total 191.16 last 359.83) total: 64768 /unlink in 194.89 seconds: 332.32 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1489 (time 1746156906.65 total 10.00 last 148.87) - open/close 3334 (time 1746156916.65 total 20.00 last 184.45) - open/close 4990 (time 1746156926.66 total 30.01 last 165.51) - open/close 6977 (time 1746156936.66 total 40.01 last 198.62) - open/close 8554 (time 1746156946.66 total 50.02 last 157.68) - open/close 10000 (time 1746156955.69 total 59.04 last 160.27) - open/close 11766 (time 1746156965.69 total 69.04 last 176.51) - open/close 13739 (time 1746156975.69 total 79.05 last 197.23) - open/close 16118 (time 1746156985.70 total 89.05 last 237.88) - open/close 17867 (time 1746156995.70 total 99.05 last 174.88) - open/close 19264 (time 1746157005.72 total 109.07 last 139.40) - open/close 20000 (time 1746157010.88 total 114.23 last 142.54) - open/close 21741 (time 1746157020.88 total 124.23 last 174.06) - open/close 23124 (time 1746157030.89 total 134.24 last 138.26) - open/close 24553 (time 1746157040.89 total 144.24 last 142.89) - open/close 25741 (time 1746157050.89 total 154.24 last 118.79) - open/close 27141 (time 1746157060.89 total 164.24 last 139.97) - open/close 28572 (time 1746157070.90 total 174.25 last 143.02) - open/close 30000 (time 1746157079.63 total 182.98 last 163.56) - open/close 31385 (time 1746157089.63 total 192.99 last 138.40) - open/close 32756 (time 1746157099.64 total 202.99 last 137.05) - open/close 34321 (time 1746157109.64 total 212.99 last 156.44) - open/close 35688 (time 1746157119.64 total 223.00 last 136.65) - open/close 37135 (time 1746157129.65 total 233.00 last 144.68) - open/close 38536 (time 1746157139.65 total 243.00 last 140.04) - open/close 40000 (time 1746157148.51 total 251.86 last 165.26) - open/close 42036 (time 1746157158.51 total 261.86 last 203.55) - open/close 43580 (time 1746157168.52 total 271.87 last 154.25) - open/close 45326 (time 1746157178.52 total 281.87 last 174.59) - open/close 47081 (time 1746157188.52 total 291.88 last 175.46) - open/close 48845 (time 1746157198.53 total 301.88 last 176.31) - open/close 50000 (time 1746157205.70 total 309.05 last 161.04) - open/close 51557 (time 1746157215.71 total 319.06 last 155.63) - open/close 53038 (time 1746157225.71 total 329.06 last 148.03) - open/close 54492 (time 1746157235.72 total 339.07 last 145.30) - open/close 55862 (time 1746157245.72 total 349.07 last 136.94) - open/close 57171 (time 1746157255.72 total 359.08 last 130.87) - open/close 58490 (time 1746157265.73 total 369.08 last 131.81) - open/close 59919 (time 1746157275.74 total 379.10 last 142.70) - open/close 61325 (time 1746157285.75 total 389.10 last 140.57) - open/close 62809 (time 1746157295.75 total 399.10 last 148.38) - open/close 64101 (time 1746157305.75 total 409.11 last 129.13) total: 64768 open/close in 414.43 seconds: 156.28 ops/second - /unlink 2107 (time 1746157322.69 total 10.01 last 210.51) - /unlink 4337 (time 1746157332.69 total 20.01 last 222.93) - /unlink 6773 (time 1746157342.69 total 30.01 last 243.60) - /unlink 8639 (time 1746157352.70 total 40.02 last 186.50) - /unlink 10000 (time 1746157360.57 total 47.89 last 172.87) - /unlink 11836 (time 1746157370.57 total 57.89 last 183.58) - /unlink 13577 (time 1746157380.58 total 67.90 last 173.96) - /unlink 15415 (time 1746157390.58 total 77.91 last 183.69) - /unlink 17421 (time 1746157400.59 total 87.91 last 200.44) - /unlink 19293 (time 1746157410.60 total 97.92 last 187.12) - /unlink 20000 (time 1746157413.62 total 100.95 last 233.54) - /unlink 21970 (time 1746157423.63 total 110.95 last 196.88) - /unlink 24798 (time 1746157433.63 total 120.95 last 282.71) - /unlink 27661 (time 1746157443.63 total 130.96 last 286.27) - /unlink 30000 (time 1746157452.05 total 139.37 last 277.85) - /unlink 32273 (time 1746157462.06 total 149.38 last 227.16) - /unlink 34634 (time 1746157472.06 total 159.38 last 236.09) - /unlink 37047 (time 1746157482.06 total 169.38 last 241.24) - /unlink 39250 (time 1746157492.06 total 179.38 last 220.27) - /unlink 40000 (time 1746157495.88 total 183.20 last 196.56) - /unlink 42378 (time 1746157505.88 total 193.20 last 237.75) - /unlink 44876 (time 1746157515.88 total 203.20 last 249.78) - /unlink 47233 (time 1746157525.88 total 213.21 last 235.61) - /unlink 50000 (time 1746157535.04 total 222.37 last 302.11) - /unlink 52999 (time 1746157545.04 total 232.37 last 299.86) - /unlink 55362 (time 1746157555.05 total 242.37 last 236.26) - /unlink 57541 (time 1746157565.05 total 252.37 last 217.83) - /unlink 59889 (time 1746157575.05 total 262.37 last 234.79) - /unlink 63275 (time 1746157585.05 total 272.37 last 338.58) total: 64768 /unlink in 276.55 seconds: 234.20 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1767 (time 1746157618.48 total 10.00 last 176.68) - open/close 3966 (time 1746157628.48 total 20.00 last 219.85) - open/close 6710 (time 1746157638.49 total 30.01 last 274.29) - open/close 8338 (time 1746157648.49 total 40.01 last 162.80) - open/close 9949 (time 1746157658.49 total 50.01 last 161.00) - open/close 12100 (time 1746157668.49 total 60.01 last 215.09) - open/close 13964 (time 1746157678.49 total 70.02 last 186.38) - open/close 15439 (time 1746157688.51 total 80.03 last 147.32) - open/close 16774 (time 1746157698.51 total 90.03 last 133.48) - open/close 18024 (time 1746157708.52 total 100.04 last 124.89) - open/close 19261 (time 1746157718.52 total 110.04 last 123.66) - open/close 20000 (time 1746157725.13 total 116.65 last 111.79) - open/close 21206 (time 1746157735.13 total 126.65 last 120.57) - open/close 22337 (time 1746157745.14 total 136.66 last 113.06) - open/close 23202 (time 1746157755.14 total 146.66 last 86.44) - open/close 24046 (time 1746157765.15 total 156.68 last 84.31) - open/close 24896 (time 1746157775.16 total 166.68 last 84.98) - open/close 26206 (time 1746157785.16 total 176.68 last 130.99) - open/close 28036 (time 1746157795.16 total 186.68 last 182.93) - open/close 29720 (time 1746157805.16 total 196.69 last 168.34) - open/close 31221 (time 1746157815.17 total 206.69 last 150.08) - open/close 32927 (time 1746157825.17 total 216.69 last 170.59) - open/close 34860 (time 1746157835.17 total 226.69 last 193.26) - open/close 36163 (time 1746157845.18 total 236.70 last 130.15) - open/close 37296 (time 1746157855.19 total 246.71 last 113.22) - open/close 38803 (time 1746157865.19 total 256.71 last 150.62) - open/close 40000 (time 1746157873.46 total 264.98 last 144.77) - open/close 41376 (time 1746157883.46 total 274.98 last 137.58) - open/close 42579 (time 1746157893.46 total 284.98 last 120.28) - open/close 43850 (time 1746157903.47 total 294.99 last 127.03) - open/close 45380 (time 1746157913.47 total 304.99 last 152.96) - open/close 46625 (time 1746157923.47 total 314.99 last 124.48) - open/close 47810 (time 1746157933.47 total 324.99 last 118.50) - open/close 48937 (time 1746157943.48 total 335.00 last 112.61) - open/close 50000 (time 1746157952.57 total 344.09 last 116.99) - open/close 51321 (time 1746157962.57 total 354.09 last 132.07) - open/close 52524 (time 1746157972.58 total 364.10 last 120.18) - open/close 53685 (time 1746157982.59 total 374.11 last 116.04) - open/close 54815 (time 1746157992.60 total 384.12 last 112.86) - open/close 55637 (time 1746158002.61 total 394.13 last 82.11) - open/close 56431 (time 1746158012.62 total 404.14 last 79.28) - open/close 57196 (time 1746158022.63 total 414.15 last 76.49) - open/close 58105 (time 1746158032.63 total 424.15 last 90.88) - open/close 59036 (time 1746158042.65 total 434.17 last 92.93) - open/close 60000 (time 1746158052.07 total 443.59 last 102.34) - open/close 61098 (time 1746158062.08 total 453.60 last 109.68) - open/close 61999 (time 1746158072.09 total 463.61 last 90.02) - open/close 62968 (time 1746158082.09 total 473.61 last 96.86) - open/close 64005 (time 1746158092.10 total 483.62 last 103.63) total: 64768 open/close in 490.76 seconds: 131.97 ops/second - /unlink 1706 (time 1746158111.38 total 10.01 last 170.49) - /unlink 3783 (time 1746158121.38 total 20.01 last 207.68) - /unlink 5640 (time 1746158131.38 total 30.01 last 185.63) - /unlink 7342 (time 1746158141.41 total 40.04 last 169.69) - /unlink 9348 (time 1746158151.42 total 50.05 last 200.48) - /unlink 10000 (time 1746158155.08 total 53.71 last 178.20) - /unlink 11634 (time 1746158165.09 total 63.72 last 163.19) - /unlink 13265 (time 1746158175.10 total 73.73 last 162.96) - /unlink 14987 (time 1746158185.10 total 83.73 last 172.19) - /unlink 16372 (time 1746158195.10 total 93.73 last 138.50) - /unlink 17946 (time 1746158205.10 total 103.73 last 157.37) - /unlink 19620 (time 1746158215.11 total 113.73 last 167.36) - /unlink 20000 (time 1746158217.20 total 115.83 last 181.22) - /unlink 21721 (time 1746158227.20 total 125.83 last 172.09) - /unlink 23632 (time 1746158237.21 total 135.84 last 190.99) - /unlink 25843 (time 1746158247.21 total 145.84 last 221.06) - /unlink 28173 (time 1746158257.21 total 155.84 last 232.91) - /unlink 30000 (time 1746158265.69 total 164.32 last 215.43) - /unlink 32178 (time 1746158275.70 total 174.32 last 217.78) - /unlink 34268 (time 1746158285.70 total 184.32 last 208.98) - /unlink 36788 (time 1746158295.70 total 194.33 last 251.97) - /unlink 39436 (time 1746158305.70 total 204.33 last 264.71) - /unlink 40000 (time 1746158308.29 total 206.92 last 217.76) - /unlink 42391 (time 1746158318.30 total 216.92 last 239.02) - /unlink 44745 (time 1746158328.30 total 226.92 last 235.36) - /unlink 47131 (time 1746158338.30 total 236.93 last 238.52) - /unlink 49666 (time 1746158348.30 total 246.93 last 253.46) - /unlink 52118 (time 1746158358.30 total 256.93 last 245.15) - /unlink 55054 (time 1746158368.31 total 266.94 last 293.43) - /unlink 57240 (time 1746158378.31 total 276.94 last 218.58) - /unlink 59475 (time 1746158388.31 total 286.94 last 223.46) - /unlink 61810 (time 1746158398.31 total 296.94 last 233.49) - /unlink 64391 (time 1746158408.32 total 306.94 last 258.01) total: 64768 /unlink in 308.26 seconds: 210.11 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server unloading modules via unload_modules_local on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2