== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:45:02 (1717011902) Stopping clients: oleg234-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg234-client.virtnet /mnt/lustre2 (opts:-f) oleg234-server: oleg234-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: detected 4 online CPUs by sysfs oleg234-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg234-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg234-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock oleg234-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1229 (time 1717011962.25 total 10.00 last 122.86) - open/close 2590 (time 1717011972.25 total 20.01 last 136.04) - open/close 3956 (time 1717011982.26 total 30.01 last 136.53) - open/close 5374 (time 1717011992.26 total 40.02 last 141.73) - open/close 6622 (time 1717012002.27 total 50.03 last 124.70) - open/close 7840 (time 1717012012.27 total 60.03 last 121.78) - open/close 9038 (time 1717012022.27 total 70.03 last 119.79) - open/close 10000 (time 1717012030.22 total 77.97 last 121.08) - open/close 11192 (time 1717012040.23 total 87.98 last 119.09) - open/close 12429 (time 1717012050.23 total 97.98 last 123.65) - open/close 13650 (time 1717012060.23 total 107.99 last 122.09) - open/close 14813 (time 1717012070.24 total 117.99 last 116.20) - open/close 15975 (time 1717012080.24 total 128.00 last 116.15) - open/close 17204 (time 1717012090.25 total 138.00 last 122.87) - open/close 18446 (time 1717012100.25 total 148.00 last 124.20) - open/close 19826 (time 1717012110.26 total 158.01 last 137.86) - open/close 21122 (time 1717012120.26 total 168.02 last 129.54) - open/close 22351 (time 1717012130.26 total 178.02 last 122.88) - open/close 23826 (time 1717012140.27 total 188.02 last 147.42) - open/close 25463 (time 1717012150.27 total 198.03 last 163.62) - open/close 26923 (time 1717012160.28 total 208.03 last 145.96) - open/close 28320 (time 1717012170.28 total 218.04 last 139.60) - open/close 29775 (time 1717012180.28 total 228.04 last 145.48) - open/close 31131 (time 1717012190.29 total 238.05 last 135.49) - open/close 32585 (time 1717012200.30 total 248.05 last 145.27) - open/close 33888 (time 1717012210.30 total 258.06 last 130.27) - open/close 35192 (time 1717012220.31 total 268.06 last 130.37) - open/close 36457 (time 1717012230.31 total 278.07 last 126.43) - open/close 37670 (time 1717012240.32 total 288.07 last 121.24) - open/close 38899 (time 1717012250.32 total 298.07 last 122.89) - open/close 40000 (time 1717012260.01 total 307.76 last 113.58) - open/close 41270 (time 1717012270.02 total 317.77 last 126.88) - open/close 42671 (time 1717012280.03 total 327.78 last 139.98) - open/close 43875 (time 1717012290.04 total 337.79 last 120.32) - open/close 45178 (time 1717012300.04 total 347.79 last 130.28) - open/close 46513 (time 1717012310.04 total 357.80 last 133.42) - open/close 47725 (time 1717012320.05 total 367.80 last 121.16) - open/close 48980 (time 1717012330.05 total 377.81 last 125.42) - open/close 50000 (time 1717012337.94 total 385.69 last 129.29) - open/close 51332 (time 1717012347.95 total 395.70 last 133.10) - open/close 52750 (time 1717012357.95 total 405.70 last 141.79) - open/close 54057 (time 1717012367.96 total 415.71 last 130.59) - open/close 55345 (time 1717012377.96 total 425.72 last 128.71) - open/close 56740 (time 1717012387.98 total 435.73 last 139.35) - open/close 58069 (time 1717012397.98 total 445.73 last 132.84) - open/close 59536 (time 1717012407.99 total 455.74 last 146.57) - open/close 60000 (time 1717012411.60 total 459.35 last 128.66) - open/close 61318 (time 1717012421.60 total 469.35 last 131.78) - open/close 62820 (time 1717012431.60 total 479.36 last 150.09) - open/close 64118 (time 1717012441.61 total 489.36 last 129.76) total: 64768 open/close in 494.66 seconds: 130.93 ops/second - /unlink 2014 (time 1717012458.63 total 10.00 last 201.36) - /unlink 4370 (time 1717012468.63 total 20.01 last 235.46) - /unlink 6650 (time 1717012478.63 total 30.01 last 227.95) - /unlink 8978 (time 1717012488.64 total 40.01 last 232.76) - /unlink 10000 (time 1717012493.09 total 44.46 last 229.50) - /unlink 12249 (time 1717012503.09 total 54.47 last 224.82) - /unlink 14340 (time 1717012513.09 total 64.47 last 209.07) - /unlink 16402 (time 1717012523.10 total 74.47 last 206.12) - /unlink 18364 (time 1717012533.10 total 84.48 last 196.15) - /unlink 20000 (time 1717012541.00 total 92.38 last 207.04) - /unlink 21799 (time 1717012551.01 total 102.38 last 179.83) - /unlink 23885 (time 1717012561.01 total 112.38 last 208.55) - /unlink 25890 (time 1717012571.02 total 122.39 last 200.37) - /unlink 28051 (time 1717012581.02 total 132.39 last 216.06) - /unlink 30000 (time 1717012588.45 total 139.82 last 262.24) - /unlink 32167 (time 1717012598.45 total 149.83 last 216.66) - /unlink 34233 (time 1717012608.45 total 159.83 last 206.52) - /unlink 36585 (time 1717012618.45 total 169.83 last 235.19) - /unlink 38957 (time 1717012628.46 total 179.83 last 237.18) - /unlink 40000 (time 1717012632.85 total 184.22 last 237.51) - /unlink 42211 (time 1717012642.85 total 194.22 last 221.08) - /unlink 44356 (time 1717012652.86 total 204.23 last 214.29) - /unlink 46291 (time 1717012662.86 total 214.24 last 193.38) - /unlink 48495 (time 1717012672.86 total 224.24 last 220.37) - /unlink 50000 (time 1717012679.06 total 230.44 last 242.83) - /unlink 52394 (time 1717012689.06 total 240.44 last 239.40) - /unlink 55055 (time 1717012699.06 total 250.44 last 266.10) - /unlink 57473 (time 1717012709.06 total 260.44 last 241.79) - /unlink 59790 (time 1717012719.07 total 270.44 last 231.66) - /unlink 62054 (time 1717012729.07 total 280.44 last 226.39) - /unlink 64310 (time 1717012739.07 total 290.44 last 225.56) total: 64768 /unlink in 292.20 seconds: 221.66 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 1396 (time 1717012769.35 total 10.00 last 139.54) - open/close 2830 (time 1717012779.35 total 20.01 last 143.36) - open/close 4068 (time 1717012789.37 total 30.02 last 123.61) - open/close 5310 (time 1717012799.37 total 40.03 last 124.10) - open/close 6537 (time 1717012809.38 total 50.03 last 122.67) - open/close 7826 (time 1717012819.39 total 60.04 last 128.77) - open/close 8993 (time 1717012829.39 total 70.05 last 116.66) - open/close 10000 (time 1717012837.90 total 78.55 last 118.37) - open/close 11239 (time 1717012847.90 total 88.56 last 123.87) - open/close 12491 (time 1717012857.90 total 98.56 last 125.15) - open/close 13758 (time 1717012867.91 total 108.57 last 126.62) - open/close 15028 (time 1717012877.91 total 118.57 last 126.99) - open/close 16295 (time 1717012887.91 total 128.57 last 126.69) - open/close 17576 (time 1717012897.91 total 138.57 last 128.07) - open/close 18842 (time 1717012907.92 total 148.58 last 126.52) - open/close 20000 (time 1717012917.27 total 157.92 last 123.90) - open/close 21198 (time 1717012927.27 total 167.93 last 119.69) - open/close 22289 (time 1717012937.28 total 177.93 last 109.08) - open/close 23412 (time 1717012947.28 total 187.94 last 112.25) - open/close 24541 (time 1717012957.29 total 197.95 last 112.76) - open/close 25704 (time 1717012967.30 total 207.95 last 116.28) - open/close 26842 (time 1717012977.30 total 217.96 last 113.74) - open/close 28092 (time 1717012987.31 total 227.96 last 124.94) - open/close 29251 (time 1717012997.31 total 237.96 last 115.90) - open/close 30000 (time 1717013003.53 total 244.19 last 120.26) - open/close 31210 (time 1717013013.54 total 254.20 last 120.88) - open/close 32403 (time 1717013023.55 total 264.21 last 119.25) - open/close 33625 (time 1717013033.56 total 274.21 last 122.09) - open/close 34862 (time 1717013043.56 total 284.22 last 123.68) - open/close 36208 (time 1717013053.56 total 294.22 last 134.57) - open/close 37548 (time 1717013063.56 total 304.22 last 133.98) - open/close 38963 (time 1717013073.57 total 314.22 last 141.47) - open/close 40000 (time 1717013081.59 total 322.24 last 129.29) - open/close 41428 (time 1717013091.59 total 332.25 last 142.77) - open/close 42708 (time 1717013101.59 total 342.25 last 127.96) - open/close 43884 (time 1717013111.60 total 352.25 last 117.53) - open/close 44990 (time 1717013121.60 total 362.26 last 110.57) - open/close 46102 (time 1717013131.61 total 372.27 last 111.09) - open/close 47326 (time 1717013141.62 total 382.28 last 122.29) - open/close 48499 (time 1717013151.62 total 392.28 last 117.27) - open/close 49736 (time 1717013161.62 total 402.28 last 123.70) - open/close 50000 (time 1717013163.88 total 404.54 last 116.84) - open/close 51292 (time 1717013173.89 total 414.54 last 129.15) - open/close 52651 (time 1717013183.89 total 424.55 last 135.84) - open/close 53848 (time 1717013193.90 total 434.55 last 119.63) - open/close 54960 (time 1717013203.90 total 444.56 last 111.12) - open/close 56015 (time 1717013213.91 total 454.56 last 105.47) - open/close 57179 (time 1717013223.91 total 464.57 last 116.33) - open/close 58399 (time 1717013233.91 total 474.57 last 121.98) - open/close 59625 (time 1717013243.92 total 484.57 last 122.57) - open/close 60000 (time 1717013246.85 total 487.51 last 127.76) - open/close 61286 (time 1717013256.86 total 497.51 last 128.53) - open/close 62520 (time 1717013266.86 total 507.52 last 123.37) - open/close 63720 (time 1717013276.87 total 517.52 last 119.93) total: 64768 open/close in 525.61 seconds: 123.22 ops/second - /unlink 2290 (time 1717013296.55 total 10.01 last 228.87) - /unlink 4447 (time 1717013306.55 total 20.01 last 215.68) - /unlink 6695 (time 1717013316.55 total 30.01 last 224.77) - /unlink 9087 (time 1717013326.55 total 40.01 last 239.19) - /unlink 10000 (time 1717013330.57 total 44.02 last 227.54) - /unlink 11979 (time 1717013340.57 total 54.02 last 197.87) - /unlink 14142 (time 1717013350.57 total 64.02 last 216.25) - /unlink 16119 (time 1717013360.57 total 74.03 last 197.69) - /unlink 18230 (time 1717013370.57 total 84.03 last 211.03) - /unlink 20000 (time 1717013378.56 total 92.01 last 221.65) - /unlink 21851 (time 1717013388.56 total 102.02 last 185.02) - /unlink 23743 (time 1717013398.57 total 112.02 last 189.16) - /unlink 25655 (time 1717013408.57 total 122.02 last 191.13) - /unlink 27495 (time 1717013418.57 total 132.03 last 183.96) - /unlink 29508 (time 1717013428.57 total 142.03 last 201.26) - /unlink 30000 (time 1717013430.88 total 144.33 last 213.27) - /unlink 31885 (time 1717013440.88 total 154.34 last 188.44) - /unlink 33871 (time 1717013450.89 total 164.34 last 198.55) - /unlink 35730 (time 1717013460.89 total 174.34 last 185.82) - /unlink 37466 (time 1717013470.89 total 184.35 last 173.58) - /unlink 39187 (time 1717013480.89 total 194.35 last 172.03) - /unlink 40000 (time 1717013485.57 total 199.03 last 173.80) - /unlink 41891 (time 1717013495.57 total 209.03 last 189.08) - /unlink 43923 (time 1717013505.58 total 219.03 last 203.11) - /unlink 46320 (time 1717013515.58 total 229.03 last 239.70) - /unlink 48663 (time 1717013525.58 total 239.03 last 234.29) - /unlink 50000 (time 1717013531.78 total 245.23 last 215.73) - /unlink 51752 (time 1717013541.78 total 255.24 last 175.09) - /unlink 53644 (time 1717013551.78 total 265.24 last 189.17) - /unlink 55644 (time 1717013561.79 total 275.24 last 199.97) - /unlink 57673 (time 1717013571.79 total 285.24 last 202.84) - /unlink 59562 (time 1717013581.79 total 295.25 last 188.86) - /unlink 60000 (time 1717013584.59 total 298.04 last 156.60) - /unlink 61848 (time 1717013594.59 total 308.05 last 184.71) - /unlink 63707 (time 1717013604.60 total 318.05 last 185.85) total: 64768 /unlink in 323.86 seconds: 199.99 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 1264 (time 1717013640.45 total 10.01 last 126.28) - open/close 2479 (time 1717013650.47 total 20.03 last 121.19) - open/close 3755 (time 1717013660.47 total 30.04 last 127.59) - open/close 4940 (time 1717013670.47 total 40.04 last 118.47) - open/close 6157 (time 1717013680.48 total 50.05 last 121.58) - open/close 7367 (time 1717013690.49 total 60.05 last 120.93) - open/close 8588 (time 1717013700.50 total 70.07 last 121.92) - open/close 9912 (time 1717013710.51 total 80.07 last 132.30) - open/close 11197 (time 1717013720.51 total 90.08 last 128.47) - open/close 12475 (time 1717013730.52 total 100.09 last 127.70) - open/close 13779 (time 1717013740.53 total 110.09 last 130.28) - open/close 14898 (time 1717013750.54 total 120.10 last 111.82) - open/close 16032 (time 1717013760.54 total 130.10 last 113.39) - open/close 17299 (time 1717013770.54 total 140.10 last 126.69) - open/close 18695 (time 1717013780.55 total 150.11 last 139.52) - open/close 20000 (time 1717013790.13 total 159.70 last 136.08) - open/close 21387 (time 1717013800.14 total 169.70 last 138.67) - open/close 22778 (time 1717013810.14 total 179.70 last 139.04) - open/close 24069 (time 1717013820.15 total 189.71 last 128.99) - open/close 25368 (time 1717013830.15 total 199.72 last 129.88) - open/close 26663 (time 1717013840.15 total 209.72 last 129.48) - open/close 27934 (time 1717013850.16 total 219.72 last 127.02) - open/close 29281 (time 1717013860.16 total 229.73 last 134.63) - open/close 30000 (time 1717013866.09 total 235.66 last 121.29) - open/close 31139 (time 1717013876.09 total 245.66 last 113.89) - open/close 32284 (time 1717013886.10 total 255.66 last 114.46) - open/close 33376 (time 1717013896.10 total 265.66 last 109.20) - open/close 34547 (time 1717013906.10 total 275.67 last 117.03) - open/close 35627 (time 1717013916.11 total 285.67 last 107.95) - open/close 36744 (time 1717013926.11 total 295.68 last 111.65) - open/close 37831 (time 1717013936.11 total 305.68 last 108.70) - open/close 38936 (time 1717013946.12 total 315.68 last 110.44) - open/close 40000 (time 1717013955.26 total 324.82 last 116.38) - open/close 41185 (time 1717013965.27 total 334.83 last 118.38) - open/close 42334 (time 1717013975.27 total 344.84 last 114.87) - open/close 43618 (time 1717013985.28 total 354.84 last 128.35) - open/close 44908 (time 1717013995.28 total 364.84 last 128.95) - open/close 46100 (time 1717014005.29 total 374.85 last 119.13) - open/close 47314 (time 1717014015.29 total 384.86 last 121.33) - open/close 48557 (time 1717014025.29 total 394.86 last 124.27) - open/close 49760 (time 1717014035.30 total 404.86 last 120.24) - open/close 50000 (time 1717014037.44 total 407.00 last 112.10) - open/close 51189 (time 1717014047.45 total 417.01 last 118.81) - open/close 52412 (time 1717014057.45 total 427.02 last 122.23) - open/close 53618 (time 1717014067.46 total 437.02 last 120.58) - open/close 54785 (time 1717014077.46 total 447.03 last 116.61) - open/close 55939 (time 1717014087.47 total 457.04 last 115.30) - open/close 57104 (time 1717014097.47 total 467.04 last 116.46) - open/close 58278 (time 1717014107.48 total 477.04 last 117.34) - open/close 59465 (time 1717014117.48 total 487.05 last 118.65) - open/close 60000 (time 1717014121.82 total 491.38 last 123.43) - open/close 61005 (time 1717014131.82 total 501.39 last 100.47) - open/close 62062 (time 1717014141.83 total 511.39 last 105.64) - open/close 63155 (time 1717014151.83 total 521.40 last 109.26) - open/close 64367 (time 1717014161.83 total 531.40 last 121.17) total: 64768 open/close in 534.52 seconds: 121.17 ops/second - /unlink 2295 (time 1717014176.67 total 10.00 last 229.45) - /unlink 4652 (time 1717014186.67 total 20.00 last 235.67) - /unlink 6800 (time 1717014196.67 total 30.00 last 214.78) - /unlink 8898 (time 1717014206.68 total 40.01 last 209.75) - /unlink 10000 (time 1717014211.69 total 45.03 last 219.59) - /unlink 12169 (time 1717014221.70 total 55.03 last 216.88) - /unlink 14963 (time 1717014231.70 total 65.03 last 279.39) - /unlink 17485 (time 1717014241.70 total 75.03 last 252.19) - /unlink 20000 (time 1717014250.82 total 84.15 last 275.76) - /unlink 22527 (time 1717014260.82 total 94.15 last 252.60) - /unlink 24877 (time 1717014270.82 total 104.15 last 234.96) - /unlink 27138 (time 1717014280.82 total 114.15 last 226.09) - /unlink 29428 (time 1717014290.82 total 124.15 last 228.98) - /unlink 30000 (time 1717014293.63 total 126.96 last 203.59) - /unlink 32334 (time 1717014303.64 total 136.97 last 233.29) - /unlink 34597 (time 1717014313.64 total 146.97 last 226.22) - /unlink 36864 (time 1717014323.64 total 156.97 last 226.63) - /unlink 39111 (time 1717014333.64 total 166.98 last 224.69) - /unlink 40000 (time 1717014337.39 total 170.72 last 237.47) - /unlink 42128 (time 1717014347.39 total 180.72 last 212.73) - /unlink 44062 (time 1717014357.40 total 190.73 last 193.28) - /unlink 46260 (time 1717014367.40 total 200.73 last 219.75) - /unlink 48400 (time 1717014377.40 total 210.73 last 213.97) - /unlink 50000 (time 1717014385.26 total 218.59 last 203.55) - /unlink 52070 (time 1717014395.26 total 228.59 last 206.94) - /unlink 53975 (time 1717014405.27 total 238.60 last 190.44) - /unlink 56106 (time 1717014415.27 total 248.60 last 213.10) - /unlink 58000 (time 1717014425.27 total 258.60 last 189.29) - /unlink 59696 (time 1717014435.28 total 268.61 last 169.47) - /unlink 61445 (time 1717014445.28 total 278.61 last 174.90) - /unlink 63376 (time 1717014455.28 total 288.61 last 193.09) total: 64768 /unlink in 295.42 seconds: 219.24 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 1102 (time 1717014491.75 total 10.00 last 110.16) - open/close 2208 (time 1717014501.75 total 20.01 last 110.58) - open/close 3541 (time 1717014511.75 total 30.01 last 133.27) - open/close 4977 (time 1717014521.75 total 40.01 last 143.58) - open/close 6525 (time 1717014531.76 total 50.01 last 154.72) - open/close 7915 (time 1717014541.76 total 60.01 last 139.00) - open/close 9361 (time 1717014551.76 total 70.02 last 144.54) - open/close 10000 (time 1717014555.91 total 74.16 last 154.26) - open/close 11223 (time 1717014565.91 total 84.17 last 122.18) - open/close 12440 (time 1717014575.92 total 94.17 last 121.65) - open/close 13729 (time 1717014585.92 total 104.17 last 128.90) - open/close 15078 (time 1717014595.92 total 114.18 last 134.84) - open/close 16313 (time 1717014605.92 total 124.18 last 123.48) - open/close 17664 (time 1717014615.93 total 134.19 last 135.02) - open/close 18984 (time 1717014625.93 total 144.19 last 131.99) - open/close 20000 (time 1717014632.98 total 151.23 last 144.16) - open/close 21402 (time 1717014642.98 total 161.24 last 140.18) - open/close 22877 (time 1717014652.99 total 171.24 last 147.41) - open/close 24181 (time 1717014662.99 total 181.24 last 130.36) - open/close 25565 (time 1717014672.99 total 191.25 last 138.35) - open/close 26945 (time 1717014683.00 total 201.25 last 137.91) - open/close 28334 (time 1717014693.01 total 211.26 last 138.78) - open/close 29755 (time 1717014703.01 total 221.26 last 142.06) - open/close 31323 (time 1717014713.01 total 231.27 last 156.74) - open/close 32669 (time 1717014723.02 total 241.27 last 134.53) - open/close 33935 (time 1717014733.02 total 251.28 last 126.56) - open/close 35141 (time 1717014743.03 total 261.29 last 120.44) - open/close 36384 (time 1717014753.04 total 271.29 last 124.24) - open/close 37893 (time 1717014763.04 total 281.30 last 150.86) - open/close 39948 (time 1717014773.04 total 291.30 last 205.46) - open/close 41635 (time 1717014783.05 total 301.30 last 168.67) - open/close 43468 (time 1717014793.05 total 311.30 last 183.24) - open/close 45311 (time 1717014803.05 total 321.31 last 184.24) - open/close 47238 (time 1717014813.06 total 331.31 last 192.62) - open/close 48823 (time 1717014823.06 total 341.32 last 158.38) - open/close 50000 (time 1717014831.24 total 349.49 last 143.99) - open/close 51368 (time 1717014841.24 total 359.49 last 136.78) - open/close 52616 (time 1717014851.24 total 369.50 last 124.74) - open/close 53944 (time 1717014861.24 total 379.50 last 132.79) - open/close 55450 (time 1717014871.25 total 389.50 last 150.51) - open/close 56728 (time 1717014881.26 total 399.51 last 127.72) - open/close 57962 (time 1717014891.26 total 409.52 last 123.33) - open/close 59197 (time 1717014901.27 total 419.52 last 123.44) - open/close 60000 (time 1717014907.47 total 425.73 last 129.40) - open/close 61389 (time 1717014917.48 total 435.73 last 138.84) - open/close 62745 (time 1717014927.48 total 445.73 last 135.55) - open/close 64105 (time 1717014937.48 total 455.73 last 135.99) total: 64768 open/close in 460.04 seconds: 140.79 ops/second - /unlink 2145 (time 1717014953.26 total 10.00 last 214.47) - /unlink 4822 (time 1717014963.27 total 20.01 last 267.59) - /unlink 6935 (time 1717014973.27 total 30.01 last 211.25) - /unlink 8785 (time 1717014983.27 total 40.01 last 184.89) - /unlink 10000 (time 1717014989.54 total 46.28 last 193.85) - /unlink 11706 (time 1717014999.54 total 56.28 last 170.57) - /unlink 13555 (time 1717015009.54 total 66.28 last 184.90) - /unlink 15727 (time 1717015019.55 total 76.29 last 217.11) - /unlink 17795 (time 1717015029.55 total 86.29 last 206.73) - /unlink 19597 (time 1717015039.56 total 96.30 last 180.05) - /unlink 20000 (time 1717015041.82 total 98.56 last 177.89) - /unlink 21917 (time 1717015051.82 total 108.56 last 191.69) - /unlink 23845 (time 1717015061.83 total 118.57 last 192.75) - /unlink 25971 (time 1717015071.83 total 128.57 last 212.55) - /unlink 28002 (time 1717015081.83 total 138.57 last 203.08) - /unlink 29960 (time 1717015091.83 total 148.57 last 195.77) - /unlink 31861 (time 1717015101.84 total 158.58 last 190.03) - /unlink 33931 (time 1717015111.84 total 168.58 last 206.99) - /unlink 35870 (time 1717015121.84 total 178.58 last 193.84) - /unlink 37919 (time 1717015131.84 total 188.58 last 204.79) - /unlink 40000 (time 1717015141.55 total 198.29 last 214.39) - /unlink 42148 (time 1717015151.55 total 208.29 last 214.73) - /unlink 44271 (time 1717015161.56 total 218.30 last 212.27) - /unlink 46235 (time 1717015171.56 total 228.30 last 196.36) - /unlink 48347 (time 1717015181.56 total 238.30 last 211.18) - /unlink 50000 (time 1717015189.37 total 246.11 last 211.55) - /unlink 52109 (time 1717015199.38 total 256.12 last 210.79) - /unlink 53905 (time 1717015209.39 total 266.13 last 179.40) - /unlink 55774 (time 1717015219.40 total 276.14 last 186.74) - /unlink 57938 (time 1717015229.41 total 286.15 last 216.12) - /unlink 60000 (time 1717015239.06 total 295.80 last 213.75) - /unlink 62274 (time 1717015249.06 total 305.80 last 227.34) - /unlink 64419 (time 1717015259.06 total 315.80 last 214.49) total: 64768 /unlink in 317.47 seconds: 204.01 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 1187 (time 1717015292.49 total 10.01 last 118.60) - open/close 2488 (time 1717015302.50 total 20.01 last 130.05) - open/close 3925 (time 1717015312.50 total 30.02 last 143.64) - open/close 5332 (time 1717015322.50 total 40.02 last 140.63) - open/close 6682 (time 1717015332.52 total 50.03 last 134.85) - open/close 8060 (time 1717015342.52 total 60.04 last 137.74) - open/close 9382 (time 1717015352.53 total 70.04 last 132.09) - open/close 10000 (time 1717015357.22 total 74.74 last 131.74) - open/close 11250 (time 1717015367.22 total 84.74 last 124.99) - open/close 12429 (time 1717015377.22 total 94.74 last 117.86) - open/close 13564 (time 1717015387.23 total 104.74 last 113.45) - open/close 14695 (time 1717015397.23 total 114.75 last 113.04) - open/close 15836 (time 1717015407.24 total 124.75 last 114.04) - open/close 16960 (time 1717015417.25 total 134.76 last 112.29) - open/close 18095 (time 1717015427.25 total 144.77 last 113.46) - open/close 19274 (time 1717015437.26 total 154.77 last 117.85) - open/close 20000 (time 1717015443.31 total 160.83 last 119.90) - open/close 21229 (time 1717015453.32 total 170.83 last 122.83) - open/close 22357 (time 1717015463.32 total 180.83 last 112.78) - open/close 23584 (time 1717015473.32 total 190.84 last 122.69) - open/close 24840 (time 1717015483.32 total 200.84 last 125.54) - open/close 26029 (time 1717015493.33 total 210.85 last 118.83) - open/close 27451 (time 1717015503.34 total 220.85 last 142.09) - open/close 28677 (time 1717015513.34 total 230.85 last 122.59) - open/close 29974 (time 1717015523.34 total 240.86 last 129.67) - open/close 31320 (time 1717015533.34 total 250.86 last 134.57) - open/close 32758 (time 1717015543.36 total 260.88 last 143.56) - open/close 33939 (time 1717015553.37 total 270.88 last 117.99) - open/close 35257 (time 1717015563.38 total 280.89 last 131.71) - open/close 36643 (time 1717015573.38 total 290.89 last 138.55) - open/close 37981 (time 1717015583.39 total 300.90 last 133.72) - open/close 39130 (time 1717015593.39 total 310.90 last 114.87) - open/close 40000 (time 1717015600.17 total 317.68 last 128.33) - open/close 41284 (time 1717015610.17 total 327.68 last 128.40) - open/close 42522 (time 1717015620.17 total 337.69 last 123.75) - open/close 43921 (time 1717015630.18 total 347.69 last 139.82) - open/close 45384 (time 1717015640.18 total 357.69 last 146.30) - open/close 46863 (time 1717015650.18 total 367.70 last 147.86) - open/close 48360 (time 1717015660.19 total 377.70 last 149.57) - open/close 49638 (time 1717015670.19 total 387.71 last 127.76) - open/close 50000 (time 1717015672.86 total 390.37 last 135.81) - open/close 51273 (time 1717015682.86 total 400.38 last 127.25) - open/close 52393 (time 1717015692.87 total 410.39 last 111.90) - open/close 53619 (time 1717015702.87 total 420.39 last 122.55) - open/close 54919 (time 1717015712.88 total 430.39 last 129.99) - open/close 56190 (time 1717015722.88 total 440.39 last 127.10) - open/close 57488 (time 1717015732.88 total 450.39 last 129.77) - open/close 58953 (time 1717015742.88 total 460.40 last 146.41) - open/close 60000 (time 1717015750.56 total 468.08 last 136.34) - open/close 61314 (time 1717015760.57 total 478.09 last 131.31) - open/close 62553 (time 1717015770.59 total 488.10 last 123.70) - open/close 63737 (time 1717015780.59 total 498.11 last 118.37) total: 64768 open/close in 507.88 seconds: 127.53 ops/second - /unlink 1978 (time 1717015802.42 total 10.00 last 197.74) - /unlink 3931 (time 1717015812.43 total 20.01 last 195.17) - /unlink 5744 (time 1717015822.43 total 30.01 last 181.26) - /unlink 7553 (time 1717015832.43 total 40.01 last 180.90) - /unlink 9570 (time 1717015842.44 total 50.02 last 201.58) - /unlink 10000 (time 1717015844.62 total 52.20 last 197.24) - /unlink 12118 (time 1717015854.62 total 62.20 last 211.76) - /unlink 14238 (time 1717015864.62 total 72.20 last 211.99) - /unlink 16232 (time 1717015874.62 total 82.20 last 199.36) - /unlink 18408 (time 1717015884.62 total 92.20 last 217.60) - /unlink 20000 (time 1717015891.12 total 98.70 last 244.98) - /unlink 22432 (time 1717015901.12 total 108.70 last 243.20) - /unlink 24673 (time 1717015911.12 total 118.71 last 224.01) - /unlink 26765 (time 1717015921.13 total 128.71 last 209.14) - /unlink 28689 (time 1717015931.13 total 138.71 last 192.40) - /unlink 30000 (time 1717015937.50 total 145.08 last 205.87) - /unlink 31945 (time 1717015947.50 total 155.08 last 194.46) - /unlink 33793 (time 1717015957.50 total 165.08 last 184.79) - /unlink 35822 (time 1717015967.50 total 175.08 last 202.88) - /unlink 38130 (time 1717015977.50 total 185.08 last 230.75) - /unlink 40000 (time 1717015985.39 total 192.98 last 236.90) - /unlink 42649 (time 1717015995.40 total 202.98 last 264.88) - /unlink 44958 (time 1717016005.40 total 212.98 last 230.89) - /unlink 46993 (time 1717016015.40 total 222.98 last 203.43) - /unlink 48907 (time 1717016025.40 total 232.98 last 191.36) - /unlink 50000 (time 1717016030.71 total 238.30 last 205.70) - /unlink 52683 (time 1717016040.72 total 248.30 last 268.25) - /unlink 54960 (time 1717016050.72 total 258.30 last 227.69) - /unlink 58098 (time 1717016060.72 total 268.30 last 313.72) - /unlink 60000 (time 1717016067.35 total 274.93 last 286.80) - /unlink 62608 (time 1717016077.35 total 284.93 last 260.76) total: 64768 /unlink in 293.82 seconds: 220.43 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server unloading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2