== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:56:17 (1746064577) Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) oleg440-server: oleg440-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: MODOPTS_LIBCFS= oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg440-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1203 (time 1746064633.89 total 10.00 last 120.24) - open/close 2270 (time 1746064643.90 total 20.01 last 106.66) - open/close 3362 (time 1746064653.90 total 30.01 last 109.17) - open/close 4768 (time 1746064663.90 total 40.01 last 140.57) - open/close 5949 (time 1746064673.91 total 50.02 last 117.99) - open/close 6928 (time 1746064683.92 total 60.03 last 97.85) - open/close 7928 (time 1746064693.92 total 70.03 last 99.97) - open/close 8976 (time 1746064703.92 total 80.03 last 104.78) - open/close 10000 (time 1746064712.74 total 88.86 last 116.05) - open/close 11003 (time 1746064722.76 total 98.87 last 100.19) - open/close 12197 (time 1746064732.77 total 108.88 last 119.28) - open/close 13307 (time 1746064742.77 total 118.88 last 110.98) - open/close 14493 (time 1746064752.77 total 128.88 last 118.57) - open/close 15699 (time 1746064762.78 total 138.89 last 120.52) - open/close 16855 (time 1746064772.78 total 148.89 last 115.54) - open/close 17987 (time 1746064782.78 total 158.89 last 113.19) - open/close 19452 (time 1746064792.79 total 168.90 last 146.39) - open/close 20000 (time 1746064797.47 total 173.59 last 117.00) - open/close 21543 (time 1746064807.48 total 183.59 last 154.25) - open/close 23304 (time 1746064817.48 total 193.60 last 175.98) - open/close 24825 (time 1746064827.49 total 203.60 last 152.05) - open/close 26430 (time 1746064837.49 total 213.60 last 160.42) - open/close 28456 (time 1746064847.49 total 223.60 last 202.59) - open/close 29762 (time 1746064857.49 total 233.61 last 130.59) - open/close 31047 (time 1746064867.50 total 243.61 last 128.39) - open/close 32341 (time 1746064877.50 total 253.61 last 129.39) - open/close 33776 (time 1746064887.50 total 263.62 last 143.47) - open/close 35531 (time 1746064897.50 total 273.62 last 175.49) - open/close 37011 (time 1746064907.51 total 283.62 last 147.92) - open/close 38377 (time 1746064917.52 total 293.63 last 136.51) - open/close 40000 (time 1746064926.79 total 302.90 last 175.10) - open/close 41795 (time 1746064936.79 total 312.90 last 179.49) - open/close 43730 (time 1746064946.79 total 322.90 last 193.42) - open/close 45404 (time 1746064956.80 total 332.91 last 167.28) - open/close 47353 (time 1746064966.80 total 342.91 last 194.84) - open/close 49101 (time 1746064976.80 total 352.92 last 174.77) - open/close 50000 (time 1746064982.51 total 358.62 last 157.64) - open/close 51669 (time 1746064992.52 total 368.63 last 166.70) - open/close 53059 (time 1746065002.52 total 378.63 last 138.97) - open/close 54068 (time 1746065012.53 total 388.64 last 100.82) - open/close 55346 (time 1746065022.53 total 398.65 last 127.74) - open/close 56818 (time 1746065032.53 total 408.65 last 147.18) - open/close 58254 (time 1746065042.54 total 418.65 last 143.54) - open/close 59768 (time 1746065052.54 total 428.65 last 151.38) - open/close 61399 (time 1746065062.54 total 438.66 last 163.04) - open/close 62876 (time 1746065072.55 total 448.66 last 147.66) - open/close 64111 (time 1746065082.55 total 458.66 last 123.46) total: 64768 open/close in 463.16 seconds: 139.84 ops/second - /unlink 2383 (time 1746065098.49 total 10.00 last 238.21) - /unlink 4564 (time 1746065108.49 total 20.00 last 218.09) - /unlink 6929 (time 1746065118.49 total 30.01 last 236.47) - /unlink 9877 (time 1746065128.49 total 40.01 last 294.70) - /unlink 12647 (time 1746065138.50 total 50.01 last 276.93) - /unlink 15076 (time 1746065148.50 total 60.02 last 242.77) - /unlink 17520 (time 1746065158.50 total 70.02 last 244.36) - /unlink 19672 (time 1746065168.51 total 80.02 last 215.12) - /unlink 21915 (time 1746065178.51 total 90.02 last 224.28) - /unlink 23918 (time 1746065188.51 total 100.02 last 200.27) - /unlink 25874 (time 1746065198.51 total 110.03 last 195.51) - /unlink 28207 (time 1746065208.52 total 120.03 last 233.21) - /unlink 30000 (time 1746065214.24 total 125.76 last 313.07) - /unlink 32743 (time 1746065224.25 total 135.76 last 274.19) - /unlink 35178 (time 1746065234.25 total 145.77 last 243.33) - /unlink 37903 (time 1746065244.26 total 155.77 last 272.40) - /unlink 40000 (time 1746065251.24 total 162.76 last 300.23) - /unlink 43417 (time 1746065261.24 total 172.76 last 341.65) - /unlink 45941 (time 1746065271.25 total 182.76 last 252.36) - /unlink 48375 (time 1746065281.25 total 192.77 last 243.31) - /unlink 50000 (time 1746065287.04 total 198.56 last 280.67) - /unlink 52798 (time 1746065297.04 total 208.56 last 279.79) - /unlink 55071 (time 1746065307.04 total 218.56 last 227.24) - /unlink 58133 (time 1746065317.04 total 228.56 last 306.14) - /unlink 60000 (time 1746065323.81 total 235.32 last 276.12) - /unlink 63182 (time 1746065333.81 total 245.32 last 318.15) total: 64768 /unlink in 249.67 seconds: 259.42 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1741 (time 1746065364.21 total 10.00 last 174.06) - open/close 3200 (time 1746065374.21 total 20.01 last 145.80) - open/close 5006 (time 1746065384.22 total 30.01 last 180.53) - open/close 6707 (time 1746065394.22 total 40.02 last 170.05) - open/close 8111 (time 1746065404.22 total 50.02 last 140.37) - open/close 9662 (time 1746065414.23 total 60.02 last 155.03) - open/close 10000 (time 1746065416.96 total 62.75 last 123.83) - open/close 11232 (time 1746065426.96 total 72.75 last 123.18) - open/close 12418 (time 1746065436.97 total 82.76 last 118.49) - open/close 13526 (time 1746065446.97 total 92.77 last 110.72) - open/close 14786 (time 1746065456.98 total 102.77 last 125.96) - open/close 16067 (time 1746065466.98 total 112.78 last 128.04) - open/close 17279 (time 1746065476.98 total 122.78 last 121.16) - open/close 18559 (time 1746065486.99 total 132.78 last 127.99) - open/close 19900 (time 1746065496.99 total 142.79 last 134.03) - open/close 21118 (time 1746065507.00 total 152.79 last 121.73) - open/close 22662 (time 1746065517.00 total 162.80 last 154.33) - open/close 24308 (time 1746065527.00 total 172.80 last 164.57) - open/close 25723 (time 1746065537.00 total 182.80 last 141.48) - open/close 27234 (time 1746065547.01 total 192.80 last 151.06) - open/close 28949 (time 1746065557.01 total 202.80 last 171.48) - open/close 30000 (time 1746065563.73 total 209.52 last 156.41) - open/close 31827 (time 1746065573.73 total 219.52 last 182.69) - open/close 34124 (time 1746065583.73 total 229.52 last 229.65) - open/close 36204 (time 1746065593.74 total 239.53 last 207.82) - open/close 38100 (time 1746065603.74 total 249.53 last 189.58) - open/close 39775 (time 1746065613.74 total 259.54 last 167.45) - open/close 41443 (time 1746065623.74 total 269.54 last 166.75) - open/close 43435 (time 1746065633.75 total 279.54 last 199.14) - open/close 45655 (time 1746065643.75 total 289.54 last 221.96) - open/close 47713 (time 1746065653.75 total 299.55 last 205.74) - open/close 49503 (time 1746065663.75 total 309.55 last 178.97) - open/close 50000 (time 1746065667.68 total 313.48 last 126.60) - open/close 51239 (time 1746065677.68 total 323.48 last 123.87) - open/close 52394 (time 1746065687.69 total 333.48 last 115.46) - open/close 53673 (time 1746065697.69 total 343.49 last 127.80) - open/close 54781 (time 1746065707.70 total 353.49 last 110.77) - open/close 56001 (time 1746065717.70 total 363.49 last 121.96) - open/close 57316 (time 1746065727.71 total 373.50 last 131.37) - open/close 58720 (time 1746065737.71 total 383.50 last 140.40) - open/close 59866 (time 1746065747.71 total 393.51 last 114.56) - open/close 60945 (time 1746065757.72 total 403.52 last 107.81) - open/close 61789 (time 1746065767.72 total 413.52 last 84.36) - open/close 62704 (time 1746065777.75 total 423.54 last 91.30) - open/close 63545 (time 1746065787.75 total 433.55 last 84.04) - open/close 64308 (time 1746065797.76 total 443.55 last 76.26) total: 64768 open/close in 447.46 seconds: 144.75 ops/second - /unlink 2186 (time 1746065813.40 total 10.00 last 218.53) - /unlink 5467 (time 1746065823.40 total 20.00 last 328.06) - /unlink 9076 (time 1746065833.41 total 30.01 last 360.80) - /unlink 10000 (time 1746065836.16 total 32.76 last 335.68) - /unlink 12391 (time 1746065846.16 total 42.76 last 239.03) - /unlink 14361 (time 1746065856.17 total 52.77 last 196.94) - /unlink 16899 (time 1746065866.17 total 62.77 last 253.62) - /unlink 19518 (time 1746065876.18 total 72.78 last 261.78) - /unlink 20000 (time 1746065878.32 total 74.92 last 225.39) - /unlink 22871 (time 1746065888.32 total 84.92 last 287.03) - /unlink 25354 (time 1746065898.32 total 94.92 last 248.15) - /unlink 27712 (time 1746065908.33 total 104.93 last 235.74) - /unlink 29753 (time 1746065918.33 total 114.93 last 204.09) - /unlink 31867 (time 1746065928.33 total 124.93 last 211.28) - /unlink 33950 (time 1746065938.34 total 134.94 last 208.25) - /unlink 35715 (time 1746065948.34 total 144.94 last 176.47) - /unlink 37472 (time 1746065958.34 total 154.94 last 175.67) - /unlink 39287 (time 1746065968.35 total 164.95 last 181.27) - /unlink 40000 (time 1746065972.49 total 169.09 last 172.11) - /unlink 41837 (time 1746065982.50 total 179.10 last 183.59) - /unlink 43792 (time 1746065992.50 total 189.10 last 195.48) - /unlink 45869 (time 1746066002.50 total 199.10 last 207.67) - /unlink 47894 (time 1746066012.51 total 209.10 last 202.47) - /unlink 49785 (time 1746066022.51 total 219.11 last 189.07) - /unlink 52068 (time 1746066032.51 total 229.11 last 228.25) - /unlink 54208 (time 1746066042.52 total 239.12 last 213.70) - /unlink 56137 (time 1746066052.52 total 249.12 last 192.88) - /unlink 58053 (time 1746066062.53 total 259.13 last 191.44) - /unlink 59995 (time 1746066072.53 total 269.13 last 194.18) - /unlink 61997 (time 1746066082.54 total 279.14 last 200.14) - /unlink 63932 (time 1746066092.54 total 289.14 last 193.45) total: 64768 /unlink in 294.08 seconds: 220.24 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1081 (time 1746066126.83 total 10.00 last 108.06) - open/close 2146 (time 1746066136.84 total 20.01 last 106.47) - open/close 3243 (time 1746066146.85 total 30.02 last 109.60) - open/close 4324 (time 1746066156.85 total 40.02 last 108.05) - open/close 5444 (time 1746066166.86 total 50.03 last 111.90) - open/close 6501 (time 1746066176.87 total 60.04 last 105.55) - open/close 7544 (time 1746066186.88 total 70.04 last 104.28) - open/close 8673 (time 1746066196.88 total 80.05 last 112.80) - open/close 9919 (time 1746066206.89 total 90.05 last 124.58) - open/close 11147 (time 1746066216.89 total 100.06 last 122.71) - open/close 12453 (time 1746066226.89 total 110.06 last 130.58) - open/close 13690 (time 1746066236.90 total 120.07 last 123.67) - open/close 14878 (time 1746066246.90 total 130.07 last 118.74) - open/close 16186 (time 1746066256.91 total 140.08 last 130.73) - open/close 17346 (time 1746066266.91 total 150.08 last 115.94) - open/close 18610 (time 1746066276.92 total 160.09 last 126.30) - open/close 19792 (time 1746066286.93 total 170.10 last 118.07) - open/close 21121 (time 1746066296.93 total 180.10 last 132.89) - open/close 22268 (time 1746066306.93 total 190.10 last 114.68) - open/close 23552 (time 1746066316.94 total 200.11 last 128.37) - open/close 24697 (time 1746066326.94 total 210.11 last 114.46) - open/close 25808 (time 1746066336.94 total 220.11 last 111.05) - open/close 27024 (time 1746066346.94 total 230.11 last 121.60) - open/close 28214 (time 1746066356.95 total 240.12 last 118.96) - open/close 29404 (time 1746066366.95 total 250.12 last 118.97) - open/close 30000 (time 1746066371.52 total 254.69 last 130.37) - open/close 31241 (time 1746066381.52 total 264.69 last 124.08) - open/close 32609 (time 1746066391.53 total 274.70 last 136.75) - open/close 33799 (time 1746066401.53 total 284.70 last 118.95) - open/close 35127 (time 1746066411.53 total 294.70 last 132.76) - open/close 36383 (time 1746066421.54 total 304.71 last 125.56) - open/close 37610 (time 1746066431.54 total 314.71 last 122.69) - open/close 38887 (time 1746066441.54 total 324.71 last 127.63) - open/close 40000 (time 1746066448.45 total 331.62 last 161.07) - open/close 41663 (time 1746066458.45 total 341.62 last 166.25) - open/close 43171 (time 1746066468.46 total 351.63 last 150.73) - open/close 44454 (time 1746066478.46 total 361.63 last 128.27) - open/close 45728 (time 1746066488.46 total 371.63 last 127.39) - open/close 46977 (time 1746066498.47 total 381.64 last 124.85) - open/close 48325 (time 1746066508.47 total 391.64 last 134.76) - open/close 49770 (time 1746066518.47 total 401.64 last 144.44) - open/close 50974 (time 1746066528.48 total 411.65 last 120.35) - open/close 52420 (time 1746066538.48 total 421.65 last 144.59) - open/close 53839 (time 1746066548.48 total 431.65 last 141.86) - open/close 55146 (time 1746066558.48 total 441.65 last 130.69) - open/close 56567 (time 1746066568.48 total 451.65 last 142.06) - open/close 58053 (time 1746066578.49 total 461.65 last 148.60) - open/close 59388 (time 1746066588.49 total 471.66 last 133.48) - open/close 60000 (time 1746066592.68 total 475.85 last 146.05) - open/close 61476 (time 1746066602.68 total 485.85 last 147.60) - open/close 62801 (time 1746066612.68 total 495.85 last 132.43) - open/close 64073 (time 1746066622.69 total 505.86 last 127.08) total: 64768 open/close in 511.43 seconds: 126.64 ops/second - /unlink 2084 (time 1746066640.01 total 10.00 last 208.32) - /unlink 4434 (time 1746066650.01 total 20.01 last 234.95) - /unlink 6969 (time 1746066660.01 total 30.01 last 253.45) - /unlink 9274 (time 1746066670.02 total 40.01 last 230.47) - /unlink 10000 (time 1746066673.04 total 43.03 last 240.29) - /unlink 12105 (time 1746066683.04 total 53.03 last 210.41) - /unlink 14215 (time 1746066693.04 total 63.04 last 210.93) - /unlink 16242 (time 1746066703.05 total 73.04 last 202.62) - /unlink 18401 (time 1746066713.05 total 83.05 last 215.77) - /unlink 20000 (time 1746066721.45 total 91.44 last 190.53) - /unlink 22028 (time 1746066731.45 total 101.45 last 202.65) - /unlink 24130 (time 1746066741.45 total 111.45 last 210.18) - /unlink 26182 (time 1746066751.46 total 121.45 last 205.12) - /unlink 28630 (time 1746066761.46 total 131.45 last 244.80) - /unlink 30000 (time 1746066767.39 total 137.38 last 231.08) - /unlink 32295 (time 1746066777.39 total 147.38 last 229.45) - /unlink 34569 (time 1746066787.39 total 157.38 last 227.39) - /unlink 36704 (time 1746066797.39 total 167.38 last 213.48) - /unlink 39136 (time 1746066807.39 total 177.39 last 243.18) - /unlink 40000 (time 1746066810.88 total 180.88 last 247.52) - /unlink 43034 (time 1746066820.88 total 190.88 last 303.39) - /unlink 45624 (time 1746066830.88 total 200.88 last 258.97) - /unlink 48337 (time 1746066840.89 total 210.88 last 271.21) - /unlink 50000 (time 1746066847.35 total 217.35 last 257.18) - /unlink 52464 (time 1746066857.35 total 227.35 last 246.39) - /unlink 54561 (time 1746066867.35 total 237.35 last 209.69) - /unlink 56719 (time 1746066877.36 total 247.35 last 215.77) - /unlink 59059 (time 1746066887.36 total 257.35 last 233.95) - /unlink 60000 (time 1746066891.80 total 261.79 last 211.98) - /unlink 61938 (time 1746066901.80 total 271.79 last 193.80) - /unlink 63713 (time 1746066911.80 total 281.79 last 177.47) total: 64768 /unlink in 286.80 seconds: 225.83 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1237 (time 1746066947.92 total 10.01 last 123.60) - open/close 2550 (time 1746066957.93 total 20.02 last 131.17) - open/close 3940 (time 1746066967.93 total 30.02 last 138.97) - open/close 5441 (time 1746066977.94 total 40.03 last 150.01) - open/close 6784 (time 1746066987.94 total 50.03 last 134.27) - open/close 8223 (time 1746066997.95 total 60.03 last 143.86) - open/close 9597 (time 1746067007.95 total 70.04 last 137.29) - open/close 10000 (time 1746067011.06 total 73.15 last 129.69) - open/close 11204 (time 1746067021.07 total 83.15 last 120.34) - open/close 12356 (time 1746067031.07 total 93.15 last 115.19) - open/close 13603 (time 1746067041.07 total 103.16 last 124.65) - open/close 14926 (time 1746067051.08 total 113.16 last 132.21) - open/close 16234 (time 1746067061.08 total 123.17 last 130.72) - open/close 17545 (time 1746067071.09 total 133.17 last 131.08) - open/close 18785 (time 1746067081.09 total 143.17 last 123.94) - open/close 20000 (time 1746067090.24 total 152.32 last 132.80) - open/close 21469 (time 1746067100.24 total 162.32 last 146.90) - open/close 22879 (time 1746067110.24 total 172.33 last 140.98) - open/close 24272 (time 1746067120.24 total 182.33 last 139.27) - open/close 25796 (time 1746067130.25 total 192.33 last 152.34) - open/close 27490 (time 1746067140.25 total 202.33 last 169.39) - open/close 29090 (time 1746067150.25 total 212.33 last 159.97) - open/close 30000 (time 1746067156.02 total 218.10 last 157.74) - open/close 32033 (time 1746067166.02 total 228.10 last 203.26) - open/close 34085 (time 1746067176.02 total 238.11 last 205.13) - open/close 35753 (time 1746067186.02 total 248.11 last 166.78) - open/close 37314 (time 1746067196.03 total 258.11 last 156.09) - open/close 38882 (time 1746067206.03 total 268.11 last 156.79) - open/close 40000 (time 1746067212.37 total 274.46 last 176.10) - open/close 41675 (time 1746067222.38 total 284.46 last 167.46) - open/close 43244 (time 1746067232.38 total 294.47 last 156.83) - open/close 45052 (time 1746067242.38 total 304.47 last 180.77) - open/close 47102 (time 1746067252.39 total 314.47 last 204.95) - open/close 48816 (time 1746067262.39 total 324.47 last 171.34) - open/close 50000 (time 1746067269.58 total 331.66 last 164.76) - open/close 51418 (time 1746067279.58 total 341.66 last 141.76) - open/close 52980 (time 1746067289.58 total 351.66 last 156.20) - open/close 54456 (time 1746067299.58 total 361.67 last 147.55) - open/close 56148 (time 1746067309.59 total 371.67 last 169.11) - open/close 57830 (time 1746067319.59 total 381.68 last 168.10) - open/close 59323 (time 1746067329.60 total 391.68 last 149.23) - open/close 60000 (time 1746067332.98 total 395.06 last 200.44) - open/close 61749 (time 1746067342.98 total 405.06 last 174.85) - open/close 63936 (time 1746067352.98 total 415.07 last 218.67) total: 64768 open/close in 419.25 seconds: 154.49 ops/second - /unlink 3378 (time 1746067368.19 total 10.00 last 337.64) - /unlink 6671 (time 1746067378.20 total 20.01 last 329.13) - /unlink 9662 (time 1746067388.20 total 30.01 last 299.02) - /unlink 12447 (time 1746067398.20 total 40.02 last 278.39) - /unlink 15633 (time 1746067408.21 total 50.02 last 318.56) - /unlink 19282 (time 1746067418.21 total 60.02 last 364.90) - /unlink 20000 (time 1746067420.80 total 62.61 last 276.81) - /unlink 23094 (time 1746067430.80 total 72.61 last 309.33) - /unlink 25202 (time 1746067440.81 total 82.62 last 210.70) - /unlink 27353 (time 1746067450.81 total 92.62 last 215.07) - /unlink 29837 (time 1746067460.81 total 102.62 last 248.36) - /unlink 32016 (time 1746067470.81 total 112.62 last 217.85) - /unlink 34503 (time 1746067480.81 total 122.63 last 248.64) - /unlink 36970 (time 1746067490.81 total 132.63 last 246.69) - /unlink 39414 (time 1746067500.82 total 142.63 last 244.39) - /unlink 40000 (time 1746067502.95 total 144.76 last 274.50) - /unlink 42862 (time 1746067512.95 total 154.76 last 286.14) - /unlink 45897 (time 1746067522.95 total 164.76 last 303.46) - /unlink 48269 (time 1746067532.95 total 174.77 last 237.18) - /unlink 50000 (time 1746067539.47 total 181.28 last 265.62) - /unlink 52496 (time 1746067549.47 total 191.28 last 249.57) - /unlink 54722 (time 1746067559.48 total 201.29 last 222.51) - /unlink 57015 (time 1746067569.48 total 211.29 last 229.30) - /unlink 59521 (time 1746067579.48 total 221.29 last 250.55) - /unlink 61968 (time 1746067589.48 total 231.29 last 244.67) total: 64768 /unlink in 240.69 seconds: 269.10 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1397 (time 1746067628.49 total 10.00 last 139.69) - open/close 2991 (time 1746067638.49 total 20.01 last 159.33) - open/close 4779 (time 1746067648.49 total 30.01 last 178.79) - open/close 6448 (time 1746067658.51 total 40.02 last 166.70) - open/close 8130 (time 1746067668.51 total 50.02 last 168.14) - open/close 10000 (time 1746067677.29 total 58.80 last 213.01) - open/close 12066 (time 1746067687.29 total 68.80 last 206.56) - open/close 13917 (time 1746067697.29 total 78.80 last 185.06) - open/close 15634 (time 1746067707.30 total 88.81 last 171.66) - open/close 17155 (time 1746067717.30 total 98.81 last 152.03) - open/close 18821 (time 1746067727.31 total 108.82 last 166.51) - open/close 20000 (time 1746067734.48 total 115.99 last 164.27) - open/close 21556 (time 1746067744.49 total 126.00 last 155.53) - open/close 22975 (time 1746067754.49 total 136.00 last 141.87) - open/close 24353 (time 1746067764.49 total 146.00 last 137.78) - open/close 25720 (time 1746067774.49 total 156.00 last 136.69) - open/close 27143 (time 1746067784.49 total 166.00 last 142.28) - open/close 28604 (time 1746067794.50 total 176.01 last 146.02) - open/close 30000 (time 1746067802.24 total 183.75 last 180.34) - open/close 31629 (time 1746067812.25 total 193.76 last 162.78) - open/close 33185 (time 1746067822.25 total 203.76 last 155.53) - open/close 34682 (time 1746067832.26 total 213.77 last 149.51) - open/close 36013 (time 1746067842.27 total 223.78 last 133.04) - open/close 37319 (time 1746067852.27 total 233.78 last 130.55) - open/close 38589 (time 1746067862.28 total 243.79 last 126.89) - open/close 40000 (time 1746067871.90 total 253.42 last 146.61) - open/close 41575 (time 1746067881.91 total 263.42 last 157.42) - open/close 43069 (time 1746067891.91 total 273.43 last 149.33) - open/close 44529 (time 1746067901.92 total 283.43 last 145.90) - open/close 45959 (time 1746067911.93 total 293.44 last 142.92) - open/close 47493 (time 1746067921.93 total 303.44 last 153.39) - open/close 49031 (time 1746067931.93 total 313.44 last 153.80) - open/close 50000 (time 1746067937.84 total 319.35 last 163.82) - open/close 51460 (time 1746067947.85 total 329.36 last 145.94) - open/close 53206 (time 1746067957.85 total 339.36 last 174.56) - open/close 54857 (time 1746067967.85 total 349.36 last 165.03) - open/close 56375 (time 1746067977.86 total 359.37 last 151.66) - open/close 58071 (time 1746067987.86 total 369.38 last 169.57) - open/close 59860 (time 1746067997.87 total 379.38 last 178.82) - open/close 61626 (time 1746068007.87 total 389.38 last 176.57) - open/close 63496 (time 1746068017.88 total 399.39 last 186.86) total: 64768 open/close in 406.99 seconds: 159.14 ops/second - /unlink 2886 (time 1746068036.82 total 10.00 last 288.59) - /unlink 5585 (time 1746068046.82 total 20.00 last 269.90) - /unlink 8678 (time 1746068056.82 total 30.00 last 309.20) - /unlink 10000 (time 1746068060.58 total 33.76 last 351.84) - /unlink 13162 (time 1746068070.58 total 43.76 last 316.15) - /unlink 16065 (time 1746068080.58 total 53.76 last 290.26) - /unlink 18877 (time 1746068090.58 total 63.77 last 281.15) - /unlink 20000 (time 1746068096.18 total 69.37 last 200.54) - /unlink 22619 (time 1746068106.18 total 79.37 last 261.85) - /unlink 25163 (time 1746068116.19 total 89.37 last 254.40) - /unlink 27721 (time 1746068126.19 total 99.37 last 255.64) - /unlink 30000 (time 1746068134.37 total 107.55 last 278.76) - /unlink 32681 (time 1746068144.37 total 117.56 last 267.91) - /unlink 35602 (time 1746068154.37 total 127.56 last 292.09) - /unlink 39222 (time 1746068164.38 total 137.56 last 361.85) - /unlink 40000 (time 1746068167.04 total 140.22 last 292.53) - /unlink 42642 (time 1746068177.04 total 150.22 last 264.20) - /unlink 45102 (time 1746068187.04 total 160.22 last 245.97) - /unlink 47695 (time 1746068197.04 total 170.22 last 259.24) - /unlink 50000 (time 1746068206.75 total 179.93 last 237.53) - /unlink 51939 (time 1746068216.75 total 189.93 last 193.90) - /unlink 54084 (time 1746068226.75 total 199.93 last 214.44) - /unlink 56257 (time 1746068236.75 total 209.94 last 217.18) - /unlink 58512 (time 1746068246.76 total 219.94 last 225.42) - /unlink 60000 (time 1746068252.22 total 225.40 last 272.41) - /unlink 62929 (time 1746068262.22 total 235.40 last 292.83) total: 64768 /unlink in 242.02 seconds: 267.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server unloading modules via unload_modules_local on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2