== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:57:53 (1746143873) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg339-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg339-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg339-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1296 (time 1746143936.44 total 10.01 last 129.49) - open/close 2548 (time 1746143946.45 total 20.01 last 125.14) - open/close 3741 (time 1746143956.45 total 30.02 last 119.25) - open/close 5105 (time 1746143966.45 total 40.02 last 136.39) - open/close 6513 (time 1746143976.46 total 50.03 last 140.65) - open/close 7810 (time 1746143986.47 total 60.04 last 129.60) - open/close 8924 (time 1746143996.48 total 70.04 last 111.31) - open/close 10000 (time 1746144003.86 total 77.42 last 145.89) - open/close 11668 (time 1746144013.86 total 87.42 last 166.77) - open/close 13192 (time 1746144023.86 total 97.42 last 152.39) - open/close 14957 (time 1746144033.86 total 107.42 last 176.47) - open/close 16814 (time 1746144043.87 total 117.43 last 185.56) - open/close 19182 (time 1746144053.87 total 127.43 last 236.70) - open/close 20000 (time 1746144056.86 total 130.42 last 273.81) - open/close 22507 (time 1746144066.86 total 140.42 last 250.67) - open/close 24288 (time 1746144076.86 total 150.42 last 178.10) - open/close 25635 (time 1746144086.86 total 160.43 last 134.63) - open/close 26967 (time 1746144096.87 total 170.43 last 133.16) - open/close 28576 (time 1746144106.87 total 180.43 last 160.87) - open/close 30000 (time 1746144116.87 total 190.43 last 142.44) - open/close 31121 (time 1746144126.87 total 200.43 last 112.08) - open/close 32141 (time 1746144136.88 total 210.44 last 101.92) - open/close 33193 (time 1746144146.88 total 220.44 last 105.18) - open/close 34430 (time 1746144156.88 total 230.45 last 123.65) - open/close 35801 (time 1746144166.89 total 240.45 last 137.02) - open/close 37132 (time 1746144176.89 total 250.46 last 133.03) - open/close 38318 (time 1746144186.90 total 260.46 last 118.56) - open/close 39521 (time 1746144196.90 total 270.47 last 120.23) - open/close 40000 (time 1746144200.29 total 273.85 last 141.35) - open/close 41341 (time 1746144210.29 total 283.86 last 134.05) - open/close 42405 (time 1746144220.30 total 293.86 last 106.39) - open/close 43679 (time 1746144230.30 total 303.87 last 127.30) - open/close 44731 (time 1746144240.31 total 313.87 last 105.12) - open/close 45858 (time 1746144250.31 total 323.88 last 112.66) - open/close 46870 (time 1746144260.33 total 333.89 last 101.09) - open/close 48042 (time 1746144270.33 total 343.89 last 117.14) - open/close 49241 (time 1746144280.33 total 353.90 last 119.87) - open/close 50000 (time 1746144286.71 total 360.28 last 118.96) - open/close 51058 (time 1746144296.72 total 370.28 last 105.79) - open/close 52128 (time 1746144306.72 total 380.28 last 106.97) - open/close 53261 (time 1746144316.72 total 390.29 last 113.23) - open/close 54760 (time 1746144326.73 total 400.29 last 149.85) - open/close 55957 (time 1746144336.73 total 410.30 last 119.65) - open/close 57203 (time 1746144346.74 total 420.30 last 124.53) - open/close 58327 (time 1746144356.74 total 430.30 last 112.37) - open/close 59338 (time 1746144366.75 total 440.31 last 101.02) - open/close 60000 (time 1746144372.54 total 446.11 last 114.27) - open/close 61218 (time 1746144382.54 total 456.11 last 121.77) - open/close 62302 (time 1746144392.55 total 466.11 last 108.33) - open/close 63494 (time 1746144402.55 total 476.12 last 119.17) - open/close 64517 (time 1746144412.55 total 486.12 last 102.29) total: 64768 open/close in 488.66 seconds: 132.54 ops/second - /unlink 2096 (time 1746144427.24 total 10.01 last 209.48) - /unlink 4405 (time 1746144437.24 total 20.01 last 230.89) - /unlink 6698 (time 1746144447.24 total 30.01 last 229.20) - /unlink 9343 (time 1746144457.24 total 40.01 last 264.48) - /unlink 10000 (time 1746144460.62 total 43.39 last 194.60) - /unlink 12605 (time 1746144470.62 total 53.39 last 260.45) - /unlink 14932 (time 1746144480.62 total 63.39 last 232.62) - /unlink 16932 (time 1746144490.63 total 73.40 last 199.91) - /unlink 19345 (time 1746144500.63 total 83.40 last 241.30) - /unlink 20000 (time 1746144503.61 total 86.38 last 219.39) - /unlink 22275 (time 1746144513.61 total 96.38 last 227.47) - /unlink 24743 (time 1746144523.62 total 106.39 last 246.74) - /unlink 26933 (time 1746144533.62 total 116.39 last 218.87) - /unlink 29137 (time 1746144543.63 total 126.40 last 220.33) - /unlink 30000 (time 1746144548.34 total 131.12 last 182.87) - /unlink 32027 (time 1746144558.35 total 141.12 last 202.68) - /unlink 34401 (time 1746144568.35 total 151.12 last 237.33) - /unlink 37475 (time 1746144578.35 total 161.12 last 307.32) - /unlink 40000 (time 1746144586.44 total 169.21 last 312.25) - /unlink 42590 (time 1746144596.44 total 179.21 last 258.98) - /unlink 45083 (time 1746144606.44 total 189.21 last 249.28) - /unlink 47571 (time 1746144616.44 total 199.21 last 248.74) - /unlink 49961 (time 1746144626.45 total 209.22 last 238.92) - /unlink 52266 (time 1746144636.45 total 219.22 last 230.43) - /unlink 54523 (time 1746144646.45 total 229.22 last 225.62) - /unlink 56587 (time 1746144656.46 total 239.23 last 206.33) - /unlink 58753 (time 1746144666.46 total 249.23 last 216.59) - /unlink 60000 (time 1746144672.44 total 255.21 last 208.31) - /unlink 62198 (time 1746144682.45 total 265.22 last 219.60) total: 64768 /unlink in 274.27 seconds: 236.15 ops/second Waiting for MDT destroys to complete - open/close 1575 (time 1746144714.23 total 10.01 last 157.42) - open/close 2812 (time 1746144724.23 total 20.01 last 123.65) - open/close 4104 (time 1746144734.24 total 30.02 last 129.08) - open/close 5528 (time 1746144744.24 total 40.02 last 142.35) - open/close 6924 (time 1746144754.24 total 50.02 last 139.59) - open/close 8249 (time 1746144764.25 total 60.02 last 132.47) - open/close 9515 (time 1746144774.25 total 70.03 last 126.51) - open/close 10000 (time 1746144777.64 total 73.42 last 143.09) - open/close 11471 (time 1746144787.64 total 83.42 last 147.10) - open/close 13060 (time 1746144797.64 total 93.42 last 158.88) - open/close 14643 (time 1746144807.65 total 103.43 last 158.20) - open/close 15976 (time 1746144817.65 total 113.43 last 133.24) - open/close 17419 (time 1746144827.66 total 123.44 last 144.20) - open/close 18791 (time 1746144837.66 total 133.44 last 137.15) - open/close 20000 (time 1746144846.57 total 142.34 last 135.83) - open/close 21311 (time 1746144856.57 total 152.35 last 131.09) - open/close 22674 (time 1746144866.58 total 162.36 last 136.12) - open/close 23857 (time 1746144876.59 total 172.37 last 118.22) - open/close 25031 (time 1746144886.59 total 182.37 last 117.33) - open/close 26221 (time 1746144896.60 total 192.38 last 118.89) - open/close 27429 (time 1746144906.61 total 202.39 last 120.72) - open/close 28746 (time 1746144916.61 total 212.39 last 131.64) - open/close 30000 (time 1746144926.45 total 222.22 last 127.53) - open/close 31390 (time 1746144936.45 total 232.23 last 138.97) - open/close 32644 (time 1746144946.45 total 242.23 last 125.40) - open/close 33959 (time 1746144956.45 total 252.23 last 131.45) - open/close 35111 (time 1746144966.46 total 262.24 last 115.14) - open/close 36406 (time 1746144976.46 total 272.24 last 129.48) - open/close 37801 (time 1746144986.47 total 282.25 last 139.35) - open/close 39059 (time 1746144996.47 total 292.25 last 125.75) - open/close 40000 (time 1746145003.49 total 299.27 last 134.08) - open/close 41430 (time 1746145013.49 total 309.27 last 142.98) - open/close 42781 (time 1746145023.49 total 319.27 last 135.08) - open/close 43996 (time 1746145033.50 total 329.28 last 121.41) - open/close 45146 (time 1746145043.50 total 339.28 last 114.98) - open/close 46471 (time 1746145053.51 total 349.28 last 132.46) - open/close 47727 (time 1746145063.51 total 359.29 last 125.50) - open/close 48997 (time 1746145073.52 total 369.30 last 126.94) - open/close 50000 (time 1746145081.97 total 377.75 last 118.67) - open/close 51144 (time 1746145091.97 total 387.75 last 114.37) - open/close 52441 (time 1746145101.98 total 397.76 last 129.66) - open/close 53538 (time 1746145111.98 total 407.76 last 109.67) - open/close 54769 (time 1746145121.98 total 417.76 last 123.06) - open/close 56060 (time 1746145131.99 total 427.77 last 129.03) - open/close 57365 (time 1746145141.99 total 437.77 last 130.43) - open/close 58604 (time 1746145152.01 total 447.79 last 123.70) - open/close 59800 (time 1746145162.04 total 457.82 last 119.25) - open/close 61179 (time 1746145172.06 total 467.84 last 137.59) - open/close 62447 (time 1746145182.11 total 477.89 last 126.18) - open/close 63738 (time 1746145192.13 total 487.91 last 128.80) total: 64768 open/close in 496.14 seconds: 130.54 ops/second - /unlink 2118 (time 1746145212.36 total 10.00 last 211.71) - /unlink 4125 (time 1746145222.36 total 20.01 last 200.62) - /unlink 6058 (time 1746145232.36 total 30.01 last 193.27) - /unlink 8078 (time 1746145242.36 total 40.01 last 201.92) - /unlink 10000 (time 1746145252.13 total 49.78 last 196.73) - /unlink 12504 (time 1746145262.14 total 59.78 last 250.35) - /unlink 14813 (time 1746145272.14 total 69.79 last 230.85) - /unlink 16829 (time 1746145282.14 total 79.79 last 201.57) - /unlink 18859 (time 1746145292.14 total 89.79 last 202.97) - /unlink 20000 (time 1746145297.32 total 94.97 last 220.45) - /unlink 22136 (time 1746145307.33 total 104.97 last 213.43) - /unlink 24105 (time 1746145317.33 total 114.98 last 196.83) - /unlink 26153 (time 1746145327.33 total 124.98 last 204.76) - /unlink 28138 (time 1746145337.33 total 134.98 last 198.47) - /unlink 30000 (time 1746145345.57 total 143.22 last 225.95) - /unlink 31839 (time 1746145355.58 total 153.23 last 183.81) - /unlink 33685 (time 1746145365.58 total 163.23 last 184.58) - /unlink 35952 (time 1746145375.58 total 173.23 last 226.61) - /unlink 37670 (time 1746145385.58 total 183.23 last 171.78) - /unlink 39511 (time 1746145395.59 total 193.24 last 183.96) - /unlink 40000 (time 1746145398.15 total 195.80 last 190.84) - /unlink 41913 (time 1746145408.16 total 205.80 last 191.26) - /unlink 44118 (time 1746145418.16 total 215.80 last 220.49) - /unlink 46394 (time 1746145428.16 total 225.81 last 227.50) - /unlink 48400 (time 1746145438.16 total 235.81 last 200.58) - /unlink 50000 (time 1746145446.33 total 243.98 last 195.91) - /unlink 52231 (time 1746145456.33 total 253.98 last 223.07) - /unlink 54470 (time 1746145466.33 total 263.98 last 223.89) - /unlink 56502 (time 1746145476.33 total 273.98 last 203.19) - /unlink 58291 (time 1746145486.34 total 283.98 last 178.80) - /unlink 60000 (time 1746145494.99 total 292.64 last 197.54) - /unlink 61698 (time 1746145504.99 total 302.64 last 169.78) - /unlink 63519 (time 1746145514.99 total 312.64 last 182.08) total: 64768 /unlink in 319.86 seconds: 202.49 ops/second Waiting for MDT destroys to complete - open/close 1026 (time 1746145540.39 total 10.01 last 102.53) - open/close 2214 (time 1746145550.40 total 20.02 last 118.67) - open/close 3510 (time 1746145560.41 total 30.03 last 129.44) - open/close 4522 (time 1746145570.41 total 40.03 last 101.17) - open/close 5581 (time 1746145580.41 total 50.03 last 105.89) - open/close 6666 (time 1746145590.42 total 60.04 last 108.43) - open/close 7831 (time 1746145600.42 total 70.04 last 116.48) - open/close 8998 (time 1746145610.43 total 80.04 last 116.65) - open/close 10000 (time 1746145619.24 total 88.86 last 113.65) - open/close 11271 (time 1746145629.25 total 98.86 last 127.07) - open/close 12661 (time 1746145639.25 total 108.86 last 138.97) - open/close 13793 (time 1746145649.26 total 118.87 last 113.09) - open/close 15014 (time 1746145659.27 total 128.88 last 121.99) - open/close 16224 (time 1746145669.27 total 138.89 last 120.97) - open/close 17409 (time 1746145679.28 total 148.89 last 118.42) - open/close 18602 (time 1746145689.28 total 158.89 last 119.28) - open/close 19755 (time 1746145699.28 total 168.90 last 115.22) - open/close 20952 (time 1746145709.29 total 178.90 last 119.67) - open/close 22134 (time 1746145719.29 total 188.91 last 118.18) - open/close 23247 (time 1746145729.29 total 198.91 last 111.24) - open/close 24380 (time 1746145739.30 total 208.92 last 113.23) - open/close 25505 (time 1746145749.30 total 218.92 last 112.48) - open/close 26630 (time 1746145759.31 total 228.93 last 112.37) - open/close 27823 (time 1746145769.31 total 238.93 last 119.29) - open/close 28949 (time 1746145779.32 total 248.94 last 112.55) - open/close 30000 (time 1746145788.21 total 257.83 last 118.19) - open/close 31043 (time 1746145798.23 total 267.84 last 104.14) - open/close 32243 (time 1746145808.23 total 277.85 last 119.98) - open/close 33357 (time 1746145818.23 total 287.85 last 111.33) - open/close 34558 (time 1746145828.23 total 297.85 last 120.09) - open/close 35821 (time 1746145838.24 total 307.86 last 126.23) - open/close 36865 (time 1746145848.24 total 317.86 last 104.37) - open/close 38117 (time 1746145858.24 total 327.86 last 125.17) - open/close 39231 (time 1746145868.25 total 337.87 last 111.34) - open/close 40000 (time 1746145875.51 total 345.13 last 105.91) - open/close 41092 (time 1746145885.51 total 355.13 last 109.17) - open/close 42201 (time 1746145895.51 total 365.13 last 110.89) - open/close 43378 (time 1746145905.52 total 375.14 last 117.58) - open/close 44542 (time 1746145915.53 total 385.15 last 116.34) - open/close 45632 (time 1746145925.53 total 395.15 last 108.97) - open/close 46724 (time 1746145935.53 total 405.15 last 109.18) - open/close 48076 (time 1746145945.54 total 415.15 last 135.20) - open/close 49278 (time 1746145955.54 total 425.16 last 120.16) - open/close 50000 (time 1746145961.74 total 431.36 last 116.38) - open/close 51102 (time 1746145971.75 total 441.36 last 110.14) - open/close 52358 (time 1746145981.75 total 451.37 last 125.58) - open/close 53696 (time 1746145991.75 total 461.37 last 133.76) - open/close 54886 (time 1746146001.76 total 471.38 last 118.91) - open/close 56309 (time 1746146011.76 total 481.38 last 142.24) - open/close 57511 (time 1746146021.77 total 491.39 last 120.12) - open/close 58711 (time 1746146031.78 total 501.40 last 119.91) - open/close 60000 (time 1746146040.44 total 510.06 last 148.82) - open/close 61176 (time 1746146050.44 total 520.06 last 117.58) - open/close 62375 (time 1746146060.44 total 530.06 last 119.88) - open/close 63576 (time 1746146070.45 total 540.07 last 120.02) total: 64768 open/close in 548.49 seconds: 118.08 ops/second - /unlink 2359 (time 1746146090.67 total 10.00 last 235.85) - /unlink 4765 (time 1746146100.68 total 20.00 last 240.58) - /unlink 7062 (time 1746146110.68 total 30.00 last 229.68) - /unlink 9384 (time 1746146120.68 total 40.00 last 232.19) - /unlink 10000 (time 1746146123.91 total 43.24 last 190.68) - /unlink 12855 (time 1746146133.91 total 53.24 last 285.46) - /unlink 15711 (time 1746146143.91 total 63.24 last 285.54) - /unlink 18374 (time 1746146153.92 total 73.25 last 266.14) - /unlink 20000 (time 1746146161.53 total 80.85 last 213.70) - /unlink 22008 (time 1746146171.53 total 90.86 last 200.75) - /unlink 23946 (time 1746146181.53 total 100.86 last 193.74) - /unlink 25996 (time 1746146191.54 total 110.86 last 204.90) - /unlink 28123 (time 1746146201.54 total 120.87 last 212.67) - /unlink 30000 (time 1746146210.08 total 129.41 last 219.74) - /unlink 32578 (time 1746146220.08 total 139.41 last 257.72) - /unlink 34861 (time 1746146230.09 total 149.41 last 228.25) - /unlink 37396 (time 1746146240.09 total 159.42 last 253.34) - /unlink 39950 (time 1746146250.09 total 169.42 last 255.35) - /unlink 42078 (time 1746146260.10 total 179.43 last 212.60) - /unlink 44851 (time 1746146270.10 total 189.43 last 277.30) - /unlink 48622 (time 1746146280.10 total 199.43 last 377.08) - /unlink 50000 (time 1746146285.08 total 204.41 last 276.75) - /unlink 53669 (time 1746146295.08 total 214.41 last 366.85) - /unlink 56802 (time 1746146305.08 total 224.41 last 313.29) - /unlink 60000 (time 1746146314.87 total 234.20 last 326.86) - /unlink 63287 (time 1746146324.87 total 244.20 last 328.56) total: 64768 /unlink in 249.32 seconds: 259.78 ops/second Waiting for MDT destroys to complete - open/close 1600 (time 1746146346.98 total 10.00 last 159.97) - open/close 3162 (time 1746146356.99 total 20.01 last 156.10) - open/close 4893 (time 1746146366.99 total 30.01 last 173.08) - open/close 6501 (time 1746146376.99 total 40.01 last 160.77) - open/close 8189 (time 1746146386.99 total 50.01 last 168.77) - open/close 9901 (time 1746146396.99 total 60.01 last 171.18) - open/close 11484 (time 1746146407.00 total 70.02 last 158.28) - open/close 13303 (time 1746146417.00 total 80.02 last 181.76) - open/close 15117 (time 1746146427.00 total 90.02 last 181.38) - open/close 16844 (time 1746146437.01 total 100.03 last 172.60) - open/close 19046 (time 1746146447.01 total 110.03 last 220.17) - open/close 20000 (time 1746146451.61 total 114.63 last 207.27) - open/close 22074 (time 1746146461.61 total 124.64 last 207.38) - open/close 23965 (time 1746146471.62 total 134.64 last 189.00) - open/close 26733 (time 1746146481.62 total 144.64 last 276.73) - open/close 28989 (time 1746146491.63 total 154.65 last 225.54) - open/close 30000 (time 1746146496.93 total 159.95 last 190.70) - open/close 32174 (time 1746146506.93 total 169.95 last 217.32) - open/close 34139 (time 1746146516.93 total 179.95 last 196.49) - open/close 35935 (time 1746146526.93 total 189.95 last 179.57) - open/close 37535 (time 1746146536.94 total 199.96 last 159.93) - open/close 39401 (time 1746146546.94 total 209.96 last 186.54) - open/close 40000 (time 1746146549.49 total 212.52 last 234.52) - open/close 42296 (time 1746146559.50 total 222.52 last 229.47) - open/close 44814 (time 1746146569.50 total 232.52 last 251.78) - open/close 47176 (time 1746146579.50 total 242.52 last 236.14) - open/close 49920 (time 1746146589.51 total 252.53 last 274.29) - open/close 53034 (time 1746146599.51 total 262.53 last 311.39) - open/close 55582 (time 1746146609.51 total 272.53 last 254.78) - open/close 58842 (time 1746146619.51 total 282.53 last 325.90) - open/close 60000 (time 1746146625.56 total 288.58 last 191.46) - open/close 62132 (time 1746146635.56 total 298.58 last 213.16) - open/close 63819 (time 1746146645.56 total 308.58 last 168.68) total: 64768 open/close in 313.47 seconds: 206.62 ops/second - /unlink 2805 (time 1746146661.62 total 10.01 last 280.35) - /unlink 5876 (time 1746146671.62 total 20.01 last 307.03) - /unlink 9014 (time 1746146681.62 total 30.01 last 313.75) - /unlink 10000 (time 1746146683.87 total 32.26 last 438.01) - /unlink 14658 (time 1746146693.87 total 42.26 last 465.79) - /unlink 19237 (time 1746146703.88 total 52.26 last 457.83) - /unlink 24294 (time 1746146713.88 total 62.26 last 505.61) - /unlink 28695 (time 1746146723.88 total 72.26 last 440.06) - /unlink 30000 (time 1746146726.48 total 74.87 last 501.32) - /unlink 35493 (time 1746146736.48 total 84.87 last 549.20) - /unlink 40000 (time 1746146745.38 total 93.77 last 506.50) - /unlink 44906 (time 1746146755.38 total 103.77 last 490.50) - /unlink 48017 (time 1746146765.39 total 113.77 last 311.06) - /unlink 50000 (time 1746146770.84 total 119.23 last 363.46) - /unlink 54174 (time 1746146780.84 total 129.23 last 417.28) - /unlink 59618 (time 1746146790.85 total 139.23 last 544.19) - /unlink 64074 (time 1746146800.85 total 149.24 last 445.41) total: 64768 /unlink in 151.52 seconds: 427.46 ops/second Waiting for MDT destroys to complete - open/close 2245 (time 1746146821.52 total 10.00 last 224.46) - open/close 3869 (time 1746146831.53 total 20.00 last 162.38) - open/close 5298 (time 1746146841.53 total 30.00 last 142.89) - open/close 7208 (time 1746146851.53 total 40.01 last 190.97) - open/close 9589 (time 1746146861.53 total 50.01 last 238.03) - open/close 11654 (time 1746146871.53 total 60.01 last 206.43) - open/close 13386 (time 1746146881.54 total 70.02 last 173.11) - open/close 14807 (time 1746146891.54 total 80.02 last 142.02) - open/close 16424 (time 1746146901.55 total 90.03 last 161.63) - open/close 17892 (time 1746146911.55 total 100.03 last 146.73) - open/close 19617 (time 1746146921.56 total 110.03 last 172.46) - open/close 21966 (time 1746146931.56 total 120.04 last 234.84) - open/close 23884 (time 1746146941.56 total 130.04 last 191.76) - open/close 26131 (time 1746146951.56 total 140.04 last 224.67) - open/close 28143 (time 1746146961.57 total 150.05 last 201.07) - open/close 30000 (time 1746146970.52 total 158.99 last 207.57) - open/close 32191 (time 1746146980.52 total 169.00 last 218.95) - open/close 33829 (time 1746146990.53 total 179.01 last 163.72) - open/close 35316 (time 1746147000.53 total 189.01 last 148.70) - open/close 37048 (time 1746147010.53 total 199.01 last 173.19) - open/close 38590 (time 1746147020.53 total 209.01 last 154.13) - open/close 40000 (time 1746147030.40 total 218.87 last 142.95) - open/close 41697 (time 1746147040.40 total 228.88 last 169.69) - open/close 43012 (time 1746147050.40 total 238.88 last 131.43) - open/close 44314 (time 1746147060.41 total 248.88 last 130.15) - open/close 45645 (time 1746147070.41 total 258.89 last 133.07) - open/close 47000 (time 1746147080.41 total 268.89 last 135.44) - open/close 48317 (time 1746147090.42 total 278.90 last 131.64) - open/close 49757 (time 1746147100.43 total 288.91 last 143.80) - open/close 51096 (time 1746147110.44 total 298.92 last 133.75) - open/close 52319 (time 1746147120.45 total 308.92 last 122.27) - open/close 53799 (time 1746147130.45 total 318.92 last 147.99) - open/close 55464 (time 1746147140.45 total 328.93 last 166.44) - open/close 56823 (time 1746147150.46 total 338.93 last 135.82) - open/close 57969 (time 1746147160.46 total 348.94 last 114.49) - open/close 59169 (time 1746147170.48 total 358.95 last 119.87) - open/close 60000 (time 1746147177.55 total 366.03 last 117.49) - open/close 61277 (time 1746147187.57 total 376.05 last 127.47) - open/close 62511 (time 1746147197.57 total 386.05 last 123.31) - open/close 63702 (time 1746147207.58 total 396.05 last 119.08) total: 64768 open/close in 405.62 seconds: 159.68 ops/second - /unlink 1818 (time 1746147229.27 total 10.00 last 181.78) - /unlink 3732 (time 1746147239.28 total 20.00 last 191.37) - /unlink 5616 (time 1746147249.28 total 30.01 last 188.36) - /unlink 7681 (time 1746147259.28 total 40.01 last 206.48) - /unlink 9959 (time 1746147269.28 total 50.01 last 227.80) - /unlink 11798 (time 1746147279.28 total 60.01 last 183.81) - /unlink 13701 (time 1746147289.29 total 70.01 last 190.24) - /unlink 15864 (time 1746147299.29 total 80.01 last 216.29) - /unlink 18179 (time 1746147309.29 total 90.02 last 231.49) - /unlink 20000 (time 1746147317.60 total 98.33 last 219.11) - /unlink 22764 (time 1746147327.60 total 108.33 last 276.39) - /unlink 25586 (time 1746147337.60 total 118.33 last 282.10) - /unlink 28637 (time 1746147347.61 total 128.34 last 304.91) - /unlink 30000 (time 1746147353.41 total 134.14 last 234.82) - /unlink 32787 (time 1746147363.41 total 144.14 last 278.70) - /unlink 35808 (time 1746147373.41 total 154.14 last 302.06) - /unlink 38385 (time 1746147383.42 total 164.14 last 257.63) - /unlink 40000 (time 1746147387.88 total 168.61 last 362.05) - /unlink 44197 (time 1746147397.88 total 178.61 last 419.61) - /unlink 47061 (time 1746147407.88 total 188.61 last 286.35) - /unlink 50000 (time 1746147414.59 total 195.31 last 438.30) - /unlink 54637 (time 1746147424.59 total 205.32 last 463.67) - /unlink 58379 (time 1746147434.59 total 215.32 last 374.14) - /unlink 60000 (time 1746147439.25 total 219.97 last 348.05) - /unlink 62914 (time 1746147449.25 total 229.98 last 291.35) total: 64768 /unlink in 237.42 seconds: 272.80 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2