== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:37:11 (1746142631) Stopping clients: oleg205-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg205-client.virtnet /mnt/lustre2 (opts:-f) oleg205-server: oleg205-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing load_modules_local oleg205-server: Loading modules from /home/green/git/lustre-release/lustre oleg205-server: detected 4 online CPUs by sysfs oleg205-server: MODOPTS_LIBCFS= oleg205-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg205-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg205-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg205-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1087 (time 1746142695.30 total 10.01 last 108.58) - open/close 2242 (time 1746142705.31 total 20.02 last 115.35) - open/close 3378 (time 1746142715.32 total 30.03 last 113.55) - open/close 4615 (time 1746142725.32 total 40.03 last 123.68) - open/close 5904 (time 1746142735.33 total 50.04 last 128.77) - open/close 7064 (time 1746142745.33 total 60.04 last 115.97) - open/close 7992 (time 1746142755.34 total 70.05 last 92.74) - open/close 8927 (time 1746142765.35 total 80.06 last 93.38) - open/close 9915 (time 1746142775.35 total 90.07 last 98.78) - open/close 11098 (time 1746142785.36 total 100.07 last 118.19) - open/close 12245 (time 1746142795.36 total 110.07 last 114.70) - open/close 13378 (time 1746142805.37 total 120.08 last 113.21) - open/close 14618 (time 1746142815.37 total 130.09 last 123.94) - open/close 15761 (time 1746142825.38 total 140.09 last 114.22) - open/close 16895 (time 1746142835.39 total 150.11 last 113.27) - open/close 18043 (time 1746142845.39 total 160.11 last 114.80) - open/close 19325 (time 1746142855.39 total 170.11 last 128.20) - open/close 20000 (time 1746142861.38 total 176.10 last 112.65) - open/close 21215 (time 1746142871.39 total 186.10 last 121.43) - open/close 22349 (time 1746142881.40 total 196.11 last 113.34) - open/close 23558 (time 1746142891.40 total 206.11 last 120.89) - open/close 24790 (time 1746142901.40 total 216.11 last 123.19) - open/close 25980 (time 1746142911.40 total 226.11 last 118.96) - open/close 27240 (time 1746142921.40 total 236.12 last 125.96) - open/close 28299 (time 1746142931.41 total 246.12 last 105.84) - open/close 29389 (time 1746142941.42 total 256.13 last 108.87) - open/close 30000 (time 1746142946.97 total 261.68 last 110.09) - open/close 31186 (time 1746142956.98 total 271.69 last 118.52) - open/close 32289 (time 1746142966.99 total 281.70 last 110.21) - open/close 33443 (time 1746142976.99 total 291.71 last 115.33) - open/close 34568 (time 1746142986.99 total 301.71 last 112.49) - open/close 35576 (time 1746142997.00 total 311.71 last 100.77) - open/close 36722 (time 1746143007.00 total 321.71 last 114.60) - open/close 37930 (time 1746143017.01 total 331.72 last 120.66) - open/close 38910 (time 1746143027.02 total 341.73 last 97.90) - open/close 39963 (time 1746143037.02 total 351.73 last 105.27) - open/close 41093 (time 1746143047.02 total 361.74 last 112.96) - open/close 42141 (time 1746143057.03 total 371.75 last 104.72) - open/close 43183 (time 1746143067.04 total 381.75 last 104.13) - open/close 44213 (time 1746143077.04 total 391.76 last 102.97) - open/close 45125 (time 1746143087.05 total 401.76 last 91.15) - open/close 46105 (time 1746143097.05 total 411.76 last 97.96) - open/close 47097 (time 1746143107.05 total 421.77 last 99.19) - open/close 48177 (time 1746143117.05 total 431.77 last 108.00) - open/close 49133 (time 1746143127.06 total 441.77 last 95.55) - open/close 50000 (time 1746143135.23 total 449.94 last 106.10) - open/close 51024 (time 1746143145.25 total 459.96 last 102.21) - open/close 51877 (time 1746143155.27 total 469.98 last 85.15) - open/close 52830 (time 1746143165.27 total 479.99 last 95.24) - open/close 53797 (time 1746143175.28 total 489.99 last 96.61) - open/close 54833 (time 1746143185.29 total 500.00 last 103.51) - open/close 56038 (time 1746143195.30 total 510.01 last 120.43) - open/close 57094 (time 1746143205.31 total 520.03 last 105.43) - open/close 58146 (time 1746143215.32 total 530.03 last 105.16) - open/close 59236 (time 1746143225.32 total 540.03 last 108.96) - open/close 60000 (time 1746143232.02 total 546.74 last 113.97) - open/close 61008 (time 1746143242.02 total 556.74 last 100.79) - open/close 62052 (time 1746143252.02 total 566.74 last 104.39) - open/close 63132 (time 1746143262.03 total 576.74 last 107.94) - open/close 64136 (time 1746143272.03 total 586.75 last 100.36) total: 64768 open/close in 593.16 seconds: 109.19 ops/second - /unlink 1752 (time 1746143290.57 total 10.00 last 175.18) - /unlink 3615 (time 1746143300.59 total 20.02 last 186.01) - /unlink 5261 (time 1746143310.59 total 30.02 last 164.59) - /unlink 7027 (time 1746143320.60 total 40.02 last 176.50) - /unlink 8811 (time 1746143330.60 total 50.03 last 178.30) - /unlink 10000 (time 1746143338.22 total 57.65 last 156.02) - /unlink 11904 (time 1746143348.22 total 67.65 last 190.37) - /unlink 13748 (time 1746143358.22 total 77.65 last 184.40) - /unlink 15713 (time 1746143368.23 total 87.65 last 196.49) - /unlink 17503 (time 1746143378.24 total 97.66 last 178.82) - /unlink 19427 (time 1746143388.24 total 107.66 last 192.34) - /unlink 20000 (time 1746143391.33 total 110.75 last 185.60) - /unlink 21667 (time 1746143401.33 total 120.75 last 166.70) - /unlink 23436 (time 1746143411.33 total 130.75 last 176.86) - /unlink 24992 (time 1746143421.33 total 140.76 last 155.59) - /unlink 26687 (time 1746143431.33 total 150.76 last 169.47) - /unlink 28550 (time 1746143441.34 total 160.76 last 186.18) - /unlink 30000 (time 1746143448.61 total 168.04 last 199.25) - /unlink 32015 (time 1746143458.62 total 178.05 last 201.36) - /unlink 34030 (time 1746143468.62 total 188.05 last 201.50) - /unlink 36195 (time 1746143478.62 total 198.05 last 216.47) - /unlink 38115 (time 1746143488.63 total 208.05 last 191.94) - /unlink 40000 (time 1746143498.48 total 217.91 last 191.26) - /unlink 42077 (time 1746143508.49 total 227.92 last 207.53) - /unlink 44033 (time 1746143518.49 total 237.92 last 195.54) - /unlink 46099 (time 1746143528.50 total 247.92 last 206.52) - /unlink 48185 (time 1746143538.50 total 257.92 last 208.59) - /unlink 50000 (time 1746143547.67 total 267.10 last 197.83) - /unlink 51887 (time 1746143557.67 total 277.10 last 188.68) - /unlink 53806 (time 1746143567.67 total 287.10 last 191.89) - /unlink 55800 (time 1746143577.68 total 297.10 last 199.34) - /unlink 57852 (time 1746143587.68 total 307.11 last 205.12) - /unlink 59805 (time 1746143597.68 total 317.11 last 195.22) - /unlink 61855 (time 1746143607.68 total 327.11 last 204.99) - /unlink 64048 (time 1746143617.68 total 337.11 last 219.28) total: 64768 /unlink in 340.42 seconds: 190.26 ops/second Waiting for MDT destroys to complete - open/close 1310 (time 1746143641.61 total 10.00 last 130.95) - open/close 2510 (time 1746143651.62 total 20.01 last 119.94) - open/close 3755 (time 1746143661.63 total 30.02 last 124.37) - open/close 4901 (time 1746143671.63 total 40.02 last 114.57) - open/close 6135 (time 1746143681.64 total 50.03 last 123.34) - open/close 7234 (time 1746143691.64 total 60.03 last 109.86) - open/close 8381 (time 1746143701.65 total 70.04 last 114.59) - open/close 9597 (time 1746143711.65 total 80.04 last 121.58) - open/close 10000 (time 1746143714.72 total 83.11 last 131.24) - open/close 11248 (time 1746143724.73 total 93.12 last 124.69) - open/close 12488 (time 1746143734.73 total 103.12 last 123.98) - open/close 13764 (time 1746143744.74 total 113.13 last 127.51) - open/close 15008 (time 1746143754.74 total 123.13 last 124.38) - open/close 16217 (time 1746143764.75 total 133.14 last 120.84) - open/close 17643 (time 1746143774.75 total 143.14 last 142.52) - open/close 18924 (time 1746143784.76 total 153.15 last 128.00) - open/close 20000 (time 1746143792.90 total 161.29 last 132.14) - open/close 21296 (time 1746143802.91 total 171.31 last 129.47) - open/close 22681 (time 1746143812.91 total 181.31 last 138.48) - open/close 24052 (time 1746143822.93 total 191.32 last 136.95) - open/close 25465 (time 1746143832.93 total 201.32 last 141.25) - open/close 26662 (time 1746143842.93 total 211.32 last 119.65) - open/close 27948 (time 1746143852.94 total 221.33 last 128.55) - open/close 29190 (time 1746143862.94 total 231.33 last 124.15) - open/close 30000 (time 1746143869.62 total 238.01 last 121.26) - open/close 31309 (time 1746143879.63 total 248.02 last 130.77) - open/close 32787 (time 1746143889.63 total 258.02 last 147.77) - open/close 34144 (time 1746143899.64 total 268.03 last 135.65) - open/close 35632 (time 1746143909.64 total 278.03 last 148.80) - open/close 36872 (time 1746143919.64 total 288.03 last 123.95) - open/close 38098 (time 1746143929.64 total 298.03 last 122.58) - open/close 39301 (time 1746143939.64 total 308.04 last 120.29) - open/close 40000 (time 1746143945.30 total 313.69 last 123.59) - open/close 41202 (time 1746143955.30 total 323.70 last 120.15) - open/close 42506 (time 1746143965.31 total 333.70 last 130.33) - open/close 43630 (time 1746143975.31 total 343.70 last 112.38) - open/close 44851 (time 1746143985.31 total 353.70 last 122.07) - open/close 46259 (time 1746143995.31 total 363.71 last 140.77) - open/close 47594 (time 1746144005.32 total 373.71 last 133.49) - open/close 48824 (time 1746144015.32 total 383.71 last 122.99) - open/close 50000 (time 1746144025.22 total 393.62 last 118.68) - open/close 51189 (time 1746144035.23 total 403.62 last 118.86) - open/close 52363 (time 1746144045.23 total 413.62 last 117.34) - open/close 53529 (time 1746144055.24 total 423.63 last 116.56) - open/close 54655 (time 1746144065.28 total 433.67 last 112.15) - open/close 55721 (time 1746144075.29 total 443.68 last 106.51) - open/close 56851 (time 1746144085.30 total 453.69 last 112.89) - open/close 58042 (time 1746144095.31 total 463.70 last 118.94) - open/close 59151 (time 1746144105.35 total 473.74 last 110.43) - open/close 60000 (time 1746144112.75 total 481.14 last 114.82) - open/close 61316 (time 1746144122.75 total 491.14 last 131.53) - open/close 62501 (time 1746144132.76 total 501.15 last 118.42) - open/close 63763 (time 1746144142.76 total 511.15 last 126.13) total: 64768 open/close in 518.69 seconds: 124.87 ops/second - /unlink 2175 (time 1746144161.87 total 10.00 last 217.40) - /unlink 4239 (time 1746144171.87 total 20.01 last 206.32) - /unlink 6147 (time 1746144181.87 total 30.01 last 190.77) - /unlink 8202 (time 1746144191.88 total 40.02 last 205.37) - /unlink 10000 (time 1746144200.21 total 48.35 last 215.69) - /unlink 11976 (time 1746144210.22 total 58.35 last 197.57) - /unlink 13867 (time 1746144220.22 total 68.35 last 189.08) - /unlink 15963 (time 1746144230.23 total 78.36 last 209.41) - /unlink 18143 (time 1746144240.23 total 88.37 last 217.86) - /unlink 20000 (time 1746144248.11 total 96.25 last 235.69) - /unlink 22187 (time 1746144258.12 total 106.26 last 218.51) - /unlink 24274 (time 1746144268.12 total 116.26 last 208.67) - /unlink 26370 (time 1746144278.12 total 126.26 last 209.55) - /unlink 28536 (time 1746144288.13 total 136.26 last 216.54) - /unlink 30000 (time 1746144295.04 total 143.18 last 211.81) - /unlink 31924 (time 1746144305.04 total 153.18 last 192.35) - /unlink 33828 (time 1746144315.04 total 163.18 last 190.38) - /unlink 35800 (time 1746144325.04 total 173.18 last 197.17) - /unlink 38198 (time 1746144335.04 total 183.18 last 239.78) - /unlink 40000 (time 1746144342.56 total 190.69 last 239.90) - /unlink 42469 (time 1746144352.56 total 200.70 last 246.85) - /unlink 44927 (time 1746144362.56 total 210.70 last 245.75) - /unlink 47289 (time 1746144372.56 total 220.70 last 236.11) - /unlink 49526 (time 1746144382.57 total 230.70 last 223.64) - /unlink 50000 (time 1746144384.83 total 232.97 last 209.12) - /unlink 52086 (time 1746144394.84 total 242.97 last 208.52) - /unlink 54566 (time 1746144404.84 total 252.97 last 247.98) - /unlink 56688 (time 1746144414.84 total 262.97 last 212.20) - /unlink 58911 (time 1746144424.84 total 272.98 last 222.22) - /unlink 60000 (time 1746144429.29 total 277.42 last 244.93) - /unlink 61903 (time 1746144439.29 total 287.43 last 190.15) - /unlink 63965 (time 1746144449.30 total 297.44 last 206.14) total: 64768 /unlink in 301.07 seconds: 215.13 ops/second Waiting for MDT destroys to complete - open/close 1410 (time 1746144473.51 total 10.00 last 140.97) - open/close 3102 (time 1746144483.52 total 20.01 last 169.11) - open/close 5043 (time 1746144493.52 total 30.01 last 194.02) - open/close 6980 (time 1746144503.52 total 40.01 last 193.65) - open/close 8712 (time 1746144513.52 total 50.02 last 173.18) - open/close 10000 (time 1746144522.29 total 58.78 last 146.98) - open/close 11348 (time 1746144532.29 total 68.78 last 134.78) - open/close 12959 (time 1746144542.29 total 78.78 last 161.04) - open/close 14394 (time 1746144552.30 total 88.79 last 143.40) - open/close 15787 (time 1746144562.30 total 98.79 last 139.25) - open/close 17233 (time 1746144572.31 total 108.80 last 144.56) - open/close 18615 (time 1746144582.31 total 118.80 last 138.12) - open/close 19885 (time 1746144592.32 total 128.81 last 126.93) - open/close 21241 (time 1746144602.32 total 138.81 last 135.57) - open/close 22729 (time 1746144612.32 total 148.82 last 148.73) - open/close 24201 (time 1746144622.33 total 158.82 last 147.19) - open/close 25755 (time 1746144632.33 total 168.82 last 155.38) - open/close 27337 (time 1746144642.33 total 178.82 last 158.12) - open/close 28986 (time 1746144652.33 total 188.83 last 164.85) - open/close 30000 (time 1746144657.46 total 193.95 last 197.94) - open/close 31324 (time 1746144667.46 total 203.95 last 132.38) - open/close 32579 (time 1746144677.46 total 213.95 last 125.48) - open/close 33945 (time 1746144687.46 total 223.95 last 136.60) - open/close 35309 (time 1746144697.47 total 233.96 last 136.27) - open/close 36594 (time 1746144707.47 total 243.96 last 128.47) - open/close 37845 (time 1746144717.48 total 253.97 last 124.98) - open/close 39174 (time 1746144727.48 total 263.98 last 132.85) - open/close 40000 (time 1746144733.55 total 270.04 last 136.29) - open/close 41634 (time 1746144743.55 total 280.04 last 163.39) - open/close 43011 (time 1746144753.55 total 290.04 last 137.68) - open/close 44268 (time 1746144763.56 total 300.05 last 125.50) - open/close 45452 (time 1746144773.57 total 310.06 last 118.33) - open/close 46897 (time 1746144783.57 total 320.06 last 144.49) - open/close 48308 (time 1746144793.57 total 330.07 last 141.04) - open/close 49602 (time 1746144803.58 total 340.07 last 129.36) - open/close 50000 (time 1746144806.87 total 343.36 last 121.06) - open/close 51167 (time 1746144816.87 total 353.36 last 116.65) - open/close 52414 (time 1746144826.88 total 363.37 last 124.61) - open/close 53654 (time 1746144836.88 total 373.37 last 123.98) - open/close 55045 (time 1746144846.88 total 383.37 last 139.07) - open/close 56352 (time 1746144856.88 total 393.37 last 130.69) - open/close 57524 (time 1746144866.89 total 403.38 last 117.15) - open/close 58733 (time 1746144876.89 total 413.38 last 120.87) - open/close 59793 (time 1746144886.89 total 423.38 last 105.97) - open/close 60911 (time 1746144896.90 total 433.39 last 111.72) - open/close 62240 (time 1746144906.90 total 443.39 last 132.85) - open/close 63444 (time 1746144916.90 total 453.39 last 120.37) - open/close 64608 (time 1746144926.91 total 463.40 last 116.33) total: 64768 open/close in 464.81 seconds: 139.34 ops/second - /unlink 2054 (time 1746144940.26 total 10.01 last 205.28) - /unlink 4175 (time 1746144950.26 total 20.01 last 212.05) - /unlink 6386 (time 1746144960.27 total 30.01 last 220.96) - /unlink 8495 (time 1746144970.27 total 40.02 last 210.79) - /unlink 10000 (time 1746144978.30 total 48.05 last 187.42) - /unlink 12041 (time 1746144988.30 total 58.05 last 204.08) - /unlink 14112 (time 1746144998.31 total 68.06 last 207.00) - /unlink 16189 (time 1746145008.31 total 78.06 last 207.63) - /unlink 18345 (time 1746145018.31 total 88.06 last 215.53) - /unlink 20000 (time 1746145026.62 total 96.37 last 199.30) - /unlink 22087 (time 1746145036.62 total 106.37 last 208.66) - /unlink 24093 (time 1746145046.62 total 116.37 last 200.58) - /unlink 26063 (time 1746145056.63 total 126.38 last 196.89) - /unlink 28149 (time 1746145066.63 total 136.38 last 208.47) - /unlink 30000 (time 1746145075.80 total 145.55 last 201.90) - /unlink 32292 (time 1746145085.81 total 155.55 last 229.08) - /unlink 34930 (time 1746145095.81 total 165.56 last 263.70) - /unlink 37343 (time 1746145105.81 total 175.56 last 241.25) - /unlink 39422 (time 1746145115.81 total 185.56 last 207.90) - /unlink 40000 (time 1746145118.46 total 188.20 last 218.71) - /unlink 42406 (time 1746145128.46 total 198.21 last 240.47) - /unlink 44932 (time 1746145138.46 total 208.21 last 252.57) - /unlink 47470 (time 1746145148.47 total 218.22 last 253.60) - /unlink 50000 (time 1746145158.03 total 227.78 last 264.66) - /unlink 52406 (time 1746145168.03 total 237.78 last 240.53) - /unlink 54583 (time 1746145178.04 total 247.78 last 217.62) - /unlink 57742 (time 1746145188.04 total 257.79 last 315.83) - /unlink 60000 (time 1746145196.70 total 266.45 last 260.53) - /unlink 62631 (time 1746145206.71 total 276.46 last 263.02) total: 64768 /unlink in 284.87 seconds: 227.36 ops/second Waiting for MDT destroys to complete - open/close 1344 (time 1746145239.36 total 10.00 last 134.36) - open/close 2852 (time 1746145249.36 total 20.00 last 150.79) - open/close 4086 (time 1746145259.36 total 30.01 last 123.38) - open/close 5267 (time 1746145269.36 total 40.01 last 118.03) - open/close 6612 (time 1746145279.38 total 50.02 last 134.32) - open/close 7881 (time 1746145289.38 total 60.03 last 126.86) - open/close 9092 (time 1746145299.38 total 70.03 last 121.09) - open/close 10000 (time 1746145306.32 total 76.96 last 130.91) - open/close 11264 (time 1746145316.32 total 86.97 last 126.36) - open/close 12750 (time 1746145326.33 total 96.98 last 148.44) - open/close 14263 (time 1746145336.34 total 106.99 last 151.17) - open/close 15876 (time 1746145346.34 total 116.99 last 161.24) - open/close 17310 (time 1746145356.35 total 127.00 last 143.27) - open/close 18714 (time 1746145366.36 total 137.00 last 140.35) - open/close 20000 (time 1746145376.04 total 146.68 last 132.85) - open/close 21479 (time 1746145386.04 total 156.69 last 147.85) - open/close 22848 (time 1746145396.04 total 166.69 last 136.90) - open/close 24364 (time 1746145406.04 total 176.69 last 151.52) - open/close 25689 (time 1746145416.05 total 186.70 last 132.40) - open/close 27132 (time 1746145426.06 total 196.71 last 144.21) - open/close 28520 (time 1746145436.06 total 206.71 last 138.75) - open/close 30000 (time 1746145445.54 total 216.19 last 156.16) - open/close 31575 (time 1746145455.54 total 226.19 last 157.46) - open/close 32906 (time 1746145465.55 total 236.20 last 133.00) - open/close 34339 (time 1746145475.55 total 246.20 last 143.28) - open/close 35831 (time 1746145485.55 total 256.20 last 149.19) - open/close 37223 (time 1746145495.57 total 266.21 last 139.01) - open/close 38525 (time 1746145505.57 total 276.22 last 130.15) - open/close 39768 (time 1746145515.57 total 286.22 last 124.26) - open/close 41068 (time 1746145525.58 total 296.23 last 129.93) - open/close 42252 (time 1746145535.59 total 306.24 last 118.28) - open/close 43410 (time 1746145545.60 total 316.25 last 115.66) - open/close 44577 (time 1746145555.61 total 326.26 last 116.62) - open/close 45961 (time 1746145565.61 total 336.26 last 138.33) - open/close 47265 (time 1746145575.61 total 346.26 last 130.38) - open/close 48663 (time 1746145585.62 total 356.27 last 139.68) - open/close 49917 (time 1746145595.63 total 366.27 last 125.35) - open/close 50852 (time 1746145605.65 total 376.30 last 93.30) - open/close 51984 (time 1746145615.65 total 386.30 last 113.16) - open/close 53339 (time 1746145625.65 total 396.30 last 135.50) - open/close 54740 (time 1746145635.65 total 406.30 last 140.07) - open/close 56086 (time 1746145645.65 total 416.30 last 134.60) - open/close 57389 (time 1746145655.66 total 426.31 last 130.23) - open/close 58882 (time 1746145665.66 total 436.31 last 149.24) - open/close 60000 (time 1746145673.73 total 444.38 last 138.55) - open/close 61314 (time 1746145683.74 total 454.39 last 131.28) - open/close 62685 (time 1746145693.74 total 464.39 last 137.09) - open/close 63968 (time 1746145703.74 total 474.39 last 128.28) total: 64768 open/close in 480.17 seconds: 134.88 ops/second - /unlink 2135 (time 1746145721.17 total 10.00 last 213.50) - /unlink 4356 (time 1746145731.17 total 20.01 last 221.95) - /unlink 6655 (time 1746145741.18 total 30.02 last 229.70) - /unlink 9019 (time 1746145751.18 total 40.02 last 236.36) - /unlink 10000 (time 1746145755.32 total 44.15 last 237.14) - /unlink 12075 (time 1746145765.33 total 54.16 last 207.43) - /unlink 14065 (time 1746145775.33 total 64.16 last 198.90) - /unlink 16109 (time 1746145785.33 total 74.16 last 204.37) - /unlink 18331 (time 1746145795.34 total 84.17 last 222.08) - /unlink 20000 (time 1746145803.03 total 91.86 last 216.95) - /unlink 22562 (time 1746145813.03 total 101.87 last 256.13) - /unlink 25201 (time 1746145823.03 total 111.87 last 263.89) - /unlink 28071 (time 1746145833.04 total 121.87 last 286.95) - /unlink 30000 (time 1746145840.83 total 129.67 last 247.40) - /unlink 32551 (time 1746145850.83 total 139.67 last 255.05) - /unlink 34953 (time 1746145860.83 total 149.67 last 240.20) - /unlink 37326 (time 1746145870.84 total 159.67 last 237.27) - /unlink 39680 (time 1746145880.84 total 169.67 last 235.32) - /unlink 42030 (time 1746145890.85 total 179.68 last 234.84) - /unlink 44512 (time 1746145900.85 total 189.68 last 248.16) - /unlink 46997 (time 1746145910.85 total 199.68 last 248.41) - /unlink 49426 (time 1746145920.85 total 209.68 last 242.89) - /unlink 50000 (time 1746145923.07 total 211.90 last 259.33) - /unlink 52376 (time 1746145933.07 total 221.90 last 237.59) - /unlink 54716 (time 1746145943.07 total 231.90 last 233.99) - /unlink 57317 (time 1746145953.07 total 241.90 last 260.05) - /unlink 60000 (time 1746145962.52 total 251.35 last 283.85) - /unlink 62346 (time 1746145972.52 total 261.36 last 234.52) total: 64768 /unlink in 271.35 seconds: 238.69 ops/second Waiting for MDT destroys to complete - open/close 1709 (time 1746146000.68 total 10.00 last 170.88) - open/close 3275 (time 1746146010.68 total 20.01 last 156.53) - open/close 4841 (time 1746146020.69 total 30.01 last 156.51) - open/close 6244 (time 1746146030.69 total 40.01 last 140.25) - open/close 7822 (time 1746146040.69 total 50.02 last 157.74) - open/close 9421 (time 1746146050.70 total 60.02 last 159.80) - open/close 10000 (time 1746146054.33 total 63.65 last 159.56) - open/close 11638 (time 1746146064.33 total 73.66 last 163.77) - open/close 13331 (time 1746146074.33 total 83.66 last 169.23) - open/close 14929 (time 1746146084.34 total 93.66 last 159.74) - open/close 16422 (time 1746146094.34 total 103.67 last 149.26) - open/close 17875 (time 1746146104.34 total 113.67 last 145.26) - open/close 19449 (time 1746146114.35 total 123.67 last 157.32) - open/close 20000 (time 1746146118.17 total 127.49 last 144.37) - open/close 21655 (time 1746146128.17 total 137.49 last 165.44) - open/close 23309 (time 1746146138.17 total 147.50 last 165.31) - open/close 24871 (time 1746146148.18 total 157.50 last 156.18) - open/close 26387 (time 1746146158.18 total 167.50 last 151.55) - open/close 27774 (time 1746146168.18 total 177.50 last 138.70) - open/close 29242 (time 1746146178.18 total 187.51 last 146.73) - open/close 30000 (time 1746146182.51 total 191.84 last 175.06) - open/close 31487 (time 1746146192.52 total 201.85 last 148.61) - open/close 33420 (time 1746146202.52 total 211.85 last 193.23) - open/close 35041 (time 1746146212.53 total 221.85 last 162.05) - open/close 36594 (time 1746146222.53 total 231.86 last 155.25) - open/close 38079 (time 1746146232.53 total 241.86 last 148.48) - open/close 39828 (time 1746146242.53 total 251.86 last 174.87) - open/close 41418 (time 1746146252.54 total 261.86 last 158.93) - open/close 42877 (time 1746146262.54 total 271.87 last 145.82) - open/close 44433 (time 1746146272.55 total 281.87 last 155.55) - open/close 46146 (time 1746146282.55 total 291.87 last 171.29) - open/close 47696 (time 1746146292.56 total 301.88 last 154.86) - open/close 49334 (time 1746146302.56 total 311.88 last 163.76) - open/close 50000 (time 1746146306.36 total 315.68 last 175.21) - open/close 51527 (time 1746146316.36 total 325.69 last 152.67) - open/close 53322 (time 1746146326.37 total 335.69 last 179.41) - open/close 54922 (time 1746146336.37 total 345.70 last 159.90) - open/close 56510 (time 1746146346.37 total 355.70 last 158.78) - open/close 58203 (time 1746146356.38 total 365.71 last 169.19) - open/close 59844 (time 1746146366.39 total 375.71 last 164.01) - open/close 61384 (time 1746146376.39 total 385.71 last 153.99) - open/close 62867 (time 1746146386.39 total 395.71 last 148.26) - open/close 64335 (time 1746146396.39 total 405.72 last 146.73) total: 64768 open/close in 408.57 seconds: 158.52 ops/second - /unlink 2763 (time 1746146410.96 total 10.00 last 276.24) - /unlink 5010 (time 1746146420.97 total 20.01 last 224.61) - /unlink 7331 (time 1746146430.97 total 30.01 last 232.02) - /unlink 10000 (time 1746146440.79 total 39.83 last 271.66) - /unlink 12293 (time 1746146450.80 total 49.84 last 229.29) - /unlink 15217 (time 1746146460.80 total 59.84 last 292.33) - /unlink 17915 (time 1746146470.80 total 69.84 last 269.64) - /unlink 20000 (time 1746146478.98 total 78.02 last 254.96) - /unlink 22666 (time 1746146488.99 total 88.03 last 266.28) - /unlink 25321 (time 1746146498.99 total 98.03 last 265.50) - /unlink 27630 (time 1746146508.99 total 108.03 last 230.90) - /unlink 30000 (time 1746146518.44 total 117.48 last 250.97) - /unlink 32458 (time 1746146528.44 total 127.48 last 245.78) - /unlink 34743 (time 1746146538.44 total 137.48 last 228.41) - /unlink 37106 (time 1746146548.44 total 147.48 last 236.27) - /unlink 39430 (time 1746146558.45 total 157.49 last 232.32) - /unlink 40000 (time 1746146560.73 total 159.77 last 249.15) - /unlink 42406 (time 1746146570.74 total 169.78 last 240.54) - /unlink 44635 (time 1746146580.74 total 179.78 last 222.75) - /unlink 46922 (time 1746146590.75 total 189.79 last 228.61) - /unlink 49287 (time 1746146600.75 total 199.79 last 236.38) - /unlink 50000 (time 1746146603.84 total 202.88 last 230.97) - /unlink 52107 (time 1746146613.84 total 212.88 last 210.65) - /unlink 54444 (time 1746146623.84 total 222.88 last 233.69) - /unlink 56661 (time 1746146633.85 total 232.89 last 221.51) - /unlink 58634 (time 1746146643.85 total 242.89 last 197.27) - /unlink 60000 (time 1746146649.06 total 248.10 last 262.06) - /unlink 62297 (time 1746146659.07 total 258.10 last 229.69) - /unlink 64729 (time 1746146669.07 total 268.11 last 243.10) total: 64768 /unlink in 268.25 seconds: 241.44 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg205-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg205-server unloading modules via unload_modules_local on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg205-client: oleg205-server: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-client: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-client: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 2