== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:20:14 (1746145214) Stopping clients: oleg423-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg423-client.virtnet /mnt/lustre2 (opts:-f) oleg423-server: oleg423-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions oleg423-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg423-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg423-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2123 (time 1746145257.96 total 10.00 last 212.20) - open/close 4009 (time 1746145267.97 total 20.01 last 188.52) - open/close 5978 (time 1746145277.97 total 30.01 last 196.85) - open/close 7533 (time 1746145287.98 total 40.02 last 155.40) - open/close 9150 (time 1746145297.98 total 50.02 last 161.68) - open/close 10000 (time 1746145301.49 total 53.53 last 242.08) - open/close 11538 (time 1746145311.50 total 63.54 last 153.67) - open/close 13422 (time 1746145321.50 total 73.54 last 188.38) - open/close 16158 (time 1746145331.50 total 83.54 last 273.57) - open/close 18573 (time 1746145341.51 total 93.55 last 241.35) - open/close 20000 (time 1746145348.79 total 100.83 last 195.93) - open/close 21971 (time 1746145358.79 total 110.83 last 197.09) - open/close 24280 (time 1746145368.79 total 120.83 last 230.84) - open/close 26003 (time 1746145378.79 total 130.84 last 172.27) - open/close 28069 (time 1746145388.80 total 140.84 last 206.52) - open/close 29706 (time 1746145398.80 total 150.84 last 163.66) - open/close 31258 (time 1746145408.81 total 160.85 last 155.13) - open/close 33108 (time 1746145418.81 total 170.85 last 184.88) - open/close 35814 (time 1746145428.82 total 180.86 last 270.41) - open/close 37705 (time 1746145438.82 total 190.86 last 189.05) - open/close 39907 (time 1746145448.82 total 200.86 last 220.16) - open/close 41603 (time 1746145458.83 total 210.87 last 169.56) - open/close 43953 (time 1746145468.83 total 220.87 last 234.94) - open/close 46416 (time 1746145478.83 total 230.87 last 246.28) - open/close 48646 (time 1746145488.83 total 240.88 last 222.88) - open/close 50000 (time 1746145497.72 total 249.77 last 152.30) - open/close 51580 (time 1746145507.73 total 259.77 last 158.00) - open/close 53675 (time 1746145517.73 total 269.77 last 209.48) - open/close 55447 (time 1746145527.73 total 279.77 last 177.20) - open/close 57111 (time 1746145537.73 total 289.77 last 166.39) - open/close 58938 (time 1746145547.73 total 299.77 last 182.62) - open/close 60000 (time 1746145555.54 total 307.58 last 135.96) - open/close 61052 (time 1746145565.55 total 317.59 last 105.11) - open/close 62120 (time 1746145575.56 total 327.60 last 106.71) - open/close 63156 (time 1746145585.56 total 337.60 last 103.57) - open/close 64420 (time 1746145595.56 total 347.61 last 126.37) total: 64768 open/close in 349.61 seconds: 185.26 ops/second - /unlink 2052 (time 1746145608.81 total 10.00 last 205.15) - /unlink 4388 (time 1746145618.81 total 20.00 last 233.57) - /unlink 6855 (time 1746145628.81 total 30.00 last 246.70) - /unlink 8986 (time 1746145638.81 total 40.00 last 213.08) - /unlink 10000 (time 1746145643.65 total 44.85 last 209.46) - /unlink 12340 (time 1746145653.65 total 54.85 last 233.96) - /unlink 14846 (time 1746145663.66 total 64.86 last 250.38) - /unlink 17039 (time 1746145673.67 total 74.86 last 219.15) - /unlink 19266 (time 1746145683.67 total 84.87 last 222.59) - /unlink 20000 (time 1746145686.45 total 87.65 last 264.22) - /unlink 22352 (time 1746145696.45 total 97.65 last 235.13) - /unlink 24633 (time 1746145706.46 total 107.65 last 228.08) - /unlink 27426 (time 1746145716.46 total 117.65 last 279.24) - /unlink 30000 (time 1746145725.61 total 126.81 last 281.16) - /unlink 32725 (time 1746145735.61 total 136.81 last 272.44) - /unlink 35316 (time 1746145745.61 total 146.81 last 259.09) - /unlink 38586 (time 1746145755.62 total 156.81 last 326.95) - /unlink 40000 (time 1746145760.73 total 161.92 last 276.71) - /unlink 43155 (time 1746145770.73 total 171.92 last 315.48) - /unlink 45400 (time 1746145780.73 total 181.92 last 224.47) - /unlink 48911 (time 1746145790.73 total 191.92 last 351.08) - /unlink 50000 (time 1746145793.25 total 194.45 last 431.52) - /unlink 52680 (time 1746145803.25 total 204.45 last 267.98) - /unlink 55267 (time 1746145813.25 total 214.45 last 258.69) - /unlink 58458 (time 1746145823.25 total 224.45 last 319.08) - /unlink 60000 (time 1746145828.17 total 229.37 last 313.60) - /unlink 62946 (time 1746145838.17 total 239.37 last 294.60) total: 64768 /unlink in 245.08 seconds: 264.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1982 (time 1746145870.94 total 10.00 last 198.19) - open/close 3795 (time 1746145880.94 total 20.01 last 181.21) - open/close 5583 (time 1746145890.94 total 30.01 last 178.78) - open/close 7232 (time 1746145900.94 total 40.01 last 164.88) - open/close 8723 (time 1746145910.94 total 50.01 last 149.07) - open/close 9978 (time 1746145920.95 total 60.01 last 125.45) - open/close 11295 (time 1746145930.95 total 70.02 last 131.68) - open/close 12599 (time 1746145940.96 total 80.02 last 130.30) - open/close 14091 (time 1746145950.96 total 90.03 last 149.11) - open/close 15334 (time 1746145960.97 total 100.03 last 124.24) - open/close 16568 (time 1746145970.97 total 110.04 last 123.35) - open/close 17904 (time 1746145980.97 total 120.04 last 133.57) - open/close 19444 (time 1746145990.98 total 130.04 last 153.97) - open/close 20000 (time 1746145995.57 total 134.63 last 121.15) - open/close 21174 (time 1746146005.57 total 144.64 last 117.34) - open/close 22311 (time 1746146015.58 total 154.64 last 113.61) - open/close 23443 (time 1746146025.59 total 164.65 last 113.11) - open/close 24708 (time 1746146035.59 total 174.66 last 126.46) - open/close 26016 (time 1746146045.60 total 184.66 last 130.72) - open/close 27270 (time 1746146055.60 total 194.67 last 125.34) - open/close 28589 (time 1746146065.61 total 204.68 last 131.79) - open/close 29928 (time 1746146075.61 total 214.68 last 133.86) - open/close 31365 (time 1746146085.61 total 224.68 last 143.67) - open/close 32656 (time 1746146095.62 total 234.68 last 129.09) - open/close 34097 (time 1746146105.62 total 244.69 last 144.03) - open/close 35670 (time 1746146115.62 total 254.69 last 157.28) - open/close 36943 (time 1746146125.62 total 264.69 last 127.29) - open/close 38007 (time 1746146135.63 total 274.69 last 106.34) - open/close 39133 (time 1746146145.63 total 284.70 last 112.55) - open/close 40000 (time 1746146152.74 total 291.81 last 122.00) - open/close 41360 (time 1746146162.75 total 301.81 last 135.93) - open/close 43182 (time 1746146172.75 total 311.81 last 182.12) - open/close 44943 (time 1746146182.75 total 321.82 last 176.09) - open/close 46597 (time 1746146192.76 total 331.82 last 165.31) - open/close 48023 (time 1746146202.76 total 341.82 last 142.59) - open/close 49395 (time 1746146212.76 total 351.83 last 137.11) - open/close 50000 (time 1746146216.03 total 355.10 last 185.16) - open/close 51545 (time 1746146226.03 total 365.10 last 154.43) - open/close 52777 (time 1746146236.04 total 375.10 last 123.14) - open/close 54196 (time 1746146246.05 total 385.11 last 141.80) - open/close 55525 (time 1746146256.05 total 395.12 last 132.80) - open/close 56737 (time 1746146266.06 total 405.13 last 121.12) - open/close 57853 (time 1746146276.06 total 415.13 last 111.59) - open/close 59532 (time 1746146286.07 total 425.13 last 167.80) - open/close 60000 (time 1746146289.44 total 428.50 last 138.93) - open/close 61737 (time 1746146299.44 total 438.50 last 173.64) - open/close 63118 (time 1746146309.45 total 448.51 last 137.97) total: 64768 open/close in 458.09 seconds: 141.39 ops/second - /unlink 2066 (time 1746146330.47 total 10.00 last 206.56) - /unlink 3993 (time 1746146340.48 total 20.01 last 192.59) - /unlink 6509 (time 1746146350.48 total 30.01 last 251.49) - /unlink 8851 (time 1746146360.49 total 40.02 last 234.12) - /unlink 10000 (time 1746146366.04 total 45.57 last 206.96) - /unlink 12047 (time 1746146376.04 total 55.57 last 204.56) - /unlink 14147 (time 1746146386.05 total 65.58 last 209.98) - /unlink 16496 (time 1746146396.05 total 75.58 last 234.88) - /unlink 19086 (time 1746146406.05 total 85.58 last 258.82) - /unlink 20000 (time 1746146409.51 total 89.04 last 264.18) - /unlink 22611 (time 1746146419.52 total 99.05 last 261.02) - /unlink 25224 (time 1746146429.52 total 109.05 last 261.29) - /unlink 27328 (time 1746146439.52 total 119.05 last 210.39) - /unlink 29743 (time 1746146449.52 total 129.05 last 241.48) - /unlink 32303 (time 1746146459.52 total 139.05 last 255.94) - /unlink 34656 (time 1746146469.52 total 149.05 last 235.24) - /unlink 37248 (time 1746146479.52 total 159.05 last 259.18) - /unlink 39305 (time 1746146489.53 total 169.06 last 205.63) - /unlink 40000 (time 1746146492.54 total 172.07 last 230.72) - /unlink 42107 (time 1746146502.54 total 182.07 last 210.63) - /unlink 44038 (time 1746146512.55 total 192.08 last 193.00) - /unlink 45846 (time 1746146522.55 total 202.08 last 180.73) - /unlink 47958 (time 1746146532.55 total 212.08 last 211.19) - /unlink 50000 (time 1746146541.84 total 221.37 last 219.89) - /unlink 52520 (time 1746146551.84 total 231.37 last 251.95) - /unlink 55140 (time 1746146561.84 total 241.37 last 261.87) - /unlink 57854 (time 1746146571.85 total 251.38 last 271.34) - /unlink 60000 (time 1746146579.93 total 259.46 last 265.51) - /unlink 63258 (time 1746146589.93 total 269.46 last 325.73) total: 64768 /unlink in 275.51 seconds: 235.08 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1469 (time 1746146622.55 total 10.00 last 146.85) - open/close 3283 (time 1746146632.55 total 20.01 last 181.31) - open/close 4612 (time 1746146642.55 total 30.01 last 132.88) - open/close 6489 (time 1746146652.55 total 40.01 last 187.70) - open/close 8214 (time 1746146662.56 total 50.01 last 172.41) - open/close 9740 (time 1746146672.56 total 60.02 last 152.55) - open/close 10000 (time 1746146674.60 total 62.06 last 127.31) - open/close 11445 (time 1746146684.61 total 72.06 last 144.43) - open/close 12706 (time 1746146694.61 total 82.07 last 126.08) - open/close 14077 (time 1746146704.62 total 92.07 last 136.98) - open/close 15548 (time 1746146714.62 total 102.08 last 147.03) - open/close 17150 (time 1746146724.62 total 112.08 last 160.19) - open/close 18785 (time 1746146734.63 total 122.09 last 163.33) - open/close 20000 (time 1746146743.13 total 130.59 last 142.94) - open/close 21388 (time 1746146753.14 total 140.59 last 138.77) - open/close 22814 (time 1746146763.14 total 150.60 last 142.53) - open/close 24521 (time 1746146773.15 total 160.60 last 170.62) - open/close 26251 (time 1746146783.15 total 170.60 last 172.99) - open/close 27696 (time 1746146793.15 total 180.61 last 144.46) - open/close 29200 (time 1746146803.16 total 190.62 last 150.24) - open/close 30000 (time 1746146810.91 total 198.36 last 103.26) - open/close 31062 (time 1746146820.91 total 208.37 last 106.18) - open/close 32158 (time 1746146830.92 total 218.37 last 109.52) - open/close 33407 (time 1746146840.92 total 228.38 last 124.85) - open/close 34700 (time 1746146850.93 total 238.38 last 129.24) - open/close 36130 (time 1746146860.93 total 248.39 last 142.92) - open/close 37412 (time 1746146870.94 total 258.39 last 128.13) - open/close 38496 (time 1746146880.95 total 268.40 last 108.30) - open/close 39567 (time 1746146890.95 total 278.41 last 107.02) - open/close 40000 (time 1746146894.58 total 282.03 last 119.57) - open/close 41316 (time 1746146904.58 total 292.03 last 131.57) - open/close 42530 (time 1746146914.58 total 302.04 last 121.35) - open/close 43550 (time 1746146924.59 total 312.05 last 101.88) - open/close 44711 (time 1746146934.60 total 322.06 last 115.99) - open/close 45834 (time 1746146944.60 total 332.06 last 112.29) - open/close 47094 (time 1746146954.61 total 342.07 last 125.92) - open/close 48064 (time 1746146964.61 total 352.07 last 96.99) - open/close 49098 (time 1746146974.61 total 362.07 last 103.37) - open/close 50000 (time 1746146983.82 total 371.28 last 97.95) - open/close 51057 (time 1746146993.83 total 381.29 last 105.57) - open/close 52212 (time 1746147003.84 total 391.30 last 115.44) - open/close 53366 (time 1746147013.84 total 401.30 last 115.39) - open/close 54590 (time 1746147023.85 total 411.31 last 122.23) - open/close 55735 (time 1746147033.86 total 421.32 last 114.42) - open/close 57004 (time 1746147043.87 total 431.33 last 126.80) - open/close 58120 (time 1746147053.88 total 441.34 last 111.49) - open/close 59306 (time 1746147063.89 total 451.35 last 118.44) - open/close 60000 (time 1746147069.88 total 457.34 last 115.89) - open/close 61214 (time 1746147079.89 total 467.35 last 121.30) - open/close 62518 (time 1746147089.89 total 477.35 last 130.34) - open/close 63825 (time 1746147099.90 total 487.35 last 130.69) total: 64768 open/close in 496.07 seconds: 130.56 ops/second - /unlink 1911 (time 1746147120.46 total 10.00 last 191.03) - /unlink 4001 (time 1746147130.46 total 20.00 last 208.98) - /unlink 6094 (time 1746147140.47 total 30.01 last 209.17) - /unlink 8122 (time 1746147150.47 total 40.01 last 202.72) - /unlink 10000 (time 1746147158.32 total 47.86 last 239.49) - /unlink 11753 (time 1746147168.32 total 57.86 last 175.19) - /unlink 13577 (time 1746147178.32 total 67.86 last 182.38) - /unlink 15363 (time 1746147188.33 total 77.87 last 178.48) - /unlink 17540 (time 1746147198.33 total 87.87 last 217.68) - /unlink 19563 (time 1746147208.34 total 97.88 last 202.15) - /unlink 20000 (time 1746147210.73 total 100.27 last 182.51) - /unlink 21713 (time 1746147220.74 total 110.28 last 171.23) - /unlink 23763 (time 1746147230.74 total 120.28 last 204.95) - /unlink 25484 (time 1746147240.74 total 130.28 last 172.06) - /unlink 27395 (time 1746147250.74 total 140.28 last 191.05) - /unlink 29327 (time 1746147260.75 total 150.29 last 193.16) - /unlink 30000 (time 1746147263.89 total 153.43 last 214.01) - /unlink 32374 (time 1746147273.89 total 163.43 last 237.37) - /unlink 34526 (time 1746147283.89 total 173.43 last 215.20) - /unlink 36662 (time 1746147293.90 total 183.44 last 213.55) - /unlink 38717 (time 1746147303.90 total 193.44 last 205.48) - /unlink 40000 (time 1746147309.58 total 199.12 last 225.73) - /unlink 42066 (time 1746147319.58 total 209.12 last 206.57) - /unlink 44373 (time 1746147329.58 total 219.12 last 230.68) - /unlink 46368 (time 1746147339.58 total 229.12 last 199.47) - /unlink 48336 (time 1746147349.58 total 239.12 last 196.80) - /unlink 50000 (time 1746147358.37 total 247.91 last 189.37) - /unlink 51910 (time 1746147368.37 total 257.91 last 190.96) - /unlink 53737 (time 1746147378.38 total 267.92 last 182.65) - /unlink 55885 (time 1746147388.38 total 277.92 last 214.72) - /unlink 57768 (time 1746147398.38 total 287.92 last 188.22) - /unlink 59954 (time 1746147408.39 total 297.93 last 218.55) - /unlink 62982 (time 1746147418.39 total 307.93 last 302.63) total: 64768 /unlink in 315.20 seconds: 205.48 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1200 (time 1746147453.27 total 10.00 last 119.94) - open/close 2567 (time 1746147463.27 total 20.01 last 136.69) - open/close 3941 (time 1746147473.27 total 30.01 last 137.40) - open/close 5321 (time 1746147483.27 total 40.01 last 137.94) - open/close 6651 (time 1746147493.28 total 50.02 last 132.93) - open/close 8014 (time 1746147503.28 total 60.02 last 136.29) - open/close 9430 (time 1746147513.28 total 70.02 last 141.56) - open/close 10000 (time 1746147516.67 total 73.41 last 168.20) - open/close 11717 (time 1746147526.67 total 83.41 last 171.68) - open/close 13020 (time 1746147536.67 total 93.41 last 130.26) - open/close 14199 (time 1746147546.67 total 103.41 last 117.89) - open/close 15511 (time 1746147556.68 total 113.41 last 131.17) - open/close 16766 (time 1746147566.68 total 123.42 last 125.47) - open/close 17892 (time 1746147576.68 total 133.42 last 112.58) - open/close 18984 (time 1746147586.69 total 143.42 last 109.15) - open/close 20000 (time 1746147595.11 total 151.84 last 120.67) - open/close 21160 (time 1746147605.11 total 161.85 last 115.98) - open/close 22413 (time 1746147615.11 total 171.85 last 125.30) - open/close 23438 (time 1746147625.12 total 181.86 last 102.40) - open/close 24319 (time 1746147635.12 total 191.86 last 88.05) - open/close 25145 (time 1746147645.13 total 201.86 last 82.58) - open/close 26119 (time 1746147655.13 total 211.87 last 97.33) - open/close 27124 (time 1746147665.14 total 221.88 last 100.40) - open/close 28269 (time 1746147675.14 total 231.88 last 114.49) - open/close 29590 (time 1746147685.15 total 241.89 last 132.01) - open/close 30000 (time 1746147687.67 total 244.41 last 162.78) - open/close 31410 (time 1746147697.67 total 254.41 last 140.99) - open/close 32732 (time 1746147707.68 total 264.42 last 132.03) - open/close 33964 (time 1746147717.69 total 274.43 last 123.07) - open/close 35488 (time 1746147727.69 total 284.43 last 152.40) - open/close 36982 (time 1746147737.70 total 294.44 last 149.32) - open/close 38565 (time 1746147747.71 total 304.44 last 158.20) - open/close 39959 (time 1746147757.71 total 314.44 last 139.39) - open/close 41315 (time 1746147767.71 total 324.45 last 135.59) - open/close 42609 (time 1746147777.71 total 334.45 last 129.37) - open/close 44023 (time 1746147787.71 total 344.45 last 141.34) - open/close 45265 (time 1746147797.71 total 354.45 last 124.19) - open/close 46544 (time 1746147807.72 total 364.46 last 127.77) - open/close 48079 (time 1746147817.73 total 374.47 last 153.46) - open/close 49628 (time 1746147827.73 total 384.47 last 154.84) - open/close 50000 (time 1746147829.75 total 386.49 last 184.39) - open/close 51875 (time 1746147839.75 total 396.49 last 187.49) - open/close 53520 (time 1746147849.75 total 406.49 last 164.41) - open/close 55394 (time 1746147859.76 total 416.50 last 187.28) - open/close 56723 (time 1746147869.77 total 426.51 last 132.83) - open/close 58102 (time 1746147879.77 total 436.51 last 137.82) - open/close 59456 (time 1746147889.77 total 446.51 last 135.38) - open/close 60000 (time 1746147893.71 total 450.45 last 138.33) - open/close 61549 (time 1746147903.71 total 460.45 last 154.86) - open/close 63043 (time 1746147913.71 total 470.45 last 149.40) total: 64768 open/close in 480.36 seconds: 134.83 ops/second - /unlink 1819 (time 1746147935.32 total 10.00 last 181.86) - /unlink 3929 (time 1746147945.32 total 20.00 last 211.00) - /unlink 5986 (time 1746147955.32 total 30.00 last 205.66) - /unlink 8146 (time 1746147965.32 total 40.01 last 215.94) - /unlink 10000 (time 1746147973.24 total 47.93 last 234.05) - /unlink 12702 (time 1746147983.24 total 57.93 last 270.19) - /unlink 14974 (time 1746147993.25 total 67.93 last 227.16) - /unlink 17159 (time 1746148003.25 total 77.93 last 218.41) - /unlink 18967 (time 1746148013.25 total 87.94 last 180.76) - /unlink 20000 (time 1746148017.92 total 92.61 last 221.08) - /unlink 22019 (time 1746148027.93 total 102.61 last 201.85) - /unlink 23890 (time 1746148037.93 total 112.61 last 187.08) - /unlink 25867 (time 1746148047.93 total 122.61 last 197.67) - /unlink 27966 (time 1746148057.93 total 132.62 last 209.85) - /unlink 30000 (time 1746148067.06 total 141.74 last 222.85) - /unlink 31960 (time 1746148077.06 total 151.75 last 195.96) - /unlink 33888 (time 1746148087.06 total 161.75 last 192.74) - /unlink 35966 (time 1746148097.06 total 171.75 last 207.79) - /unlink 37999 (time 1746148107.07 total 181.75 last 203.18) - /unlink 40000 (time 1746148116.85 total 191.54 last 204.57) - /unlink 42516 (time 1746148126.85 total 201.54 last 251.53) - /unlink 45170 (time 1746148136.86 total 211.55 last 265.22) - /unlink 47816 (time 1746148146.87 total 221.55 last 264.49) - /unlink 50000 (time 1746148155.24 total 229.92 last 260.91) - /unlink 53515 (time 1746148165.24 total 239.92 last 351.42) - /unlink 56745 (time 1746148175.24 total 249.93 last 322.90) - /unlink 59871 (time 1746148185.24 total 259.93 last 312.56) - /unlink 63296 (time 1746148195.24 total 269.93 last 342.49) total: 64768 /unlink in 274.52 seconds: 235.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1349 (time 1746148227.66 total 10.00 last 134.90) - open/close 2715 (time 1746148237.67 total 20.00 last 136.59) - open/close 3940 (time 1746148247.67 total 30.00 last 122.47) - open/close 5182 (time 1746148257.67 total 40.01 last 124.12) - open/close 6443 (time 1746148267.68 total 50.01 last 126.06) - open/close 7840 (time 1746148277.68 total 60.02 last 139.62) - open/close 9425 (time 1746148287.69 total 70.03 last 158.33) - open/close 10000 (time 1746148291.26 total 73.60 last 161.22) - open/close 11011 (time 1746148301.27 total 83.60 last 101.04) - open/close 12215 (time 1746148311.27 total 93.61 last 120.31) - open/close 13323 (time 1746148321.28 total 103.62 last 110.73) - open/close 14352 (time 1746148331.28 total 113.62 last 102.87) - open/close 15526 (time 1746148341.29 total 123.62 last 117.38) - open/close 16688 (time 1746148351.29 total 133.63 last 116.11) - open/close 17944 (time 1746148361.29 total 143.63 last 125.60) - open/close 19283 (time 1746148371.30 total 153.63 last 133.81) - open/close 20000 (time 1746148377.71 total 160.04 last 111.91) - open/close 21107 (time 1746148387.72 total 170.06 last 110.54) - open/close 22415 (time 1746148397.73 total 180.06 last 130.74) - open/close 23657 (time 1746148407.73 total 190.07 last 124.14) - open/close 24926 (time 1746148417.73 total 200.07 last 126.89) - open/close 26041 (time 1746148427.73 total 210.07 last 111.47) - open/close 27106 (time 1746148437.76 total 220.09 last 106.26) - open/close 28234 (time 1746148447.76 total 230.10 last 112.73) - open/close 29399 (time 1746148457.76 total 240.10 last 116.49) - open/close 30000 (time 1746148463.43 total 245.76 last 106.14) - open/close 31104 (time 1746148473.43 total 255.76 last 110.39) - open/close 32349 (time 1746148483.43 total 265.76 last 124.47) - open/close 34178 (time 1746148493.43 total 275.77 last 182.87) - open/close 35964 (time 1746148503.43 total 285.77 last 178.55) - open/close 38111 (time 1746148513.43 total 295.77 last 214.69) - open/close 39932 (time 1746148523.44 total 305.77 last 182.04) - open/close 41360 (time 1746148533.44 total 315.77 last 142.79) - open/close 42706 (time 1746148543.44 total 325.77 last 134.60) - open/close 44071 (time 1746148553.44 total 335.77 last 136.48) - open/close 45545 (time 1746148563.44 total 345.78 last 147.36) - open/close 46851 (time 1746148573.45 total 355.78 last 130.54) - open/close 48111 (time 1746148583.45 total 365.78 last 125.99) - open/close 49490 (time 1746148593.45 total 375.78 last 137.87) - open/close 50000 (time 1746148597.20 total 379.53 last 136.15) - open/close 51342 (time 1746148607.20 total 389.53 last 134.18) - open/close 52555 (time 1746148617.20 total 399.53 last 121.27) - open/close 53986 (time 1746148627.20 total 409.54 last 143.03) - open/close 55311 (time 1746148637.21 total 419.54 last 132.45) - open/close 56919 (time 1746148647.21 total 429.55 last 160.72) - open/close 58322 (time 1746148657.22 total 439.55 last 140.23) - open/close 59728 (time 1746148667.22 total 449.55 last 140.59) - open/close 60000 (time 1746148669.25 total 451.58 last 133.91) - open/close 61588 (time 1746148679.25 total 461.59 last 158.72) - open/close 63095 (time 1746148689.26 total 471.59 last 150.66) - open/close 64537 (time 1746148699.26 total 481.59 last 144.18) total: 64768 open/close in 483.34 seconds: 134.00 ops/second - /unlink 2625 (time 1746148712.32 total 10.00 last 262.48) - /unlink 5086 (time 1746148722.33 total 20.01 last 245.89) - /unlink 7196 (time 1746148732.33 total 30.01 last 210.98) - /unlink 9245 (time 1746148742.33 total 40.01 last 204.81) - /unlink 10000 (time 1746148745.62 total 43.30 last 229.79) - /unlink 12412 (time 1746148755.62 total 53.30 last 241.18) - /unlink 14765 (time 1746148765.62 total 63.30 last 235.27) - /unlink 16771 (time 1746148775.63 total 73.31 last 200.50) - /unlink 19017 (time 1746148785.63 total 83.31 last 224.57) - /unlink 20000 (time 1746148790.07 total 87.76 last 221.10) - /unlink 22107 (time 1746148800.08 total 97.76 last 210.64) - /unlink 23864 (time 1746148810.08 total 107.76 last 175.63) - /unlink 25781 (time 1746148820.08 total 117.76 last 191.69) - /unlink 27920 (time 1746148830.08 total 127.76 last 213.88) - /unlink 29997 (time 1746148840.08 total 137.76 last 207.69) - /unlink 32080 (time 1746148850.08 total 147.77 last 208.27) - /unlink 34508 (time 1746148860.09 total 157.77 last 242.75) - /unlink 36919 (time 1746148870.09 total 167.77 last 241.09) - /unlink 38808 (time 1746148880.09 total 177.78 last 188.76) - /unlink 40000 (time 1746148886.78 total 184.46 last 178.35) - /unlink 41839 (time 1746148896.78 total 194.47 last 183.76) - /unlink 44175 (time 1746148906.79 total 204.47 last 233.59) - /unlink 46232 (time 1746148916.79 total 214.47 last 205.61) - /unlink 48566 (time 1746148926.79 total 224.47 last 233.36) - /unlink 50000 (time 1746148933.06 total 230.75 last 228.60) - /unlink 52654 (time 1746148943.07 total 240.75 last 265.38) - /unlink 55312 (time 1746148953.07 total 250.75 last 265.75) - /unlink 58096 (time 1746148963.07 total 260.75 last 278.38) - /unlink 60000 (time 1746148969.59 total 267.27 last 292.11) - /unlink 62289 (time 1746148979.59 total 277.27 last 228.85) - /unlink 64669 (time 1746148989.59 total 287.27 last 237.99) total: 64768 /unlink in 287.66 seconds: 225.15 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server unloading modules via unload_modules_local on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2