== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:27:00 (1765103220) Stopping clients: oleg103-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg103-client.virtnet /mnt/lustre2 (opts:-f) oleg103-server: oleg103-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing load_modules_local oleg103-server: Loading modules from /home/green/git/lustre-release/lustre oleg103-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg103-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg103-server: detected 4 online CPUs by sysfs oleg103-server: MODOPTS_LIBCFS= oleg103-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg103-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg103-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 655 (time 1765103331.07 total 10.02 last 65.40) - open/close 1489 (time 1765103341.08 total 20.02 last 83.36) - open/close 2216 (time 1765103351.08 total 30.02 last 72.70) - open/close 2998 (time 1765103361.08 total 40.03 last 78.16) - open/close 3784 (time 1765103371.09 total 50.03 last 78.56) - open/close 4569 (time 1765103381.09 total 60.04 last 78.45) - open/close 5329 (time 1765103391.10 total 70.04 last 75.97) - open/close 6049 (time 1765103401.10 total 80.04 last 71.97) - open/close 6813 (time 1765103411.11 total 90.05 last 76.37) - open/close 7575 (time 1765103421.11 total 100.05 last 76.16) - open/close 8304 (time 1765103431.12 total 110.06 last 72.86) - open/close 9029 (time 1765103441.14 total 120.08 last 72.36) - open/close 9742 (time 1765103451.14 total 130.08 last 71.30) - open/close 10000 (time 1765103454.76 total 133.70 last 71.24) - open/close 10698 (time 1765103464.78 total 143.73 last 69.62) - open/close 11434 (time 1765103474.79 total 153.73 last 73.53) - open/close 12170 (time 1765103484.79 total 163.74 last 73.59) - open/close 12955 (time 1765103494.81 total 173.75 last 78.39) - open/close 13743 (time 1765103504.81 total 183.75 last 78.79) - open/close 14485 (time 1765103514.82 total 193.76 last 74.15) - open/close 15270 (time 1765103524.82 total 203.76 last 78.46) - open/close 15923 (time 1765103534.83 total 213.77 last 65.28) - open/close 16606 (time 1765103544.83 total 223.77 last 68.25) - open/close 17397 (time 1765103554.84 total 233.78 last 79.03) - open/close 18160 (time 1765103564.86 total 243.80 last 76.17) - open/close 18906 (time 1765103574.86 total 253.80 last 74.59) - open/close 19620 (time 1765103584.86 total 263.81 last 71.37) - open/close 20000 (time 1765103590.55 total 269.49 last 66.82) - open/close 20546 (time 1765103600.56 total 279.50 last 54.54) - open/close 21236 (time 1765103610.59 total 289.53 last 68.82) - open/close 22007 (time 1765103620.59 total 299.54 last 77.04) - open/close 22756 (time 1765103630.60 total 309.55 last 74.83) - open/close 23481 (time 1765103640.61 total 319.55 last 72.45) - open/close 24210 (time 1765103650.62 total 329.56 last 72.87) - open/close 24939 (time 1765103660.63 total 339.57 last 72.83) - open/close 25718 (time 1765103670.63 total 349.57 last 77.89) - open/close 26335 (time 1765103680.63 total 359.58 last 61.66) - open/close 27080 (time 1765103690.64 total 369.59 last 74.42) - open/close 27807 (time 1765103700.66 total 379.61 last 72.56) - open/close 28585 (time 1765103710.66 total 389.61 last 77.79) - open/close 29339 (time 1765103720.67 total 399.62 last 75.33) - open/close 30000 (time 1765103729.76 total 408.70 last 72.76) - open/close 30807 (time 1765103739.77 total 418.71 last 80.64) - open/close 31553 (time 1765103749.77 total 428.72 last 74.55) - open/close 32318 (time 1765103759.77 total 438.72 last 76.50) - open/close 33067 (time 1765103769.78 total 448.72 last 74.84) - open/close 33855 (time 1765103779.79 total 458.74 last 78.70) - open/close 34628 (time 1765103789.80 total 468.74 last 77.24) - open/close 35431 (time 1765103799.81 total 478.76 last 80.19) - open/close 36213 (time 1765103809.83 total 488.77 last 78.10) - open/close 36985 (time 1765103819.83 total 498.78 last 77.15) - open/close 37725 (time 1765103829.84 total 508.78 last 73.95) - open/close 38511 (time 1765103839.85 total 518.79 last 78.54) - open/close 39248 (time 1765103849.85 total 528.79 last 73.70) - open/close 40000 (time 1765103859.47 total 538.41 last 78.21) - open/close 40788 (time 1765103869.47 total 548.41 last 78.78) - open/close 41605 (time 1765103879.48 total 558.43 last 81.56) - open/close 42416 (time 1765103889.49 total 568.43 last 81.06) - open/close 43214 (time 1765103899.49 total 578.44 last 79.77) - open/close 44033 (time 1765103909.50 total 588.44 last 81.86) - open/close 44930 (time 1765103919.50 total 598.44 last 89.69) - open/close 45739 (time 1765103929.50 total 608.44 last 80.90) - open/close 46516 (time 1765103939.50 total 618.45 last 77.67) - open/close 47265 (time 1765103949.51 total 628.45 last 74.88) - open/close 48076 (time 1765103959.51 total 638.45 last 81.08) - open/close 48841 (time 1765103969.52 total 648.46 last 76.44) - open/close 49596 (time 1765103979.52 total 658.46 last 75.48) - open/close 50000 (time 1765103983.93 total 662.87 last 91.65) - open/close 50923 (time 1765103993.93 total 672.88 last 92.26) - open/close 51791 (time 1765104003.94 total 682.88 last 86.78) - open/close 52601 (time 1765104013.94 total 692.88 last 80.99) - open/close 53388 (time 1765104023.95 total 702.89 last 78.59) - open/close 54207 (time 1765104033.95 total 712.90 last 81.89) - open/close 55044 (time 1765104043.96 total 722.90 last 83.64) - open/close 55865 (time 1765104053.96 total 732.91 last 82.07) - open/close 56753 (time 1765104063.97 total 742.92 last 88.71) - open/close 57604 (time 1765104073.98 total 752.93 last 85.01) - open/close 58478 (time 1765104084.00 total 762.94 last 87.30) - open/close 59353 (time 1765104094.00 total 772.94 last 87.45) - open/close 60000 (time 1765104101.09 total 780.03 last 91.32) - open/close 60942 (time 1765104111.10 total 790.04 last 94.10) - open/close 61788 (time 1765104121.10 total 800.04 last 84.58) - open/close 62658 (time 1765104131.10 total 810.04 last 86.98) - open/close 63479 (time 1765104141.11 total 820.05 last 82.05) - open/close 64382 (time 1765104151.11 total 830.05 last 90.30) total: 64768 open/close in 833.76 seconds: 77.68 ops/second - /unlink 956 (time 1765104167.19 total 10.01 last 95.50) - /unlink 1899 (time 1765104177.20 total 20.01 last 94.26) - /unlink 2869 (time 1765104187.20 total 30.02 last 96.96) - /unlink 3842 (time 1765104197.21 total 40.03 last 97.19) - /unlink 4768 (time 1765104207.22 total 50.04 last 92.54) - /unlink 5655 (time 1765104217.23 total 60.04 last 88.63) - /unlink 6546 (time 1765104227.23 total 70.05 last 89.08) - /unlink 7390 (time 1765104237.23 total 80.05 last 84.38) - /unlink 8392 (time 1765104247.23 total 90.05 last 100.19) - /unlink 9489 (time 1765104257.23 total 100.05 last 109.69) - /unlink 10000 (time 1765104261.39 total 104.20 last 123.05) - /unlink 10984 (time 1765104271.39 total 114.21 last 98.33) - /unlink 11978 (time 1765104281.40 total 124.22 last 99.31) - /unlink 13058 (time 1765104291.41 total 134.23 last 107.95) - /unlink 14111 (time 1765104301.42 total 144.24 last 105.17) - /unlink 15073 (time 1765104311.43 total 154.25 last 96.08) - /unlink 15986 (time 1765104321.44 total 164.25 last 91.26) - /unlink 16952 (time 1765104331.45 total 174.26 last 96.50) - /unlink 18058 (time 1765104341.46 total 184.27 last 110.49) - /unlink 19257 (time 1765104351.46 total 194.28 last 119.86) - /unlink 20000 (time 1765104357.64 total 200.46 last 120.14) - /unlink 21210 (time 1765104367.65 total 210.47 last 120.97) - /unlink 22504 (time 1765104377.65 total 220.47 last 129.39) - /unlink 23795 (time 1765104387.66 total 230.47 last 128.99) - /unlink 24957 (time 1765104397.66 total 240.48 last 116.17) - /unlink 26246 (time 1765104407.66 total 250.48 last 128.87) - /unlink 27445 (time 1765104417.67 total 260.49 last 119.83) - /unlink 28441 (time 1765104427.68 total 270.50 last 99.51) - /unlink 29552 (time 1765104437.68 total 280.50 last 111.09) - /unlink 30000 (time 1765104441.36 total 284.18 last 121.63) - /unlink 31180 (time 1765104451.36 total 294.18 last 117.97) - /unlink 32125 (time 1765104461.36 total 304.18 last 94.50) - /unlink 33244 (time 1765104471.37 total 314.19 last 111.84) - /unlink 34304 (time 1765104481.37 total 324.19 last 105.97) - /unlink 35270 (time 1765104491.37 total 334.19 last 96.58) - /unlink 36303 (time 1765104501.38 total 344.20 last 103.26) - /unlink 37393 (time 1765104511.38 total 354.20 last 108.96) - /unlink 38418 (time 1765104521.39 total 364.21 last 102.40) - /unlink 39395 (time 1765104531.40 total 374.21 last 97.66) - /unlink 40000 (time 1765104537.18 total 379.99 last 104.67) - /unlink 40988 (time 1765104547.19 total 390.01 last 98.67) - /unlink 41854 (time 1765104557.21 total 400.03 last 86.41) - /unlink 42725 (time 1765104567.22 total 410.04 last 87.00) - /unlink 43743 (time 1765104577.23 total 420.04 last 101.76) - /unlink 44836 (time 1765104587.24 total 430.06 last 109.15) - /unlink 45866 (time 1765104597.24 total 440.06 last 102.98) - /unlink 46830 (time 1765104607.24 total 450.06 last 96.38) - /unlink 47808 (time 1765104617.25 total 460.06 last 97.78) - /unlink 48879 (time 1765104627.25 total 470.07 last 107.08) - /unlink 50000 (time 1765104636.36 total 479.18 last 122.99) - /unlink 51024 (time 1765104646.37 total 489.19 last 102.32) - /unlink 52223 (time 1765104656.38 total 499.20 last 119.79) - /unlink 53298 (time 1765104666.38 total 509.20 last 107.46) - /unlink 54354 (time 1765104676.39 total 519.21 last 105.52) - /unlink 55626 (time 1765104686.40 total 529.21 last 127.14) - /unlink 56823 (time 1765104696.40 total 539.22 last 119.65) - /unlink 57939 (time 1765104706.41 total 549.23 last 111.49) - /unlink 58917 (time 1765104716.41 total 559.23 last 97.79) - /unlink 59808 (time 1765104726.41 total 569.23 last 89.09) - /unlink 61004 (time 1765104736.41 total 579.23 last 119.58) - /unlink 62116 (time 1765104746.42 total 589.24 last 111.10) - /unlink 63457 (time 1765104756.42 total 599.24 last 134.09) - /unlink 64614 (time 1765104766.42 total 609.24 last 115.70) total: 64768 /unlink in 610.78 seconds: 106.04 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1132 (time 1765104800.33 total 10.00 last 113.19) - open/close 2318 (time 1765104810.34 total 20.01 last 118.52) - open/close 3478 (time 1765104820.35 total 30.02 last 115.91) - open/close 4515 (time 1765104830.35 total 40.02 last 103.67) - open/close 5498 (time 1765104840.36 total 50.02 last 98.25) - open/close 6587 (time 1765104850.36 total 60.02 last 108.90) - open/close 7735 (time 1765104860.36 total 70.03 last 114.73) - open/close 8792 (time 1765104870.36 total 80.03 last 105.69) - open/close 9893 (time 1765104880.37 total 90.03 last 110.08) - open/close 10925 (time 1765104890.37 total 100.04 last 103.17) - open/close 11989 (time 1765104900.37 total 110.04 last 106.38) - open/close 13046 (time 1765104910.38 total 120.05 last 105.59) - open/close 14167 (time 1765104920.39 total 130.06 last 111.97) - open/close 15258 (time 1765104930.39 total 140.06 last 109.09) - open/close 16342 (time 1765104940.39 total 150.06 last 108.38) - open/close 17643 (time 1765104950.40 total 160.06 last 130.08) - open/close 18661 (time 1765104960.40 total 170.07 last 101.72) - open/close 19744 (time 1765104970.42 total 180.09 last 108.13) - open/close 20000 (time 1765104972.54 total 182.20 last 120.95) - open/close 21087 (time 1765104982.54 total 192.21 last 108.64) - open/close 22264 (time 1765104992.55 total 202.22 last 117.63) - open/close 23442 (time 1765105002.55 total 212.22 last 117.76) - open/close 24454 (time 1765105012.55 total 222.22 last 101.20) - open/close 25463 (time 1765105022.55 total 232.22 last 100.88) - open/close 26660 (time 1765105032.56 total 242.23 last 119.64) - open/close 28014 (time 1765105042.56 total 252.23 last 135.35) - open/close 29334 (time 1765105052.56 total 262.23 last 131.98) - open/close 30000 (time 1765105057.21 total 266.88 last 143.36) - open/close 31868 (time 1765105067.21 total 276.88 last 186.72) - open/close 33443 (time 1765105077.22 total 286.88 last 157.47) - open/close 34962 (time 1765105087.22 total 296.89 last 151.84) - open/close 36169 (time 1765105097.22 total 306.89 last 120.65) - open/close 37496 (time 1765105107.22 total 316.89 last 132.68) - open/close 38677 (time 1765105117.24 total 326.90 last 117.97) - open/close 39825 (time 1765105127.24 total 336.91 last 114.73) - open/close 41060 (time 1765105137.25 total 346.91 last 123.44) - open/close 42166 (time 1765105147.25 total 356.92 last 110.59) - open/close 43374 (time 1765105157.25 total 366.92 last 120.76) - open/close 44544 (time 1765105167.26 total 376.92 last 116.95) - open/close 45820 (time 1765105177.26 total 386.93 last 127.55) - open/close 46899 (time 1765105187.26 total 396.93 last 107.85) - open/close 48125 (time 1765105197.26 total 406.93 last 122.59) - open/close 49364 (time 1765105207.28 total 416.94 last 123.75) - open/close 50000 (time 1765105213.07 total 422.73 last 109.83) - open/close 51216 (time 1765105223.08 total 432.74 last 121.49) - open/close 52414 (time 1765105233.09 total 442.75 last 119.67) - open/close 53675 (time 1765105243.09 total 452.76 last 126.07) - open/close 54999 (time 1765105253.09 total 462.76 last 132.40) - open/close 56230 (time 1765105263.09 total 472.76 last 123.10) - open/close 57524 (time 1765105273.09 total 482.76 last 129.37) - open/close 58953 (time 1765105283.09 total 492.76 last 142.87) - open/close 60000 (time 1765105290.08 total 499.75 last 149.81) - open/close 61443 (time 1765105300.09 total 509.75 last 144.24) - open/close 62877 (time 1765105310.09 total 519.76 last 143.35) - open/close 64466 (time 1765105320.09 total 529.76 last 158.90) total: 64768 open/close in 531.94 seconds: 121.76 ops/second - /unlink 2387 (time 1765105334.47 total 10.00 last 238.67) - /unlink 4756 (time 1765105344.48 total 20.00 last 236.82) - /unlink 7323 (time 1765105354.48 total 30.01 last 256.61) - /unlink 9606 (time 1765105364.48 total 40.01 last 228.27) - /unlink 11923 (time 1765105374.48 total 50.01 last 231.64) - /unlink 14476 (time 1765105384.49 total 60.02 last 255.20) - /unlink 17202 (time 1765105394.49 total 70.02 last 272.53) - /unlink 20000 (time 1765105403.01 total 78.54 last 328.21) - /unlink 22753 (time 1765105413.02 total 88.55 last 275.24) - /unlink 25800 (time 1765105423.02 total 98.55 last 304.69) - /unlink 29011 (time 1765105433.02 total 108.55 last 320.99) - /unlink 30000 (time 1765105435.82 total 111.35 last 353.60) - /unlink 33946 (time 1765105445.82 total 121.35 last 394.57) - /unlink 37093 (time 1765105455.82 total 131.35 last 314.63) - /unlink 39665 (time 1765105465.82 total 141.35 last 257.15) - /unlink 42273 (time 1765105475.82 total 151.35 last 260.78) - /unlink 44489 (time 1765105485.82 total 161.35 last 221.58) - /unlink 47062 (time 1765105495.83 total 171.35 last 257.25) - /unlink 49880 (time 1765105505.83 total 181.36 last 281.69) - /unlink 52756 (time 1765105515.83 total 191.36 last 287.59) - /unlink 55595 (time 1765105525.83 total 201.36 last 283.81) - /unlink 58068 (time 1765105535.84 total 211.36 last 247.23) - /unlink 60000 (time 1765105542.25 total 217.78 last 301.14) - /unlink 63086 (time 1765105552.25 total 227.78 last 308.52) total: 64768 /unlink in 232.77 seconds: 278.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2000 (time 1765105587.11 total 10.00 last 199.97) - open/close 3761 (time 1765105597.12 total 20.00 last 176.08) - open/close 5737 (time 1765105607.12 total 30.01 last 197.54) - open/close 7626 (time 1765105617.12 total 40.01 last 188.86) - open/close 9311 (time 1765105627.12 total 50.01 last 168.47) - open/close 10000 (time 1765105630.93 total 53.82 last 180.84) - open/close 11996 (time 1765105640.93 total 63.82 last 199.60) - open/close 13986 (time 1765105650.94 total 73.82 last 198.91) - open/close 15908 (time 1765105660.94 total 83.83 last 192.15) - open/close 18264 (time 1765105670.94 total 93.83 last 235.58) - open/close 20000 (time 1765105678.69 total 101.58 last 223.96) - open/close 21957 (time 1765105688.70 total 111.58 last 195.63) - open/close 24029 (time 1765105698.70 total 121.59 last 207.13) - open/close 26021 (time 1765105708.72 total 131.60 last 198.87) - open/close 27981 (time 1765105718.72 total 141.61 last 195.95) - open/close 29986 (time 1765105728.72 total 151.61 last 200.48) - open/close 31996 (time 1765105738.72 total 161.61 last 200.97) - open/close 33869 (time 1765105748.73 total 171.61 last 187.21) - open/close 35546 (time 1765105758.73 total 181.61 last 167.66) - open/close 37477 (time 1765105768.74 total 191.62 last 192.93) - open/close 39625 (time 1765105778.74 total 201.63 last 214.76) - open/close 42072 (time 1765105788.74 total 211.63 last 244.63) - open/close 44435 (time 1765105798.74 total 221.63 last 236.24) - open/close 46828 (time 1765105808.75 total 231.64 last 239.18) - open/close 49259 (time 1765105818.75 total 241.64 last 243.04) - open/close 50000 (time 1765105821.34 total 244.23 last 286.20) - open/close 52143 (time 1765105831.35 total 254.23 last 214.18) - open/close 53613 (time 1765105841.35 total 264.24 last 146.93) - open/close 55386 (time 1765105851.35 total 274.24 last 177.30) - open/close 57408 (time 1765105861.37 total 284.25 last 201.87) - open/close 58880 (time 1765105871.37 total 294.26 last 147.13) - open/close 60000 (time 1765105879.29 total 302.17 last 141.51) - open/close 61906 (time 1765105889.29 total 312.18 last 190.53) - open/close 63728 (time 1765105899.30 total 322.19 last 182.06) total: 64768 open/close in 329.34 seconds: 196.66 ops/second - /unlink 2794 (time 1765105918.46 total 10.00 last 279.28) - /unlink 5750 (time 1765105928.46 total 20.01 last 295.57) - /unlink 8883 (time 1765105938.47 total 30.01 last 313.23) - /unlink 10000 (time 1765105941.66 total 33.21 last 349.19) - /unlink 13686 (time 1765105951.66 total 43.21 last 368.57) - /unlink 18091 (time 1765105961.67 total 53.21 last 440.48) - /unlink 20000 (time 1765105966.45 total 57.99 last 398.86) - /unlink 23783 (time 1765105976.45 total 68.00 last 378.20) - /unlink 27088 (time 1765105986.46 total 78.00 last 330.44) - /unlink 30000 (time 1765105995.41 total 86.95 last 325.37) - /unlink 34179 (time 1765106005.41 total 96.95 last 417.89) - /unlink 38996 (time 1765106015.41 total 106.95 last 481.64) - /unlink 40000 (time 1765106017.86 total 109.40 last 409.80) - /unlink 44018 (time 1765106027.86 total 119.40 last 401.75) - /unlink 48751 (time 1765106037.86 total 129.40 last 473.25) - /unlink 50000 (time 1765106040.40 total 131.95 last 490.92) - /unlink 55147 (time 1765106050.40 total 141.95 last 514.64) - /unlink 60000 (time 1765106060.22 total 151.77 last 494.22) total: 64768 /unlink in 159.78 seconds: 405.35 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2630 (time 1765106096.97 total 10.00 last 262.99) - open/close 5026 (time 1765106106.97 total 20.00 last 239.58) - open/close 7107 (time 1765106116.97 total 30.01 last 208.01) - open/close 9867 (time 1765106126.97 total 40.01 last 275.92) - open/close 12801 (time 1765106136.98 total 50.01 last 293.35) - open/close 15656 (time 1765106146.98 total 60.02 last 285.34) - open/close 18353 (time 1765106156.98 total 70.02 last 269.64) - open/close 20000 (time 1765106163.56 total 76.59 last 250.45) - open/close 22308 (time 1765106173.56 total 86.59 last 230.78) - open/close 24555 (time 1765106183.56 total 96.60 last 224.68) - open/close 27647 (time 1765106193.57 total 106.60 last 309.06) - open/close 30000 (time 1765106201.46 total 114.49 last 298.10) - open/close 32455 (time 1765106211.46 total 124.50 last 245.40) - open/close 35605 (time 1765106221.46 total 134.50 last 314.98) - open/close 39368 (time 1765106231.47 total 144.50 last 376.27) - open/close 42739 (time 1765106241.47 total 154.50 last 337.02) - open/close 46375 (time 1765106251.47 total 164.50 last 363.53) - open/close 49935 (time 1765106261.47 total 174.51 last 355.92) - open/close 53121 (time 1765106271.47 total 184.51 last 318.55) - open/close 56689 (time 1765106281.48 total 194.51 last 356.74) - open/close 60000 (time 1765106289.12 total 202.15 last 433.07) - open/close 64084 (time 1765106299.12 total 212.16 last 408.33) total: 64768 open/close in 214.18 seconds: 302.40 ops/second - /unlink 5650 (time 1765106312.56 total 10.00 last 565.00) - /unlink 10000 (time 1765106318.76 total 16.20 last 701.06) - /unlink 15930 (time 1765106328.77 total 26.21 last 592.42) - /unlink 20000 (time 1765106335.68 total 33.12 last 589.11) - /unlink 25442 (time 1765106345.68 total 43.13 last 544.10) - /unlink 30000 (time 1765106352.99 total 50.43 last 623.87) - /unlink 36939 (time 1765106362.99 total 60.43 last 693.80) - /unlink 40000 (time 1765106367.10 total 64.55 last 744.00) - /unlink 46674 (time 1765106377.10 total 74.55 last 667.40) - /unlink 50000 (time 1765106381.83 total 79.28 last 703.13) - /unlink 56821 (time 1765106391.83 total 89.28 last 682.10) - /unlink 60000 (time 1765106396.49 total 93.94 last 682.40) total: 64768 /unlink in 102.00 seconds: 634.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4099 (time 1765106435.71 total 10.00 last 409.83) - open/close 8812 (time 1765106445.71 total 20.00 last 471.25) - open/close 10000 (time 1765106448.09 total 22.38 last 500.01) - open/close 14521 (time 1765106458.09 total 32.38 last 452.00) - open/close 19157 (time 1765106468.09 total 42.38 last 463.55) - open/close 23356 (time 1765106478.09 total 52.38 last 419.86) - open/close 27488 (time 1765106488.09 total 62.38 last 413.14) - open/close 30000 (time 1765106493.45 total 67.74 last 468.70) - open/close 35046 (time 1765106503.46 total 77.75 last 504.51) - open/close 39373 (time 1765106513.46 total 87.75 last 432.62) - open/close 43245 (time 1765106523.46 total 97.75 last 387.15) - open/close 47287 (time 1765106533.46 total 107.75 last 404.12) - open/close 50000 (time 1765106540.18 total 114.47 last 403.90) - open/close 53862 (time 1765106550.18 total 124.47 last 386.16) - open/close 57665 (time 1765106560.18 total 134.47 last 380.27) - open/close 60000 (time 1765106565.44 total 139.73 last 443.93) total: 64768 open/close in 149.43 seconds: 433.44 ops/second - /unlink 7866 (time 1765106586.43 total 10.00 last 786.54) - /unlink 10000 (time 1765106588.83 total 12.41 last 887.35) - /unlink 17562 (time 1765106598.83 total 22.41 last 756.13) - /unlink 20000 (time 1765106602.56 total 26.13 last 654.13) - /unlink 26831 (time 1765106612.56 total 36.13 last 683.07) - /unlink 30000 (time 1765106616.93 total 40.51 last 724.84) - /unlink 38382 (time 1765106626.93 total 50.51 last 838.10) - /unlink 45733 (time 1765106636.93 total 60.51 last 735.00) - /unlink 50000 (time 1765106642.62 total 66.20 last 749.83) - /unlink 58209 (time 1765106652.63 total 76.20 last 820.75) - /unlink 60000 (time 1765106654.95 total 78.53 last 769.66) total: 64768 /unlink in 84.25 seconds: 768.77 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg103-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg103-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg103-server stop mds service on oleg103-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg103-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing unload_modules_local oleg103-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg103-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg103-client: oleg103-client: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-server: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-client: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-server: ssh exited with exit code 2