== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:31:33 (1778535093) Stopping clients: oleg158-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg158-client.virtnet /mnt/lustre2 (opts:-f) oleg158-server: oleg158-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing load_modules_local oleg158-server: Loading modules from /home/green/git/lustre-release/lustre oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg158-server: detected 4 online CPUs by sysfs oleg158-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg158-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg158-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg158-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg158-client.virtnet: -o user_xattr,flock oleg158-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16232 424 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 699 (time 1778535238.28 total 10.01 last 69.84) - open/close 1462 (time 1778535248.28 total 20.01 last 76.27) - open/close 2306 (time 1778535258.30 total 30.03 last 84.29) - open/close 3079 (time 1778535268.31 total 40.03 last 77.24) - open/close 3888 (time 1778535278.31 total 50.03 last 80.89) - open/close 4666 (time 1778535288.31 total 60.04 last 77.74) - open/close 5466 (time 1778535298.33 total 70.06 last 79.85) - open/close 6185 (time 1778535308.35 total 80.07 last 71.81) - open/close 6835 (time 1778535318.35 total 90.08 last 64.99) - open/close 7431 (time 1778535328.35 total 100.08 last 59.58) - open/close 8095 (time 1778535338.35 total 110.08 last 66.40) - open/close 8787 (time 1778535348.36 total 120.09 last 69.14) - open/close 9540 (time 1778535358.38 total 130.10 last 75.18) - open/close 10000 (time 1778535364.11 total 135.83 last 80.28) - open/close 10841 (time 1778535374.11 total 145.84 last 84.05) - open/close 11657 (time 1778535384.15 total 155.88 last 81.31) - open/close 12509 (time 1778535394.15 total 165.88 last 85.18) - open/close 13341 (time 1778535404.15 total 175.88 last 83.18) - open/close 14168 (time 1778535414.16 total 185.89 last 82.63) - open/close 15077 (time 1778535424.16 total 195.89 last 90.87) - open/close 16045 (time 1778535434.17 total 205.89 last 96.78) - open/close 16860 (time 1778535444.17 total 215.90 last 81.48) - open/close 17768 (time 1778535454.17 total 225.90 last 90.77) - open/close 18686 (time 1778535464.17 total 235.90 last 91.79) - open/close 19613 (time 1778535474.18 total 245.91 last 92.64) - open/close 20000 (time 1778535477.88 total 249.61 last 104.56) - open/close 20998 (time 1778535487.90 total 259.62 last 99.64) - open/close 22000 (time 1778535497.90 total 269.62 last 100.19) - open/close 22998 (time 1778535507.90 total 279.63 last 99.76) - open/close 23932 (time 1778535517.91 total 289.64 last 93.33) - open/close 24839 (time 1778535527.92 total 299.65 last 90.62) - open/close 25699 (time 1778535537.92 total 309.65 last 85.95) - open/close 26573 (time 1778535547.93 total 319.66 last 87.33) - open/close 27441 (time 1778535557.93 total 329.66 last 86.80) - open/close 28323 (time 1778535567.93 total 339.66 last 88.17) - open/close 29313 (time 1778535577.93 total 349.66 last 99.00) - open/close 30000 (time 1778535585.85 total 357.58 last 86.76) - open/close 31008 (time 1778535595.85 total 367.58 last 100.79) - open/close 31988 (time 1778535605.87 total 377.60 last 97.81) - open/close 32958 (time 1778535615.88 total 387.60 last 96.98) - open/close 33890 (time 1778535625.88 total 397.60 last 93.20) - open/close 34821 (time 1778535635.90 total 407.63 last 92.90) - open/close 35771 (time 1778535645.90 total 417.63 last 95.00) - open/close 36674 (time 1778535655.90 total 427.63 last 90.25) - open/close 37570 (time 1778535665.91 total 437.63 last 89.56) - open/close 38407 (time 1778535675.91 total 447.64 last 83.69) - open/close 39308 (time 1778535685.91 total 457.64 last 90.05) - open/close 40000 (time 1778535694.01 total 465.74 last 85.44) - open/close 40863 (time 1778535704.02 total 475.75 last 86.26) - open/close 41852 (time 1778535714.04 total 485.77 last 98.63) - open/close 42795 (time 1778535724.05 total 495.78 last 94.27) - open/close 43660 (time 1778535734.05 total 505.78 last 86.48) - open/close 44555 (time 1778535744.06 total 515.79 last 89.42) - open/close 45421 (time 1778535754.09 total 525.81 last 86.36) - open/close 46371 (time 1778535764.09 total 535.82 last 94.95) - open/close 47408 (time 1778535774.09 total 545.82 last 103.69) - open/close 48357 (time 1778535784.10 total 555.83 last 94.81) - open/close 49401 (time 1778535794.11 total 565.84 last 104.35) - open/close 50000 (time 1778535800.28 total 572.01 last 97.00) - open/close 50849 (time 1778535810.28 total 582.01 last 84.90) - open/close 51788 (time 1778535820.29 total 592.02 last 93.80) - open/close 52958 (time 1778535830.30 total 602.02 last 116.98) - open/close 53980 (time 1778535840.30 total 612.03 last 102.16) - open/close 55037 (time 1778535850.30 total 622.03 last 105.64) - open/close 56009 (time 1778535860.34 total 632.07 last 96.84) - open/close 56922 (time 1778535870.34 total 642.07 last 91.28) - open/close 57860 (time 1778535880.35 total 652.07 last 93.79) - open/close 58795 (time 1778535890.35 total 662.08 last 93.45) - open/close 59889 (time 1778535900.35 total 672.08 last 109.39) - open/close 60990 (time 1778535910.35 total 682.08 last 110.09) - open/close 62000 (time 1778535920.36 total 692.09 last 100.93) - open/close 63073 (time 1778535930.36 total 702.09 last 107.28) - open/close 64123 (time 1778535940.37 total 712.10 last 104.89) total: 64768 open/close in 718.09 seconds: 90.19 ops/second - /unlink 2101 (time 1778535958.89 total 10.00 last 210.02) - /unlink 4311 (time 1778535968.89 total 20.00 last 220.98) - /unlink 6352 (time 1778535978.89 total 30.01 last 204.09) - /unlink 8574 (time 1778535988.89 total 40.01 last 222.20) - /unlink 10000 (time 1778535994.89 total 46.01 last 237.66) - /unlink 11986 (time 1778536004.90 total 56.01 last 198.56) - /unlink 13851 (time 1778536014.90 total 66.01 last 186.45) - /unlink 16058 (time 1778536024.90 total 76.01 last 220.66) - /unlink 18275 (time 1778536034.90 total 86.02 last 221.59) - /unlink 20000 (time 1778536043.32 total 94.44 last 204.87) - /unlink 21962 (time 1778536053.33 total 104.44 last 196.13) - /unlink 23917 (time 1778536063.33 total 114.45 last 195.41) - /unlink 25885 (time 1778536073.34 total 124.45 last 196.72) - /unlink 28025 (time 1778536083.34 total 134.45 last 213.92) - /unlink 29876 (time 1778536093.34 total 144.46 last 185.04) - /unlink 31694 (time 1778536103.34 total 154.46 last 181.79) - /unlink 33906 (time 1778536113.35 total 164.46 last 221.14) - /unlink 35932 (time 1778536123.35 total 174.46 last 202.56) - /unlink 38051 (time 1778536133.35 total 184.46 last 211.88) - /unlink 40000 (time 1778536142.92 total 194.03 last 203.76) - /unlink 42102 (time 1778536152.92 total 204.03 last 210.18) - /unlink 44385 (time 1778536162.92 total 214.03 last 228.30) - /unlink 46655 (time 1778536172.92 total 224.03 last 226.94) - /unlink 48760 (time 1778536182.92 total 234.03 last 210.45) - /unlink 50000 (time 1778536188.64 total 239.75 last 216.93) - /unlink 52161 (time 1778536198.64 total 249.76 last 216.00) - /unlink 54094 (time 1778536208.65 total 259.76 last 193.21) - /unlink 56125 (time 1778536218.66 total 269.77 last 202.94) - /unlink 58380 (time 1778536228.66 total 279.77 last 225.44) - /unlink 60000 (time 1778536236.57 total 287.68 last 204.81) - /unlink 62437 (time 1778536246.57 total 297.68 last 243.68) - /unlink 64643 (time 1778536256.57 total 307.68 last 220.59) total: 64768 /unlink in 308.21 seconds: 210.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1243 (time 1778536294.36 total 10.00 last 124.28) - open/close 2386 (time 1778536304.36 total 20.00 last 114.27) - open/close 3701 (time 1778536314.36 total 30.00 last 131.50) - open/close 4979 (time 1778536324.37 total 40.01 last 127.71) - open/close 6305 (time 1778536334.38 total 50.02 last 132.53) - open/close 7514 (time 1778536344.38 total 60.02 last 120.81) - open/close 8199 (time 1778536354.38 total 70.02 last 68.50) - open/close 8773 (time 1778536364.39 total 80.03 last 57.36) - open/close 9414 (time 1778536374.40 total 90.05 last 64.01) - open/close 9977 (time 1778536384.54 total 100.18 last 55.55) - open/close 10606 (time 1778536394.54 total 110.18 last 62.89) - open/close 11261 (time 1778536404.54 total 120.18 last 65.49) - open/close 12087 (time 1778536414.56 total 130.20 last 82.48) - open/close 12969 (time 1778536424.56 total 140.20 last 88.19) - open/close 13819 (time 1778536434.57 total 150.21 last 84.91) - open/close 14780 (time 1778536444.57 total 160.21 last 96.08) - open/close 15756 (time 1778536454.57 total 170.22 last 97.58) - open/close 16673 (time 1778536464.58 total 180.22 last 91.65) - open/close 17568 (time 1778536474.58 total 190.22 last 89.47) - open/close 18470 (time 1778536484.59 total 200.23 last 90.16) - open/close 19398 (time 1778536494.59 total 210.24 last 92.73) - open/close 20000 (time 1778536501.58 total 217.22 last 86.19) - open/close 20895 (time 1778536511.59 total 227.23 last 89.41) - open/close 21818 (time 1778536521.59 total 237.23 last 92.28) - open/close 22675 (time 1778536531.60 total 247.24 last 85.65) - open/close 23591 (time 1778536541.60 total 257.25 last 91.53) - open/close 24491 (time 1778536551.60 total 267.25 last 90.00) - open/close 25330 (time 1778536561.62 total 277.26 last 83.79) - open/close 26236 (time 1778536571.63 total 287.27 last 90.51) - open/close 27103 (time 1778536581.64 total 297.28 last 86.61) - open/close 27982 (time 1778536591.64 total 307.28 last 87.87) - open/close 28803 (time 1778536601.64 total 317.29 last 82.08) - open/close 29676 (time 1778536611.64 total 327.29 last 87.30) - open/close 30000 (time 1778536616.17 total 331.81 last 71.61) - open/close 30868 (time 1778536626.18 total 341.82 last 86.70) - open/close 31693 (time 1778536636.19 total 351.84 last 82.39) - open/close 32497 (time 1778536646.20 total 361.84 last 80.35) - open/close 33388 (time 1778536656.20 total 371.85 last 89.06) - open/close 34249 (time 1778536666.21 total 381.85 last 86.10) - open/close 35129 (time 1778536676.21 total 391.85 last 87.94) - open/close 36041 (time 1778536686.22 total 401.86 last 91.15) - open/close 36928 (time 1778536696.23 total 411.87 last 88.58) - open/close 37869 (time 1778536706.24 total 421.88 last 94.02) - open/close 38739 (time 1778536716.25 total 431.89 last 86.92) - open/close 39605 (time 1778536726.25 total 441.89 last 86.57) - open/close 40000 (time 1778536730.45 total 446.09 last 94.13) - open/close 40831 (time 1778536740.45 total 456.09 last 83.09) - open/close 41658 (time 1778536750.46 total 466.10 last 82.66) - open/close 42525 (time 1778536760.46 total 476.10 last 86.65) - open/close 43400 (time 1778536770.47 total 486.11 last 87.38) - open/close 44199 (time 1778536780.48 total 496.12 last 79.87) - open/close 45032 (time 1778536790.49 total 506.13 last 83.22) - open/close 45902 (time 1778536800.50 total 516.14 last 86.91) - open/close 46606 (time 1778536810.51 total 526.15 last 70.32) - open/close 47454 (time 1778536820.52 total 536.16 last 84.74) - open/close 48385 (time 1778536830.52 total 546.16 last 93.07) - open/close 49362 (time 1778536840.52 total 556.16 last 97.68) - open/close 50000 (time 1778536846.94 total 562.58 last 99.41) - open/close 50971 (time 1778536856.94 total 572.58 last 97.06) - open/close 51845 (time 1778536866.96 total 582.60 last 87.29) - open/close 52833 (time 1778536876.98 total 592.62 last 98.60) - open/close 53762 (time 1778536886.98 total 602.62 last 92.86) - open/close 54668 (time 1778536896.99 total 612.63 last 90.50) - open/close 55565 (time 1778536906.99 total 622.63 last 89.69) - open/close 56453 (time 1778536917.00 total 632.64 last 88.75) - open/close 57307 (time 1778536927.00 total 642.64 last 85.37) - open/close 58220 (time 1778536937.01 total 652.65 last 91.23) - open/close 59011 (time 1778536947.01 total 662.65 last 79.09) - open/close 59996 (time 1778536957.02 total 672.66 last 98.41) - open/close 60901 (time 1778536967.02 total 682.66 last 90.49) - open/close 61742 (time 1778536977.03 total 692.67 last 84.03) - open/close 62657 (time 1778536987.03 total 702.67 last 91.45) - open/close 63515 (time 1778536997.04 total 712.68 last 85.75) - open/close 64315 (time 1778537007.05 total 722.69 last 79.91) total: 64768 open/close in 727.72 seconds: 89.00 ops/second - /unlink 1496 (time 1778537025.22 total 10.00 last 149.53) - /unlink 2955 (time 1778537035.23 total 20.01 last 145.84) - /unlink 4675 (time 1778537045.23 total 30.01 last 171.99) - /unlink 6511 (time 1778537055.23 total 40.01 last 183.59) - /unlink 8080 (time 1778537065.23 total 50.01 last 156.85) - /unlink 9819 (time 1778537075.24 total 60.02 last 173.81) - /unlink 11569 (time 1778537085.24 total 70.02 last 174.97) - /unlink 13142 (time 1778537095.24 total 80.02 last 157.30) - /unlink 14806 (time 1778537105.24 total 90.02 last 166.34) - /unlink 16348 (time 1778537115.25 total 100.03 last 154.11) - /unlink 18020 (time 1778537125.26 total 110.05 last 166.95) - /unlink 19573 (time 1778537135.26 total 120.05 last 155.30) - /unlink 20000 (time 1778537138.08 total 122.86 last 151.59) - /unlink 21638 (time 1778537148.09 total 132.87 last 163.71) - /unlink 23205 (time 1778537158.09 total 142.87 last 156.70) - /unlink 24603 (time 1778537168.09 total 152.87 last 139.76) - /unlink 26107 (time 1778537178.10 total 162.88 last 150.30) - /unlink 27680 (time 1778537188.10 total 172.88 last 157.30) - /unlink 29290 (time 1778537198.10 total 182.88 last 160.98) - /unlink 30000 (time 1778537203.20 total 187.98 last 139.20) - /unlink 31752 (time 1778537213.20 total 197.98 last 175.20) - /unlink 33484 (time 1778537223.20 total 207.98 last 173.20) - /unlink 35223 (time 1778537233.20 total 217.98 last 173.86) - /unlink 37014 (time 1778537243.23 total 228.01 last 178.57) - /unlink 39007 (time 1778537253.23 total 238.02 last 199.22) - /unlink 40000 (time 1778537258.82 total 243.60 last 177.71) - /unlink 41876 (time 1778537268.83 total 253.61 last 187.53) - /unlink 43742 (time 1778537278.83 total 263.61 last 186.58) - /unlink 45582 (time 1778537288.83 total 273.61 last 183.94) - /unlink 47291 (time 1778537298.83 total 283.61 last 170.89) - /unlink 49186 (time 1778537308.83 total 293.62 last 189.46) - /unlink 50000 (time 1778537313.80 total 298.58 last 163.87) - /unlink 51803 (time 1778537323.81 total 308.59 last 180.22) - /unlink 53753 (time 1778537333.81 total 318.59 last 194.87) - /unlink 56042 (time 1778537343.81 total 328.59 last 228.89) - /unlink 58138 (time 1778537353.82 total 338.60 last 209.53) - /unlink 60000 (time 1778537363.02 total 347.80 last 202.30) - /unlink 61958 (time 1778537373.02 total 357.81 last 195.73) - /unlink 64032 (time 1778537383.03 total 367.81 last 207.33) total: 64768 /unlink in 371.72 seconds: 174.24 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1158 (time 1778537415.74 total 10.01 last 115.68) - open/close 2274 (time 1778537425.75 total 20.02 last 111.55) - open/close 3370 (time 1778537435.75 total 30.02 last 109.59) - open/close 4588 (time 1778537445.75 total 40.02 last 121.77) - open/close 5719 (time 1778537455.75 total 50.02 last 113.08) - open/close 6845 (time 1778537465.75 total 60.02 last 112.57) - open/close 7918 (time 1778537475.76 total 70.03 last 107.23) - open/close 8989 (time 1778537485.76 total 80.03 last 107.06) - open/close 9997 (time 1778537495.78 total 90.05 last 100.67) - open/close 11077 (time 1778537505.78 total 100.05 last 107.94) - open/close 12085 (time 1778537515.79 total 110.05 last 100.78) - open/close 13221 (time 1778537525.79 total 120.06 last 113.56) - open/close 14401 (time 1778537535.79 total 130.06 last 117.98) - open/close 15659 (time 1778537545.80 total 140.07 last 125.68) - open/close 16780 (time 1778537555.81 total 150.08 last 112.03) - open/close 17791 (time 1778537565.81 total 160.08 last 101.08) - open/close 18753 (time 1778537575.81 total 170.08 last 96.16) - open/close 19837 (time 1778537585.82 total 180.09 last 108.29) - open/close 20897 (time 1778537595.82 total 190.09 last 105.99) - open/close 21928 (time 1778537605.83 total 200.10 last 102.99) - open/close 23023 (time 1778537615.83 total 210.10 last 109.50) - open/close 24100 (time 1778537625.84 total 220.11 last 107.64) - open/close 25177 (time 1778537635.84 total 230.11 last 107.67) - open/close 26309 (time 1778537645.84 total 240.11 last 113.20) - open/close 27536 (time 1778537655.86 total 250.13 last 122.52) - open/close 28632 (time 1778537665.86 total 260.13 last 109.57) - open/close 29730 (time 1778537675.88 total 270.15 last 109.60) - open/close 30000 (time 1778537678.57 total 272.84 last 100.43) - open/close 31078 (time 1778537688.58 total 282.84 last 107.72) - open/close 32142 (time 1778537698.58 total 292.85 last 106.38) - open/close 33256 (time 1778537708.60 total 302.87 last 111.16) - open/close 34233 (time 1778537718.60 total 312.87 last 97.70) - open/close 35319 (time 1778537728.60 total 322.87 last 108.59) - open/close 36519 (time 1778537738.61 total 332.88 last 119.91) - open/close 37665 (time 1778537748.62 total 342.89 last 114.44) - open/close 38740 (time 1778537758.62 total 352.89 last 107.47) - open/close 39893 (time 1778537768.63 total 362.90 last 115.18) - open/close 41045 (time 1778537778.64 total 372.90 last 115.19) - open/close 42228 (time 1778537788.64 total 382.91 last 118.23) - open/close 43293 (time 1778537798.65 total 392.91 last 106.46) - open/close 44511 (time 1778537808.65 total 402.92 last 121.69) - open/close 45678 (time 1778537818.67 total 412.93 last 116.57) - open/close 46789 (time 1778537828.67 total 422.94 last 111.08) - open/close 47890 (time 1778537838.68 total 432.94 last 110.01) - open/close 49046 (time 1778537848.68 total 442.95 last 115.57) - open/close 50000 (time 1778537856.99 total 451.26 last 114.72) - open/close 51047 (time 1778537867.00 total 461.27 last 104.66) - open/close 52187 (time 1778537877.00 total 471.27 last 113.95) - open/close 53384 (time 1778537887.00 total 481.27 last 119.67) - open/close 54446 (time 1778537897.01 total 491.28 last 106.17) - open/close 55574 (time 1778537907.01 total 501.28 last 112.74) - open/close 56640 (time 1778537917.03 total 511.30 last 106.41) - open/close 57757 (time 1778537927.03 total 521.30 last 111.69) - open/close 58887 (time 1778537937.03 total 531.30 last 112.98) - open/close 59944 (time 1778537947.04 total 541.31 last 105.61) - open/close 61054 (time 1778537957.05 total 551.32 last 110.93) - open/close 62227 (time 1778537967.05 total 561.32 last 117.28) - open/close 63312 (time 1778537977.05 total 571.32 last 108.45) - open/close 64470 (time 1778537987.06 total 581.32 last 115.79) total: 64768 open/close in 584.07 seconds: 110.89 ops/second - /unlink 2278 (time 1778538002.11 total 10.00 last 227.73) - /unlink 4369 (time 1778538012.12 total 20.01 last 209.04) - /unlink 6514 (time 1778538022.12 total 30.01 last 214.46) - /unlink 8646 (time 1778538032.13 total 40.02 last 212.99) - /unlink 10000 (time 1778538038.52 total 46.41 last 211.98) - /unlink 12002 (time 1778538048.52 total 56.41 last 200.10) - /unlink 14046 (time 1778538058.53 total 66.41 last 204.31) - /unlink 16508 (time 1778538068.53 total 76.41 last 246.19) - /unlink 18958 (time 1778538078.53 total 86.42 last 244.96) - /unlink 20000 (time 1778538083.03 total 90.92 last 231.44) - /unlink 22166 (time 1778538093.03 total 100.92 last 216.56) - /unlink 24768 (time 1778538103.03 total 110.92 last 260.14) - /unlink 27254 (time 1778538113.04 total 120.93 last 248.52) - /unlink 29882 (time 1778538123.04 total 130.93 last 262.73) - /unlink 32310 (time 1778538133.04 total 140.93 last 242.78) - /unlink 34775 (time 1778538143.04 total 150.93 last 246.45) - /unlink 37455 (time 1778538153.04 total 160.93 last 267.96) - /unlink 40000 (time 1778538163.03 total 170.92 last 254.90) - /unlink 42694 (time 1778538173.03 total 180.92 last 269.33) - /unlink 45253 (time 1778538183.03 total 190.92 last 255.80) - /unlink 47848 (time 1778538193.03 total 200.92 last 259.49) - /unlink 50000 (time 1778538200.68 total 208.57 last 281.35) - /unlink 52657 (time 1778538210.69 total 218.58 last 265.63) - /unlink 55608 (time 1778538220.69 total 228.58 last 295.04) - /unlink 58305 (time 1778538230.69 total 238.58 last 269.66) - /unlink 60000 (time 1778538236.94 total 244.83 last 271.18) - /unlink 62886 (time 1778538246.94 total 254.83 last 288.53) total: 64768 /unlink in 261.02 seconds: 248.13 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1713 (time 1778538285.43 total 10.00 last 171.23) - open/close 3474 (time 1778538295.43 total 20.00 last 176.10) - open/close 5282 (time 1778538305.43 total 30.00 last 180.79) - open/close 7193 (time 1778538315.43 total 40.01 last 191.03) - open/close 9162 (time 1778538325.44 total 50.02 last 196.67) - open/close 10000 (time 1778538329.88 total 54.45 last 189.00) - open/close 11877 (time 1778538339.88 total 64.46 last 187.63) - open/close 14671 (time 1778538349.88 total 74.46 last 279.35) - open/close 17293 (time 1778538359.89 total 84.46 last 262.10) - open/close 19565 (time 1778538369.89 total 94.47 last 227.13) - open/close 21530 (time 1778538379.89 total 104.47 last 196.41) - open/close 23939 (time 1778538389.90 total 114.47 last 240.83) - open/close 26543 (time 1778538399.90 total 124.48 last 260.34) - open/close 29083 (time 1778538409.90 total 134.48 last 253.97) - open/close 30000 (time 1778538413.27 total 137.85 last 272.13) - open/close 32302 (time 1778538423.27 total 147.85 last 230.13) - open/close 35172 (time 1778538433.27 total 157.85 last 286.95) - open/close 37824 (time 1778538443.27 total 167.85 last 265.19) - open/close 40000 (time 1778538452.85 total 177.42 last 227.34) - open/close 42228 (time 1778538462.85 total 187.43 last 222.74) - open/close 44197 (time 1778538472.85 total 197.43 last 196.85) - open/close 46371 (time 1778538482.85 total 207.43 last 217.38) - open/close 48622 (time 1778538492.85 total 217.43 last 225.05) - open/close 50000 (time 1778538499.96 total 224.54 last 193.96) - open/close 51873 (time 1778538509.96 total 234.54 last 187.27) - open/close 53857 (time 1778538519.96 total 244.54 last 198.39) - open/close 55788 (time 1778538529.97 total 254.55 last 192.90) - open/close 57654 (time 1778538539.97 total 264.55 last 186.56) - open/close 59996 (time 1778538549.98 total 274.55 last 234.11) - open/close 62092 (time 1778538559.98 total 284.56 last 209.54) - open/close 63931 (time 1778538569.98 total 294.56 last 183.86) total: 64768 open/close in 299.23 seconds: 216.45 ops/second - /unlink 3501 (time 1778538586.30 total 10.00 last 349.98) - /unlink 6919 (time 1778538596.30 total 20.00 last 341.80) - /unlink 10000 (time 1778538604.29 total 28.00 last 385.23) - /unlink 14283 (time 1778538614.30 total 38.00 last 428.21) - /unlink 18648 (time 1778538624.30 total 48.00 last 436.50) - /unlink 20000 (time 1778538627.87 total 51.58 last 378.18) - /unlink 23664 (time 1778538637.87 total 61.58 last 366.38) - /unlink 27439 (time 1778538647.87 total 71.58 last 377.45) - /unlink 30000 (time 1778538654.77 total 78.47 last 371.49) - /unlink 33153 (time 1778538664.77 total 88.48 last 315.25) - /unlink 36410 (time 1778538674.77 total 98.48 last 325.61) - /unlink 40000 (time 1778538684.43 total 108.14 last 371.51) - /unlink 44056 (time 1778538694.43 total 118.14 last 405.58) - /unlink 49008 (time 1778538704.44 total 128.14 last 495.13) - /unlink 50000 (time 1778538706.78 total 130.49 last 422.36) - /unlink 54169 (time 1778538716.78 total 140.49 last 416.88) - /unlink 59905 (time 1778538726.78 total 150.49 last 573.60) total: 64768 /unlink in 159.16 seconds: 406.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2703 (time 1778538771.42 total 10.00 last 270.20) - open/close 5256 (time 1778538781.42 total 20.01 last 255.24) - open/close 7803 (time 1778538791.43 total 30.01 last 254.60) - open/close 9911 (time 1778538801.43 total 40.01 last 210.76) - open/close 13055 (time 1778538811.43 total 50.01 last 314.38) - open/close 16157 (time 1778538821.43 total 60.01 last 310.19) - open/close 19046 (time 1778538831.43 total 70.01 last 288.85) - open/close 20000 (time 1778538834.18 total 72.76 last 346.96) - open/close 23521 (time 1778538844.18 total 82.77 last 352.05) - open/close 26547 (time 1778538854.18 total 92.77 last 302.55) - open/close 29524 (time 1778538864.19 total 102.77 last 297.60) - open/close 32903 (time 1778538874.19 total 112.77 last 337.88) - open/close 36228 (time 1778538884.19 total 122.77 last 332.46) - open/close 39515 (time 1778538894.19 total 132.77 last 328.63) - open/close 42723 (time 1778538904.19 total 142.78 last 320.72) - open/close 46083 (time 1778538914.20 total 152.78 last 335.92) - open/close 49521 (time 1778538924.20 total 162.78 last 343.76) - open/close 53279 (time 1778538934.20 total 172.78 last 375.68) - open/close 57103 (time 1778538944.20 total 182.78 last 382.36) - open/close 60000 (time 1778538951.80 total 190.38 last 381.16) - open/close 63873 (time 1778538961.80 total 200.39 last 387.20) total: 64768 open/close in 202.82 seconds: 319.34 ops/second - /unlink 6813 (time 1778538975.60 total 10.00 last 681.27) - /unlink 10000 (time 1778538980.03 total 14.44 last 718.39) - /unlink 15853 (time 1778538990.04 total 24.44 last 585.26) - /unlink 20000 (time 1778538996.26 total 30.67 last 665.84) - /unlink 25770 (time 1778539006.26 total 40.67 last 576.96) - /unlink 30000 (time 1778539012.84 total 47.25 last 642.97) - /unlink 36578 (time 1778539022.84 total 57.25 last 657.72) - /unlink 40000 (time 1778539028.35 total 62.75 last 621.90) - /unlink 46839 (time 1778539038.35 total 72.75 last 683.86) - /unlink 50000 (time 1778539042.92 total 77.32 last 691.89) - /unlink 57092 (time 1778539052.92 total 87.32 last 709.16) - /unlink 60000 (time 1778539057.32 total 91.72 last 660.45) total: 64768 /unlink in 98.02 seconds: 660.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg158-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing unload_modules_local oleg158-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg158-server: LNET ready to unload oleg158-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2