== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:42:49 (1772469769) Stopping clients: oleg305-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg305-client.virtnet /mnt/lustre2 (opts:-f) oleg305-server: oleg305-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing load_modules_local oleg305-server: Loading modules from /home/green/git/lustre-release/lustre oleg305-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg305-server: detected 4 online CPUs by sysfs oleg305-server: MODOPTS_LIBCFS= oleg305-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg305-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg305-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 937 (time 1772469903.02 total 10.00 last 93.67) - open/close 1794 (time 1772469913.03 total 20.01 last 85.64) - open/close 2555 (time 1772469923.03 total 30.02 last 76.04) - open/close 3532 (time 1772469933.04 total 40.03 last 97.61) - open/close 4533 (time 1772469943.05 total 50.03 last 100.03) - open/close 5290 (time 1772469953.08 total 60.06 last 75.49) - open/close 5950 (time 1772469963.08 total 70.07 last 65.98) - open/close 6602 (time 1772469973.09 total 80.07 last 65.15) - open/close 7408 (time 1772469983.10 total 90.09 last 80.49) - open/close 8248 (time 1772469993.11 total 100.09 last 83.98) - open/close 8952 (time 1772470003.11 total 110.09 last 70.37) - open/close 9607 (time 1772470013.11 total 120.10 last 65.48) - open/close 10000 (time 1772470018.18 total 125.16 last 77.57) - open/close 10881 (time 1772470028.18 total 135.17 last 88.05) - open/close 11742 (time 1772470038.19 total 145.18 last 86.02) - open/close 12654 (time 1772470048.20 total 155.18 last 91.16) - open/close 13425 (time 1772470058.20 total 165.18 last 77.09) - open/close 14180 (time 1772470068.20 total 175.19 last 75.47) - open/close 14920 (time 1772470078.21 total 185.19 last 73.97) - open/close 15514 (time 1772470088.21 total 195.20 last 59.37) - open/close 16191 (time 1772470098.23 total 205.21 last 67.61) - open/close 16961 (time 1772470108.23 total 215.22 last 76.95) - open/close 17562 (time 1772470118.25 total 225.24 last 59.99) - open/close 18199 (time 1772470128.28 total 235.27 last 63.50) - open/close 18837 (time 1772470138.28 total 245.27 last 63.79) - open/close 19591 (time 1772470148.29 total 255.27 last 75.35) - open/close 20000 (time 1772470152.91 total 259.89 last 88.56) - open/close 20787 (time 1772470162.91 total 269.90 last 78.65) - open/close 21600 (time 1772470172.93 total 279.91 last 81.20) - open/close 22488 (time 1772470182.93 total 289.91 last 88.79) - open/close 23323 (time 1772470192.93 total 299.91 last 83.47) - open/close 24202 (time 1772470202.94 total 309.92 last 87.84) - open/close 25075 (time 1772470212.94 total 319.92 last 87.27) - open/close 25931 (time 1772470222.94 total 329.93 last 85.59) - open/close 26671 (time 1772470232.95 total 339.94 last 73.93) - open/close 27501 (time 1772470242.96 total 349.94 last 82.95) - open/close 28271 (time 1772470252.97 total 359.95 last 76.94) - open/close 29079 (time 1772470262.97 total 369.95 last 80.77) - open/close 29854 (time 1772470272.97 total 379.96 last 77.47) - open/close 30636 (time 1772470282.98 total 389.96 last 78.15) - open/close 31428 (time 1772470292.98 total 399.96 last 79.20) - open/close 32214 (time 1772470303.00 total 409.98 last 78.46) - open/close 32978 (time 1772470313.01 total 419.99 last 76.31) - open/close 33812 (time 1772470323.01 total 430.00 last 83.37) - open/close 34621 (time 1772470333.02 total 440.00 last 80.88) - open/close 35514 (time 1772470343.02 total 450.01 last 89.25) - open/close 36379 (time 1772470353.02 total 460.01 last 86.48) - open/close 37184 (time 1772470363.03 total 470.01 last 80.46) - open/close 38087 (time 1772470373.04 total 480.02 last 90.23) - open/close 38930 (time 1772470383.04 total 490.02 last 84.29) - open/close 39686 (time 1772470393.04 total 500.03 last 75.57) - open/close 40000 (time 1772470396.94 total 503.93 last 80.54) - open/close 40821 (time 1772470406.95 total 513.93 last 82.02) - open/close 41684 (time 1772470416.96 total 523.94 last 86.25) - open/close 42537 (time 1772470426.96 total 533.94 last 85.30) - open/close 43429 (time 1772470436.96 total 543.95 last 89.15) - open/close 44113 (time 1772470446.96 total 553.95 last 68.38) - open/close 44852 (time 1772470456.97 total 563.95 last 73.88) - open/close 45430 (time 1772470466.99 total 573.97 last 57.68) - open/close 46188 (time 1772470476.99 total 583.98 last 75.78) - open/close 47161 (time 1772470487.00 total 593.98 last 97.22) - open/close 48131 (time 1772470497.01 total 603.99 last 96.90) - open/close 48710 (time 1772470507.01 total 614.00 last 57.88) - open/close 49505 (time 1772470517.06 total 624.04 last 79.13) - open/close 50000 (time 1772470523.42 total 630.40 last 77.84) - open/close 50792 (time 1772470533.43 total 640.41 last 79.15) - open/close 51647 (time 1772470543.43 total 650.41 last 85.46) - open/close 52495 (time 1772470553.45 total 660.43 last 84.65) - open/close 53423 (time 1772470563.45 total 670.44 last 92.75) - open/close 54145 (time 1772470573.46 total 680.44 last 72.16) - open/close 54767 (time 1772470583.47 total 690.45 last 62.14) - open/close 55582 (time 1772470593.47 total 700.45 last 81.49) - open/close 56508 (time 1772470603.48 total 710.46 last 92.54) - open/close 57278 (time 1772470613.48 total 720.47 last 76.94) - open/close 57949 (time 1772470623.49 total 730.48 last 67.03) - open/close 58881 (time 1772470633.50 total 740.48 last 93.16) - open/close 59789 (time 1772470643.50 total 750.48 last 90.78) - open/close 60000 (time 1772470646.09 total 753.08 last 81.41) - open/close 60921 (time 1772470656.10 total 763.08 last 92.06) - open/close 61727 (time 1772470666.10 total 773.08 last 80.58) - open/close 62670 (time 1772470676.11 total 783.09 last 94.21) - open/close 63438 (time 1772470686.11 total 793.09 last 76.79) - open/close 64233 (time 1772470696.12 total 803.10 last 79.46) total: 64768 open/close in 809.50 seconds: 80.01 ops/second - /unlink 828 (time 1772470717.63 total 10.00 last 82.76) - /unlink 1598 (time 1772470727.68 total 20.06 last 76.61) - /unlink 2404 (time 1772470737.80 total 30.17 last 79.69) - /unlink 3233 (time 1772470747.80 total 40.17 last 82.88) - /unlink 4156 (time 1772470757.80 total 50.17 last 92.28) - /unlink 5058 (time 1772470767.81 total 60.18 last 90.13) - /unlink 5971 (time 1772470777.81 total 70.18 last 91.29) - /unlink 6781 (time 1772470787.81 total 80.19 last 80.96) - /unlink 7636 (time 1772470797.82 total 90.19 last 85.46) - /unlink 8436 (time 1772470807.83 total 100.21 last 79.91) - /unlink 9198 (time 1772470817.83 total 110.21 last 76.18) - /unlink 9951 (time 1772470827.84 total 120.21 last 75.25) - /unlink 10920 (time 1772470837.85 total 130.22 last 96.82) - /unlink 11831 (time 1772470847.85 total 140.23 last 91.07) - /unlink 12659 (time 1772470857.86 total 150.23 last 82.76) - /unlink 13449 (time 1772470867.86 total 160.23 last 78.97) - /unlink 14244 (time 1772470877.86 total 170.24 last 79.48) - /unlink 15101 (time 1772470887.86 total 180.24 last 85.69) - /unlink 15916 (time 1772470897.87 total 190.24 last 81.45) - /unlink 16838 (time 1772470907.88 total 200.25 last 92.11) - /unlink 17792 (time 1772470917.88 total 210.26 last 95.38) - /unlink 18691 (time 1772470927.90 total 220.27 last 89.76) - /unlink 19434 (time 1772470937.90 total 230.27 last 74.28) - /unlink 20000 (time 1772470945.40 total 237.77 last 75.48) - /unlink 20706 (time 1772470955.41 total 247.79 last 70.50) - /unlink 21539 (time 1772470965.41 total 257.79 last 83.28) - /unlink 22449 (time 1772470975.42 total 267.79 last 90.96) - /unlink 23254 (time 1772470985.42 total 277.80 last 80.47) - /unlink 24157 (time 1772470995.43 total 287.80 last 90.25) - /unlink 25037 (time 1772471005.43 total 297.81 last 87.95) - /unlink 25867 (time 1772471015.44 total 307.81 last 82.94) - /unlink 26662 (time 1772471025.45 total 317.82 last 79.42) - /unlink 27555 (time 1772471035.46 total 327.83 last 89.25) - /unlink 28283 (time 1772471045.46 total 337.83 last 72.78) - /unlink 29177 (time 1772471055.46 total 347.84 last 89.35) - /unlink 30000 (time 1772471063.50 total 355.87 last 102.46) - /unlink 30905 (time 1772471073.50 total 365.88 last 90.42) - /unlink 31834 (time 1772471083.51 total 375.89 last 92.85) - /unlink 32644 (time 1772471093.51 total 385.89 last 80.97) - /unlink 33439 (time 1772471103.52 total 395.89 last 79.49) - /unlink 34180 (time 1772471113.52 total 405.89 last 74.09) - /unlink 35103 (time 1772471123.52 total 415.90 last 92.26) - /unlink 35899 (time 1772471133.53 total 425.90 last 79.56) - /unlink 36740 (time 1772471143.53 total 435.90 last 84.09) - /unlink 37494 (time 1772471153.53 total 445.91 last 75.35) - /unlink 37927 (time 1772471163.54 total 455.92 last 43.28) - /unlink 38778 (time 1772471173.55 total 465.93 last 85.01) - /unlink 39720 (time 1772471183.55 total 475.93 last 94.20) - /unlink 40000 (time 1772471186.15 total 478.52 last 107.87) - /unlink 40929 (time 1772471196.15 total 488.53 last 92.85) - /unlink 41805 (time 1772471206.15 total 498.53 last 87.58) - /unlink 42711 (time 1772471216.16 total 508.53 last 90.59) - /unlink 43590 (time 1772471226.16 total 518.54 last 87.84) - /unlink 44394 (time 1772471236.17 total 528.54 last 80.38) - /unlink 45196 (time 1772471246.17 total 538.54 last 80.17) - /unlink 45891 (time 1772471256.17 total 548.55 last 69.49) - /unlink 46553 (time 1772471266.18 total 558.55 last 66.15) - /unlink 47420 (time 1772471276.18 total 568.55 last 86.69) - /unlink 48253 (time 1772471286.18 total 578.56 last 83.27) - /unlink 49121 (time 1772471296.19 total 588.56 last 86.77) - /unlink 49885 (time 1772471306.19 total 598.57 last 76.34) - /unlink 50659 (time 1772471316.19 total 608.57 last 77.39) - /unlink 51584 (time 1772471326.21 total 618.59 last 92.34) - /unlink 52469 (time 1772471336.29 total 628.67 last 87.78) - /unlink 53401 (time 1772471346.30 total 638.67 last 93.17) - /unlink 54277 (time 1772471356.31 total 648.68 last 87.53) - /unlink 55071 (time 1772471366.31 total 658.68 last 79.36) - /unlink 55818 (time 1772471376.31 total 668.69 last 74.68) - /unlink 56736 (time 1772471386.32 total 678.69 last 91.77) - /unlink 57688 (time 1772471396.32 total 688.70 last 95.13) - /unlink 58545 (time 1772471406.33 total 698.71 last 85.64) - /unlink 59307 (time 1772471416.33 total 708.71 last 76.18) - /unlink 60000 (time 1772471424.26 total 716.63 last 87.45) - /unlink 60808 (time 1772471434.26 total 726.64 last 80.78) - /unlink 61601 (time 1772471444.26 total 736.64 last 79.27) - /unlink 62392 (time 1772471454.27 total 746.64 last 79.08) - /unlink 63196 (time 1772471464.28 total 756.65 last 80.32) - /unlink 64061 (time 1772471474.29 total 766.66 last 86.41) total: 64768 /unlink in 774.78 seconds: 83.60 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 930 (time 1772471521.68 total 10.01 last 92.88) - open/close 1643 (time 1772471531.68 total 20.02 last 71.25) - open/close 2579 (time 1772471541.71 total 30.05 last 93.34) - open/close 3431 (time 1772471551.71 total 40.05 last 85.18) - open/close 4071 (time 1772471561.74 total 50.08 last 63.81) - open/close 5087 (time 1772471571.74 total 60.08 last 101.60) - open/close 6140 (time 1772471581.75 total 70.08 last 105.27) - open/close 7020 (time 1772471591.77 total 80.11 last 87.79) - open/close 7838 (time 1772471601.79 total 90.13 last 81.62) - open/close 8793 (time 1772471611.80 total 100.13 last 95.46) - open/close 9815 (time 1772471621.81 total 110.14 last 102.10) - open/close 10868 (time 1772471631.81 total 120.14 last 105.28) - open/close 11900 (time 1772471641.82 total 130.15 last 103.11) - open/close 12707 (time 1772471651.83 total 140.16 last 80.61) - open/close 13621 (time 1772471661.83 total 150.17 last 91.35) - open/close 14420 (time 1772471671.85 total 160.18 last 79.80) - open/close 15150 (time 1772471681.85 total 170.18 last 73.00) - open/close 15943 (time 1772471691.85 total 180.19 last 79.26) - open/close 16843 (time 1772471701.86 total 190.19 last 89.96) - open/close 17702 (time 1772471711.86 total 200.19 last 85.87) - open/close 18474 (time 1772471721.86 total 210.20 last 77.18) - open/close 19378 (time 1772471731.87 total 220.20 last 90.34) - open/close 20000 (time 1772471740.93 total 229.27 last 68.64) - open/close 20485 (time 1772471750.93 total 239.27 last 48.49) - open/close 21095 (time 1772471760.94 total 249.27 last 60.98) - open/close 21583 (time 1772471770.94 total 259.28 last 48.77) - open/close 22257 (time 1772471780.95 total 269.28 last 67.35) - open/close 22925 (time 1772471790.96 total 279.30 last 66.71) - open/close 23818 (time 1772471800.97 total 289.30 last 89.25) - open/close 24817 (time 1772471810.97 total 299.30 last 99.88) - open/close 25708 (time 1772471820.98 total 309.32 last 88.98) - open/close 26678 (time 1772471830.98 total 319.32 last 96.98) - open/close 27476 (time 1772471840.99 total 329.33 last 79.75) - open/close 28326 (time 1772471851.00 total 339.33 last 84.95) - open/close 29144 (time 1772471861.00 total 349.34 last 81.76) - open/close 29891 (time 1772471871.00 total 359.34 last 74.70) - open/close 30488 (time 1772471881.00 total 369.34 last 59.69) - open/close 31167 (time 1772471891.02 total 379.36 last 67.77) - open/close 31916 (time 1772471901.02 total 389.36 last 74.90) - open/close 32694 (time 1772471911.04 total 399.37 last 77.68) - open/close 33377 (time 1772471921.04 total 409.38 last 68.28) - open/close 34236 (time 1772471931.05 total 419.38 last 85.82) - open/close 35104 (time 1772471941.05 total 429.39 last 86.78) - open/close 36040 (time 1772471951.06 total 439.39 last 93.56) - open/close 36966 (time 1772471961.06 total 449.39 last 92.56) - open/close 37991 (time 1772471971.07 total 459.40 last 102.44) - open/close 38797 (time 1772471981.07 total 469.40 last 80.59) - open/close 39723 (time 1772471991.07 total 479.41 last 92.53) - open/close 40000 (time 1772471994.53 total 482.87 last 80.13) - open/close 40697 (time 1772472004.54 total 492.87 last 69.66) - open/close 41511 (time 1772472014.54 total 502.87 last 81.39) - open/close 42437 (time 1772472024.55 total 512.88 last 92.51) - open/close 43121 (time 1772472034.56 total 522.89 last 68.33) - open/close 43957 (time 1772472044.57 total 532.90 last 83.52) - open/close 45036 (time 1772472054.57 total 542.90 last 107.89) - open/close 45966 (time 1772472064.57 total 552.91 last 92.95) - open/close 46973 (time 1772472074.58 total 562.92 last 100.60) - open/close 47891 (time 1772472084.59 total 572.92 last 91.78) - open/close 48715 (time 1772472094.59 total 582.92 last 82.38) - open/close 49488 (time 1772472104.59 total 592.93 last 77.26) - open/close 50000 (time 1772472111.67 total 600.00 last 72.41) - open/close 50846 (time 1772472121.67 total 610.00 last 84.59) - open/close 51729 (time 1772472131.67 total 620.00 last 88.28) - open/close 52568 (time 1772472141.68 total 630.01 last 83.82) - open/close 53539 (time 1772472151.68 total 640.01 last 97.10) - open/close 54257 (time 1772472161.69 total 650.02 last 71.73) - open/close 55155 (time 1772472171.69 total 660.03 last 89.75) - open/close 56033 (time 1772472181.71 total 670.04 last 87.70) - open/close 56880 (time 1772472191.72 total 680.06 last 84.57) - open/close 57659 (time 1772472201.73 total 690.06 last 77.86) - open/close 58558 (time 1772472211.74 total 700.07 last 89.79) - open/close 59336 (time 1772472221.77 total 710.10 last 77.57) - open/close 60000 (time 1772472231.30 total 719.63 last 69.69) - open/close 60696 (time 1772472241.32 total 729.65 last 69.46) - open/close 61525 (time 1772472251.32 total 739.65 last 82.89) - open/close 62362 (time 1772472261.33 total 749.66 last 83.62) - open/close 63197 (time 1772472271.33 total 759.67 last 83.45) - open/close 64020 (time 1772472281.35 total 769.69 last 82.14) total: 64768 open/close in 778.25 seconds: 83.22 ops/second - /unlink 915 (time 1772472304.21 total 10.02 last 91.29) - /unlink 1747 (time 1772472314.22 total 20.04 last 83.08) - /unlink 2567 (time 1772472324.23 total 30.05 last 81.91) - /unlink 3442 (time 1772472334.24 total 40.06 last 87.41) - /unlink 4312 (time 1772472344.25 total 50.07 last 86.94) - /unlink 4925 (time 1772472354.27 total 60.09 last 61.16) - /unlink 5663 (time 1772472364.29 total 70.10 last 73.71) - /unlink 6412 (time 1772472374.29 total 80.10 last 74.89) - /unlink 6795 (time 1772472384.33 total 90.15 last 38.12) - /unlink 7398 (time 1772472394.34 total 100.16 last 60.26) - /unlink 8146 (time 1772472404.35 total 110.17 last 74.71) - /unlink 8940 (time 1772472414.35 total 120.17 last 79.39) - /unlink 9847 (time 1772472424.37 total 130.19 last 90.52) - /unlink 10777 (time 1772472434.38 total 140.19 last 92.97) - /unlink 11685 (time 1772472444.39 total 150.20 last 90.73) - /unlink 12452 (time 1772472454.39 total 160.20 last 76.67) - /unlink 13281 (time 1772472464.40 total 170.21 last 82.85) - /unlink 14110 (time 1772472474.40 total 180.21 last 82.86) - /unlink 14901 (time 1772472484.40 total 190.22 last 79.07) - /unlink 15786 (time 1772472494.42 total 200.23 last 88.39) - /unlink 16479 (time 1772472504.42 total 210.23 last 69.28) - /unlink 17209 (time 1772472514.44 total 220.26 last 72.85) - /unlink 18004 (time 1772472524.45 total 230.26 last 79.44) - /unlink 18903 (time 1772472534.45 total 240.26 last 89.90) - /unlink 19835 (time 1772472544.46 total 250.27 last 93.14) - /unlink 20765 (time 1772472554.46 total 260.27 last 92.98) - /unlink 21671 (time 1772472564.46 total 270.28 last 90.56) - /unlink 22463 (time 1772472574.47 total 280.29 last 79.13) - /unlink 23246 (time 1772472584.48 total 290.29 last 78.24) - /unlink 23918 (time 1772472594.49 total 300.30 last 67.15) - /unlink 24689 (time 1772472604.49 total 310.31 last 77.03) - /unlink 25383 (time 1772472614.50 total 320.31 last 69.36) - /unlink 26119 (time 1772472624.50 total 330.32 last 73.59) - /unlink 26964 (time 1772472634.51 total 340.32 last 84.44) - /unlink 27936 (time 1772472644.53 total 350.35 last 96.97) - /unlink 28811 (time 1772472654.54 total 360.35 last 87.43) - /unlink 29536 (time 1772472664.55 total 370.36 last 72.44) - /unlink 30000 (time 1772472669.49 total 375.31 last 93.89) - /unlink 30893 (time 1772472679.50 total 385.31 last 89.25) - /unlink 31805 (time 1772472689.50 total 395.31 last 91.18) - /unlink 32725 (time 1772472699.51 total 405.32 last 91.95) - /unlink 33390 (time 1772472709.51 total 415.33 last 66.46) - /unlink 34162 (time 1772472719.52 total 425.33 last 77.17) - /unlink 34971 (time 1772472729.53 total 435.35 last 80.76) - /unlink 35771 (time 1772472739.55 total 445.37 last 79.87) - /unlink 36338 (time 1772472749.56 total 455.37 last 56.66) - /unlink 37223 (time 1772472759.57 total 465.39 last 88.37) - /unlink 38115 (time 1772472769.58 total 475.39 last 89.18) - /unlink 38978 (time 1772472779.58 total 485.40 last 86.25) - /unlink 39900 (time 1772472789.59 total 495.40 last 92.16) - /unlink 40659 (time 1772472799.60 total 505.41 last 75.80) - /unlink 41469 (time 1772472809.61 total 515.42 last 80.93) - /unlink 42138 (time 1772472819.62 total 525.43 last 66.84) - /unlink 42995 (time 1772472829.63 total 535.44 last 85.60) - /unlink 43834 (time 1772472839.64 total 545.46 last 83.78) - /unlink 44638 (time 1772472849.65 total 555.46 last 80.36) - /unlink 45233 (time 1772472859.68 total 565.49 last 59.31) - /unlink 45955 (time 1772472869.69 total 575.51 last 72.10) - /unlink 46857 (time 1772472879.70 total 585.51 last 90.19) - /unlink 47672 (time 1772472889.70 total 595.51 last 81.48) - /unlink 48508 (time 1772472899.70 total 605.52 last 83.54) - /unlink 49384 (time 1772472909.71 total 615.52 last 87.54) - /unlink 50000 (time 1772472916.51 total 622.33 last 90.58) - /unlink 50908 (time 1772472926.52 total 632.34 last 90.69) - /unlink 51832 (time 1772472936.54 total 642.35 last 92.27) - /unlink 52606 (time 1772472946.54 total 652.36 last 77.35) - /unlink 53368 (time 1772472956.55 total 662.36 last 76.15) - /unlink 54081 (time 1772472966.56 total 672.38 last 71.20) - /unlink 54737 (time 1772472976.57 total 682.39 last 65.55) - /unlink 55580 (time 1772472986.58 total 692.39 last 84.27) - /unlink 56356 (time 1772472996.58 total 702.40 last 77.55) - /unlink 57109 (time 1772473006.58 total 712.40 last 75.30) - /unlink 57910 (time 1772473016.59 total 722.41 last 80.01) - /unlink 58699 (time 1772473026.62 total 732.44 last 78.66) - /unlink 59477 (time 1772473036.65 total 742.47 last 77.56) - /unlink 60000 (time 1772473044.90 total 750.72 last 63.39) - /unlink 60557 (time 1772473054.91 total 760.73 last 55.67) - /unlink 61276 (time 1772473064.93 total 770.74 last 71.79) - /unlink 62125 (time 1772473074.93 total 780.75 last 84.84) - /unlink 62989 (time 1772473084.94 total 790.75 last 86.37) - /unlink 63812 (time 1772473094.97 total 800.78 last 82.07) - /unlink 64644 (time 1772473104.98 total 810.79 last 83.08) total: 64768 /unlink in 813.33 seconds: 79.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 772 (time 1772473148.19 total 10.00 last 77.18) - open/close 1469 (time 1772473158.20 total 20.01 last 69.67) - open/close 2135 (time 1772473168.21 total 30.02 last 66.50) - open/close 2921 (time 1772473178.21 total 40.02 last 78.59) - open/close 3817 (time 1772473188.21 total 50.02 last 89.58) - open/close 4685 (time 1772473198.23 total 60.03 last 86.71) - open/close 5276 (time 1772473208.24 total 70.05 last 59.04) - open/close 5882 (time 1772473218.29 total 80.10 last 60.29) - open/close 6405 (time 1772473228.30 total 90.11 last 52.22) - open/close 7037 (time 1772473238.31 total 100.12 last 63.17) - open/close 7567 (time 1772473248.32 total 110.13 last 52.96) - open/close 8569 (time 1772473258.32 total 120.13 last 100.19) - open/close 9526 (time 1772473268.32 total 130.13 last 95.67) - open/close 10000 (time 1772473273.28 total 135.09 last 95.52) - open/close 10804 (time 1772473283.29 total 145.10 last 80.35) - open/close 11440 (time 1772473293.30 total 155.11 last 63.54) - open/close 12075 (time 1772473303.30 total 165.11 last 63.49) - open/close 12945 (time 1772473313.30 total 175.11 last 86.95) - open/close 13733 (time 1772473323.35 total 185.16 last 78.41) - open/close 14407 (time 1772473333.36 total 195.17 last 67.33) - open/close 15270 (time 1772473343.37 total 205.18 last 86.28) - open/close 16032 (time 1772473353.38 total 215.18 last 76.13) - open/close 17023 (time 1772473363.38 total 225.19 last 99.05) - open/close 17863 (time 1772473373.38 total 235.19 last 83.98) - open/close 18810 (time 1772473383.39 total 245.20 last 94.60) - open/close 19758 (time 1772473393.40 total 255.21 last 94.78) - open/close 20000 (time 1772473397.54 total 259.35 last 58.45) - open/close 20826 (time 1772473407.55 total 269.36 last 82.51) - open/close 21742 (time 1772473417.55 total 279.36 last 91.55) - open/close 22650 (time 1772473427.56 total 289.36 last 90.78) - open/close 23520 (time 1772473437.56 total 299.37 last 86.95) - open/close 24441 (time 1772473447.57 total 309.38 last 92.05) - open/close 25431 (time 1772473457.57 total 319.38 last 98.97) - open/close 26351 (time 1772473467.57 total 329.38 last 91.95) - open/close 27286 (time 1772473477.58 total 339.39 last 93.43) - open/close 28197 (time 1772473487.59 total 349.40 last 91.05) - open/close 29150 (time 1772473497.59 total 359.40 last 95.24) - open/close 30000 (time 1772473505.50 total 367.31 last 107.56) - open/close 31025 (time 1772473515.50 total 377.31 last 102.44) - open/close 31941 (time 1772473525.51 total 387.32 last 91.53) - open/close 32905 (time 1772473535.51 total 397.32 last 96.34) - open/close 33873 (time 1772473545.52 total 407.33 last 96.77) - open/close 34783 (time 1772473555.52 total 417.33 last 90.94) - open/close 35626 (time 1772473565.53 total 427.34 last 84.23) - open/close 36454 (time 1772473575.53 total 437.34 last 82.79) - open/close 37427 (time 1772473585.54 total 447.35 last 97.23) - open/close 38339 (time 1772473595.55 total 457.36 last 91.10) - open/close 39280 (time 1772473605.56 total 467.37 last 94.00) - open/close 40000 (time 1772473612.66 total 474.47 last 101.42) - open/close 40977 (time 1772473622.67 total 484.48 last 97.65) - open/close 41958 (time 1772473632.67 total 494.48 last 98.07) - open/close 42960 (time 1772473642.67 total 504.48 last 100.15) - open/close 44163 (time 1772473652.68 total 514.49 last 120.21) - open/close 45240 (time 1772473662.69 total 524.50 last 107.65) - open/close 46186 (time 1772473672.69 total 534.50 last 94.57) - open/close 47183 (time 1772473682.69 total 544.50 last 99.70) - open/close 48256 (time 1772473692.69 total 554.50 last 107.27) - open/close 49373 (time 1772473702.70 total 564.51 last 111.65) - open/close 50000 (time 1772473708.23 total 570.03 last 113.45) - open/close 51107 (time 1772473718.23 total 580.04 last 110.62) - open/close 52154 (time 1772473728.24 total 590.05 last 104.66) - open/close 53179 (time 1772473738.24 total 600.05 last 102.49) - open/close 54101 (time 1772473748.25 total 610.06 last 92.10) - open/close 55009 (time 1772473758.25 total 620.06 last 90.79) - open/close 56054 (time 1772473768.26 total 630.06 last 104.44) - open/close 57088 (time 1772473778.26 total 640.07 last 103.37) - open/close 58122 (time 1772473788.26 total 650.07 last 103.37) - open/close 59482 (time 1772473798.27 total 660.08 last 135.87) - open/close 60000 (time 1772473804.87 total 666.68 last 78.49) - open/close 60838 (time 1772473814.87 total 676.68 last 83.77) - open/close 61804 (time 1772473824.88 total 686.68 last 96.60) - open/close 62743 (time 1772473834.89 total 696.70 last 93.79) - open/close 63715 (time 1772473844.89 total 706.70 last 97.19) - open/close 64654 (time 1772473854.89 total 716.70 last 93.85) total: 64768 open/close in 717.91 seconds: 90.22 ops/second - /unlink 952 (time 1772473869.11 total 10.00 last 95.19) - /unlink 2069 (time 1772473879.11 total 20.00 last 111.69) - /unlink 3226 (time 1772473889.13 total 30.02 last 115.54) - /unlink 4299 (time 1772473899.13 total 40.02 last 107.27) - /unlink 5144 (time 1772473909.13 total 50.02 last 84.47) - /unlink 6217 (time 1772473919.14 total 60.03 last 107.26) - /unlink 7284 (time 1772473929.15 total 70.03 last 106.63) - /unlink 8573 (time 1772473939.15 total 80.04 last 128.84) - /unlink 9526 (time 1772473949.15 total 90.04 last 95.26) - /unlink 10000 (time 1772473953.98 total 94.87 last 98.19) - /unlink 10937 (time 1772473963.99 total 104.87 last 93.65) - /unlink 11957 (time 1772473973.99 total 114.87 last 101.98) - /unlink 12897 (time 1772473984.01 total 124.90 last 93.78) - /unlink 13904 (time 1772473994.02 total 134.90 last 100.63) - /unlink 14959 (time 1772474004.02 total 144.91 last 105.49) - /unlink 15933 (time 1772474014.02 total 154.91 last 97.35) - /unlink 16960 (time 1772474024.03 total 164.92 last 102.66) - /unlink 17915 (time 1772474034.04 total 174.93 last 95.41) - /unlink 18936 (time 1772474044.05 total 184.94 last 101.99) - /unlink 19978 (time 1772474054.05 total 194.94 last 104.17) - /unlink 21043 (time 1772474064.06 total 204.95 last 106.44) - /unlink 22186 (time 1772474074.08 total 214.97 last 114.04) - /unlink 23469 (time 1772474084.08 total 224.97 last 128.27) - /unlink 24497 (time 1772474094.09 total 234.98 last 102.73) - /unlink 25597 (time 1772474104.09 total 244.98 last 109.95) - /unlink 26627 (time 1772474114.10 total 254.98 last 102.97) - /unlink 27655 (time 1772474124.11 total 264.99 last 102.70) - /unlink 28746 (time 1772474134.11 total 275.00 last 109.02) - /unlink 29902 (time 1772474144.12 total 285.00 last 115.57) - /unlink 31004 (time 1772474154.12 total 295.00 last 110.20) - /unlink 32155 (time 1772474164.13 total 305.02 last 114.94) - /unlink 33248 (time 1772474174.13 total 315.02 last 109.27) - /unlink 34475 (time 1772474184.14 total 325.02 last 122.64) - /unlink 35632 (time 1772474194.14 total 335.03 last 115.64) - /unlink 36715 (time 1772474204.15 total 345.03 last 108.27) - /unlink 37816 (time 1772474214.16 total 355.05 last 109.97) - /unlink 39068 (time 1772474224.16 total 365.05 last 125.18) - /unlink 40000 (time 1772474231.33 total 372.22 last 129.95) - /unlink 41198 (time 1772474241.34 total 382.23 last 119.66) - /unlink 42381 (time 1772474251.35 total 392.24 last 118.23) - /unlink 43481 (time 1772474261.35 total 402.24 last 109.96) - /unlink 44649 (time 1772474271.36 total 412.25 last 116.72) - /unlink 45850 (time 1772474281.37 total 422.26 last 119.92) - /unlink 47029 (time 1772474291.38 total 432.27 last 117.84) - /unlink 48254 (time 1772474301.38 total 442.27 last 122.50) - /unlink 49465 (time 1772474311.39 total 452.27 last 121.00) - /unlink 50000 (time 1772474316.14 total 457.03 last 112.60) - /unlink 51020 (time 1772474326.14 total 467.03 last 101.96) - /unlink 52126 (time 1772474336.15 total 477.03 last 110.57) - /unlink 53337 (time 1772474346.15 total 487.04 last 121.03) - /unlink 54605 (time 1772474356.15 total 497.04 last 126.76) - /unlink 56552 (time 1772474366.16 total 507.04 last 194.67) - /unlink 58962 (time 1772474376.16 total 517.04 last 240.97) - /unlink 60000 (time 1772474381.30 total 522.18 last 201.93) - /unlink 61999 (time 1772474391.30 total 532.19 last 199.78) - /unlink 64095 (time 1772474401.30 total 542.19 last 209.59) total: 64768 /unlink in 545.96 seconds: 118.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1283 (time 1772474436.99 total 10.00 last 128.26) - open/close 2492 (time 1772474446.99 total 20.00 last 120.89) - open/close 3675 (time 1772474456.99 total 30.01 last 118.27) - open/close 4962 (time 1772474467.01 total 40.02 last 128.49) - open/close 6297 (time 1772474477.01 total 50.03 last 133.42) - open/close 7580 (time 1772474487.01 total 60.03 last 128.28) - open/close 8979 (time 1772474497.01 total 70.03 last 139.90) - open/close 10000 (time 1772474503.93 total 76.95 last 147.57) - open/close 11385 (time 1772474513.93 total 86.95 last 138.48) - open/close 12649 (time 1772474523.93 total 96.95 last 126.39) - open/close 14070 (time 1772474533.94 total 106.95 last 142.06) - open/close 15338 (time 1772474543.94 total 116.96 last 126.75) - open/close 16629 (time 1772474553.95 total 126.96 last 129.03) - open/close 18083 (time 1772474563.95 total 136.97 last 145.36) - open/close 19533 (time 1772474573.95 total 146.97 last 144.99) - open/close 20000 (time 1772474577.06 total 150.08 last 150.02) - open/close 21457 (time 1772474587.06 total 160.08 last 145.69) - open/close 23115 (time 1772474597.07 total 170.08 last 165.77) - open/close 24576 (time 1772474607.07 total 180.09 last 146.02) - open/close 25846 (time 1772474617.07 total 190.09 last 127.00) - open/close 27370 (time 1772474627.07 total 200.09 last 152.38) - open/close 28914 (time 1772474637.08 total 210.10 last 154.30) - open/close 30000 (time 1772474643.02 total 216.03 last 182.91) - open/close 31850 (time 1772474653.02 total 226.04 last 184.93) - open/close 33494 (time 1772474663.02 total 236.04 last 164.39) - open/close 34925 (time 1772474673.03 total 246.04 last 143.03) - open/close 36214 (time 1772474683.03 total 256.05 last 128.86) - open/close 37613 (time 1772474693.03 total 266.05 last 139.88) - open/close 38997 (time 1772474703.03 total 276.05 last 138.36) - open/close 40000 (time 1772474709.63 total 282.65 last 152.08) - open/close 41439 (time 1772474719.64 total 292.65 last 143.81) - open/close 42985 (time 1772474729.64 total 302.66 last 154.46) - open/close 44542 (time 1772474739.65 total 312.67 last 155.62) - open/close 46216 (time 1772474749.66 total 322.67 last 167.27) - open/close 47568 (time 1772474759.66 total 332.67 last 135.20) - open/close 48980 (time 1772474769.66 total 342.68 last 141.17) - open/close 50000 (time 1772474777.56 total 350.58 last 129.03) - open/close 51333 (time 1772474787.57 total 360.58 last 133.28) - open/close 52820 (time 1772474797.57 total 370.59 last 148.59) - open/close 54270 (time 1772474807.58 total 380.60 last 144.90) - open/close 55574 (time 1772474817.58 total 390.60 last 130.37) - open/close 56903 (time 1772474827.60 total 400.61 last 132.71) - open/close 58295 (time 1772474837.60 total 410.62 last 139.12) - open/close 59667 (time 1772474847.61 total 420.63 last 137.10) - open/close 60000 (time 1772474850.28 total 423.30 last 124.63) - open/close 61441 (time 1772474860.28 total 433.30 last 144.07) - open/close 62970 (time 1772474870.29 total 443.30 last 152.87) - open/close 64371 (time 1772474880.29 total 453.31 last 140.05) total: 64768 open/close in 456.18 seconds: 141.98 ops/second - /unlink 2717 (time 1772474895.04 total 10.00 last 271.60) - /unlink 5267 (time 1772474905.04 total 20.00 last 254.98) - /unlink 8102 (time 1772474915.04 total 30.01 last 283.47) - /unlink 10000 (time 1772474921.90 total 36.86 last 276.71) - /unlink 13229 (time 1772474931.90 total 46.87 last 322.82) - /unlink 16150 (time 1772474941.90 total 56.87 last 292.10) - /unlink 19269 (time 1772474951.90 total 66.87 last 311.87) - /unlink 22774 (time 1772474961.90 total 76.87 last 350.49) - /unlink 25681 (time 1772474971.90 total 86.87 last 290.66) - /unlink 28562 (time 1772474981.91 total 96.87 last 288.03) - /unlink 30000 (time 1772474987.70 total 102.67 last 248.18) - /unlink 32550 (time 1772474997.71 total 112.67 last 254.83) - /unlink 34963 (time 1772475007.71 total 122.68 last 241.18) - /unlink 37527 (time 1772475017.71 total 132.68 last 256.37) - /unlink 40000 (time 1772475025.81 total 140.78 last 305.30) - /unlink 43331 (time 1772475035.82 total 150.78 last 333.04) - /unlink 46708 (time 1772475045.82 total 160.78 last 337.65) - /unlink 49873 (time 1772475055.82 total 170.79 last 316.38) - /unlink 54040 (time 1772475065.82 total 180.79 last 416.67) - /unlink 57584 (time 1772475075.82 total 190.79 last 354.38) - /unlink 60000 (time 1772475082.45 total 197.41 last 364.85) - /unlink 63552 (time 1772475092.45 total 207.41 last 355.18) total: 64768 /unlink in 211.46 seconds: 306.29 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2267 (time 1772475126.78 total 10.01 last 226.55) - open/close 4556 (time 1772475136.78 total 20.01 last 228.90) - open/close 6231 (time 1772475146.78 total 30.01 last 167.38) - open/close 8004 (time 1772475156.79 total 40.02 last 177.26) - open/close 10000 (time 1772475166.20 total 49.42 last 212.14) - open/close 12482 (time 1772475176.20 total 59.43 last 248.19) - open/close 15337 (time 1772475186.20 total 69.43 last 285.42) - open/close 17981 (time 1772475196.20 total 79.43 last 264.32) - open/close 20000 (time 1772475205.35 total 88.58 last 220.71) - open/close 21729 (time 1772475215.35 total 98.58 last 172.88) - open/close 23587 (time 1772475225.37 total 108.59 last 185.52) - open/close 25329 (time 1772475235.37 total 118.60 last 174.15) - open/close 27228 (time 1772475245.37 total 128.60 last 189.86) - open/close 29545 (time 1772475255.37 total 138.60 last 231.61) - open/close 30000 (time 1772475257.51 total 140.74 last 212.89) - open/close 32037 (time 1772475267.52 total 150.75 last 203.58) - open/close 33880 (time 1772475277.52 total 160.75 last 184.18) - open/close 35640 (time 1772475287.52 total 170.75 last 175.99) - open/close 37451 (time 1772475297.53 total 180.76 last 180.99) - open/close 39048 (time 1772475307.54 total 190.77 last 159.52) - open/close 40000 (time 1772475313.28 total 196.51 last 165.84) - open/close 41875 (time 1772475323.28 total 206.51 last 187.48) - open/close 43969 (time 1772475333.29 total 216.52 last 209.23) - open/close 46090 (time 1772475343.29 total 226.52 last 212.06) - open/close 48076 (time 1772475353.30 total 236.53 last 198.53) - open/close 50000 (time 1772475362.38 total 245.61 last 211.76) - open/close 52063 (time 1772475372.38 total 255.61 last 206.28) - open/close 54032 (time 1772475382.39 total 265.62 last 196.74) - open/close 56212 (time 1772475392.40 total 275.62 last 217.91) - open/close 58884 (time 1772475402.40 total 285.63 last 267.17) - open/close 60000 (time 1772475406.01 total 289.24 last 308.75) - open/close 62344 (time 1772475416.01 total 299.24 last 234.36) - open/close 64477 (time 1772475426.01 total 309.24 last 213.25) total: 64768 open/close in 310.13 seconds: 208.84 ops/second - /unlink 4642 (time 1772475438.43 total 10.00 last 463.98) - /unlink 9375 (time 1772475448.43 total 20.01 last 473.28) - /unlink 10000 (time 1772475450.47 total 22.05 last 305.90) - /unlink 13289 (time 1772475460.47 total 32.05 last 328.81) - /unlink 16526 (time 1772475470.48 total 42.05 last 323.58) - /unlink 20000 (time 1772475479.17 total 50.74 last 399.75) - /unlink 23976 (time 1772475489.17 total 60.75 last 397.44) - /unlink 28707 (time 1772475499.17 total 70.75 last 473.10) - /unlink 30000 (time 1772475501.82 total 73.40 last 488.60) - /unlink 35104 (time 1772475511.82 total 83.40 last 510.35) - /unlink 39154 (time 1772475521.82 total 93.40 last 404.96) - /unlink 43847 (time 1772475531.82 total 103.40 last 469.27) - /unlink 49510 (time 1772475541.82 total 113.40 last 566.29) - /unlink 55245 (time 1772475551.82 total 123.40 last 573.48) - /unlink 60000 (time 1772475561.59 total 133.17 last 486.85) total: 64768 /unlink in 141.56 seconds: 457.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg305-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg305-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server stop mds service on oleg305-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg305-server unloading modules via unload_modules_local on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing unload_modules_local oleg305-server: modules unloaded. pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2