== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:03:43 (1763510623) Stopping clients: oleg220-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg220-client.virtnet /mnt/lustre2 (opts:-f) oleg220-server: oleg220-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg220-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg220-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 757 (time 1763510718.51 total 10.01 last 75.65) - open/close 1407 (time 1763510728.53 total 20.03 last 64.86) - open/close 2096 (time 1763510738.54 total 30.04 last 68.85) - open/close 2791 (time 1763510748.54 total 40.04 last 69.49) - open/close 3445 (time 1763510758.55 total 50.05 last 65.35) - open/close 4145 (time 1763510768.55 total 60.05 last 70.00) - open/close 4811 (time 1763510778.55 total 70.05 last 66.60) - open/close 5512 (time 1763510788.55 total 80.05 last 70.07) - open/close 6245 (time 1763510798.55 total 90.05 last 73.28) - open/close 7015 (time 1763510808.56 total 100.06 last 76.98) - open/close 7763 (time 1763510818.56 total 110.06 last 74.78) - open/close 8479 (time 1763510828.57 total 120.07 last 71.53) - open/close 9174 (time 1763510838.57 total 130.07 last 69.48) - open/close 9819 (time 1763510848.57 total 140.07 last 64.49) - open/close 10000 (time 1763510851.09 total 142.58 last 72.10) - open/close 10719 (time 1763510861.10 total 152.60 last 71.80) - open/close 11399 (time 1763510871.12 total 162.62 last 67.88) - open/close 12103 (time 1763510881.12 total 172.62 last 70.37) - open/close 12869 (time 1763510891.13 total 182.63 last 76.56) - open/close 13619 (time 1763510901.14 total 192.64 last 74.92) - open/close 14368 (time 1763510911.15 total 202.65 last 74.76) - open/close 15061 (time 1763510921.16 total 212.66 last 69.27) - open/close 15695 (time 1763510931.17 total 222.67 last 63.34) - open/close 16325 (time 1763510941.17 total 232.67 last 62.96) - open/close 16954 (time 1763510951.19 total 242.69 last 62.80) - open/close 17619 (time 1763510961.20 total 252.70 last 66.42) - open/close 18320 (time 1763510971.22 total 262.72 last 69.99) - open/close 19055 (time 1763510981.22 total 272.72 last 73.48) - open/close 19760 (time 1763510991.23 total 282.73 last 70.43) - open/close 20000 (time 1763510995.16 total 286.65 last 61.15) - open/close 20680 (time 1763511005.17 total 296.67 last 67.88) - open/close 21341 (time 1763511015.18 total 306.68 last 66.08) - open/close 22037 (time 1763511025.19 total 316.69 last 69.47) - open/close 22699 (time 1763511035.20 total 326.70 last 66.18) - open/close 23373 (time 1763511045.20 total 336.70 last 67.36) - open/close 24059 (time 1763511055.22 total 346.72 last 68.47) - open/close 24700 (time 1763511065.25 total 356.75 last 63.94) - open/close 25410 (time 1763511075.26 total 366.76 last 70.94) - open/close 26132 (time 1763511085.27 total 376.77 last 72.13) - open/close 26858 (time 1763511095.27 total 386.77 last 72.56) - open/close 27502 (time 1763511105.27 total 396.77 last 64.40) - open/close 28167 (time 1763511115.29 total 406.79 last 66.42) - open/close 28867 (time 1763511125.29 total 416.79 last 69.96) - open/close 29573 (time 1763511135.29 total 426.79 last 70.60) - open/close 30000 (time 1763511142.14 total 433.64 last 62.38) - open/close 30648 (time 1763511152.14 total 443.64 last 64.77) - open/close 31301 (time 1763511162.16 total 453.66 last 65.21) - open/close 31921 (time 1763511172.17 total 463.67 last 61.91) - open/close 32613 (time 1763511182.17 total 473.67 last 69.20) - open/close 33260 (time 1763511192.18 total 483.68 last 64.66) - open/close 33846 (time 1763511202.18 total 493.68 last 58.59) - open/close 34538 (time 1763511212.18 total 503.68 last 69.19) - open/close 35238 (time 1763511222.22 total 513.72 last 69.72) - open/close 35931 (time 1763511232.23 total 523.73 last 69.25) - open/close 36593 (time 1763511242.24 total 533.74 last 66.12) - open/close 37228 (time 1763511252.25 total 543.75 last 63.46) - open/close 37894 (time 1763511262.25 total 553.75 last 66.55) - open/close 38591 (time 1763511272.26 total 563.76 last 69.65) - open/close 39229 (time 1763511282.28 total 573.78 last 63.67) - open/close 39924 (time 1763511292.28 total 583.78 last 69.49) - open/close 40649 (time 1763511302.29 total 593.79 last 72.42) - open/close 41298 (time 1763511312.30 total 603.80 last 64.83) - open/close 41897 (time 1763511322.31 total 613.81 last 59.84) - open/close 42572 (time 1763511332.33 total 623.82 last 67.42) - open/close 43266 (time 1763511342.34 total 633.84 last 69.30) - open/close 44037 (time 1763511352.35 total 643.85 last 77.02) - open/close 44759 (time 1763511362.35 total 653.85 last 72.17) - open/close 45422 (time 1763511372.37 total 663.86 last 66.23) - open/close 46047 (time 1763511382.37 total 673.87 last 62.44) - open/close 46714 (time 1763511392.38 total 683.88 last 66.67) - open/close 47398 (time 1763511402.39 total 693.89 last 68.33) - open/close 48016 (time 1763511412.40 total 703.90 last 61.76) - open/close 48671 (time 1763511422.41 total 713.91 last 65.41) - open/close 49379 (time 1763511432.41 total 723.91 last 70.77) - open/close 50000 (time 1763511441.54 total 733.04 last 68.03) - open/close 50661 (time 1763511451.57 total 743.07 last 65.93) - open/close 51309 (time 1763511461.58 total 753.08 last 64.74) - open/close 51966 (time 1763511471.59 total 763.09 last 65.64) - open/close 52601 (time 1763511481.61 total 773.11 last 63.33) - open/close 53239 (time 1763511491.62 total 783.12 last 63.77) - open/close 53917 (time 1763511501.63 total 793.13 last 67.73) - open/close 54550 (time 1763511511.64 total 803.14 last 63.21) - open/close 55273 (time 1763511521.64 total 813.14 last 72.30) - open/close 55897 (time 1763511531.65 total 823.15 last 62.33) - open/close 56525 (time 1763511541.67 total 833.17 last 62.67) - open/close 57126 (time 1763511551.68 total 843.18 last 60.05) - open/close 57756 (time 1763511561.68 total 853.18 last 62.99) - open/close 58357 (time 1763511571.68 total 863.18 last 60.10) - open/close 58991 (time 1763511581.70 total 873.20 last 63.32) - open/close 59662 (time 1763511591.71 total 883.21 last 67.03) - open/close 60000 (time 1763511596.53 total 888.03 last 70.12) - open/close 60603 (time 1763511606.54 total 898.03 last 60.25) - open/close 61223 (time 1763511616.54 total 908.04 last 61.99) - open/close 61845 (time 1763511626.55 total 918.04 last 62.14) - open/close 62459 (time 1763511636.55 total 928.05 last 61.35) - open/close 63075 (time 1763511646.56 total 938.06 last 61.55) - open/close 63708 (time 1763511656.56 total 948.06 last 63.28) - open/close 64441 (time 1763511666.58 total 958.07 last 73.22) total: 64768 open/close in 962.41 seconds: 67.30 ops/second - /unlink 693 (time 1763511684.44 total 10.01 last 69.25) - /unlink 1381 (time 1763511694.45 total 20.02 last 68.73) - /unlink 1979 (time 1763511704.46 total 30.03 last 59.74) - /unlink 2613 (time 1763511714.48 total 40.04 last 63.33) - /unlink 3308 (time 1763511724.48 total 50.04 last 69.49) - /unlink 3991 (time 1763511734.49 total 60.05 last 68.23) - /unlink 4631 (time 1763511744.51 total 70.07 last 63.85) - /unlink 5225 (time 1763511754.51 total 80.08 last 59.39) - /unlink 5880 (time 1763511764.51 total 90.08 last 65.48) - /unlink 6525 (time 1763511774.54 total 100.10 last 64.35) - /unlink 7203 (time 1763511784.55 total 110.11 last 67.72) - /unlink 7844 (time 1763511794.55 total 120.12 last 64.07) - /unlink 8473 (time 1763511804.55 total 130.12 last 62.89) - /unlink 9207 (time 1763511814.57 total 140.13 last 73.30) - /unlink 9826 (time 1763511824.57 total 150.13 last 61.88) - /unlink 10000 (time 1763511827.43 total 152.99 last 60.91) - /unlink 10613 (time 1763511837.43 total 163.00 last 61.26) - /unlink 11213 (time 1763511847.45 total 173.01 last 59.92) - /unlink 11809 (time 1763511857.45 total 183.01 last 59.58) - /unlink 12321 (time 1763511867.46 total 193.02 last 51.16) - /unlink 12928 (time 1763511877.47 total 203.03 last 60.63) - /unlink 13563 (time 1763511887.48 total 213.04 last 63.44) - /unlink 14200 (time 1763511897.48 total 223.05 last 63.67) - /unlink 14846 (time 1763511907.51 total 233.07 last 64.45) - /unlink 15514 (time 1763511917.53 total 243.09 last 66.66) - /unlink 16099 (time 1763511927.53 total 253.10 last 58.46) - /unlink 16702 (time 1763511937.54 total 263.10 last 60.27) - /unlink 17312 (time 1763511947.54 total 273.10 last 61.00) - /unlink 17919 (time 1763511957.55 total 283.11 last 60.64) - /unlink 18516 (time 1763511967.56 total 293.12 last 59.65) - /unlink 19123 (time 1763511977.57 total 303.13 last 60.63) - /unlink 19787 (time 1763511987.57 total 313.13 last 66.39) - /unlink 20000 (time 1763511991.01 total 316.57 last 61.95) - /unlink 20670 (time 1763512001.02 total 326.58 last 66.93) - /unlink 21221 (time 1763512011.07 total 336.64 last 54.81) - /unlink 21788 (time 1763512021.09 total 346.65 last 56.61) - /unlink 22363 (time 1763512031.10 total 356.67 last 57.41) - /unlink 22932 (time 1763512041.12 total 366.68 last 56.83) - /unlink 23498 (time 1763512051.12 total 376.68 last 56.59) - /unlink 24059 (time 1763512061.12 total 386.68 last 56.10) - /unlink 24654 (time 1763512071.15 total 396.71 last 59.32) - /unlink 25240 (time 1763512081.16 total 406.73 last 58.52) - /unlink 25782 (time 1763512091.17 total 416.73 last 54.17) - /unlink 26313 (time 1763512101.19 total 426.75 last 53.01) - /unlink 26825 (time 1763512111.20 total 436.76 last 51.13) - /unlink 27416 (time 1763512121.20 total 446.77 last 59.09) - /unlink 27979 (time 1763512131.21 total 456.78 last 56.23) - /unlink 28521 (time 1763512141.22 total 466.78 last 54.19) - /unlink 29117 (time 1763512151.22 total 476.78 last 59.59) - /unlink 29680 (time 1763512161.23 total 486.80 last 56.21) - /unlink 30000 (time 1763512166.67 total 492.24 last 58.82) - /unlink 30566 (time 1763512176.68 total 502.24 last 56.58) - /unlink 31145 (time 1763512186.69 total 512.25 last 57.84) - /unlink 31748 (time 1763512196.69 total 522.26 last 60.25) - /unlink 32304 (time 1763512206.71 total 532.27 last 55.53) - /unlink 32829 (time 1763512216.73 total 542.29 last 52.37) - /unlink 33341 (time 1763512226.74 total 552.31 last 51.15) - /unlink 33899 (time 1763512236.76 total 562.32 last 55.71) - /unlink 34478 (time 1763512246.76 total 572.32 last 57.90) - /unlink 35064 (time 1763512256.76 total 582.33 last 58.58) - /unlink 35687 (time 1763512266.77 total 592.34 last 62.23) - /unlink 36199 (time 1763512276.78 total 602.34 last 51.17) - /unlink 36703 (time 1763512286.79 total 612.36 last 50.33) - /unlink 37290 (time 1763512296.81 total 622.37 last 58.61) - /unlink 37889 (time 1763512306.82 total 632.38 last 59.85) - /unlink 38508 (time 1763512316.83 total 642.39 last 61.82) - /unlink 39172 (time 1763512326.84 total 652.41 last 66.31) - /unlink 39757 (time 1763512336.85 total 662.42 last 58.44) - /unlink 40000 (time 1763512340.84 total 666.40 last 61.05) - /unlink 40593 (time 1763512350.85 total 676.41 last 59.22) - /unlink 41201 (time 1763512360.86 total 686.42 last 60.73) - /unlink 41806 (time 1763512370.86 total 696.43 last 60.48) - /unlink 42426 (time 1763512380.87 total 706.43 last 61.96) - /unlink 43047 (time 1763512390.87 total 716.44 last 62.09) - /unlink 43640 (time 1763512400.89 total 726.45 last 59.22) - /unlink 44261 (time 1763512410.90 total 736.46 last 62.04) - /unlink 44844 (time 1763512420.90 total 746.46 last 58.28) - /unlink 45401 (time 1763512430.91 total 756.47 last 55.65) - /unlink 46012 (time 1763512440.91 total 766.47 last 61.08) - /unlink 46645 (time 1763512450.92 total 776.48 last 63.24) - /unlink 47196 (time 1763512460.92 total 786.49 last 55.08) - /unlink 47810 (time 1763512470.93 total 796.49 last 61.35) - /unlink 48422 (time 1763512480.94 total 806.50 last 61.14) - /unlink 49064 (time 1763512490.95 total 816.51 last 64.13) - /unlink 49670 (time 1763512500.97 total 826.53 last 60.47) - /unlink 50000 (time 1763512506.54 total 832.10 last 59.25) - /unlink 50635 (time 1763512516.55 total 842.11 last 63.43) - /unlink 51224 (time 1763512526.55 total 852.12 last 58.87) - /unlink 51810 (time 1763512536.57 total 862.14 last 58.51) - /unlink 52410 (time 1763512546.58 total 872.15 last 59.92) - /unlink 53071 (time 1763512556.59 total 882.16 last 66.04) - /unlink 53701 (time 1763512566.60 total 892.17 last 62.93) - /unlink 54320 (time 1763512576.62 total 902.18 last 61.81) - /unlink 54820 (time 1763512586.63 total 912.20 last 49.93) - /unlink 55360 (time 1763512596.63 total 922.20 last 53.99) - /unlink 55955 (time 1763512606.64 total 932.20 last 59.48) - /unlink 56560 (time 1763512616.65 total 942.21 last 60.46) - /unlink 57185 (time 1763512626.65 total 952.21 last 62.47) - /unlink 57793 (time 1763512636.65 total 962.22 last 60.78) - /unlink 58385 (time 1763512646.65 total 972.22 last 59.19) - /unlink 58979 (time 1763512656.66 total 982.22 last 59.38) - /unlink 59567 (time 1763512666.66 total 992.22 last 58.80) - /unlink 60000 (time 1763512674.24 total 999.80 last 57.15) - /unlink 60629 (time 1763512684.25 total 1009.82 last 62.79) - /unlink 61232 (time 1763512694.27 total 1019.83 last 60.21) - /unlink 61908 (time 1763512704.27 total 1029.83 last 67.58) - /unlink 62558 (time 1763512714.28 total 1039.85 last 64.92) - /unlink 63211 (time 1763512724.29 total 1049.86 last 65.23) - /unlink 63864 (time 1763512734.30 total 1059.87 last 65.23) - /unlink 64457 (time 1763512744.33 total 1069.89 last 59.14) total: 64768 /unlink in 1074.92 seconds: 60.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 676 (time 1763512788.49 total 10.01 last 67.53) - open/close 1335 (time 1763512798.50 total 20.01 last 65.88) - open/close 1980 (time 1763512808.50 total 30.02 last 64.46) - open/close 2606 (time 1763512818.51 total 40.02 last 62.59) - open/close 3196 (time 1763512828.52 total 50.04 last 58.91) - open/close 3875 (time 1763512838.52 total 60.04 last 67.89) - open/close 4491 (time 1763512848.54 total 70.06 last 61.48) - open/close 5085 (time 1763512858.55 total 80.07 last 59.32) - open/close 5726 (time 1763512868.56 total 90.07 last 64.07) - open/close 6379 (time 1763512878.57 total 100.08 last 65.26) - open/close 6999 (time 1763512888.57 total 110.09 last 61.97) - open/close 7638 (time 1763512898.58 total 120.09 last 63.87) - open/close 8272 (time 1763512908.59 total 130.10 last 63.34) - open/close 8918 (time 1763512918.60 total 140.12 last 64.49) - open/close 9582 (time 1763512928.61 total 150.13 last 66.34) - open/close 10000 (time 1763512934.87 total 156.39 last 66.79) - open/close 10644 (time 1763512944.87 total 166.39 last 64.37) - open/close 11268 (time 1763512954.88 total 176.39 last 62.38) - open/close 11912 (time 1763512964.89 total 186.40 last 64.35) - open/close 12627 (time 1763512974.89 total 196.41 last 71.48) - open/close 13356 (time 1763512984.89 total 206.41 last 72.87) - open/close 14067 (time 1763512994.90 total 216.41 last 71.07) - open/close 14758 (time 1763513004.91 total 226.43 last 69.02) - open/close 15471 (time 1763513014.91 total 236.43 last 71.28) - open/close 16165 (time 1763513024.92 total 246.43 last 69.39) - open/close 16817 (time 1763513034.92 total 256.43 last 65.19) - open/close 17429 (time 1763513044.92 total 266.44 last 61.15) - open/close 18146 (time 1763513054.93 total 276.45 last 71.65) - open/close 18858 (time 1763513064.95 total 286.47 last 71.08) - open/close 19555 (time 1763513074.95 total 296.47 last 69.68) - open/close 20000 (time 1763513081.57 total 303.08 last 67.26) - open/close 20703 (time 1763513091.58 total 313.10 last 70.19) - open/close 21403 (time 1763513101.60 total 323.11 last 69.91) - open/close 22067 (time 1763513111.60 total 333.12 last 66.37) - open/close 22771 (time 1763513121.61 total 343.13 last 70.31) - open/close 23445 (time 1763513131.62 total 353.13 last 67.38) - open/close 24078 (time 1763513141.62 total 363.14 last 63.26) - open/close 24735 (time 1763513151.63 total 373.15 last 65.63) - open/close 25413 (time 1763513161.64 total 383.15 last 67.77) - open/close 26102 (time 1763513171.65 total 393.16 last 68.83) - open/close 26792 (time 1763513181.66 total 403.18 last 68.91) - open/close 27470 (time 1763513191.67 total 413.18 last 67.75) - open/close 28157 (time 1763513201.68 total 423.20 last 68.61) - open/close 28835 (time 1763513211.69 total 433.21 last 67.72) - open/close 29466 (time 1763513221.71 total 443.23 last 62.97) - open/close 30000 (time 1763513229.99 total 451.50 last 64.54) - open/close 30609 (time 1763513239.99 total 461.50 last 60.89) - open/close 31332 (time 1763513249.99 total 471.51 last 72.28) - open/close 32049 (time 1763513260.00 total 481.51 last 71.66) - open/close 32757 (time 1763513270.01 total 491.53 last 70.69) - open/close 33423 (time 1763513280.01 total 501.53 last 66.59) - open/close 34061 (time 1763513290.02 total 511.53 last 63.79) - open/close 34696 (time 1763513300.03 total 521.54 last 63.42) - open/close 35306 (time 1763513310.03 total 531.55 last 60.99) - open/close 35911 (time 1763513320.03 total 541.55 last 60.49) - open/close 36575 (time 1763513330.04 total 551.56 last 66.33) - open/close 37155 (time 1763513340.06 total 561.57 last 57.91) - open/close 37747 (time 1763513350.06 total 571.57 last 59.19) - open/close 38426 (time 1763513360.06 total 581.58 last 67.88) - open/close 39111 (time 1763513370.07 total 591.59 last 68.41) - open/close 39757 (time 1763513380.08 total 601.60 last 64.54) - open/close 40000 (time 1763513383.64 total 605.16 last 68.29) - open/close 40614 (time 1763513393.65 total 615.17 last 61.33) - open/close 41268 (time 1763513403.67 total 625.18 last 65.32) - open/close 41947 (time 1763513413.69 total 635.20 last 67.75) - open/close 42585 (time 1763513423.70 total 645.21 last 63.73) - open/close 43208 (time 1763513433.72 total 655.23 last 62.19) - open/close 43892 (time 1763513443.74 total 665.26 last 68.20) - open/close 44535 (time 1763513453.75 total 675.27 last 64.24) - open/close 45180 (time 1763513463.76 total 685.27 last 64.48) - open/close 45832 (time 1763513473.76 total 695.28 last 65.17) - open/close 46387 (time 1763513483.78 total 705.29 last 55.42) - open/close 46920 (time 1763513493.79 total 715.30 last 53.25) - open/close 47523 (time 1763513503.79 total 725.31 last 60.25) - open/close 48123 (time 1763513513.80 total 735.32 last 59.94) - open/close 48709 (time 1763513523.82 total 745.34 last 58.49) - open/close 49328 (time 1763513533.83 total 755.34 last 61.88) - open/close 49993 (time 1763513543.83 total 765.35 last 66.46) - open/close 50648 (time 1763513553.84 total 775.35 last 65.48) - open/close 51345 (time 1763513563.84 total 785.35 last 69.68) - open/close 52011 (time 1763513573.84 total 795.36 last 66.56) - open/close 52681 (time 1763513583.86 total 805.37 last 66.92) - open/close 53312 (time 1763513593.89 total 815.40 last 62.91) - open/close 54007 (time 1763513603.91 total 825.43 last 69.34) - open/close 54647 (time 1763513613.91 total 835.43 last 63.98) - open/close 55238 (time 1763513623.92 total 845.44 last 59.06) - open/close 55901 (time 1763513633.92 total 855.44 last 66.29) - open/close 56604 (time 1763513643.93 total 865.44 last 70.27) - open/close 57257 (time 1763513653.93 total 875.44 last 65.30) - open/close 57923 (time 1763513663.93 total 885.45 last 66.56) - open/close 58536 (time 1763513673.94 total 895.46 last 61.25) - open/close 59216 (time 1763513683.95 total 905.47 last 67.92) - open/close 59893 (time 1763513693.96 total 915.48 last 67.64) - open/close 60563 (time 1763513703.97 total 925.49 last 66.96) - open/close 61246 (time 1763513713.97 total 935.49 last 68.27) - open/close 61914 (time 1763513723.98 total 945.50 last 66.75) - open/close 62572 (time 1763513733.99 total 955.51 last 65.72) - open/close 63175 (time 1763513744.01 total 965.52 last 60.20) - open/close 63847 (time 1763513754.02 total 975.54 last 67.11) - open/close 64507 (time 1763513764.02 total 985.54 last 66.00) total: 64768 open/close in 989.32 seconds: 65.47 ops/second - /unlink 643 (time 1763513781.27 total 10.01 last 64.25) - /unlink 1231 (time 1763513791.28 total 20.03 last 58.70) - /unlink 1881 (time 1763513801.29 total 30.03 last 64.97) - /unlink 2575 (time 1763513811.31 total 40.05 last 69.28) - /unlink 3303 (time 1763513821.32 total 50.06 last 72.74) - /unlink 4022 (time 1763513831.32 total 60.06 last 71.88) - /unlink 4761 (time 1763513841.33 total 70.07 last 73.84) - /unlink 5497 (time 1763513851.33 total 80.07 last 73.57) - /unlink 6239 (time 1763513861.34 total 90.08 last 74.14) - /unlink 6915 (time 1763513871.34 total 100.08 last 67.57) - /unlink 7684 (time 1763513881.36 total 110.10 last 76.79) - /unlink 8441 (time 1763513891.37 total 120.11 last 75.62) - /unlink 9133 (time 1763513901.38 total 130.12 last 69.13) - /unlink 9840 (time 1763513911.38 total 140.12 last 70.69) - /unlink 10000 (time 1763513913.49 total 142.23 last 75.70) - /unlink 10747 (time 1763513923.50 total 152.24 last 74.65) - /unlink 11395 (time 1763513933.51 total 162.25 last 64.73) - /unlink 12170 (time 1763513943.52 total 172.26 last 77.42) - /unlink 12898 (time 1763513953.53 total 182.27 last 72.76) - /unlink 13712 (time 1763513963.53 total 192.27 last 81.37) - /unlink 14466 (time 1763513973.54 total 202.28 last 75.31) - /unlink 15184 (time 1763513983.55 total 212.29 last 71.74) - /unlink 15953 (time 1763513993.55 total 222.29 last 76.90) - /unlink 16640 (time 1763514003.56 total 232.30 last 68.64) - /unlink 17279 (time 1763514013.57 total 242.32 last 63.82) - /unlink 18003 (time 1763514023.58 total 252.32 last 72.34) - /unlink 18803 (time 1763514033.58 total 262.32 last 80.00) - /unlink 19564 (time 1763514043.58 total 272.32 last 76.09) - /unlink 20000 (time 1763514049.43 total 278.17 last 74.56) - /unlink 20730 (time 1763514059.43 total 288.17 last 73.00) - /unlink 21387 (time 1763514069.43 total 298.18 last 65.68) - /unlink 22095 (time 1763514079.44 total 308.18 last 70.77) - /unlink 22818 (time 1763514089.46 total 318.20 last 72.13) - /unlink 23523 (time 1763514099.47 total 328.21 last 70.44) - /unlink 24163 (time 1763514109.47 total 338.21 last 63.99) - /unlink 24798 (time 1763514119.48 total 348.22 last 63.46) - /unlink 25477 (time 1763514129.49 total 358.23 last 67.85) - /unlink 26093 (time 1763514139.50 total 368.24 last 61.54) - /unlink 26784 (time 1763514149.51 total 378.25 last 69.03) - /unlink 27472 (time 1763514159.51 total 388.26 last 68.76) - /unlink 28117 (time 1763514169.52 total 398.27 last 64.43) - /unlink 28748 (time 1763514179.53 total 408.28 last 63.05) - /unlink 29314 (time 1763514189.54 total 418.28 last 56.57) - /unlink 29965 (time 1763514199.55 total 428.29 last 65.04) - /unlink 30590 (time 1763514209.55 total 438.29 last 62.50) - /unlink 31292 (time 1763514219.56 total 448.30 last 70.14) - /unlink 31954 (time 1763514229.56 total 458.31 last 66.15) - /unlink 32537 (time 1763514239.58 total 468.32 last 58.21) - /unlink 33109 (time 1763514249.60 total 478.34 last 57.08) - /unlink 33693 (time 1763514259.62 total 488.36 last 58.32) - /unlink 34322 (time 1763514269.63 total 498.38 last 62.78) - /unlink 34926 (time 1763514279.64 total 508.39 last 60.34) - /unlink 35570 (time 1763514289.65 total 518.39 last 64.35) - /unlink 36206 (time 1763514299.65 total 528.40 last 63.59) - /unlink 36885 (time 1763514309.67 total 538.42 last 67.76) - /unlink 37608 (time 1763514319.67 total 548.42 last 72.30) - /unlink 38309 (time 1763514329.68 total 558.43 last 70.03) - /unlink 38959 (time 1763514339.69 total 568.43 last 64.99) - /unlink 39606 (time 1763514349.69 total 578.44 last 64.66) - /unlink 40000 (time 1763514355.96 total 584.70 last 62.91) - /unlink 40692 (time 1763514365.96 total 594.70 last 69.16) - /unlink 41396 (time 1763514375.97 total 604.72 last 70.32) - /unlink 42062 (time 1763514385.98 total 614.72 last 66.56) - /unlink 42717 (time 1763514395.99 total 624.73 last 65.45) - /unlink 43356 (time 1763514405.99 total 634.73 last 63.87) - /unlink 44026 (time 1763514416.00 total 644.74 last 66.95) - /unlink 44675 (time 1763514426.01 total 654.76 last 64.80) - /unlink 45396 (time 1763514436.02 total 664.76 last 72.04) - /unlink 46093 (time 1763514446.03 total 674.77 last 69.66) - /unlink 46758 (time 1763514456.03 total 684.77 last 66.49) - /unlink 47448 (time 1763514466.03 total 694.77 last 68.99) - /unlink 48072 (time 1763514476.04 total 704.79 last 62.33) - /unlink 48695 (time 1763514486.05 total 714.79 last 62.26) - /unlink 49306 (time 1763514496.06 total 724.80 last 61.04) - /unlink 49946 (time 1763514506.07 total 734.81 last 63.94) - /unlink 50593 (time 1763514516.08 total 744.83 last 64.61) - /unlink 51250 (time 1763514526.09 total 754.83 last 65.69) - /unlink 51904 (time 1763514536.09 total 764.83 last 65.38) - /unlink 52526 (time 1763514546.10 total 774.84 last 62.15) - /unlink 53156 (time 1763514556.11 total 784.85 last 62.94) - /unlink 53783 (time 1763514566.11 total 794.85 last 62.68) - /unlink 54446 (time 1763514576.11 total 804.85 last 66.29) - /unlink 55099 (time 1763514586.13 total 814.87 last 65.18) - /unlink 55790 (time 1763514596.15 total 824.89 last 68.98) - /unlink 56497 (time 1763514606.16 total 834.90 last 70.65) - /unlink 57144 (time 1763514616.16 total 844.90 last 64.68) - /unlink 57785 (time 1763514626.16 total 854.90 last 64.07) - /unlink 58473 (time 1763514636.17 total 864.91 last 68.77) - /unlink 59159 (time 1763514646.18 total 874.92 last 68.51) - /unlink 59808 (time 1763514656.19 total 884.94 last 64.81) - /unlink 60000 (time 1763514658.89 total 887.64 last 71.09) - /unlink 60647 (time 1763514668.90 total 897.64 last 64.69) - /unlink 61308 (time 1763514678.90 total 907.65 last 66.04) - /unlink 61962 (time 1763514688.91 total 917.65 last 65.39) - /unlink 62626 (time 1763514698.92 total 927.66 last 66.32) - /unlink 63324 (time 1763514708.93 total 937.68 last 69.69) - /unlink 64017 (time 1763514718.94 total 947.69 last 69.24) - /unlink 64659 (time 1763514728.95 total 957.70 last 64.13) total: 64768 /unlink in 959.59 seconds: 67.50 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 694 (time 1763514769.35 total 10.01 last 69.32) - open/close 1386 (time 1763514779.36 total 20.02 last 69.14) - open/close 2076 (time 1763514789.38 total 30.04 last 68.85) - open/close 2768 (time 1763514799.40 total 40.06 last 69.05) - open/close 3497 (time 1763514809.42 total 50.08 last 72.76) - open/close 4198 (time 1763514819.43 total 60.09 last 70.07) - open/close 4889 (time 1763514829.43 total 70.09 last 69.08) - open/close 5596 (time 1763514839.45 total 80.11 last 70.60) - open/close 6331 (time 1763514849.45 total 90.11 last 73.48) - open/close 7080 (time 1763514859.46 total 100.12 last 74.82) - open/close 7853 (time 1763514869.47 total 110.13 last 77.23) - open/close 8592 (time 1763514879.47 total 120.13 last 73.88) - open/close 9323 (time 1763514889.47 total 130.13 last 73.09) - open/close 10000 (time 1763514898.56 total 139.22 last 74.52) - open/close 10724 (time 1763514908.57 total 149.23 last 72.33) - open/close 11507 (time 1763514918.58 total 159.24 last 78.22) - open/close 12452 (time 1763514928.59 total 169.25 last 94.39) - open/close 13338 (time 1763514938.59 total 179.25 last 88.57) - open/close 14239 (time 1763514948.60 total 189.26 last 90.01) - open/close 15246 (time 1763514958.61 total 199.27 last 100.63) - open/close 16257 (time 1763514968.61 total 209.27 last 101.06) - open/close 17090 (time 1763514978.63 total 219.29 last 83.20) - open/close 18082 (time 1763514988.64 total 229.30 last 99.08) - open/close 19072 (time 1763514998.64 total 239.30 last 99.00) - open/close 19941 (time 1763515008.65 total 249.31 last 86.82) - open/close 20836 (time 1763515018.67 total 259.33 last 89.33) - open/close 21749 (time 1763515028.67 total 269.33 last 91.27) - open/close 22714 (time 1763515038.67 total 279.33 last 96.48) - open/close 23602 (time 1763515048.67 total 289.33 last 88.78) - open/close 24484 (time 1763515058.68 total 299.34 last 88.16) - open/close 25384 (time 1763515068.68 total 309.34 last 89.99) - open/close 26318 (time 1763515078.69 total 319.35 last 93.31) - open/close 27115 (time 1763515088.70 total 329.36 last 79.64) - open/close 27959 (time 1763515098.70 total 339.36 last 84.33) - open/close 28752 (time 1763515108.72 total 349.38 last 79.21) - open/close 29595 (time 1763515118.72 total 359.38 last 84.30) - open/close 30000 (time 1763515123.16 total 363.82 last 91.09) - open/close 30860 (time 1763515133.17 total 373.83 last 85.97) - open/close 31727 (time 1763515143.17 total 383.83 last 86.65) - open/close 32780 (time 1763515153.18 total 393.83 last 105.25) - open/close 33826 (time 1763515163.18 total 403.84 last 104.59) - open/close 34933 (time 1763515173.18 total 413.84 last 110.70) - open/close 35864 (time 1763515183.18 total 423.84 last 93.10) - open/close 36858 (time 1763515193.18 total 433.84 last 99.38) - open/close 37990 (time 1763515203.18 total 443.84 last 113.18) - open/close 38929 (time 1763515213.19 total 453.85 last 93.80) - open/close 39821 (time 1763515223.19 total 463.85 last 89.19) - open/close 40000 (time 1763515225.28 total 465.94 last 85.70) - open/close 40853 (time 1763515235.30 total 475.96 last 85.16) - open/close 41894 (time 1763515245.30 total 485.96 last 104.06) - open/close 42809 (time 1763515255.31 total 495.97 last 91.42) - open/close 43897 (time 1763515265.31 total 505.97 last 108.77) - open/close 45191 (time 1763515275.31 total 515.97 last 129.40) - open/close 46344 (time 1763515285.32 total 525.98 last 115.23) - open/close 47652 (time 1763515295.32 total 535.98 last 130.75) - open/close 48749 (time 1763515305.33 total 545.99 last 109.61) - open/close 49829 (time 1763515315.34 total 556.00 last 107.94) - open/close 51265 (time 1763515325.34 total 566.00 last 143.56) - open/close 52612 (time 1763515335.34 total 576.00 last 134.64) - open/close 53861 (time 1763515345.35 total 586.01 last 124.83) - open/close 55213 (time 1763515355.35 total 596.01 last 135.19) - open/close 56257 (time 1763515365.35 total 606.01 last 104.37) - open/close 57568 (time 1763515375.37 total 616.02 last 130.95) - open/close 58751 (time 1763515385.37 total 626.03 last 118.27) - open/close 60000 (time 1763515393.38 total 634.04 last 155.81) - open/close 61649 (time 1763515403.38 total 644.04 last 164.88) - open/close 63071 (time 1763515413.39 total 654.05 last 142.15) - open/close 64450 (time 1763515423.39 total 664.05 last 137.84) total: 64768 open/close in 666.58 seconds: 97.17 ops/second - /unlink 1423 (time 1763515438.00 total 10.00 last 142.24) - /unlink 2807 (time 1763515448.00 total 20.01 last 138.36) - /unlink 4238 (time 1763515458.01 total 30.01 last 143.05) - /unlink 5755 (time 1763515468.01 total 40.01 last 151.67) - /unlink 7169 (time 1763515478.01 total 50.02 last 141.35) - /unlink 8974 (time 1763515488.01 total 60.02 last 180.50) - /unlink 10000 (time 1763515491.57 total 63.57 last 288.61) - /unlink 13427 (time 1763515501.57 total 73.57 last 342.66) - /unlink 16506 (time 1763515511.57 total 83.57 last 307.82) - /unlink 19594 (time 1763515521.57 total 93.58 last 308.72) - /unlink 22333 (time 1763515531.58 total 103.58 last 273.81) - /unlink 25684 (time 1763515541.58 total 113.58 last 335.09) - /unlink 28921 (time 1763515551.58 total 123.58 last 323.59) - /unlink 30000 (time 1763515554.91 total 126.92 last 323.82) - /unlink 33350 (time 1763515564.91 total 136.92 last 334.98) - /unlink 35764 (time 1763515574.91 total 146.92 last 241.39) - /unlink 38082 (time 1763515584.91 total 156.92 last 231.75) - /unlink 40000 (time 1763515591.58 total 163.59 last 287.63) - /unlink 42469 (time 1763515601.58 total 173.59 last 246.88) - /unlink 45437 (time 1763515611.59 total 183.59 last 296.73) - /unlink 48102 (time 1763515621.59 total 193.59 last 266.44) - /unlink 50000 (time 1763515627.89 total 199.89 last 301.18) - /unlink 52986 (time 1763515637.89 total 209.90 last 298.58) - /unlink 55867 (time 1763515647.89 total 219.90 last 288.09) - /unlink 59243 (time 1763515657.90 total 229.90 last 337.45) - /unlink 60000 (time 1763515660.59 total 232.60 last 280.83) - /unlink 63336 (time 1763515670.59 total 242.60 last 333.58) total: 64768 /unlink in 246.83 seconds: 262.40 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1700 (time 1763515709.46 total 10.00 last 169.97) - open/close 3857 (time 1763515719.47 total 20.01 last 215.63) - open/close 5660 (time 1763515729.47 total 30.01 last 180.29) - open/close 8047 (time 1763515739.47 total 40.01 last 238.62) - open/close 10000 (time 1763515748.67 total 49.21 last 212.22) - open/close 12171 (time 1763515758.67 total 59.21 last 217.05) - open/close 14341 (time 1763515768.68 total 69.22 last 216.87) - open/close 16280 (time 1763515778.68 total 79.22 last 193.87) - open/close 18606 (time 1763515788.69 total 89.23 last 232.43) - open/close 20000 (time 1763515795.31 total 95.85 last 210.38) - open/close 22233 (time 1763515805.32 total 105.85 last 223.29) - open/close 24300 (time 1763515815.32 total 115.86 last 206.64) - open/close 26190 (time 1763515825.32 total 125.86 last 188.92) - open/close 28294 (time 1763515835.33 total 135.87 last 210.22) - open/close 30000 (time 1763515844.16 total 144.70 last 193.26) - open/close 32969 (time 1763515854.16 total 154.70 last 296.86) - open/close 36399 (time 1763515864.16 total 164.70 last 342.92) - open/close 38838 (time 1763515874.16 total 174.70 last 243.85) - open/close 40000 (time 1763515879.53 total 180.07 last 216.51) - open/close 42103 (time 1763515889.53 total 190.07 last 210.26) - open/close 44119 (time 1763515899.54 total 200.08 last 201.55) - open/close 46562 (time 1763515909.54 total 210.08 last 244.23) - open/close 49192 (time 1763515919.54 total 220.08 last 262.90) - open/close 50000 (time 1763515923.80 total 224.34 last 189.76) - open/close 52654 (time 1763515933.80 total 234.34 last 265.34) - open/close 54980 (time 1763515943.81 total 244.35 last 232.52) - open/close 56922 (time 1763515953.81 total 254.35 last 194.13) - open/close 59699 (time 1763515963.81 total 264.35 last 277.65) - open/close 62141 (time 1763515973.81 total 274.35 last 244.19) - open/close 64719 (time 1763515983.82 total 284.36 last 257.53) total: 64768 open/close in 284.57 seconds: 227.60 ops/second - /unlink 4084 (time 1763515995.76 total 10.00 last 408.38) - /unlink 8603 (time 1763516005.76 total 20.00 last 451.79) - /unlink 10000 (time 1763516008.21 total 22.45 last 570.79) - /unlink 16136 (time 1763516018.21 total 32.45 last 613.53) - /unlink 20000 (time 1763516025.05 total 39.30 last 564.53) - /unlink 25154 (time 1763516035.05 total 49.30 last 515.40) - /unlink 29725 (time 1763516045.06 total 59.30 last 456.98) - /unlink 33557 (time 1763516055.06 total 69.30 last 383.16) - /unlink 37665 (time 1763516065.06 total 79.30 last 410.76) - /unlink 40000 (time 1763516071.33 total 85.57 last 372.58) - /unlink 43970 (time 1763516081.33 total 95.57 last 396.98) - /unlink 49297 (time 1763516091.34 total 105.59 last 531.77) - /unlink 53319 (time 1763516101.34 total 115.59 last 402.18) - /unlink 57971 (time 1763516111.35 total 125.59 last 465.16) - /unlink 60000 (time 1763516114.92 total 129.16 last 568.22) total: 64768 /unlink in 136.58 seconds: 474.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4372 (time 1763516151.11 total 10.00 last 437.13) - open/close 7348 (time 1763516161.11 total 20.00 last 297.58) - open/close 9265 (time 1763516171.11 total 30.01 last 191.64) - open/close 10000 (time 1763516174.00 total 32.89 last 254.65) - open/close 12395 (time 1763516184.00 total 42.89 last 239.47) - open/close 14219 (time 1763516194.01 total 52.90 last 182.28) - open/close 16120 (time 1763516204.01 total 62.90 last 190.09) - open/close 18834 (time 1763516214.01 total 72.90 last 271.37) - open/close 20000 (time 1763516218.03 total 76.92 last 290.18) - open/close 22901 (time 1763516228.03 total 86.92 last 290.06) - open/close 25901 (time 1763516238.03 total 96.92 last 299.96) - open/close 29557 (time 1763516248.03 total 106.93 last 365.50) - open/close 32839 (time 1763516258.03 total 116.93 last 328.18) - open/close 35735 (time 1763516268.04 total 126.93 last 289.46) - open/close 38405 (time 1763516278.04 total 136.93 last 266.95) - open/close 40000 (time 1763516282.76 total 141.65 last 337.88) - open/close 43123 (time 1763516292.76 total 151.65 last 312.27) - open/close 46412 (time 1763516302.76 total 161.66 last 328.85) - open/close 49109 (time 1763516312.77 total 171.66 last 269.59) - open/close 50000 (time 1763516317.43 total 176.33 last 190.86) - open/close 52397 (time 1763516327.44 total 186.33 last 239.64) - open/close 56039 (time 1763516337.44 total 196.33 last 364.17) - open/close 59361 (time 1763516347.44 total 206.33 last 332.19) - open/close 60000 (time 1763516350.68 total 209.57 last 197.41) - open/close 62805 (time 1763516360.68 total 219.57 last 280.49) total: 64768 open/close in 224.99 seconds: 287.87 ops/second - /unlink 6223 (time 1763516377.46 total 10.00 last 622.27) - /unlink 10000 (time 1763516384.13 total 16.67 last 566.10) - /unlink 14734 (time 1763516394.13 total 26.67 last 473.35) - /unlink 19917 (time 1763516404.13 total 36.67 last 518.23) - /unlink 26657 (time 1763516414.14 total 46.68 last 673.95) - /unlink 30000 (time 1763516418.46 total 51.00 last 772.57) - /unlink 36206 (time 1763516428.46 total 61.00 last 620.49) - /unlink 40000 (time 1763516434.58 total 67.12 last 620.54) - /unlink 46820 (time 1763516444.59 total 77.13 last 681.33) - /unlink 50000 (time 1763516449.56 total 82.10 last 639.52) - /unlink 57623 (time 1763516459.56 total 92.10 last 762.26) - /unlink 60000 (time 1763516463.50 total 96.04 last 603.32) total: 64768 /unlink in 102.71 seconds: 630.62 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg220-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg220-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server stop mds service on oleg220-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing unload_modules_local oleg220-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2