== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:45:30 (1753742730) Stopping clients: oleg259-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg259-client.virtnet /mnt/lustre2 (opts:-f) oleg259-server: oleg259-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing load_modules_local oleg259-server: Loading modules from /home/green/git/lustre-release/lustre oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg259-server: detected 4 online CPUs by sysfs oleg259-server: MODOPTS_LIBCFS= oleg259-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg259-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg259-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 577 (time 1753742836.42 total 10.01 last 57.64) - open/close 1106 (time 1753742846.44 total 20.03 last 52.77) - open/close 1703 (time 1753742856.45 total 30.04 last 59.68) - open/close 2273 (time 1753742866.46 total 40.05 last 56.95) - open/close 2864 (time 1753742876.46 total 50.05 last 59.06) - open/close 3462 (time 1753742886.47 total 60.06 last 59.75) - open/close 4074 (time 1753742896.47 total 70.06 last 61.19) - open/close 4672 (time 1753742906.48 total 80.08 last 59.73) - open/close 5230 (time 1753742916.49 total 90.08 last 55.78) - open/close 5830 (time 1753742926.49 total 100.08 last 59.98) - open/close 6451 (time 1753742936.51 total 110.10 last 62.00) - open/close 6976 (time 1753742946.52 total 120.12 last 52.42) - open/close 7560 (time 1753742956.54 total 130.13 last 58.31) - open/close 8115 (time 1753742966.54 total 140.13 last 55.50) - open/close 8734 (time 1753742976.55 total 150.14 last 61.84) - open/close 9350 (time 1753742986.56 total 160.15 last 61.52) - open/close 9926 (time 1753742996.60 total 170.19 last 57.39) - open/close 10427 (time 1753743006.61 total 180.20 last 50.06) - open/close 11073 (time 1753743016.62 total 190.21 last 64.55) - open/close 11718 (time 1753743026.63 total 200.22 last 64.43) - open/close 12379 (time 1753743036.63 total 210.22 last 66.06) - open/close 12976 (time 1753743046.64 total 220.23 last 59.64) - open/close 13616 (time 1753743056.65 total 230.24 last 63.95) - open/close 14285 (time 1753743066.66 total 240.25 last 66.83) - open/close 14886 (time 1753743076.66 total 250.26 last 60.08) - open/close 15489 (time 1753743086.69 total 260.28 last 60.17) - open/close 15981 (time 1753743096.70 total 270.29 last 49.14) - open/close 16446 (time 1753743106.70 total 280.29 last 46.49) - open/close 17024 (time 1753743116.71 total 290.30 last 57.74) - open/close 17589 (time 1753743126.72 total 300.31 last 56.48) - open/close 18206 (time 1753743136.73 total 310.32 last 61.63) - open/close 18807 (time 1753743146.75 total 320.34 last 59.98) - open/close 19416 (time 1753743156.76 total 330.35 last 60.79) - open/close 19979 (time 1753743166.79 total 340.38 last 56.13) - open/close 20547 (time 1753743176.80 total 350.39 last 56.75) - open/close 21054 (time 1753743186.80 total 360.39 last 50.70) - open/close 21672 (time 1753743196.80 total 370.40 last 61.79) - open/close 22227 (time 1753743206.82 total 380.42 last 55.39) - open/close 22764 (time 1753743216.86 total 390.45 last 53.52) - open/close 23384 (time 1753743226.89 total 400.48 last 61.82) - open/close 23959 (time 1753743236.89 total 410.49 last 57.46) - open/close 24295 (time 1753743246.90 total 420.49 last 33.57) - open/close 24800 (time 1753743256.91 total 430.50 last 50.45) - open/close 25375 (time 1753743266.92 total 440.52 last 57.43) - open/close 25998 (time 1753743276.95 total 450.54 last 62.13) - open/close 26557 (time 1753743286.96 total 460.55 last 55.85) - open/close 27142 (time 1753743296.97 total 470.56 last 58.43) - open/close 27802 (time 1753743306.98 total 480.57 last 65.95) - open/close 28436 (time 1753743317.00 total 490.59 last 63.30) - open/close 29051 (time 1753743327.01 total 500.60 last 61.42) - open/close 29728 (time 1753743337.03 total 510.62 last 67.57) - open/close 30000 (time 1753743342.22 total 515.82 last 52.36) - open/close 30553 (time 1753743352.24 total 525.83 last 55.24) - open/close 31094 (time 1753743362.24 total 535.83 last 54.06) - open/close 31660 (time 1753743372.25 total 545.84 last 56.56) - open/close 32294 (time 1753743382.25 total 555.84 last 63.38) - open/close 32917 (time 1753743392.27 total 565.86 last 62.21) - open/close 33541 (time 1753743402.27 total 575.86 last 62.38) - open/close 34145 (time 1753743412.27 total 585.86 last 60.38) - open/close 34777 (time 1753743422.31 total 595.90 last 62.99) - open/close 35414 (time 1753743432.31 total 605.90 last 63.66) - open/close 36066 (time 1753743442.32 total 615.91 last 65.17) - open/close 36717 (time 1753743452.32 total 625.92 last 65.06) - open/close 37281 (time 1753743462.33 total 635.92 last 56.36) - open/close 37892 (time 1753743472.33 total 645.92 last 61.09) - open/close 38556 (time 1753743482.34 total 655.93 last 66.35) - open/close 39221 (time 1753743492.35 total 665.95 last 66.41) - open/close 39885 (time 1753743502.36 total 675.95 last 66.35) - open/close 40477 (time 1753743512.38 total 685.97 last 59.09) - open/close 41100 (time 1753743522.38 total 695.97 last 62.30) - open/close 41703 (time 1753743532.39 total 705.98 last 60.28) - open/close 42277 (time 1753743542.39 total 715.98 last 57.37) - open/close 42836 (time 1753743552.40 total 725.99 last 55.85) - open/close 43476 (time 1753743562.41 total 736.00 last 63.94) - open/close 44135 (time 1753743572.42 total 746.01 last 65.84) - open/close 44768 (time 1753743582.42 total 756.02 last 63.26) - open/close 45387 (time 1753743592.43 total 766.02 last 61.89) - open/close 45887 (time 1753743602.44 total 776.03 last 49.94) - open/close 46462 (time 1753743612.44 total 786.04 last 57.47) - open/close 47094 (time 1753743622.45 total 796.04 last 63.17) - open/close 47713 (time 1753743632.46 total 806.05 last 61.84) - open/close 48371 (time 1753743642.47 total 816.06 last 65.73) - open/close 49037 (time 1753743652.48 total 826.07 last 66.52) - open/close 49504 (time 1753743662.50 total 836.09 last 46.61) - open/close 50000 (time 1753743671.98 total 845.57 last 52.31) - open/close 50605 (time 1753743681.99 total 855.58 last 60.46) - open/close 51203 (time 1753743691.99 total 865.58 last 59.78) - open/close 51851 (time 1753743702.01 total 875.60 last 64.67) - open/close 52501 (time 1753743712.03 total 885.62 last 64.88) - open/close 53176 (time 1753743722.03 total 895.62 last 67.49) - open/close 53783 (time 1753743732.03 total 905.62 last 60.70) - open/close 54360 (time 1753743742.04 total 915.63 last 57.69) - open/close 54984 (time 1753743752.04 total 925.63 last 62.40) - open/close 55582 (time 1753743762.04 total 935.63 last 59.75) - open/close 56148 (time 1753743772.05 total 945.64 last 56.57) - open/close 56716 (time 1753743782.07 total 955.66 last 56.71) - open/close 57297 (time 1753743792.07 total 965.66 last 58.09) - open/close 57947 (time 1753743802.10 total 975.69 last 64.76) - open/close 58490 (time 1753743812.11 total 985.71 last 54.24) - open/close 59092 (time 1753743822.14 total 995.73 last 60.04) - open/close 59647 (time 1753743832.17 total 1005.76 last 55.36) - open/close 60000 (time 1753743838.77 total 1012.36 last 53.46) - open/close 60525 (time 1753743848.77 total 1022.37 last 52.47) - open/close 61062 (time 1753743858.78 total 1032.37 last 53.65) - open/close 61648 (time 1753743868.81 total 1042.40 last 58.47) - open/close 62266 (time 1753743878.82 total 1052.41 last 61.74) - open/close 62885 (time 1753743888.83 total 1062.43 last 61.79) - open/close 63506 (time 1753743898.85 total 1072.44 last 62.02) - open/close 64130 (time 1753743908.85 total 1082.44 last 62.38) - open/close 64698 (time 1753743918.86 total 1092.45 last 56.77) total: 64768 open/close in 1093.68 seconds: 59.22 ops/second - /unlink 549 (time 1753743934.51 total 10.01 last 54.82) - /unlink 1134 (time 1753743944.51 total 20.02 last 58.48) - /unlink 1735 (time 1753743954.51 total 30.02 last 60.08) - /unlink 2369 (time 1753743964.52 total 40.03 last 63.36) - /unlink 3001 (time 1753743974.53 total 50.03 last 63.15) - /unlink 3600 (time 1753743984.54 total 60.05 last 59.80) - /unlink 4254 (time 1753743994.54 total 70.05 last 65.40) - /unlink 4873 (time 1753744004.56 total 80.07 last 61.81) - /unlink 5468 (time 1753744014.58 total 90.08 last 59.39) - /unlink 6053 (time 1753744024.61 total 100.12 last 58.30) - /unlink 6617 (time 1753744034.61 total 110.12 last 56.39) - /unlink 7213 (time 1753744044.61 total 120.12 last 59.60) - /unlink 7809 (time 1753744054.62 total 130.13 last 59.57) - /unlink 8454 (time 1753744064.62 total 140.13 last 64.47) - /unlink 9081 (time 1753744074.63 total 150.14 last 62.64) - /unlink 9704 (time 1753744084.64 total 160.15 last 62.25) - /unlink 10000 (time 1753744088.99 total 164.50 last 68.03) - /unlink 10666 (time 1753744098.99 total 174.50 last 66.58) - /unlink 11309 (time 1753744109.01 total 184.52 last 64.21) - /unlink 11935 (time 1753744119.03 total 194.54 last 62.47) - /unlink 12601 (time 1753744129.04 total 204.55 last 66.53) - /unlink 13196 (time 1753744139.05 total 214.56 last 59.41) - /unlink 13815 (time 1753744149.06 total 224.57 last 61.83) - /unlink 14389 (time 1753744159.08 total 234.59 last 57.32) - /unlink 14999 (time 1753744169.10 total 244.61 last 60.87) - /unlink 15655 (time 1753744179.10 total 254.61 last 65.59) - /unlink 16284 (time 1753744189.10 total 264.61 last 62.90) - /unlink 16909 (time 1753744199.10 total 274.61 last 62.48) - /unlink 17497 (time 1753744209.10 total 284.61 last 58.80) - /unlink 18055 (time 1753744219.12 total 294.62 last 55.74) - /unlink 18668 (time 1753744229.13 total 304.64 last 61.22) - /unlink 19329 (time 1753744239.13 total 314.64 last 66.08) - /unlink 19999 (time 1753744249.14 total 324.65 last 66.96) - /unlink 20621 (time 1753744259.14 total 334.65 last 62.16) - /unlink 21212 (time 1753744269.16 total 344.67 last 59.02) - /unlink 21842 (time 1753744279.17 total 354.67 last 62.95) - /unlink 22527 (time 1753744289.19 total 364.69 last 68.37) - /unlink 23211 (time 1753744299.19 total 374.70 last 68.34) - /unlink 23878 (time 1753744309.20 total 384.71 last 66.66) - /unlink 24553 (time 1753744319.22 total 394.73 last 67.37) - /unlink 25244 (time 1753744329.23 total 404.73 last 69.06) - /unlink 25877 (time 1753744339.23 total 414.74 last 63.25) - /unlink 26555 (time 1753744349.24 total 424.75 last 67.77) - /unlink 27233 (time 1753744359.25 total 434.75 last 67.74) - /unlink 27910 (time 1753744369.26 total 444.77 last 67.61) - /unlink 28560 (time 1753744379.26 total 454.77 last 64.99) - /unlink 29196 (time 1753744389.27 total 464.78 last 63.52) - /unlink 30000 (time 1753744399.24 total 474.75 last 80.65) - /unlink 30758 (time 1753744409.25 total 484.76 last 75.71) - /unlink 31437 (time 1753744419.27 total 494.77 last 67.82) - /unlink 32116 (time 1753744429.27 total 504.78 last 67.85) - /unlink 32839 (time 1753744439.28 total 514.79 last 72.23) - /unlink 33461 (time 1753744449.29 total 524.80 last 62.15) - /unlink 34167 (time 1753744459.30 total 534.81 last 70.56) - /unlink 34890 (time 1753744469.31 total 544.82 last 72.20) - /unlink 35523 (time 1753744479.33 total 554.83 last 63.21) - /unlink 36141 (time 1753744489.34 total 564.85 last 61.71) - /unlink 36760 (time 1753744499.34 total 574.85 last 61.88) - /unlink 37417 (time 1753744509.37 total 584.88 last 65.53) - /unlink 38007 (time 1753744519.39 total 594.89 last 58.90) - /unlink 38660 (time 1753744529.39 total 604.90 last 65.25) - /unlink 39300 (time 1753744539.41 total 614.91 last 63.93) - /unlink 39943 (time 1753744549.42 total 624.93 last 64.19) - /unlink 40595 (time 1753744559.43 total 634.94 last 65.12) - /unlink 41265 (time 1753744569.45 total 644.96 last 66.91) - /unlink 41968 (time 1753744579.47 total 654.98 last 70.17) - /unlink 42649 (time 1753744589.48 total 664.98 last 68.05) - /unlink 43339 (time 1753744599.49 total 675.00 last 68.89) - /unlink 44049 (time 1753744609.49 total 685.00 last 70.99) - /unlink 44739 (time 1753744619.50 total 695.01 last 68.95) - /unlink 45483 (time 1753744629.51 total 705.02 last 74.31) - /unlink 46103 (time 1753744639.52 total 715.02 last 61.98) - /unlink 46792 (time 1753744649.52 total 725.02 last 68.90) - /unlink 47504 (time 1753744659.52 total 735.03 last 71.17) - /unlink 48207 (time 1753744669.54 total 745.05 last 70.14) - /unlink 48932 (time 1753744679.55 total 755.06 last 72.46) - /unlink 49706 (time 1753744689.55 total 765.06 last 77.38) - /unlink 50000 (time 1753744693.18 total 768.69 last 80.99) - /unlink 50850 (time 1753744703.19 total 778.70 last 84.93) - /unlink 51610 (time 1753744713.19 total 788.70 last 75.95) - /unlink 52397 (time 1753744723.20 total 798.71 last 78.66) - /unlink 53171 (time 1753744733.20 total 808.71 last 77.37) - /unlink 53869 (time 1753744743.21 total 818.72 last 69.77) - /unlink 54568 (time 1753744753.21 total 828.72 last 69.90) - /unlink 55269 (time 1753744763.23 total 838.73 last 69.99) - /unlink 56036 (time 1753744773.23 total 848.73 last 76.70) - /unlink 56843 (time 1753744783.23 total 858.74 last 80.65) - /unlink 57624 (time 1753744793.23 total 868.74 last 78.10) - /unlink 58394 (time 1753744803.24 total 878.75 last 76.96) - /unlink 59118 (time 1753744813.25 total 888.76 last 72.30) - /unlink 59839 (time 1753744823.26 total 898.77 last 72.04) - /unlink 60000 (time 1753744825.45 total 900.96 last 73.56) - /unlink 60715 (time 1753744835.45 total 910.96 last 71.48) - /unlink 61473 (time 1753744845.45 total 920.96 last 75.77) - /unlink 62199 (time 1753744855.46 total 930.96 last 72.59) - /unlink 62924 (time 1753744865.46 total 940.97 last 72.44) - /unlink 63637 (time 1753744875.48 total 950.99 last 71.20) - /unlink 64343 (time 1753744885.49 total 961.00 last 70.54) total: 64768 /unlink in 967.19 seconds: 66.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 682 (time 1753744928.76 total 10.00 last 68.20) - open/close 1378 (time 1753744938.77 total 20.01 last 69.51) - open/close 2121 (time 1753744948.78 total 30.02 last 74.28) - open/close 2903 (time 1753744958.79 total 40.03 last 78.10) - open/close 3616 (time 1753744968.79 total 50.03 last 71.29) - open/close 4323 (time 1753744978.82 total 60.06 last 70.48) - open/close 5046 (time 1753744988.82 total 70.06 last 72.29) - open/close 5730 (time 1753744998.83 total 80.07 last 68.35) - open/close 6379 (time 1753745008.84 total 90.08 last 64.86) - open/close 7014 (time 1753745018.84 total 100.08 last 63.47) - open/close 7660 (time 1753745028.85 total 110.09 last 64.54) - open/close 8302 (time 1753745038.86 total 120.10 last 64.16) - open/close 8908 (time 1753745048.88 total 130.12 last 60.46) - open/close 9592 (time 1753745058.88 total 140.12 last 68.40) - open/close 10000 (time 1753745064.75 total 145.99 last 69.51) - open/close 10677 (time 1753745074.76 total 156.00 last 67.66) - open/close 11321 (time 1753745084.77 total 166.01 last 64.31) - open/close 11964 (time 1753745094.77 total 176.01 last 64.28) - open/close 12677 (time 1753745104.78 total 186.02 last 71.27) - open/close 13375 (time 1753745114.79 total 196.03 last 69.74) - open/close 14036 (time 1753745124.79 total 206.03 last 66.07) - open/close 14722 (time 1753745134.80 total 216.04 last 68.57) - open/close 15361 (time 1753745144.80 total 226.04 last 63.86) - open/close 15984 (time 1753745154.81 total 236.05 last 62.27) - open/close 16645 (time 1753745164.82 total 246.06 last 66.05) - open/close 17361 (time 1753745174.82 total 256.06 last 71.55) - open/close 18086 (time 1753745184.83 total 266.07 last 72.46) - open/close 18760 (time 1753745194.83 total 276.07 last 67.36) - open/close 19407 (time 1753745204.84 total 286.08 last 64.67) - open/close 20000 (time 1753745213.10 total 294.34 last 71.75) - open/close 20666 (time 1753745223.11 total 304.35 last 66.59) - open/close 21375 (time 1753745233.12 total 314.36 last 70.78) - open/close 22108 (time 1753745243.14 total 324.38 last 73.21) - open/close 22791 (time 1753745253.15 total 334.39 last 68.23) - open/close 23413 (time 1753745263.17 total 344.41 last 62.06) - open/close 24032 (time 1753745273.17 total 354.41 last 61.86) - open/close 24702 (time 1753745283.19 total 364.43 last 66.93) - open/close 25348 (time 1753745293.19 total 374.43 last 64.58) - open/close 26014 (time 1753745303.19 total 384.43 last 66.58) - open/close 26720 (time 1753745313.20 total 394.44 last 70.53) - open/close 27406 (time 1753745323.21 total 404.45 last 68.57) - open/close 28103 (time 1753745333.21 total 414.45 last 69.66) - open/close 28787 (time 1753745343.21 total 424.45 last 68.39) - open/close 29485 (time 1753745353.22 total 434.46 last 69.73) - open/close 30000 (time 1753745361.02 total 442.26 last 66.05) - open/close 30628 (time 1753745371.03 total 452.27 last 62.72) - open/close 31296 (time 1753745381.04 total 462.28 last 66.76) - open/close 31974 (time 1753745391.05 total 472.29 last 67.70) - open/close 32590 (time 1753745401.05 total 482.29 last 61.59) - open/close 33206 (time 1753745411.05 total 492.30 last 61.60) - open/close 33857 (time 1753745421.06 total 502.30 last 65.05) - open/close 34540 (time 1753745431.07 total 512.31 last 68.27) - open/close 35203 (time 1753745441.08 total 522.32 last 66.23) - open/close 35858 (time 1753745451.09 total 532.33 last 65.42) - open/close 36502 (time 1753745461.09 total 542.33 last 64.37) - open/close 37138 (time 1753745471.10 total 552.34 last 63.58) - open/close 37868 (time 1753745481.11 total 562.35 last 72.93) - open/close 38587 (time 1753745491.11 total 572.35 last 71.87) - open/close 39202 (time 1753745501.11 total 582.35 last 61.48) - open/close 39882 (time 1753745511.12 total 592.36 last 67.97) - open/close 40514 (time 1753745521.13 total 602.37 last 63.11) - open/close 41170 (time 1753745531.14 total 612.38 last 65.54) - open/close 41828 (time 1753745541.15 total 622.39 last 65.74) - open/close 42500 (time 1753745551.15 total 632.39 last 67.18) - open/close 43160 (time 1753745561.16 total 642.40 last 65.93) - open/close 43813 (time 1753745571.16 total 652.40 last 65.28) - open/close 44418 (time 1753745581.17 total 662.41 last 60.48) - open/close 45062 (time 1753745591.17 total 672.41 last 64.36) - open/close 45735 (time 1753745601.18 total 682.42 last 67.26) - open/close 46429 (time 1753745611.18 total 692.42 last 69.37) - open/close 47106 (time 1753745621.19 total 702.43 last 67.67) - open/close 47783 (time 1753745631.19 total 712.43 last 67.66) - open/close 48447 (time 1753745641.20 total 722.44 last 66.38) - open/close 49139 (time 1753745651.21 total 732.45 last 69.10) - open/close 49740 (time 1753745661.22 total 742.46 last 60.03) - open/close 50000 (time 1753745665.06 total 746.30 last 67.73) - open/close 50683 (time 1753745675.07 total 756.31 last 68.26) - open/close 51388 (time 1753745685.08 total 766.32 last 70.44) - open/close 52091 (time 1753745695.08 total 776.32 last 70.30) - open/close 52794 (time 1753745705.09 total 786.33 last 70.21) - open/close 53576 (time 1753745715.12 total 796.36 last 77.99) - open/close 54326 (time 1753745725.13 total 806.37 last 74.92) - open/close 55059 (time 1753745735.14 total 816.38 last 73.24) - open/close 55788 (time 1753745745.14 total 826.38 last 72.88) - open/close 56464 (time 1753745755.15 total 836.39 last 67.56) - open/close 57222 (time 1753745765.15 total 846.39 last 75.79) - open/close 57995 (time 1753745775.15 total 856.39 last 77.30) - open/close 58730 (time 1753745785.15 total 866.39 last 73.45) - open/close 59496 (time 1753745795.17 total 876.41 last 76.51) - open/close 60000 (time 1753745802.65 total 883.89 last 67.30) - open/close 60663 (time 1753745812.66 total 893.90 last 66.26) - open/close 61329 (time 1753745822.67 total 903.91 last 66.54) - open/close 62002 (time 1753745832.68 total 913.92 last 67.27) - open/close 62631 (time 1753745842.68 total 923.92 last 62.86) - open/close 63318 (time 1753745852.69 total 933.93 last 68.64) - open/close 63993 (time 1753745862.70 total 943.94 last 67.45) - open/close 64691 (time 1753745872.72 total 953.96 last 69.67) total: 64768 open/close in 955.25 seconds: 67.80 ops/second - /unlink 650 (time 1753745887.49 total 10.00 last 64.97) - /unlink 1245 (time 1753745897.51 total 20.02 last 59.41) - /unlink 1931 (time 1753745907.51 total 30.03 last 68.56) - /unlink 2586 (time 1753745917.53 total 40.04 last 65.42) - /unlink 3242 (time 1753745927.57 total 50.08 last 65.32) - /unlink 3965 (time 1753745937.57 total 60.09 last 72.26) - /unlink 4670 (time 1753745947.58 total 70.09 last 70.48) - /unlink 5423 (time 1753745957.60 total 80.11 last 75.11) - /unlink 6315 (time 1753745967.62 total 90.13 last 89.05) - /unlink 7116 (time 1753745977.62 total 100.13 last 80.09) - /unlink 7918 (time 1753745987.63 total 110.14 last 80.16) - /unlink 8684 (time 1753745997.63 total 120.14 last 76.56) - /unlink 9439 (time 1753746007.63 total 130.15 last 75.47) - /unlink 10000 (time 1753746015.28 total 137.80 last 73.34) - /unlink 10752 (time 1753746025.30 total 147.81 last 75.08) - /unlink 11465 (time 1753746035.32 total 157.83 last 71.16) - /unlink 12260 (time 1753746045.33 total 167.84 last 79.40) - /unlink 13060 (time 1753746055.33 total 177.85 last 79.99) - /unlink 13868 (time 1753746065.35 total 187.86 last 80.69) - /unlink 14786 (time 1753746075.35 total 197.86 last 91.76) - /unlink 15641 (time 1753746085.36 total 207.87 last 85.44) - /unlink 16512 (time 1753746095.36 total 217.88 last 87.05) - /unlink 17298 (time 1753746105.37 total 227.88 last 78.58) - /unlink 18070 (time 1753746115.37 total 237.88 last 77.15) - /unlink 18828 (time 1753746125.38 total 247.90 last 75.72) - /unlink 19615 (time 1753746135.40 total 257.91 last 78.59) - /unlink 20000 (time 1753746140.30 total 262.82 last 78.46) - /unlink 20763 (time 1753746150.32 total 272.83 last 76.21) - /unlink 21592 (time 1753746160.32 total 282.84 last 82.83) - /unlink 22337 (time 1753746170.33 total 292.84 last 74.44) - /unlink 23091 (time 1753746180.34 total 302.85 last 75.32) - /unlink 23873 (time 1753746190.34 total 312.86 last 78.19) - /unlink 24808 (time 1753746200.35 total 322.86 last 93.42) - /unlink 25637 (time 1753746210.36 total 332.87 last 82.82) - /unlink 26438 (time 1753746220.38 total 342.90 last 79.93) - /unlink 27341 (time 1753746230.39 total 352.90 last 90.24) - /unlink 28205 (time 1753746240.40 total 362.91 last 86.33) - /unlink 29040 (time 1753746250.42 total 372.93 last 83.31) - /unlink 29845 (time 1753746260.43 total 382.94 last 80.47) - /unlink 30720 (time 1753746270.44 total 392.95 last 87.41) - /unlink 31620 (time 1753746280.45 total 402.96 last 89.88) - /unlink 32575 (time 1753746290.45 total 412.96 last 95.49) - /unlink 33658 (time 1753746300.45 total 422.96 last 108.29) - /unlink 34648 (time 1753746310.46 total 432.97 last 98.91) - /unlink 35626 (time 1753746320.48 total 442.99 last 97.63) - /unlink 36693 (time 1753746330.48 total 452.99 last 106.68) - /unlink 37638 (time 1753746340.49 total 463.00 last 94.44) - /unlink 38593 (time 1753746350.50 total 473.01 last 95.39) - /unlink 39489 (time 1753746360.50 total 483.01 last 89.57) - /unlink 40000 (time 1753746365.91 total 488.42 last 94.44) - /unlink 40906 (time 1753746375.92 total 498.43 last 90.51) - /unlink 41890 (time 1753746385.93 total 508.44 last 98.29) - /unlink 42858 (time 1753746395.94 total 518.46 last 96.69) - /unlink 43779 (time 1753746405.95 total 528.46 last 92.06) - /unlink 44833 (time 1753746415.96 total 538.47 last 105.27) - /unlink 45719 (time 1753746425.96 total 548.47 last 88.60) - /unlink 46836 (time 1753746435.96 total 558.47 last 111.70) - /unlink 47864 (time 1753746445.97 total 568.48 last 102.73) - /unlink 48852 (time 1753746455.98 total 578.49 last 98.71) - /unlink 49663 (time 1753746465.98 total 588.49 last 81.08) - /unlink 50000 (time 1753746470.29 total 592.80 last 78.19) - /unlink 50809 (time 1753746480.30 total 602.81 last 80.84) - /unlink 51695 (time 1753746490.30 total 612.81 last 88.59) - /unlink 52725 (time 1753746500.31 total 622.82 last 102.92) - /unlink 53621 (time 1753746510.31 total 632.82 last 89.56) - /unlink 54479 (time 1753746520.31 total 642.82 last 85.78) - /unlink 55384 (time 1753746530.32 total 652.83 last 90.46) - /unlink 56308 (time 1753746540.32 total 662.83 last 92.39) - /unlink 57303 (time 1753746550.32 total 672.84 last 99.43) - /unlink 58419 (time 1753746560.33 total 682.84 last 111.58) - /unlink 59283 (time 1753746570.33 total 692.84 last 86.36) - /unlink 60000 (time 1753746578.18 total 700.69 last 91.40) - /unlink 60820 (time 1753746588.18 total 710.69 last 82.00) - /unlink 61744 (time 1753746598.18 total 720.69 last 92.40) - /unlink 62708 (time 1753746608.18 total 730.69 last 96.36) - /unlink 63710 (time 1753746618.18 total 740.70 last 100.17) - /unlink 64590 (time 1753746628.19 total 750.70 last 87.97) total: 64768 /unlink in 752.75 seconds: 86.04 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1132 (time 1753746660.41 total 10.01 last 113.14) - open/close 1983 (time 1753746670.42 total 20.01 last 85.02) - open/close 2986 (time 1753746680.43 total 30.02 last 100.26) - open/close 4169 (time 1753746690.44 total 40.03 last 118.18) - open/close 5101 (time 1753746700.44 total 50.04 last 93.13) - open/close 6026 (time 1753746710.45 total 60.04 last 92.48) - open/close 7034 (time 1753746720.45 total 70.04 last 100.75) - open/close 8102 (time 1753746730.46 total 80.05 last 106.73) - open/close 9176 (time 1753746740.47 total 90.06 last 107.28) - open/close 10000 (time 1753746749.02 total 98.62 last 96.32) - open/close 11199 (time 1753746759.03 total 108.62 last 119.84) - open/close 12246 (time 1753746769.04 total 118.63 last 104.63) - open/close 13182 (time 1753746779.04 total 128.63 last 93.54) - open/close 14114 (time 1753746789.04 total 138.64 last 93.18) - open/close 14922 (time 1753746799.05 total 148.64 last 80.75) - open/close 15749 (time 1753746809.05 total 158.64 last 82.69) - open/close 16593 (time 1753746819.06 total 168.65 last 84.33) - open/close 17406 (time 1753746829.06 total 178.66 last 81.26) - open/close 18153 (time 1753746839.08 total 188.67 last 74.60) - open/close 18931 (time 1753746849.08 total 198.68 last 77.76) - open/close 19790 (time 1753746859.09 total 208.68 last 85.85) - open/close 20000 (time 1753746861.18 total 210.78 last 100.24) - open/close 20943 (time 1753746871.19 total 220.78 last 94.28) - open/close 21915 (time 1753746881.19 total 230.78 last 97.16) - open/close 23170 (time 1753746891.19 total 240.78 last 125.46) - open/close 24185 (time 1753746901.20 total 250.79 last 101.43) - open/close 25135 (time 1753746911.20 total 260.79 last 94.98) - open/close 26050 (time 1753746921.21 total 270.80 last 91.44) - open/close 26949 (time 1753746931.22 total 280.81 last 89.80) - open/close 28007 (time 1753746941.22 total 290.81 last 105.77) - open/close 29261 (time 1753746951.23 total 300.82 last 125.29) - open/close 30000 (time 1753746957.68 total 307.28 last 114.54) - open/close 31068 (time 1753746967.69 total 317.28 last 106.75) - open/close 32271 (time 1753746977.69 total 327.28 last 120.25) - open/close 33677 (time 1753746987.69 total 337.29 last 140.55) - open/close 34835 (time 1753746997.70 total 347.29 last 115.72) - open/close 35972 (time 1753747007.71 total 357.30 last 113.63) - open/close 37111 (time 1753747017.72 total 367.31 last 113.80) - open/close 38206 (time 1753747027.72 total 377.31 last 109.49) - open/close 39413 (time 1753747037.72 total 387.31 last 120.69) - open/close 40000 (time 1753747042.68 total 392.27 last 118.35) - open/close 41012 (time 1753747052.68 total 402.27 last 101.15) - open/close 42231 (time 1753747062.68 total 412.27 last 121.89) - open/close 43488 (time 1753747072.69 total 422.28 last 125.63) - open/close 44556 (time 1753747082.69 total 432.29 last 106.74) - open/close 46028 (time 1753747092.69 total 442.29 last 147.17) - open/close 47647 (time 1753747102.70 total 452.29 last 161.83) - open/close 48893 (time 1753747112.70 total 462.29 last 124.57) - open/close 50000 (time 1753747120.76 total 470.36 last 137.31) - open/close 51546 (time 1753747130.77 total 480.37 last 154.44) - open/close 52757 (time 1753747140.78 total 490.37 last 121.01) - open/close 53902 (time 1753747150.79 total 500.38 last 114.42) - open/close 55248 (time 1753747160.79 total 510.38 last 134.55) - open/close 56430 (time 1753747170.80 total 520.39 last 118.12) - open/close 57429 (time 1753747180.80 total 530.40 last 99.86) - open/close 58554 (time 1753747190.81 total 540.40 last 112.45) - open/close 59655 (time 1753747200.81 total 550.40 last 110.07) - open/close 60000 (time 1753747204.18 total 553.77 last 102.33) - open/close 61208 (time 1753747214.19 total 563.78 last 120.71) - open/close 62413 (time 1753747224.19 total 573.78 last 120.50) - open/close 63462 (time 1753747234.19 total 583.78 last 104.89) - open/close 64593 (time 1753747244.20 total 593.79 last 113.03) total: 64768 open/close in 595.49 seconds: 108.77 ops/second - /unlink 2311 (time 1753747257.99 total 10.00 last 231.09) - /unlink 4303 (time 1753747268.00 total 20.00 last 199.16) - /unlink 6673 (time 1753747278.00 total 30.00 last 236.98) - /unlink 8596 (time 1753747288.00 total 40.01 last 192.24) - /unlink 10000 (time 1753747294.71 total 46.71 last 209.34) - /unlink 12344 (time 1753747304.71 total 56.71 last 234.38) - /unlink 14448 (time 1753747314.71 total 66.72 last 210.29) - /unlink 16548 (time 1753747324.71 total 76.72 last 210.00) - /unlink 18861 (time 1753747334.71 total 86.72 last 231.29) - /unlink 20000 (time 1753747341.05 total 93.05 last 179.86) - /unlink 22060 (time 1753747351.05 total 103.05 last 205.99) - /unlink 24045 (time 1753747361.05 total 113.05 last 198.47) - /unlink 26076 (time 1753747371.05 total 123.06 last 203.01) - /unlink 28230 (time 1753747381.05 total 133.06 last 215.40) - /unlink 30000 (time 1753747388.58 total 140.59 last 235.11) - /unlink 32427 (time 1753747398.58 total 150.59 last 242.65) - /unlink 34859 (time 1753747408.59 total 160.59 last 243.13) - /unlink 37574 (time 1753747418.59 total 170.59 last 271.44) - /unlink 39811 (time 1753747428.59 total 180.59 last 223.67) - /unlink 42438 (time 1753747438.59 total 190.60 last 262.66) - /unlink 45074 (time 1753747448.59 total 200.60 last 263.53) - /unlink 47841 (time 1753747458.59 total 210.60 last 276.69) - /unlink 50000 (time 1753747465.99 total 217.99 last 292.03) - /unlink 53820 (time 1753747475.99 total 227.99 last 381.99) - /unlink 58080 (time 1753747485.99 total 237.99 last 425.99) - /unlink 60000 (time 1753747490.30 total 242.31 last 445.04) total: 64768 /unlink in 251.69 seconds: 257.33 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2271 (time 1753747529.88 total 10.00 last 227.08) - open/close 3880 (time 1753747539.88 total 20.00 last 160.84) - open/close 5409 (time 1753747549.88 total 30.00 last 152.90) - open/close 7092 (time 1753747559.88 total 40.01 last 168.27) - open/close 8762 (time 1753747569.88 total 50.01 last 166.99) - open/close 10000 (time 1753747574.78 total 54.90 last 252.78) - open/close 12618 (time 1753747584.78 total 64.91 last 261.72) - open/close 14515 (time 1753747594.79 total 74.91 last 189.64) - open/close 16688 (time 1753747604.79 total 84.91 last 217.23) - open/close 19850 (time 1753747614.79 total 94.92 last 316.15) - open/close 22987 (time 1753747624.79 total 104.92 last 313.63) - open/close 26506 (time 1753747634.80 total 114.92 last 351.85) - open/close 30000 (time 1753747644.78 total 124.91 last 349.83) - open/close 33308 (time 1753747654.78 total 134.91 last 330.75) - open/close 35973 (time 1753747664.79 total 144.92 last 266.27) - open/close 39043 (time 1753747674.79 total 154.92 last 306.99) - open/close 40000 (time 1753747678.71 total 158.83 last 244.55) - open/close 43031 (time 1753747688.71 total 168.83 last 303.00) - open/close 45750 (time 1753747698.71 total 178.84 last 271.84) - open/close 48372 (time 1753747708.71 total 188.84 last 262.18) - open/close 50000 (time 1753747715.18 total 195.31 last 251.61) - open/close 51849 (time 1753747725.19 total 205.31 last 184.86) - open/close 54349 (time 1753747735.19 total 215.31 last 249.94) - open/close 56235 (time 1753747745.26 total 225.38 last 187.32) - open/close 58111 (time 1753747755.26 total 235.38 last 187.54) - open/close 60000 (time 1753747764.21 total 244.33 last 211.11) - open/close 61457 (time 1753747774.21 total 254.34 last 145.60) - open/close 62917 (time 1753747784.22 total 264.34 last 145.94) - open/close 64565 (time 1753747794.22 total 274.35 last 164.71) total: 64768 open/close in 275.78 seconds: 234.85 ops/second - /unlink 2623 (time 1753747807.76 total 10.00 last 262.24) - /unlink 5985 (time 1753747817.76 total 20.00 last 336.19) - /unlink 9295 (time 1753747827.76 total 30.00 last 330.95) - /unlink 12114 (time 1753747837.77 total 40.01 last 281.87) - /unlink 15316 (time 1753747847.77 total 50.01 last 320.19) - /unlink 18759 (time 1753747857.77 total 60.01 last 344.22) - /unlink 20000 (time 1753747862.34 total 64.58 last 271.27) - /unlink 23257 (time 1753747872.34 total 74.58 last 325.68) - /unlink 26531 (time 1753747882.35 total 84.59 last 327.33) - /unlink 30000 (time 1753747891.53 total 93.77 last 377.65) - /unlink 33542 (time 1753747901.53 total 103.77 last 354.16) - /unlink 38044 (time 1753747911.53 total 113.77 last 450.16) - /unlink 40000 (time 1753747915.33 total 117.57 last 515.34) - /unlink 45182 (time 1753747925.33 total 127.57 last 518.20) - /unlink 50000 (time 1753747933.75 total 135.99 last 571.94) - /unlink 55553 (time 1753747943.76 total 145.99 last 555.22) - /unlink 60000 (time 1753747952.06 total 154.30 last 535.65) - /unlink 64317 (time 1753747962.06 total 164.30 last 431.66) total: 64768 /unlink in 165.42 seconds: 391.53 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1747 (time 1753747994.28 total 10.01 last 174.58) - open/close 3754 (time 1753748004.29 total 20.02 last 200.43) - open/close 5728 (time 1753748014.29 total 30.02 last 197.36) - open/close 7831 (time 1753748024.29 total 40.03 last 210.23) - open/close 10000 (time 1753748031.03 total 46.76 last 322.26) - open/close 13613 (time 1753748041.03 total 56.76 last 361.27) - open/close 17056 (time 1753748051.03 total 66.76 last 344.28) - open/close 20000 (time 1753748060.57 total 76.30 last 308.49) - open/close 22869 (time 1753748070.57 total 86.31 last 286.77) - open/close 26416 (time 1753748080.58 total 96.31 last 354.59) - open/close 30000 (time 1753748089.96 total 105.69 last 382.01) - open/close 34486 (time 1753748099.96 total 115.69 last 448.54) - open/close 39067 (time 1753748109.96 total 125.69 last 458.07) - open/close 43510 (time 1753748119.96 total 135.69 last 444.22) - open/close 47752 (time 1753748129.96 total 145.70 last 424.18) - open/close 50000 (time 1753748135.15 total 150.89 last 433.08) - open/close 54835 (time 1753748145.16 total 160.89 last 483.41) - open/close 59614 (time 1753748155.16 total 170.89 last 477.82) - open/close 64153 (time 1753748165.16 total 180.89 last 453.87) total: 64768 open/close in 182.64 seconds: 354.62 ops/second - /unlink 7464 (time 1753748178.24 total 10.00 last 746.33) - /unlink 10000 (time 1753748181.23 total 12.99 last 848.20) - /unlink 16892 (time 1753748191.23 total 22.99 last 689.16) - /unlink 20000 (time 1753748196.06 total 27.82 last 644.31) - /unlink 26974 (time 1753748206.06 total 37.82 last 697.01) - /unlink 30000 (time 1753748210.27 total 42.02 last 719.86) - /unlink 37873 (time 1753748220.27 total 52.03 last 787.21) - /unlink 40000 (time 1753748222.63 total 54.39 last 900.03) - /unlink 47604 (time 1753748232.63 total 64.39 last 760.38) - /unlink 50000 (time 1753748235.96 total 67.72 last 720.23) - /unlink 57587 (time 1753748245.96 total 77.72 last 758.69) - /unlink 60000 (time 1753748249.13 total 80.89 last 760.77) total: 64768 /unlink in 87.15 seconds: 743.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg259-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server stop mds service on oleg259-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg259-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing unload_modules_local oleg259-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg259-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2