== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 02:31:35 (1771399895) Stopping clients: oleg211-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg211-client.virtnet /mnt/lustre2 (opts:-f) oleg211-server: oleg211-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing load_modules_local oleg211-server: Loading modules from /home/green/git/lustre-release/lustre oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg211-server: detected 4 online CPUs by sysfs oleg211-server: MODOPTS_LIBCFS= oleg211-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg211-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg211-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 869 (time 1771400037.08 total 10.00 last 86.86) - open/close 1612 (time 1771400047.09 total 20.01 last 74.24) - open/close 2237 (time 1771400057.11 total 30.03 last 62.38) - open/close 2725 (time 1771400067.13 total 40.06 last 48.69) - open/close 3423 (time 1771400077.14 total 50.06 last 69.74) - open/close 4051 (time 1771400087.14 total 60.07 last 62.79) - open/close 4771 (time 1771400097.15 total 70.07 last 71.95) - open/close 5446 (time 1771400107.16 total 80.09 last 67.41) - open/close 6343 (time 1771400117.16 total 90.09 last 89.68) - open/close 7247 (time 1771400127.17 total 100.09 last 90.34) - open/close 8037 (time 1771400137.17 total 110.10 last 78.99) - open/close 8833 (time 1771400147.18 total 120.10 last 79.54) - open/close 9718 (time 1771400157.18 total 130.11 last 88.48) - open/close 10000 (time 1771400160.43 total 133.35 last 86.94) - open/close 10900 (time 1771400170.43 total 143.36 last 89.93) - open/close 11790 (time 1771400180.44 total 153.36 last 88.95) - open/close 12650 (time 1771400190.46 total 163.38 last 85.85) - open/close 13215 (time 1771400200.46 total 173.38 last 56.49) - open/close 13958 (time 1771400210.48 total 183.40 last 74.18) - open/close 14700 (time 1771400220.49 total 193.41 last 74.08) - open/close 15452 (time 1771400230.49 total 203.41 last 75.20) - open/close 16047 (time 1771400240.49 total 213.42 last 59.48) - open/close 16777 (time 1771400250.50 total 223.42 last 72.98) - open/close 17496 (time 1771400260.50 total 233.42 last 71.87) - open/close 18232 (time 1771400270.51 total 243.43 last 73.58) - open/close 19044 (time 1771400280.52 total 253.44 last 81.10) - open/close 19802 (time 1771400290.53 total 263.45 last 75.70) - open/close 20000 (time 1771400293.41 total 266.34 last 68.68) - open/close 20783 (time 1771400303.42 total 276.34 last 78.26) - open/close 21476 (time 1771400313.43 total 286.35 last 69.20) - open/close 22224 (time 1771400323.45 total 296.37 last 74.68) - open/close 22912 (time 1771400333.45 total 306.38 last 68.77) - open/close 23534 (time 1771400343.46 total 316.38 last 62.17) - open/close 24339 (time 1771400353.46 total 326.39 last 80.44) - open/close 25097 (time 1771400363.50 total 336.42 last 75.55) - open/close 25839 (time 1771400373.51 total 346.43 last 74.12) - open/close 26640 (time 1771400383.51 total 356.43 last 80.09) - open/close 27273 (time 1771400393.51 total 366.43 last 63.30) - open/close 28055 (time 1771400403.52 total 376.44 last 78.12) - open/close 28720 (time 1771400413.53 total 386.45 last 66.46) - open/close 29468 (time 1771400423.53 total 396.45 last 74.77) - open/close 30000 (time 1771400430.61 total 403.54 last 75.13) - open/close 30775 (time 1771400440.63 total 413.56 last 77.35) - open/close 31398 (time 1771400450.65 total 423.57 last 62.20) - open/close 32039 (time 1771400460.65 total 433.58 last 64.07) - open/close 32775 (time 1771400470.65 total 443.58 last 73.59) - open/close 33375 (time 1771400480.66 total 453.58 last 59.96) - open/close 33930 (time 1771400490.67 total 463.59 last 55.47) - open/close 34466 (time 1771400500.67 total 473.59 last 53.58) - open/close 35151 (time 1771400510.68 total 483.60 last 68.42) - open/close 35903 (time 1771400520.69 total 493.61 last 75.15) - open/close 36545 (time 1771400530.71 total 503.63 last 64.07) - open/close 37210 (time 1771400540.72 total 513.65 last 66.41) - open/close 37939 (time 1771400550.72 total 523.65 last 72.90) - open/close 38655 (time 1771400560.73 total 533.65 last 71.57) - open/close 39435 (time 1771400570.73 total 543.65 last 77.97) - open/close 40000 (time 1771400579.48 total 552.40 last 64.58) - open/close 40730 (time 1771400589.49 total 562.41 last 72.94) - open/close 41599 (time 1771400599.49 total 572.41 last 86.89) - open/close 42474 (time 1771400609.49 total 582.42 last 87.45) - open/close 43325 (time 1771400619.51 total 592.43 last 84.99) - open/close 44166 (time 1771400629.52 total 602.45 last 83.97) - open/close 45046 (time 1771400639.52 total 612.45 last 87.99) - open/close 45870 (time 1771400649.53 total 622.45 last 82.34) - open/close 46541 (time 1771400659.54 total 632.47 last 67.01) - open/close 47062 (time 1771400669.55 total 642.47 last 52.06) - open/close 47488 (time 1771400679.55 total 652.48 last 42.60) - open/close 48221 (time 1771400689.57 total 662.49 last 73.19) - open/close 49005 (time 1771400699.57 total 672.49 last 78.36) - open/close 49665 (time 1771400709.58 total 682.50 last 65.97) - open/close 50000 (time 1771400714.76 total 687.69 last 64.59) - open/close 50618 (time 1771400724.77 total 697.69 last 61.77) - open/close 51294 (time 1771400734.78 total 707.70 last 67.54) - open/close 52044 (time 1771400744.78 total 717.71 last 74.95) - open/close 52827 (time 1771400754.79 total 727.71 last 78.27) - open/close 53569 (time 1771400764.79 total 737.72 last 74.16) - open/close 54230 (time 1771400774.80 total 747.73 last 66.02) - open/close 54986 (time 1771400784.82 total 757.74 last 75.50) - open/close 55736 (time 1771400794.82 total 767.74 last 74.98) - open/close 56551 (time 1771400804.83 total 777.75 last 81.46) - open/close 57358 (time 1771400814.84 total 787.76 last 80.62) - open/close 58065 (time 1771400824.85 total 797.77 last 70.62) - open/close 58849 (time 1771400834.86 total 807.78 last 78.30) - open/close 59614 (time 1771400844.87 total 817.80 last 76.38) - open/close 60000 (time 1771400850.48 total 823.40 last 68.89) - open/close 60813 (time 1771400860.48 total 833.41 last 81.25) - open/close 61605 (time 1771400870.49 total 843.41 last 79.15) - open/close 62253 (time 1771400880.49 total 853.41 last 64.80) - open/close 62960 (time 1771400890.50 total 863.42 last 70.63) - open/close 63634 (time 1771400900.50 total 873.42 last 67.39) - open/close 64394 (time 1771400910.51 total 883.43 last 75.97) total: 64768 open/close in 889.46 seconds: 72.82 ops/second - /unlink 771 (time 1771400931.27 total 10.01 last 77.06) - /unlink 1430 (time 1771400941.28 total 20.01 last 65.86) - /unlink 2203 (time 1771400951.28 total 30.01 last 77.29) - /unlink 3027 (time 1771400961.29 total 40.02 last 82.36) - /unlink 3725 (time 1771400971.29 total 50.02 last 69.79) - /unlink 4323 (time 1771400981.29 total 60.02 last 59.80) - /unlink 4839 (time 1771400991.29 total 70.02 last 51.58) - /unlink 5531 (time 1771401001.32 total 80.05 last 69.04) - /unlink 6212 (time 1771401011.35 total 90.08 last 67.89) - /unlink 6873 (time 1771401021.38 total 100.11 last 65.87) - /unlink 7559 (time 1771401031.39 total 110.13 last 68.51) - /unlink 8242 (time 1771401041.40 total 120.13 last 68.29) - /unlink 8959 (time 1771401051.40 total 130.13 last 71.64) - /unlink 9764 (time 1771401061.41 total 140.14 last 80.44) - /unlink 10000 (time 1771401064.43 total 143.17 last 78.06) - /unlink 10765 (time 1771401074.44 total 153.17 last 76.43) - /unlink 11535 (time 1771401084.45 total 163.18 last 76.98) - /unlink 12278 (time 1771401094.45 total 173.18 last 74.28) - /unlink 13004 (time 1771401104.46 total 183.19 last 72.54) - /unlink 13775 (time 1771401114.47 total 193.20 last 77.01) - /unlink 14553 (time 1771401124.47 total 203.21 last 77.76) - /unlink 15159 (time 1771401134.48 total 213.21 last 60.54) - /unlink 15869 (time 1771401144.49 total 223.22 last 70.96) - /unlink 16495 (time 1771401154.51 total 233.24 last 62.45) - /unlink 17279 (time 1771401164.52 total 243.25 last 78.34) - /unlink 18009 (time 1771401174.52 total 253.26 last 72.97) - /unlink 18736 (time 1771401184.54 total 263.27 last 72.61) - /unlink 19547 (time 1771401194.55 total 273.28 last 81.02) - /unlink 20000 (time 1771401200.83 total 279.56 last 72.11) - /unlink 20668 (time 1771401210.84 total 289.57 last 66.76) - /unlink 21434 (time 1771401220.84 total 299.57 last 76.57) - /unlink 22181 (time 1771401230.85 total 309.58 last 74.62) - /unlink 22927 (time 1771401240.88 total 319.61 last 74.38) - /unlink 23649 (time 1771401250.89 total 329.62 last 72.14) - /unlink 24309 (time 1771401260.89 total 339.62 last 65.97) - /unlink 25102 (time 1771401270.90 total 349.63 last 79.23) - /unlink 25784 (time 1771401280.91 total 359.64 last 68.15) - /unlink 26531 (time 1771401290.92 total 369.65 last 74.63) - /unlink 27287 (time 1771401300.95 total 379.68 last 75.37) - /unlink 28051 (time 1771401310.97 total 389.70 last 76.26) - /unlink 28845 (time 1771401320.99 total 399.72 last 79.24) - /unlink 29516 (time 1771401330.99 total 409.72 last 67.07) - /unlink 30000 (time 1771401338.57 total 417.30 last 63.85) - /unlink 30749 (time 1771401348.57 total 427.31 last 74.88) - /unlink 31292 (time 1771401358.59 total 437.32 last 54.24) - /unlink 31893 (time 1771401368.59 total 447.32 last 60.10) - /unlink 32670 (time 1771401378.59 total 457.32 last 77.70) - /unlink 33413 (time 1771401388.60 total 467.33 last 74.23) - /unlink 34151 (time 1771401398.61 total 477.34 last 73.70) - /unlink 34834 (time 1771401408.62 total 487.35 last 68.21) - /unlink 35523 (time 1771401418.63 total 497.36 last 68.89) - /unlink 36275 (time 1771401428.66 total 507.39 last 74.96) - /unlink 36997 (time 1771401438.67 total 517.40 last 72.10) - /unlink 37720 (time 1771401448.68 total 527.41 last 72.22) - /unlink 38296 (time 1771401458.69 total 537.42 last 57.56) - /unlink 38983 (time 1771401468.70 total 547.43 last 68.62) - /unlink 39638 (time 1771401478.73 total 557.46 last 65.31) - /unlink 40000 (time 1771401484.00 total 562.73 last 68.69) - /unlink 40703 (time 1771401494.02 total 572.75 last 70.17) - /unlink 41448 (time 1771401504.03 total 582.76 last 74.43) - /unlink 42207 (time 1771401514.04 total 592.77 last 75.83) - /unlink 42954 (time 1771401524.04 total 602.77 last 74.67) - /unlink 43705 (time 1771401534.05 total 612.78 last 75.05) - /unlink 44514 (time 1771401544.06 total 622.79 last 80.77) - /unlink 45368 (time 1771401554.07 total 632.80 last 85.38) - /unlink 46152 (time 1771401564.08 total 642.81 last 78.26) - /unlink 46884 (time 1771401574.08 total 652.82 last 73.19) - /unlink 47765 (time 1771401584.09 total 662.82 last 88.07) - /unlink 48616 (time 1771401594.09 total 672.82 last 85.08) - /unlink 49451 (time 1771401604.10 total 682.83 last 83.43) - /unlink 50000 (time 1771401610.95 total 689.68 last 80.18) - /unlink 50888 (time 1771401620.95 total 699.68 last 88.78) - /unlink 51736 (time 1771401630.95 total 709.68 last 84.76) - /unlink 52683 (time 1771401640.96 total 719.69 last 94.67) - /unlink 53532 (time 1771401650.96 total 729.69 last 84.89) - /unlink 54340 (time 1771401660.96 total 739.69 last 80.78) - /unlink 55112 (time 1771401670.97 total 749.70 last 77.12) - /unlink 55940 (time 1771401680.99 total 759.72 last 82.66) - /unlink 56625 (time 1771401691.01 total 769.74 last 68.35) - /unlink 57353 (time 1771401701.01 total 779.74 last 72.77) - /unlink 58167 (time 1771401711.02 total 789.75 last 81.34) - /unlink 58927 (time 1771401721.03 total 799.76 last 75.91) - /unlink 59726 (time 1771401731.04 total 809.77 last 79.87) - /unlink 60000 (time 1771401734.26 total 812.99 last 84.94) - /unlink 60861 (time 1771401744.27 total 823.00 last 86.06) - /unlink 61619 (time 1771401754.28 total 833.01 last 75.72) - /unlink 62421 (time 1771401764.29 total 843.02 last 80.10) - /unlink 63259 (time 1771401774.29 total 853.02 last 83.77) - /unlink 64149 (time 1771401784.32 total 863.05 last 88.74) total: 64768 /unlink in 871.48 seconds: 74.32 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 734 (time 1771401830.35 total 10.01 last 73.29) - open/close 1638 (time 1771401840.35 total 20.02 last 90.38) - open/close 2592 (time 1771401850.35 total 30.02 last 95.38) - open/close 3501 (time 1771401860.36 total 40.03 last 90.82) - open/close 4353 (time 1771401870.36 total 50.03 last 85.20) - open/close 5305 (time 1771401880.37 total 60.03 last 95.16) - open/close 6210 (time 1771401890.37 total 70.03 last 90.49) - open/close 7023 (time 1771401900.38 total 80.04 last 81.22) - open/close 7986 (time 1771401910.38 total 90.05 last 96.23) - open/close 8841 (time 1771401920.39 total 100.05 last 85.47) - open/close 9753 (time 1771401930.40 total 110.07 last 91.08) - open/close 10000 (time 1771401933.19 total 112.86 last 88.48) - open/close 10831 (time 1771401943.20 total 122.86 last 83.05) - open/close 11708 (time 1771401953.22 total 132.88 last 87.56) - open/close 12492 (time 1771401963.22 total 142.88 last 78.40) - open/close 13335 (time 1771401973.22 total 152.89 last 84.25) - open/close 14094 (time 1771401983.24 total 162.90 last 75.78) - open/close 14963 (time 1771401993.24 total 172.91 last 86.88) - open/close 15822 (time 1771402003.25 total 182.91 last 85.86) - open/close 16677 (time 1771402013.25 total 192.91 last 85.50) - open/close 17438 (time 1771402023.25 total 202.92 last 76.05) - open/close 18352 (time 1771402033.27 total 212.93 last 91.26) - open/close 19176 (time 1771402043.27 total 222.94 last 82.36) - open/close 19957 (time 1771402053.28 total 232.95 last 78.03) - open/close 20879 (time 1771402063.28 total 242.95 last 92.19) - open/close 21756 (time 1771402073.29 total 252.96 last 87.59) - open/close 22471 (time 1771402083.30 total 262.96 last 71.48) - open/close 23065 (time 1771402093.32 total 272.98 last 59.29) - open/close 23857 (time 1771402103.32 total 282.98 last 79.20) - open/close 24709 (time 1771402113.33 total 292.99 last 85.13) - open/close 25505 (time 1771402123.33 total 303.00 last 79.53) - open/close 26325 (time 1771402133.35 total 313.01 last 81.90) - open/close 27091 (time 1771402143.35 total 323.02 last 76.57) - open/close 27955 (time 1771402153.36 total 333.03 last 86.27) - open/close 28772 (time 1771402163.37 total 343.03 last 81.69) - open/close 29733 (time 1771402173.40 total 353.07 last 95.77) - open/close 30000 (time 1771402176.72 total 356.39 last 80.44) - open/close 30866 (time 1771402186.72 total 366.39 last 86.57) - open/close 31718 (time 1771402196.73 total 376.39 last 85.16) - open/close 32645 (time 1771402206.73 total 386.40 last 92.63) - open/close 33566 (time 1771402216.75 total 396.41 last 92.00) - open/close 34381 (time 1771402226.78 total 406.44 last 81.26) - open/close 35088 (time 1771402236.78 total 416.45 last 70.66) - open/close 35989 (time 1771402246.79 total 426.45 last 90.06) - open/close 36914 (time 1771402256.80 total 436.46 last 92.38) - open/close 37813 (time 1771402266.81 total 446.47 last 89.80) - open/close 38741 (time 1771402276.81 total 456.48 last 92.76) - open/close 39705 (time 1771402286.82 total 466.48 last 96.35) - open/close 40000 (time 1771402289.76 total 469.42 last 100.39) - open/close 40979 (time 1771402299.76 total 479.43 last 97.87) - open/close 41892 (time 1771402309.77 total 489.43 last 91.23) - open/close 42727 (time 1771402319.77 total 499.44 last 83.48) - open/close 43764 (time 1771402329.78 total 509.44 last 103.65) - open/close 44788 (time 1771402339.81 total 519.48 last 102.05) - open/close 45821 (time 1771402349.82 total 529.48 last 103.25) - open/close 46765 (time 1771402359.82 total 539.48 last 94.37) - open/close 47702 (time 1771402369.82 total 549.49 last 93.69) - open/close 48575 (time 1771402379.83 total 559.49 last 87.26) - open/close 49512 (time 1771402389.83 total 569.50 last 93.62) - open/close 50000 (time 1771402395.41 total 575.07 last 87.56) - open/close 50973 (time 1771402405.41 total 585.07 last 97.28) - open/close 52033 (time 1771402415.41 total 595.08 last 105.97) - open/close 52959 (time 1771402425.42 total 605.08 last 92.53) - open/close 53871 (time 1771402435.42 total 615.09 last 91.18) - open/close 54813 (time 1771402445.42 total 625.09 last 94.18) - open/close 55657 (time 1771402455.43 total 635.09 last 84.37) - open/close 56518 (time 1771402465.44 total 645.10 last 86.02) - open/close 57454 (time 1771402475.49 total 655.16 last 93.07) - open/close 58243 (time 1771402485.51 total 665.18 last 78.76) - open/close 58996 (time 1771402495.51 total 675.18 last 75.28) - open/close 59912 (time 1771402505.52 total 685.18 last 91.56) - open/close 60906 (time 1771402515.53 total 695.19 last 99.32) - open/close 61839 (time 1771402525.53 total 705.19 last 93.29) - open/close 62776 (time 1771402535.53 total 715.20 last 93.67) - open/close 63698 (time 1771402545.53 total 725.20 last 92.20) - open/close 64593 (time 1771402555.54 total 735.20 last 89.45) total: 64768 open/close in 737.44 seconds: 87.83 ops/second - /unlink 812 (time 1771402571.34 total 10.02 last 81.08) - /unlink 1673 (time 1771402581.34 total 20.02 last 86.08) - /unlink 2519 (time 1771402591.35 total 30.03 last 84.49) - /unlink 3434 (time 1771402601.36 total 40.04 last 91.46) - /unlink 4428 (time 1771402611.36 total 50.04 last 99.38) - /unlink 5469 (time 1771402621.36 total 60.04 last 104.05) - /unlink 6401 (time 1771402631.37 total 70.05 last 93.13) - /unlink 7410 (time 1771402641.38 total 80.06 last 100.81) - /unlink 8388 (time 1771402651.39 total 90.06 last 97.75) - /unlink 9346 (time 1771402661.39 total 100.07 last 95.77) - /unlink 10000 (time 1771402668.46 total 107.14 last 92.46) - /unlink 11012 (time 1771402678.47 total 117.14 last 101.16) - /unlink 11987 (time 1771402688.48 total 127.16 last 97.37) - /unlink 12950 (time 1771402698.48 total 137.16 last 96.30) - /unlink 13987 (time 1771402708.50 total 147.18 last 103.48) - /unlink 14936 (time 1771402718.51 total 157.19 last 94.79) - /unlink 15893 (time 1771402728.52 total 167.20 last 95.61) - /unlink 16969 (time 1771402738.52 total 177.20 last 107.57) - /unlink 17855 (time 1771402748.53 total 187.21 last 88.54) - /unlink 18832 (time 1771402758.54 total 197.21 last 97.66) - /unlink 19836 (time 1771402768.55 total 207.22 last 100.29) - /unlink 20808 (time 1771402778.56 total 217.23 last 97.08) - /unlink 21674 (time 1771402788.56 total 227.24 last 86.55) - /unlink 22552 (time 1771402798.57 total 237.24 last 87.78) - /unlink 23509 (time 1771402808.57 total 247.25 last 95.65) - /unlink 24434 (time 1771402818.59 total 257.26 last 92.37) - /unlink 25409 (time 1771402828.59 total 267.26 last 97.49) - /unlink 26318 (time 1771402838.59 total 277.27 last 90.86) - /unlink 27204 (time 1771402848.59 total 287.27 last 88.57) - /unlink 28241 (time 1771402858.60 total 297.27 last 103.68) - /unlink 29091 (time 1771402868.60 total 307.28 last 84.97) - /unlink 30000 (time 1771402877.83 total 316.51 last 98.46) - /unlink 30862 (time 1771402887.84 total 326.52 last 86.12) - /unlink 31883 (time 1771402897.87 total 336.54 last 101.83) - /unlink 32858 (time 1771402907.87 total 346.55 last 97.49) - /unlink 33814 (time 1771402917.87 total 356.55 last 95.60) - /unlink 34515 (time 1771402927.89 total 366.56 last 69.98) - /unlink 35383 (time 1771402937.89 total 376.57 last 86.74) - /unlink 36333 (time 1771402947.90 total 386.58 last 94.95) - /unlink 37307 (time 1771402957.91 total 396.58 last 97.34) - /unlink 38280 (time 1771402967.91 total 406.58 last 97.29) - /unlink 39404 (time 1771402977.93 total 416.60 last 112.16) - /unlink 40000 (time 1771402983.76 total 422.43 last 102.26) - /unlink 41011 (time 1771402993.76 total 432.44 last 101.05) - /unlink 42062 (time 1771403003.76 total 442.44 last 105.06) - /unlink 43050 (time 1771403013.78 total 452.45 last 98.70) - /unlink 43982 (time 1771403023.78 total 462.46 last 93.17) - /unlink 44914 (time 1771403033.78 total 472.46 last 93.17) - /unlink 45777 (time 1771403043.80 total 482.48 last 86.15) - /unlink 46617 (time 1771403053.81 total 492.48 last 83.95) - /unlink 47568 (time 1771403063.81 total 502.49 last 95.05) - /unlink 48560 (time 1771403073.81 total 512.49 last 99.16) - /unlink 49531 (time 1771403083.83 total 522.50 last 96.99) - /unlink 50000 (time 1771403088.16 total 526.83 last 108.26) - /unlink 51111 (time 1771403098.16 total 536.84 last 111.09) - /unlink 52145 (time 1771403108.16 total 546.84 last 103.37) - /unlink 53091 (time 1771403118.17 total 556.84 last 94.55) - /unlink 54126 (time 1771403128.17 total 566.85 last 103.45) - /unlink 55190 (time 1771403138.17 total 576.85 last 106.38) - /unlink 56225 (time 1771403148.18 total 586.86 last 103.42) - /unlink 57179 (time 1771403158.19 total 596.86 last 95.36) - /unlink 58341 (time 1771403168.19 total 606.86 last 116.19) - /unlink 59388 (time 1771403178.19 total 616.87 last 104.66) - /unlink 60000 (time 1771403183.41 total 622.09 last 117.28) - /unlink 61116 (time 1771403193.41 total 632.09 last 111.57) - /unlink 62204 (time 1771403203.42 total 642.09 last 108.75) - /unlink 63307 (time 1771403213.42 total 652.10 last 110.26) - /unlink 64318 (time 1771403223.42 total 662.10 last 101.06) total: 64768 /unlink in 666.39 seconds: 97.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1175 (time 1771403259.58 total 10.01 last 117.43) - open/close 2277 (time 1771403269.58 total 20.01 last 110.14) - open/close 3307 (time 1771403279.59 total 30.01 last 102.98) - open/close 4333 (time 1771403289.59 total 40.02 last 102.56) - open/close 5364 (time 1771403299.61 total 50.03 last 102.95) - open/close 6419 (time 1771403309.61 total 60.04 last 105.46) - open/close 7509 (time 1771403319.61 total 70.04 last 108.99) - open/close 8544 (time 1771403329.62 total 80.05 last 103.38) - open/close 9527 (time 1771403339.63 total 90.05 last 98.25) - open/close 10000 (time 1771403344.31 total 94.74 last 100.92) - open/close 11073 (time 1771403354.32 total 104.74 last 107.26) - open/close 12205 (time 1771403364.32 total 114.75 last 113.15) - open/close 13280 (time 1771403374.32 total 124.75 last 107.48) - open/close 14162 (time 1771403384.33 total 134.75 last 88.17) - open/close 15302 (time 1771403394.33 total 144.76 last 113.92) - open/close 16439 (time 1771403404.34 total 154.76 last 113.65) - open/close 17494 (time 1771403414.34 total 164.77 last 105.46) - open/close 18691 (time 1771403424.34 total 174.77 last 119.69) - open/close 19794 (time 1771403434.36 total 184.78 last 110.16) - open/close 20991 (time 1771403444.36 total 194.78 last 119.68) - open/close 22293 (time 1771403454.36 total 204.79 last 130.12) - open/close 23320 (time 1771403464.37 total 214.80 last 102.64) - open/close 24213 (time 1771403474.37 total 224.80 last 89.30) - open/close 25332 (time 1771403484.38 total 234.80 last 111.82) - open/close 26498 (time 1771403494.39 total 244.81 last 116.51) - open/close 27701 (time 1771403504.39 total 254.82 last 120.27) - open/close 28870 (time 1771403514.40 total 264.82 last 116.83) - open/close 30000 (time 1771403524.01 total 274.44 last 117.51) - open/close 31242 (time 1771403534.02 total 284.45 last 124.11) - open/close 32534 (time 1771403544.03 total 294.45 last 129.11) - open/close 33733 (time 1771403554.04 total 304.46 last 119.77) - open/close 34866 (time 1771403564.04 total 314.47 last 113.25) - open/close 36132 (time 1771403574.04 total 324.47 last 126.57) - open/close 37125 (time 1771403584.04 total 334.47 last 99.30) - open/close 38206 (time 1771403594.05 total 344.48 last 108.04) - open/close 39408 (time 1771403604.06 total 354.48 last 120.09) - open/close 40000 (time 1771403608.96 total 359.39 last 120.66) - open/close 41284 (time 1771403618.98 total 369.40 last 128.24) - open/close 42633 (time 1771403628.98 total 379.40 last 134.89) - open/close 43877 (time 1771403638.99 total 389.41 last 124.27) - open/close 45112 (time 1771403648.99 total 399.42 last 123.47) - open/close 46325 (time 1771403659.00 total 409.43 last 121.18) - open/close 47573 (time 1771403669.00 total 419.43 last 124.79) - open/close 48656 (time 1771403679.01 total 429.43 last 108.25) - open/close 49780 (time 1771403689.01 total 439.44 last 112.36) - open/close 51030 (time 1771403699.01 total 449.44 last 124.98) - open/close 52122 (time 1771403709.02 total 459.45 last 109.11) - open/close 53280 (time 1771403719.02 total 469.45 last 115.78) - open/close 54705 (time 1771403729.02 total 479.45 last 142.46) - open/close 56094 (time 1771403739.03 total 489.46 last 138.83) - open/close 57237 (time 1771403749.03 total 499.46 last 114.29) - open/close 58321 (time 1771403759.04 total 509.47 last 108.27) - open/close 59439 (time 1771403769.05 total 519.48 last 111.67) - open/close 60000 (time 1771403773.31 total 523.74 last 131.73) - open/close 61491 (time 1771403783.31 total 533.74 last 149.08) - open/close 62997 (time 1771403793.31 total 543.74 last 150.58) - open/close 64532 (time 1771403803.32 total 553.74 last 153.49) total: 64768 open/close in 555.34 seconds: 116.63 ops/second - /unlink 2637 (time 1771403816.62 total 10.00 last 263.64) - /unlink 5202 (time 1771403826.62 total 20.00 last 256.47) - /unlink 8335 (time 1771403836.62 total 30.00 last 313.26) - /unlink 10000 (time 1771403842.10 total 35.48 last 304.02) - /unlink 12654 (time 1771403852.10 total 45.48 last 265.39) - /unlink 15234 (time 1771403862.10 total 55.49 last 257.89) - /unlink 17578 (time 1771403872.11 total 65.49 last 234.25) - /unlink 20000 (time 1771403880.58 total 73.96 last 285.91) - /unlink 23340 (time 1771403890.58 total 83.96 last 333.98) - /unlink 26623 (time 1771403900.58 total 93.96 last 328.26) - /unlink 29593 (time 1771403910.58 total 103.97 last 296.98) - /unlink 32367 (time 1771403920.58 total 113.97 last 277.34) - /unlink 35167 (time 1771403930.58 total 123.97 last 279.97) - /unlink 38449 (time 1771403940.58 total 133.97 last 328.20) - /unlink 40000 (time 1771403945.82 total 139.20 last 296.48) - /unlink 42602 (time 1771403955.82 total 149.20 last 260.15) - /unlink 45896 (time 1771403965.82 total 159.21 last 329.25) - /unlink 48589 (time 1771403975.82 total 169.21 last 269.23) - /unlink 50000 (time 1771403981.44 total 174.83 last 251.09) - /unlink 52920 (time 1771403991.45 total 184.83 last 291.95) - /unlink 55532 (time 1771404001.45 total 194.83 last 261.11) - /unlink 58229 (time 1771404011.46 total 204.84 last 269.41) - /unlink 60000 (time 1771404017.36 total 210.75 last 299.91) - /unlink 62837 (time 1771404027.37 total 220.75 last 283.69) total: 64768 /unlink in 227.58 seconds: 284.60 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1766 (time 1771404061.78 total 10.00 last 176.55) - open/close 3633 (time 1771404071.78 total 20.00 last 186.70) - open/close 5232 (time 1771404081.78 total 30.01 last 159.85) - open/close 6839 (time 1771404091.78 total 40.01 last 160.67) - open/close 8442 (time 1771404101.79 total 50.01 last 160.21) - open/close 10000 (time 1771404111.72 total 59.95 last 156.79) - open/close 11604 (time 1771404121.73 total 69.95 last 160.35) - open/close 13368 (time 1771404131.73 total 79.96 last 176.25) - open/close 15222 (time 1771404141.74 total 89.97 last 185.30) - open/close 16816 (time 1771404151.74 total 99.97 last 159.33) - open/close 18308 (time 1771404161.75 total 109.97 last 149.16) - open/close 19834 (time 1771404171.75 total 119.98 last 152.54) - open/close 21306 (time 1771404181.75 total 129.98 last 147.17) - open/close 22923 (time 1771404191.76 total 139.98 last 161.66) - open/close 24477 (time 1771404201.76 total 149.98 last 155.37) - open/close 26055 (time 1771404211.76 total 159.99 last 157.75) - open/close 27813 (time 1771404221.76 total 169.99 last 175.79) - open/close 29493 (time 1771404231.76 total 179.99 last 167.98) - open/close 30000 (time 1771404234.64 total 182.87 last 176.06) - open/close 31715 (time 1771404244.65 total 192.87 last 171.43) - open/close 33558 (time 1771404254.65 total 202.88 last 184.22) - open/close 35108 (time 1771404264.65 total 212.88 last 155.00) - open/close 36590 (time 1771404274.65 total 222.88 last 148.17) - open/close 37944 (time 1771404284.66 total 232.89 last 135.31) - open/close 39551 (time 1771404294.66 total 242.89 last 160.61) - open/close 40000 (time 1771404296.87 total 245.10 last 203.10) - open/close 42030 (time 1771404306.88 total 255.10 last 203.00) - open/close 43921 (time 1771404316.88 total 265.10 last 189.07) - open/close 45979 (time 1771404326.88 total 275.11 last 205.73) - open/close 47869 (time 1771404336.88 total 285.11 last 188.99) - open/close 49538 (time 1771404346.88 total 295.11 last 166.83) - open/close 50000 (time 1771404349.29 total 297.52 last 192.02) - open/close 51518 (time 1771404359.29 total 307.52 last 151.80) - open/close 53211 (time 1771404369.29 total 317.52 last 169.26) - open/close 54772 (time 1771404379.29 total 327.52 last 156.10) - open/close 56382 (time 1771404389.29 total 337.52 last 160.98) - open/close 58334 (time 1771404399.30 total 347.52 last 195.15) - open/close 60000 (time 1771404407.70 total 355.93 last 198.28) - open/close 62195 (time 1771404417.70 total 365.93 last 219.46) - open/close 63750 (time 1771404427.71 total 375.94 last 155.29) total: 64768 open/close in 382.17 seconds: 169.47 ops/second - /unlink 3181 (time 1771404445.66 total 10.00 last 318.08) - /unlink 6955 (time 1771404455.66 total 20.00 last 377.33) - /unlink 10000 (time 1771404464.87 total 29.20 last 330.96) - /unlink 13036 (time 1771404474.87 total 39.21 last 303.53) - /unlink 16676 (time 1771404484.87 total 49.21 last 363.97) - /unlink 20000 (time 1771404494.82 total 59.16 last 333.92) - /unlink 23814 (time 1771404504.83 total 69.16 last 381.25) - /unlink 27932 (time 1771404514.83 total 79.17 last 411.70) - /unlink 30000 (time 1771404518.84 total 83.18 last 515.49) - /unlink 35406 (time 1771404528.84 total 93.18 last 540.56) - /unlink 40000 (time 1771404537.56 total 101.90 last 526.94) - /unlink 44744 (time 1771404547.56 total 111.90 last 474.35) - /unlink 50000 (time 1771404556.73 total 121.06 last 573.50) - /unlink 54926 (time 1771404566.73 total 131.07 last 492.52) - /unlink 59787 (time 1771404576.73 total 141.07 last 486.04) total: 64768 /unlink in 150.23 seconds: 431.13 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3691 (time 1771404614.06 total 10.01 last 368.67) - open/close 6862 (time 1771404624.06 total 20.02 last 316.99) - open/close 9374 (time 1771404634.06 total 30.02 last 251.19) - open/close 13137 (time 1771404644.06 total 40.02 last 376.24) - open/close 16521 (time 1771404654.07 total 50.02 last 338.38) - open/close 20000 (time 1771404663.03 total 58.98 last 387.98) - open/close 23847 (time 1771404673.03 total 68.99 last 384.62) - open/close 27949 (time 1771404683.04 total 78.99 last 410.14) - open/close 30000 (time 1771404689.18 total 85.13 last 333.79) - open/close 33998 (time 1771404699.18 total 95.13 last 399.75) - open/close 38053 (time 1771404709.18 total 105.14 last 405.41) - open/close 40000 (time 1771404714.74 total 110.69 last 350.53) - open/close 43482 (time 1771404724.74 total 120.69 last 348.18) - open/close 45386 (time 1771404734.74 total 130.69 last 190.39) - open/close 46494 (time 1771404744.74 total 140.69 last 110.78) - open/close 47572 (time 1771404754.74 total 150.69 last 107.79) - open/close 48751 (time 1771404764.74 total 160.70 last 117.89) - open/close 50000 (time 1771404770.89 total 166.84 last 203.34) - open/close 51688 (time 1771404780.90 total 176.85 last 168.60) - open/close 53353 (time 1771404790.90 total 186.85 last 166.44) - open/close 55054 (time 1771404800.90 total 196.85 last 170.08) - open/close 56541 (time 1771404810.91 total 206.86 last 148.61) - open/close 57981 (time 1771404820.91 total 216.87 last 143.92) - open/close 59417 (time 1771404830.92 total 226.87 last 143.57) - open/close 60000 (time 1771404835.13 total 231.08 last 138.28) - open/close 61546 (time 1771404845.13 total 241.09 last 154.57) - open/close 63330 (time 1771404855.14 total 251.09 last 178.33) total: 64768 open/close in 256.94 seconds: 252.08 ops/second - /unlink 3094 (time 1771404872.54 total 10.01 last 309.17) - /unlink 5478 (time 1771404882.54 total 20.01 last 238.39) - /unlink 7870 (time 1771404892.54 total 30.01 last 239.15) - /unlink 10000 (time 1771404900.71 total 38.18 last 260.76) - /unlink 12480 (time 1771404910.71 total 48.18 last 247.93) - /unlink 14314 (time 1771404920.71 total 58.19 last 183.32) - /unlink 16452 (time 1771404930.71 total 68.19 last 213.78) - /unlink 18202 (time 1771404940.74 total 78.21 last 174.59) - /unlink 18763 (time 1771404950.74 total 88.21 last 56.09) - /unlink 19551 (time 1771404960.74 total 98.21 last 78.80) - /unlink 20000 (time 1771404964.90 total 102.38 last 107.80) - /unlink 20967 (time 1771404974.91 total 112.38 last 96.65) - /unlink 21900 (time 1771404984.93 total 122.40 last 93.14) - /unlink 22826 (time 1771404994.93 total 132.40 last 92.57) - /unlink 23707 (time 1771405004.94 total 142.41 last 88.03) - /unlink 24637 (time 1771405014.95 total 152.42 last 92.91) - /unlink 25908 (time 1771405024.95 total 162.42 last 127.08) - /unlink 27106 (time 1771405034.98 total 172.46 last 119.39) - /unlink 28125 (time 1771405044.99 total 182.46 last 101.83) - /unlink 29258 (time 1771405054.99 total 192.46 last 113.28) - /unlink 30000 (time 1771405061.76 total 199.23 last 109.71) - /unlink 31263 (time 1771405071.76 total 209.23 last 126.25) - /unlink 32647 (time 1771405081.76 total 219.23 last 138.39) - /unlink 33821 (time 1771405091.76 total 229.24 last 117.35) - /unlink 35033 (time 1771405101.77 total 239.24 last 121.18) - /unlink 36193 (time 1771405111.78 total 249.25 last 115.89) - /unlink 37388 (time 1771405121.78 total 259.25 last 119.47) - /unlink 38624 (time 1771405131.78 total 269.25 last 123.58) - /unlink 39927 (time 1771405141.79 total 279.26 last 130.14) - /unlink 41301 (time 1771405151.79 total 289.26 last 137.40) - /unlink 42649 (time 1771405161.79 total 299.27 last 134.78) - /unlink 43796 (time 1771405171.80 total 309.27 last 114.64) - /unlink 45018 (time 1771405181.80 total 319.28 last 122.14) - /unlink 46359 (time 1771405191.80 total 329.28 last 134.09) - /unlink 47387 (time 1771405201.81 total 339.29 last 102.70) - /unlink 48642 (time 1771405211.82 total 349.29 last 125.46) - /unlink 50000 (time 1771405221.28 total 358.75 last 143.57) - /unlink 51309 (time 1771405231.28 total 368.75 last 130.86) - /unlink 52610 (time 1771405241.28 total 378.76 last 130.05) - /unlink 53791 (time 1771405251.28 total 388.76 last 118.10) - /unlink 55019 (time 1771405261.30 total 398.77 last 122.65) - /unlink 56133 (time 1771405271.30 total 408.77 last 111.40) - /unlink 57398 (time 1771405281.31 total 418.79 last 126.28) - /unlink 58669 (time 1771405291.32 total 428.79 last 127.07) - /unlink 59841 (time 1771405301.32 total 438.79 last 117.17) - /unlink 61132 (time 1771405311.32 total 448.79 last 129.08) - /unlink 62356 (time 1771405321.33 total 458.80 last 122.33) - /unlink 63480 (time 1771405331.33 total 468.80 last 112.36) total: 64768 /unlink in 477.81 seconds: 135.55 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg211-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg211-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server unloading modules via unload_modules_local on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing unload_modules_local oleg211-server: modules unloaded. pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2