== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:43:24 (1770684204) Stopping clients: oleg104-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg104-client.virtnet /mnt/lustre2 (opts:-f) oleg104-server: oleg104-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg104-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg104-server: detected 4 online CPUs by sysfs oleg104-server: MODOPTS_LIBCFS= oleg104-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg104-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg104-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32649 393 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 378 (time 1770684386.17 total 10.01 last 37.75) - open/close 783 (time 1770684396.17 total 20.02 last 40.48) - open/close 1300 (time 1770684406.20 total 30.05 last 51.53) - open/close 1776 (time 1770684416.21 total 40.06 last 47.57) - open/close 2275 (time 1770684426.22 total 50.07 last 49.86) - open/close 2784 (time 1770684436.22 total 60.07 last 50.88) - open/close 3335 (time 1770684446.24 total 70.09 last 54.99) - open/close 3885 (time 1770684456.25 total 80.10 last 54.94) - open/close 4391 (time 1770684466.27 total 90.12 last 50.53) - open/close 4905 (time 1770684476.28 total 100.12 last 51.36) - open/close 5368 (time 1770684486.28 total 110.13 last 46.28) - open/close 5833 (time 1770684496.29 total 120.14 last 46.45) - open/close 6352 (time 1770684506.31 total 130.16 last 51.82) - open/close 6823 (time 1770684516.32 total 140.16 last 47.06) - open/close 7246 (time 1770684526.33 total 150.18 last 42.22) - open/close 7711 (time 1770684536.35 total 160.20 last 46.43) - open/close 8282 (time 1770684546.35 total 170.20 last 57.09) - open/close 8787 (time 1770684556.36 total 180.21 last 50.42) - open/close 9232 (time 1770684566.37 total 190.22 last 44.49) - open/close 9772 (time 1770684576.40 total 200.25 last 53.82) - open/close 10000 (time 1770684580.99 total 204.83 last 49.74) - open/close 10505 (time 1770684591.00 total 214.85 last 50.44) - open/close 11000 (time 1770684601.01 total 224.86 last 49.42) - open/close 11522 (time 1770684611.03 total 234.88 last 52.14) - open/close 11996 (time 1770684621.04 total 244.89 last 47.34) - open/close 12453 (time 1770684631.05 total 254.90 last 45.64) - open/close 12889 (time 1770684641.06 total 264.91 last 43.58) - open/close 13300 (time 1770684651.06 total 274.91 last 41.08) - open/close 13712 (time 1770684661.06 total 284.91 last 41.20) - open/close 14204 (time 1770684671.08 total 294.92 last 49.14) - open/close 14719 (time 1770684681.08 total 304.92 last 51.50) - open/close 15234 (time 1770684691.08 total 314.93 last 51.47) - open/close 15640 (time 1770684701.09 total 324.94 last 40.55) - open/close 16117 (time 1770684711.11 total 334.95 last 47.65) - open/close 16484 (time 1770684721.12 total 344.97 last 36.65) - open/close 16957 (time 1770684731.15 total 354.99 last 47.17) - open/close 17405 (time 1770684741.15 total 365.00 last 44.77) - open/close 17787 (time 1770684751.15 total 375.00 last 38.20) - open/close 18195 (time 1770684761.16 total 385.01 last 40.78) - open/close 18700 (time 1770684771.17 total 395.02 last 50.42) - open/close 19115 (time 1770684781.20 total 405.05 last 41.38) - open/close 19511 (time 1770684791.23 total 415.07 last 39.51) - open/close 19890 (time 1770684801.25 total 425.10 last 37.80) - open/close 20000 (time 1770684803.77 total 427.62 last 43.65) - open/close 20359 (time 1770684813.79 total 437.64 last 35.82) - open/close 20792 (time 1770684823.80 total 447.65 last 43.29) - open/close 21192 (time 1770684833.81 total 457.66 last 39.95) - open/close 21654 (time 1770684843.82 total 467.67 last 46.17) - open/close 22099 (time 1770684853.82 total 477.67 last 44.49) - open/close 22641 (time 1770684863.84 total 487.69 last 54.08) - open/close 23142 (time 1770684873.89 total 497.73 last 49.88) - open/close 23524 (time 1770684883.90 total 507.75 last 38.13) - open/close 23943 (time 1770684893.93 total 517.77 last 41.81) - open/close 24392 (time 1770684903.94 total 527.79 last 44.82) - open/close 24848 (time 1770684913.96 total 537.80 last 45.54) - open/close 25283 (time 1770684923.97 total 547.82 last 43.45) - open/close 25727 (time 1770684933.97 total 557.82 last 44.37) - open/close 26229 (time 1770684943.99 total 567.84 last 50.11) - open/close 26760 (time 1770684954.00 total 577.84 last 53.08) - open/close 27179 (time 1770684964.00 total 587.85 last 41.88) - open/close 27619 (time 1770684974.03 total 597.88 last 43.89) - open/close 28013 (time 1770684984.04 total 607.89 last 39.33) - open/close 28438 (time 1770684994.05 total 617.90 last 42.45) - open/close 28878 (time 1770685004.06 total 627.90 last 43.99) - open/close 29309 (time 1770685014.08 total 637.93 last 43.01) - open/close 29791 (time 1770685024.08 total 647.93 last 48.18) - open/close 30000 (time 1770685029.28 total 653.13 last 40.17) - open/close 30402 (time 1770685039.29 total 663.14 last 40.17) - open/close 30697 (time 1770685049.35 total 673.20 last 29.33) - open/close 31024 (time 1770685059.36 total 683.21 last 32.66) - open/close 31362 (time 1770685069.39 total 693.23 last 33.72) - open/close 31778 (time 1770685079.39 total 703.24 last 41.58) - open/close 32173 (time 1770685089.39 total 713.24 last 39.50) - open/close 32551 (time 1770685099.40 total 723.25 last 37.76) - open/close 32872 (time 1770685109.47 total 733.32 last 31.88) - open/close 33206 (time 1770685119.49 total 743.33 last 33.34) - open/close 33611 (time 1770685129.50 total 753.35 last 40.45) - open/close 34010 (time 1770685139.50 total 763.35 last 39.88) - open/close 34443 (time 1770685149.54 total 773.39 last 43.15) - open/close 34881 (time 1770685159.55 total 783.39 last 43.76) - open/close 35406 (time 1770685169.55 total 793.40 last 52.46) - open/close 35945 (time 1770685179.56 total 803.41 last 53.88) - open/close 36362 (time 1770685189.57 total 813.42 last 41.64) - open/close 36778 (time 1770685199.57 total 823.42 last 41.60) - open/close 37251 (time 1770685209.58 total 833.43 last 47.27) - open/close 37715 (time 1770685219.59 total 843.44 last 46.36) - open/close 38168 (time 1770685229.60 total 853.45 last 45.24) - open/close 38518 (time 1770685239.60 total 863.45 last 34.99) - open/close 39004 (time 1770685249.62 total 873.47 last 48.53) - open/close 39530 (time 1770685259.62 total 883.47 last 52.58) - open/close 39859 (time 1770685269.64 total 893.49 last 32.83) - open/close 40000 (time 1770685273.09 total 896.94 last 40.89) - open/close 40424 (time 1770685283.10 total 906.95 last 42.38) - open/close 40911 (time 1770685293.10 total 916.95 last 48.69) - open/close 41442 (time 1770685303.11 total 926.96 last 53.06) - open/close 41811 (time 1770685313.15 total 937.00 last 36.75) - open/close 42231 (time 1770685323.15 total 947.00 last 41.99) - open/close 42657 (time 1770685333.16 total 957.01 last 42.55) - open/close 43162 (time 1770685343.16 total 967.01 last 50.49) - open/close 43590 (time 1770685353.20 total 977.05 last 42.63) - open/close 43973 (time 1770685363.21 total 987.06 last 38.27) - open/close 44346 (time 1770685373.22 total 997.07 last 37.24) - open/close 44696 (time 1770685383.24 total 1007.09 last 34.95) - open/close 45101 (time 1770685393.25 total 1017.10 last 40.46) - open/close 45454 (time 1770685403.25 total 1027.10 last 35.30) - open/close 45838 (time 1770685413.26 total 1037.11 last 38.35) - open/close 46255 (time 1770685423.30 total 1047.15 last 41.55) - open/close 46629 (time 1770685433.32 total 1057.17 last 37.31) - open/close 47075 (time 1770685443.33 total 1067.17 last 44.59) - open/close 47485 (time 1770685453.33 total 1077.18 last 40.99) - open/close 47890 (time 1770685463.34 total 1087.19 last 40.46) - open/close 48436 (time 1770685473.35 total 1097.19 last 54.56) - open/close 48984 (time 1770685483.36 total 1107.21 last 54.74) - open/close 49530 (time 1770685493.43 total 1117.28 last 54.21) - open/close 49891 (time 1770685503.43 total 1127.28 last 36.08) - open/close 50000 (time 1770685506.07 total 1129.91 last 41.41) - open/close 50417 (time 1770685516.08 total 1139.92 last 41.66) - open/close 50870 (time 1770685526.10 total 1149.95 last 45.19) - open/close 51286 (time 1770685536.10 total 1159.95 last 41.60) - open/close 51733 (time 1770685546.10 total 1169.95 last 44.69) - open/close 52121 (time 1770685556.14 total 1179.99 last 38.66) - open/close 52526 (time 1770685566.15 total 1189.99 last 40.47) - open/close 52929 (time 1770685576.16 total 1200.01 last 40.24) - open/close 53296 (time 1770685586.16 total 1210.01 last 36.69) - open/close 53692 (time 1770685596.20 total 1220.05 last 39.46) - open/close 54107 (time 1770685606.21 total 1230.05 last 41.47) - open/close 54527 (time 1770685616.24 total 1240.09 last 41.86) - open/close 54937 (time 1770685626.24 total 1250.09 last 41.00) - open/close 55329 (time 1770685636.24 total 1260.09 last 39.19) - open/close 55819 (time 1770685646.26 total 1270.11 last 48.91) - open/close 56214 (time 1770685656.27 total 1280.12 last 39.46) - open/close 56627 (time 1770685666.28 total 1290.13 last 41.26) - open/close 57027 (time 1770685676.30 total 1300.15 last 39.91) - open/close 57470 (time 1770685686.33 total 1310.18 last 44.19) - open/close 57966 (time 1770685696.33 total 1320.18 last 49.59) - open/close 58470 (time 1770685706.36 total 1330.21 last 50.24) - open/close 58884 (time 1770685716.37 total 1340.21 last 41.38) - open/close 59282 (time 1770685726.37 total 1350.22 last 39.78) - open/close 59682 (time 1770685736.39 total 1360.24 last 39.94) - open/close 60000 (time 1770685745.06 total 1368.91 last 36.65) - open/close 60383 (time 1770685755.07 total 1378.92 last 38.29) - open/close 60830 (time 1770685765.07 total 1388.92 last 44.69) - open/close 61332 (time 1770685775.08 total 1398.92 last 50.18) - open/close 61812 (time 1770685785.08 total 1408.93 last 47.95) - open/close 62294 (time 1770685795.09 total 1418.93 last 48.20) - open/close 62761 (time 1770685805.09 total 1428.94 last 46.67) - open/close 63256 (time 1770685815.11 total 1438.96 last 49.43) - open/close 63790 (time 1770685825.11 total 1448.96 last 53.40) - open/close 64248 (time 1770685835.13 total 1458.98 last 45.70) - open/close 64726 (time 1770685845.14 total 1468.99 last 47.74) total: 64768 open/close in 1469.66 seconds: 44.07 ops/second - /unlink 490 (time 1770685862.04 total 10.01 last 48.93) - /unlink 948 (time 1770685872.05 total 20.03 last 45.74) - /unlink 1469 (time 1770685882.07 total 30.04 last 52.01) - /unlink 2001 (time 1770685892.09 total 40.07 last 53.09) - /unlink 2479 (time 1770685902.10 total 50.08 last 47.76) - /unlink 2965 (time 1770685912.11 total 60.09 last 48.55) - /unlink 3418 (time 1770685922.11 total 70.09 last 45.30) - /unlink 3915 (time 1770685932.11 total 80.09 last 49.69) - /unlink 4392 (time 1770685942.13 total 90.11 last 47.59) - /unlink 4905 (time 1770685952.16 total 100.14 last 51.16) - /unlink 5385 (time 1770685962.17 total 110.15 last 47.95) - /unlink 5879 (time 1770685972.18 total 120.16 last 49.36) - /unlink 6347 (time 1770685982.20 total 130.18 last 46.70) - /unlink 6876 (time 1770685992.21 total 140.19 last 52.85) - /unlink 7456 (time 1770686002.24 total 150.22 last 57.80) - /unlink 8001 (time 1770686012.26 total 160.24 last 54.40) - /unlink 8520 (time 1770686022.28 total 170.26 last 51.79) - /unlink 9034 (time 1770686032.30 total 180.28 last 51.31) - /unlink 9587 (time 1770686042.31 total 190.29 last 55.25) - /unlink 10000 (time 1770686049.03 total 197.01 last 61.47) - /unlink 10532 (time 1770686059.04 total 207.01 last 53.16) - /unlink 11087 (time 1770686069.04 total 217.02 last 55.48) - /unlink 11681 (time 1770686079.06 total 227.03 last 59.31) - /unlink 12224 (time 1770686089.06 total 237.04 last 54.26) - /unlink 12761 (time 1770686099.12 total 247.10 last 53.37) - /unlink 13301 (time 1770686109.13 total 257.11 last 53.96) - /unlink 13825 (time 1770686119.15 total 267.13 last 52.31) - /unlink 14381 (time 1770686129.16 total 277.14 last 55.54) - /unlink 14929 (time 1770686139.16 total 287.14 last 54.79) - /unlink 15508 (time 1770686149.19 total 297.16 last 57.75) - /unlink 16095 (time 1770686159.21 total 307.18 last 58.58) - /unlink 16681 (time 1770686169.23 total 317.20 last 58.49) - /unlink 17275 (time 1770686179.23 total 327.21 last 59.38) - /unlink 17827 (time 1770686189.23 total 337.21 last 55.17) - /unlink 18418 (time 1770686199.24 total 347.21 last 59.09) - /unlink 19009 (time 1770686209.24 total 357.21 last 59.10) - /unlink 19583 (time 1770686219.25 total 367.23 last 57.34) - /unlink 20000 (time 1770686226.96 total 374.94 last 54.07) - /unlink 20536 (time 1770686236.98 total 384.96 last 53.51) - /unlink 21082 (time 1770686246.98 total 394.96 last 54.60) - /unlink 21594 (time 1770686257.00 total 404.97 last 51.11) - /unlink 22076 (time 1770686267.01 total 414.99 last 48.12) - /unlink 22598 (time 1770686277.03 total 425.01 last 52.09) - /unlink 23080 (time 1770686287.04 total 435.02 last 48.15) - /unlink 23571 (time 1770686297.05 total 445.02 last 49.08) - /unlink 24011 (time 1770686307.07 total 455.05 last 43.91) - /unlink 24521 (time 1770686317.09 total 465.06 last 50.91) - /unlink 25036 (time 1770686327.10 total 475.08 last 51.43) - /unlink 25569 (time 1770686337.12 total 485.10 last 53.19) - /unlink 26109 (time 1770686347.14 total 495.11 last 53.91) - /unlink 26664 (time 1770686357.17 total 505.15 last 55.32) - /unlink 27113 (time 1770686367.18 total 515.16 last 44.84) - /unlink 27619 (time 1770686377.19 total 525.17 last 50.56) - /unlink 28170 (time 1770686387.20 total 535.18 last 55.02) - /unlink 28714 (time 1770686397.22 total 545.20 last 54.31) - /unlink 29157 (time 1770686407.24 total 555.21 last 44.23) - /unlink 29626 (time 1770686417.25 total 565.23 last 46.84) - /unlink 30000 (time 1770686425.03 total 573.01 last 48.05) - /unlink 30571 (time 1770686435.04 total 583.02 last 57.06) - /unlink 31166 (time 1770686445.04 total 593.02 last 59.49) - /unlink 31790 (time 1770686455.05 total 603.03 last 62.35) - /unlink 32356 (time 1770686465.05 total 613.03 last 56.58) - /unlink 32854 (time 1770686475.07 total 623.04 last 49.74) - /unlink 33402 (time 1770686485.09 total 633.07 last 54.66) - /unlink 33922 (time 1770686495.09 total 643.07 last 51.99) - /unlink 34444 (time 1770686505.10 total 653.08 last 52.16) - /unlink 35011 (time 1770686515.10 total 663.08 last 56.69) - /unlink 35525 (time 1770686525.12 total 673.10 last 51.32) - /unlink 35980 (time 1770686535.12 total 683.10 last 45.49) - /unlink 36414 (time 1770686545.13 total 693.11 last 43.36) - /unlink 36945 (time 1770686555.14 total 703.12 last 53.05) - /unlink 37552 (time 1770686565.14 total 713.12 last 60.69) - /unlink 38114 (time 1770686575.14 total 723.12 last 56.17) - /unlink 38641 (time 1770686585.16 total 733.14 last 52.61) - /unlink 39172 (time 1770686595.16 total 743.14 last 53.08) - /unlink 39713 (time 1770686605.20 total 753.17 last 53.92) - /unlink 40000 (time 1770686610.53 total 758.51 last 53.83) - /unlink 40566 (time 1770686620.54 total 768.52 last 56.53) - /unlink 41127 (time 1770686630.54 total 778.52 last 56.08) - /unlink 41665 (time 1770686640.57 total 788.55 last 53.65) - /unlink 42222 (time 1770686650.59 total 798.56 last 55.62) - /unlink 42806 (time 1770686660.59 total 808.57 last 58.36) - /unlink 43353 (time 1770686670.60 total 818.58 last 54.67) - /unlink 43901 (time 1770686680.61 total 828.58 last 54.77) - /unlink 44436 (time 1770686690.61 total 838.59 last 53.48) - /unlink 44985 (time 1770686700.63 total 848.61 last 54.79) - /unlink 45496 (time 1770686710.64 total 858.62 last 51.03) - /unlink 46052 (time 1770686720.66 total 868.64 last 55.51) - /unlink 46590 (time 1770686730.67 total 878.65 last 53.72) - /unlink 47138 (time 1770686740.68 total 888.66 last 54.75) - /unlink 47666 (time 1770686750.68 total 898.66 last 52.79) - /unlink 48186 (time 1770686760.70 total 908.68 last 51.92) - /unlink 48658 (time 1770686770.71 total 918.69 last 47.16) - /unlink 49135 (time 1770686780.71 total 928.69 last 47.68) - /unlink 49715 (time 1770686790.71 total 938.69 last 57.98) - /unlink 50000 (time 1770686796.07 total 944.05 last 53.18) - /unlink 50612 (time 1770686806.09 total 954.06 last 61.12) - /unlink 51122 (time 1770686816.11 total 964.09 last 50.88) - /unlink 51685 (time 1770686826.13 total 974.10 last 56.20) - /unlink 52220 (time 1770686836.13 total 984.11 last 53.48) - /unlink 52774 (time 1770686846.14 total 994.12 last 55.32) - /unlink 53315 (time 1770686856.15 total 1004.13 last 54.05) - /unlink 53856 (time 1770686866.16 total 1014.14 last 54.05) - /unlink 54359 (time 1770686876.19 total 1024.17 last 50.17) - /unlink 54921 (time 1770686886.19 total 1034.17 last 56.18) - /unlink 55464 (time 1770686896.21 total 1044.19 last 54.20) - /unlink 56102 (time 1770686906.22 total 1054.20 last 63.74) - /unlink 56700 (time 1770686916.23 total 1064.21 last 59.75) - /unlink 57366 (time 1770686926.24 total 1074.22 last 66.50) - /unlink 58048 (time 1770686936.25 total 1084.23 last 68.15) - /unlink 58654 (time 1770686946.27 total 1094.25 last 60.48) - /unlink 59332 (time 1770686956.28 total 1104.26 last 67.71) - /unlink 59953 (time 1770686966.29 total 1114.27 last 62.05) - /unlink 60590 (time 1770686976.29 total 1124.27 last 63.69) - /unlink 61204 (time 1770686986.30 total 1134.28 last 61.35) - /unlink 61834 (time 1770686996.31 total 1144.29 last 62.91) - /unlink 62560 (time 1770687006.31 total 1154.29 last 72.60) - /unlink 63174 (time 1770687016.32 total 1164.30 last 61.35) - /unlink 63850 (time 1770687026.33 total 1174.30 last 67.57) - /unlink 64620 (time 1770687036.33 total 1184.31 last 76.98) total: 64768 /unlink in 1186.60 seconds: 54.58 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 607 (time 1770687076.39 total 10.02 last 60.56) - open/close 1280 (time 1770687086.40 total 20.03 last 67.26) - open/close 1989 (time 1770687096.40 total 30.03 last 70.86) - open/close 2689 (time 1770687106.42 total 40.05 last 69.89) - open/close 3363 (time 1770687116.44 total 50.07 last 67.25) - open/close 4026 (time 1770687126.45 total 60.08 last 66.25) - open/close 4770 (time 1770687136.46 total 70.09 last 74.34) - open/close 5507 (time 1770687146.46 total 80.09 last 73.68) - open/close 6286 (time 1770687156.47 total 90.10 last 77.83) - open/close 7001 (time 1770687166.47 total 100.10 last 71.46) - open/close 7707 (time 1770687176.48 total 110.11 last 70.54) - open/close 8430 (time 1770687186.50 total 120.13 last 72.18) - open/close 9216 (time 1770687196.51 total 130.14 last 78.52) - open/close 9984 (time 1770687206.51 total 140.14 last 76.77) - open/close 10638 (time 1770687216.52 total 150.15 last 65.35) - open/close 11351 (time 1770687226.52 total 160.16 last 71.27) - open/close 12143 (time 1770687236.53 total 170.16 last 79.19) - open/close 12954 (time 1770687246.53 total 180.16 last 81.04) - open/close 13649 (time 1770687256.54 total 190.17 last 69.48) - open/close 14469 (time 1770687266.54 total 200.17 last 82.00) - open/close 15166 (time 1770687276.55 total 210.18 last 69.57) - open/close 15892 (time 1770687286.56 total 220.19 last 72.55) - open/close 16670 (time 1770687296.57 total 230.20 last 77.75) - open/close 17383 (time 1770687306.57 total 240.20 last 71.27) - open/close 18189 (time 1770687316.58 total 250.21 last 80.53) - open/close 19187 (time 1770687326.58 total 260.21 last 99.76) - open/close 19840 (time 1770687336.59 total 270.22 last 65.24) - open/close 20000 (time 1770687339.09 total 272.72 last 64.16) - open/close 20725 (time 1770687349.09 total 282.72 last 72.46) - open/close 21492 (time 1770687359.10 total 292.73 last 76.65) - open/close 22279 (time 1770687369.10 total 302.73 last 78.70) - open/close 23033 (time 1770687379.10 total 312.73 last 75.40) - open/close 23793 (time 1770687389.11 total 322.74 last 75.89) - open/close 24538 (time 1770687399.12 total 332.75 last 74.46) - open/close 25304 (time 1770687409.14 total 342.77 last 76.45) - open/close 26054 (time 1770687419.14 total 352.77 last 74.98) - open/close 26867 (time 1770687429.14 total 362.77 last 81.29) - open/close 27753 (time 1770687439.15 total 372.78 last 88.50) - open/close 28517 (time 1770687449.16 total 382.79 last 76.37) - open/close 29253 (time 1770687459.16 total 392.80 last 73.55) - open/close 30000 (time 1770687468.49 total 402.12 last 80.10) - open/close 30686 (time 1770687478.49 total 412.12 last 68.58) - open/close 31288 (time 1770687488.50 total 422.13 last 60.17) - open/close 32034 (time 1770687498.50 total 432.14 last 74.55) - open/close 32763 (time 1770687508.51 total 442.14 last 72.85) - open/close 33535 (time 1770687518.52 total 452.15 last 77.11) - open/close 34227 (time 1770687528.53 total 462.16 last 69.14) - open/close 34979 (time 1770687538.54 total 472.17 last 75.16) - open/close 35649 (time 1770687548.54 total 482.17 last 66.96) - open/close 36332 (time 1770687558.55 total 492.18 last 68.26) - open/close 37030 (time 1770687568.56 total 502.20 last 69.69) - open/close 37710 (time 1770687578.57 total 512.20 last 67.95) - open/close 38463 (time 1770687588.58 total 522.21 last 75.21) - open/close 39160 (time 1770687598.58 total 532.22 last 69.69) - open/close 39793 (time 1770687608.60 total 542.23 last 63.21) - open/close 40000 (time 1770687611.64 total 545.27 last 68.00) - open/close 40704 (time 1770687621.65 total 555.28 last 70.35) - open/close 41442 (time 1770687631.66 total 565.29 last 73.76) - open/close 42153 (time 1770687641.66 total 575.29 last 71.09) - open/close 42844 (time 1770687651.66 total 585.30 last 69.05) - open/close 43626 (time 1770687661.66 total 595.30 last 78.20) - open/close 44480 (time 1770687671.67 total 605.30 last 85.33) - open/close 45264 (time 1770687681.68 total 615.31 last 78.32) - open/close 46030 (time 1770687691.69 total 625.32 last 76.54) - open/close 46723 (time 1770687701.69 total 635.32 last 69.29) - open/close 47482 (time 1770687711.70 total 645.33 last 75.88) - open/close 48196 (time 1770687721.71 total 655.34 last 71.33) - open/close 48956 (time 1770687731.71 total 665.35 last 75.94) - open/close 49681 (time 1770687741.72 total 675.35 last 72.43) - open/close 50000 (time 1770687746.24 total 679.88 last 70.56) - open/close 50761 (time 1770687756.25 total 689.88 last 76.03) - open/close 51486 (time 1770687766.26 total 699.89 last 72.45) - open/close 52229 (time 1770687776.27 total 709.90 last 74.22) - open/close 52976 (time 1770687786.28 total 719.91 last 74.65) - open/close 53730 (time 1770687796.30 total 729.93 last 75.25) - open/close 54534 (time 1770687806.31 total 739.94 last 80.32) - open/close 55329 (time 1770687816.32 total 749.95 last 79.43) - open/close 56126 (time 1770687826.32 total 759.95 last 79.66) - open/close 56877 (time 1770687836.32 total 769.95 last 75.09) - open/close 57620 (time 1770687846.33 total 779.96 last 74.24) - open/close 58371 (time 1770687856.33 total 789.96 last 75.07) - open/close 59029 (time 1770687866.34 total 799.97 last 65.75) - open/close 59781 (time 1770687876.35 total 809.98 last 75.16) - open/close 60000 (time 1770687879.25 total 812.88 last 75.53) - open/close 60887 (time 1770687889.25 total 822.89 last 88.63) - open/close 61645 (time 1770687899.27 total 832.90 last 75.67) - open/close 62419 (time 1770687909.29 total 842.92 last 77.28) - open/close 63157 (time 1770687919.29 total 852.93 last 73.75) - open/close 63938 (time 1770687929.30 total 862.93 last 78.02) - open/close 64740 (time 1770687939.31 total 872.94 last 80.14) total: 64768 open/close in 873.21 seconds: 74.17 ops/second - /unlink 901 (time 1770687952.34 total 10.00 last 90.09) - /unlink 1698 (time 1770687962.35 total 20.01 last 79.62) - /unlink 2465 (time 1770687972.36 total 30.02 last 76.63) - /unlink 3310 (time 1770687982.36 total 40.02 last 84.46) - /unlink 4204 (time 1770687992.37 total 50.04 last 89.31) - /unlink 5108 (time 1770688002.38 total 60.04 last 90.33) - /unlink 6008 (time 1770688012.39 total 70.05 last 89.92) - /unlink 6962 (time 1770688022.40 total 80.06 last 95.36) - /unlink 7935 (time 1770688032.40 total 90.06 last 97.28) - /unlink 8836 (time 1770688042.40 total 100.06 last 90.04) - /unlink 9734 (time 1770688052.41 total 110.07 last 89.72) - /unlink 10000 (time 1770688055.21 total 112.87 last 95.01) - /unlink 10858 (time 1770688065.22 total 122.88 last 85.74) - /unlink 11681 (time 1770688075.23 total 132.89 last 82.23) - /unlink 12457 (time 1770688085.23 total 142.89 last 77.60) - /unlink 13245 (time 1770688095.24 total 152.90 last 78.74) - /unlink 14096 (time 1770688105.24 total 162.90 last 85.05) - /unlink 15003 (time 1770688115.25 total 172.91 last 90.66) - /unlink 15898 (time 1770688125.25 total 182.91 last 89.44) - /unlink 16706 (time 1770688135.25 total 192.91 last 80.80) - /unlink 17504 (time 1770688145.25 total 202.91 last 79.79) - /unlink 18319 (time 1770688155.26 total 212.92 last 81.47) - /unlink 19246 (time 1770688165.28 total 222.94 last 92.53) - /unlink 20000 (time 1770688173.67 total 231.33 last 89.78) - /unlink 21019 (time 1770688183.67 total 241.33 last 101.89) - /unlink 22005 (time 1770688193.68 total 251.34 last 98.51) - /unlink 23093 (time 1770688203.69 total 261.35 last 108.77) - /unlink 24273 (time 1770688213.69 total 271.35 last 117.96) - /unlink 25930 (time 1770688223.69 total 281.35 last 165.67) - /unlink 27611 (time 1770688233.70 total 291.37 last 167.88) - /unlink 28821 (time 1770688243.71 total 301.37 last 120.99) - /unlink 29998 (time 1770688253.72 total 311.38 last 117.59) - /unlink 31490 (time 1770688263.72 total 321.38 last 149.18) - /unlink 32903 (time 1770688273.72 total 331.38 last 141.22) - /unlink 34173 (time 1770688283.72 total 341.38 last 126.98) - /unlink 35213 (time 1770688293.72 total 351.38 last 104.00) - /unlink 36271 (time 1770688303.73 total 361.39 last 105.76) - /unlink 37573 (time 1770688313.73 total 371.39 last 130.14) - /unlink 38951 (time 1770688323.74 total 381.40 last 137.73) - /unlink 40000 (time 1770688332.93 total 390.59 last 114.13) - /unlink 41226 (time 1770688342.94 total 400.60 last 122.51) - /unlink 42609 (time 1770688352.94 total 410.60 last 138.20) - /unlink 43879 (time 1770688362.95 total 420.61 last 126.95) - /unlink 45028 (time 1770688372.95 total 430.61 last 114.89) - /unlink 46327 (time 1770688382.96 total 440.62 last 129.79) - /unlink 47699 (time 1770688392.96 total 450.62 last 137.18) - /unlink 48809 (time 1770688402.96 total 460.62 last 110.95) - /unlink 50000 (time 1770688412.48 total 470.14 last 125.10) - /unlink 51361 (time 1770688422.49 total 480.15 last 136.07) - /unlink 52684 (time 1770688432.49 total 490.15 last 132.25) - /unlink 53775 (time 1770688442.51 total 500.17 last 108.90) - /unlink 54847 (time 1770688452.51 total 510.17 last 107.15) - /unlink 56069 (time 1770688462.52 total 520.18 last 122.09) - /unlink 57697 (time 1770688472.53 total 530.19 last 162.73) - /unlink 58916 (time 1770688482.53 total 540.19 last 121.90) - /unlink 60000 (time 1770688492.25 total 549.91 last 111.46) - /unlink 61157 (time 1770688502.27 total 559.93 last 115.48) - /unlink 62439 (time 1770688512.28 total 569.94 last 128.06) - /unlink 64463 (time 1770688522.28 total 579.94 last 202.36) total: 64768 /unlink in 581.23 seconds: 111.43 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1436 (time 1770688553.77 total 10.01 last 143.49) - open/close 2600 (time 1770688563.78 total 20.02 last 116.24) - open/close 3779 (time 1770688573.79 total 30.03 last 117.85) - open/close 5041 (time 1770688583.80 total 40.04 last 126.08) - open/close 6181 (time 1770688593.80 total 50.04 last 113.98) - open/close 7666 (time 1770688603.80 total 60.04 last 148.45) - open/close 9204 (time 1770688613.81 total 70.04 last 153.73) - open/close 10000 (time 1770688619.41 total 75.65 last 142.09) - open/close 11384 (time 1770688629.41 total 85.65 last 138.32) - open/close 13127 (time 1770688639.42 total 95.65 last 174.29) - open/close 14762 (time 1770688649.42 total 105.66 last 163.40) - open/close 16272 (time 1770688659.42 total 115.66 last 151.00) - open/close 18101 (time 1770688669.42 total 125.66 last 182.86) - open/close 20000 (time 1770688679.33 total 135.56 last 191.78) - open/close 21613 (time 1770688689.33 total 145.57 last 161.25) - open/close 23087 (time 1770688699.33 total 155.57 last 147.34) - open/close 24782 (time 1770688709.34 total 165.58 last 169.39) - open/close 26153 (time 1770688719.34 total 175.58 last 137.04) - open/close 27373 (time 1770688729.35 total 185.58 last 121.97) - open/close 28670 (time 1770688739.35 total 195.59 last 129.63) - open/close 29917 (time 1770688749.35 total 205.59 last 124.67) - open/close 31674 (time 1770688759.36 total 215.59 last 175.66) - open/close 33230 (time 1770688769.36 total 225.59 last 155.58) - open/close 34767 (time 1770688779.36 total 235.60 last 153.64) - open/close 36727 (time 1770688789.36 total 245.60 last 195.98) - open/close 38337 (time 1770688799.37 total 255.61 last 160.91) - open/close 39828 (time 1770688809.37 total 265.61 last 149.02) - open/close 41791 (time 1770688819.38 total 275.62 last 196.21) - open/close 43430 (time 1770688829.38 total 285.62 last 163.88) - open/close 45008 (time 1770688839.38 total 295.62 last 157.77) - open/close 47108 (time 1770688849.38 total 305.62 last 209.95) - open/close 49070 (time 1770688859.39 total 315.62 last 196.13) - open/close 50000 (time 1770688863.55 total 319.79 last 223.31) - open/close 51882 (time 1770688873.55 total 329.79 last 188.17) - open/close 54032 (time 1770688883.56 total 339.79 last 214.94) - open/close 55955 (time 1770688893.56 total 349.80 last 192.26) - open/close 58060 (time 1770688903.56 total 359.80 last 210.47) - open/close 60000 (time 1770688913.16 total 369.40 last 202.03) - open/close 62534 (time 1770688923.16 total 379.40 last 253.33) total: 64768 open/close in 387.48 seconds: 167.15 ops/second - /unlink 5019 (time 1770688943.13 total 10.00 last 501.84) - /unlink 10000 (time 1770688952.76 total 19.63 last 517.53) - /unlink 15292 (time 1770688962.76 total 29.63 last 529.19) - /unlink 19961 (time 1770688972.76 total 39.63 last 466.67) - /unlink 24620 (time 1770688982.76 total 49.63 last 465.84) - /unlink 29852 (time 1770688992.76 total 59.63 last 523.17) - /unlink 35397 (time 1770689002.76 total 69.63 last 554.50) - /unlink 40000 (time 1770689010.78 total 77.65 last 574.08) - /unlink 45425 (time 1770689020.78 total 87.65 last 542.49) - /unlink 50000 (time 1770689029.91 total 96.78 last 500.96) - /unlink 54939 (time 1770689039.92 total 106.79 last 493.51) - /unlink 59357 (time 1770689049.92 total 116.79 last 441.73) - /unlink 64555 (time 1770689059.92 total 126.79 last 519.79) total: 64768 /unlink in 127.12 seconds: 509.51 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3911 (time 1770689090.84 total 10.00 last 391.02) - open/close 7521 (time 1770689100.84 total 20.00 last 360.98) - open/close 10000 (time 1770689108.63 total 27.79 last 318.25) - open/close 13422 (time 1770689118.63 total 37.79 last 342.13) - open/close 16669 (time 1770689128.63 total 47.80 last 324.66) - open/close 20000 (time 1770689138.26 total 57.42 last 345.94) - open/close 24196 (time 1770689148.26 total 67.42 last 419.59) - open/close 28400 (time 1770689158.26 total 77.43 last 420.36) - open/close 30000 (time 1770689162.63 total 81.80 last 366.10) - open/close 33810 (time 1770689172.63 total 91.80 last 381.00) - open/close 37390 (time 1770689182.63 total 101.80 last 357.98) - open/close 40000 (time 1770689191.01 total 110.17 last 311.54) - open/close 42393 (time 1770689201.01 total 120.18 last 239.24) - open/close 45452 (time 1770689211.02 total 130.18 last 305.79) - open/close 48717 (time 1770689221.02 total 140.18 last 326.44) - open/close 50000 (time 1770689225.73 total 144.89 last 272.57) - open/close 53575 (time 1770689235.73 total 154.89 last 357.48) - open/close 56598 (time 1770689245.73 total 164.89 last 302.25) - open/close 59826 (time 1770689255.73 total 174.89 last 322.70) - open/close 62899 (time 1770689265.73 total 184.89 last 307.30) total: 64768 open/close in 190.28 seconds: 340.38 ops/second - /unlink 6467 (time 1770689282.54 total 10.00 last 646.67) - /unlink 10000 (time 1770689289.88 total 17.34 last 481.23) - /unlink 15003 (time 1770689299.88 total 27.34 last 500.23) - /unlink 20000 (time 1770689308.80 total 36.26 last 560.18) - /unlink 25219 (time 1770689318.80 total 46.26 last 521.86) - /unlink 30000 (time 1770689328.07 total 55.53 last 516.07) - /unlink 35731 (time 1770689338.08 total 65.54 last 572.70) - /unlink 40000 (time 1770689345.22 total 72.68 last 597.32) - /unlink 45830 (time 1770689355.22 total 82.68 last 582.96) - /unlink 50000 (time 1770689362.78 total 90.24 last 551.67) - /unlink 54982 (time 1770689372.79 total 100.25 last 497.78) - /unlink 60000 (time 1770689382.45 total 109.91 last 519.53) total: 64768 /unlink in 117.71 seconds: 550.23 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3136 (time 1770689419.06 total 10.00 last 313.57) - open/close 6200 (time 1770689429.07 total 20.01 last 306.27) - open/close 9691 (time 1770689439.07 total 30.01 last 349.06) - open/close 13285 (time 1770689449.07 total 40.01 last 359.39) - open/close 17128 (time 1770689459.07 total 50.01 last 384.24) - open/close 20000 (time 1770689467.14 total 58.08 last 355.79) - open/close 22728 (time 1770689477.14 total 68.08 last 272.79) - open/close 25899 (time 1770689487.14 total 78.08 last 317.05) - open/close 28853 (time 1770689497.14 total 88.09 last 295.32) - open/close 30000 (time 1770689501.72 total 92.66 last 250.47) - open/close 33367 (time 1770689511.73 total 102.67 last 336.65) - open/close 37102 (time 1770689521.73 total 112.67 last 373.41) - open/close 40000 (time 1770689529.48 total 120.42 last 373.89) - open/close 43335 (time 1770689539.48 total 130.42 last 333.47) - open/close 47136 (time 1770689549.48 total 140.42 last 380.01) - open/close 50000 (time 1770689557.92 total 148.86 last 339.42) - open/close 52486 (time 1770689567.92 total 158.86 last 248.53) - open/close 55222 (time 1770689577.92 total 168.86 last 273.57) - open/close 57700 (time 1770689587.96 total 178.90 last 246.89) - open/close 59665 (time 1770689597.97 total 188.91 last 196.40) - open/close 61658 (time 1770689607.97 total 198.91 last 199.27) - open/close 63485 (time 1770689617.97 total 208.91 last 182.63) total: 64768 open/close in 214.64 seconds: 301.75 ops/second - /unlink 4529 (time 1770689635.50 total 10.00 last 452.88) - /unlink 8330 (time 1770689645.50 total 20.00 last 380.06) - /unlink 10000 (time 1770689649.81 total 24.32 last 386.87) - /unlink 13779 (time 1770689659.82 total 34.32 last 377.80) - /unlink 17696 (time 1770689669.82 total 44.33 last 391.44) - /unlink 20000 (time 1770689674.55 total 49.05 last 487.85) - /unlink 24932 (time 1770689684.55 total 59.05 last 493.13) - /unlink 28955 (time 1770689694.55 total 69.05 last 402.24) - /unlink 30000 (time 1770689697.56 total 72.06 last 347.44) - /unlink 34987 (time 1770689707.56 total 82.06 last 498.63) - /unlink 39354 (time 1770689717.56 total 92.06 last 436.68) - /unlink 43944 (time 1770689727.56 total 102.06 last 458.97) - /unlink 49284 (time 1770689737.56 total 112.06 last 533.97) - /unlink 54855 (time 1770689747.56 total 122.06 last 557.05) - /unlink 59011 (time 1770689757.56 total 132.07 last 415.57) - /unlink 60000 (time 1770689759.69 total 134.20 last 463.47) - /unlink 64355 (time 1770689769.69 total 144.20 last 435.48) total: 64768 /unlink in 144.89 seconds: 447.03 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg104-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing unload_modules_local oleg104-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg104-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2