== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:30:32 (1773941432) Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg260-client.virtnet /mnt/lustre2 (opts:-f) oleg260-server: oleg260-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg260-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4482 (time 1773941477.47 total 10.00 last 448.15) - open/close 8884 (time 1773941487.47 total 20.00 last 440.12) - open/close 10000 (time 1773941490.35 total 22.88 last 387.31) - open/close 14573 (time 1773941500.35 total 32.89 last 457.26) - open/close 19097 (time 1773941510.35 total 42.89 last 452.33) - open/close 20000 (time 1773941512.37 total 44.91 last 446.52) - open/close 23976 (time 1773941522.39 total 54.92 last 397.15) - open/close 28181 (time 1773941532.39 total 64.92 last 420.43) - open/close 30000 (time 1773941536.25 total 68.79 last 470.33) - open/close 34314 (time 1773941546.26 total 78.79 last 431.36) - open/close 38753 (time 1773941556.26 total 88.79 last 443.88) - open/close 40000 (time 1773941559.17 total 91.71 last 427.31) - open/close 44395 (time 1773941569.17 total 101.71 last 439.49) - open/close 48354 (time 1773941579.18 total 111.71 last 395.80) - open/close 50000 (time 1773941585.16 total 117.69 last 275.19) - open/close 53026 (time 1773941595.16 total 127.69 last 302.56) - open/close 56427 (time 1773941605.16 total 137.70 last 340.03) - open/close 60000 (time 1773941614.38 total 146.92 last 387.52) - open/close 63024 (time 1773941624.38 total 156.92 last 302.35) total: 64768 open/close in 162.74 seconds: 397.99 ops/second - /unlink 5246 (time 1773941641.87 total 10.00 last 524.54) - /unlink 9586 (time 1773941651.87 total 20.00 last 433.95) - /unlink 14961 (time 1773941661.87 total 30.00 last 537.47) - /unlink 20000 (time 1773941671.11 total 39.24 last 545.59) - /unlink 24742 (time 1773941681.11 total 49.24 last 474.14) - /unlink 30000 (time 1773941690.39 total 58.52 last 566.46) - /unlink 35006 (time 1773941700.39 total 68.52 last 500.57) - /unlink 40000 (time 1773941709.63 total 77.77 last 540.30) - /unlink 45473 (time 1773941719.64 total 87.77 last 547.26) - /unlink 50000 (time 1773941728.20 total 96.33 last 528.89) - /unlink 55879 (time 1773941738.20 total 106.33 last 587.83) - /unlink 60000 (time 1773941746.66 total 114.80 last 486.65) - /unlink 64515 (time 1773941756.67 total 124.80 last 451.39) total: 64768 /unlink in 125.48 seconds: 516.16 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3234 (time 1773941791.11 total 10.00 last 323.39) - open/close 6402 (time 1773941801.11 total 20.00 last 316.75) - open/close 9608 (time 1773941811.11 total 30.00 last 320.59) - open/close 12540 (time 1773941821.11 total 40.00 last 293.19) - open/close 15323 (time 1773941831.11 total 50.00 last 278.27) - open/close 17998 (time 1773941841.11 total 60.00 last 267.46) - open/close 20000 (time 1773941847.73 total 66.62 last 302.67) - open/close 22801 (time 1773941857.73 total 76.62 last 280.01) - open/close 25244 (time 1773941867.73 total 86.62 last 244.25) - open/close 27541 (time 1773941877.74 total 96.63 last 229.67) - open/close 29901 (time 1773941887.74 total 106.63 last 235.98) - open/close 32274 (time 1773941897.74 total 116.63 last 237.26) - open/close 35109 (time 1773941907.74 total 126.63 last 283.50) - open/close 37965 (time 1773941917.74 total 136.63 last 285.55) - open/close 40000 (time 1773941924.85 total 143.74 last 286.12) - open/close 42523 (time 1773941934.85 total 153.74 last 252.28) - open/close 45000 (time 1773941944.85 total 163.74 last 247.65) - open/close 47786 (time 1773941954.86 total 173.75 last 278.58) - open/close 50000 (time 1773941962.18 total 181.07 last 302.13) - open/close 53028 (time 1773941972.18 total 191.07 last 302.77) - open/close 56238 (time 1773941982.19 total 201.08 last 320.96) - open/close 58731 (time 1773941992.19 total 211.08 last 249.25) - open/close 60000 (time 1773941996.94 total 215.84 last 266.74) - open/close 62554 (time 1773942006.95 total 225.84 last 255.32) total: 64768 open/close in 234.39 seconds: 276.33 ops/second - /unlink 6222 (time 1773942027.14 total 10.00 last 622.17) - /unlink 10000 (time 1773942034.34 total 17.19 last 525.34) - /unlink 14920 (time 1773942044.34 total 27.19 last 491.97) - /unlink 20000 (time 1773942054.22 total 37.08 last 513.89) - /unlink 25443 (time 1773942064.22 total 47.08 last 544.23) - /unlink 30000 (time 1773942073.23 total 56.08 last 506.05) - /unlink 35277 (time 1773942083.23 total 66.08 last 527.69) - /unlink 40000 (time 1773942092.73 total 75.59 last 496.91) - /unlink 45097 (time 1773942102.73 total 85.59 last 509.64) - /unlink 49875 (time 1773942112.74 total 95.60 last 477.39) - /unlink 54179 (time 1773942122.74 total 105.60 last 430.32) - /unlink 58742 (time 1773942132.74 total 115.60 last 456.27) - /unlink 60000 (time 1773942135.76 total 118.62 last 417.08) total: 64768 /unlink in 127.97 seconds: 506.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2358 (time 1773942173.13 total 10.01 last 235.67) - open/close 5314 (time 1773942183.13 total 20.01 last 295.59) - open/close 7963 (time 1773942193.13 total 30.01 last 264.83) - open/close 10000 (time 1773942201.58 total 38.46 last 241.10) - open/close 12482 (time 1773942211.58 total 48.46 last 248.11) - open/close 14747 (time 1773942221.58 total 58.46 last 226.50) - open/close 17115 (time 1773942231.58 total 68.46 last 236.78) - open/close 19394 (time 1773942241.58 total 78.46 last 227.87) - open/close 20000 (time 1773942243.59 total 80.47 last 301.85) - open/close 22808 (time 1773942253.59 total 90.47 last 280.76) - open/close 25853 (time 1773942263.59 total 100.47 last 304.45) - open/close 29650 (time 1773942273.59 total 110.48 last 379.66) - open/close 32146 (time 1773942283.60 total 120.48 last 249.59) - open/close 34877 (time 1773942293.60 total 130.48 last 273.07) - open/close 38826 (time 1773942303.60 total 140.48 last 394.88) - open/close 40000 (time 1773942308.30 total 145.18 last 249.67) - open/close 42323 (time 1773942318.30 total 155.18 last 232.29) - open/close 45746 (time 1773942328.30 total 165.18 last 342.28) - open/close 50000 (time 1773942338.14 total 175.02 last 432.22) - open/close 54351 (time 1773942348.14 total 185.02 last 435.06) - open/close 58112 (time 1773942358.15 total 195.03 last 375.96) - open/close 60000 (time 1773942365.82 total 202.70 last 245.98) - open/close 63702 (time 1773942375.82 total 212.70 last 370.16) total: 64768 open/close in 215.52 seconds: 300.51 ops/second - /unlink 6802 (time 1773942390.54 total 10.00 last 680.17) - /unlink 10000 (time 1773942395.95 total 15.41 last 590.86) - /unlink 17341 (time 1773942405.95 total 25.41 last 734.06) - /unlink 20000 (time 1773942409.74 total 29.21 last 701.12) - /unlink 26797 (time 1773942419.74 total 39.21 last 679.67) - /unlink 30000 (time 1773942424.30 total 43.76 last 703.19) - /unlink 35470 (time 1773942434.30 total 53.76 last 546.97) - /unlink 40000 (time 1773942441.73 total 61.20 last 609.38) - /unlink 46179 (time 1773942451.73 total 71.20 last 617.79) - /unlink 50000 (time 1773942459.59 total 79.05 last 486.65) - /unlink 55177 (time 1773942469.59 total 89.05 last 517.65) - /unlink 60000 (time 1773942479.18 total 98.65 last 502.65) total: 64768 /unlink in 108.24 seconds: 598.40 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2685 (time 1773942516.06 total 10.00 last 268.45) - open/close 5492 (time 1773942526.06 total 20.00 last 280.67) - open/close 8135 (time 1773942536.07 total 30.00 last 264.29) - open/close 10000 (time 1773942542.98 total 36.92 last 269.54) - open/close 12617 (time 1773942552.99 total 46.93 last 261.62) - open/close 14949 (time 1773942562.99 total 56.93 last 233.14) - open/close 17779 (time 1773942573.01 total 66.95 last 282.36) - open/close 20000 (time 1773942582.30 total 76.24 last 239.11) - open/close 22638 (time 1773942592.30 total 86.24 last 263.72) - open/close 25417 (time 1773942602.31 total 96.25 last 277.79) - open/close 27793 (time 1773942612.31 total 106.25 last 237.52) - open/close 30000 (time 1773942621.53 total 115.47 last 239.30) - open/close 32399 (time 1773942631.54 total 125.47 last 239.87) - open/close 35100 (time 1773942641.54 total 135.47 last 270.09) - open/close 38097 (time 1773942651.54 total 145.48 last 299.65) - open/close 40000 (time 1773942658.78 total 152.72 last 262.75) - open/close 42757 (time 1773942668.78 total 162.72 last 275.61) - open/close 45279 (time 1773942678.79 total 172.73 last 252.10) - open/close 47839 (time 1773942688.79 total 182.73 last 255.90) - open/close 50000 (time 1773942696.89 total 190.82 last 267.01) - open/close 52825 (time 1773942706.89 total 200.83 last 282.40) - open/close 55351 (time 1773942716.90 total 210.83 last 252.38) - open/close 57723 (time 1773942726.98 total 220.91 last 235.35) - open/close 60000 (time 1773942736.27 total 230.20 last 245.09) - open/close 62347 (time 1773942746.27 total 240.21 last 234.64) - open/close 64503 (time 1773942756.27 total 250.21 last 215.54) total: 64768 open/close in 251.53 seconds: 257.50 ops/second - /unlink 4453 (time 1773942769.67 total 10.00 last 445.21) - /unlink 8842 (time 1773942779.67 total 20.00 last 438.87) - /unlink 10000 (time 1773942781.95 total 22.27 last 509.62) - /unlink 14512 (time 1773942791.95 total 32.28 last 451.14) - /unlink 18525 (time 1773942801.95 total 42.28 last 401.23) - /unlink 20000 (time 1773942805.31 total 45.64 last 439.14) - /unlink 24469 (time 1773942815.31 total 55.64 last 446.88) - /unlink 29417 (time 1773942825.31 total 65.64 last 494.75) - /unlink 33980 (time 1773942835.31 total 75.64 last 456.28) - /unlink 38167 (time 1773942845.31 total 85.64 last 418.70) - /unlink 40000 (time 1773942850.16 total 90.49 last 377.86) - /unlink 44368 (time 1773942860.16 total 100.49 last 436.78) - /unlink 48541 (time 1773942870.16 total 110.49 last 417.29) - /unlink 50000 (time 1773942873.04 total 113.37 last 506.80) - /unlink 54587 (time 1773942883.04 total 123.37 last 458.62) - /unlink 59196 (time 1773942893.04 total 133.37 last 460.88) - /unlink 64496 (time 1773942903.04 total 143.37 last 529.91) total: 64768 /unlink in 143.87 seconds: 450.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1936 (time 1773942936.50 total 10.00 last 193.55) - open/close 3970 (time 1773942946.50 total 20.00 last 203.35) - open/close 4951 (time 1773942956.51 total 30.01 last 98.05) - open/close 7586 (time 1773942966.51 total 40.01 last 263.50) - open/close 10000 (time 1773942972.57 total 46.07 last 398.03) - open/close 14524 (time 1773942982.57 total 56.08 last 452.34) - open/close 18893 (time 1773942992.57 total 66.08 last 436.85) - open/close 20000 (time 1773942996.08 total 69.58 last 315.63) - open/close 23082 (time 1773943006.08 total 79.59 last 308.14) - open/close 27191 (time 1773943016.08 total 89.59 last 410.88) - open/close 30000 (time 1773943022.64 total 96.14 last 428.54) - open/close 34839 (time 1773943032.64 total 106.14 last 483.84) - open/close 39460 (time 1773943042.64 total 116.14 last 462.06) - open/close 44086 (time 1773943052.64 total 126.15 last 462.52) - open/close 47533 (time 1773943062.64 total 136.15 last 344.68) - open/close 50000 (time 1773943069.42 total 142.93 last 363.91) - open/close 53618 (time 1773943079.42 total 152.93 last 361.73) - open/close 57845 (time 1773943089.43 total 162.93 last 422.68) - open/close 60000 (time 1773943095.54 total 169.04 last 352.45) - open/close 63717 (time 1773943105.54 total 179.04 last 371.68) total: 64768 open/close in 182.16 seconds: 355.56 ops/second - /unlink 6369 (time 1773943120.03 total 10.00 last 636.90) - /unlink 10000 (time 1773943125.50 total 15.47 last 663.71) - /unlink 17120 (time 1773943135.50 total 25.47 last 711.92) - /unlink 20000 (time 1773943139.44 total 29.41 last 731.16) - /unlink 26887 (time 1773943149.44 total 39.41 last 688.69) - /unlink 30000 (time 1773943154.06 total 44.03 last 673.44) - /unlink 36595 (time 1773943164.06 total 54.03 last 659.42) - /unlink 40000 (time 1773943171.46 total 61.44 last 459.86) - /unlink 44558 (time 1773943181.47 total 71.44 last 455.76) - /unlink 49381 (time 1773943191.47 total 81.44 last 482.26) - /unlink 54851 (time 1773943201.47 total 91.44 last 546.87) - /unlink 59034 (time 1773943211.47 total 101.44 last 418.25) - /unlink 60000 (time 1773943213.63 total 103.60 last 447.44) - /unlink 64431 (time 1773943223.63 total 113.60 last 443.03) total: 64768 /unlink in 114.37 seconds: 566.28 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server unloading modules via unload_modules_local on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local oleg260-server: modules unloaded. pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2