== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:43:51 (1762166631) Stopping clients: oleg204-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg204-client.virtnet /mnt/lustre2 (opts:-f) oleg204-server: oleg204-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg204-server: detected 4 online CPUs by sysfs oleg204-server: MODOPTS_LIBCFS= oleg204-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg204-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1344 (time 1762166691.55 total 10.00 last 134.37) - open/close 2569 (time 1762166701.56 total 20.01 last 122.46) - open/close 3857 (time 1762166711.56 total 30.01 last 128.76) - open/close 5627 (time 1762166721.56 total 40.01 last 176.95) - open/close 7398 (time 1762166731.57 total 50.02 last 177.00) - open/close 8886 (time 1762166741.57 total 60.02 last 148.77) - open/close 10000 (time 1762166748.93 total 67.38 last 151.36) - open/close 11399 (time 1762166758.94 total 77.38 last 139.83) - open/close 12894 (time 1762166768.94 total 87.39 last 149.45) - open/close 14241 (time 1762166778.94 total 97.39 last 134.67) - open/close 15610 (time 1762166788.95 total 107.40 last 136.79) - open/close 16926 (time 1762166798.95 total 117.40 last 131.57) - open/close 18444 (time 1762166808.96 total 127.40 last 151.73) - open/close 20000 (time 1762166818.07 total 136.51 last 170.83) - open/close 21606 (time 1762166828.07 total 146.51 last 160.60) - open/close 23025 (time 1762166838.07 total 156.52 last 141.86) - open/close 24982 (time 1762166848.07 total 166.52 last 195.68) - open/close 27079 (time 1762166858.08 total 176.53 last 209.52) - open/close 28966 (time 1762166868.08 total 186.53 last 188.67) - open/close 30000 (time 1762166871.50 total 189.94 last 302.56) - open/close 32397 (time 1762166881.50 total 199.95 last 239.59) - open/close 34632 (time 1762166891.50 total 209.95 last 223.50) - open/close 36902 (time 1762166901.50 total 219.95 last 226.95) - open/close 38613 (time 1762166911.51 total 229.95 last 171.05) - open/close 40000 (time 1762166918.33 total 236.78 last 203.32) - open/close 41793 (time 1762166928.33 total 246.78 last 179.29) - open/close 43668 (time 1762166938.33 total 256.78 last 187.47) - open/close 45821 (time 1762166948.33 total 266.78 last 215.23) - open/close 47826 (time 1762166958.34 total 276.79 last 200.42) - open/close 49647 (time 1762166968.35 total 286.80 last 181.88) - open/close 50000 (time 1762166970.97 total 289.42 last 134.53) - open/close 52107 (time 1762166980.98 total 299.42 last 210.65) - open/close 54081 (time 1762166990.98 total 309.43 last 197.37) - open/close 55829 (time 1762167000.98 total 319.43 last 174.70) - open/close 57579 (time 1762167010.98 total 329.43 last 174.99) - open/close 59606 (time 1762167020.98 total 339.43 last 202.69) - open/close 60000 (time 1762167023.49 total 341.94 last 157.43) - open/close 62017 (time 1762167033.49 total 351.94 last 201.68) - open/close 64363 (time 1762167043.49 total 361.94 last 234.54) total: 64768 open/close in 363.47 seconds: 178.19 ops/second - /unlink 3890 (time 1762167056.65 total 10.00 last 389.00) - /unlink 7794 (time 1762167066.65 total 20.00 last 390.38) - /unlink 10000 (time 1762167073.26 total 26.61 last 333.75) - /unlink 13509 (time 1762167083.26 total 36.61 last 350.87) - /unlink 17336 (time 1762167093.26 total 46.61 last 382.70) - /unlink 20000 (time 1762167100.54 total 53.89 last 366.03) - /unlink 23597 (time 1762167110.54 total 63.89 last 359.69) - /unlink 27650 (time 1762167120.54 total 73.89 last 405.26) - /unlink 30000 (time 1762167126.97 total 80.32 last 365.49) - /unlink 34118 (time 1762167136.97 total 90.32 last 411.76) - /unlink 37877 (time 1762167146.97 total 100.32 last 375.88) - /unlink 40000 (time 1762167153.28 total 106.63 last 336.35) - /unlink 43851 (time 1762167163.28 total 116.64 last 385.03) - /unlink 48661 (time 1762167173.28 total 126.64 last 480.99) - /unlink 50000 (time 1762167175.97 total 129.32 last 498.67) - /unlink 54209 (time 1762167185.97 total 139.32 last 420.85) - /unlink 58164 (time 1762167195.97 total 149.32 last 395.41) - /unlink 60000 (time 1762167201.61 total 154.96 last 325.59) - /unlink 63991 (time 1762167211.61 total 164.96 last 399.05) total: 64768 /unlink in 167.10 seconds: 387.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1861 (time 1762167241.81 total 10.01 last 185.99) - open/close 3605 (time 1762167251.81 total 20.01 last 174.40) - open/close 6200 (time 1762167261.82 total 30.01 last 259.43) - open/close 9421 (time 1762167271.82 total 40.01 last 322.02) - open/close 10000 (time 1762167274.03 total 42.22 last 262.37) - open/close 12932 (time 1762167284.03 total 52.22 last 293.09) - open/close 15257 (time 1762167294.03 total 62.22 last 232.44) - open/close 18106 (time 1762167304.04 total 72.23 last 284.83) - open/close 20000 (time 1762167311.74 total 79.93 last 245.98) - open/close 22644 (time 1762167321.74 total 89.93 last 264.32) - open/close 25613 (time 1762167331.74 total 99.93 last 296.82) - open/close 28233 (time 1762167341.74 total 109.94 last 261.92) - open/close 30000 (time 1762167349.93 total 118.12 last 215.79) - open/close 32781 (time 1762167359.97 total 128.16 last 277.11) - open/close 35692 (time 1762167369.97 total 138.16 last 291.04) - open/close 38372 (time 1762167379.97 total 148.16 last 267.95) - open/close 40000 (time 1762167385.49 total 153.68 last 295.24) - open/close 42283 (time 1762167395.49 total 163.68 last 228.22) - open/close 44289 (time 1762167405.49 total 173.68 last 200.58) - open/close 45795 (time 1762167415.50 total 183.69 last 150.49) - open/close 47509 (time 1762167425.50 total 193.69 last 171.34) - open/close 49770 (time 1762167435.50 total 203.70 last 226.04) - open/close 52231 (time 1762167445.51 total 213.70 last 246.05) - open/close 54749 (time 1762167455.51 total 223.70 last 251.74) - open/close 57292 (time 1762167465.51 total 233.70 last 254.25) - open/close 60000 (time 1762167475.22 total 243.41 last 278.93) - open/close 62502 (time 1762167485.22 total 253.41 last 250.17) - open/close 64694 (time 1762167495.22 total 263.41 last 219.17) total: 64768 open/close in 263.71 seconds: 245.60 ops/second - /unlink 4549 (time 1762167507.15 total 10.00 last 454.82) - /unlink 8194 (time 1762167517.16 total 20.00 last 364.43) - /unlink 10000 (time 1762167522.50 total 25.35 last 338.06) - /unlink 13336 (time 1762167532.50 total 35.35 last 333.59) - /unlink 16518 (time 1762167542.50 total 45.35 last 318.18) - /unlink 20000 (time 1762167550.12 total 52.97 last 456.87) - /unlink 24093 (time 1762167560.18 total 63.03 last 406.90) - /unlink 28634 (time 1762167570.18 total 73.03 last 454.06) - /unlink 30000 (time 1762167573.82 total 76.67 last 374.94) - /unlink 34330 (time 1762167583.82 total 86.67 last 432.95) - /unlink 39208 (time 1762167593.83 total 96.67 last 487.71) - /unlink 45286 (time 1762167603.83 total 106.67 last 607.80) - /unlink 50000 (time 1762167610.44 total 113.29 last 712.52) - /unlink 55486 (time 1762167620.44 total 123.29 last 548.57) - /unlink 60000 (time 1762167626.52 total 129.37 last 743.08) total: 64768 /unlink in 135.73 seconds: 477.18 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4219 (time 1762167673.92 total 10.00 last 421.82) - open/close 8167 (time 1762167683.92 total 20.00 last 394.73) - open/close 10000 (time 1762167687.76 total 23.84 last 477.71) - open/close 14075 (time 1762167697.76 total 33.84 last 407.43) - open/close 18373 (time 1762167707.76 total 43.84 last 429.72) - open/close 20000 (time 1762167711.44 total 47.53 last 441.54) - open/close 23898 (time 1762167721.45 total 57.53 last 389.78) - open/close 27939 (time 1762167731.45 total 67.53 last 404.06) - open/close 30000 (time 1762167736.05 total 72.14 last 447.40) - open/close 34087 (time 1762167746.05 total 82.14 last 408.64) - open/close 39019 (time 1762167756.06 total 92.14 last 493.11) - open/close 40000 (time 1762167758.19 total 94.28 last 459.55) - open/close 44333 (time 1762167768.19 total 104.28 last 433.23) - open/close 48481 (time 1762167778.19 total 114.28 last 414.77) - open/close 50000 (time 1762167781.51 total 117.59 last 458.07) - open/close 54810 (time 1762167791.51 total 127.60 last 480.93) - open/close 59590 (time 1762167801.51 total 137.60 last 477.98) - open/close 63838 (time 1762167811.51 total 147.60 last 424.66) total: 64768 open/close in 149.56 seconds: 433.06 ops/second - /unlink 6432 (time 1762167824.85 total 10.00 last 643.16) - /unlink 10000 (time 1762167830.37 total 15.53 last 645.34) - /unlink 16443 (time 1762167840.38 total 25.53 last 644.26) - /unlink 20000 (time 1762167845.49 total 30.64 last 696.00) - /unlink 26902 (time 1762167855.49 total 40.64 last 690.12) - /unlink 30000 (time 1762167860.07 total 45.23 last 675.92) - /unlink 36821 (time 1762167870.07 total 55.23 last 682.05) - /unlink 40000 (time 1762167874.44 total 59.59 last 727.83) - /unlink 46526 (time 1762167884.44 total 69.59 last 652.57) - /unlink 50000 (time 1762167889.42 total 74.58 last 697.16) - /unlink 57856 (time 1762167899.42 total 84.58 last 785.52) - /unlink 60000 (time 1762167902.66 total 87.82 last 662.09) total: 64768 /unlink in 93.61 seconds: 691.87 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4435 (time 1762167938.39 total 10.00 last 443.45) - open/close 9041 (time 1762167948.39 total 20.00 last 460.57) - open/close 14329 (time 1762167958.39 total 30.00 last 528.78) - open/close 19206 (time 1762167968.39 total 40.00 last 487.67) - open/close 24285 (time 1762167978.39 total 50.00 last 507.82) - open/close 28994 (time 1762167988.39 total 60.01 last 470.82) - open/close 30000 (time 1762167990.84 total 62.45 last 411.54) - open/close 34337 (time 1762168000.84 total 72.45 last 433.67) - open/close 39462 (time 1762168010.84 total 82.45 last 512.40) - open/close 44456 (time 1762168020.84 total 92.45 last 499.34) - open/close 46927 (time 1762168030.85 total 102.47 last 246.73) - open/close 47640 (time 1762168040.86 total 112.47 last 71.28) - open/close 48290 (time 1762168050.86 total 122.47 last 64.99) - open/close 48860 (time 1762168060.87 total 132.48 last 56.95) - open/close 49541 (time 1762168070.87 total 142.49 last 68.05) - open/close 50000 (time 1762168076.00 total 147.61 last 89.59) - open/close 51139 (time 1762168086.00 total 157.61 last 113.89) - open/close 52786 (time 1762168096.00 total 167.61 last 164.68) - open/close 54373 (time 1762168106.00 total 177.62 last 158.68) - open/close 56525 (time 1762168116.01 total 187.62 last 215.09) - open/close 58158 (time 1762168126.01 total 197.63 last 163.16) - open/close 59779 (time 1762168136.02 total 207.63 last 162.05) - open/close 61267 (time 1762168146.02 total 217.63 last 148.77) - open/close 63155 (time 1762168156.02 total 227.64 last 188.72) total: 64768 open/close in 236.00 seconds: 274.44 ops/second - /unlink 3760 (time 1762168175.90 total 10.00 last 376.00) - /unlink 7154 (time 1762168185.90 total 20.00 last 339.35) - /unlink 9204 (time 1762168195.90 total 30.00 last 204.96) - /unlink 10000 (time 1762168199.01 total 33.12 last 255.74) - /unlink 12162 (time 1762168209.02 total 43.13 last 215.98) - /unlink 13523 (time 1762168219.02 total 53.13 last 136.08) - /unlink 15059 (time 1762168229.02 total 63.13 last 153.60) - /unlink 16902 (time 1762168239.03 total 73.13 last 184.20) - /unlink 18663 (time 1762168249.03 total 83.13 last 176.08) - /unlink 20000 (time 1762168256.51 total 90.61 last 178.80) - /unlink 21705 (time 1762168266.51 total 100.61 last 170.47) - /unlink 23410 (time 1762168276.51 total 110.61 last 170.49) - /unlink 25180 (time 1762168286.51 total 120.62 last 176.97) - /unlink 26714 (time 1762168296.52 total 130.62 last 153.34) - /unlink 29034 (time 1762168306.52 total 140.62 last 231.99) - /unlink 30000 (time 1762168310.66 total 144.76 last 233.19) - /unlink 33060 (time 1762168320.67 total 154.77 last 305.78) - /unlink 35405 (time 1762168330.67 total 164.77 last 234.43) - /unlink 37619 (time 1762168340.67 total 174.78 last 221.31) - /unlink 39608 (time 1762168350.68 total 184.78 last 198.77) - /unlink 40000 (time 1762168353.01 total 187.12 last 167.95) - /unlink 42045 (time 1762168363.01 total 197.12 last 204.49) - /unlink 43911 (time 1762168373.01 total 207.12 last 186.57) - /unlink 45741 (time 1762168383.01 total 217.12 last 182.99) - /unlink 47372 (time 1762168393.02 total 227.12 last 163.06) - /unlink 49388 (time 1762168403.02 total 237.12 last 201.58) - /unlink 53036 (time 1762168413.02 total 247.12 last 364.75) - /unlink 57181 (time 1762168423.02 total 257.12 last 414.48) - /unlink 59971 (time 1762168433.02 total 267.13 last 278.92) - /unlink 62541 (time 1762168443.03 total 277.13 last 256.90) total: 64768 /unlink in 283.97 seconds: 228.08 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2250 (time 1762168480.60 total 10.00 last 224.94) - open/close 4136 (time 1762168490.61 total 20.01 last 188.55) - open/close 6012 (time 1762168500.61 total 30.01 last 187.54) - open/close 7963 (time 1762168510.61 total 40.01 last 195.06) - open/close 10000 (time 1762168520.37 total 49.77 last 208.71) - open/close 11999 (time 1762168530.37 total 59.77 last 199.84) - open/close 13922 (time 1762168540.38 total 69.78 last 192.26) - open/close 15401 (time 1762168550.38 total 79.78 last 147.80) - open/close 17241 (time 1762168560.39 total 89.79 last 183.88) - open/close 19206 (time 1762168570.39 total 99.79 last 196.44) - open/close 20000 (time 1762168572.85 total 102.25 last 322.46) - open/close 22383 (time 1762168582.86 total 112.26 last 238.27) - open/close 24836 (time 1762168592.86 total 122.26 last 245.11) - open/close 26606 (time 1762168602.87 total 132.27 last 176.95) - open/close 29163 (time 1762168612.87 total 142.27 last 255.66) - open/close 30000 (time 1762168618.01 total 147.41 last 162.74) - open/close 31812 (time 1762168628.02 total 157.42 last 181.12) - open/close 33841 (time 1762168638.02 total 167.42 last 202.81) - open/close 36283 (time 1762168648.02 total 177.42 last 244.18) - open/close 39464 (time 1762168658.02 total 187.42 last 318.07) - open/close 42511 (time 1762168668.02 total 197.42 last 304.66) - open/close 45844 (time 1762168678.02 total 207.42 last 333.27) - open/close 48246 (time 1762168688.03 total 217.43 last 240.12) - open/close 50000 (time 1762168694.33 total 223.73 last 278.14) - open/close 52864 (time 1762168704.33 total 233.73 last 286.37) - open/close 55126 (time 1762168714.33 total 243.73 last 226.19) - open/close 57667 (time 1762168724.34 total 253.74 last 254.06) - open/close 59996 (time 1762168734.34 total 263.74 last 232.83) - open/close 62920 (time 1762168744.34 total 273.74 last 292.31) total: 64768 open/close in 280.16 seconds: 231.18 ops/second - /unlink 3869 (time 1762168762.46 total 10.00 last 386.85) - /unlink 8057 (time 1762168772.47 total 20.00 last 418.72) - /unlink 10000 (time 1762168776.81 total 24.35 last 447.28) - /unlink 13847 (time 1762168786.81 total 34.35 last 384.64) - /unlink 17459 (time 1762168796.81 total 44.35 last 361.19) - /unlink 20000 (time 1762168801.51 total 49.05 last 540.69) - /unlink 24810 (time 1762168811.51 total 59.05 last 480.89) - /unlink 28917 (time 1762168821.51 total 69.05 last 410.67) - /unlink 30000 (time 1762168824.02 total 71.56 last 432.33) - /unlink 34288 (time 1762168834.09 total 81.63 last 425.82) - /unlink 38710 (time 1762168844.09 total 91.63 last 442.20) - /unlink 40000 (time 1762168847.93 total 95.46 last 336.11) - /unlink 43787 (time 1762168857.93 total 105.47 last 378.67) - /unlink 47987 (time 1762168867.93 total 115.47 last 419.88) - /unlink 50000 (time 1762168873.10 total 120.64 last 389.20) - /unlink 54914 (time 1762168883.10 total 130.64 last 491.33) - /unlink 59299 (time 1762168893.11 total 140.64 last 438.44) - /unlink 64431 (time 1762168903.11 total 150.64 last 513.13) total: 64768 /unlink in 151.17 seconds: 428.44 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local oleg204-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg204-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2