== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:46:58 (1758808018) Stopping clients: oleg234-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg234-client.virtnet /mnt/lustre2 (opts:-f) oleg234-server: oleg234-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg234-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg234-server: detected 4 online CPUs by sysfs oleg234-server: MODOPTS_LIBCFS= oleg234-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg234-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1914 (time 1758808071.56 total 10.00 last 191.39) - open/close 3795 (time 1758808081.56 total 20.00 last 188.04) - open/close 5731 (time 1758808091.56 total 30.01 last 193.56) - open/close 7381 (time 1758808101.57 total 40.01 last 164.95) - open/close 8832 (time 1758808111.57 total 50.01 last 145.02) - open/close 10000 (time 1758808120.73 total 59.17 last 127.56) - open/close 11188 (time 1758808130.73 total 69.18 last 118.74) - open/close 12284 (time 1758808140.74 total 79.18 last 109.57) - open/close 13475 (time 1758808150.74 total 89.18 last 119.05) - open/close 14752 (time 1758808160.74 total 99.19 last 127.64) - open/close 16256 (time 1758808170.75 total 109.19 last 150.34) - open/close 17848 (time 1758808180.75 total 119.20 last 159.13) - open/close 19374 (time 1758808190.76 total 129.20 last 152.50) - open/close 20000 (time 1758808195.53 total 133.97 last 131.17) - open/close 21564 (time 1758808205.54 total 143.98 last 156.34) - open/close 23376 (time 1758808215.54 total 153.98 last 181.14) - open/close 25587 (time 1758808225.54 total 163.99 last 221.02) - open/close 27894 (time 1758808235.54 total 173.99 last 230.64) - open/close 29844 (time 1758808245.55 total 183.99 last 194.96) - open/close 31843 (time 1758808255.55 total 194.00 last 199.76) - open/close 33695 (time 1758808265.55 total 204.00 last 185.19) - open/close 35428 (time 1758808275.56 total 214.00 last 173.24) - open/close 37079 (time 1758808285.57 total 224.01 last 164.87) - open/close 38385 (time 1758808295.58 total 234.02 last 130.54) - open/close 39919 (time 1758808305.58 total 244.03 last 153.30) - open/close 41715 (time 1758808315.59 total 254.03 last 179.46) - open/close 43203 (time 1758808325.60 total 264.05 last 148.60) - open/close 44918 (time 1758808335.61 total 274.05 last 171.43) - open/close 46468 (time 1758808345.61 total 284.05 last 154.98) - open/close 47812 (time 1758808355.61 total 294.06 last 134.33) - open/close 49309 (time 1758808365.62 total 304.06 last 149.63) - open/close 50000 (time 1758808369.39 total 307.83 last 183.23) - open/close 51563 (time 1758808379.39 total 317.83 last 156.29) - open/close 53314 (time 1758808389.39 total 327.84 last 175.05) - open/close 54969 (time 1758808399.40 total 337.84 last 165.46) - open/close 56628 (time 1758808409.40 total 347.84 last 165.82) - open/close 58452 (time 1758808419.41 total 357.86 last 182.19) - open/close 59915 (time 1758808429.42 total 367.86 last 146.20) - open/close 60922 (time 1758808439.42 total 377.87 last 100.65) - open/close 62111 (time 1758808449.45 total 387.89 last 118.62) - open/close 63555 (time 1758808459.45 total 397.89 last 144.40) total: 64768 open/close in 405.79 seconds: 159.61 ops/second - /unlink 2460 (time 1758808479.34 total 10.00 last 245.97) - /unlink 5162 (time 1758808489.35 total 20.00 last 270.18) - /unlink 8325 (time 1758808499.35 total 30.00 last 316.28) - /unlink 10000 (time 1758808504.68 total 35.34 last 314.02) - /unlink 12566 (time 1758808514.68 total 45.34 last 256.50) - /unlink 15237 (time 1758808524.69 total 55.34 last 267.07) - /unlink 18215 (time 1758808534.69 total 65.34 last 297.76) - /unlink 20000 (time 1758808540.59 total 71.24 last 302.48) - /unlink 23180 (time 1758808550.59 total 81.25 last 317.95) - /unlink 26471 (time 1758808560.59 total 91.25 last 329.07) - /unlink 29693 (time 1758808570.59 total 101.25 last 322.20) - /unlink 33464 (time 1758808580.59 total 111.25 last 377.06) - /unlink 37880 (time 1758808590.59 total 121.25 last 441.60) - /unlink 40000 (time 1758808595.06 total 125.72 last 474.43) - /unlink 44710 (time 1758808605.06 total 135.72 last 470.99) - /unlink 49308 (time 1758808615.06 total 145.72 last 459.70) - /unlink 53668 (time 1758808625.06 total 155.72 last 435.96) - /unlink 58757 (time 1758808635.06 total 165.72 last 508.89) - /unlink 60000 (time 1758808637.94 total 168.59 last 432.87) total: 64768 /unlink in 178.02 seconds: 363.83 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3620 (time 1758808680.07 total 10.00 last 361.96) - open/close 7119 (time 1758808690.07 total 20.00 last 349.84) - open/close 10000 (time 1758808698.65 total 28.58 last 335.76) - open/close 13851 (time 1758808708.65 total 38.58 last 385.09) - open/close 17621 (time 1758808718.66 total 48.59 last 376.94) - open/close 20000 (time 1758808724.93 total 54.86 last 378.91) - open/close 23460 (time 1758808734.94 total 64.87 last 345.95) - open/close 27241 (time 1758808744.94 total 74.87 last 378.09) - open/close 30000 (time 1758808753.20 total 83.13 last 334.03) - open/close 32968 (time 1758808763.20 total 93.13 last 296.79) - open/close 36018 (time 1758808773.20 total 103.13 last 305.00) - open/close 39821 (time 1758808783.20 total 113.13 last 380.23) - open/close 42981 (time 1758808793.20 total 123.13 last 315.99) - open/close 46397 (time 1758808803.20 total 133.13 last 341.57) - open/close 50000 (time 1758808812.50 total 142.43 last 387.56) - open/close 53743 (time 1758808822.50 total 152.43 last 374.27) - open/close 57450 (time 1758808832.50 total 162.43 last 370.63) - open/close 60000 (time 1758808840.33 total 170.26 last 325.65) - open/close 63392 (time 1758808850.33 total 180.26 last 339.16) total: 64768 open/close in 184.03 seconds: 351.93 ops/second - /unlink 6157 (time 1758808865.55 total 10.00 last 615.66) - /unlink 10000 (time 1758808871.93 total 16.38 last 602.60) - /unlink 15983 (time 1758808881.93 total 26.38 last 598.28) - /unlink 20000 (time 1758808888.02 total 32.46 last 660.24) - /unlink 26065 (time 1758808898.02 total 42.46 last 606.40) - /unlink 30000 (time 1758808904.85 total 49.29 last 576.19) - /unlink 35626 (time 1758808914.85 total 59.29 last 562.55) - /unlink 40000 (time 1758808922.94 total 67.39 last 540.23) - /unlink 45770 (time 1758808932.95 total 77.39 last 576.97) - /unlink 50000 (time 1758808940.22 total 84.67 last 581.55) - /unlink 55571 (time 1758808950.22 total 94.67 last 557.06) - /unlink 60000 (time 1758808957.91 total 102.36 last 575.62) total: 64768 /unlink in 109.14 seconds: 593.42 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4865 (time 1758808994.76 total 10.00 last 486.47) - open/close 9253 (time 1758809004.76 total 20.00 last 438.71) - open/close 13307 (time 1758809014.76 total 30.00 last 405.39) - open/close 17632 (time 1758809024.76 total 40.00 last 432.42) - open/close 20000 (time 1758809030.62 total 45.87 last 403.80) - open/close 23892 (time 1758809040.63 total 55.87 last 389.11) - open/close 28271 (time 1758809050.63 total 65.87 last 437.88) - open/close 30000 (time 1758809054.06 total 69.30 last 503.79) - open/close 34687 (time 1758809064.06 total 79.30 last 468.68) - open/close 39006 (time 1758809074.06 total 89.30 last 431.89) - open/close 40000 (time 1758809076.28 total 91.52 last 448.49) - open/close 43693 (time 1758809086.28 total 101.52 last 369.27) - open/close 47304 (time 1758809096.28 total 111.52 last 361.06) - open/close 50000 (time 1758809104.51 total 119.76 last 327.34) - open/close 52692 (time 1758809114.52 total 129.76 last 269.15) - open/close 55524 (time 1758809124.52 total 139.76 last 283.13) - open/close 58260 (time 1758809134.52 total 149.77 last 273.52) - open/close 60000 (time 1758809139.95 total 155.20 last 320.39) - open/close 63969 (time 1758809149.95 total 165.20 last 396.87) total: 64768 open/close in 166.90 seconds: 388.06 ops/second - /unlink 7139 (time 1758809163.02 total 10.00 last 713.86) - /unlink 10000 (time 1758809167.62 total 14.60 last 622.19) - /unlink 16200 (time 1758809177.62 total 24.60 last 619.92) - /unlink 20000 (time 1758809184.36 total 31.34 last 563.56) - /unlink 26686 (time 1758809194.36 total 41.34 last 668.52) - /unlink 30000 (time 1758809199.22 total 46.20 last 682.42) - /unlink 36928 (time 1758809209.22 total 56.20 last 692.78) - /unlink 40000 (time 1758809213.39 total 60.37 last 735.96) - /unlink 47771 (time 1758809223.39 total 70.38 last 777.09) - /unlink 50000 (time 1758809226.04 total 73.02 last 843.14) - /unlink 57520 (time 1758809236.04 total 83.02 last 751.95) - /unlink 60000 (time 1758809239.36 total 86.34 last 747.38) total: 64768 /unlink in 92.41 seconds: 700.90 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4803 (time 1758809278.30 total 10.00 last 480.26) - open/close 9589 (time 1758809288.30 total 20.00 last 478.57) - open/close 14437 (time 1758809298.30 total 30.00 last 484.77) - open/close 19444 (time 1758809308.30 total 40.00 last 500.67) - open/close 24667 (time 1758809318.30 total 50.00 last 522.28) - open/close 29958 (time 1758809328.30 total 60.00 last 529.03) - open/close 35154 (time 1758809338.30 total 70.01 last 519.56) - open/close 40000 (time 1758809347.32 total 79.03 last 537.05) - open/close 45108 (time 1758809357.32 total 89.03 last 510.79) - open/close 49932 (time 1758809367.32 total 99.03 last 482.34) - open/close 54814 (time 1758809377.32 total 109.03 last 488.18) - open/close 59580 (time 1758809387.33 total 119.03 last 476.52) - open/close 64152 (time 1758809397.33 total 129.03 last 457.15) total: 64768 open/close in 130.31 seconds: 497.02 ops/second - /unlink 8025 (time 1758809409.94 total 10.00 last 802.47) - /unlink 10000 (time 1758809412.24 total 12.31 last 856.54) - /unlink 17146 (time 1758809422.26 total 22.32 last 713.77) - /unlink 20000 (time 1758809426.37 total 26.43 last 693.51) - /unlink 27750 (time 1758809436.37 total 36.43 last 774.98) - /unlink 30000 (time 1758809438.91 total 38.97 last 886.02) - /unlink 36815 (time 1758809448.91 total 48.97 last 681.48) - /unlink 40000 (time 1758809453.34 total 53.40 last 718.68) - /unlink 47612 (time 1758809463.34 total 63.41 last 761.16) - /unlink 50000 (time 1758809466.41 total 66.48 last 777.58) - /unlink 57712 (time 1758809476.42 total 76.48 last 771.12) - /unlink 60000 (time 1758809479.33 total 79.39 last 785.07) total: 64768 /unlink in 86.57 seconds: 748.11 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4578 (time 1758809517.32 total 10.00 last 457.74) - open/close 9327 (time 1758809527.32 total 20.00 last 474.83) - open/close 14352 (time 1758809537.32 total 30.00 last 502.50) - open/close 19131 (time 1758809547.32 total 40.00 last 477.81) - open/close 24215 (time 1758809557.33 total 50.01 last 508.36) - open/close 29243 (time 1758809567.33 total 60.01 last 502.78) - open/close 34243 (time 1758809577.33 total 70.01 last 499.97) - open/close 38969 (time 1758809587.33 total 80.01 last 472.57) - open/close 43901 (time 1758809597.33 total 90.01 last 493.14) - open/close 48327 (time 1758809607.33 total 100.01 last 442.54) - open/close 50000 (time 1758809610.79 total 103.47 last 482.95) - open/close 54800 (time 1758809620.79 total 113.47 last 479.96) - open/close 59346 (time 1758809630.79 total 123.48 last 454.59) - open/close 63632 (time 1758809640.88 total 133.56 last 424.91) total: 64768 open/close in 136.18 seconds: 475.61 ops/second - /unlink 7396 (time 1758809655.00 total 10.00 last 739.57) - /unlink 10000 (time 1758809658.93 total 13.93 last 662.50) - /unlink 17693 (time 1758809668.93 total 23.93 last 769.28) - /unlink 20000 (time 1758809671.19 total 26.19 last 1019.44) - /unlink 26959 (time 1758809681.19 total 36.19 last 695.85) - /unlink 30000 (time 1758809685.53 total 40.53 last 701.48) - /unlink 37832 (time 1758809695.53 total 50.53 last 783.15) - /unlink 40000 (time 1758809698.53 total 53.53 last 722.32) - /unlink 47455 (time 1758809708.53 total 63.53 last 745.44) - /unlink 50000 (time 1758809711.84 total 66.84 last 768.69) - /unlink 57293 (time 1758809721.84 total 76.84 last 729.28) - /unlink 60000 (time 1758809725.67 total 80.67 last 707.56) total: 64768 /unlink in 86.96 seconds: 744.77 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg234-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2