== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:09:43 (1758733783) Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:-f) oleg420-server: oleg420-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: MODOPTS_LIBCFS= oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg420-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg420-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 977 (time 1758733860.52 total 10.00 last 97.67) - open/close 1952 (time 1758733870.52 total 20.01 last 97.44) - open/close 2880 (time 1758733880.53 total 30.02 last 92.74) - open/close 3910 (time 1758733890.53 total 40.02 last 102.95) - open/close 4901 (time 1758733900.53 total 50.02 last 99.09) - open/close 6066 (time 1758733910.54 total 60.03 last 116.42) - open/close 7122 (time 1758733920.55 total 70.04 last 105.50) - open/close 8080 (time 1758733930.57 total 80.05 last 95.65) - open/close 9030 (time 1758733940.58 total 90.06 last 94.91) - open/close 9951 (time 1758733950.58 total 100.06 last 92.10) - open/close 10832 (time 1758733960.58 total 110.07 last 88.08) - open/close 11929 (time 1758733970.58 total 120.07 last 109.69) - open/close 12829 (time 1758733980.60 total 130.09 last 89.83) - open/close 13737 (time 1758733990.60 total 140.09 last 90.77) - open/close 14974 (time 1758734000.61 total 150.09 last 123.63) - open/close 16230 (time 1758734010.61 total 160.10 last 125.52) - open/close 17612 (time 1758734020.62 total 170.10 last 138.16) - open/close 18849 (time 1758734030.62 total 180.11 last 123.62) - open/close 20000 (time 1758734037.99 total 187.47 last 156.31) - open/close 21435 (time 1758734047.99 total 197.48 last 143.48) - open/close 22834 (time 1758734057.99 total 207.48 last 139.87) - open/close 24185 (time 1758734068.00 total 217.49 last 134.99) - open/close 25527 (time 1758734078.00 total 227.49 last 134.18) - open/close 27211 (time 1758734088.01 total 237.49 last 168.31) - open/close 28523 (time 1758734098.01 total 247.50 last 131.15) - open/close 29914 (time 1758734108.01 total 257.50 last 139.06) - open/close 31082 (time 1758734118.02 total 267.50 last 116.77) - open/close 32529 (time 1758734128.02 total 277.51 last 144.65) - open/close 34279 (time 1758734138.02 total 287.51 last 174.93) - open/close 35825 (time 1758734148.03 total 297.51 last 154.55) - open/close 37049 (time 1758734158.03 total 307.52 last 122.31) - open/close 38155 (time 1758734168.04 total 317.52 last 110.58) - open/close 39139 (time 1758734178.04 total 327.53 last 98.33) - open/close 40000 (time 1758734185.90 total 335.38 last 109.64) - open/close 41179 (time 1758734195.90 total 345.39 last 117.87) - open/close 42263 (time 1758734205.91 total 355.39 last 108.32) - open/close 43335 (time 1758734215.91 total 365.40 last 107.15) - open/close 44536 (time 1758734225.92 total 375.41 last 119.96) - open/close 45792 (time 1758734235.92 total 385.41 last 125.58) - open/close 47325 (time 1758734245.92 total 395.41 last 153.29) - open/close 48662 (time 1758734255.92 total 405.41 last 133.69) - open/close 49623 (time 1758734265.93 total 415.42 last 96.04) - open/close 50000 (time 1758734270.23 total 419.71 last 87.76) - open/close 51027 (time 1758734280.23 total 429.72 last 102.65) - open/close 52077 (time 1758734290.24 total 439.72 last 104.95) - open/close 53509 (time 1758734300.24 total 449.72 last 143.19) - open/close 55149 (time 1758734310.24 total 459.73 last 163.99) - open/close 56785 (time 1758734320.24 total 469.73 last 163.55) - open/close 58293 (time 1758734330.24 total 479.73 last 150.80) - open/close 59758 (time 1758734340.25 total 489.73 last 146.43) - open/close 61257 (time 1758734350.25 total 499.74 last 149.83) - open/close 62660 (time 1758734360.25 total 509.74 last 140.25) - open/close 64183 (time 1758734370.26 total 519.75 last 152.24) total: 64768 open/close in 523.71 seconds: 123.67 ops/second - /unlink 2974 (time 1758734386.12 total 10.00 last 297.38) - /unlink 5896 (time 1758734396.12 total 20.00 last 292.19) - /unlink 8440 (time 1758734406.13 total 30.00 last 254.34) - /unlink 10000 (time 1758734413.10 total 36.98 last 223.65) - /unlink 13075 (time 1758734423.10 total 46.98 last 307.46) - /unlink 16304 (time 1758734433.11 total 56.98 last 322.82) - /unlink 19139 (time 1758734443.11 total 66.99 last 283.39) - /unlink 20000 (time 1758734446.04 total 69.92 last 293.48) - /unlink 22725 (time 1758734456.05 total 79.92 last 272.43) - /unlink 25384 (time 1758734466.05 total 89.92 last 265.86) - /unlink 28691 (time 1758734476.05 total 99.93 last 330.66) - /unlink 30000 (time 1758734480.57 total 104.45 last 289.47) - /unlink 32939 (time 1758734490.57 total 114.45 last 293.81) - /unlink 36087 (time 1758734500.58 total 124.45 last 314.74) - /unlink 38730 (time 1758734510.58 total 134.45 last 264.25) - /unlink 40000 (time 1758734514.89 total 138.77 last 294.26) - /unlink 43178 (time 1758734524.89 total 148.77 last 317.79) - /unlink 45997 (time 1758734534.89 total 158.77 last 281.87) - /unlink 48843 (time 1758734544.90 total 168.78 last 284.48) - /unlink 50000 (time 1758734548.47 total 172.34 last 324.29) - /unlink 52631 (time 1758734558.47 total 182.34 last 263.08) - /unlink 55048 (time 1758734568.47 total 192.35 last 241.66) - /unlink 57395 (time 1758734578.47 total 202.35 last 234.67) - /unlink 60000 (time 1758734588.10 total 211.97 last 270.63) - /unlink 62305 (time 1758734598.10 total 221.97 last 230.50) - /unlink 64528 (time 1758734608.10 total 231.97 last 222.29) total: 64768 /unlink in 232.97 seconds: 278.01 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1594 (time 1758734640.09 total 10.00 last 159.39) - open/close 3322 (time 1758734650.10 total 20.01 last 172.72) - open/close 4754 (time 1758734660.10 total 30.01 last 143.13) - open/close 6095 (time 1758734670.11 total 40.02 last 134.03) - open/close 7617 (time 1758734680.11 total 50.02 last 152.15) - open/close 9236 (time 1758734690.11 total 60.02 last 161.88) - open/close 10000 (time 1758734695.35 total 65.25 last 145.98) - open/close 11450 (time 1758734705.35 total 75.26 last 144.94) - open/close 13442 (time 1758734715.35 total 85.26 last 199.19) - open/close 15061 (time 1758734725.35 total 95.26 last 161.89) - open/close 16829 (time 1758734735.36 total 105.27 last 176.68) - open/close 18294 (time 1758734745.36 total 115.27 last 146.45) - open/close 19985 (time 1758734755.36 total 125.27 last 169.06) - open/close 21730 (time 1758734765.36 total 135.27 last 174.50) - open/close 23700 (time 1758734775.37 total 145.28 last 196.92) - open/close 25547 (time 1758734785.37 total 155.28 last 184.70) - open/close 27105 (time 1758734795.37 total 165.28 last 155.79) - open/close 28733 (time 1758734805.37 total 175.28 last 162.79) - open/close 30000 (time 1758734813.30 total 183.21 last 159.71) - open/close 31639 (time 1758734823.30 total 193.21 last 163.90) - open/close 33521 (time 1758734833.31 total 203.21 last 188.15) - open/close 35251 (time 1758734843.31 total 213.21 last 173.00) - open/close 37105 (time 1758734853.31 total 223.22 last 185.29) - open/close 38737 (time 1758734863.32 total 233.22 last 163.14) - open/close 40000 (time 1758734870.08 total 239.99 last 186.64) - open/close 41839 (time 1758734880.08 total 249.99 last 183.90) - open/close 43931 (time 1758734890.09 total 259.99 last 209.18) - open/close 45887 (time 1758734900.09 total 270.00 last 195.49) - open/close 48019 (time 1758734910.09 total 280.00 last 213.12) - open/close 50000 (time 1758734918.27 total 288.18 last 242.29) - open/close 52233 (time 1758734928.27 total 298.18 last 223.24) - open/close 54561 (time 1758734938.28 total 308.18 last 232.77) - open/close 56719 (time 1758734948.28 total 318.18 last 215.76) - open/close 58918 (time 1758734958.28 total 328.19 last 219.86) - open/close 60000 (time 1758734964.17 total 334.08 last 183.59) - open/close 61909 (time 1758734974.17 total 344.08 last 190.87) - open/close 64040 (time 1758734984.18 total 354.08 last 213.08) total: 64768 open/close in 357.75 seconds: 181.04 ops/second - /unlink 4183 (time 1758734999.58 total 10.00 last 418.28) - /unlink 8475 (time 1758735009.58 total 20.00 last 429.13) - /unlink 10000 (time 1758735013.27 total 23.69 last 413.58) - /unlink 13205 (time 1758735023.27 total 33.69 last 320.48) - /unlink 17174 (time 1758735033.27 total 43.69 last 396.86) - /unlink 20000 (time 1758735040.88 total 51.31 last 371.09) - /unlink 23678 (time 1758735050.88 total 61.31 last 367.77) - /unlink 28370 (time 1758735060.89 total 71.31 last 469.10) - /unlink 30000 (time 1758735064.35 total 74.77 last 470.47) - /unlink 33982 (time 1758735074.35 total 84.78 last 398.11) - /unlink 39984 (time 1758735084.35 total 94.78 last 600.15) - /unlink 44427 (time 1758735094.36 total 104.78 last 444.22) - /unlink 49429 (time 1758735104.36 total 114.78 last 500.18) - /unlink 54336 (time 1758735114.36 total 124.78 last 490.70) - /unlink 58516 (time 1758735124.36 total 134.78 last 417.92) - /unlink 60000 (time 1758735127.99 total 138.41 last 408.61) total: 64768 /unlink in 147.75 seconds: 438.37 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2744 (time 1758735164.30 total 10.00 last 274.39) - open/close 5706 (time 1758735174.31 total 20.00 last 296.16) - open/close 7935 (time 1758735184.31 total 30.01 last 222.80) - open/close 10000 (time 1758735193.16 total 38.86 last 233.28) - open/close 13439 (time 1758735203.16 total 48.86 last 343.82) - open/close 17625 (time 1758735213.16 total 58.86 last 418.57) - open/close 20000 (time 1758735220.19 total 65.88 last 338.27) - open/close 23617 (time 1758735230.19 total 75.88 last 361.65) - open/close 26927 (time 1758735240.19 total 85.88 last 330.98) - open/close 30000 (time 1758735248.35 total 94.04 last 376.55) - open/close 33523 (time 1758735258.35 total 104.05 last 352.27) - open/close 36990 (time 1758735268.35 total 114.05 last 346.68) - open/close 39823 (time 1758735278.35 total 124.05 last 283.22) - open/close 43031 (time 1758735288.39 total 134.09 last 319.58) - open/close 46470 (time 1758735298.39 total 144.09 last 343.88) - open/close 49895 (time 1758735308.39 total 154.09 last 342.50) - open/close 53567 (time 1758735318.40 total 164.09 last 367.09) - open/close 56727 (time 1758735328.40 total 174.09 last 315.99) - open/close 59785 (time 1758735338.40 total 184.09 last 305.71) - open/close 62972 (time 1758735348.47 total 194.17 last 316.34) total: 64768 open/close in 199.35 seconds: 324.90 ops/second - /unlink 7059 (time 1758735364.99 total 10.00 last 705.80) - /unlink 10000 (time 1758735369.58 total 14.59 last 640.37) - /unlink 15949 (time 1758735379.58 total 24.60 last 594.81) - /unlink 20000 (time 1758735387.07 total 32.08 last 541.18) - /unlink 25713 (time 1758735397.07 total 42.08 last 571.24) - /unlink 30000 (time 1758735404.88 total 49.89 last 549.28) - /unlink 36291 (time 1758735414.88 total 59.89 last 629.07) - /unlink 40000 (time 1758735420.91 total 65.93 last 614.30) - /unlink 46825 (time 1758735430.92 total 75.93 last 682.43) - /unlink 50000 (time 1758735435.66 total 80.67 last 669.49) - /unlink 57405 (time 1758735445.66 total 90.67 last 740.46) - /unlink 60000 (time 1758735449.00 total 94.01 last 777.69) total: 64768 /unlink in 101.80 seconds: 636.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5155 (time 1758735488.04 total 10.00 last 515.40) - open/close 10000 (time 1758735497.54 total 19.50 last 510.04) - open/close 14521 (time 1758735507.54 total 29.50 last 451.99) - open/close 18809 (time 1758735517.54 total 39.50 last 428.77) - open/close 20000 (time 1758735520.16 total 42.12 last 456.03) - open/close 24646 (time 1758735530.16 total 52.12 last 464.56) - open/close 29425 (time 1758735540.16 total 62.12 last 477.88) - open/close 33968 (time 1758735550.16 total 72.12 last 454.29) - open/close 38523 (time 1758735560.16 total 82.12 last 455.43) - open/close 40000 (time 1758735563.37 total 85.33 last 460.06) - open/close 44470 (time 1758735573.37 total 95.33 last 446.98) - open/close 49151 (time 1758735583.37 total 105.33 last 468.03) - open/close 53583 (time 1758735593.37 total 115.33 last 443.19) - open/close 58388 (time 1758735603.37 total 125.33 last 480.45) - open/close 60000 (time 1758735606.78 total 128.74 last 472.50) - open/close 64736 (time 1758735616.87 total 138.83 last 469.60) total: 64768 open/close in 138.89 seconds: 466.34 ops/second - /unlink 7589 (time 1758735628.24 total 10.00 last 758.79) - /unlink 10000 (time 1758735630.99 total 12.76 last 875.58) - /unlink 17700 (time 1758735640.99 total 22.76 last 769.96) - /unlink 20000 (time 1758735644.36 total 26.12 last 682.88) - /unlink 27723 (time 1758735654.36 total 36.12 last 772.23) - /unlink 30000 (time 1758735656.72 total 38.49 last 964.40) - /unlink 37880 (time 1758735666.72 total 48.49 last 787.99) - /unlink 40000 (time 1758735669.53 total 51.30 last 753.95) - /unlink 48308 (time 1758735679.53 total 61.30 last 830.73) - /unlink 50000 (time 1758735681.65 total 63.41 last 801.38) - /unlink 57740 (time 1758735691.65 total 73.41 last 773.99) - /unlink 60000 (time 1758735694.55 total 76.32 last 777.60) total: 64768 /unlink in 82.61 seconds: 784.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4915 (time 1758735732.40 total 10.00 last 491.41) - open/close 8910 (time 1758735742.40 total 20.00 last 399.41) - open/close 10000 (time 1758735745.03 total 22.63 last 414.36) - open/close 13701 (time 1758735755.03 total 32.64 last 370.07) - open/close 18152 (time 1758735765.04 total 42.64 last 445.03) - open/close 20000 (time 1758735768.97 total 46.58 last 469.25) - open/close 24127 (time 1758735778.98 total 56.58 last 412.62) - open/close 28827 (time 1758735789.00 total 66.60 last 468.96) - open/close 30000 (time 1758735791.43 total 69.03 last 481.72) - open/close 34641 (time 1758735801.43 total 79.04 last 464.07) - open/close 39335 (time 1758735811.43 total 89.04 last 469.37) - open/close 43482 (time 1758735821.44 total 99.04 last 414.64) - open/close 47810 (time 1758735831.44 total 109.04 last 432.79) - open/close 50000 (time 1758735836.20 total 113.80 last 460.03) - open/close 54728 (time 1758735846.20 total 123.80 last 472.79) - open/close 59610 (time 1758735856.20 total 133.80 last 488.12) - open/close 64374 (time 1758735866.20 total 143.80 last 476.35) total: 64768 open/close in 144.73 seconds: 447.51 ops/second - /unlink 7734 (time 1758735878.41 total 10.00 last 773.24) - /unlink 10000 (time 1758735881.49 total 13.09 last 734.76) - /unlink 16965 (time 1758735891.49 total 23.09 last 696.43) - /unlink 20000 (time 1758735895.33 total 26.93 last 790.10) - /unlink 27075 (time 1758735905.33 total 36.93 last 707.46) - /unlink 30000 (time 1758735909.98 total 41.57 last 629.91) - /unlink 37666 (time 1758735919.98 total 51.57 last 766.54) - /unlink 40000 (time 1758735922.77 total 54.36 last 837.49) - /unlink 47593 (time 1758735932.77 total 64.36 last 759.28) - /unlink 50000 (time 1758735936.34 total 67.93 last 674.14) - /unlink 57797 (time 1758735946.34 total 77.93 last 779.67) - /unlink 60000 (time 1758735949.38 total 80.97 last 723.82) total: 64768 /unlink in 86.59 seconds: 747.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing unload_modules_local oleg420-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg420-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2