== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:44:07 (1772167447) Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f) oleg241-server: oleg241-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg241-server: detected 4 online CPUs by sysfs oleg241-server: MODOPTS_LIBCFS= oleg241-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg241-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg241-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3705 (time 1772167492.70 total 10.00 last 370.45) - open/close 7095 (time 1772167502.70 total 20.00 last 338.99) - open/close 10000 (time 1772167509.75 total 27.05 last 412.29) - open/close 13726 (time 1772167519.75 total 37.05 last 372.57) - open/close 17061 (time 1772167529.75 total 47.05 last 333.43) - open/close 20000 (time 1772167539.12 total 56.42 last 313.68) - open/close 23010 (time 1772167549.12 total 66.42 last 300.97) - open/close 26864 (time 1772167559.12 total 76.42 last 385.30) - open/close 30000 (time 1772167568.17 total 85.47 last 346.60) - open/close 33754 (time 1772167578.17 total 95.47 last 375.39) - open/close 37105 (time 1772167588.17 total 105.47 last 335.05) - open/close 40000 (time 1772167597.60 total 114.90 last 307.14) - open/close 43022 (time 1772167607.60 total 124.90 last 302.16) - open/close 46013 (time 1772167617.60 total 134.90 last 299.01) - open/close 48862 (time 1772167627.61 total 144.90 last 284.87) - open/close 50000 (time 1772167630.15 total 147.45 last 447.37) - open/close 53291 (time 1772167640.15 total 157.45 last 329.08) - open/close 56393 (time 1772167650.15 total 167.45 last 310.11) - open/close 59566 (time 1772167660.15 total 177.45 last 317.28) - open/close 62606 (time 1772167670.15 total 187.45 last 303.96) total: 64768 open/close in 193.95 seconds: 333.94 ops/second - /unlink 5283 (time 1772167688.59 total 10.00 last 528.24) - /unlink 10000 (time 1772167696.97 total 18.39 last 562.59) - /unlink 15323 (time 1772167706.97 total 28.39 last 532.26) - /unlink 20000 (time 1772167714.83 total 36.24 last 595.53) - /unlink 25683 (time 1772167724.83 total 46.24 last 568.23) - /unlink 30000 (time 1772167733.63 total 55.04 last 490.67) - /unlink 35653 (time 1772167743.63 total 65.04 last 565.25) - /unlink 40000 (time 1772167751.73 total 73.15 last 536.16) - /unlink 45740 (time 1772167761.74 total 83.15 last 573.87) - /unlink 50000 (time 1772167769.67 total 91.08 last 537.20) - /unlink 54939 (time 1772167779.67 total 101.09 last 493.63) - /unlink 60000 (time 1772167787.98 total 109.39 last 609.36) total: 64768 /unlink in 118.45 seconds: 546.77 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2581 (time 1772167826.45 total 10.00 last 258.09) - open/close 5328 (time 1772167836.45 total 20.00 last 274.64) - open/close 8027 (time 1772167846.45 total 30.00 last 269.85) - open/close 10000 (time 1772167852.68 total 36.24 last 316.47) - open/close 12804 (time 1772167862.69 total 46.24 last 280.30) - open/close 15341 (time 1772167872.69 total 56.24 last 253.67) - open/close 17945 (time 1772167882.69 total 66.24 last 260.38) - open/close 20000 (time 1772167890.60 total 74.15 last 259.78) - open/close 23049 (time 1772167900.60 total 84.15 last 304.90) - open/close 25666 (time 1772167910.60 total 94.16 last 261.64) - open/close 28152 (time 1772167920.61 total 104.16 last 248.50) - open/close 30000 (time 1772167928.60 total 112.16 last 231.03) - open/close 32686 (time 1772167938.61 total 122.16 last 268.54) - open/close 35672 (time 1772167948.61 total 132.16 last 298.58) - open/close 38226 (time 1772167958.61 total 142.17 last 255.30) - open/close 40000 (time 1772167967.04 total 150.59 last 210.51) - open/close 42479 (time 1772167977.04 total 160.60 last 247.85) - open/close 45254 (time 1772167987.04 total 170.60 last 277.39) - open/close 47760 (time 1772167997.05 total 180.60 last 250.49) - open/close 50000 (time 1772168007.02 total 190.57 last 224.70) - open/close 52400 (time 1772168017.02 total 200.57 last 240.00) - open/close 54639 (time 1772168027.02 total 210.58 last 223.82) - open/close 57011 (time 1772168037.03 total 220.58 last 237.12) - open/close 59325 (time 1772168047.03 total 230.58 last 231.34) - open/close 60000 (time 1772168049.83 total 233.39 last 240.72) - open/close 62234 (time 1772168059.84 total 243.39 last 223.31) - open/close 64361 (time 1772168069.84 total 253.39 last 212.68) total: 64768 open/close in 255.27 seconds: 253.72 ops/second - /unlink 4474 (time 1772168083.76 total 10.00 last 447.34) - /unlink 9213 (time 1772168093.76 total 20.00 last 473.82) - /unlink 14146 (time 1772168103.76 total 30.00 last 493.26) - /unlink 19633 (time 1772168113.76 total 40.00 last 548.65) - /unlink 25181 (time 1772168123.76 total 50.01 last 554.74) - /unlink 30000 (time 1772168131.68 total 57.92 last 608.53) - /unlink 36186 (time 1772168141.68 total 67.93 last 618.54) - /unlink 40000 (time 1772168148.64 total 74.89 last 547.76) - /unlink 46430 (time 1772168158.65 total 84.90 last 642.59) - /unlink 50000 (time 1772168164.36 total 90.60 last 625.36) - /unlink 56501 (time 1772168174.36 total 100.60 last 650.05) - /unlink 60000 (time 1772168180.14 total 106.39 last 604.91) total: 64768 /unlink in 115.90 seconds: 558.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3547 (time 1772168218.95 total 10.00 last 354.63) - open/close 6887 (time 1772168228.96 total 20.01 last 333.88) - open/close 9990 (time 1772168238.96 total 30.01 last 310.25) - open/close 12545 (time 1772168248.96 total 40.01 last 255.39) - open/close 15371 (time 1772168258.96 total 50.01 last 282.53) - open/close 17829 (time 1772168268.97 total 60.02 last 245.70) - open/close 20000 (time 1772168278.05 total 69.10 last 239.06) - open/close 22587 (time 1772168288.05 total 79.10 last 258.67) - open/close 25097 (time 1772168298.05 total 89.10 last 250.92) - open/close 27281 (time 1772168308.06 total 99.11 last 218.35) - open/close 29591 (time 1772168318.06 total 109.11 last 230.96) - open/close 31984 (time 1772168328.06 total 119.11 last 239.22) - open/close 34221 (time 1772168338.07 total 129.11 last 223.63) - open/close 36564 (time 1772168348.07 total 139.12 last 234.16) - open/close 38892 (time 1772168358.07 total 149.12 last 232.73) - open/close 40000 (time 1772168363.16 total 154.21 last 217.70) - open/close 42184 (time 1772168373.16 total 164.21 last 218.37) - open/close 44703 (time 1772168383.17 total 174.21 last 251.89) - open/close 47311 (time 1772168393.17 total 184.22 last 260.70) - open/close 49883 (time 1772168403.17 total 194.22 last 257.19) - open/close 52372 (time 1772168413.17 total 204.22 last 248.88) - open/close 55058 (time 1772168423.17 total 214.22 last 268.51) - open/close 57496 (time 1772168433.17 total 224.22 last 243.79) - open/close 59744 (time 1772168443.18 total 234.22 last 224.76) - open/close 62547 (time 1772168453.18 total 244.23 last 280.23) total: 64768 open/close in 251.66 seconds: 257.36 ops/second - /unlink 4814 (time 1772168472.09 total 10.00 last 481.39) - /unlink 9871 (time 1772168482.10 total 20.01 last 505.30) - /unlink 14742 (time 1772168492.10 total 30.01 last 487.07) - /unlink 19158 (time 1772168502.10 total 40.01 last 441.56) - /unlink 23259 (time 1772168512.11 total 50.02 last 409.55) - /unlink 28013 (time 1772168522.11 total 60.02 last 475.40) - /unlink 30000 (time 1772168526.20 total 64.11 last 486.06) - /unlink 35622 (time 1772168536.20 total 74.11 last 562.15) - /unlink 40000 (time 1772168543.00 total 80.91 last 643.74) - /unlink 45979 (time 1772168553.00 total 90.91 last 597.88) - /unlink 50000 (time 1772168558.57 total 96.48 last 721.97) - /unlink 55261 (time 1772168568.58 total 106.49 last 525.92) - /unlink 60000 (time 1772168577.58 total 115.49 last 526.42) - /unlink 64210 (time 1772168587.58 total 125.49 last 420.97) total: 64768 /unlink in 127.36 seconds: 508.56 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2807 (time 1772168618.45 total 10.00 last 280.64) - open/close 6028 (time 1772168628.45 total 20.00 last 322.07) - open/close 9259 (time 1772168638.45 total 30.01 last 323.02) - open/close 10000 (time 1772168640.52 total 32.07 last 358.17) - open/close 12963 (time 1772168650.52 total 42.08 last 296.22) - open/close 15366 (time 1772168660.52 total 52.08 last 240.27) - open/close 17763 (time 1772168670.52 total 62.08 last 239.68) - open/close 20000 (time 1772168679.52 total 71.07 last 248.74) - open/close 22291 (time 1772168689.52 total 81.07 last 229.05) - open/close 24696 (time 1772168699.52 total 91.08 last 240.45) - open/close 27744 (time 1772168709.56 total 101.11 last 303.72) - open/close 30000 (time 1772168718.11 total 109.66 last 263.85) - open/close 32340 (time 1772168728.11 total 119.66 last 233.97) - open/close 34743 (time 1772168738.11 total 129.67 last 240.19) - open/close 37826 (time 1772168748.11 total 139.67 last 308.28) - open/close 40000 (time 1772168756.74 total 148.29 last 252.10) - open/close 42394 (time 1772168766.74 total 158.29 last 239.36) - open/close 45178 (time 1772168776.74 total 168.30 last 278.34) - open/close 47980 (time 1772168786.74 total 178.30 last 280.12) - open/close 50000 (time 1772168795.59 total 187.15 last 228.21) - open/close 52282 (time 1772168805.60 total 197.15 last 228.17) - open/close 55725 (time 1772168815.60 total 207.16 last 344.16) - open/close 57485 (time 1772168825.60 total 217.16 last 175.97) - open/close 59734 (time 1772168835.61 total 227.16 last 224.80) - open/close 60718 (time 1772168845.62 total 237.17 last 98.29) - open/close 61639 (time 1772168855.64 total 247.19 last 91.92) - open/close 64092 (time 1772168865.64 total 257.19 last 245.28) total: 64768 open/close in 259.34 seconds: 249.74 ops/second - /unlink 6198 (time 1772168879.27 total 10.00 last 619.78) - /unlink 10000 (time 1772168885.04 total 15.78 last 657.88) - /unlink 14748 (time 1772168895.05 total 25.79 last 474.54) - /unlink 18358 (time 1772168905.05 total 35.79 last 360.89) - /unlink 20000 (time 1772168908.72 total 39.45 last 448.38) - /unlink 25851 (time 1772168918.72 total 49.45 last 585.04) - /unlink 30000 (time 1772168925.68 total 56.42 last 595.51) - /unlink 37145 (time 1772168935.68 total 66.42 last 714.48) - /unlink 40000 (time 1772168939.70 total 70.44 last 710.12) - /unlink 45011 (time 1772168949.71 total 80.44 last 501.02) - /unlink 49202 (time 1772168959.71 total 90.44 last 419.07) - /unlink 50000 (time 1772168961.81 total 92.55 last 379.24) - /unlink 54399 (time 1772168971.81 total 102.55 last 439.82) - /unlink 59462 (time 1772168981.81 total 112.55 last 506.25) - /unlink 62573 (time 1772168991.81 total 122.55 last 311.07) total: 64768 /unlink in 128.38 seconds: 504.51 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4848 (time 1772169026.44 total 10.00 last 484.75) - open/close 10000 (time 1772169036.08 total 19.65 last 534.13) - open/close 15298 (time 1772169046.08 total 29.65 last 529.77) - open/close 20000 (time 1772169055.50 total 39.06 last 499.58) - open/close 25535 (time 1772169065.50 total 49.06 last 553.44) - open/close 30000 (time 1772169073.69 total 57.25 last 545.00) - open/close 35518 (time 1772169083.69 total 67.25 last 551.73) - open/close 40000 (time 1772169091.59 total 75.15 last 567.49) - open/close 45516 (time 1772169101.59 total 85.15 last 551.50) - open/close 50000 (time 1772169110.03 total 93.59 last 531.22) - open/close 55512 (time 1772169120.03 total 103.60 last 551.18) - open/close 60000 (time 1772169128.53 total 112.10 last 527.87) total: 64768 open/close in 120.97 seconds: 535.39 ops/second - /unlink 7155 (time 1772169148.79 total 10.00 last 715.47) - /unlink 10000 (time 1772169153.03 total 14.24 last 670.94) - /unlink 17295 (time 1772169163.03 total 24.24 last 729.48) - /unlink 20000 (time 1772169166.73 total 27.93 last 732.38) - /unlink 27335 (time 1772169176.73 total 37.94 last 733.38) - /unlink 30000 (time 1772169180.53 total 41.74 last 700.44) - /unlink 37244 (time 1772169190.53 total 51.74 last 724.37) - /unlink 40000 (time 1772169195.07 total 56.28 last 607.65) - /unlink 47322 (time 1772169205.07 total 66.28 last 732.14) - /unlink 50000 (time 1772169209.10 total 70.31 last 664.33) - /unlink 57064 (time 1772169219.10 total 80.31 last 706.40) - /unlink 60000 (time 1772169223.29 total 84.50 last 700.43) total: 64768 /unlink in 91.79 seconds: 705.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server unloading modules via unload_modules_local on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing unload_modules_local oleg241-server: modules unloaded. pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2