== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:08:55 (1760137735) Stopping clients: oleg156-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg156-client.virtnet /mnt/lustre2 (opts:-f) oleg156-server: oleg156-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg156-server: detected 4 online CPUs by sysfs oleg156-server: MODOPTS_LIBCFS= oleg156-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg156-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg156-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg156-client.virtnet: -o user_xattr,flock 192.168.201.156@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4564 (time 1760137779.94 total 10.00 last 456.35) - open/close 9004 (time 1760137789.95 total 20.00 last 443.95) - open/close 10000 (time 1760137792.19 total 22.25 last 443.51) - open/close 14152 (time 1760137802.19 total 32.25 last 415.20) - open/close 18619 (time 1760137812.19 total 42.25 last 446.66) - open/close 20000 (time 1760137815.25 total 45.30 last 452.33) - open/close 24604 (time 1760137825.25 total 55.30 last 460.37) - open/close 29121 (time 1760137835.25 total 65.30 last 451.61) - open/close 30000 (time 1760137837.28 total 67.34 last 432.56) - open/close 34195 (time 1760137847.28 total 77.34 last 419.47) - open/close 38577 (time 1760137857.28 total 87.34 last 438.18) - open/close 40000 (time 1760137860.40 total 90.46 last 456.42) - open/close 44393 (time 1760137870.40 total 100.46 last 439.23) - open/close 48633 (time 1760137880.40 total 110.46 last 423.97) - open/close 50000 (time 1760137883.84 total 113.89 last 398.03) - open/close 53430 (time 1760137893.84 total 123.89 last 342.99) - open/close 57204 (time 1760137903.84 total 133.89 last 377.33) - open/close 60000 (time 1760137911.06 total 141.12 last 387.05) - open/close 63765 (time 1760137921.06 total 151.12 last 376.47) total: 64768 open/close in 153.88 seconds: 420.90 ops/second - /unlink 7920 (time 1760137935.26 total 10.00 last 792.00) - /unlink 10000 (time 1760137937.62 total 12.35 last 883.36) - /unlink 17895 (time 1760137947.62 total 22.35 last 789.48) - /unlink 20000 (time 1760137950.54 total 25.28 last 719.19) - /unlink 27459 (time 1760137960.55 total 35.28 last 745.84) - /unlink 30000 (time 1760137963.78 total 38.52 last 784.82) - /unlink 37921 (time 1760137973.81 total 48.55 last 789.82) - /unlink 40000 (time 1760137976.59 total 51.33 last 747.53) - /unlink 47809 (time 1760137986.59 total 61.33 last 780.88) - /unlink 50000 (time 1760137989.95 total 64.68 last 653.24) - /unlink 57406 (time 1760137999.95 total 74.69 last 740.51) - /unlink 60000 (time 1760138003.59 total 78.33 last 712.44) total: 64768 /unlink in 84.35 seconds: 767.83 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4764 (time 1760138039.77 total 10.00 last 476.32) - open/close 9520 (time 1760138049.77 total 20.00 last 475.57) - open/close 14309 (time 1760138059.77 total 30.00 last 478.82) - open/close 19087 (time 1760138069.77 total 40.01 last 477.72) - open/close 23762 (time 1760138079.77 total 50.01 last 467.42) - open/close 28390 (time 1760138089.77 total 60.01 last 462.77) - open/close 30000 (time 1760138093.12 total 63.36 last 480.88) - open/close 34695 (time 1760138103.12 total 73.36 last 469.48) - open/close 39483 (time 1760138113.12 total 83.36 last 478.74) - open/close 44065 (time 1760138123.12 total 93.36 last 458.15) - open/close 48602 (time 1760138133.13 total 103.36 last 453.69) - open/close 50000 (time 1760138136.23 total 106.47 last 450.11) - open/close 54576 (time 1760138146.23 total 116.47 last 457.54) - open/close 59061 (time 1760138156.23 total 126.47 last 448.46) - open/close 60000 (time 1760138158.29 total 128.52 last 456.55) - open/close 64645 (time 1760138168.29 total 138.52 last 464.47) total: 64768 open/close in 138.89 seconds: 466.33 ops/second - /unlink 7883 (time 1760138180.05 total 10.00 last 788.28) - /unlink 10000 (time 1760138182.59 total 12.54 last 833.36) - /unlink 17702 (time 1760138192.59 total 22.54 last 770.13) - /unlink 20000 (time 1760138195.75 total 25.70 last 727.80) - /unlink 27893 (time 1760138205.75 total 35.70 last 789.23) - /unlink 30000 (time 1760138208.37 total 38.32 last 803.67) - /unlink 37863 (time 1760138218.37 total 48.32 last 786.25) - /unlink 40000 (time 1760138221.25 total 51.21 last 740.64) - /unlink 47862 (time 1760138231.25 total 61.21 last 786.14) - /unlink 50000 (time 1760138233.87 total 63.83 last 816.42) - /unlink 57949 (time 1760138243.87 total 73.83 last 794.87) - /unlink 60000 (time 1760138246.55 total 76.50 last 767.45) total: 64768 /unlink in 82.82 seconds: 782.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4364 (time 1760138284.30 total 10.00 last 436.29) - open/close 8648 (time 1760138294.30 total 20.00 last 428.39) - open/close 10000 (time 1760138297.14 total 22.84 last 475.77) - open/close 13751 (time 1760138307.14 total 32.85 last 375.08) - open/close 17406 (time 1760138317.14 total 42.85 last 365.39) - open/close 20000 (time 1760138324.37 total 50.08 last 358.91) - open/close 24071 (time 1760138334.37 total 60.08 last 406.95) - open/close 27869 (time 1760138344.37 total 70.08 last 379.75) - open/close 30000 (time 1760138350.09 total 75.80 last 372.48) - open/close 33223 (time 1760138360.10 total 85.80 last 322.20) - open/close 37498 (time 1760138370.10 total 95.81 last 427.42) - open/close 40000 (time 1760138376.00 total 101.71 last 423.99) - open/close 44281 (time 1760138386.00 total 111.71 last 428.06) - open/close 48209 (time 1760138396.00 total 121.71 last 392.77) - open/close 50000 (time 1760138400.04 total 125.75 last 443.43) - open/close 54247 (time 1760138410.04 total 135.75 last 424.69) - open/close 58474 (time 1760138420.04 total 145.75 last 422.62) - open/close 60000 (time 1760138423.95 total 149.65 last 391.08) - open/close 63722 (time 1760138433.95 total 159.65 last 372.18) total: 64768 open/close in 162.27 seconds: 399.15 ops/second - /unlink 7727 (time 1760138447.89 total 10.00 last 772.67) - /unlink 10000 (time 1760138451.12 total 13.23 last 702.99) - /unlink 17513 (time 1760138461.12 total 23.23 last 751.25) - /unlink 20000 (time 1760138464.14 total 26.25 last 823.83) - /unlink 25757 (time 1760138474.14 total 36.25 last 575.63) - /unlink 30000 (time 1760138481.63 total 43.74 last 566.49) - /unlink 35749 (time 1760138491.63 total 53.74 last 574.90) - /unlink 40000 (time 1760138499.94 total 62.06 last 511.34) - /unlink 46039 (time 1760138509.94 total 72.06 last 603.87) - /unlink 50000 (time 1760138517.28 total 79.39 last 540.00) - /unlink 56999 (time 1760138527.28 total 89.39 last 699.89) - /unlink 60000 (time 1760138532.06 total 94.17 last 627.72) total: 64768 /unlink in 102.94 seconds: 629.20 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3017 (time 1760138574.36 total 10.00 last 301.63) - open/close 5940 (time 1760138584.36 total 20.00 last 292.25) - open/close 9942 (time 1760138594.36 total 30.01 last 400.12) - open/close 12899 (time 1760138604.36 total 40.01 last 295.68) - open/close 14773 (time 1760138614.36 total 50.01 last 187.39) - open/close 19437 (time 1760138624.36 total 60.01 last 466.35) - open/close 24068 (time 1760138634.36 total 70.01 last 463.02) - open/close 28673 (time 1760138644.37 total 80.01 last 460.45) - open/close 30000 (time 1760138647.35 total 83.00 last 444.53) - open/close 34505 (time 1760138657.35 total 93.00 last 450.47) - open/close 38914 (time 1760138667.35 total 103.00 last 440.86) - open/close 40000 (time 1760138669.67 total 105.32 last 468.56) - open/close 44496 (time 1760138679.67 total 115.32 last 449.51) - open/close 48731 (time 1760138689.73 total 125.38 last 420.94) - open/close 50000 (time 1760138692.80 total 128.45 last 413.23) - open/close 54574 (time 1760138702.81 total 138.45 last 457.32) - open/close 59014 (time 1760138712.81 total 148.45 last 443.96) - open/close 60000 (time 1760138715.11 total 150.76 last 427.70) - open/close 64420 (time 1760138725.11 total 160.76 last 441.96) total: 64768 open/close in 161.60 seconds: 400.78 ops/second - /unlink 7725 (time 1760138737.32 total 10.00 last 772.48) - /unlink 10000 (time 1760138740.98 total 13.67 last 620.45) - /unlink 15393 (time 1760138750.98 total 23.67 last 539.29) - /unlink 20000 (time 1760138757.96 total 30.65 last 660.16) - /unlink 25011 (time 1760138767.96 total 40.65 last 501.06) - /unlink 30000 (time 1760138775.54 total 48.23 last 658.03) - /unlink 36732 (time 1760138785.55 total 58.23 last 673.06) - /unlink 40000 (time 1760138791.05 total 63.73 last 594.00) - /unlink 45170 (time 1760138801.05 total 73.73 last 516.89) - /unlink 49897 (time 1760138811.05 total 83.73 last 472.68) - /unlink 54400 (time 1760138821.05 total 93.74 last 450.27) - /unlink 59383 (time 1760138831.05 total 103.74 last 498.25) - /unlink 63929 (time 1760138841.05 total 113.74 last 454.50) total: 64768 /unlink in 114.87 seconds: 563.82 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2147 (time 1760138873.93 total 10.00 last 214.62) - open/close 4274 (time 1760138883.93 total 20.00 last 212.70) - open/close 7434 (time 1760138893.93 total 30.01 last 315.89) - open/close 9683 (time 1760138903.94 total 40.01 last 224.83) - open/close 12043 (time 1760138913.94 total 50.01 last 235.99) - open/close 15312 (time 1760138923.94 total 60.01 last 326.84) - open/close 19859 (time 1760138933.94 total 70.01 last 454.69) - open/close 23756 (time 1760138943.94 total 80.01 last 389.66) - open/close 26207 (time 1760138953.94 total 90.02 last 245.07) - open/close 28811 (time 1760138963.94 total 100.02 last 260.33) - open/close 30000 (time 1760138968.86 total 104.93 last 241.99) - open/close 32723 (time 1760138978.86 total 114.94 last 272.19) - open/close 35407 (time 1760138988.86 total 124.94 last 268.36) - open/close 37703 (time 1760138998.87 total 134.95 last 229.36) - open/close 40000 (time 1760139008.06 total 144.14 last 250.01) - open/close 42324 (time 1760139018.06 total 154.14 last 232.34) - open/close 44631 (time 1760139028.06 total 164.14 last 230.65) - open/close 46987 (time 1760139038.07 total 174.14 last 235.51) - open/close 49300 (time 1760139048.07 total 184.15 last 231.26) - open/close 52019 (time 1760139058.07 total 194.15 last 271.79) - open/close 54326 (time 1760139068.08 total 204.15 last 230.67) - open/close 58527 (time 1760139078.08 total 214.15 last 420.03) - open/close 60000 (time 1760139081.21 total 217.28 last 470.93) - open/close 64657 (time 1760139091.21 total 227.28 last 465.63) total: 64768 open/close in 227.63 seconds: 284.53 ops/second - /unlink 8408 (time 1760139102.74 total 10.00 last 840.71) - /unlink 17037 (time 1760139112.74 total 20.00 last 862.84) - /unlink 20000 (time 1760139116.29 total 23.55 last 834.59) - /unlink 27496 (time 1760139126.29 total 33.55 last 749.58) - /unlink 30000 (time 1760139129.65 total 36.91 last 745.76) - /unlink 38186 (time 1760139139.65 total 46.91 last 818.21) - /unlink 40000 (time 1760139142.06 total 49.32 last 754.28) - /unlink 48123 (time 1760139152.06 total 59.32 last 812.26) - /unlink 50000 (time 1760139154.14 total 61.40 last 903.52) - /unlink 56477 (time 1760139164.14 total 71.40 last 647.69) - /unlink 60000 (time 1760139171.57 total 78.83 last 474.19) total: 64768 /unlink in 87.39 seconds: 741.16 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg156-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing unload_modules_local oleg156-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2