== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:26:38 (1760451998) Stopping clients: oleg431-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg431-client.virtnet /mnt/lustre2 (opts:-f) oleg431-server: oleg431-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg431-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg431-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg431-client.virtnet: -o user_xattr,flock oleg431-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1910 (time 1760452052.64 total 10.00 last 190.99) - open/close 3612 (time 1760452062.64 total 20.00 last 170.16) - open/close 6056 (time 1760452072.64 total 30.01 last 244.31) - open/close 8611 (time 1760452082.65 total 40.01 last 255.38) - open/close 10000 (time 1760452087.66 total 45.02 last 277.37) - open/close 12387 (time 1760452097.66 total 55.02 last 238.65) - open/close 14323 (time 1760452107.66 total 65.02 last 193.54) - open/close 16231 (time 1760452117.66 total 75.02 last 190.79) - open/close 18496 (time 1760452127.66 total 85.02 last 226.49) - open/close 20000 (time 1760452134.20 total 91.56 last 230.16) - open/close 22360 (time 1760452144.20 total 101.56 last 235.98) - open/close 24486 (time 1760452154.20 total 111.57 last 212.49) - open/close 26318 (time 1760452164.21 total 121.57 last 183.13) - open/close 27978 (time 1760452174.21 total 131.58 last 165.88) - open/close 29688 (time 1760452184.22 total 141.58 last 170.92) - open/close 31829 (time 1760452194.22 total 151.58 last 214.03) - open/close 33906 (time 1760452204.22 total 161.59 last 207.65) - open/close 36444 (time 1760452214.22 total 171.59 last 253.79) - open/close 38630 (time 1760452224.23 total 181.59 last 218.53) - open/close 40000 (time 1760452230.24 total 187.60 last 227.97) - open/close 41748 (time 1760452240.24 total 197.61 last 174.67) - open/close 43438 (time 1760452250.25 total 207.61 last 168.89) - open/close 45131 (time 1760452260.25 total 217.61 last 169.28) - open/close 47013 (time 1760452270.26 total 227.62 last 188.08) - open/close 49051 (time 1760452280.26 total 237.62 last 203.79) - open/close 50000 (time 1760452284.51 total 241.87 last 223.43) - open/close 52058 (time 1760452294.51 total 251.87 last 205.69) - open/close 53567 (time 1760452304.52 total 261.88 last 150.85) - open/close 55183 (time 1760452314.52 total 271.88 last 161.52) - open/close 57078 (time 1760452324.53 total 281.89 last 189.36) - open/close 58804 (time 1760452334.53 total 291.89 last 172.56) - open/close 60000 (time 1760452340.91 total 298.28 last 187.36) - open/close 61889 (time 1760452350.92 total 308.28 last 188.80) - open/close 64015 (time 1760452360.92 total 318.29 last 212.51) total: 64768 open/close in 323.12 seconds: 200.45 ops/second - /unlink 2972 (time 1760452377.61 total 10.00 last 297.15) - /unlink 6422 (time 1760452387.61 total 20.00 last 344.98) - /unlink 9881 (time 1760452397.61 total 30.00 last 345.86) - /unlink 13566 (time 1760452407.61 total 40.01 last 368.41) - /unlink 17086 (time 1760452417.62 total 50.01 last 351.85) - /unlink 20000 (time 1760452426.34 total 58.73 last 334.06) - /unlink 23623 (time 1760452436.34 total 68.73 last 362.26) - /unlink 27039 (time 1760452446.34 total 78.74 last 341.59) - /unlink 30000 (time 1760452454.65 total 87.04 last 356.51) - /unlink 33709 (time 1760452464.65 total 97.04 last 370.86) - /unlink 37309 (time 1760452474.65 total 107.04 last 359.93) - /unlink 40000 (time 1760452482.54 total 114.93 last 341.08) - /unlink 43497 (time 1760452492.54 total 124.94 last 349.58) - /unlink 46698 (time 1760452502.54 total 134.94 last 320.09) - /unlink 50000 (time 1760452512.00 total 144.40 last 349.04) - /unlink 54230 (time 1760452522.00 total 154.40 last 422.97) - /unlink 57914 (time 1760452532.01 total 164.40 last 368.34) - /unlink 60000 (time 1760452538.04 total 170.43 last 345.95) - /unlink 64717 (time 1760452548.04 total 180.43 last 471.55) total: 64768 /unlink in 180.62 seconds: 358.59 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2296 (time 1760452576.24 total 10.00 last 229.53) - open/close 4213 (time 1760452586.25 total 20.01 last 191.60) - open/close 6539 (time 1760452596.25 total 30.01 last 232.52) - open/close 8888 (time 1760452606.26 total 40.01 last 234.83) - open/close 10000 (time 1760452610.51 total 44.27 last 261.43) - open/close 12465 (time 1760452620.51 total 54.27 last 246.45) - open/close 14606 (time 1760452630.51 total 64.27 last 214.06) - open/close 16947 (time 1760452640.51 total 74.27 last 234.09) - open/close 19426 (time 1760452650.51 total 84.27 last 247.89) - open/close 20000 (time 1760452652.83 total 86.59 last 248.25) - open/close 22248 (time 1760452662.83 total 96.59 last 224.76) - open/close 24331 (time 1760452672.83 total 106.59 last 208.30) - open/close 26548 (time 1760452682.83 total 116.59 last 221.66) - open/close 28755 (time 1760452692.83 total 126.59 last 220.61) - open/close 30000 (time 1760452699.94 total 133.70 last 175.10) - open/close 31893 (time 1760452709.95 total 143.71 last 189.22) - open/close 33920 (time 1760452719.95 total 153.71 last 202.66) - open/close 35922 (time 1760452729.95 total 163.71 last 200.12) - open/close 37997 (time 1760452739.96 total 173.72 last 207.41) - open/close 40000 (time 1760452747.68 total 181.44 last 259.27) - open/close 42763 (time 1760452757.68 total 191.44 last 276.27) - open/close 45464 (time 1760452767.69 total 201.45 last 270.08) - open/close 47913 (time 1760452777.69 total 211.45 last 244.84) - open/close 50000 (time 1760452786.36 total 220.12 last 240.64) - open/close 52210 (time 1760452796.36 total 230.12 last 220.97) - open/close 54338 (time 1760452806.36 total 240.12 last 212.78) - open/close 56875 (time 1760452816.36 total 250.12 last 253.70) - open/close 59181 (time 1760452826.37 total 260.13 last 230.51) - open/close 60000 (time 1760452829.80 total 263.56 last 238.29) - open/close 62571 (time 1760452839.80 total 273.56 last 257.08) total: 64768 open/close in 281.14 seconds: 230.38 ops/second - /unlink 5821 (time 1760452858.89 total 10.00 last 582.06) - /unlink 10000 (time 1760452866.85 total 17.96 last 525.00) - /unlink 14789 (time 1760452876.85 total 27.96 last 478.88) - /unlink 20000 (time 1760452884.72 total 35.83 last 662.11) - /unlink 26063 (time 1760452894.72 total 45.83 last 606.28) - /unlink 30000 (time 1760452900.85 total 51.96 last 641.99) - /unlink 35385 (time 1760452910.85 total 61.96 last 538.49) - /unlink 40000 (time 1760452919.12 total 70.23 last 558.29) - /unlink 45878 (time 1760452929.12 total 80.23 last 587.79) - /unlink 50000 (time 1760452936.27 total 87.38 last 576.72) - /unlink 56769 (time 1760452946.28 total 97.40 last 675.71) - /unlink 60000 (time 1760452950.95 total 102.06 last 692.37) total: 64768 /unlink in 109.35 seconds: 592.28 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4074 (time 1760452985.36 total 10.00 last 407.32) - open/close 8745 (time 1760452995.36 total 20.00 last 467.04) - open/close 10000 (time 1760452997.98 total 22.62 last 479.19) - open/close 14777 (time 1760453007.98 total 32.62 last 477.66) - open/close 19145 (time 1760453017.98 total 42.62 last 436.77) - open/close 23996 (time 1760453027.98 total 52.62 last 485.07) - open/close 28454 (time 1760453037.98 total 62.63 last 445.74) - open/close 30000 (time 1760453041.22 total 65.87 last 477.09) - open/close 35094 (time 1760453051.22 total 75.87 last 509.39) - open/close 39667 (time 1760453061.22 total 85.87 last 457.22) - open/close 44572 (time 1760453071.22 total 95.87 last 490.50) - open/close 49136 (time 1760453081.22 total 105.87 last 456.40) - open/close 53573 (time 1760453091.22 total 115.87 last 443.66) - open/close 58143 (time 1760453101.23 total 125.87 last 456.96) - open/close 60000 (time 1760453105.68 total 130.33 last 416.71) - open/close 64612 (time 1760453115.78 total 140.42 last 456.81) total: 64768 open/close in 140.75 seconds: 460.17 ops/second - /unlink 7387 (time 1760453127.41 total 10.00 last 738.63) - /unlink 10000 (time 1760453130.31 total 12.89 last 903.27) - /unlink 18152 (time 1760453140.31 total 22.89 last 815.19) - /unlink 20000 (time 1760453143.18 total 25.77 last 643.37) - /unlink 27480 (time 1760453153.18 total 35.77 last 747.97) - /unlink 30000 (time 1760453155.76 total 38.35 last 976.00) - /unlink 37798 (time 1760453165.76 total 48.35 last 779.70) - /unlink 40000 (time 1760453169.36 total 51.95 last 612.22) - /unlink 47780 (time 1760453179.36 total 61.95 last 777.96) - /unlink 50000 (time 1760453181.72 total 64.30 last 941.94) - /unlink 58177 (time 1760453191.72 total 74.30 last 817.64) - /unlink 60000 (time 1760453194.13 total 76.72 last 755.32) total: 64768 /unlink in 82.05 seconds: 789.37 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4771 (time 1760453231.87 total 10.00 last 477.05) - open/close 9971 (time 1760453241.87 total 20.00 last 519.90) - open/close 14510 (time 1760453251.87 total 30.00 last 453.81) - open/close 19579 (time 1760453261.87 total 40.01 last 506.85) - open/close 23903 (time 1760453271.88 total 50.01 last 432.28) - open/close 28424 (time 1760453281.88 total 60.01 last 452.07) - open/close 30000 (time 1760453285.33 total 63.46 last 457.13) - open/close 34538 (time 1760453295.33 total 73.46 last 453.64) - open/close 38601 (time 1760453305.33 total 83.46 last 406.25) - open/close 40000 (time 1760453308.94 total 87.07 last 387.92) - open/close 44125 (time 1760453318.94 total 97.07 last 412.46) - open/close 48588 (time 1760453328.94 total 107.07 last 446.26) - open/close 50000 (time 1760453332.10 total 110.23 last 446.34) - open/close 54743 (time 1760453342.10 total 120.23 last 474.27) - open/close 59900 (time 1760453352.10 total 130.23 last 515.66) total: 64768 open/close in 140.07 seconds: 462.39 ops/second - /unlink 7613 (time 1760453373.26 total 10.00 last 761.20) - /unlink 10000 (time 1760453376.16 total 12.90 last 822.06) - /unlink 17212 (time 1760453386.16 total 22.91 last 721.16) - /unlink 20000 (time 1760453390.22 total 26.96 last 687.94) - /unlink 27927 (time 1760453400.22 total 36.96 last 792.69) - /unlink 30000 (time 1760453403.26 total 40.01 last 680.23) - /unlink 38342 (time 1760453413.27 total 50.01 last 834.09) - /unlink 40000 (time 1760453415.73 total 52.47 last 672.71) - /unlink 48153 (time 1760453425.73 total 62.47 last 815.26) - /unlink 50000 (time 1760453427.82 total 64.56 last 886.13) - /unlink 58489 (time 1760453437.82 total 74.56 last 848.38) - /unlink 60000 (time 1760453439.99 total 76.73 last 697.31) total: 64768 /unlink in 82.18 seconds: 788.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4999 (time 1760453477.74 total 10.00 last 499.88) - open/close 9916 (time 1760453487.75 total 20.00 last 491.67) - open/close 14383 (time 1760453497.75 total 30.00 last 446.70) - open/close 19071 (time 1760453507.75 total 40.00 last 468.74) - open/close 23550 (time 1760453517.75 total 50.00 last 447.88) - open/close 28804 (time 1760453527.75 total 60.00 last 525.35) - open/close 30000 (time 1760453530.05 total 62.30 last 520.01) - open/close 34997 (time 1760453540.05 total 72.30 last 499.69) - open/close 39886 (time 1760453550.05 total 82.30 last 488.88) - open/close 44651 (time 1760453560.05 total 92.30 last 476.47) - open/close 49765 (time 1760453570.05 total 102.31 last 511.33) - open/close 54172 (time 1760453580.05 total 112.31 last 440.69) - open/close 58400 (time 1760453590.05 total 122.31 last 422.76) - open/close 60000 (time 1760453593.98 total 126.24 last 407.10) - open/close 64113 (time 1760453603.98 total 136.24 last 411.30) total: 64768 open/close in 137.57 seconds: 470.80 ops/second - /unlink 8358 (time 1760453616.69 total 10.00 last 835.76) - /unlink 10000 (time 1760453619.17 total 12.49 last 660.72) - /unlink 17473 (time 1760453629.18 total 22.49 last 747.13) - /unlink 20000 (time 1760453632.63 total 25.95 last 730.65) - /unlink 27134 (time 1760453642.64 total 35.95 last 713.11) - /unlink 30000 (time 1760453646.27 total 39.58 last 790.31) - /unlink 38271 (time 1760453656.27 total 49.58 last 827.09) - /unlink 45495 (time 1760453666.27 total 59.58 last 722.34) - /unlink 50000 (time 1760453672.33 total 65.64 last 743.38) - /unlink 57125 (time 1760453682.33 total 75.64 last 712.43) - /unlink 60000 (time 1760453686.63 total 79.94 last 668.39) total: 64768 /unlink in 86.30 seconds: 750.53 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg431-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing unload_modules_local oleg431-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg431-server: LNET ready to unload oleg431-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2