== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:52:09 (1754405529) Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) oleg430-server: oleg430-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg430-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg430-server: detected 4 online CPUs by sysfs oleg430-server: MODOPTS_LIBCFS= oleg430-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg430-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg430-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 940 (time 1754405580.17 total 10.01 last 93.94) - open/close 1683 (time 1754405590.19 total 20.02 last 74.19) - open/close 2391 (time 1754405600.19 total 30.03 last 70.77) - open/close 3149 (time 1754405610.20 total 40.03 last 75.76) - open/close 3883 (time 1754405620.20 total 50.03 last 73.38) - open/close 4629 (time 1754405630.20 total 60.04 last 74.56) - open/close 5493 (time 1754405640.21 total 70.04 last 86.35) - open/close 6317 (time 1754405650.21 total 80.05 last 82.36) - open/close 7155 (time 1754405660.22 total 90.05 last 83.79) - open/close 7994 (time 1754405670.22 total 100.05 last 83.88) - open/close 8813 (time 1754405680.22 total 110.05 last 81.89) - open/close 9688 (time 1754405690.23 total 120.06 last 87.44) - open/close 10000 (time 1754405693.83 total 123.67 last 86.48) - open/close 10852 (time 1754405703.83 total 133.67 last 85.19) - open/close 11686 (time 1754405713.84 total 143.68 last 83.34) - open/close 12528 (time 1754405723.85 total 153.68 last 84.17) - open/close 13360 (time 1754405733.85 total 163.69 last 83.15) - open/close 14190 (time 1754405743.86 total 173.69 last 82.96) - open/close 15063 (time 1754405753.86 total 183.70 last 87.22) - open/close 15935 (time 1754405763.87 total 193.71 last 87.14) - open/close 17005 (time 1754405773.87 total 203.71 last 106.98) - open/close 17963 (time 1754405783.88 total 213.72 last 95.72) - open/close 18868 (time 1754405793.89 total 223.72 last 90.46) - open/close 19815 (time 1754405803.89 total 233.72 last 94.68) - open/close 20775 (time 1754405813.89 total 243.73 last 95.95) - open/close 21635 (time 1754405823.89 total 253.73 last 85.99) - open/close 22452 (time 1754405833.93 total 263.76 last 81.45) - open/close 23409 (time 1754405843.93 total 273.76 last 95.67) - open/close 24376 (time 1754405853.94 total 283.77 last 96.63) - open/close 25358 (time 1754405863.94 total 293.77 last 98.17) - open/close 26309 (time 1754405873.95 total 303.78 last 95.03) - open/close 27249 (time 1754405883.95 total 313.78 last 93.99) - open/close 28217 (time 1754405893.95 total 323.79 last 96.74) - open/close 29221 (time 1754405903.96 total 333.80 last 100.32) - open/close 30000 (time 1754405912.03 total 341.86 last 96.54) - open/close 30899 (time 1754405922.04 total 351.87 last 89.85) - open/close 31821 (time 1754405932.04 total 361.87 last 92.17) - open/close 32725 (time 1754405942.04 total 371.88 last 90.37) - open/close 33687 (time 1754405952.05 total 381.88 last 96.15) - open/close 34723 (time 1754405962.05 total 391.88 last 103.59) - open/close 35690 (time 1754405972.05 total 401.89 last 96.65) - open/close 36576 (time 1754405982.07 total 411.90 last 88.46) - open/close 37482 (time 1754405992.07 total 421.91 last 90.57) - open/close 38340 (time 1754406002.08 total 431.92 last 85.73) - open/close 39381 (time 1754406012.09 total 441.92 last 104.04) - open/close 40000 (time 1754406018.39 total 448.23 last 98.16) - open/close 40991 (time 1754406028.39 total 458.23 last 99.09) - open/close 42057 (time 1754406038.40 total 468.23 last 106.58) - open/close 43024 (time 1754406048.40 total 478.24 last 96.65) - open/close 43949 (time 1754406058.41 total 488.25 last 92.41) - open/close 44865 (time 1754406068.41 total 498.25 last 91.58) - open/close 45819 (time 1754406078.41 total 508.25 last 95.39) - open/close 46727 (time 1754406088.42 total 518.26 last 90.72) - open/close 47787 (time 1754406098.43 total 528.27 last 105.89) - open/close 48797 (time 1754406108.44 total 538.28 last 100.93) - open/close 49877 (time 1754406118.45 total 548.28 last 107.93) - open/close 50954 (time 1754406128.45 total 558.28 last 107.70) - open/close 52069 (time 1754406138.46 total 568.29 last 111.42) - open/close 53202 (time 1754406148.46 total 578.29 last 113.26) - open/close 54435 (time 1754406158.46 total 588.30 last 123.24) - open/close 55591 (time 1754406168.47 total 598.30 last 115.56) - open/close 56758 (time 1754406178.47 total 608.31 last 116.63) - open/close 57723 (time 1754406188.47 total 618.31 last 96.49) - open/close 58850 (time 1754406198.48 total 628.31 last 112.64) - open/close 60000 (time 1754406207.13 total 636.97 last 132.91) - open/close 61099 (time 1754406217.13 total 646.97 last 109.89) - open/close 62192 (time 1754406227.14 total 656.97 last 109.27) - open/close 63293 (time 1754406237.14 total 666.98 last 110.02) - open/close 64431 (time 1754406247.15 total 676.99 last 113.70) total: 64768 open/close in 679.53 seconds: 95.31 ops/second - /unlink 1335 (time 1754406261.68 total 10.00 last 133.45) - /unlink 2680 (time 1754406271.68 total 20.01 last 134.44) - /unlink 4238 (time 1754406281.68 total 30.01 last 155.79) - /unlink 5679 (time 1754406291.69 total 40.01 last 144.07) - /unlink 7067 (time 1754406301.69 total 50.02 last 138.68) - /unlink 8145 (time 1754406311.69 total 60.02 last 107.80) - /unlink 9623 (time 1754406321.70 total 70.03 last 147.71) - /unlink 10000 (time 1754406324.45 total 72.77 last 137.26) - /unlink 11442 (time 1754406334.45 total 82.77 last 144.19) - /unlink 13061 (time 1754406344.45 total 92.78 last 161.84) - /unlink 14514 (time 1754406354.46 total 102.78 last 145.19) - /unlink 16215 (time 1754406364.46 total 112.79 last 170.09) - /unlink 18754 (time 1754406374.46 total 122.79 last 253.86) - /unlink 20000 (time 1754406380.36 total 128.69 last 211.22) - /unlink 22347 (time 1754406390.36 total 138.69 last 234.68) - /unlink 24794 (time 1754406400.36 total 148.69 last 244.61) - /unlink 27274 (time 1754406410.37 total 158.70 last 247.84) - /unlink 29992 (time 1754406420.37 total 168.70 last 271.77) - /unlink 33129 (time 1754406430.37 total 178.70 last 313.64) - /unlink 36870 (time 1754406440.37 total 188.70 last 374.08) - /unlink 40000 (time 1754406450.18 total 198.51 last 319.07) - /unlink 43588 (time 1754406460.18 total 208.51 last 358.80) - /unlink 46458 (time 1754406470.19 total 218.51 last 286.96) - /unlink 49317 (time 1754406480.19 total 228.51 last 285.89) - /unlink 50000 (time 1754406482.49 total 230.81 last 296.67) - /unlink 52568 (time 1754406492.49 total 240.81 last 256.80) - /unlink 54972 (time 1754406502.49 total 250.82 last 240.34) - /unlink 57215 (time 1754406512.49 total 260.82 last 224.27) - /unlink 59846 (time 1754406522.49 total 270.82 last 263.06) - /unlink 62566 (time 1754406532.50 total 280.82 last 271.96) total: 64768 /unlink in 288.31 seconds: 224.65 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1386 (time 1754406573.81 total 10.00 last 138.56) - open/close 2993 (time 1754406583.81 total 20.01 last 160.64) - open/close 4413 (time 1754406593.81 total 30.01 last 141.99) - open/close 6138 (time 1754406603.81 total 40.01 last 172.49) - open/close 7843 (time 1754406613.82 total 50.01 last 170.43) - open/close 10000 (time 1754406623.60 total 59.79 last 220.60) - open/close 12519 (time 1754406633.60 total 69.79 last 251.85) - open/close 14461 (time 1754406643.60 total 79.79 last 194.18) - open/close 16267 (time 1754406653.60 total 89.79 last 180.55) - open/close 18440 (time 1754406663.60 total 99.80 last 217.25) - open/close 20000 (time 1754406670.86 total 107.05 last 215.00) - open/close 21702 (time 1754406680.86 total 117.05 last 170.19) - open/close 23514 (time 1754406690.86 total 127.05 last 181.18) - open/close 25625 (time 1754406700.86 total 137.06 last 211.04) - open/close 27742 (time 1754406710.87 total 147.06 last 211.68) - open/close 29914 (time 1754406720.87 total 157.06 last 217.12) - open/close 32101 (time 1754406730.87 total 167.07 last 218.57) - open/close 33821 (time 1754406740.88 total 177.07 last 171.98) - open/close 35504 (time 1754406750.88 total 187.07 last 168.24) - open/close 37454 (time 1754406760.88 total 197.07 last 194.97) - open/close 39212 (time 1754406770.89 total 207.08 last 175.71) - open/close 40000 (time 1754406775.38 total 211.58 last 175.20) - open/close 41755 (time 1754406785.39 total 221.58 last 175.44) - open/close 43777 (time 1754406795.39 total 231.58 last 202.14) - open/close 45701 (time 1754406805.40 total 241.59 last 192.27) - open/close 47835 (time 1754406815.40 total 251.59 last 213.35) - open/close 50000 (time 1754406824.36 total 260.56 last 241.50) - open/close 52531 (time 1754406834.36 total 270.56 last 253.09) - open/close 55251 (time 1754406844.37 total 280.56 last 271.91) - open/close 57639 (time 1754406854.37 total 290.57 last 238.69) - open/close 59449 (time 1754406864.37 total 300.57 last 180.95) - open/close 60000 (time 1754406866.92 total 303.11 last 216.39) - open/close 62562 (time 1754406876.92 total 313.12 last 256.17) - open/close 64505 (time 1754406886.93 total 323.12 last 194.21) total: 64768 open/close in 324.54 seconds: 199.57 ops/second - /unlink 3255 (time 1754406899.99 total 10.00 last 325.40) - /unlink 6425 (time 1754406910.00 total 20.00 last 316.95) - /unlink 10000 (time 1754406919.69 total 29.70 last 368.70) - /unlink 13781 (time 1754406929.69 total 39.70 last 378.08) - /unlink 18064 (time 1754406939.70 total 49.70 last 428.17) - /unlink 20000 (time 1754406944.20 total 54.21 last 429.68) - /unlink 23698 (time 1754406954.20 total 64.21 last 369.75) - /unlink 27625 (time 1754406964.20 total 74.21 last 392.61) - /unlink 30000 (time 1754406970.40 total 80.41 last 383.36) - /unlink 33154 (time 1754406980.40 total 90.41 last 315.32) - /unlink 36825 (time 1754406990.40 total 100.41 last 367.05) - /unlink 40000 (time 1754406999.08 total 109.09 last 365.99) - /unlink 43203 (time 1754407009.08 total 119.09 last 320.26) - /unlink 47375 (time 1754407019.08 total 129.09 last 417.19) - /unlink 50000 (time 1754407024.60 total 134.60 last 475.91) - /unlink 53862 (time 1754407034.60 total 144.61 last 385.96) - /unlink 57327 (time 1754407044.60 total 154.61 last 346.49) - /unlink 60000 (time 1754407051.05 total 161.06 last 414.68) - /unlink 63889 (time 1754407061.05 total 171.06 last 388.81) total: 64768 /unlink in 173.75 seconds: 372.77 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2079 (time 1754407092.33 total 10.00 last 207.84) - open/close 4245 (time 1754407102.33 total 20.00 last 216.59) - open/close 6218 (time 1754407112.33 total 30.00 last 197.30) - open/close 8557 (time 1754407122.33 total 40.01 last 233.84) - open/close 10000 (time 1754407128.92 total 46.59 last 219.02) - open/close 12174 (time 1754407138.92 total 56.60 last 217.39) - open/close 14490 (time 1754407148.92 total 66.60 last 231.58) - open/close 16464 (time 1754407158.93 total 76.60 last 197.26) - open/close 18646 (time 1754407168.93 total 86.60 last 218.18) - open/close 20000 (time 1754407175.90 total 93.58 last 194.12) - open/close 21986 (time 1754407185.91 total 103.58 last 198.56) - open/close 23875 (time 1754407195.91 total 113.58 last 188.85) - open/close 25855 (time 1754407205.91 total 123.59 last 197.92) - open/close 27778 (time 1754407215.92 total 133.60 last 192.17) - open/close 30000 (time 1754407225.60 total 143.28 last 229.52) - open/close 32554 (time 1754407235.60 total 153.28 last 255.32) - open/close 35007 (time 1754407245.61 total 163.28 last 245.24) - open/close 37515 (time 1754407255.61 total 173.28 last 250.73) - open/close 39846 (time 1754407265.61 total 183.29 last 233.03) - open/close 42248 (time 1754407275.62 total 193.29 last 240.09) - open/close 44527 (time 1754407285.62 total 203.29 last 227.85) - open/close 46804 (time 1754407295.62 total 213.29 last 227.69) - open/close 49131 (time 1754407305.62 total 223.30 last 232.69) - open/close 50000 (time 1754407308.80 total 226.47 last 273.56) - open/close 52466 (time 1754407318.80 total 236.47 last 246.58) - open/close 55018 (time 1754407328.80 total 246.48 last 255.10) - open/close 57110 (time 1754407338.80 total 256.48 last 209.15) - open/close 60000 (time 1754407348.24 total 265.91 last 306.35) - open/close 62744 (time 1754407358.24 total 275.92 last 274.32) total: 64768 open/close in 282.38 seconds: 229.36 ops/second - /unlink 5725 (time 1754407376.18 total 10.00 last 572.50) - /unlink 10000 (time 1754407383.68 total 17.50 last 569.75) - /unlink 15304 (time 1754407393.68 total 27.50 last 530.38) - /unlink 20000 (time 1754407403.37 total 37.19 last 484.76) - /unlink 24736 (time 1754407413.37 total 47.19 last 473.53) - /unlink 30000 (time 1754407423.31 total 57.14 last 529.42) - /unlink 35713 (time 1754407433.31 total 67.14 last 571.28) - /unlink 40000 (time 1754407440.73 total 74.55 last 578.15) - /unlink 45173 (time 1754407450.73 total 84.55 last 517.29) - /unlink 50000 (time 1754407459.71 total 93.53 last 537.47) - /unlink 56078 (time 1754407469.71 total 103.53 last 607.78) - /unlink 60000 (time 1754407477.16 total 110.98 last 526.52) total: 64768 /unlink in 119.25 seconds: 543.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2516 (time 1754407519.71 total 10.00 last 251.52) - open/close 5443 (time 1754407529.71 total 20.01 last 292.63) - open/close 8236 (time 1754407539.72 total 30.01 last 279.26) - open/close 10000 (time 1754407545.26 total 35.56 last 317.92) - open/close 12785 (time 1754407555.26 total 45.56 last 278.48) - open/close 15938 (time 1754407565.27 total 55.56 last 315.24) - open/close 19533 (time 1754407575.27 total 65.56 last 359.47) - open/close 23254 (time 1754407585.27 total 75.56 last 372.03) - open/close 26745 (time 1754407595.27 total 85.56 last 349.08) - open/close 29812 (time 1754407605.27 total 95.56 last 306.67) - open/close 32677 (time 1754407615.27 total 105.56 last 286.48) - open/close 35910 (time 1754407625.27 total 115.56 last 323.27) - open/close 38788 (time 1754407635.28 total 125.57 last 287.70) - open/close 40000 (time 1754407640.27 total 130.56 last 242.67) - open/close 42788 (time 1754407650.27 total 140.56 last 278.74) - open/close 45687 (time 1754407660.27 total 150.56 last 289.88) - open/close 48526 (time 1754407670.28 total 160.57 last 283.84) - open/close 50000 (time 1754407675.43 total 165.72 last 285.79) - open/close 53114 (time 1754407685.43 total 175.72 last 311.37) - open/close 56453 (time 1754407695.43 total 185.73 last 333.87) - open/close 59696 (time 1754407705.44 total 195.73 last 324.21) - open/close 62622 (time 1754407715.44 total 205.73 last 292.57) total: 64768 open/close in 212.51 seconds: 304.77 ops/second - /unlink 5925 (time 1754407733.66 total 10.00 last 592.49) - /unlink 10000 (time 1754407740.31 total 16.65 last 612.55) - /unlink 16508 (time 1754407750.31 total 26.65 last 650.75) - /unlink 20000 (time 1754407755.93 total 32.27 last 621.54) - /unlink 26116 (time 1754407765.93 total 42.27 last 611.60) - /unlink 30000 (time 1754407772.30 total 48.64 last 609.79) - /unlink 36993 (time 1754407782.30 total 58.64 last 699.27) - /unlink 40000 (time 1754407785.99 total 62.34 last 813.82) - /unlink 46930 (time 1754407795.99 total 72.34 last 692.97) - /unlink 50000 (time 1754407800.55 total 76.89 last 674.56) - /unlink 57562 (time 1754407810.55 total 86.89 last 756.13) - /unlink 60000 (time 1754407813.93 total 90.28 last 719.95) total: 64768 /unlink in 96.18 seconds: 673.39 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4385 (time 1754407850.32 total 10.00 last 438.48) - open/close 8634 (time 1754407860.33 total 20.00 last 424.84) - open/close 10000 (time 1754407863.74 total 23.42 last 399.74) - open/close 13822 (time 1754407873.74 total 33.42 last 382.12) - open/close 17399 (time 1754407883.75 total 43.43 last 357.55) - open/close 20000 (time 1754407890.09 total 49.77 last 409.99) - open/close 24735 (time 1754407900.09 total 59.77 last 473.50) - open/close 29363 (time 1754407910.09 total 69.77 last 462.72) - open/close 33644 (time 1754407920.10 total 79.77 last 428.03) - open/close 38038 (time 1754407930.10 total 89.77 last 439.32) - open/close 40000 (time 1754407934.72 total 94.39 last 424.87) - open/close 44272 (time 1754407944.72 total 104.39 last 427.11) - open/close 48756 (time 1754407954.72 total 114.40 last 448.32) - open/close 50000 (time 1754407957.62 total 117.30 last 428.58) - open/close 54295 (time 1754407967.62 total 127.30 last 429.48) - open/close 58490 (time 1754407977.62 total 137.30 last 419.42) - open/close 60000 (time 1754407981.39 total 141.07 last 401.15) - open/close 64280 (time 1754407991.39 total 151.07 last 427.96) total: 64768 open/close in 152.28 seconds: 425.31 ops/second - /unlink 7574 (time 1754408003.98 total 10.00 last 757.39) - /unlink 10000 (time 1754408006.98 total 13.00 last 809.48) - /unlink 18234 (time 1754408016.98 total 23.00 last 823.34) - /unlink 20000 (time 1754408019.01 total 25.03 last 868.50) - /unlink 27889 (time 1754408029.01 total 35.03 last 788.82) - /unlink 30000 (time 1754408032.43 total 38.45 last 618.31) - /unlink 36970 (time 1754408042.43 total 48.45 last 697.00) - /unlink 40000 (time 1754408046.30 total 52.31 last 783.29) - /unlink 46558 (time 1754408056.30 total 62.32 last 655.61) - /unlink 48525 (time 1754408066.31 total 72.33 last 196.46) - /unlink 50000 (time 1754408075.69 total 81.71 last 157.30) - /unlink 51360 (time 1754408085.69 total 91.71 last 135.98) - /unlink 52996 (time 1754408095.70 total 101.71 last 163.51) - /unlink 56807 (time 1754408105.70 total 111.72 last 380.99) - /unlink 60000 (time 1754408113.07 total 119.09 last 432.94) - /unlink 64267 (time 1754408123.08 total 129.09 last 426.65) total: 64768 /unlink in 130.29 seconds: 497.11 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local oleg430-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg430-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2