== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:27:14 (1778534834) Stopping clients: oleg332-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg332-client.virtnet /mnt/lustre2 (opts:-f) oleg332-server: oleg332-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg332-server: detected 4 online CPUs by sysfs oleg332-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg332-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock oleg332-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1236 (time 1778534892.43 total 10.00 last 123.55) - open/close 2401 (time 1778534902.44 total 20.01 last 116.48) - open/close 3795 (time 1778534912.44 total 30.01 last 139.36) - open/close 5224 (time 1778534922.44 total 40.01 last 142.90) - open/close 6801 (time 1778534932.44 total 50.01 last 157.64) - open/close 8300 (time 1778534942.45 total 60.02 last 149.79) - open/close 9863 (time 1778534952.46 total 70.02 last 156.23) - open/close 11559 (time 1778534962.46 total 80.03 last 169.56) - open/close 13318 (time 1778534972.46 total 90.03 last 175.87) - open/close 15158 (time 1778534982.46 total 100.03 last 183.92) - open/close 16571 (time 1778534992.46 total 110.03 last 141.28) - open/close 17943 (time 1778535002.47 total 120.03 last 137.20) - open/close 19552 (time 1778535012.47 total 130.04 last 160.85) - open/close 20000 (time 1778535015.59 total 133.16 last 143.51) - open/close 21410 (time 1778535025.60 total 143.17 last 140.90) - open/close 22788 (time 1778535035.60 total 153.17 last 137.72) - open/close 24291 (time 1778535045.61 total 163.18 last 150.23) - open/close 25649 (time 1778535055.61 total 173.18 last 135.75) - open/close 27474 (time 1778535065.62 total 183.19 last 182.37) - open/close 29291 (time 1778535075.62 total 193.19 last 181.58) - open/close 30000 (time 1778535079.02 total 196.59 last 208.83) - open/close 32069 (time 1778535089.02 total 206.59 last 206.83) - open/close 34125 (time 1778535099.02 total 216.59 last 205.57) - open/close 36615 (time 1778535109.03 total 226.60 last 248.88) - open/close 38986 (time 1778535119.03 total 236.60 last 237.06) - open/close 40000 (time 1778535123.11 total 240.68 last 248.84) - open/close 42219 (time 1778535133.11 total 250.68 last 221.85) - open/close 45084 (time 1778535143.11 total 260.68 last 286.46) - open/close 48025 (time 1778535153.11 total 270.68 last 294.02) - open/close 50000 (time 1778535160.29 total 277.86 last 275.15) - open/close 52005 (time 1778535170.30 total 287.87 last 200.37) - open/close 54388 (time 1778535180.30 total 297.87 last 238.19) - open/close 56363 (time 1778535190.31 total 307.87 last 197.43) - open/close 58253 (time 1778535200.31 total 317.88 last 188.94) - open/close 60000 (time 1778535208.52 total 326.09 last 212.62) - open/close 62247 (time 1778535218.53 total 336.10 last 224.67) total: 64768 open/close in 345.22 seconds: 187.61 ops/second - /unlink 4012 (time 1778535239.32 total 10.00 last 401.18) - /unlink 8351 (time 1778535249.32 total 20.00 last 433.81) - /unlink 10000 (time 1778535252.92 total 23.60 last 457.95) - /unlink 13967 (time 1778535262.92 total 33.60 last 396.68) - /unlink 17609 (time 1778535272.92 total 43.60 last 364.17) - /unlink 20000 (time 1778535279.60 total 50.28 last 358.40) - /unlink 24653 (time 1778535289.60 total 60.28 last 465.24) - /unlink 29780 (time 1778535299.60 total 70.28 last 512.63) - /unlink 34623 (time 1778535309.60 total 80.28 last 484.24) - /unlink 39518 (time 1778535319.60 total 90.28 last 489.48) - /unlink 43295 (time 1778535329.61 total 100.29 last 377.30) - /unlink 46425 (time 1778535339.61 total 110.29 last 312.98) - /unlink 49385 (time 1778535349.61 total 120.29 last 295.99) - /unlink 53154 (time 1778535359.62 total 130.30 last 376.77) - /unlink 56816 (time 1778535369.62 total 140.30 last 366.15) - /unlink 60000 (time 1778535378.38 total 149.06 last 363.36) - /unlink 64444 (time 1778535388.38 total 159.06 last 444.36) total: 64768 /unlink in 159.68 seconds: 405.62 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2456 (time 1778535420.29 total 10.00 last 245.60) - open/close 4813 (time 1778535430.29 total 20.00 last 235.70) - open/close 7157 (time 1778535440.29 total 30.00 last 234.39) - open/close 9803 (time 1778535450.29 total 40.00 last 264.60) - open/close 12819 (time 1778535460.29 total 50.00 last 301.50) - open/close 15782 (time 1778535470.29 total 60.01 last 296.18) - open/close 18625 (time 1778535480.30 total 70.01 last 284.22) - open/close 20000 (time 1778535486.02 total 75.74 last 240.19) - open/close 22169 (time 1778535496.02 total 85.74 last 216.84) - open/close 24301 (time 1778535506.03 total 95.74 last 213.14) - open/close 26791 (time 1778535516.03 total 105.74 last 248.93) - open/close 29291 (time 1778535526.03 total 115.75 last 249.95) - open/close 30000 (time 1778535528.49 total 118.21 last 287.87) - open/close 32912 (time 1778535538.50 total 128.21 last 291.07) - open/close 35792 (time 1778535548.51 total 138.22 last 287.81) - open/close 38720 (time 1778535558.51 total 148.22 last 292.76) - open/close 40000 (time 1778535562.03 total 151.74 last 363.69) - open/close 43592 (time 1778535572.03 total 161.74 last 359.16) - open/close 46637 (time 1778535582.03 total 171.74 last 304.48) - open/close 49774 (time 1778535592.03 total 181.74 last 313.65) - open/close 52603 (time 1778535602.03 total 191.75 last 282.85) - open/close 56141 (time 1778535612.03 total 201.75 last 353.79) - open/close 60000 (time 1778535621.02 total 210.73 last 429.49) - open/close 63858 (time 1778535631.02 total 220.73 last 385.76) total: 64768 open/close in 223.18 seconds: 290.20 ops/second - /unlink 5029 (time 1778535644.88 total 10.00 last 502.83) - /unlink 10000 (time 1778535654.01 total 19.13 last 544.57) - /unlink 15970 (time 1778535664.01 total 29.13 last 596.92) - /unlink 20000 (time 1778535670.96 total 36.08 last 579.63) - /unlink 25832 (time 1778535680.96 total 46.08 last 583.14) - /unlink 30000 (time 1778535688.31 total 53.43 last 567.27) - /unlink 34910 (time 1778535698.31 total 63.43 last 490.89) - /unlink 40000 (time 1778535707.46 total 72.58 last 556.38) - /unlink 44759 (time 1778535717.47 total 82.59 last 475.72) - /unlink 49825 (time 1778535727.47 total 92.59 last 506.60) - /unlink 55407 (time 1778535737.47 total 102.59 last 558.11) - /unlink 60000 (time 1778535745.70 total 110.82 last 557.72) total: 64768 /unlink in 118.79 seconds: 545.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4092 (time 1778535784.37 total 10.00 last 409.12) - open/close 7695 (time 1778535794.37 total 20.00 last 360.25) - open/close 10000 (time 1778535800.60 total 26.24 last 369.71) - open/close 14523 (time 1778535810.60 total 36.24 last 452.30) - open/close 17831 (time 1778535820.60 total 46.24 last 330.74) - open/close 20000 (time 1778535827.27 total 52.91 last 325.26) - open/close 23201 (time 1778535837.27 total 62.91 last 320.06) - open/close 27294 (time 1778535847.28 total 72.91 last 409.21) - open/close 30000 (time 1778535853.95 total 79.58 last 405.64) - open/close 34247 (time 1778535863.95 total 89.58 last 424.67) - open/close 38128 (time 1778535873.95 total 99.59 last 387.98) - open/close 40000 (time 1778535878.71 total 104.34 last 393.70) - open/close 44226 (time 1778535888.71 total 114.34 last 422.58) - open/close 47996 (time 1778535898.71 total 124.34 last 376.95) - open/close 50000 (time 1778535903.43 total 129.07 last 424.19) - open/close 54177 (time 1778535913.43 total 139.07 last 417.63) - open/close 58536 (time 1778535923.58 total 149.21 last 429.72) - open/close 60000 (time 1778535926.89 total 152.53 last 441.95) - open/close 64444 (time 1778535936.89 total 162.53 last 444.33) total: 64768 open/close in 163.19 seconds: 396.88 ops/second - /unlink 7157 (time 1778535948.84 total 10.00 last 715.64) - /unlink 10000 (time 1778535952.46 total 13.63 last 783.54) - /unlink 17366 (time 1778535962.47 total 23.63 last 736.57) - /unlink 20000 (time 1778535965.62 total 26.78 last 834.93) - /unlink 27885 (time 1778535975.62 total 36.78 last 788.47) - /unlink 30000 (time 1778535978.67 total 39.84 last 692.47) - /unlink 37229 (time 1778535988.68 total 49.84 last 722.83) - /unlink 40000 (time 1778535992.72 total 53.88 last 685.73) - /unlink 46995 (time 1778536002.72 total 63.88 last 699.46) - /unlink 50000 (time 1778536006.59 total 67.75 last 776.06) - /unlink 56898 (time 1778536016.59 total 77.75 last 689.74) - /unlink 60000 (time 1778536020.66 total 81.82 last 762.93) total: 64768 /unlink in 87.67 seconds: 738.76 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4732 (time 1778536055.30 total 10.00 last 473.12) - open/close 9986 (time 1778536065.30 total 20.00 last 525.37) - open/close 15232 (time 1778536075.30 total 30.00 last 524.59) - open/close 19954 (time 1778536085.30 total 40.00 last 472.12) - open/close 25077 (time 1778536095.30 total 50.01 last 512.20) - open/close 29887 (time 1778536105.30 total 60.01 last 480.98) - open/close 34938 (time 1778536115.31 total 70.01 last 504.96) - open/close 39733 (time 1778536125.31 total 80.01 last 479.43) - open/close 44008 (time 1778536135.31 total 90.01 last 427.46) - open/close 48564 (time 1778536145.31 total 100.01 last 455.53) - open/close 50000 (time 1778536148.16 total 102.87 last 503.53) - open/close 55201 (time 1778536158.16 total 112.87 last 520.10) - open/close 59583 (time 1778536168.16 total 122.87 last 438.18) - open/close 63930 (time 1778536178.16 total 132.87 last 434.66) total: 64768 open/close in 134.67 seconds: 480.94 ops/second - /unlink 8611 (time 1778536191.24 total 10.00 last 861.06) - /unlink 17363 (time 1778536201.24 total 20.00 last 875.19) - /unlink 20000 (time 1778536204.40 total 23.16 last 834.29) - /unlink 27775 (time 1778536214.41 total 33.16 last 777.42) - /unlink 30000 (time 1778536217.26 total 36.02 last 779.84) - /unlink 37590 (time 1778536227.26 total 46.02 last 758.99) - /unlink 40000 (time 1778536230.23 total 48.99 last 811.07) - /unlink 47366 (time 1778536240.23 total 58.99 last 736.59) - /unlink 50000 (time 1778536244.53 total 63.28 last 613.01) - /unlink 57782 (time 1778536254.53 total 73.28 last 778.12) - /unlink 60000 (time 1778536257.32 total 76.08 last 793.02) total: 64768 /unlink in 81.70 seconds: 792.71 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5313 (time 1778536296.88 total 10.00 last 531.22) - open/close 10000 (time 1778536305.34 total 18.46 last 554.19) - open/close 15473 (time 1778536315.34 total 28.46 last 547.20) - open/close 20000 (time 1778536323.55 total 36.67 last 551.52) - open/close 25317 (time 1778536333.55 total 46.67 last 531.62) - open/close 29882 (time 1778536343.55 total 56.67 last 456.47) - open/close 31890 (time 1778536353.56 total 66.68 last 200.65) - open/close 33077 (time 1778536363.56 total 76.68 last 118.67) - open/close 34392 (time 1778536373.57 total 86.68 last 131.46) - open/close 36000 (time 1778536383.57 total 96.69 last 160.70) - open/close 37335 (time 1778536393.58 total 106.70 last 133.43) - open/close 38778 (time 1778536403.58 total 116.70 last 144.25) - open/close 39406 (time 1778536413.60 total 126.72 last 62.69) - open/close 39959 (time 1778536423.63 total 136.75 last 55.12) - open/close 40568 (time 1778536433.63 total 146.75 last 60.86) - open/close 41490 (time 1778536443.64 total 156.76 last 92.11) - open/close 42382 (time 1778536453.65 total 166.77 last 89.15) - open/close 43185 (time 1778536463.65 total 176.77 last 80.26) - open/close 44030 (time 1778536473.66 total 186.78 last 84.45) - open/close 45065 (time 1778536483.68 total 196.80 last 103.33) - open/close 45957 (time 1778536493.68 total 206.80 last 89.15) - open/close 46808 (time 1778536503.69 total 216.80 last 85.08) - open/close 47680 (time 1778536513.69 total 226.81 last 87.18) - open/close 48533 (time 1778536523.69 total 236.81 last 85.28) - open/close 49358 (time 1778536533.69 total 246.81 last 82.46) - open/close 50000 (time 1778536541.08 total 254.20 last 86.89) - open/close 51011 (time 1778536551.09 total 264.21 last 101.01) - open/close 51910 (time 1778536561.09 total 274.21 last 89.87) - open/close 52822 (time 1778536571.10 total 284.22 last 91.13) - open/close 53769 (time 1778536581.12 total 294.24 last 94.57) - open/close 54547 (time 1778536591.12 total 304.24 last 77.78) - open/close 55439 (time 1778536601.13 total 314.25 last 89.06) - open/close 56346 (time 1778536611.14 total 324.26 last 90.63) - open/close 57200 (time 1778536621.15 total 334.27 last 85.35) - open/close 58115 (time 1778536631.15 total 344.27 last 91.44) - open/close 58998 (time 1778536641.16 total 354.28 last 88.28) - open/close 59967 (time 1778536651.17 total 364.29 last 96.80) - open/close 60890 (time 1778536661.18 total 374.30 last 92.21) - open/close 61824 (time 1778536671.19 total 384.31 last 93.32) - open/close 62859 (time 1778536681.19 total 394.31 last 103.48) - open/close 63758 (time 1778536691.20 total 404.32 last 89.82) - open/close 64653 (time 1778536701.20 total 414.32 last 89.45) total: 64768 open/close in 415.45 seconds: 155.90 ops/second - /unlink 1624 (time 1778536715.00 total 10.00 last 162.40) - /unlink 3201 (time 1778536725.01 total 20.01 last 157.59) - /unlink 4833 (time 1778536735.01 total 30.01 last 163.10) - /unlink 6459 (time 1778536745.02 total 40.01 last 162.59) - /unlink 8128 (time 1778536755.02 total 50.02 last 166.86) - /unlink 9880 (time 1778536765.02 total 60.02 last 175.18) - /unlink 11777 (time 1778536775.02 total 70.02 last 189.66) - /unlink 13382 (time 1778536785.03 total 80.03 last 160.38) - /unlink 14938 (time 1778536795.03 total 90.03 last 155.57) - /unlink 16601 (time 1778536805.03 total 100.03 last 166.24) - /unlink 18160 (time 1778536815.04 total 110.04 last 155.73) - /unlink 19734 (time 1778536825.05 total 120.05 last 157.33) - /unlink 21285 (time 1778536835.05 total 130.05 last 155.05) - /unlink 22911 (time 1778536845.05 total 140.05 last 162.58) - /unlink 24572 (time 1778536855.06 total 150.05 last 166.05) - /unlink 26121 (time 1778536865.06 total 160.06 last 154.86) - /unlink 27865 (time 1778536875.06 total 170.06 last 174.33) - /unlink 29362 (time 1778536885.07 total 180.07 last 149.58) - /unlink 30000 (time 1778536889.77 total 184.77 last 135.69) - /unlink 31501 (time 1778536899.78 total 194.78 last 150.04) - /unlink 33019 (time 1778536909.79 total 204.79 last 151.64) - /unlink 34583 (time 1778536919.79 total 214.79 last 156.30) - /unlink 36125 (time 1778536929.80 total 224.80 last 154.14) - /unlink 37622 (time 1778536939.80 total 234.80 last 149.69) - /unlink 39104 (time 1778536949.80 total 244.80 last 148.13) - /unlink 40000 (time 1778536955.22 total 250.22 last 165.42) - /unlink 41580 (time 1778536965.22 total 260.22 last 157.96) - /unlink 43266 (time 1778536975.22 total 270.22 last 168.60) - /unlink 44973 (time 1778536985.22 total 280.22 last 170.67) - /unlink 46735 (time 1778536995.22 total 290.22 last 176.20) - /unlink 48387 (time 1778537005.23 total 300.22 last 165.19) - /unlink 50000 (time 1778537014.26 total 309.26 last 178.52) - /unlink 51836 (time 1778537024.26 total 319.26 last 183.54) - /unlink 53699 (time 1778537034.27 total 329.26 last 186.28) - /unlink 55855 (time 1778537044.27 total 339.26 last 215.60) - /unlink 58303 (time 1778537054.27 total 349.27 last 244.74) - /unlink 60000 (time 1778537061.75 total 356.75 last 226.90) - /unlink 61933 (time 1778537071.75 total 366.75 last 193.27) - /unlink 63930 (time 1778537081.75 total 376.75 last 199.67) total: 64768 /unlink in 381.29 seconds: 169.87 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local oleg332-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg332-server: LNET ready to unload oleg332-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2