== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:05:34 (1777421134) Stopping clients: oleg243-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg243-client.virtnet /mnt/lustre2 (opts:-f) oleg243-server: oleg243-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg243-server: detected 4 online CPUs by sysfs oleg243-server: MODOPTS_LIBCFS= oleg243-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg243-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32583 327 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2812 (time 1777421184.35 total 10.00 last 281.16) - open/close 5515 (time 1777421194.35 total 20.00 last 270.26) - open/close 8499 (time 1777421204.36 total 30.01 last 298.19) - open/close 10000 (time 1777421209.42 total 35.07 last 296.59) - open/close 12300 (time 1777421219.42 total 45.08 last 229.89) - open/close 15262 (time 1777421229.42 total 55.08 last 296.19) - open/close 17887 (time 1777421239.42 total 65.08 last 262.50) - open/close 20000 (time 1777421249.02 total 74.68 last 220.09) - open/close 22290 (time 1777421259.02 total 84.68 last 228.97) - open/close 24700 (time 1777421269.02 total 94.68 last 240.99) - open/close 26976 (time 1777421279.03 total 104.68 last 227.53) - open/close 28779 (time 1777421289.03 total 114.68 last 180.24) - open/close 30000 (time 1777421295.67 total 121.33 last 183.82) - open/close 32623 (time 1777421305.67 total 131.33 last 262.28) - open/close 35257 (time 1777421315.68 total 141.33 last 263.35) - open/close 38230 (time 1777421325.68 total 151.33 last 297.24) - open/close 40000 (time 1777421334.17 total 159.82 last 208.52) - open/close 41893 (time 1777421344.17 total 169.82 last 189.26) - open/close 43934 (time 1777421354.17 total 179.82 last 204.10) - open/close 46287 (time 1777421364.17 total 189.82 last 235.26) - open/close 49205 (time 1777421374.17 total 199.82 last 291.79) - open/close 50000 (time 1777421376.47 total 202.13 last 345.29) - open/close 52956 (time 1777421386.48 total 212.13 last 295.52) - open/close 55235 (time 1777421396.48 total 222.13 last 227.85) - open/close 57228 (time 1777421406.48 total 232.14 last 199.25) - open/close 59402 (time 1777421416.48 total 242.14 last 217.39) - open/close 60000 (time 1777421420.10 total 245.75 last 165.46) - open/close 62436 (time 1777421430.10 total 255.75 last 243.50) - open/close 64373 (time 1777421440.10 total 265.76 last 193.63) total: 64768 open/close in 267.93 seconds: 241.73 ops/second - /unlink 4674 (time 1777421453.95 total 10.00 last 467.19) - /unlink 9560 (time 1777421463.95 total 20.01 last 488.47) - /unlink 13069 (time 1777421473.95 total 30.01 last 350.83) - /unlink 16393 (time 1777421483.95 total 40.01 last 332.34) - /unlink 20000 (time 1777421493.87 total 49.92 last 363.91) - /unlink 23405 (time 1777421503.87 total 59.92 last 340.48) - /unlink 26464 (time 1777421513.87 total 69.93 last 305.78) - /unlink 29732 (time 1777421523.87 total 79.93 last 326.79) - /unlink 32825 (time 1777421533.87 total 89.93 last 309.25) - /unlink 36049 (time 1777421543.87 total 99.93 last 322.35) - /unlink 38690 (time 1777421553.87 total 109.93 last 264.09) - /unlink 40000 (time 1777421558.51 total 114.56 last 282.75) - /unlink 43683 (time 1777421568.51 total 124.57 last 368.25) - /unlink 46596 (time 1777421578.51 total 134.57 last 291.21) - /unlink 50000 (time 1777421587.83 total 143.88 last 365.39) - /unlink 53794 (time 1777421597.83 total 153.88 last 379.38) - /unlink 57817 (time 1777421607.83 total 163.89 last 402.25) - /unlink 60000 (time 1777421615.35 total 171.41 last 290.28) - /unlink 63362 (time 1777421625.35 total 181.41 last 336.14) total: 64768 /unlink in 184.54 seconds: 350.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1923 (time 1777421658.66 total 10.00 last 192.22) - open/close 4077 (time 1777421668.66 total 20.01 last 215.36) - open/close 6713 (time 1777421678.66 total 30.01 last 263.60) - open/close 9364 (time 1777421688.66 total 40.01 last 265.08) - open/close 10000 (time 1777421691.73 total 43.07 last 207.34) - open/close 12082 (time 1777421701.73 total 53.08 last 208.18) - open/close 13724 (time 1777421711.73 total 63.08 last 164.18) - open/close 15466 (time 1777421721.73 total 73.08 last 174.15) - open/close 17317 (time 1777421731.73 total 83.08 last 185.10) - open/close 19121 (time 1777421741.74 total 93.08 last 180.31) - open/close 20000 (time 1777421746.79 total 98.14 last 174.01) - open/close 21731 (time 1777421756.79 total 108.14 last 173.09) - open/close 23637 (time 1777421766.80 total 118.14 last 190.51) - open/close 26080 (time 1777421776.80 total 128.14 last 244.22) - open/close 27901 (time 1777421786.80 total 138.15 last 182.06) - open/close 29453 (time 1777421796.81 total 148.15 last 155.11) - open/close 30000 (time 1777421801.63 total 152.97 last 113.44) - open/close 31162 (time 1777421811.63 total 162.98 last 116.15) - open/close 32404 (time 1777421821.64 total 172.98 last 124.15) - open/close 33643 (time 1777421831.64 total 182.98 last 123.89) - open/close 35121 (time 1777421841.64 total 192.98 last 147.78) - open/close 36993 (time 1777421851.64 total 202.99 last 187.18) - open/close 38800 (time 1777421861.64 total 212.99 last 180.66) - open/close 40000 (time 1777421870.58 total 221.92 last 134.33) - open/close 41439 (time 1777421880.59 total 231.93 last 143.75) - open/close 42723 (time 1777421890.59 total 241.94 last 128.35) - open/close 44102 (time 1777421900.59 total 251.94 last 137.88) - open/close 46040 (time 1777421910.59 total 261.94 last 193.79) - open/close 47473 (time 1777421920.60 total 271.94 last 143.21) - open/close 49191 (time 1777421930.61 total 281.95 last 171.65) - open/close 50000 (time 1777421936.37 total 287.72 last 140.38) - open/close 51700 (time 1777421946.37 total 297.72 last 169.98) - open/close 53223 (time 1777421956.37 total 307.72 last 152.26) - open/close 54641 (time 1777421966.38 total 317.73 last 141.68) - open/close 56341 (time 1777421976.39 total 327.73 last 169.91) - open/close 57707 (time 1777421986.40 total 337.74 last 136.47) - open/close 59254 (time 1777421996.40 total 347.75 last 154.64) - open/close 60000 (time 1777422000.67 total 352.02 last 174.63) - open/close 62536 (time 1777422010.68 total 362.02 last 253.52) - open/close 64381 (time 1777422020.68 total 372.03 last 184.42) total: 64768 open/close in 374.33 seconds: 173.03 ops/second - /unlink 3292 (time 1777422034.71 total 10.00 last 329.13) - /unlink 7078 (time 1777422044.71 total 20.00 last 378.55) - /unlink 10000 (time 1777422054.34 total 29.63 last 303.49) - /unlink 14416 (time 1777422064.34 total 39.63 last 441.57) - /unlink 19199 (time 1777422074.34 total 49.63 last 478.25) - /unlink 24764 (time 1777422084.34 total 59.63 last 556.50) - /unlink 30000 (time 1777422093.44 total 68.73 last 575.91) - /unlink 35627 (time 1777422103.44 total 78.73 last 562.68) - /unlink 40000 (time 1777422110.47 total 85.76 last 621.33) - /unlink 46660 (time 1777422120.48 total 95.77 last 665.89) - /unlink 50000 (time 1777422125.72 total 101.01 last 637.04) - /unlink 57375 (time 1777422135.72 total 111.01 last 737.43) - /unlink 60000 (time 1777422139.88 total 115.17 last 631.56) total: 64768 /unlink in 122.34 seconds: 529.42 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3939 (time 1777422182.72 total 10.00 last 393.86) - open/close 8242 (time 1777422192.72 total 20.00 last 430.25) - open/close 10000 (time 1777422197.12 total 24.40 last 399.73) - open/close 13837 (time 1777422207.12 total 34.40 last 383.66) - open/close 18043 (time 1777422217.12 total 44.40 last 420.55) - open/close 20000 (time 1777422221.61 total 48.89 last 436.32) - open/close 24792 (time 1777422231.61 total 58.89 last 479.13) - open/close 29157 (time 1777422241.61 total 68.89 last 436.48) - open/close 30000 (time 1777422243.82 total 71.10 last 381.76) - open/close 34325 (time 1777422253.82 total 81.10 last 432.46) - open/close 38836 (time 1777422263.83 total 91.11 last 450.80) - open/close 40000 (time 1777422266.42 total 93.70 last 448.59) - open/close 44756 (time 1777422276.42 total 103.70 last 475.56) - open/close 48569 (time 1777422286.43 total 113.70 last 381.22) - open/close 50000 (time 1777422290.45 total 117.72 last 355.95) - open/close 53819 (time 1777422300.45 total 127.73 last 381.83) - open/close 57911 (time 1777422310.45 total 137.73 last 409.17) - open/close 60000 (time 1777422315.99 total 143.27 last 377.04) - open/close 63801 (time 1777422325.99 total 153.27 last 380.01) total: 64768 open/close in 156.40 seconds: 414.11 ops/second - /unlink 6673 (time 1777422340.51 total 10.00 last 667.22) - /unlink 10000 (time 1777422345.39 total 14.87 last 682.88) - /unlink 16152 (time 1777422355.39 total 24.87 last 615.13) - /unlink 20000 (time 1777422361.61 total 31.10 last 617.96) - /unlink 27185 (time 1777422371.61 total 41.10 last 718.47) - /unlink 30000 (time 1777422375.76 total 45.25 last 678.71) - /unlink 37353 (time 1777422385.76 total 55.25 last 735.22) - /unlink 40000 (time 1777422390.53 total 60.02 last 555.50) - /unlink 46384 (time 1777422400.53 total 70.02 last 638.36) - /unlink 50000 (time 1777422405.61 total 75.10 last 711.09) - /unlink 57534 (time 1777422415.61 total 85.10 last 753.34) - /unlink 60000 (time 1777422418.62 total 88.11 last 819.42) total: 64768 /unlink in 95.33 seconds: 679.39 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2943 (time 1777422457.25 total 10.00 last 294.29) - open/close 5704 (time 1777422467.25 total 20.00 last 276.06) - open/close 9561 (time 1777422477.25 total 30.00 last 385.68) - open/close 14373 (time 1777422487.25 total 40.00 last 481.14) - open/close 18915 (time 1777422497.26 total 50.00 last 454.17) - open/close 20000 (time 1777422499.67 total 52.42 last 448.48) - open/close 24426 (time 1777422509.68 total 62.43 last 442.49) - open/close 28273 (time 1777422519.68 total 72.43 last 384.69) - open/close 30000 (time 1777422523.81 total 76.56 last 417.71) - open/close 34452 (time 1777422533.81 total 86.56 last 445.17) - open/close 38515 (time 1777422543.81 total 96.56 last 406.26) - open/close 40000 (time 1777422546.88 total 99.63 last 484.04) - open/close 45315 (time 1777422556.88 total 109.63 last 531.46) - open/close 50000 (time 1777422565.98 total 118.73 last 515.14) - open/close 54986 (time 1777422575.98 total 128.73 last 498.54) - open/close 59582 (time 1777422585.98 total 138.73 last 459.57) - open/close 64293 (time 1777422595.98 total 148.73 last 471.03) total: 64768 open/close in 150.14 seconds: 431.38 ops/second - /unlink 7487 (time 1777422608.79 total 10.00 last 748.48) - /unlink 10000 (time 1777422612.35 total 13.56 last 705.86) - /unlink 16944 (time 1777422622.35 total 23.56 last 694.35) - /unlink 20000 (time 1777422626.78 total 28.00 last 688.69) - /unlink 27860 (time 1777422636.79 total 38.00 last 785.98) - /unlink 30000 (time 1777422639.84 total 41.06 last 700.37) - /unlink 37331 (time 1777422649.84 total 51.06 last 733.01) - /unlink 40000 (time 1777422652.84 total 54.06 last 889.72) - /unlink 47604 (time 1777422662.84 total 64.06 last 760.39) - /unlink 50000 (time 1777422666.14 total 67.35 last 727.16) - /unlink 57652 (time 1777422676.14 total 77.35 last 765.20) - /unlink 60000 (time 1777422678.76 total 79.98 last 894.50) total: 64768 /unlink in 85.92 seconds: 753.85 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4993 (time 1777422715.91 total 10.00 last 499.29) - open/close 9213 (time 1777422725.91 total 20.00 last 421.88) - open/close 10000 (time 1777422728.14 total 22.23 last 354.04) - open/close 14181 (time 1777422738.14 total 32.23 last 418.09) - open/close 18678 (time 1777422748.14 total 42.23 last 449.63) - open/close 20000 (time 1777422751.22 total 45.31 last 428.47) - open/close 24155 (time 1777422761.23 total 55.31 last 415.43) - open/close 28433 (time 1777422771.23 total 65.32 last 427.75) - open/close 30000 (time 1777422774.27 total 68.36 last 514.45) - open/close 34942 (time 1777422784.27 total 78.36 last 494.19) - open/close 39070 (time 1777422794.27 total 88.36 last 412.77) - open/close 40000 (time 1777422796.74 total 90.83 last 377.74) - open/close 44265 (time 1777422806.74 total 100.83 last 426.42) - open/close 48676 (time 1777422816.74 total 110.83 last 441.06) - open/close 50000 (time 1777422819.59 total 113.68 last 464.18) - open/close 54466 (time 1777422829.59 total 123.68 last 446.57) - open/close 58922 (time 1777422839.59 total 133.68 last 445.58) - open/close 60000 (time 1777422842.05 total 136.14 last 438.46) total: 64768 open/close in 145.94 seconds: 443.80 ops/second - /unlink 7756 (time 1777422863.18 total 10.00 last 775.54) - /unlink 10000 (time 1777422866.02 total 12.84 last 790.30) - /unlink 18144 (time 1777422876.02 total 22.84 last 814.24) - /unlink 20000 (time 1777422878.53 total 25.35 last 738.70) - /unlink 27296 (time 1777422888.54 total 35.36 last 729.49) - /unlink 30000 (time 1777422892.86 total 39.68 last 625.46) - /unlink 36733 (time 1777422902.86 total 49.68 last 673.24) - /unlink 40000 (time 1777422907.00 total 53.82 last 790.11) - /unlink 47603 (time 1777422917.00 total 63.82 last 760.17) - /unlink 50000 (time 1777422920.33 total 67.15 last 719.13) - /unlink 57309 (time 1777422930.33 total 77.15 last 730.90) - /unlink 60000 (time 1777422933.89 total 80.71 last 756.87) total: 64768 /unlink in 87.01 seconds: 744.37 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server unloading modules via unload_modules_local on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2