== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:31:45 (1751319105) Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f) oleg241-server: oleg241-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: detected 4 online CPUs by sysfs oleg241-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg241-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg241-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock oleg241-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16169 329 15840 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 4925 (time 1751319140.74 total 10.00 last 492.50) - open/close 9741 (time 1751319150.74 total 20.00 last 481.60) - open/close 14775 (time 1751319160.74 total 30.00 last 503.32) - open/close 19545 (time 1751319170.74 total 40.00 last 476.98) - open/close 25323 (time 1751319180.74 total 50.00 last 577.74) - open/close 30000 (time 1751319189.06 total 58.32 last 562.63) - open/close 35985 (time 1751319199.06 total 68.32 last 598.42) - open/close 40000 (time 1751319206.12 total 75.39 last 568.06) - open/close 45505 (time 1751319216.13 total 85.39 last 550.45) - open/close 50000 (time 1751319223.66 total 92.92 last 596.85) - open/close 56036 (time 1751319233.66 total 102.92 last 603.54) - open/close 60000 (time 1751319240.42 total 109.68 last 586.02) total: 64768 open/close in 118.08 seconds: 548.50 ops/second - /unlink 10000 (time 1751319259.43 total 9.83 last 1017.00) - /unlink 20000 (time 1751319269.21 total 19.61 last 1023.22) - /unlink 30000 (time 1751319278.96 total 29.36 last 1025.69) - /unlink 40000 (time 1751319288.44 total 38.84 last 1054.46) - /unlink 50000 (time 1751319297.96 total 48.35 last 1051.05) - /unlink 60000 (time 1751319307.72 total 58.12 last 1023.62) total: 64768 /unlink in 62.78 seconds: 1031.59 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4998 (time 1751319360.02 total 10.05 last 497.31) - open/close 10000 (time 1751319369.94 total 19.96 last 504.66) - open/close 15144 (time 1751319379.94 total 29.96 last 514.36) - open/close 19954 (time 1751319389.94 total 39.96 last 480.96) - open/close 24983 (time 1751319399.94 total 49.96 last 502.87) - open/close 30000 (time 1751319409.83 total 59.86 last 507.07) - open/close 35051 (time 1751319419.83 total 69.86 last 505.06) - open/close 40000 (time 1751319429.63 total 79.66 last 504.99) - open/close 45084 (time 1751319439.64 total 89.66 last 508.32) - open/close 50000 (time 1751319449.45 total 99.48 last 500.64) - open/close 55246 (time 1751319459.46 total 109.48 last 524.57) - open/close 60000 (time 1751319468.78 total 118.81 last 509.75) total: 64768 open/close in 128.31 seconds: 504.79 ops/second - /unlink 8667 (time 1751319489.13 total 10.00 last 866.61) - /unlink 16938 (time 1751319499.13 total 20.00 last 827.07) - /unlink 20000 (time 1751319502.63 total 23.50 last 876.40) - /unlink 28526 (time 1751319512.63 total 33.50 last 852.59) - /unlink 37294 (time 1751319522.63 total 43.50 last 876.79) - /unlink 40000 (time 1751319525.77 total 46.64 last 861.02) - /unlink 48977 (time 1751319535.77 total 56.64 last 897.61) - /unlink 57923 (time 1751319545.78 total 66.64 last 894.14) - /unlink 60000 (time 1751319547.98 total 68.85 last 941.86) total: 64768 /unlink in 74.03 seconds: 874.94 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5447 (time 1751319590.59 total 10.00 last 544.64) - open/close 10000 (time 1751319598.76 total 18.17 last 557.33) - open/close 15487 (time 1751319608.76 total 28.17 last 548.62) - open/close 20000 (time 1751319617.13 total 36.54 last 539.41) - open/close 25415 (time 1751319627.13 total 46.54 last 541.48) - open/close 30000 (time 1751319635.54 total 54.95 last 545.11) - open/close 35348 (time 1751319645.54 total 64.95 last 534.73) - open/close 40000 (time 1751319654.18 total 73.59 last 538.55) - open/close 44986 (time 1751319664.18 total 83.59 last 498.56) - open/close 49404 (time 1751319674.21 total 93.62 last 440.56) - open/close 54311 (time 1751319684.21 total 103.62 last 490.62) - open/close 58437 (time 1751319694.21 total 113.62 last 412.58) - open/close 60000 (time 1751319697.34 total 116.75 last 499.54) total: 64768 open/close in 126.07 seconds: 513.74 ops/second - /unlink 9160 (time 1751319717.52 total 10.00 last 915.96) - /unlink 18117 (time 1751319727.52 total 20.00 last 895.65) - /unlink 20000 (time 1751319729.65 total 22.13 last 883.24) - /unlink 28716 (time 1751319739.65 total 32.13 last 871.51) - /unlink 37558 (time 1751319749.65 total 42.14 last 884.11) - /unlink 40000 (time 1751319752.38 total 44.86 last 895.32) - /unlink 49001 (time 1751319762.38 total 54.86 last 900.05) - /unlink 58136 (time 1751319772.38 total 64.86 last 913.42) - /unlink 60000 (time 1751319774.42 total 66.90 last 914.68) total: 64768 /unlink in 71.98 seconds: 899.81 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3206 (time 1751319821.23 total 10.00 last 320.48) - open/close 6403 (time 1751319831.23 total 20.01 last 319.65) - open/close 10000 (time 1751319839.17 total 27.95 last 452.76) - open/close 14740 (time 1751319849.17 total 37.95 last 473.95) - open/close 17788 (time 1751319859.18 total 47.95 last 304.74) - open/close 20000 (time 1751319866.78 total 55.56 last 290.74) - open/close 22896 (time 1751319876.79 total 65.56 last 289.54) - open/close 25860 (time 1751319886.79 total 75.56 last 296.34) - open/close 29161 (time 1751319896.79 total 85.57 last 330.09) - open/close 32272 (time 1751319906.79 total 95.57 last 311.00) - open/close 35434 (time 1751319916.80 total 105.57 last 316.09) - open/close 38529 (time 1751319926.80 total 115.57 last 309.49) - open/close 40000 (time 1751319932.12 total 120.90 last 276.33) - open/close 43309 (time 1751319942.12 total 130.90 last 330.86) - open/close 45845 (time 1751319952.12 total 140.90 last 253.56) - open/close 48422 (time 1751319962.12 total 150.90 last 257.69) - open/close 50000 (time 1751319968.74 total 157.52 last 238.38) - open/close 52827 (time 1751319978.74 total 167.52 last 282.70) - open/close 55263 (time 1751319988.74 total 177.52 last 243.58) - open/close 57717 (time 1751319998.75 total 187.52 last 245.33) - open/close 60000 (time 1751320008.21 total 196.99 last 241.10) - open/close 62495 (time 1751320018.22 total 206.99 last 249.42) total: 64768 open/close in 216.29 seconds: 299.45 ops/second - /unlink 4471 (time 1751320038.80 total 10.00 last 447.05) - /unlink 9518 (time 1751320048.80 total 20.00 last 504.60) - /unlink 14060 (time 1751320058.81 total 30.00 last 454.13) - /unlink 18633 (time 1751320068.81 total 40.01 last 457.20) - /unlink 20000 (time 1751320071.70 total 42.90 last 471.97) - /unlink 24740 (time 1751320081.71 total 52.91 last 473.92) - /unlink 29379 (time 1751320091.71 total 62.91 last 463.86) - /unlink 34404 (time 1751320101.71 total 72.91 last 502.46) - /unlink 39104 (time 1751320111.71 total 82.91 last 469.95) - /unlink 40000 (time 1751320113.74 total 84.94 last 441.86) - /unlink 44588 (time 1751320123.74 total 94.94 last 458.78) - /unlink 49173 (time 1751320133.74 total 104.94 last 458.47) - /unlink 54253 (time 1751320143.74 total 114.94 last 507.94) - /unlink 59516 (time 1751320153.74 total 124.94 last 526.30) - /unlink 64335 (time 1751320163.74 total 134.94 last 481.90) total: 64768 /unlink in 135.89 seconds: 476.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2525 (time 1751320197.13 total 10.00 last 252.43) - open/close 5027 (time 1751320207.14 total 20.00 last 250.17) - open/close 7579 (time 1751320217.14 total 30.00 last 255.20) - open/close 10000 (time 1751320227.13 total 40.00 last 242.26) - open/close 12760 (time 1751320237.13 total 50.00 last 275.98) - open/close 15219 (time 1751320247.13 total 60.00 last 245.84) - open/close 17654 (time 1751320257.13 total 70.00 last 243.48) - open/close 20000 (time 1751320266.79 total 79.65 last 243.02) - open/close 22507 (time 1751320276.79 total 89.66 last 250.65) - open/close 25056 (time 1751320286.79 total 99.66 last 254.87) - open/close 27477 (time 1751320296.79 total 109.66 last 242.08) - open/close 29952 (time 1751320306.79 total 119.66 last 247.45) - open/close 32451 (time 1751320316.80 total 129.66 last 249.83) - open/close 34839 (time 1751320326.80 total 139.67 last 238.73) - open/close 37174 (time 1751320336.80 total 149.67 last 233.43) - open/close 39595 (time 1751320346.80 total 159.67 last 242.02) - open/close 42066 (time 1751320356.81 total 169.67 last 247.06) - open/close 44489 (time 1751320366.81 total 179.68 last 242.27) - open/close 46977 (time 1751320376.81 total 189.68 last 248.72) - open/close 49362 (time 1751320386.81 total 199.68 last 238.46) - open/close 50000 (time 1751320389.44 total 202.30 last 243.29) - open/close 52337 (time 1751320399.44 total 212.31 last 233.65) - open/close 54771 (time 1751320409.44 total 222.31 last 243.34) - open/close 57164 (time 1751320419.44 total 232.31 last 239.26) - open/close 59562 (time 1751320429.44 total 242.31 last 239.77) - open/close 62014 (time 1751320439.45 total 252.31 last 245.12) - open/close 64367 (time 1751320449.45 total 262.32 last 235.27) total: 64768 open/close in 264.04 seconds: 245.30 ops/second - /unlink 4570 (time 1751320462.41 total 10.00 last 456.93) - /unlink 9187 (time 1751320472.41 total 20.00 last 461.65) - /unlink 13773 (time 1751320482.41 total 30.00 last 458.59) - /unlink 18279 (time 1751320492.41 total 40.00 last 450.59) - /unlink 20000 (time 1751320496.25 total 43.84 last 448.26) - /unlink 24574 (time 1751320506.26 total 53.84 last 457.29) - /unlink 29380 (time 1751320516.26 total 63.85 last 480.56) - /unlink 33856 (time 1751320526.26 total 73.85 last 447.50) - /unlink 38480 (time 1751320536.26 total 83.85 last 462.35) - /unlink 40000 (time 1751320539.43 total 87.02 last 479.97) - /unlink 44784 (time 1751320549.43 total 97.02 last 478.30) - /unlink 49407 (time 1751320559.43 total 107.02 last 462.29) - /unlink 54047 (time 1751320569.43 total 117.02 last 464.00) - /unlink 58767 (time 1751320579.43 total 127.02 last 472.00) - /unlink 60000 (time 1751320582.03 total 129.62 last 473.73) - /unlink 64646 (time 1751320592.03 total 139.62 last 464.54) total: 64768 /unlink in 139.90 seconds: 462.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server LNET ready to unload unloading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing unload_modules_local oleg241-server: LNET ready to unload modules unloaded. pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2