== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:25:31 (1729088731) Stopping clients: oleg135-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg135-client.virtnet /mnt/lustre2 (opts:-f) oleg135-server: oleg135-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: Force libcfs to create 2 CPU partitions oleg135-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg135-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg135-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg135-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1589 (time 1729088783.59 total 10.00 last 158.86) - open/close 3075 (time 1729088793.60 total 20.00 last 148.57) - open/close 4644 (time 1729088803.60 total 30.01 last 156.79) - open/close 6428 (time 1729088813.61 total 40.01 last 178.34) - open/close 8073 (time 1729088823.61 total 50.02 last 164.45) - open/close 9665 (time 1729088833.61 total 60.02 last 159.19) - open/close 11155 (time 1729088843.61 total 70.02 last 148.96) - open/close 12628 (time 1729088853.61 total 80.02 last 147.29) - open/close 14302 (time 1729088863.62 total 90.02 last 167.37) - open/close 15856 (time 1729088873.62 total 100.03 last 155.36) - open/close 17485 (time 1729088883.62 total 110.03 last 162.85) - open/close 19127 (time 1729088893.62 total 120.03 last 164.17) - open/close 20000 (time 1729088898.63 total 125.04 last 174.40) - open/close 21812 (time 1729088908.63 total 135.04 last 181.20) - open/close 23600 (time 1729088918.63 total 145.04 last 178.76) - open/close 25604 (time 1729088928.63 total 155.04 last 200.39) - open/close 27542 (time 1729088938.64 total 165.04 last 193.72) - open/close 29478 (time 1729088948.64 total 175.05 last 193.44) - open/close 30000 (time 1729088950.82 total 177.22 last 240.27) - open/close 31746 (time 1729088960.82 total 187.23 last 174.53) - open/close 33614 (time 1729088970.82 total 197.23 last 186.72) - open/close 35688 (time 1729088980.83 total 207.24 last 207.33) - open/close 37997 (time 1729088990.83 total 217.24 last 230.85) - open/close 40000 (time 1729088999.73 total 226.13 last 225.17) - open/close 41929 (time 1729089009.73 total 236.14 last 192.87) - open/close 44684 (time 1729089019.73 total 246.14 last 275.49) - open/close 46596 (time 1729089029.73 total 256.14 last 191.16) - open/close 48640 (time 1729089039.73 total 266.14 last 204.39) - open/close 50000 (time 1729089048.11 total 274.52 last 162.34) - open/close 52095 (time 1729089058.11 total 284.52 last 209.46) - open/close 53853 (time 1729089068.11 total 294.52 last 175.76) - open/close 56327 (time 1729089078.11 total 304.52 last 247.33) - open/close 58798 (time 1729089088.12 total 314.53 last 247.01) - open/close 60000 (time 1729089093.10 total 319.51 last 241.39) - open/close 62949 (time 1729089103.10 total 329.51 last 294.87) total: 64768 open/close in 337.35 seconds: 191.99 ops/second - /unlink 4550 (time 1729089121.95 total 10.00 last 454.88) - /unlink 9501 (time 1729089131.95 total 20.00 last 495.05) - /unlink 13382 (time 1729089141.95 total 30.00 last 388.07) - /unlink 17315 (time 1729089151.96 total 40.01 last 393.25) - /unlink 20000 (time 1729089158.43 total 46.48 last 414.50) - /unlink 24078 (time 1729089168.43 total 56.48 last 407.74) - /unlink 28899 (time 1729089178.44 total 66.49 last 482.06) - /unlink 30000 (time 1729089181.14 total 69.19 last 407.25) - /unlink 34343 (time 1729089191.14 total 79.19 last 434.28) - /unlink 38924 (time 1729089201.14 total 89.19 last 458.04) - /unlink 40000 (time 1729089203.46 total 91.51 last 464.50) - /unlink 45155 (time 1729089213.46 total 101.51 last 515.49) - /unlink 49650 (time 1729089223.46 total 111.51 last 449.29) - /unlink 53339 (time 1729089233.46 total 121.51 last 368.83) - /unlink 57145 (time 1729089243.47 total 131.52 last 380.47) - /unlink 60000 (time 1729089250.98 total 139.03 last 380.07) - /unlink 63894 (time 1729089260.98 total 149.03 last 389.36) total: 64768 /unlink in 151.58 seconds: 427.29 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2009 (time 1729089290.23 total 10.00 last 200.89) - open/close 4174 (time 1729089300.23 total 20.00 last 216.49) - open/close 6550 (time 1729089310.24 total 30.00 last 237.58) - open/close 9211 (time 1729089320.24 total 40.00 last 266.07) - open/close 10000 (time 1729089322.67 total 42.44 last 323.74) - open/close 12753 (time 1729089332.68 total 52.44 last 275.25) - open/close 15601 (time 1729089342.68 total 62.44 last 284.77) - open/close 18639 (time 1729089352.68 total 72.45 last 303.64) - open/close 20000 (time 1729089357.86 total 77.63 last 262.66) - open/close 22659 (time 1729089367.86 total 87.63 last 265.87) - open/close 25166 (time 1729089377.86 total 97.63 last 250.69) - open/close 27428 (time 1729089387.87 total 107.63 last 226.18) - open/close 29446 (time 1729089397.87 total 117.63 last 201.80) - open/close 30000 (time 1729089400.95 total 120.72 last 179.49) - open/close 31726 (time 1729089410.96 total 130.72 last 172.52) - open/close 34034 (time 1729089420.96 total 140.73 last 230.76) - open/close 36431 (time 1729089430.96 total 150.73 last 239.70) - open/close 39176 (time 1729089440.96 total 160.73 last 274.49) - open/close 40000 (time 1729089444.58 total 164.35 last 227.32) - open/close 42635 (time 1729089454.59 total 174.35 last 263.41) - open/close 45235 (time 1729089464.59 total 184.35 last 259.98) - open/close 48477 (time 1729089474.59 total 194.36 last 324.17) - open/close 50000 (time 1729089479.39 total 199.15 last 317.56) - open/close 52659 (time 1729089489.39 total 209.15 last 265.90) - open/close 54843 (time 1729089499.39 total 219.15 last 218.37) - open/close 56752 (time 1729089509.39 total 229.15 last 190.88) - open/close 59606 (time 1729089519.39 total 239.16 last 285.34) - open/close 63001 (time 1729089529.39 total 249.16 last 339.41) total: 64768 open/close in 254.78 seconds: 254.22 ops/second - /unlink 5339 (time 1729089545.94 total 10.00 last 533.88) - /unlink 10000 (time 1729089553.76 total 17.82 last 596.41) - /unlink 17159 (time 1729089563.76 total 27.82 last 715.84) - /unlink 20000 (time 1729089567.23 total 31.28 last 819.36) - /unlink 27571 (time 1729089577.23 total 41.28 last 757.05) - /unlink 30000 (time 1729089580.26 total 44.32 last 799.89) - /unlink 36692 (time 1729089590.26 total 54.32 last 669.18) - /unlink 40000 (time 1729089595.61 total 59.67 last 618.48) - /unlink 48206 (time 1729089605.61 total 69.67 last 820.57) - /unlink 50000 (time 1729089607.87 total 71.93 last 793.95) - /unlink 59203 (time 1729089617.87 total 81.93 last 920.29) total: 64768 /unlink in 87.47 seconds: 740.45 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5139 (time 1729089656.00 total 10.00 last 513.86) - open/close 9653 (time 1729089666.00 total 20.00 last 451.31) - open/close 14424 (time 1729089676.00 total 30.00 last 477.01) - open/close 19869 (time 1729089686.00 total 40.00 last 544.48) - open/close 25778 (time 1729089696.00 total 50.00 last 590.90) - open/close 30000 (time 1729089703.13 total 57.13 last 592.36) - open/close 35141 (time 1729089713.13 total 67.13 last 514.07) - open/close 40000 (time 1729089722.60 total 76.61 last 512.94) - open/close 45223 (time 1729089732.60 total 86.61 last 522.28) - open/close 50000 (time 1729089742.29 total 96.29 last 493.28) - open/close 55192 (time 1729089752.29 total 106.29 last 519.18) - open/close 60000 (time 1729089761.31 total 115.32 last 532.60) - open/close 64705 (time 1729089771.31 total 125.32 last 470.42) total: 64768 open/close in 125.46 seconds: 516.25 ops/second - /unlink 8177 (time 1729089782.33 total 10.00 last 817.66) - /unlink 10000 (time 1729089784.76 total 12.43 last 750.70) - /unlink 18208 (time 1729089794.76 total 22.43 last 820.78) - /unlink 20000 (time 1729089797.28 total 24.95 last 710.22) - /unlink 26312 (time 1729089807.28 total 34.95 last 631.14) - /unlink 30000 (time 1729089812.67 total 40.34 last 684.05) - /unlink 36962 (time 1729089822.67 total 50.34 last 696.19) - /unlink 40000 (time 1729089826.65 total 54.33 last 763.23) - /unlink 48358 (time 1729089836.65 total 64.33 last 835.79) - /unlink 50000 (time 1729089838.83 total 66.51 last 752.93) - /unlink 58014 (time 1729089848.83 total 76.51 last 801.32) - /unlink 60000 (time 1729089851.37 total 79.04 last 784.29) total: 64768 /unlink in 84.91 seconds: 762.77 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4433 (time 1729089886.77 total 10.00 last 443.27) - open/close 8802 (time 1729089896.78 total 20.00 last 436.84) - open/close 10000 (time 1729089899.34 total 22.56 last 467.81) - open/close 14413 (time 1729089909.34 total 32.56 last 441.22) - open/close 19032 (time 1729089919.34 total 42.57 last 461.80) - open/close 20000 (time 1729089921.51 total 44.73 last 446.97) - open/close 24761 (time 1729089931.51 total 54.73 last 476.01) - open/close 29448 (time 1729089941.51 total 64.74 last 468.65) - open/close 34124 (time 1729089951.51 total 74.74 last 467.56) - open/close 37736 (time 1729089961.51 total 84.74 last 361.11) - open/close 40000 (time 1729089969.91 total 93.13 last 269.72) - open/close 44426 (time 1729089979.91 total 103.13 last 442.57) - open/close 49635 (time 1729089989.91 total 113.13 last 520.85) - open/close 53174 (time 1729089999.91 total 123.13 last 353.89) - open/close 57484 (time 1729090009.91 total 133.14 last 430.95) - open/close 60000 (time 1729090015.19 total 138.42 last 476.03) total: 64768 open/close in 147.50 seconds: 439.09 ops/second - /unlink 8221 (time 1729090035.07 total 10.00 last 822.04) - /unlink 18028 (time 1729090045.07 total 20.00 last 980.69) - /unlink 27884 (time 1729090055.07 total 30.00 last 985.53) - /unlink 30000 (time 1729090057.34 total 32.27 last 932.67) - /unlink 39202 (time 1729090067.34 total 42.27 last 920.13) - /unlink 48310 (time 1729090077.34 total 52.27 last 910.77) - /unlink 56097 (time 1729090087.34 total 62.27 last 778.64) - /unlink 60000 (time 1729090092.05 total 66.98 last 828.23) total: 64768 /unlink in 72.85 seconds: 889.03 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4789 (time 1729090136.76 total 10.00 last 478.85) - open/close 9690 (time 1729090146.76 total 20.00 last 490.04) - open/close 14519 (time 1729090156.76 total 30.00 last 482.85) - open/close 19235 (time 1729090166.76 total 40.01 last 471.49) - open/close 20000 (time 1729090168.91 total 42.16 last 355.70) - open/close 23442 (time 1729090178.91 total 52.16 last 344.12) - open/close 28055 (time 1729090188.92 total 62.16 last 461.22) - open/close 30000 (time 1729090193.12 total 66.36 last 462.81) - open/close 34512 (time 1729090203.12 total 76.37 last 451.10) - open/close 39213 (time 1729090213.12 total 86.37 last 470.06) - open/close 44006 (time 1729090223.12 total 96.37 last 479.29) - open/close 47771 (time 1729090233.12 total 106.37 last 376.49) - open/close 50000 (time 1729090240.81 total 114.06 last 289.83) - open/close 52860 (time 1729090250.81 total 124.06 last 285.96) - open/close 55631 (time 1729090260.89 total 134.13 last 275.02) - open/close 58612 (time 1729090270.89 total 144.14 last 298.05) - open/close 60000 (time 1729090275.54 total 148.78 last 298.61) - open/close 63120 (time 1729090285.54 total 158.79 last 311.93) total: 64768 open/close in 164.63 seconds: 393.42 ops/second - /unlink 5044 (time 1729090302.41 total 10.00 last 504.36) - /unlink 9223 (time 1729090312.41 total 20.00 last 417.83) - /unlink 14466 (time 1729090322.41 total 30.00 last 524.25) - /unlink 19193 (time 1729090332.41 total 40.00 last 472.69) - /unlink 24886 (time 1729090342.41 total 50.00 last 569.28) - /unlink 29606 (time 1729090352.41 total 60.00 last 471.96) - /unlink 34589 (time 1729090362.42 total 70.01 last 498.19) - /unlink 40000 (time 1729090372.08 total 79.67 last 560.02) - /unlink 46142 (time 1729090382.08 total 89.67 last 614.12) - /unlink 50000 (time 1729090388.27 total 95.86 last 623.25) - /unlink 57084 (time 1729090398.27 total 105.86 last 708.33) - /unlink 60000 (time 1729090403.06 total 110.65 last 609.24) total: 64768 /unlink in 118.49 seconds: 546.62 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server unloading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2