== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:52:46 (1747828366) Stopping clients: oleg140-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg140-client.virtnet /mnt/lustre2 (opts:-f) oleg140-server: oleg140-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: detected 4 online CPUs by sysfs oleg140-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg140-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg140-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock oleg140-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16169 329 15840 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 4152 (time 1747828403.74 total 10.00 last 415.12) - open/close 7308 (time 1747828413.74 total 20.00 last 315.53) - open/close 10000 (time 1747828421.41 total 27.67 last 351.36) - open/close 13222 (time 1747828431.41 total 37.67 last 322.17) - open/close 16503 (time 1747828441.41 total 47.67 last 328.01) - open/close 20000 (time 1747828449.78 total 56.04 last 417.98) - open/close 24816 (time 1747828459.78 total 66.04 last 481.59) - open/close 29779 (time 1747828469.78 total 76.04 last 496.30) - open/close 35356 (time 1747828479.78 total 86.04 last 557.66) - open/close 40000 (time 1747828488.62 total 94.88 last 525.28) - open/close 45127 (time 1747828498.62 total 104.88 last 512.61) - open/close 50000 (time 1747828507.85 total 114.11 last 528.21) - open/close 54638 (time 1747828517.85 total 124.11 last 463.75) - open/close 58836 (time 1747828527.85 total 134.11 last 419.76) - open/close 60000 (time 1747828530.86 total 137.12 last 386.35) total: 64768 open/close in 147.05 seconds: 440.44 ops/second - /unlink 8433 (time 1747828551.65 total 10.00 last 843.25) - /unlink 16889 (time 1747828561.65 total 20.00 last 845.54) - /unlink 20000 (time 1747828565.08 total 23.43 last 908.46) - /unlink 28684 (time 1747828575.08 total 33.43 last 868.39) - /unlink 37276 (time 1747828585.08 total 43.43 last 859.15) - /unlink 40000 (time 1747828588.07 total 46.42 last 909.95) - /unlink 49056 (time 1747828598.07 total 56.42 last 905.58) - /unlink 58213 (time 1747828608.07 total 66.42 last 915.69) total: 64768 /unlink in 72.79 seconds: 889.81 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5298 (time 1747828652.71 total 10.00 last 529.79) - open/close 10000 (time 1747828660.87 total 18.15 last 576.59) - open/close 15574 (time 1747828670.87 total 28.16 last 557.32) - open/close 20000 (time 1747828678.83 total 36.11 last 556.15) - open/close 25253 (time 1747828688.83 total 46.12 last 525.24) - open/close 30000 (time 1747828697.76 total 55.05 last 531.33) - open/close 35312 (time 1747828707.76 total 65.05 last 531.16) - open/close 40000 (time 1747828716.65 total 73.94 last 527.41) - open/close 45486 (time 1747828726.65 total 83.94 last 548.55) - open/close 50000 (time 1747828734.97 total 92.26 last 542.77) - open/close 55638 (time 1747828745.02 total 102.30 last 561.28) - open/close 60000 (time 1747828752.67 total 109.96 last 569.68) total: 64768 open/close in 118.82 seconds: 545.08 ops/second - /unlink 9590 (time 1747828772.33 total 10.00 last 958.91) - /unlink 20000 (time 1747828782.21 total 19.89 last 1053.11) - /unlink 30000 (time 1747828791.71 total 29.38 last 1053.46) - /unlink 39106 (time 1747828801.71 total 39.38 last 910.57) - /unlink 48313 (time 1747828811.71 total 49.38 last 920.65) - /unlink 57521 (time 1747828821.71 total 59.38 last 920.71) - /unlink 60000 (time 1747828824.33 total 62.00 last 946.39) total: 64768 /unlink in 67.37 seconds: 961.34 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5280 (time 1747828872.95 total 10.00 last 527.94) - open/close 10000 (time 1747828881.65 total 18.70 last 542.86) - open/close 15394 (time 1747828891.65 total 28.70 last 539.39) - open/close 20000 (time 1747828900.65 total 37.70 last 511.54) - open/close 25268 (time 1747828910.65 total 47.70 last 526.77) - open/close 30000 (time 1747828919.82 total 56.87 last 516.17) - open/close 35323 (time 1747828929.82 total 66.87 last 532.21) - open/close 40000 (time 1747828938.71 total 75.76 last 526.29) - open/close 45265 (time 1747828948.71 total 85.76 last 526.45) - open/close 50000 (time 1747828957.97 total 95.02 last 511.14) - open/close 55208 (time 1747828967.97 total 105.02 last 520.78) - open/close 60000 (time 1747828977.13 total 114.18 last 523.51) total: 64768 open/close in 123.27 seconds: 525.41 ops/second - /unlink 9206 (time 1747828997.07 total 10.00 last 920.51) - /unlink 18548 (time 1747829007.07 total 20.00 last 934.15) - /unlink 27848 (time 1747829017.07 total 30.00 last 929.95) - /unlink 30000 (time 1747829019.36 total 32.28 last 942.97) - /unlink 39286 (time 1747829029.36 total 42.28 last 928.55) - /unlink 48617 (time 1747829039.36 total 52.28 last 933.09) - /unlink 57678 (time 1747829049.36 total 62.28 last 906.09) - /unlink 60000 (time 1747829051.76 total 64.68 last 967.47) total: 64768 /unlink in 69.57 seconds: 930.93 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3854 (time 1747829098.86 total 10.00 last 385.27) - open/close 8525 (time 1747829108.86 total 20.00 last 467.07) - open/close 10000 (time 1747829111.67 total 22.81 last 525.31) - open/close 14699 (time 1747829121.67 total 32.81 last 469.86) - open/close 19560 (time 1747829131.67 total 42.81 last 486.06) - open/close 24370 (time 1747829141.67 total 52.81 last 480.99) - open/close 27685 (time 1747829151.67 total 62.81 last 331.47) - open/close 30000 (time 1747829158.14 total 69.29 last 357.77) - open/close 33942 (time 1747829168.15 total 79.29 last 394.16) - open/close 38214 (time 1747829178.15 total 89.29 last 427.13) - open/close 40000 (time 1747829182.84 total 93.98 last 380.66) - open/close 44047 (time 1747829192.84 total 103.98 last 404.64) - open/close 47335 (time 1747829202.84 total 113.98 last 328.79) - open/close 50000 (time 1747829210.20 total 121.34 last 362.14) - open/close 53285 (time 1747829220.20 total 131.34 last 328.43) - open/close 55770 (time 1747829230.20 total 141.35 last 248.42) - open/close 58272 (time 1747829240.21 total 151.35 last 250.16) - open/close 60000 (time 1747829245.81 total 156.96 last 308.14) - open/close 62421 (time 1747829255.82 total 166.96 last 242.01) total: 64768 open/close in 175.34 seconds: 369.38 ops/second - /unlink 4924 (time 1747829275.41 total 10.00 last 492.30) - /unlink 10000 (time 1747829284.88 total 19.48 last 535.80) - /unlink 15344 (time 1747829294.88 total 29.48 last 534.38) - /unlink 20000 (time 1747829302.96 total 37.55 last 576.69) - /unlink 25904 (time 1747829312.96 total 47.55 last 590.37) - /unlink 30000 (time 1747829318.99 total 53.59 last 678.52) - /unlink 35843 (time 1747829328.99 total 63.59 last 584.29) - /unlink 40000 (time 1747829334.20 total 68.80 last 797.99) - /unlink 45105 (time 1747829344.20 total 78.80 last 510.49) - /unlink 50000 (time 1747829353.44 total 88.03 last 530.16) - /unlink 55581 (time 1747829363.44 total 98.03 last 558.01) - /unlink 60000 (time 1747829370.72 total 105.32 last 606.48) total: 64768 /unlink in 113.88 seconds: 568.75 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2545 (time 1747829409.14 total 10.00 last 254.46) - open/close 5320 (time 1747829419.15 total 20.01 last 277.39) - open/close 8005 (time 1747829429.15 total 30.01 last 268.43) - open/close 10000 (time 1747829436.90 total 37.76 last 257.40) - open/close 12603 (time 1747829446.90 total 47.76 last 260.29) - open/close 15193 (time 1747829456.90 total 57.76 last 258.98) - open/close 17814 (time 1747829466.90 total 67.76 last 262.06) - open/close 20000 (time 1747829474.73 total 75.59 last 279.34) - open/close 23046 (time 1747829484.73 total 85.59 last 304.54) - open/close 25624 (time 1747829494.73 total 95.59 last 257.72) - open/close 28214 (time 1747829504.73 total 105.59 last 258.96) - open/close 30000 (time 1747829511.09 total 111.95 last 280.83) - open/close 33096 (time 1747829521.10 total 121.96 last 309.49) - open/close 35758 (time 1747829531.10 total 131.96 last 266.11) - open/close 38241 (time 1747829541.10 total 141.96 last 248.25) - open/close 40000 (time 1747829547.92 total 148.78 last 257.96) - open/close 44184 (time 1747829557.92 total 158.78 last 418.39) - open/close 47002 (time 1747829567.92 total 168.78 last 281.73) - open/close 50000 (time 1747829575.62 total 176.48 last 389.44) - open/close 53731 (time 1747829585.62 total 186.48 last 373.05) - open/close 56270 (time 1747829595.62 total 196.48 last 253.88) - open/close 58689 (time 1747829605.63 total 206.49 last 241.89) - open/close 60000 (time 1747829611.00 total 211.86 last 244.07) - open/close 62459 (time 1747829621.00 total 221.86 last 245.81) total: 64768 open/close in 231.43 seconds: 279.86 ops/second - /unlink 4735 (time 1747829641.86 total 10.00 last 473.49) - /unlink 9569 (time 1747829651.86 total 20.00 last 483.33) - /unlink 14891 (time 1747829661.86 total 30.00 last 532.17) - /unlink 20000 (time 1747829671.68 total 39.83 last 520.02) - /unlink 25110 (time 1747829681.68 total 49.83 last 510.90) - /unlink 30000 (time 1747829691.52 total 59.66 last 497.34) - /unlink 35998 (time 1747829701.52 total 69.66 last 599.67) - /unlink 40000 (time 1747829709.33 total 77.47 last 512.62) - /unlink 44613 (time 1747829719.33 total 87.47 last 461.23) - /unlink 49700 (time 1747829729.33 total 97.47 last 508.64) - /unlink 54495 (time 1747829739.33 total 107.47 last 479.46) - /unlink 59386 (time 1747829749.33 total 117.47 last 489.04) - /unlink 64256 (time 1747829759.33 total 127.48 last 486.98) total: 64768 /unlink in 128.58 seconds: 503.71 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server LNET ready to unload unloading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local oleg140-server: LNET ready to unload modules unloaded. pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2