== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:08:32 (1744535312) Stopping clients: oleg402-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg402-client.virtnet /mnt/lustre2 (opts:-f) oleg402-server: oleg402-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing load_modules_local oleg402-server: Loading modules from /home/green/git/lustre-release/lustre oleg402-server: detected 4 online CPUs by sysfs oleg402-server: MODOPTS_LIBCFS= oleg402-server: Force libcfs to create 2 CPU partitions oleg402-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg402-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg402-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg402-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3246 (time 1744535348.89 total 10.00 last 324.59) - open/close 5897 (time 1744535358.89 total 20.00 last 265.09) - open/close 9281 (time 1744535368.89 total 30.00 last 338.39) - open/close 14078 (time 1744535378.89 total 40.00 last 479.66) - open/close 18424 (time 1744535388.89 total 50.00 last 434.60) - open/close 20000 (time 1744535392.88 total 53.98 last 395.85) - open/close 23683 (time 1744535402.88 total 63.98 last 368.27) - open/close 28546 (time 1744535412.88 total 73.98 last 486.28) - open/close 30000 (time 1744535415.20 total 76.31 last 625.88) - open/close 35129 (time 1744535425.20 total 86.31 last 512.82) - open/close 40000 (time 1744535434.05 total 95.16 last 550.17) - open/close 45554 (time 1744535444.06 total 105.17 last 555.25) - open/close 50000 (time 1744535453.25 total 114.36 last 483.58) - open/close 55293 (time 1744535463.25 total 124.36 last 529.18) - open/close 59429 (time 1744535473.25 total 134.36 last 413.57) - open/close 63689 (time 1744535483.26 total 144.36 last 425.97) total: 64768 open/close in 146.97 seconds: 440.70 ops/second - /unlink 7355 (time 1744535496.70 total 10.00 last 735.47) - /unlink 10000 (time 1744535501.32 total 14.62 last 572.96) - /unlink 16620 (time 1744535511.32 total 24.62 last 662.00) - /unlink 20000 (time 1744535518.35 total 31.65 last 480.54) - /unlink 24898 (time 1744535528.35 total 41.65 last 489.71) - /unlink 29583 (time 1744535538.35 total 51.65 last 468.49) - /unlink 34757 (time 1744535548.35 total 61.65 last 517.36) - /unlink 40000 (time 1744535557.04 total 70.34 last 603.47) - /unlink 47509 (time 1744535567.04 total 80.34 last 750.83) - /unlink 50000 (time 1744535570.56 total 83.86 last 707.86) - /unlink 56705 (time 1744535580.56 total 93.86 last 670.38) - /unlink 60000 (time 1744535584.74 total 98.04 last 789.76) total: 64768 /unlink in 102.74 seconds: 630.40 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4717 (time 1744535622.39 total 10.00 last 471.62) - open/close 9780 (time 1744535632.39 total 20.00 last 506.28) - open/close 14921 (time 1744535642.39 total 30.00 last 514.07) - open/close 20000 (time 1744535652.29 total 39.90 last 513.40) - open/close 25170 (time 1744535662.29 total 49.90 last 516.94) - open/close 30000 (time 1744535671.32 total 58.93 last 534.95) - open/close 35084 (time 1744535681.32 total 68.93 last 508.36) - open/close 40000 (time 1744535690.90 total 78.51 last 512.92) - open/close 45314 (time 1744535700.90 total 88.51 last 531.33) - open/close 50000 (time 1744535709.83 total 97.44 last 525.06) - open/close 54186 (time 1744535719.83 total 107.44 last 418.51) - open/close 58693 (time 1744535729.83 total 117.44 last 450.65) - open/close 60000 (time 1744535732.51 total 120.12 last 488.34) total: 64768 open/close in 129.09 seconds: 501.72 ops/second - /unlink 9041 (time 1744535752.33 total 10.00 last 904.01) - /unlink 18077 (time 1744535762.33 total 20.00 last 903.54) - /unlink 27831 (time 1744535772.33 total 30.00 last 975.39) - /unlink 30000 (time 1744535774.62 total 32.29 last 946.96) - /unlink 39356 (time 1744535784.62 total 42.29 last 935.56) - /unlink 48841 (time 1744535794.62 total 52.29 last 948.49) - /unlink 58030 (time 1744535804.63 total 62.29 last 918.87) - /unlink 60000 (time 1744535806.77 total 64.43 last 920.45) total: 64768 /unlink in 69.07 seconds: 937.65 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 6620 (time 1744535852.57 total 10.00 last 661.93) - open/close 10000 (time 1744535857.63 total 15.06 last 668.30) - open/close 16496 (time 1744535867.63 total 25.06 last 649.54) - open/close 20000 (time 1744535873.10 total 30.53 last 640.36) - open/close 26433 (time 1744535883.10 total 40.53 last 643.22) - open/close 30000 (time 1744535888.64 total 46.07 last 644.70) - open/close 35753 (time 1744535898.64 total 56.07 last 575.27) - open/close 40000 (time 1744535906.96 total 64.39 last 510.37) - open/close 44868 (time 1744535916.96 total 74.39 last 486.72) - open/close 49524 (time 1744535926.96 total 84.39 last 465.58) - open/close 54320 (time 1744535936.96 total 94.39 last 479.46) - open/close 59151 (time 1744535946.97 total 104.39 last 483.02) - open/close 64731 (time 1744535956.97 total 114.40 last 557.92) total: 64768 open/close in 114.45 seconds: 565.90 ops/second - /unlink 10000 (time 1744535967.61 total 9.82 last 1018.46) - /unlink 20000 (time 1744535977.49 total 19.70 last 1012.01) - /unlink 30000 (time 1744535986.98 total 29.19 last 1053.87) - /unlink 40000 (time 1744535996.92 total 39.13 last 1006.26) - /unlink 50000 (time 1744536006.53 total 48.74 last 1040.62) - /unlink 60000 (time 1744536015.85 total 58.06 last 1072.85) total: 64768 /unlink in 62.38 seconds: 1038.23 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5474 (time 1744536067.92 total 10.00 last 547.32) - open/close 10000 (time 1744536076.39 total 18.48 last 533.83) - open/close 15055 (time 1744536086.40 total 28.48 last 505.45) - open/close 20000 (time 1744536096.24 total 38.32 last 502.33) - open/close 25115 (time 1744536106.24 total 48.33 last 511.47) - open/close 29891 (time 1744536116.24 total 58.33 last 477.60) - open/close 35026 (time 1744536126.24 total 68.33 last 513.44) - open/close 39843 (time 1744536136.24 total 78.33 last 481.66) - open/close 44637 (time 1744536146.24 total 88.33 last 479.29) - open/close 48724 (time 1744536156.25 total 98.33 last 408.58) - open/close 50000 (time 1744536159.42 total 101.50 last 402.74) - open/close 54038 (time 1744536169.42 total 111.50 last 403.72) - open/close 58543 (time 1744536179.42 total 121.50 last 450.49) - open/close 60000 (time 1744536182.28 total 124.37 last 508.70) - open/close 63538 (time 1744536192.28 total 134.37 last 353.73) total: 64768 open/close in 137.82 seconds: 469.95 ops/second - /unlink 6043 (time 1744536206.70 total 10.00 last 604.23) - /unlink 10000 (time 1744536211.83 total 15.13 last 771.27) - /unlink 18236 (time 1744536221.83 total 25.13 last 823.40) - /unlink 20000 (time 1744536223.84 total 27.14 last 878.90) - /unlink 28371 (time 1744536233.84 total 37.14 last 837.03) - /unlink 30000 (time 1744536236.05 total 39.36 last 735.03) - /unlink 37483 (time 1744536246.05 total 49.36 last 748.29) - /unlink 40000 (time 1744536248.87 total 52.18 last 892.17) - /unlink 49367 (time 1744536258.87 total 62.18 last 936.68) - /unlink 59012 (time 1744536268.87 total 72.18 last 964.44) total: 64768 /unlink in 78.18 seconds: 828.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5609 (time 1744536308.86 total 10.00 last 560.84) - open/close 10000 (time 1744536316.83 total 17.97 last 551.03) - open/close 15338 (time 1744536326.83 total 27.97 last 533.71) - open/close 20000 (time 1744536335.07 total 36.21 last 565.69) - open/close 25535 (time 1744536345.07 total 46.21 last 553.40) - open/close 30000 (time 1744536353.36 total 54.50 last 538.68) - open/close 35559 (time 1744536363.36 total 64.50 last 555.82) - open/close 40000 (time 1744536371.55 total 72.70 last 542.21) - open/close 45450 (time 1744536381.55 total 82.70 last 544.95) - open/close 50000 (time 1744536390.12 total 91.26 last 531.02) - open/close 54926 (time 1744536400.12 total 101.27 last 492.56) - open/close 60000 (time 1744536409.63 total 110.77 last 533.95) total: 64768 open/close in 119.61 seconds: 541.48 ops/second - /unlink 9176 (time 1744536429.27 total 10.00 last 917.55) - /unlink 17508 (time 1744536439.27 total 20.00 last 833.20) - /unlink 20000 (time 1744536442.07 total 22.80 last 889.15) - /unlink 29081 (time 1744536452.07 total 32.80 last 908.00) - /unlink 37956 (time 1744536462.07 total 42.81 last 887.38) - /unlink 40000 (time 1744536465.22 total 45.96 last 649.02) - /unlink 46816 (time 1744536475.22 total 55.96 last 681.54) - /unlink 50000 (time 1744536478.75 total 59.48 last 903.33) - /unlink 59317 (time 1744536488.75 total 69.48 last 931.69) total: 64768 /unlink in 75.11 seconds: 862.31 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server unloading modules via unload_modules_local on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2