== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 03:34:15 (1749540855) Stopping clients: oleg424-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg424-client.virtnet /mnt/lustre2 (opts:-f) oleg424-server: oleg424-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions oleg424-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg424-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg424-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3703 (time 1749540897.95 total 10.00 last 370.25) - open/close 7541 (time 1749540907.95 total 20.00 last 383.74) - open/close 10000 (time 1749540913.16 total 25.21 last 471.82) - open/close 14284 (time 1749540923.16 total 35.22 last 428.34) - open/close 19356 (time 1749540933.16 total 45.22 last 507.17) - open/close 24516 (time 1749540943.16 total 55.22 last 515.98) - open/close 29485 (time 1749540953.16 total 65.22 last 496.86) - open/close 33063 (time 1749540963.17 total 75.22 last 357.74) - open/close 36983 (time 1749540973.17 total 85.22 last 392.00) - open/close 40000 (time 1749540981.76 total 93.81 last 351.23) - open/close 43841 (time 1749540991.76 total 103.81 last 384.06) - open/close 47450 (time 1749541001.76 total 113.81 last 360.90) - open/close 50000 (time 1749541008.72 total 120.77 last 366.43) - open/close 53134 (time 1749541018.72 total 130.77 last 313.30) - open/close 56899 (time 1749541028.72 total 140.77 last 376.49) - open/close 60000 (time 1749541038.00 total 150.06 last 334.01) - open/close 63593 (time 1749541048.01 total 160.06 last 359.23) total: 64768 open/close in 163.20 seconds: 396.85 ops/second - /unlink 6459 (time 1749541062.06 total 10.00 last 645.77) - /unlink 10000 (time 1749541068.20 total 16.13 last 577.41) - /unlink 15752 (time 1749541078.20 total 26.14 last 575.18) - /unlink 20000 (time 1749541085.65 total 33.59 last 569.78) - /unlink 26103 (time 1749541095.65 total 43.59 last 610.24) - /unlink 30000 (time 1749541101.85 total 49.79 last 628.53) - /unlink 36500 (time 1749541111.85 total 59.79 last 650.00) - /unlink 40000 (time 1749541117.50 total 65.44 last 620.05) - /unlink 46416 (time 1749541127.50 total 75.44 last 641.55) - /unlink 50000 (time 1749541133.20 total 81.13 last 629.03) - /unlink 56613 (time 1749541143.20 total 91.14 last 661.24) - /unlink 60000 (time 1749541149.24 total 97.18 last 560.60) total: 64768 /unlink in 103.06 seconds: 628.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3107 (time 1749541186.21 total 10.00 last 310.66) - open/close 5812 (time 1749541196.22 total 20.01 last 270.38) - open/close 8487 (time 1749541206.22 total 30.01 last 267.49) - open/close 10000 (time 1749541211.64 total 35.43 last 279.16) - open/close 12901 (time 1749541221.64 total 45.43 last 289.98) - open/close 16641 (time 1749541231.64 total 55.43 last 373.97) - open/close 20000 (time 1749541240.97 total 64.76 last 359.88) - open/close 23405 (time 1749541250.98 total 74.77 last 340.44) - open/close 26645 (time 1749541260.98 total 84.77 last 323.99) - open/close 29961 (time 1749541270.98 total 94.77 last 331.51) - open/close 33087 (time 1749541280.98 total 104.77 last 312.49) - open/close 36108 (time 1749541290.98 total 114.77 last 302.08) - open/close 39594 (time 1749541300.99 total 124.78 last 348.50) - open/close 42628 (time 1749541310.99 total 134.78 last 303.34) - open/close 45728 (time 1749541320.99 total 144.78 last 309.93) - open/close 48709 (time 1749541330.99 total 154.78 last 298.09) - open/close 50000 (time 1749541335.75 total 159.54 last 271.12) - open/close 53175 (time 1749541345.75 total 169.55 last 317.42) - open/close 56463 (time 1749541355.76 total 179.55 last 328.71) - open/close 59394 (time 1749541365.76 total 189.55 last 293.09) - open/close 62805 (time 1749541375.76 total 199.55 last 341.09) total: 64768 open/close in 204.75 seconds: 316.32 ops/second - /unlink 6829 (time 1749541391.85 total 10.00 last 682.86) - /unlink 10000 (time 1749541396.99 total 15.15 last 616.37) - /unlink 16046 (time 1749541406.99 total 25.15 last 604.57) - /unlink 20000 (time 1749541413.34 total 31.49 last 623.48) - /unlink 26351 (time 1749541423.34 total 41.49 last 635.08) - /unlink 30000 (time 1749541429.08 total 47.23 last 635.53) - /unlink 36547 (time 1749541439.08 total 57.23 last 654.65) - /unlink 40000 (time 1749541444.73 total 62.89 last 610.57) - /unlink 46591 (time 1749541454.73 total 72.89 last 659.06) - /unlink 50000 (time 1749541459.79 total 77.95 last 673.72) - /unlink 58414 (time 1749541469.80 total 87.95 last 841.37) - /unlink 60000 (time 1749541471.80 total 89.95 last 790.47) total: 64768 /unlink in 95.06 seconds: 681.37 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5064 (time 1749541506.87 total 10.00 last 506.34) - open/close 10000 (time 1749541515.96 total 19.09 last 542.97) - open/close 15526 (time 1749541525.96 total 29.09 last 552.60) - open/close 20000 (time 1749541534.28 total 37.41 last 538.08) - open/close 24908 (time 1749541544.28 total 47.41 last 490.77) - open/close 29980 (time 1749541554.28 total 57.41 last 507.14) - open/close 35376 (time 1749541564.28 total 67.41 last 539.59) - open/close 40000 (time 1749541573.66 total 76.80 last 492.59) - open/close 44967 (time 1749541583.67 total 86.80 last 496.62) - open/close 50000 (time 1749541593.49 total 96.62 last 512.34) - open/close 55148 (time 1749541603.49 total 106.62 last 514.72) - open/close 60000 (time 1749541612.89 total 116.02 last 516.46) total: 64768 open/close in 125.33 seconds: 516.78 ops/second - /unlink 8941 (time 1749541633.08 total 10.00 last 894.07) - /unlink 18147 (time 1749541643.08 total 20.00 last 920.52) - /unlink 20000 (time 1749541645.31 total 22.23 last 831.32) - /unlink 28802 (time 1749541655.31 total 32.23 last 880.10) - /unlink 37576 (time 1749541665.31 total 42.23 last 877.37) - /unlink 40000 (time 1749541667.90 total 44.82 last 936.65) - /unlink 48922 (time 1749541677.90 total 54.82 last 892.19) - /unlink 57802 (time 1749541687.90 total 64.82 last 887.99) - /unlink 60000 (time 1749541690.37 total 67.29 last 890.05) total: 64768 /unlink in 72.51 seconds: 893.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5424 (time 1749541732.00 total 10.00 last 542.33) - open/close 10000 (time 1749541740.73 total 18.74 last 523.90) - open/close 15244 (time 1749541750.73 total 28.74 last 524.39) - open/close 20000 (time 1749541760.45 total 38.45 last 489.50) - open/close 23934 (time 1749541770.45 total 48.45 last 393.39) - open/close 28661 (time 1749541780.45 total 58.45 last 472.62) - open/close 30000 (time 1749541783.62 total 61.63 last 422.00) - open/close 34714 (time 1749541793.63 total 71.63 last 471.37) - open/close 39522 (time 1749541803.63 total 81.63 last 480.75) - open/close 44254 (time 1749541813.63 total 91.63 last 473.18) - open/close 49175 (time 1749541823.63 total 101.63 last 492.10) - open/close 54398 (time 1749541833.63 total 111.63 last 522.22) - open/close 59695 (time 1749541843.68 total 121.68 last 527.10) total: 64768 open/close in 131.26 seconds: 493.43 ops/second - /unlink 9032 (time 1749541864.30 total 10.00 last 903.13) - /unlink 17737 (time 1749541874.30 total 20.00 last 870.42) - /unlink 20000 (time 1749541877.26 total 22.97 last 762.67) - /unlink 29051 (time 1749541887.27 total 32.97 last 905.02) - /unlink 38320 (time 1749541897.27 total 42.97 last 926.90) - /unlink 47081 (time 1749541907.27 total 52.97 last 875.97) - /unlink 50000 (time 1749541911.09 total 56.79 last 763.42) - /unlink 57375 (time 1749541921.09 total 66.80 last 737.40) - /unlink 60000 (time 1749541925.10 total 70.81 last 654.42) total: 64768 /unlink in 76.02 seconds: 852.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5685 (time 1749541962.26 total 10.00 last 568.41) - open/close 10000 (time 1749541969.99 total 17.73 last 558.46) - open/close 14741 (time 1749541979.99 total 27.73 last 474.04) - open/close 19016 (time 1749541989.99 total 37.73 last 427.45) - open/close 24418 (time 1749541999.99 total 47.73 last 540.13) - open/close 29097 (time 1749542009.99 total 57.73 last 467.88) - open/close 30000 (time 1749542012.84 total 60.58 last 317.23) - open/close 33657 (time 1749542022.84 total 70.58 last 365.68) - open/close 37141 (time 1749542032.84 total 80.58 last 348.36) - open/close 40000 (time 1749542039.20 total 86.94 last 449.32) - open/close 43598 (time 1749542049.20 total 96.94 last 359.78) - open/close 47696 (time 1749542059.20 total 106.94 last 409.75) - open/close 50000 (time 1749542065.70 total 113.44 last 354.76) - open/close 53608 (time 1749542075.70 total 123.44 last 360.79) - open/close 57309 (time 1749542085.70 total 133.44 last 370.08) - open/close 60000 (time 1749542092.61 total 140.35 last 389.54) - open/close 64177 (time 1749542102.61 total 150.35 last 417.68) total: 64768 open/close in 151.75 seconds: 426.80 ops/second - /unlink 7848 (time 1749542114.87 total 10.00 last 784.77) - /unlink 10000 (time 1749542117.40 total 12.53 last 849.42) - /unlink 18760 (time 1749542127.41 total 22.53 last 875.93) - /unlink 27052 (time 1749542137.41 total 32.54 last 829.13) - /unlink 30000 (time 1749542140.68 total 35.81 last 901.20) - /unlink 38677 (time 1749542150.68 total 45.81 last 867.61) - /unlink 47612 (time 1749542160.68 total 55.81 last 893.48) - /unlink 50000 (time 1749542163.34 total 58.47 last 897.50) - /unlink 59427 (time 1749542173.34 total 68.47 last 942.64) total: 64768 /unlink in 74.14 seconds: 873.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server unloading modules via unload_modules_local on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2