== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:45:10 (1744170310) Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f) oleg333-server: oleg333-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing load_modules_local oleg333-server: Loading modules from /home/green/git/lustre-release/lustre oleg333-server: detected 4 online CPUs by sysfs oleg333-server: MODOPTS_LIBCFS= oleg333-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg333-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg333-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32587 331 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4348 (time 1744170345.37 total 10.00 last 434.76) - open/close 8979 (time 1744170355.37 total 20.00 last 463.08) - open/close 13783 (time 1744170365.37 total 30.00 last 480.28) - open/close 17889 (time 1744170375.37 total 40.00 last 410.60) - open/close 20000 (time 1744170380.75 total 45.38 last 392.78) - open/close 24197 (time 1744170390.75 total 55.38 last 419.64) - open/close 28428 (time 1744170400.75 total 65.38 last 423.09) - open/close 30000 (time 1744170404.41 total 69.04 last 429.35) - open/close 34471 (time 1744170414.41 total 79.04 last 447.06) - open/close 39113 (time 1744170424.41 total 89.04 last 464.16) - open/close 43626 (time 1744170434.41 total 99.04 last 451.27) - open/close 48324 (time 1744170444.41 total 109.04 last 469.78) - open/close 50000 (time 1744170447.38 total 112.01 last 565.43) - open/close 54870 (time 1744170457.38 total 122.01 last 486.94) - open/close 60000 (time 1744170467.20 total 131.83 last 522.51) total: 64768 open/close in 141.33 seconds: 458.27 ops/second - /unlink 7262 (time 1744170487.59 total 10.00 last 726.18) - /unlink 10000 (time 1744170491.55 total 13.96 last 691.24) - /unlink 17412 (time 1744170501.55 total 23.96 last 741.15) - /unlink 20000 (time 1744170505.15 total 27.56 last 719.64) - /unlink 27109 (time 1744170515.15 total 37.56 last 710.89) - /unlink 30000 (time 1744170519.29 total 41.70 last 698.24) - /unlink 37505 (time 1744170529.29 total 51.70 last 750.43) - /unlink 40000 (time 1744170532.99 total 55.39 last 675.30) - /unlink 47854 (time 1744170542.99 total 65.39 last 785.40) - /unlink 50000 (time 1744170545.43 total 67.84 last 877.02) - /unlink 58822 (time 1744170555.43 total 77.84 last 882.13) total: 64768 /unlink in 84.26 seconds: 768.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5222 (time 1744170588.49 total 10.00 last 522.12) - open/close 10000 (time 1744170598.06 total 19.57 last 499.37) - open/close 14928 (time 1744170608.06 total 29.57 last 492.73) - open/close 20000 (time 1744170617.85 total 39.36 last 518.11) - open/close 25289 (time 1744170627.85 total 49.36 last 528.86) - open/close 30000 (time 1744170636.76 total 58.27 last 528.68) - open/close 35297 (time 1744170646.76 total 68.27 last 529.64) - open/close 40000 (time 1744170655.65 total 77.16 last 529.14) - open/close 45371 (time 1744170665.65 total 87.16 last 537.06) - open/close 50000 (time 1744170674.68 total 96.19 last 512.53) - open/close 55242 (time 1744170684.69 total 106.20 last 524.12) - open/close 60000 (time 1744170693.72 total 115.23 last 526.71) total: 64768 open/close in 124.18 seconds: 521.56 ops/second - /unlink 8926 (time 1744170713.51 total 10.00 last 892.53) - /unlink 18119 (time 1744170723.51 total 20.00 last 919.24) - /unlink 20000 (time 1744170725.53 total 22.02 last 930.04) - /unlink 28635 (time 1744170735.53 total 32.02 last 863.49) - /unlink 37530 (time 1744170745.53 total 42.02 last 889.47) - /unlink 40000 (time 1744170748.48 total 44.97 last 838.51) - /unlink 49142 (time 1744170758.48 total 54.97 last 914.16) - /unlink 58633 (time 1744170768.48 total 64.97 last 949.08) total: 64768 /unlink in 71.51 seconds: 905.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5662 (time 1744170814.46 total 10.00 last 566.11) - open/close 10000 (time 1744170822.36 total 17.90 last 549.40) - open/close 15655 (time 1744170832.36 total 27.90 last 565.49) - open/close 20000 (time 1744170840.17 total 35.71 last 556.22) - open/close 25504 (time 1744170850.17 total 45.71 last 550.31) - open/close 30000 (time 1744170858.25 total 53.78 last 556.98) - open/close 35536 (time 1744170868.25 total 63.78 last 553.57) - open/close 40000 (time 1744170876.21 total 71.74 last 560.70) - open/close 45348 (time 1744170886.21 total 81.75 last 534.74) - open/close 50000 (time 1744170895.24 total 90.78 last 514.90) - open/close 55437 (time 1744170905.24 total 100.78 last 543.67) - open/close 60000 (time 1744170913.75 total 109.28 last 536.74) total: 64768 open/close in 118.27 seconds: 547.62 ops/second - /unlink 9475 (time 1744170933.57 total 10.00 last 947.47) - /unlink 18861 (time 1744170943.57 total 20.00 last 938.55) - /unlink 28337 (time 1744170953.57 total 30.00 last 947.50) - /unlink 37338 (time 1744170963.57 total 40.00 last 900.06) - /unlink 40000 (time 1744170966.75 total 43.18 last 838.06) - /unlink 48959 (time 1744170976.75 total 53.18 last 895.90) - /unlink 58112 (time 1744170986.75 total 63.18 last 915.29) total: 64768 /unlink in 70.41 seconds: 919.88 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5538 (time 1744171034.81 total 10.00 last 553.72) - open/close 10000 (time 1744171042.85 total 18.05 last 554.69) - open/close 15457 (time 1744171052.85 total 28.05 last 545.67) - open/close 20000 (time 1744171061.19 total 36.38 last 544.93) - open/close 25151 (time 1744171071.19 total 46.38 last 515.04) - open/close 29579 (time 1744171081.20 total 56.39 last 442.66) - open/close 33167 (time 1744171091.20 total 66.39 last 358.79) - open/close 36760 (time 1744171101.20 total 76.39 last 359.26) - open/close 39991 (time 1744171111.20 total 86.39 last 323.06) - open/close 44282 (time 1744171121.20 total 96.39 last 429.05) - open/close 49285 (time 1744171131.20 total 106.39 last 500.22) - open/close 54291 (time 1744171141.20 total 116.39 last 500.59) - open/close 58304 (time 1744171151.20 total 126.39 last 401.23) - open/close 60000 (time 1744171157.27 total 132.46 last 279.63) - open/close 64431 (time 1744171167.27 total 142.46 last 443.01) total: 64768 open/close in 143.30 seconds: 451.98 ops/second - /unlink 9207 (time 1744171178.95 total 10.00 last 920.66) - /unlink 18012 (time 1744171188.96 total 20.00 last 880.39) - /unlink 20000 (time 1744171191.03 total 22.08 last 957.62) - /unlink 28432 (time 1744171201.03 total 32.08 last 843.10) - /unlink 35610 (time 1744171211.03 total 42.08 last 717.66) - /unlink 40000 (time 1744171217.61 total 48.65 last 668.05) - /unlink 48893 (time 1744171227.61 total 58.65 last 889.23) - /unlink 56510 (time 1744171237.61 total 68.65 last 761.65) - /unlink 60000 (time 1744171241.97 total 73.01 last 800.74) total: 64768 /unlink in 78.16 seconds: 828.62 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3945 (time 1744171280.78 total 10.00 last 394.46) - open/close 8242 (time 1744171290.78 total 20.00 last 429.60) - open/close 10000 (time 1744171294.65 total 23.88 last 453.56) - open/close 14515 (time 1744171304.66 total 33.88 last 451.39) - open/close 18325 (time 1744171314.66 total 43.88 last 380.88) - open/close 20000 (time 1744171319.87 total 49.10 last 321.32) - open/close 24308 (time 1744171329.87 total 59.10 last 430.79) - open/close 29151 (time 1744171339.87 total 69.10 last 484.28) - open/close 33567 (time 1744171349.87 total 79.10 last 441.60) - open/close 37210 (time 1744171359.87 total 89.10 last 364.25) - open/close 40000 (time 1744171367.25 total 96.47 last 378.51) - open/close 43528 (time 1744171377.25 total 106.47 last 352.77) - open/close 46980 (time 1744171387.25 total 116.47 last 345.20) - open/close 50000 (time 1744171396.45 total 125.68 last 328.11) - open/close 54134 (time 1744171406.45 total 135.68 last 413.38) - open/close 57321 (time 1744171416.45 total 145.68 last 318.62) - open/close 59924 (time 1744171426.45 total 155.68 last 260.27) - open/close 62640 (time 1744171436.46 total 165.68 last 271.56) total: 64768 open/close in 169.52 seconds: 382.07 ops/second - /unlink 6024 (time 1744171451.38 total 10.00 last 602.40) - /unlink 10000 (time 1744171458.82 total 17.44 last 534.44) - /unlink 16033 (time 1744171468.82 total 27.44 last 603.28) - /unlink 20000 (time 1744171476.96 total 35.59 last 486.90) - /unlink 25169 (time 1744171486.96 total 45.59 last 516.85) - /unlink 30000 (time 1744171495.74 total 54.36 last 550.56) - /unlink 35178 (time 1744171505.74 total 64.36 last 517.77) - /unlink 40000 (time 1744171514.68 total 73.31 last 539.10) - /unlink 45783 (time 1744171524.68 total 83.31 last 578.29) - /unlink 50000 (time 1744171532.88 total 91.51 last 514.46) - /unlink 55081 (time 1744171542.88 total 101.51 last 508.04) - /unlink 60000 (time 1744171552.01 total 110.63 last 538.91) total: 64768 /unlink in 117.62 seconds: 550.64 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server unloading modules via unload_modules_local on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2