== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:54:58 (1715460898) Stopping clients: oleg304-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg304-client.virtnet /mnt/lustre2 (opts:-f) oleg304-server: oleg304-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing load_modules_local oleg304-server: Loading modules from /home/green/git/lustre-release/lustre oleg304-server: detected 4 online CPUs by sysfs oleg304-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg304-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg304-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg304-client.virtnet: -o user_xattr,flock oleg304-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 3248 (time 1715460938.45 total 10.00 last 324.77) - open/close 6215 (time 1715460948.45 total 20.00 last 296.59) - open/close 10000 (time 1715460957.62 total 29.17 last 412.83) - open/close 14338 (time 1715460967.62 total 39.18 last 433.69) - open/close 17804 (time 1715460977.62 total 49.18 last 346.57) - open/close 20000 (time 1715460981.94 total 53.49 last 508.81) - open/close 25399 (time 1715460991.94 total 63.49 last 539.88) - open/close 30000 (time 1715461000.49 total 72.04 last 538.35) - open/close 35324 (time 1715461010.49 total 82.04 last 532.31) - open/close 40000 (time 1715461019.18 total 90.73 last 538.13) - open/close 45269 (time 1715461029.18 total 100.73 last 526.83) - open/close 50000 (time 1715461038.30 total 109.85 last 518.70) - open/close 55156 (time 1715461048.30 total 119.85 last 515.57) - open/close 60000 (time 1715461057.51 total 129.06 last 525.98) total: 64768 open/close in 138.16 seconds: 468.78 ops/second - /unlink 9359 (time 1715461077.40 total 10.00 last 935.75) - /unlink 17198 (time 1715461087.41 total 20.00 last 783.83) - /unlink 20000 (time 1715461090.41 total 23.00 last 934.17) - /unlink 28894 (time 1715461100.41 total 33.00 last 889.32) - /unlink 37775 (time 1715461110.41 total 43.00 last 888.09) - /unlink 40000 (time 1715461112.78 total 45.38 last 937.07) - /unlink 49546 (time 1715461122.78 total 55.38 last 954.57) - /unlink 59208 (time 1715461132.78 total 65.38 last 966.16) total: 64768 /unlink in 70.97 seconds: 912.64 ops/second - open/close 4821 (time 1715461149.29 total 10.00 last 482.06) - open/close 10000 (time 1715461158.96 total 19.67 last 535.48) - open/close 15382 (time 1715461168.97 total 29.67 last 538.12) - open/close 19212 (time 1715461178.97 total 39.68 last 382.87) - open/close 20000 (time 1715461182.23 total 42.94 last 241.67) - open/close 23197 (time 1715461192.23 total 52.94 last 319.60) - open/close 25763 (time 1715461202.24 total 62.94 last 256.56) - open/close 28970 (time 1715461212.24 total 72.94 last 320.67) - open/close 30000 (time 1715461215.75 total 76.46 last 292.89) - open/close 33220 (time 1715461225.75 total 86.46 last 321.94) - open/close 36861 (time 1715461235.75 total 96.46 last 364.08) - open/close 39890 (time 1715461245.76 total 106.46 last 302.86) - open/close 43008 (time 1715461255.76 total 116.47 last 311.73) - open/close 45720 (time 1715461265.84 total 126.55 last 269.05) - open/close 48583 (time 1715461275.84 total 136.55 last 286.22) - open/close 50000 (time 1715461280.98 total 141.68 last 275.96) - open/close 53876 (time 1715461290.98 total 151.69 last 387.54) - open/close 57233 (time 1715461300.98 total 161.69 last 335.69) - open/close 59959 (time 1715461310.99 total 171.70 last 272.20) - open/close 63193 (time 1715461320.99 total 181.70 last 323.38) total: 64768 open/close in 186.75 seconds: 346.82 ops/second - /unlink 6040 (time 1715461336.86 total 10.00 last 603.90) - /unlink 10000 (time 1715461343.12 total 16.26 last 632.42) - /unlink 16223 (time 1715461353.12 total 26.26 last 622.28) - /unlink 20000 (time 1715461358.16 total 31.31 last 749.07) - /unlink 24996 (time 1715461368.16 total 41.31 last 499.52) - /unlink 29919 (time 1715461378.17 total 51.31 last 492.25) - /unlink 35535 (time 1715461388.17 total 61.31 last 561.60) - /unlink 40000 (time 1715461396.34 total 69.48 last 546.13) - /unlink 46225 (time 1715461406.34 total 79.48 last 622.47) - /unlink 50000 (time 1715461411.63 total 84.77 last 713.84) - /unlink 55455 (time 1715461421.63 total 94.77 last 545.49) - /unlink 60000 (time 1715461428.86 total 102.01 last 628.27) total: 64768 /unlink in 108.22 seconds: 598.49 ops/second - open/close 3540 (time 1715461446.07 total 10.00 last 353.97) - open/close 8018 (time 1715461456.07 total 20.00 last 447.76) - open/close 10000 (time 1715461462.29 total 26.22 last 318.62) - open/close 14322 (time 1715461472.29 total 36.22 last 432.17) - open/close 18527 (time 1715461482.29 total 46.22 last 420.47) - open/close 20000 (time 1715461486.87 total 50.80 last 321.96) - open/close 23506 (time 1715461496.87 total 60.80 last 350.57) - open/close 26679 (time 1715461506.87 total 70.80 last 317.27) - open/close 30000 (time 1715461516.67 total 80.61 last 338.67) - open/close 33914 (time 1715461526.68 total 90.61 last 391.34) - open/close 38461 (time 1715461536.68 total 100.61 last 454.58) - open/close 40000 (time 1715461540.26 total 104.20 last 429.20) - open/close 43945 (time 1715461550.27 total 114.20 last 394.45) - open/close 47716 (time 1715461560.27 total 124.20 last 377.00) - open/close 50000 (time 1715461567.20 total 131.13 last 329.60) - open/close 53509 (time 1715461577.20 total 141.13 last 350.88) - open/close 57334 (time 1715461587.20 total 151.13 last 382.41) - open/close 60000 (time 1715461594.24 total 158.17 last 378.81) - open/close 63926 (time 1715461604.24 total 168.17 last 392.56) total: 64768 open/close in 170.72 seconds: 379.39 ops/second - /unlink 6202 (time 1715461617.72 total 10.00 last 620.10) - /unlink 10000 (time 1715461623.00 total 15.28 last 719.48) - /unlink 17682 (time 1715461633.01 total 25.28 last 768.09) - /unlink 20000 (time 1715461635.85 total 28.13 last 814.50) - /unlink 26718 (time 1715461645.85 total 38.13 last 671.78) - /unlink 30000 (time 1715461650.77 total 43.05 last 666.66) - /unlink 35615 (time 1715461660.78 total 53.05 last 561.41) - /unlink 40000 (time 1715461668.90 total 61.18 last 539.60) - /unlink 45204 (time 1715461678.90 total 71.18 last 520.33) - /unlink 50000 (time 1715461686.62 total 78.89 last 621.80) - /unlink 57582 (time 1715461696.62 total 88.89 last 758.16) - /unlink 60000 (time 1715461699.83 total 92.10 last 753.10) total: 64768 /unlink in 100.32 seconds: 645.60 ops/second - open/close 3200 (time 1715461719.33 total 10.00 last 319.91) - open/close 6737 (time 1715461729.33 total 20.00 last 353.65) - open/close 10000 (time 1715461739.02 total 29.70 last 336.58) - open/close 12797 (time 1715461749.03 total 39.70 last 279.62) - open/close 15659 (time 1715461759.03 total 49.70 last 286.19) - open/close 20000 (time 1715461768.83 total 59.51 last 442.66) - open/close 23469 (time 1715461778.88 total 69.55 last 345.48) - open/close 28309 (time 1715461788.88 total 79.55 last 483.96) - open/close 30000 (time 1715461794.45 total 85.13 last 303.31) - open/close 34008 (time 1715461804.45 total 95.13 last 400.74) - open/close 37172 (time 1715461814.45 total 105.13 last 316.38) - open/close 40000 (time 1715461821.90 total 112.58 last 379.76) - open/close 43343 (time 1715461831.90 total 122.58 last 334.26) - open/close 46330 (time 1715461841.90 total 132.58 last 298.63) - open/close 48953 (time 1715461851.91 total 142.58 last 262.25) - open/close 50000 (time 1715461855.57 total 146.24 last 286.15) - open/close 52554 (time 1715461865.57 total 156.24 last 255.36) - open/close 55157 (time 1715461875.57 total 166.24 last 260.27) - open/close 58182 (time 1715461885.57 total 176.24 last 302.43) - open/close 60000 (time 1715461891.89 total 182.56 last 287.65) - open/close 62865 (time 1715461901.89 total 192.57 last 286.40) total: 64768 open/close in 199.31 seconds: 324.96 ops/second - /unlink 4754 (time 1715461919.84 total 10.00 last 475.34) - /unlink 10000 (time 1715461929.35 total 19.51 last 551.87) - /unlink 16033 (time 1715461939.35 total 29.51 last 603.21) - /unlink 20000 (time 1715461947.24 total 37.40 last 502.67) - /unlink 25517 (time 1715461957.24 total 47.40 last 551.62) - /unlink 30000 (time 1715461966.40 total 56.56 last 489.47) - /unlink 34881 (time 1715461976.40 total 66.56 last 488.09) - /unlink 39797 (time 1715461986.40 total 76.56 last 491.50) - /unlink 45114 (time 1715461996.40 total 86.57 last 531.58) - /unlink 50000 (time 1715462004.36 total 94.52 last 614.11) - /unlink 55875 (time 1715462014.36 total 104.52 last 587.47) - /unlink 60000 (time 1715462021.95 total 112.11 last 543.54) total: 64768 /unlink in 120.69 seconds: 536.67 ops/second - open/close 2518 (time 1715462041.74 total 10.00 last 251.73) - open/close 5183 (time 1715462051.74 total 20.00 last 266.50) - open/close 7918 (time 1715462061.74 total 30.00 last 273.48) - open/close 10000 (time 1715462069.27 total 37.54 last 276.22) - open/close 12569 (time 1715462079.28 total 47.54 last 256.83) - open/close 15121 (time 1715462089.28 total 57.55 last 255.14) - open/close 17669 (time 1715462099.28 total 67.55 last 254.74) - open/close 20000 (time 1715462108.53 total 76.80 last 252.00) - open/close 22869 (time 1715462118.53 total 86.80 last 286.85) - open/close 25344 (time 1715462128.54 total 96.80 last 247.42) - open/close 27838 (time 1715462138.54 total 106.81 last 249.33) - open/close 30000 (time 1715462146.20 total 114.46 last 282.30) - open/close 32789 (time 1715462156.20 total 124.47 last 278.86) - open/close 35560 (time 1715462166.20 total 134.47 last 277.08) - open/close 38663 (time 1715462176.20 total 144.47 last 310.18) - open/close 40000 (time 1715462181.69 total 149.96 last 243.68) - open/close 42578 (time 1715462191.69 total 159.96 last 257.74) - open/close 45133 (time 1715462201.70 total 169.96 last 255.41) - open/close 47834 (time 1715462211.70 total 179.97 last 270.02) - open/close 50000 (time 1715462220.19 total 188.45 last 255.18) - open/close 52606 (time 1715462230.19 total 198.46 last 260.53) - open/close 55111 (time 1715462240.19 total 208.46 last 250.49) - open/close 57677 (time 1715462250.19 total 218.46 last 256.56) - open/close 60000 (time 1715462258.74 total 227.01 last 271.62) - open/close 62481 (time 1715462268.75 total 237.01 last 248.06) total: 64768 open/close in 246.26 seconds: 263.01 ops/second - /unlink 4786 (time 1715462289.25 total 10.00 last 478.57) - /unlink 9856 (time 1715462299.25 total 20.00 last 506.95) - /unlink 14650 (time 1715462309.25 total 30.00 last 479.33) - /unlink 19594 (time 1715462319.25 total 40.00 last 494.37) - /unlink 24510 (time 1715462329.26 total 50.01 last 491.53) - /unlink 29967 (time 1715462339.26 total 60.01 last 545.65) - /unlink 34886 (time 1715462349.26 total 70.01 last 491.84) - /unlink 39735 (time 1715462359.26 total 80.01 last 484.87) - /unlink 44728 (time 1715462369.26 total 90.01 last 499.18) - /unlink 49699 (time 1715462379.26 total 100.01 last 497.07) - /unlink 54839 (time 1715462389.26 total 110.01 last 513.99) - /unlink 60000 (time 1715462397.88 total 118.62 last 599.20) total: 64768 /unlink in 125.96 seconds: 514.21 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg304-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg304-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg304-server stop mds service on oleg304-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg304-server LNET ready to unload unloading modules on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing unload_modules_local oleg304-server: LNET ready to unload modules unloaded. pdsh@oleg304-client: oleg304-client: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-client: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2