== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 02:35:55 (1747809355) Stopping clients: oleg417-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg417-client.virtnet /mnt/lustre2 (opts:-f) oleg417-server: oleg417-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg417-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg417-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg417-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5325 (time 1747809395.48 total 10.00 last 532.41) - open/close 10000 (time 1747809404.46 total 18.98 last 520.95) - open/close 15470 (time 1747809414.46 total 28.98 last 546.98) - open/close 20000 (time 1747809422.63 total 37.15 last 554.19) - open/close 25453 (time 1747809432.63 total 47.15 last 545.26) - open/close 30000 (time 1747809440.99 total 55.51 last 543.96) - open/close 35393 (time 1747809450.99 total 65.51 last 539.29) - open/close 40000 (time 1747809459.60 total 74.12 last 535.04) - open/close 45355 (time 1747809469.60 total 84.12 last 535.47) - open/close 50000 (time 1747809478.72 total 93.24 last 509.31) - open/close 55175 (time 1747809488.72 total 103.24 last 517.49) - open/close 60000 (time 1747809497.69 total 112.20 last 538.35) total: 64768 open/close in 121.52 seconds: 532.98 ops/second - /unlink 9473 (time 1747809517.87 total 10.00 last 947.27) - /unlink 19165 (time 1747809527.87 total 20.00 last 969.18) - /unlink 28853 (time 1747809537.87 total 30.00 last 968.74) - /unlink 38825 (time 1747809547.87 total 40.00 last 997.16) - /unlink 48829 (time 1747809557.87 total 50.00 last 1000.34) - /unlink 58649 (time 1747809567.87 total 60.00 last 981.94) total: 64768 /unlink in 66.30 seconds: 976.89 ops/second Waiting for MDT destroys to complete - open/close 5378 (time 1747809679.21 total 10.00 last 537.75) - open/close 10000 (time 1747809687.90 total 18.69 last 531.87) - open/close 15363 (time 1747809697.90 total 28.69 last 536.29) - open/close 20000 (time 1747809706.71 total 37.50 last 526.54) - open/close 25345 (time 1747809716.71 total 47.50 last 534.46) - open/close 30000 (time 1747809725.63 total 56.42 last 521.82) - open/close 35162 (time 1747809735.63 total 66.42 last 516.14) - open/close 40000 (time 1747809745.20 total 76.00 last 505.25) - open/close 45073 (time 1747809755.21 total 86.00 last 507.18) - open/close 49102 (time 1747809765.21 total 96.00 last 402.90) - open/close 50000 (time 1747809767.52 total 98.31 last 388.38) - open/close 53266 (time 1747809777.52 total 108.31 last 326.52) - open/close 56389 (time 1747809787.52 total 118.31 last 312.24) - open/close 60000 (time 1747809795.85 total 126.65 last 433.45) - open/close 63573 (time 1747809805.86 total 136.65 last 357.25) total: 64768 open/close in 140.98 seconds: 459.41 ops/second - /unlink 5213 (time 1747809821.44 total 10.00 last 521.29) - /unlink 10000 (time 1747809829.10 total 17.66 last 624.93) - /unlink 16039 (time 1747809839.10 total 27.66 last 603.82) - /unlink 20000 (time 1747809844.62 total 33.18 last 718.21) - /unlink 25059 (time 1747809854.62 total 43.18 last 505.89) - /unlink 30000 (time 1747809863.63 total 52.19 last 548.34) - /unlink 36064 (time 1747809873.63 total 62.19 last 606.34) - /unlink 40000 (time 1747809878.76 total 67.32 last 767.54) - /unlink 48691 (time 1747809888.76 total 77.32 last 869.07) - /unlink 55163 (time 1747809898.76 total 87.32 last 647.14) - /unlink 60000 (time 1747809905.42 total 93.98 last 726.53) total: 64768 /unlink in 99.94 seconds: 648.04 ops/second Waiting for MDT destroys to complete - open/close 3383 (time 1747809958.87 total 10.00 last 338.23) - open/close 8145 (time 1747809968.87 total 20.00 last 476.14) - open/close 10000 (time 1747809972.22 total 23.35 last 553.70) - open/close 14395 (time 1747809982.22 total 33.36 last 439.43) - open/close 19296 (time 1747809992.22 total 43.36 last 490.08) - open/close 20000 (time 1747809994.52 total 45.66 last 305.79) - open/close 24630 (time 1747810004.53 total 55.66 last 462.97) - open/close 29869 (time 1747810014.53 total 65.66 last 523.77) - open/close 34319 (time 1747810024.53 total 75.66 last 444.97) - open/close 37888 (time 1747810034.53 total 85.66 last 356.88) - open/close 40000 (time 1747810039.77 total 90.90 last 402.85) - open/close 43914 (time 1747810049.77 total 100.91 last 391.36) - open/close 47750 (time 1747810059.77 total 110.91 last 383.58) - open/close 50000 (time 1747810065.84 total 116.97 last 371.17) - open/close 52859 (time 1747810075.84 total 126.97 last 285.82) - open/close 55670 (time 1747810085.84 total 136.97 last 281.06) - open/close 60000 (time 1747810094.96 total 146.09 last 474.76) - open/close 63150 (time 1747810104.96 total 156.10 last 314.94) total: 64768 open/close in 160.94 seconds: 402.44 ops/second - /unlink 5625 (time 1747810120.93 total 10.00 last 562.42) - /unlink 10000 (time 1747810127.91 total 16.99 last 626.37) - /unlink 16474 (time 1747810137.91 total 26.99 last 647.38) - /unlink 20000 (time 1747810143.32 total 32.40 last 651.27) - /unlink 26746 (time 1747810153.33 total 42.40 last 674.54) - /unlink 30000 (time 1747810159.07 total 48.15 last 566.01) - /unlink 34958 (time 1747810169.08 total 58.15 last 495.75) - /unlink 40000 (time 1747810177.58 total 66.66 last 592.67) - /unlink 48219 (time 1747810187.58 total 76.66 last 821.88) - /unlink 54868 (time 1747810197.58 total 86.66 last 664.84) - /unlink 60000 (time 1747810204.08 total 93.16 last 789.83) total: 64768 /unlink in 99.07 seconds: 653.77 ops/second Waiting for MDT destroys to complete - open/close 3142 (time 1747810275.73 total 10.00 last 314.19) - open/close 6089 (time 1747810285.73 total 20.00 last 294.69) - open/close 9674 (time 1747810295.73 total 30.00 last 358.44) - open/close 14688 (time 1747810305.73 total 40.00 last 501.33) - open/close 20000 (time 1747810315.26 total 49.53 last 557.51) - open/close 25420 (time 1747810325.26 total 59.53 last 541.97) - open/close 30000 (time 1747810335.20 total 69.47 last 460.64) - open/close 34346 (time 1747810345.21 total 79.48 last 434.55) - open/close 38965 (time 1747810355.21 total 89.48 last 461.81) - open/close 40000 (time 1747810357.46 total 91.73 last 459.44) - open/close 43249 (time 1747810367.46 total 101.73 last 324.84) - open/close 46128 (time 1747810377.47 total 111.74 last 287.79) - open/close 50000 (time 1747810386.22 total 120.50 last 442.06) - open/close 53892 (time 1747810396.23 total 130.50 last 389.17) - open/close 57212 (time 1747810406.23 total 140.50 last 331.93) - open/close 60000 (time 1747810414.50 total 148.77 last 337.05) - open/close 63326 (time 1747810424.50 total 158.77 last 332.59) total: 64768 open/close in 164.23 seconds: 394.36 ops/second - /unlink 5913 (time 1747810441.26 total 10.00 last 591.27) - /unlink 10000 (time 1747810447.27 total 16.01 last 679.69) - /unlink 16028 (time 1747810457.27 total 26.02 last 602.70) - /unlink 20000 (time 1747810463.37 total 32.12 last 651.00) - /unlink 26226 (time 1747810473.37 total 42.12 last 622.55) - /unlink 30000 (time 1747810480.70 total 49.44 last 515.12) - /unlink 35058 (time 1747810490.70 total 59.45 last 505.73) - /unlink 40000 (time 1747810500.21 total 68.95 last 519.95) - /unlink 45106 (time 1747810510.21 total 78.95 last 510.56) - /unlink 50000 (time 1747810518.58 total 87.32 last 584.70) - /unlink 55373 (time 1747810528.58 total 97.32 last 537.29) - /unlink 60000 (time 1747810538.10 total 106.84 last 485.90) total: 64768 /unlink in 116.00 seconds: 558.33 ops/second Waiting for MDT destroys to complete - open/close 3415 (time 1747810569.87 total 10.00 last 341.40) - open/close 6598 (time 1747810579.87 total 20.00 last 318.27) - open/close 10000 (time 1747810588.60 total 28.74 last 389.62) - open/close 12970 (time 1747810598.61 total 38.74 last 296.89) - open/close 16711 (time 1747810608.61 total 48.74 last 373.98) - open/close 20000 (time 1747810618.56 total 58.69 last 330.53) - open/close 24784 (time 1747810628.56 total 68.69 last 478.37) - open/close 29541 (time 1747810638.56 total 78.70 last 475.63) - open/close 33467 (time 1747810648.56 total 88.70 last 392.53) - open/close 38393 (time 1747810658.56 total 98.70 last 492.52) - open/close 40000 (time 1747810661.73 total 101.86 last 508.37) - open/close 45434 (time 1747810671.73 total 111.86 last 543.39) - open/close 50000 (time 1747810680.36 total 120.49 last 528.86) - open/close 55004 (time 1747810690.36 total 130.49 last 500.34) - open/close 59472 (time 1747810700.36 total 140.49 last 446.77) - open/close 63585 (time 1747810710.36 total 150.50 last 411.24) total: 64768 open/close in 153.11 seconds: 423.02 ops/second - /unlink 7571 (time 1747810723.80 total 10.00 last 757.02) - /unlink 10000 (time 1747810727.22 total 13.41 last 711.50) - /unlink 17180 (time 1747810737.22 total 23.42 last 717.90) - /unlink 20000 (time 1747810740.41 total 26.61 last 883.86) - /unlink 28381 (time 1747810750.41 total 36.61 last 838.07) - /unlink 30000 (time 1747810753.04 total 39.24 last 614.42) - /unlink 37010 (time 1747810763.04 total 49.24 last 701.00) - /unlink 40000 (time 1747810766.64 total 52.84 last 831.60) - /unlink 48934 (time 1747810776.64 total 62.84 last 893.32) - /unlink 58731 (time 1747810786.64 total 72.84 last 979.68) total: 64768 /unlink in 79.02 seconds: 819.62 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg417-server unloading modules via unload_modules_local on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2