== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:09:13 (1744535353) Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f) oleg422-server: oleg422-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing load_modules_local oleg422-server: Loading modules from /home/green/git/lustre-release/lustre oleg422-server: detected 4 online CPUs by sysfs oleg422-server: MODOPTS_LIBCFS= oleg422-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg422-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg422-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg422-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4246 (time 1744535395.08 total 10.00 last 424.53) - open/close 7970 (time 1744535405.08 total 20.00 last 372.31) - open/close 10000 (time 1744535409.67 total 24.59 last 442.82) - open/close 15818 (time 1744535419.67 total 34.59 last 581.72) - open/close 20000 (time 1744535427.64 total 42.56 last 524.46) - open/close 26062 (time 1744535437.64 total 52.56 last 606.18) - open/close 30000 (time 1744535444.05 total 58.97 last 614.92) - open/close 35616 (time 1744535454.05 total 68.97 last 561.52) - open/close 40000 (time 1744535461.69 total 76.61 last 573.77) - open/close 45133 (time 1744535471.69 total 86.61 last 513.23) - open/close 50000 (time 1744535481.60 total 96.52 last 491.35) - open/close 54931 (time 1744535491.60 total 106.52 last 493.03) - open/close 59246 (time 1744535501.60 total 116.52 last 431.40) - open/close 60000 (time 1744535503.76 total 118.68 last 348.83) - open/close 63988 (time 1744535513.76 total 128.68 last 398.77) total: 64768 open/close in 131.51 seconds: 492.50 ops/second - /unlink 4817 (time 1744535527.69 total 10.00 last 481.68) - /unlink 9788 (time 1744535537.69 total 20.00 last 496.99) - /unlink 15285 (time 1744535547.69 total 30.00 last 549.68) - /unlink 20000 (time 1744535555.63 total 37.94 last 593.80) - /unlink 29133 (time 1744535565.63 total 47.94 last 913.23) - /unlink 37975 (time 1744535575.63 total 57.94 last 884.18) - /unlink 40000 (time 1744535578.27 total 60.59 last 766.49) - /unlink 49504 (time 1744535588.27 total 70.59 last 950.38) - /unlink 59640 (time 1744535598.27 total 80.59 last 1013.56) total: 64768 /unlink in 85.72 seconds: 755.60 ops/second Waiting for MDT destroys to complete - open/close 5388 (time 1744535683.27 total 10.00 last 538.72) - open/close 10000 (time 1744535691.68 total 18.41 last 548.42) - open/close 15654 (time 1744535701.68 total 28.41 last 565.36) - open/close 20000 (time 1744535709.54 total 36.28 last 552.45) - open/close 24688 (time 1744535719.54 total 46.28 last 468.78) - open/close 29435 (time 1744535729.54 total 56.28 last 474.66) - open/close 34858 (time 1744535739.54 total 66.28 last 542.26) - open/close 40000 (time 1744535749.13 total 75.87 last 536.38) - open/close 45583 (time 1744535759.13 total 85.87 last 558.24) - open/close 50000 (time 1744535767.14 total 93.88 last 551.39) - open/close 55763 (time 1744535777.14 total 103.88 last 576.23) - open/close 60000 (time 1744535784.74 total 111.48 last 557.73) total: 64768 open/close in 119.78 seconds: 540.73 ops/second - /unlink 9871 (time 1744535803.89 total 10.00 last 987.01) - /unlink 20000 (time 1744535813.81 total 19.93 last 1020.61) - /unlink 30000 (time 1744535823.07 total 29.18 last 1080.14) - /unlink 40000 (time 1744535831.97 total 38.09 last 1123.30) - /unlink 50000 (time 1744535840.58 total 46.69 last 1162.39) - /unlink 60000 (time 1744535849.01 total 55.12 last 1185.54) total: 64768 /unlink in 59.17 seconds: 1094.54 ops/second Waiting for MDT destroys to complete - open/close 6099 (time 1744535963.06 total 10.00 last 609.82) - open/close 10000 (time 1744535969.20 total 16.14 last 635.11) - open/close 16274 (time 1744535979.20 total 26.14 last 627.35) - open/close 20000 (time 1744535984.95 total 31.89 last 648.69) - open/close 26248 (time 1744535994.95 total 41.89 last 624.77) - open/close 30000 (time 1744536000.96 total 47.90 last 623.65) - open/close 36309 (time 1744536010.96 total 57.91 last 630.82) - open/close 40000 (time 1744536016.73 total 63.67 last 640.62) - open/close 46431 (time 1744536026.73 total 73.67 last 643.07) - open/close 50000 (time 1744536032.28 total 79.22 last 642.29) - open/close 56083 (time 1744536042.28 total 89.23 last 608.24) - open/close 60000 (time 1744536049.57 total 96.52 last 537.32) - open/close 64700 (time 1744536059.58 total 106.52 last 469.92) total: 64768 open/close in 106.64 seconds: 607.36 ops/second - /unlink 9462 (time 1744536070.56 total 10.00 last 946.12) - /unlink 19390 (time 1744536080.56 total 20.01 last 992.18) - /unlink 28399 (time 1744536090.56 total 30.01 last 900.83) - /unlink 37808 (time 1744536100.57 total 40.01 last 940.79) - /unlink 40000 (time 1744536103.04 total 42.48 last 885.96) - /unlink 49673 (time 1744536113.04 total 52.48 last 967.23) - /unlink 58990 (time 1744536123.04 total 62.48 last 931.63) total: 64768 /unlink in 68.31 seconds: 948.08 ops/second Waiting for MDT destroys to complete - open/close 5266 (time 1744536234.38 total 10.00 last 526.58) - open/close 9574 (time 1744536244.38 total 20.00 last 430.73) - open/close 14934 (time 1744536254.38 total 30.00 last 535.95) - open/close 20000 (time 1744536263.56 total 39.18 last 552.30) - open/close 25534 (time 1744536273.56 total 49.18 last 553.36) - open/close 30000 (time 1744536282.93 total 58.55 last 476.28) - open/close 35332 (time 1744536292.94 total 68.55 last 533.13) - open/close 40000 (time 1744536301.68 total 77.29 last 534.12) - open/close 45506 (time 1744536311.68 total 87.29 last 550.59) - open/close 50000 (time 1744536320.27 total 95.89 last 522.74) - open/close 55342 (time 1744536330.27 total 105.89 last 534.16) - open/close 60000 (time 1744536338.67 total 114.29 last 554.47) total: 64768 open/close in 123.00 seconds: 526.56 ops/second - /unlink 9712 (time 1744536358.26 total 10.00 last 970.79) - /unlink 19624 (time 1744536368.26 total 20.00 last 991.14) - /unlink 29445 (time 1744536378.26 total 30.00 last 982.10) - /unlink 39226 (time 1744536388.26 total 40.01 last 978.03) - /unlink 48855 (time 1744536398.26 total 50.01 last 962.86) - /unlink 58799 (time 1744536408.26 total 60.01 last 994.37) total: 64768 /unlink in 65.72 seconds: 985.57 ops/second Waiting for MDT destroys to complete - open/close 5522 (time 1744536519.72 total 10.00 last 552.17) - open/close 10000 (time 1744536528.68 total 18.96 last 499.62) - open/close 14949 (time 1744536538.68 total 28.96 last 494.84) - open/close 19931 (time 1744536548.68 total 38.97 last 498.07) - open/close 25168 (time 1744536558.69 total 48.97 last 523.61) - open/close 30000 (time 1744536567.59 total 57.87 last 542.83) - open/close 35068 (time 1744536577.59 total 67.87 last 506.73) - open/close 39940 (time 1744536587.59 total 77.87 last 487.13) - open/close 44513 (time 1744536597.59 total 87.87 last 457.30) - open/close 49598 (time 1744536607.59 total 97.87 last 508.46) - open/close 53437 (time 1744536617.59 total 107.87 last 383.87) - open/close 57210 (time 1744536627.59 total 117.88 last 377.16) - open/close 60000 (time 1744536633.33 total 123.62 last 486.13) total: 64768 open/close in 133.35 seconds: 485.71 ops/second - /unlink 7789 (time 1744536654.20 total 10.00 last 778.79) - /unlink 10000 (time 1744536656.84 total 12.63 last 839.68) - /unlink 19311 (time 1744536666.84 total 22.63 last 931.08) - /unlink 28979 (time 1744536676.84 total 32.64 last 966.77) - /unlink 38131 (time 1744536686.84 total 42.64 last 915.10) - /unlink 40000 (time 1744536689.15 total 44.95 last 807.07) - /unlink 46998 (time 1744536699.16 total 54.95 last 699.60) - /unlink 50000 (time 1744536703.91 total 59.71 last 631.74) - /unlink 58722 (time 1744536713.91 total 69.71 last 872.19) total: 64768 /unlink in 77.21 seconds: 838.87 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg422-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg422-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server stop mds service on oleg422-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server stop mds service on oleg422-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server unloading modules via unload_modules_local on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2