== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:47:53 (1718927273) Stopping clients: oleg323-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg323-client.virtnet /mnt/lustre2 (opts:-f) oleg323-server: oleg323-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: detected 4 online CPUs by sysfs oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg323-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg323-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg323-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock oleg323-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5770 (time 1718927314.10 total 10.00 last 576.99) - open/close 10000 (time 1718927321.60 total 17.50 last 563.93) - open/close 15844 (time 1718927331.60 total 27.50 last 584.31) - open/close 20000 (time 1718927339.00 total 34.91 last 561.18) - open/close 23395 (time 1718927349.01 total 44.91 last 339.44) - open/close 27973 (time 1718927359.01 total 54.92 last 457.54) - open/close 30000 (time 1718927362.98 total 58.89 last 510.51) - open/close 34920 (time 1718927372.98 total 68.89 last 491.97) - open/close 40000 (time 1718927382.13 total 78.04 last 555.28) - open/close 44472 (time 1718927392.13 total 88.04 last 447.10) - open/close 48805 (time 1718927402.14 total 98.04 last 433.22) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 99.65 seconds: 498.70 ops/second - /unlink 10000 (time 1718927413.98 total 9.37 last 1067.71) - /unlink 20000 (time 1718927423.41 total 18.80 last 1060.29) - /unlink 30000 (time 1718927433.37 total 28.76 last 1004.13) - /unlink 38082 (time 1718927443.37 total 38.76 last 808.19) - /unlink 40000 (time 1718927445.74 total 41.13 last 806.69) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 50.02 seconds: 993.60 ops/second - open/close 6330 (time 1718927465.40 total 10.00 last 632.96) - open/close 10000 (time 1718927471.36 total 15.96 last 615.57) - open/close 16116 (time 1718927481.36 total 25.96 last 611.57) - open/close 20000 (time 1718927487.64 total 32.24 last 618.41) - open/close 26020 (time 1718927497.64 total 42.24 last 601.93) - open/close 30000 (time 1718927504.17 total 48.77 last 610.10) - open/close 34898 (time 1718927514.17 total 58.77 last 489.78) - open/close 40000 (time 1718927522.70 total 67.30 last 598.30) - open/close 44803 (time 1718927532.70 total 77.30 last 480.27) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 86.23 seconds: 576.32 ops/second - /unlink 10000 (time 1718927552.38 total 9.85 last 1015.51) - /unlink 19893 (time 1718927562.38 total 19.85 last 989.21) - /unlink 30000 (time 1718927571.72 total 29.19 last 1082.15) - /unlink 40000 (time 1718927580.90 total 38.36 last 1090.33) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 47.09 seconds: 1055.34 ops/second - open/close 6099 (time 1718927600.46 total 10.00 last 609.88) - open/close 10000 (time 1718927606.77 total 16.30 last 618.89) - open/close 16186 (time 1718927616.77 total 26.30 last 618.59) - open/close 20000 (time 1718927623.07 total 32.60 last 605.44) - open/close 26082 (time 1718927633.07 total 42.60 last 608.18) - open/close 30000 (time 1718927639.42 total 48.96 last 616.55) - open/close 36056 (time 1718927649.42 total 58.96 last 605.59) - open/close 40000 (time 1718927655.97 total 65.51 last 602.11) - open/close 46057 (time 1718927665.97 total 75.51 last 605.63) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 81.56 seconds: 609.31 ops/second - /unlink 10000 (time 1718927681.90 total 8.94 last 1118.83) - /unlink 20000 (time 1718927691.02 total 18.06 last 1095.95) - /unlink 30000 (time 1718927699.75 total 26.79 last 1146.23) - /unlink 40000 (time 1718927708.29 total 35.33 last 1170.80) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 43.54 seconds: 1141.33 ops/second - open/close 3008 (time 1718927727.73 total 10.00 last 300.79) - open/close 6385 (time 1718927737.73 total 20.00 last 337.69) - open/close 10000 (time 1718927745.77 total 28.04 last 449.67) - open/close 13697 (time 1718927755.77 total 38.04 last 369.66) - open/close 16661 (time 1718927765.78 total 48.04 last 296.35) - open/close 19425 (time 1718927775.78 total 58.04 last 276.36) - open/close 20000 (time 1718927777.84 total 60.11 last 278.68) - open/close 22697 (time 1718927787.84 total 70.11 last 269.69) - open/close 25734 (time 1718927797.84 total 80.11 last 303.63) - open/close 28417 (time 1718927807.85 total 90.11 last 268.24) - open/close 30000 (time 1718927813.45 total 95.72 last 282.49) - open/close 32845 (time 1718927823.45 total 105.72 last 284.44) - open/close 35585 (time 1718927833.46 total 115.72 last 273.91) - open/close 38419 (time 1718927843.46 total 125.73 last 283.30) - open/close 40000 (time 1718927848.87 total 131.14 last 291.95) - open/close 42784 (time 1718927858.88 total 141.14 last 278.30) - open/close 45946 (time 1718927868.88 total 151.15 last 316.13) open(/mnt/lustre/d106.conf-sanity/f-47862) error: No space left on device total: 47862 open/close in 158.49 seconds: 301.99 ops/second - /unlink 5170 (time 1718927887.56 total 10.00 last 516.98) - /unlink 10000 (time 1718927896.31 total 18.76 last 551.48) - /unlink 15270 (time 1718927906.32 total 28.76 last 526.94) - /unlink 20000 (time 1718927915.73 total 38.17 last 502.38) - /unlink 24991 (time 1718927925.73 total 48.18 last 499.07) - /unlink 30000 (time 1718927932.06 total 54.50 last 791.74) - /unlink 36487 (time 1718927942.06 total 64.50 last 648.69) - /unlink 40000 (time 1718927948.52 total 70.97 last 543.36) - /unlink 45054 (time 1718927958.52 total 80.97 last 505.40) unlink(/mnt/lustre/d106.conf-sanity/f-47862) error: No such file or directory total: 47862 /unlink in 86.37 seconds: 554.15 ops/second - open/close 2657 (time 1718927975.21 total 10.00 last 265.70) - open/close 5332 (time 1718927985.21 total 20.00 last 267.44) - open/close 8060 (time 1718927995.22 total 30.00 last 272.73) - open/close 10000 (time 1718928002.34 total 37.13 last 272.14) - open/close 12752 (time 1718928012.34 total 47.13 last 275.19) - open/close 15555 (time 1718928022.35 total 57.14 last 280.24) - open/close 18196 (time 1718928032.35 total 67.14 last 264.01) - open/close 20000 (time 1718928039.02 total 73.81 last 270.45) - open/close 22666 (time 1718928049.02 total 83.81 last 266.49) - open/close 25227 (time 1718928059.03 total 93.82 last 256.01) - open/close 28037 (time 1718928069.03 total 103.82 last 280.92) - open/close 30000 (time 1718928076.24 total 111.02 last 272.45) - open/close 32617 (time 1718928086.24 total 121.03 last 261.63) - open/close 35288 (time 1718928096.24 total 131.03 last 267.04) - open/close 37929 (time 1718928106.24 total 141.03 last 264.08) - open/close 40000 (time 1718928113.97 total 148.76 last 268.10) - open/close 42884 (time 1718928123.98 total 158.77 last 288.11) - open/close 45484 (time 1718928133.98 total 168.77 last 259.94) open(/mnt/lustre/d106.conf-sanity/f-47862) error: No space left on device total: 47862 open/close in 177.88 seconds: 269.07 ops/second - /unlink 5005 (time 1718928154.36 total 10.00 last 500.47) - /unlink 9990 (time 1718928164.37 total 20.00 last 498.46) - /unlink 14969 (time 1718928174.37 total 30.00 last 497.85) - /unlink 20000 (time 1718928184.29 total 39.93 last 506.92) - /unlink 24969 (time 1718928194.29 total 49.93 last 496.88) - /unlink 29962 (time 1718928204.29 total 59.93 last 499.29) - /unlink 35114 (time 1718928214.29 total 69.93 last 515.11) - /unlink 40000 (time 1718928223.90 total 79.53 last 508.82) - /unlink 45132 (time 1718928233.90 total 89.53 last 513.12) unlink(/mnt/lustre/d106.conf-sanity/f-47862) error: No such file or directory total: 47862 /unlink in 91.72 seconds: 521.84 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg323-server LNET ready to unload unloading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local oleg323-server: LNET ready to unload modules unloaded. pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2