== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:21:51 (1715332911) Stopping clients: oleg119-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg119-client.virtnet /mnt/lustre2 (opts:-f) oleg119-server: oleg119-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing load_modules_local oleg119-server: Loading modules from /home/green/git/lustre-release/lustre oleg119-server: detected 4 online CPUs by sysfs oleg119-server: Force libcfs to create 2 CPU partitions oleg119-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg119-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg119-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg119-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1409 (time 1715332969.47 total 10.00 last 140.88) - open/close 2943 (time 1715332979.48 total 20.00 last 153.37) - open/close 4572 (time 1715332989.48 total 30.01 last 162.86) - open/close 6170 (time 1715332999.49 total 40.02 last 159.63) - open/close 7846 (time 1715333009.50 total 50.03 last 167.44) - open/close 9636 (time 1715333019.50 total 60.03 last 178.91) - open/close 10000 (time 1715333021.51 total 62.04 last 181.67) - open/close 11642 (time 1715333031.51 total 72.04 last 164.20) - open/close 13445 (time 1715333041.51 total 82.04 last 180.25) - open/close 15184 (time 1715333051.51 total 92.04 last 173.85) - open/close 17068 (time 1715333061.52 total 102.04 last 188.37) - open/close 18787 (time 1715333071.52 total 112.05 last 171.84) - open/close 20000 (time 1715333078.36 total 118.89 last 177.33) - open/close 21738 (time 1715333088.36 total 128.89 last 173.74) - open/close 23439 (time 1715333098.37 total 138.89 last 170.04) - open/close 25360 (time 1715333108.37 total 148.90 last 192.06) - open/close 27209 (time 1715333118.37 total 158.90 last 184.83) - open/close 28896 (time 1715333128.38 total 168.90 last 168.63) - open/close 30000 (time 1715333134.61 total 175.13 last 177.17) - open/close 31516 (time 1715333144.61 total 185.14 last 151.55) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 191.67 seconds: 169.87 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 3086 (time 1715333162.74 total 10.00 last 308.48) - /unlink 6613 (time 1715333172.74 total 20.01 last 352.65) - /unlink 10000 (time 1715333181.76 total 29.02 last 375.71) - /unlink 13871 (time 1715333191.76 total 39.02 last 387.02) - /unlink 17925 (time 1715333201.76 total 49.02 last 405.36) - /unlink 20000 (time 1715333206.37 total 53.63 last 450.03) - /unlink 24327 (time 1715333216.37 total 63.63 last 432.69) - /unlink 27899 (time 1715333226.37 total 73.64 last 357.13) - /unlink 30000 (time 1715333231.66 total 78.92 last 397.58) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 84.82 seconds: 383.89 ops/second Waiting for MDT destroys to complete - open/close 2066 (time 1715333258.28 total 10.00 last 206.54) - open/close 4548 (time 1715333268.29 total 20.01 last 248.07) - open/close 7055 (time 1715333278.29 total 30.01 last 250.69) - open/close 9159 (time 1715333288.29 total 40.01 last 210.36) - open/close 10000 (time 1715333292.25 total 43.98 last 212.06) - open/close 11911 (time 1715333302.26 total 53.98 last 191.03) - open/close 13840 (time 1715333312.26 total 63.98 last 192.85) - open/close 16046 (time 1715333322.26 total 73.98 last 220.55) - open/close 17859 (time 1715333332.27 total 83.99 last 181.26) - open/close 19986 (time 1715333342.27 total 93.99 last 212.69) - open/close 22844 (time 1715333352.27 total 103.99 last 285.75) - open/close 25829 (time 1715333362.27 total 113.99 last 298.48) - open/close 28835 (time 1715333372.27 total 123.99 last 300.56) - open/close 30000 (time 1715333376.15 total 127.87 last 300.34) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 135.17 seconds: 240.88 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6495 (time 1715333394.41 total 10.00 last 649.48) - /unlink 10000 (time 1715333400.45 total 16.04 last 580.20) - /unlink 15204 (time 1715333410.46 total 26.04 last 520.36) - /unlink 20000 (time 1715333419.90 total 35.48 last 507.95) - /unlink 24840 (time 1715333429.90 total 45.49 last 483.94) - /unlink 30000 (time 1715333438.33 total 53.92 last 612.09) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 59.50 seconds: 547.20 ops/second Waiting for MDT destroys to complete - open/close 2317 (time 1715333463.33 total 10.00 last 231.60) - open/close 5061 (time 1715333473.33 total 20.01 last 274.36) - open/close 8835 (time 1715333483.33 total 30.01 last 377.36) - open/close 10000 (time 1715333486.55 total 33.22 last 362.38) - open/close 12845 (time 1715333496.55 total 43.22 last 284.46) - open/close 15225 (time 1715333506.55 total 53.23 last 237.94) - open/close 18011 (time 1715333516.55 total 63.23 last 278.57) - open/close 20000 (time 1715333522.57 total 69.24 last 330.72) - open/close 22935 (time 1715333532.57 total 79.24 last 293.49) - open/close 25554 (time 1715333542.57 total 89.24 last 261.85) - open/close 27898 (time 1715333552.57 total 99.25 last 234.31) - open/close 30000 (time 1715333559.98 total 106.66 last 283.61) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 114.47 seconds: 284.45 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5381 (time 1715333578.83 total 10.00 last 538.01) - /unlink 10000 (time 1715333586.91 total 18.08 last 571.93) - /unlink 15418 (time 1715333596.91 total 28.08 last 541.73) - /unlink 20000 (time 1715333604.93 total 36.10 last 570.98) - /unlink 26062 (time 1715333614.94 total 46.11 last 606.13) - /unlink 30000 (time 1715333621.00 total 52.17 last 648.88) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 55.81 seconds: 583.44 ops/second Waiting for MDT destroys to complete - open/close 3034 (time 1715333643.98 total 10.00 last 303.37) - open/close 6112 (time 1715333653.98 total 20.00 last 307.72) - open/close 9664 (time 1715333663.98 total 30.01 last 355.13) - open/close 13742 (time 1715333673.98 total 40.01 last 407.79) - open/close 17700 (time 1715333683.98 total 50.01 last 395.79) - open/close 20000 (time 1715333689.63 total 55.66 last 406.97) - open/close 24157 (time 1715333699.63 total 65.66 last 415.65) - open/close 28582 (time 1715333709.63 total 75.66 last 442.49) - open/close 30000 (time 1715333712.90 total 78.92 last 434.34) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 84.14 seconds: 386.95 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8181 (time 1715333729.04 total 10.00 last 817.92) - /unlink 10000 (time 1715333731.19 total 12.15 last 846.08) - /unlink 19483 (time 1715333741.19 total 22.15 last 948.27) - /unlink 29549 (time 1715333751.19 total 32.15 last 1006.53) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 35.49 seconds: 917.48 ops/second Waiting for MDT destroys to complete - open/close 5431 (time 1715333793.14 total 10.00 last 543.04) - open/close 10000 (time 1715333802.55 total 19.41 last 485.57) - open/close 14231 (time 1715333812.55 total 29.41 last 423.08) - open/close 19039 (time 1715333822.55 total 39.41 last 480.80) - open/close 24652 (time 1715333832.55 total 49.41 last 561.25) - open/close 30000 (time 1715333842.04 total 58.90 last 563.69) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 63.63 seconds: 511.71 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1715333857.55 total 9.92 last 1008.31) - /unlink 20000 (time 1715333866.96 total 19.33 last 1061.96) - /unlink 30000 (time 1715333876.70 total 29.07 last 1027.57) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 31.46 seconds: 1035.10 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg119-server unloading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2