== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:49:40 (1734259780) Stopping clients: oleg309-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg309-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg309-client: oleg309-server: ssh exited with exit code 2 oleg309-server: oleg309-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg309-server' oleg309-server: oleg309-server.virtnet: executing load_modules_local oleg309-server: Loading modules from /home/green/git/lustre-release/lustre oleg309-server: detected 4 online CPUs by sysfs oleg309-server: Force libcfs to create 2 CPU partitions oleg309-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg309-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg309-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg309-server: oleg309-server.virtnet: executing set_default_debug -1 all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg309-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg309-server: oleg309-server.virtnet: executing set_default_debug -1 all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg309-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug -1 all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2351 (time 1734259842.69 total 10.00 last 235.03) - open/close 4756 (time 1734259852.69 total 20.00 last 240.48) - open/close 7051 (time 1734259862.70 total 30.01 last 229.44) - open/close 9145 (time 1734259872.70 total 40.01 last 209.33) - open/close 10000 (time 1734259876.77 total 44.08 last 210.00) - open/close 11959 (time 1734259886.77 total 54.08 last 195.87) - open/close 13846 (time 1734259896.78 total 64.09 last 188.56) - open/close 15752 (time 1734259906.79 total 74.10 last 190.35) - open/close 17876 (time 1734259916.80 total 84.11 last 212.17) - open/close 20000 (time 1734259926.17 total 93.48 last 226.74) - open/close 22174 (time 1734259936.17 total 103.48 last 217.39) - open/close 24279 (time 1734259946.18 total 113.49 last 210.41) - open/close 26402 (time 1734259956.18 total 123.49 last 212.27) - open/close 28378 (time 1734259966.18 total 133.49 last 197.56) - open/close 30000 (time 1734259973.65 total 140.96 last 217.21) - open/close 32111 (time 1734259983.65 total 150.96 last 211.04) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 152.92 seconds: 212.92 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4314 (time 1734259996.77 total 10.00 last 431.39) - /unlink 9489 (time 1734260006.77 total 20.00 last 517.47) - /unlink 14352 (time 1734260016.77 total 30.00 last 486.26) - /unlink 18877 (time 1734260026.77 total 40.00 last 452.48) - /unlink 20000 (time 1734260029.29 total 42.52 last 445.95) - /unlink 24748 (time 1734260039.29 total 52.52 last 474.72) - /unlink 29238 (time 1734260049.29 total 62.52 last 448.97) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 68.94 seconds: 472.30 ops/second Waiting for MDT destroys to complete oleg309-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg309-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 - open/close 3197 (time 1734260070.81 total 10.00 last 319.68) - open/close 5965 (time 1734260080.81 total 20.00 last 276.75) - open/close 8760 (time 1734260090.81 total 30.00 last 279.43) - open/close 10000 (time 1734260095.49 total 34.68 last 265.16) - open/close 12783 (time 1734260105.49 total 44.68 last 278.23) - open/close 15652 (time 1734260115.49 total 54.69 last 286.84) - open/close 18286 (time 1734260125.49 total 64.69 last 263.39) - open/close 20000 (time 1734260132.30 total 71.49 last 251.76) - open/close 22497 (time 1734260142.30 total 81.50 last 249.66) - open/close 25310 (time 1734260152.30 total 91.50 last 281.24) - open/close 28502 (time 1734260162.31 total 101.50 last 319.17) - open/close 30000 (time 1734260167.35 total 106.54 last 297.19) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 115.55 seconds: 281.79 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5331 (time 1734260187.34 total 10.00 last 533.07) - /unlink 10000 (time 1734260195.88 total 18.54 last 546.85) - /unlink 14908 (time 1734260205.88 total 28.54 last 490.73) - /unlink 19941 (time 1734260215.88 total 38.54 last 503.26) - /unlink 24887 (time 1734260225.89 total 48.54 last 494.57) - /unlink 29900 (time 1734260235.89 total 58.54 last 501.18) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 64.11 seconds: 507.86 ops/second Waiting for MDT destroys to complete oleg309-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg309-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 - open/close 2988 (time 1734260258.42 total 10.00 last 298.69) - open/close 5863 (time 1734260268.42 total 20.01 last 287.46) - open/close 8372 (time 1734260278.42 total 30.01 last 250.88) - open/close 10000 (time 1734260285.05 total 36.64 last 245.57) - open/close 12601 (time 1734260295.05 total 46.64 last 260.06) - open/close 14835 (time 1734260305.06 total 56.64 last 223.26) - open/close 17344 (time 1734260315.06 total 66.64 last 250.86) - open/close 19819 (time 1734260325.06 total 76.65 last 247.47) - open/close 22361 (time 1734260335.06 total 86.65 last 254.18) - open/close 24864 (time 1734260345.06 total 96.65 last 250.30) - open/close 27645 (time 1734260355.07 total 106.65 last 278.03) - open/close 30000 (time 1734260364.75 total 116.34 last 243.13) - open/close 32471 (time 1734260374.75 total 126.34 last 247.05) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 126.73 seconds: 256.93 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4210 (time 1734260386.21 total 10.00 last 420.92) - /unlink 8360 (time 1734260396.22 total 20.00 last 414.94) - /unlink 10000 (time 1734260400.21 total 24.00 last 410.28) - /unlink 13886 (time 1734260410.21 total 34.00 last 388.57) - /unlink 18440 (time 1734260420.21 total 44.00 last 455.38) - /unlink 20000 (time 1734260423.56 total 47.35 last 465.56) - /unlink 24302 (time 1734260433.57 total 57.35 last 430.10) - /unlink 28475 (time 1734260443.57 total 67.36 last 417.29) - /unlink 30000 (time 1734260447.35 total 71.14 last 403.13) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 76.63 seconds: 424.87 ops/second Waiting for MDT destroys to complete oleg309-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg309-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 - open/close 2598 (time 1734260474.18 total 10.00 last 259.77) - open/close 5459 (time 1734260484.18 total 20.00 last 286.04) - open/close 8132 (time 1734260494.18 total 30.01 last 267.25) - open/close 10000 (time 1734260501.02 total 36.84 last 273.15) - open/close 12874 (time 1734260511.02 total 46.85 last 287.30) - open/close 15671 (time 1734260521.03 total 56.85 last 279.63) - open/close 18697 (time 1734260531.03 total 66.85 last 302.59) - open/close 20000 (time 1734260534.92 total 70.74 last 334.96) - open/close 22758 (time 1734260544.92 total 80.74 last 275.76) - open/close 25108 (time 1734260554.92 total 90.75 last 234.91) - open/close 27670 (time 1734260564.92 total 100.75 last 256.14) - open/close 30000 (time 1734260573.22 total 109.04 last 280.98) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 118.01 seconds: 275.90 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5507 (time 1734260593.18 total 10.00 last 550.69) - /unlink 10000 (time 1734260602.06 total 18.88 last 505.92) - /unlink 14850 (time 1734260612.06 total 28.88 last 484.99) - /unlink 20000 (time 1734260621.42 total 38.24 last 550.33) - /unlink 25029 (time 1734260631.42 total 48.24 last 502.83) - /unlink 29881 (time 1734260641.42 total 58.24 last 485.20) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 63.00 seconds: 516.81 ops/second Waiting for MDT destroys to complete oleg309-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 oleg309-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 - open/close 3317 (time 1734260664.12 total 10.00 last 331.64) - open/close 6221 (time 1734260674.12 total 20.01 last 290.29) - open/close 9093 (time 1734260684.12 total 30.01 last 287.09) - open/close 10000 (time 1734260686.99 total 32.88 last 316.21) - open/close 13036 (time 1734260696.99 total 42.88 last 303.59) - open/close 15933 (time 1734260706.99 total 52.88 last 289.61) - open/close 19293 (time 1734260716.99 total 62.88 last 336.00) - open/close 20000 (time 1734260719.04 total 64.93 last 345.61) - open/close 23587 (time 1734260729.04 total 74.93 last 358.70) - open/close 27146 (time 1734260739.04 total 84.93 last 355.90) - open/close 30000 (time 1734260746.14 total 92.02 last 402.13) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 98.25 seconds: 331.39 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7408 (time 1734260763.41 total 10.00 last 740.79) - /unlink 10000 (time 1734260766.91 total 13.51 last 739.13) - /unlink 19537 (time 1734260776.91 total 23.51 last 953.62) - /unlink 29254 (time 1734260786.91 total 33.51 last 971.66) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 36.61 seconds: 889.30 ops/second Waiting for MDT destroys to complete oleg309-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg309-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg309-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg309-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg309-server stop mds service on oleg309-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg309-server stop mds service on oleg309-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg309-server unloading modules via unload_modules_local on: 'oleg309-server' oleg309-server: oleg309-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg309-client: oleg309-client: ssh exited with exit code 2 pdsh@oleg309-client: oleg309-server: ssh exited with exit code 2 pdsh@oleg309-client: oleg309-client: ssh exited with exit code 2 pdsh@oleg309-client: oleg309-server: ssh exited with exit code 2