== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:19:05 (1723252745) Stopping clients: oleg332-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg332-client.virtnet /mnt/lustre2 (opts:-f) oleg332-server: oleg332-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg332-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg332-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg332-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1964 (time 1723252803.85 total 10.00 last 196.39) - open/close 3755 (time 1723252813.85 total 20.00 last 179.06) - open/close 6371 (time 1723252823.86 total 30.01 last 261.49) - open/close 9362 (time 1723252833.86 total 40.01 last 298.99) - open/close 10000 (time 1723252836.26 total 42.41 last 266.05) - open/close 14458 (time 1723252846.26 total 52.41 last 445.79) - open/close 17836 (time 1723252856.26 total 62.41 last 337.75) - open/close 20000 (time 1723252866.24 total 72.40 last 216.73) - open/close 22628 (time 1723252876.24 total 82.40 last 262.78) - open/close 25594 (time 1723252886.25 total 92.40 last 296.52) - open/close 28134 (time 1723252896.25 total 102.41 last 253.82) - open/close 30000 (time 1723252904.28 total 110.43 last 232.48) - open/close 32541 (time 1723252914.29 total 120.44 last 253.99) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 120.56 seconds: 270.08 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4014 (time 1723252925.55 total 10.00 last 401.38) - /unlink 8073 (time 1723252935.55 total 20.00 last 405.82) - /unlink 10000 (time 1723252939.68 total 24.13 last 466.92) - /unlink 14159 (time 1723252949.68 total 34.13 last 415.82) - /unlink 18578 (time 1723252959.68 total 44.13 last 441.84) - /unlink 20000 (time 1723252962.59 total 47.04 last 489.74) - /unlink 24934 (time 1723252972.59 total 57.04 last 493.34) - /unlink 29853 (time 1723252982.59 total 67.04 last 491.84) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 72.66 seconds: 448.08 ops/second Waiting for MDT destroys to complete oleg332-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg332-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 - open/close 2576 (time 1723253006.61 total 10.00 last 257.51) - open/close 5546 (time 1723253016.61 total 20.01 last 296.91) - open/close 9254 (time 1723253026.61 total 30.01 last 370.75) - open/close 12802 (time 1723253036.61 total 40.01 last 354.76) - open/close 16043 (time 1723253046.61 total 50.01 last 324.08) - open/close 18863 (time 1723253056.61 total 60.01 last 281.94) - open/close 20000 (time 1723253060.72 total 64.12 last 276.95) - open/close 23766 (time 1723253070.72 total 74.12 last 376.50) - open/close 26994 (time 1723253080.72 total 84.12 last 322.75) - open/close 29457 (time 1723253090.73 total 94.13 last 246.21) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 101.63 seconds: 320.37 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7630 (time 1723253109.19 total 10.00 last 762.97) - /unlink 10000 (time 1723253113.27 total 14.08 last 581.57) - /unlink 18075 (time 1723253123.27 total 24.08 last 807.42) - /unlink 20000 (time 1723253125.75 total 26.56 last 776.08) - /unlink 28848 (time 1723253135.75 total 36.56 last 884.69) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 41.19 seconds: 790.56 ops/second Waiting for MDT destroys to complete oleg332-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 oleg332-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 - open/close 3250 (time 1723253172.17 total 10.00 last 324.95) - open/close 6437 (time 1723253182.17 total 20.00 last 318.69) - open/close 10000 (time 1723253192.14 total 29.97 last 357.56) - open/close 14453 (time 1723253202.14 total 39.97 last 445.26) - open/close 18422 (time 1723253212.14 total 49.97 last 396.84) - open/close 20000 (time 1723253215.80 total 53.63 last 430.83) - open/close 23680 (time 1723253225.80 total 63.63 last 367.94) - open/close 27742 (time 1723253235.81 total 73.63 last 406.16) - open/close 30000 (time 1723253241.68 total 79.51 last 384.39) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 87.41 seconds: 372.51 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6328 (time 1723253260.57 total 10.00 last 632.75) - /unlink 10000 (time 1723253267.20 total 16.62 last 554.38) - /unlink 17461 (time 1723253277.20 total 26.62 last 746.10) - /unlink 20000 (time 1723253281.06 total 30.49 last 657.56) - /unlink 27238 (time 1723253291.06 total 40.49 last 723.80) - /unlink 30000 (time 1723253294.65 total 44.08 last 768.19) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 47.01 seconds: 692.68 ops/second Waiting for MDT destroys to complete oleg332-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg332-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 - open/close 3723 (time 1723253322.30 total 10.00 last 372.23) - open/close 7010 (time 1723253332.30 total 20.00 last 328.70) - open/close 10000 (time 1723253339.92 total 27.62 last 392.45) - open/close 14682 (time 1723253349.92 total 37.62 last 468.15) - open/close 19397 (time 1723253359.92 total 47.62 last 471.43) - open/close 24709 (time 1723253369.92 total 57.63 last 531.12) - open/close 29287 (time 1723253379.92 total 67.63 last 457.79) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 74.61 seconds: 436.41 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9314 (time 1723253397.74 total 10.00 last 931.26) - /unlink 19021 (time 1723253407.74 total 20.00 last 970.67) - /unlink 28747 (time 1723253417.75 total 30.00 last 972.47) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.17 seconds: 952.85 ops/second Waiting for MDT destroys to complete oleg332-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg332-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 - open/close 5206 (time 1723253459.79 total 10.00 last 520.51) - open/close 10000 (time 1723253468.70 total 18.91 last 537.89) - open/close 15438 (time 1723253478.70 total 28.92 last 543.75) - open/close 20000 (time 1723253486.71 total 36.92 last 570.08) - open/close 25042 (time 1723253496.71 total 46.92 last 504.16) - open/close 29807 (time 1723253506.71 total 56.92 last 476.42) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 61.91 seconds: 525.95 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1723253521.99 total 9.40 last 1063.86) - /unlink 20000 (time 1723253530.85 total 18.25 last 1129.55) - /unlink 30000 (time 1723253540.30 total 27.70 last 1057.99) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 30.04 seconds: 1084.03 ops/second Waiting for MDT destroys to complete oleg332-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg332-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg332-server unloading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2