== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:29:10 (1734258550) Stopping clients: oleg124-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg124-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2 oleg124-server: oleg124-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: detected 4 online CPUs by sysfs oleg124-server: Force libcfs to create 2 CPU partitions oleg124-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg124-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg124-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg124-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg124-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1348 (time 1734258615.07 total 10.01 last 134.72) - open/close 2715 (time 1734258625.07 total 20.01 last 136.62) - open/close 4081 (time 1734258635.07 total 30.01 last 136.60) - open/close 5399 (time 1734258645.07 total 40.01 last 131.78) - open/close 6810 (time 1734258655.08 total 50.02 last 141.05) - open/close 8256 (time 1734258665.08 total 60.02 last 144.52) - open/close 9748 (time 1734258675.08 total 70.02 last 149.19) - open/close 11160 (time 1734258685.09 total 80.03 last 141.14) - open/close 12837 (time 1734258695.09 total 90.03 last 167.70) - open/close 14316 (time 1734258705.09 total 100.03 last 147.84) - open/close 15776 (time 1734258715.09 total 110.03 last 145.96) - open/close 17059 (time 1734258725.10 total 120.04 last 128.25) - open/close 18344 (time 1734258735.10 total 130.04 last 128.47) - open/close 19630 (time 1734258745.10 total 140.04 last 128.55) - open/close 20000 (time 1734258747.86 total 142.80 last 134.33) - open/close 21373 (time 1734258757.86 total 152.81 last 137.21) - open/close 22701 (time 1734258767.87 total 162.81 last 132.74) - open/close 24106 (time 1734258777.87 total 172.81 last 140.48) - open/close 25609 (time 1734258787.87 total 182.81 last 150.27) - open/close 27271 (time 1734258797.88 total 192.82 last 166.13) - open/close 28824 (time 1734258807.88 total 202.82 last 155.23) - open/close 30000 (time 1734258816.29 total 211.23 last 139.87) - open/close 31638 (time 1734258826.29 total 221.23 last 163.77) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 226.83 seconds: 143.83 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 2954 (time 1734258843.41 total 10.00 last 295.38) - /unlink 5776 (time 1734258853.41 total 20.00 last 282.12) - /unlink 8419 (time 1734258863.41 total 30.00 last 264.29) - /unlink 10000 (time 1734258868.79 total 35.39 last 293.76) - /unlink 12768 (time 1734258878.80 total 45.39 last 276.69) - /unlink 15580 (time 1734258888.80 total 55.39 last 281.07) - /unlink 18539 (time 1734258898.81 total 65.40 last 295.80) - /unlink 20000 (time 1734258903.98 total 70.57 last 282.39) - /unlink 22857 (time 1734258913.98 total 80.58 last 285.57) - /unlink 25921 (time 1734258923.99 total 90.58 last 306.35) - /unlink 29326 (time 1734258933.99 total 100.58 last 340.45) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 109.90 seconds: 296.88 ops/second Waiting for MDT destroys to complete oleg124-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg124-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 - open/close 1713 (time 1734258962.69 total 10.00 last 171.30) - open/close 3132 (time 1734258972.69 total 20.00 last 141.87) - open/close 4582 (time 1734258982.70 total 30.01 last 144.88) - open/close 6061 (time 1734258992.71 total 40.02 last 147.81) - open/close 7545 (time 1734259002.71 total 50.02 last 148.37) - open/close 9056 (time 1734259012.71 total 60.02 last 151.09) - open/close 10000 (time 1734259018.89 total 66.20 last 152.65) - open/close 11678 (time 1734259028.90 total 76.21 last 167.76) - open/close 13078 (time 1734259038.90 total 86.21 last 139.92) - open/close 14533 (time 1734259048.91 total 96.21 last 145.46) - open/close 16060 (time 1734259058.91 total 106.22 last 152.65) - open/close 17486 (time 1734259068.91 total 116.22 last 142.53) - open/close 18945 (time 1734259078.92 total 126.23 last 145.81) - open/close 20000 (time 1734259085.88 total 133.19 last 151.62) - open/close 21544 (time 1734259095.88 total 143.19 last 154.31) - open/close 23000 (time 1734259105.89 total 153.19 last 145.58) - open/close 24425 (time 1734259115.89 total 163.20 last 142.44) - open/close 25774 (time 1734259125.90 total 173.21 last 134.77) - open/close 27090 (time 1734259135.90 total 183.21 last 131.55) - open/close 28462 (time 1734259145.91 total 193.22 last 137.11) - open/close 29761 (time 1734259155.92 total 203.23 last 129.82) - open/close 30989 (time 1734259165.92 total 213.23 last 122.74) - open/close 32302 (time 1734259175.93 total 223.24 last 131.21) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 225.67 seconds: 144.57 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 2560 (time 1734259190.45 total 10.00 last 255.94) - /unlink 4896 (time 1734259200.45 total 20.00 last 233.57) - /unlink 7370 (time 1734259210.45 total 30.01 last 247.35) - /unlink 9761 (time 1734259220.45 total 40.01 last 239.05) - /unlink 12243 (time 1734259230.46 total 50.01 last 248.16) - /unlink 14739 (time 1734259240.46 total 60.01 last 249.58) - /unlink 17360 (time 1734259250.46 total 70.01 last 262.06) - /unlink 20000 (time 1734259259.59 total 79.15 last 288.91) - /unlink 22832 (time 1734259269.60 total 89.15 last 283.14) - /unlink 25922 (time 1734259279.60 total 99.15 last 308.98) - /unlink 29097 (time 1734259289.60 total 109.15 last 317.50) - /unlink 30000 (time 1734259292.70 total 112.26 last 290.64) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 121.66 seconds: 268.18 ops/second Waiting for MDT destroys to complete oleg124-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 oleg124-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 - open/close 1461 (time 1734259323.95 total 10.00 last 146.08) - open/close 2945 (time 1734259333.95 total 20.00 last 148.39) - open/close 4249 (time 1734259343.96 total 30.00 last 130.38) - open/close 5784 (time 1734259353.96 total 40.01 last 153.47) - open/close 7245 (time 1734259363.96 total 50.01 last 146.09) - open/close 8585 (time 1734259373.96 total 60.01 last 133.98) - open/close 10000 (time 1734259382.65 total 68.69 last 162.91) - open/close 11699 (time 1734259392.65 total 78.70 last 169.82) - open/close 13328 (time 1734259402.66 total 88.70 last 162.84) - open/close 14982 (time 1734259412.66 total 98.71 last 165.29) - open/close 16732 (time 1734259422.67 total 108.72 last 174.90) - open/close 18463 (time 1734259432.67 total 118.72 last 173.05) - open/close 20000 (time 1734259441.81 total 127.86 last 168.19) - open/close 21488 (time 1734259451.82 total 137.86 last 148.68) - open/close 23211 (time 1734259461.82 total 147.86 last 172.30) - open/close 25136 (time 1734259471.82 total 157.87 last 192.46) - open/close 27068 (time 1734259481.82 total 167.87 last 193.19) - open/close 29069 (time 1734259491.82 total 177.87 last 200.07) - open/close 30000 (time 1734259496.40 total 182.45 last 203.39) - open/close 31919 (time 1734259506.40 total 192.45 last 191.81) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 196.19 seconds: 166.30 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3718 (time 1734259521.28 total 10.00 last 371.76) - /unlink 7492 (time 1734259531.29 total 20.00 last 377.38) - /unlink 10000 (time 1734259538.91 total 27.63 last 328.81) - /unlink 13020 (time 1734259548.92 total 37.63 last 301.85) - /unlink 15690 (time 1734259558.92 total 47.63 last 266.98) - /unlink 18655 (time 1734259568.92 total 57.64 last 296.49) - /unlink 20000 (time 1734259573.32 total 62.04 last 305.60) - /unlink 23102 (time 1734259583.32 total 72.04 last 310.15) - /unlink 26033 (time 1734259593.32 total 82.04 last 293.04) - /unlink 29174 (time 1734259603.32 total 92.04 last 314.08) - /unlink 30000 (time 1734259606.00 total 94.71 last 309.02) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 102.04 seconds: 319.73 ops/second Waiting for MDT destroys to complete oleg124-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 oleg124-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 - open/close 1627 (time 1734259629.86 total 10.01 last 162.59) - open/close 3378 (time 1734259639.86 total 20.01 last 175.09) - open/close 5099 (time 1734259649.86 total 30.01 last 172.06) - open/close 6819 (time 1734259659.87 total 40.02 last 171.91) - open/close 8464 (time 1734259669.87 total 50.02 last 164.46) - open/close 10000 (time 1734259679.03 total 59.18 last 167.73) - open/close 11828 (time 1734259689.03 total 69.18 last 182.71) - open/close 13580 (time 1734259699.04 total 79.18 last 175.13) - open/close 15267 (time 1734259709.04 total 89.19 last 168.67) - open/close 16793 (time 1734259719.04 total 99.19 last 152.56) - open/close 18343 (time 1734259729.05 total 109.20 last 154.89) - open/close 19960 (time 1734259739.05 total 119.20 last 161.65) - open/close 21773 (time 1734259749.06 total 129.20 last 181.23) - open/close 23487 (time 1734259759.06 total 139.20 last 171.38) - open/close 25084 (time 1734259769.06 total 149.21 last 159.66) - open/close 26980 (time 1734259779.06 total 159.21 last 189.60) - open/close 28906 (time 1734259789.06 total 169.21 last 192.57) - open/close 30000 (time 1734259794.01 total 174.16 last 220.90) - open/close 31816 (time 1734259804.02 total 184.17 last 181.44) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 188.64 seconds: 172.95 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3816 (time 1734259819.92 total 10.00 last 381.42) - /unlink 7406 (time 1734259829.92 total 20.01 last 358.92) - /unlink 10000 (time 1734259837.97 total 28.06 last 322.06) - /unlink 13181 (time 1734259847.98 total 38.06 last 317.99) - /unlink 16825 (time 1734259857.98 total 48.07 last 364.33) - /unlink 20000 (time 1734259867.88 total 57.97 last 320.72) - /unlink 23128 (time 1734259877.88 total 67.97 last 312.72) - /unlink 26580 (time 1734259887.88 total 77.97 last 345.12) - /unlink 30000 (time 1734259897.18 total 87.27 last 367.65) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 94.72 seconds: 344.44 ops/second Waiting for MDT destroys to complete oleg124-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg124-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 - open/close 2161 (time 1734259919.39 total 10.00 last 216.07) - open/close 4324 (time 1734259929.39 total 20.00 last 216.26) - open/close 6359 (time 1734259939.39 total 30.00 last 203.46) - open/close 8484 (time 1734259949.39 total 40.01 last 212.45) - open/close 10000 (time 1734259956.37 total 46.98 last 217.29) - open/close 12251 (time 1734259966.37 total 56.99 last 225.05) - open/close 14340 (time 1734259976.38 total 66.99 last 208.76) - open/close 16477 (time 1734259986.38 total 76.99 last 213.68) - open/close 18657 (time 1734259996.38 total 87.00 last 217.95) - open/close 20000 (time 1734260002.43 total 93.04 last 222.12) - open/close 22240 (time 1734260012.43 total 103.04 last 223.97) - open/close 24422 (time 1734260022.43 total 113.04 last 218.18) - open/close 26589 (time 1734260032.43 total 123.05 last 216.67) - open/close 28692 (time 1734260042.43 total 133.05 last 210.21) - open/close 30000 (time 1734260048.54 total 139.16 last 214.09) - open/close 31999 (time 1734260058.55 total 149.16 last 199.82) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 152.44 seconds: 214.02 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3293 (time 1734260073.12 total 10.00 last 329.29) - /unlink 6662 (time 1734260083.13 total 20.01 last 336.74) - /unlink 10000 (time 1734260091.94 total 28.81 last 378.91) - /unlink 13711 (time 1734260101.94 total 38.82 last 371.04) - /unlink 17555 (time 1734260111.94 total 48.82 last 384.40) - /unlink 20000 (time 1734260118.45 total 55.33 last 375.16) - /unlink 24137 (time 1734260128.45 total 65.33 last 413.70) - /unlink 28481 (time 1734260138.46 total 75.33 last 434.38) - /unlink 30000 (time 1734260141.74 total 78.62 last 461.91) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 84.60 seconds: 385.64 ops/second Waiting for MDT destroys to complete oleg124-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg124-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg124-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server unloading modules via unload_modules_local on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2