== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:49:56 (1734259796) Stopping clients: oleg304-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg304-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2 oleg304-server: oleg304-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing load_modules_local oleg304-server: Loading modules from /home/green/git/lustre-release/lustre oleg304-server: detected 4 online CPUs by sysfs oleg304-server: Force libcfs to create 2 CPU partitions oleg304-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg304-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg304-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg304-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg304-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2449 (time 1734259853.04 total 10.00 last 244.82) - open/close 4756 (time 1734259863.05 total 20.01 last 230.66) - open/close 6844 (time 1734259873.05 total 30.01 last 208.74) - open/close 8837 (time 1734259883.05 total 40.01 last 199.23) - open/close 10000 (time 1734259889.27 total 46.23 last 187.15) - open/close 11848 (time 1734259899.27 total 56.23 last 184.77) - open/close 13855 (time 1734259909.27 total 66.23 last 200.66) - open/close 16079 (time 1734259919.27 total 76.23 last 222.33) - open/close 18278 (time 1734259929.28 total 86.24 last 219.83) - open/close 20000 (time 1734259936.93 total 93.89 last 224.85) - open/close 22086 (time 1734259946.94 total 103.90 last 208.58) - open/close 24187 (time 1734259956.94 total 113.90 last 210.06) - open/close 26204 (time 1734259966.94 total 123.90 last 201.66) - open/close 28345 (time 1734259976.94 total 133.90 last 214.09) - open/close 30000 (time 1734259984.71 total 141.67 last 213.07) - open/close 32291 (time 1734259994.71 total 151.67 last 229.10) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 152.79 seconds: 213.11 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5031 (time 1734260006.89 total 10.00 last 503.08) - /unlink 9678 (time 1734260016.89 total 20.00 last 464.69) - /unlink 14067 (time 1734260026.89 total 30.00 last 438.86) - /unlink 18812 (time 1734260036.90 total 40.01 last 474.31) - /unlink 20000 (time 1734260039.62 total 42.73 last 435.94) - /unlink 24471 (time 1734260049.62 total 52.73 last 447.05) - /unlink 30000 (time 1734260059.34 total 62.45 last 569.03) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 66.65 seconds: 488.55 ops/second Waiting for MDT destroys to complete oleg304-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg304-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 - open/close 2738 (time 1734260085.01 total 10.00 last 273.76) - open/close 5367 (time 1734260095.01 total 20.00 last 262.88) - open/close 8055 (time 1734260105.01 total 30.00 last 268.74) - open/close 10000 (time 1734260112.00 total 37.00 last 278.18) - open/close 12592 (time 1734260122.00 total 47.00 last 259.18) - open/close 15121 (time 1734260132.00 total 57.00 last 252.90) - open/close 17559 (time 1734260142.00 total 67.00 last 243.79) - open/close 20000 (time 1734260150.93 total 75.93 last 273.31) - open/close 23038 (time 1734260160.94 total 85.93 last 303.79) - open/close 25908 (time 1734260170.94 total 95.93 last 286.93) - open/close 28875 (time 1734260180.94 total 105.93 last 296.65) - open/close 30000 (time 1734260184.66 total 109.65 last 302.49) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 117.36 seconds: 277.43 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5040 (time 1734260203.42 total 10.00 last 503.94) - /unlink 10000 (time 1734260212.91 total 19.49 last 522.62) - /unlink 14968 (time 1734260222.91 total 29.49 last 496.73) - /unlink 19880 (time 1734260232.91 total 39.49 last 491.13) - /unlink 24770 (time 1734260242.91 total 49.49 last 488.99) - /unlink 29783 (time 1734260252.92 total 59.50 last 501.26) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 64.53 seconds: 504.57 ops/second Waiting for MDT destroys to complete oleg304-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg304-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 - open/close 2513 (time 1734260274.76 total 10.00 last 251.20) - open/close 4901 (time 1734260284.76 total 20.00 last 238.79) - open/close 7427 (time 1734260294.76 total 30.01 last 252.58) - open/close 9580 (time 1734260304.76 total 40.01 last 215.27) - open/close 11993 (time 1734260314.76 total 50.01 last 241.30) - open/close 14404 (time 1734260324.76 total 60.01 last 241.07) - open/close 17102 (time 1734260334.76 total 70.01 last 269.79) - open/close 19853 (time 1734260344.76 total 80.01 last 275.02) - open/close 22934 (time 1734260354.77 total 90.01 last 308.05) - open/close 25620 (time 1734260364.77 total 100.02 last 268.52) - open/close 28305 (time 1734260374.77 total 110.02 last 268.44) - open/close 30000 (time 1734260381.22 total 116.46 last 263.02) - open/close 32516 (time 1734260391.22 total 126.47 last 251.50) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 126.67 seconds: 257.05 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4271 (time 1734260402.51 total 10.00 last 427.06) - /unlink 8378 (time 1734260412.52 total 20.00 last 410.62) - /unlink 10000 (time 1734260415.98 total 23.47 last 467.77) - /unlink 14516 (time 1734260425.98 total 33.47 last 451.57) - /unlink 18786 (time 1734260435.99 total 43.47 last 426.89) - /unlink 20000 (time 1734260439.01 total 46.50 last 401.81) - /unlink 24240 (time 1734260449.01 total 56.50 last 423.96) - /unlink 28891 (time 1734260459.01 total 66.50 last 464.98) - /unlink 30000 (time 1734260461.23 total 68.72 last 499.21) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 74.23 seconds: 438.65 ops/second Waiting for MDT destroys to complete oleg304-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg304-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 - open/close 3132 (time 1734260486.36 total 10.00 last 313.10) - open/close 5657 (time 1734260496.37 total 20.01 last 252.43) - open/close 8433 (time 1734260506.37 total 30.01 last 277.59) - open/close 10000 (time 1734260512.27 total 35.91 last 265.46) - open/close 12691 (time 1734260522.27 total 45.91 last 269.02) - open/close 15649 (time 1734260532.27 total 55.91 last 295.75) - open/close 18371 (time 1734260542.28 total 65.92 last 272.14) - open/close 20000 (time 1734260548.94 total 72.58 last 244.30) - open/close 22943 (time 1734260558.95 total 82.59 last 294.20) - open/close 25975 (time 1734260568.95 total 92.59 last 303.20) - open/close 29149 (time 1734260578.95 total 102.59 last 317.36) - open/close 30000 (time 1734260581.33 total 104.97 last 358.07) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 112.25 seconds: 290.08 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5945 (time 1734260599.59 total 10.00 last 594.42) - /unlink 10000 (time 1734260607.12 total 17.53 last 538.34) - /unlink 16006 (time 1734260617.12 total 27.53 last 600.55) - /unlink 20000 (time 1734260623.38 total 33.79 last 638.47) - /unlink 25642 (time 1734260633.38 total 43.79 last 564.11) - /unlink 30000 (time 1734260641.19 total 51.60 last 558.08) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 55.47 seconds: 586.94 ops/second Waiting for MDT destroys to complete oleg304-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg304-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 - open/close 3488 (time 1734260672.75 total 10.00 last 348.73) - open/close 6958 (time 1734260682.75 total 20.00 last 346.93) - open/close 10000 (time 1734260691.02 total 28.27 last 367.94) - open/close 13483 (time 1734260701.02 total 38.27 last 348.30) - open/close 16846 (time 1734260711.02 total 48.27 last 336.26) - open/close 20000 (time 1734260719.65 total 56.90 last 365.50) - open/close 23954 (time 1734260729.65 total 66.90 last 395.30) - open/close 27885 (time 1734260739.65 total 76.91 last 392.99) - open/close 30000 (time 1734260744.42 total 81.67 last 443.67) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 87.65 seconds: 371.46 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7312 (time 1734260761.31 total 10.00 last 731.15) - /unlink 10000 (time 1734260764.55 total 13.24 last 829.21) - /unlink 19166 (time 1734260774.55 total 23.24 last 916.57) - /unlink 29511 (time 1734260784.55 total 33.24 last 1034.47) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 36.43 seconds: 893.71 ops/second Waiting for MDT destroys to complete oleg304-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg304-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg304-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg304-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg304-server stop mds service on oleg304-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg304-server stop mds service on oleg304-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg304-server unloading modules via unload_modules_local on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg304-client: oleg304-client: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-client: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2