== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:28:50 (1734258530) Stopping clients: oleg326-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg326-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2 oleg326-server: oleg326-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing load_modules_local oleg326-server: Loading modules from /home/green/git/lustre-release/lustre oleg326-server: detected 4 online CPUs by sysfs oleg326-server: Force libcfs to create 2 CPU partitions oleg326-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg326-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg326-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg326-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2071 (time 1734258583.27 total 10.00 last 207.08) - open/close 3899 (time 1734258593.28 total 20.00 last 182.75) - open/close 5746 (time 1734258603.28 total 30.01 last 184.65) - open/close 7626 (time 1734258613.28 total 40.01 last 187.98) - open/close 9606 (time 1734258623.29 total 50.01 last 197.88) - open/close 11716 (time 1734258633.30 total 60.02 last 210.80) - open/close 13511 (time 1734258643.30 total 70.03 last 179.40) - open/close 15196 (time 1734258653.31 total 80.03 last 168.44) - open/close 16988 (time 1734258663.31 total 90.04 last 179.15) - open/close 18576 (time 1734258673.32 total 100.04 last 158.68) - open/close 20000 (time 1734258681.93 total 108.66 last 165.25) - open/close 21733 (time 1734258691.93 total 118.66 last 173.29) - open/close 23403 (time 1734258701.94 total 128.66 last 166.95) - open/close 24979 (time 1734258711.94 total 138.67 last 157.55) - open/close 26667 (time 1734258721.94 total 148.67 last 168.79) - open/close 28512 (time 1734258731.94 total 158.67 last 184.44) - open/close 30000 (time 1734258739.97 total 166.70 last 185.38) - open/close 31979 (time 1734258749.97 total 176.70 last 197.86) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 179.86 seconds: 181.39 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3498 (time 1734258764.51 total 10.00 last 349.76) - /unlink 7488 (time 1734258774.51 total 20.00 last 398.93) - /unlink 10000 (time 1734258782.23 total 27.72 last 325.43) - /unlink 13160 (time 1734258792.24 total 37.72 last 315.92) - /unlink 16293 (time 1734258802.24 total 47.73 last 313.28) - /unlink 19700 (time 1734258812.24 total 57.73 last 340.60) - /unlink 23591 (time 1734258822.24 total 67.73 last 389.08) - /unlink 27538 (time 1734258832.24 total 77.73 last 394.67) - /unlink 30000 (time 1734258837.98 total 83.46 last 429.35) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 90.43 seconds: 360.79 ops/second Waiting for MDT destroys to complete oleg326-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg326-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 - open/close 2181 (time 1734258869.03 total 10.00 last 218.03) - open/close 3992 (time 1734258879.04 total 20.01 last 181.05) - open/close 5756 (time 1734258889.04 total 30.01 last 176.38) - open/close 7529 (time 1734258899.04 total 40.01 last 177.28) - open/close 9196 (time 1734258909.04 total 50.01 last 166.64) - open/close 10000 (time 1734258913.97 total 54.94 last 163.07) - open/close 11657 (time 1734258923.98 total 64.95 last 165.63) - open/close 13330 (time 1734258933.98 total 74.95 last 167.27) - open/close 15053 (time 1734258943.98 total 84.95 last 172.29) - open/close 16850 (time 1734258953.98 total 94.95 last 179.66) - open/close 18675 (time 1734258963.98 total 104.95 last 182.46) - open/close 20000 (time 1734258971.53 total 112.50 last 175.45) - open/close 21720 (time 1734258981.53 total 122.51 last 171.99) - open/close 23615 (time 1734258991.54 total 132.51 last 189.45) - open/close 25584 (time 1734259001.54 total 142.51 last 196.83) - open/close 27859 (time 1734259011.54 total 152.51 last 227.45) - open/close 29810 (time 1734259021.55 total 162.52 last 195.01) - open/close 31605 (time 1734259031.55 total 172.52 last 179.40) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 178.48 seconds: 182.80 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3216 (time 1734259049.04 total 10.00 last 321.56) - /unlink 6744 (time 1734259059.04 total 20.00 last 352.74) - /unlink 10000 (time 1734259069.00 total 29.96 last 326.87) - /unlink 13116 (time 1734259079.00 total 39.97 last 311.54) - /unlink 16262 (time 1734259089.01 total 49.97 last 314.55) - /unlink 19359 (time 1734259099.01 total 59.97 last 309.67) - /unlink 20000 (time 1734259101.12 total 62.08 last 303.15) - /unlink 23021 (time 1734259111.12 total 72.08 last 302.08) - /unlink 27119 (time 1734259121.12 total 82.09 last 409.72) - /unlink 30000 (time 1734259128.92 total 89.88 last 369.78) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 97.59 seconds: 334.33 ops/second Waiting for MDT destroys to complete oleg326-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg326-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 - open/close 1843 (time 1734259158.47 total 10.01 last 184.13) - open/close 3868 (time 1734259168.47 total 20.01 last 202.43) - open/close 5948 (time 1734259178.47 total 30.01 last 207.96) - open/close 7831 (time 1734259188.48 total 40.02 last 188.19) - open/close 9694 (time 1734259198.48 total 50.02 last 186.30) - open/close 11582 (time 1734259208.48 total 60.02 last 188.77) - open/close 13328 (time 1734259218.48 total 70.02 last 174.59) - open/close 15235 (time 1734259228.49 total 80.03 last 190.62) - open/close 17127 (time 1734259238.49 total 90.03 last 189.11) - open/close 19502 (time 1734259248.49 total 100.03 last 237.49) - open/close 22022 (time 1734259258.49 total 110.04 last 251.92) - open/close 24200 (time 1734259268.50 total 120.04 last 217.72) - open/close 26084 (time 1734259278.50 total 130.05 last 188.28) - open/close 28293 (time 1734259288.51 total 140.05 last 220.86) - open/close 30000 (time 1734259297.62 total 149.16 last 187.27) - open/close 31951 (time 1734259307.62 total 159.16 last 195.10) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 162.78 seconds: 200.42 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3459 (time 1734259322.66 total 10.00 last 345.85) - /unlink 8128 (time 1734259332.66 total 20.00 last 466.82) - /unlink 10000 (time 1734259337.06 total 24.40 last 425.29) - /unlink 14270 (time 1734259347.06 total 34.41 last 426.96) - /unlink 17747 (time 1734259357.07 total 44.41 last 347.64) - /unlink 20000 (time 1734259363.32 total 50.66 last 360.45) - /unlink 23644 (time 1734259373.32 total 60.66 last 364.39) - /unlink 27380 (time 1734259383.32 total 70.66 last 373.56) - /unlink 30000 (time 1734259390.62 total 77.96 last 358.91) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 84.49 seconds: 386.15 ops/second Waiting for MDT destroys to complete oleg326-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg326-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 - open/close 1907 (time 1734259418.01 total 10.00 last 190.61) - open/close 3854 (time 1734259428.01 total 20.01 last 194.68) - open/close 5783 (time 1734259438.01 total 30.01 last 192.87) - open/close 7713 (time 1734259448.01 total 40.01 last 193.00) - open/close 9547 (time 1734259458.01 total 50.01 last 183.39) - open/close 10000 (time 1734259460.50 total 52.50 last 182.11) - open/close 11903 (time 1734259470.50 total 62.50 last 190.24) - open/close 13851 (time 1734259480.50 total 72.50 last 194.73) - open/close 15975 (time 1734259490.51 total 82.51 last 212.31) - open/close 17978 (time 1734259500.51 total 92.51 last 200.23) - open/close 19759 (time 1734259510.52 total 102.51 last 178.03) - open/close 21899 (time 1734259520.52 total 112.52 last 213.92) - open/close 24236 (time 1734259530.52 total 122.52 last 233.67) - open/close 26642 (time 1734259540.52 total 132.52 last 240.50) - open/close 28615 (time 1734259550.53 total 142.53 last 197.26) - open/close 30000 (time 1734259557.03 total 149.03 last 213.04) - open/close 32090 (time 1734259567.03 total 159.03 last 208.96) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 161.80 seconds: 201.64 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3839 (time 1734259581.29 total 10.00 last 383.88) - /unlink 8260 (time 1734259591.29 total 20.00 last 442.10) - /unlink 10000 (time 1734259595.21 total 23.92 last 444.30) - /unlink 15044 (time 1734259605.21 total 33.92 last 504.34) - /unlink 20000 (time 1734259614.51 total 43.22 last 532.78) - /unlink 25445 (time 1734259624.51 total 53.22 last 544.38) - /unlink 30000 (time 1734259633.48 total 62.19 last 508.04) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 67.86 seconds: 480.80 ops/second Waiting for MDT destroys to complete oleg326-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg326-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 - open/close 2595 (time 1734259656.23 total 10.00 last 259.47) - open/close 4977 (time 1734259666.23 total 20.00 last 238.17) - open/close 7520 (time 1734259676.23 total 30.00 last 254.27) - open/close 10000 (time 1734259685.95 total 39.72 last 255.34) - open/close 12567 (time 1734259695.95 total 49.72 last 256.64) - open/close 15456 (time 1734259705.95 total 59.72 last 288.82) - open/close 18127 (time 1734259715.95 total 69.72 last 267.08) - open/close 20000 (time 1734259723.57 total 77.34 last 245.91) - open/close 22358 (time 1734259733.57 total 87.34 last 235.68) - open/close 24680 (time 1734259743.57 total 97.34 last 232.18) - open/close 26969 (time 1734259753.57 total 107.35 last 228.88) - open/close 29305 (time 1734259763.58 total 117.35 last 233.55) - open/close 30000 (time 1734259766.42 total 120.19 last 244.60) - open/close 32464 (time 1734259776.42 total 130.19 last 246.40) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 130.79 seconds: 249.46 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4969 (time 1734259788.09 total 10.00 last 496.84) - /unlink 9780 (time 1734259798.09 total 20.00 last 481.08) - /unlink 13720 (time 1734259808.09 total 30.00 last 393.91) - /unlink 18154 (time 1734259818.09 total 40.00 last 443.37) - /unlink 20000 (time 1734259822.04 total 43.95 last 467.78) - /unlink 25578 (time 1734259832.04 total 53.95 last 557.72) - /unlink 30000 (time 1734259840.16 total 62.07 last 544.48) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 66.70 seconds: 489.14 ops/second Waiting for MDT destroys to complete oleg326-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 oleg326-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg326-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg326-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg326-server stop mds service on oleg326-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg326-server unloading modules via unload_modules_local on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg326-client: oleg326-client: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-client: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2