== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:30:13 (1734258613) Stopping clients: oleg311-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg311-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 oleg311-server: oleg311-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing load_modules_local oleg311-server: Loading modules from /home/green/git/lustre-release/lustre oleg311-server: detected 4 online CPUs by sysfs oleg311-server: Force libcfs to create 2 CPU partitions oleg311-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg311-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg311-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg311-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg311-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1453 (time 1734258673.09 total 10.00 last 145.24) - open/close 2901 (time 1734258683.09 total 20.01 last 144.74) - open/close 4487 (time 1734258693.09 total 30.01 last 158.56) - open/close 5971 (time 1734258703.10 total 40.01 last 148.39) - open/close 7463 (time 1734258713.10 total 50.02 last 149.15) - open/close 9050 (time 1734258723.10 total 60.02 last 158.61) - open/close 10000 (time 1734258728.80 total 65.72 last 166.71) - open/close 11627 (time 1734258738.81 total 75.72 last 162.64) - open/close 13353 (time 1734258748.81 total 85.72 last 172.60) - open/close 15023 (time 1734258758.81 total 95.72 last 166.98) - open/close 16792 (time 1734258768.81 total 105.73 last 176.82) - open/close 18506 (time 1734258778.82 total 115.73 last 171.31) - open/close 20000 (time 1734258788.72 total 125.64 last 150.89) - open/close 21495 (time 1734258798.72 total 135.64 last 149.46) - open/close 23021 (time 1734258808.73 total 145.65 last 152.48) - open/close 24838 (time 1734258818.73 total 155.65 last 181.63) - open/close 26521 (time 1734258828.74 total 165.65 last 168.22) - open/close 28457 (time 1734258838.74 total 175.66 last 193.55) - open/close 30000 (time 1734258847.82 total 184.74 last 169.91) - open/close 31961 (time 1734258857.83 total 194.74 last 196.05) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 198.46 seconds: 164.40 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3473 (time 1734258872.86 total 10.00 last 347.26) - /unlink 6208 (time 1734258882.87 total 20.00 last 273.47) - /unlink 9138 (time 1734258892.87 total 30.00 last 292.95) - /unlink 10000 (time 1734258895.85 total 32.98 last 289.28) - /unlink 12788 (time 1734258905.85 total 42.99 last 278.63) - /unlink 15514 (time 1734258915.85 total 52.99 last 272.60) - /unlink 18318 (time 1734258925.86 total 62.99 last 280.31) - /unlink 20000 (time 1734258932.11 total 69.24 last 269.13) - /unlink 22913 (time 1734258942.11 total 79.25 last 291.21) - /unlink 25985 (time 1734258952.11 total 89.25 last 307.17) - /unlink 28997 (time 1734258962.11 total 99.25 last 301.16) - /unlink 30000 (time 1734258965.14 total 102.28 last 330.66) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 111.31 seconds: 293.10 ops/second Waiting for MDT destroys to complete oleg311-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 oleg311-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 - open/close 1848 (time 1734258992.61 total 10.00 last 184.80) - open/close 3702 (time 1734259002.62 total 20.00 last 185.39) - open/close 5768 (time 1734259012.62 total 30.00 last 206.56) - open/close 7496 (time 1734259022.62 total 40.01 last 172.70) - open/close 9173 (time 1734259032.63 total 50.01 last 167.64) - open/close 10000 (time 1734259037.68 total 55.06 last 163.69) - open/close 11578 (time 1734259047.68 total 65.07 last 157.78) - open/close 13270 (time 1734259057.68 total 75.07 last 169.19) - open/close 14872 (time 1734259067.68 total 85.07 last 160.15) - open/close 16425 (time 1734259077.69 total 95.07 last 155.22) - open/close 17962 (time 1734259087.69 total 105.08 last 153.65) - open/close 19559 (time 1734259097.69 total 115.08 last 159.69) - open/close 20000 (time 1734259100.59 total 117.97 last 152.38) - open/close 21556 (time 1734259110.59 total 127.98 last 155.49) - open/close 23445 (time 1734259120.60 total 137.98 last 188.82) - open/close 25144 (time 1734259130.60 total 147.99 last 169.83) - open/close 26775 (time 1734259140.61 total 157.99 last 162.99) - open/close 28407 (time 1734259150.61 total 168.00 last 163.17) - open/close 30000 (time 1734259159.95 total 177.34 last 170.50) - open/close 31793 (time 1734259169.96 total 187.34 last 179.25) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 191.27 seconds: 170.57 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3359 (time 1734259185.37 total 10.00 last 335.87) - /unlink 6393 (time 1734259195.37 total 20.00 last 303.33) - /unlink 9715 (time 1734259205.38 total 30.00 last 332.18) - /unlink 12711 (time 1734259215.38 total 40.01 last 299.53) - /unlink 15856 (time 1734259225.38 total 50.01 last 314.40) - /unlink 19186 (time 1734259235.38 total 60.01 last 332.90) - /unlink 20000 (time 1734259238.01 total 62.64 last 309.72) - /unlink 24378 (time 1734259248.01 total 72.64 last 437.68) - /unlink 29095 (time 1734259258.02 total 82.65 last 471.57) - /unlink 30000 (time 1734259260.06 total 84.68 last 443.99) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 91.04 seconds: 358.39 ops/second Waiting for MDT destroys to complete oleg311-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 oleg311-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 - open/close 2093 (time 1734259286.47 total 10.01 last 209.19) - open/close 4119 (time 1734259296.47 total 20.01 last 202.59) - open/close 6047 (time 1734259306.47 total 30.01 last 192.80) - open/close 8013 (time 1734259316.48 total 40.01 last 196.48) - open/close 10000 (time 1734259325.74 total 49.27 last 214.67) - open/close 12321 (time 1734259335.74 total 59.27 last 232.08) - open/close 14525 (time 1734259345.74 total 69.27 last 220.32) - open/close 16320 (time 1734259355.75 total 79.28 last 179.42) - open/close 18182 (time 1734259365.75 total 89.28 last 186.20) - open/close 20000 (time 1734259375.18 total 98.71 last 192.68) - open/close 21787 (time 1734259385.18 total 108.71 last 178.66) - open/close 23710 (time 1734259395.19 total 118.72 last 192.25) - open/close 25670 (time 1734259405.19 total 128.72 last 195.92) - open/close 27544 (time 1734259415.19 total 138.72 last 187.40) - open/close 29403 (time 1734259425.19 total 148.72 last 185.85) - open/close 30000 (time 1734259428.20 total 151.73 last 198.47) - open/close 31897 (time 1734259438.20 total 161.73 last 189.69) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 165.56 seconds: 197.07 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3724 (time 1734259453.38 total 10.00 last 372.39) - /unlink 7172 (time 1734259463.38 total 20.00 last 344.77) - /unlink 10000 (time 1734259471.14 total 27.76 last 364.30) - /unlink 13927 (time 1734259481.14 total 37.77 last 392.64) - /unlink 18198 (time 1734259491.14 total 47.77 last 427.04) - /unlink 20000 (time 1734259495.86 total 52.48 last 382.31) - /unlink 23531 (time 1734259505.86 total 62.48 last 352.97) - /unlink 27436 (time 1734259515.86 total 72.48 last 390.46) - /unlink 30000 (time 1734259522.00 total 78.62 last 417.93) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 84.07 seconds: 388.07 ops/second Waiting for MDT destroys to complete oleg311-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 oleg311-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 - open/close 2123 (time 1734259547.41 total 10.00 last 212.27) - open/close 4312 (time 1734259557.41 total 20.00 last 218.89) - open/close 6519 (time 1734259567.41 total 30.01 last 220.61) - open/close 8432 (time 1734259577.41 total 40.01 last 191.23) - open/close 10000 (time 1734259583.99 total 46.59 last 238.36) - open/close 12360 (time 1734259594.00 total 56.59 last 235.91) - open/close 15011 (time 1734259604.00 total 66.59 last 265.04) - open/close 17868 (time 1734259614.00 total 76.60 last 285.66) - open/close 20000 (time 1734259621.78 total 84.37 last 274.12) - open/close 22565 (time 1734259631.78 total 94.37 last 256.47) - open/close 25199 (time 1734259641.78 total 104.38 last 263.37) - open/close 27821 (time 1734259651.78 total 114.38 last 262.14) - open/close 30000 (time 1734259660.45 total 123.04 last 251.42) - open/close 32531 (time 1734259670.45 total 133.04 last 253.10) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 133.42 seconds: 244.54 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4809 (time 1734259682.02 total 10.00 last 480.88) - /unlink 9783 (time 1734259692.02 total 20.00 last 497.37) - /unlink 15337 (time 1734259702.02 total 30.00 last 555.31) - /unlink 20000 (time 1734259710.32 total 38.30 last 562.32) - /unlink 24974 (time 1734259720.32 total 48.30 last 497.38) - /unlink 29663 (time 1734259730.32 total 58.30 last 468.89) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 65.31 seconds: 499.54 ops/second Waiting for MDT destroys to complete oleg311-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 oleg311-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 - open/close 2126 (time 1734259754.72 total 10.01 last 212.48) - open/close 4377 (time 1734259764.73 total 20.01 last 225.00) - open/close 6955 (time 1734259774.73 total 30.01 last 257.79) - open/close 9646 (time 1734259784.73 total 40.01 last 269.03) - open/close 12373 (time 1734259794.73 total 50.02 last 272.63) - open/close 14535 (time 1734259804.74 total 60.02 last 216.15) - open/close 16874 (time 1734259814.74 total 70.02 last 233.83) - open/close 19473 (time 1734259824.74 total 80.02 last 259.87) - open/close 22349 (time 1734259834.74 total 90.02 last 287.56) - open/close 25194 (time 1734259844.74 total 100.02 last 284.45) - open/close 28144 (time 1734259854.74 total 110.03 last 294.95) - open/close 30000 (time 1734259861.17 total 116.45 last 288.84) - open/close 32519 (time 1734259871.17 total 126.45 last 251.87) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 126.92 seconds: 257.06 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4534 (time 1734259882.72 total 10.00 last 453.40) - /unlink 8831 (time 1734259892.72 total 20.00 last 429.61) - /unlink 10000 (time 1734259895.72 total 23.00 last 390.03) - /unlink 14168 (time 1734259905.72 total 33.00 last 416.78) - /unlink 19094 (time 1734259915.72 total 43.00 last 492.59) - /unlink 24452 (time 1734259925.72 total 53.00 last 535.77) - /unlink 29643 (time 1734259935.72 total 63.00 last 519.08) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 69.07 seconds: 472.39 ops/second Waiting for MDT destroys to complete oleg311-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg311-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server unloading modules via unload_modules_local on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2