-----============= acceptance-small: conf-sanity ============----- Wed Jan 15 18:14:23 EST 2025 excepting tests: 32newtarball 110 Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg131-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg131-server oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg131-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg131-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:15:41 (1736982941) Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg131-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock oleg131-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16169 329 15840 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 3077 (time 1736982987.94 total 10.00 last 307.66) - open/close 6138 (time 1736982997.94 total 20.00 last 306.04) - open/close 8693 (time 1736983007.94 total 30.00 last 255.46) - open/close 10000 (time 1736983012.95 total 35.01 last 260.93) - open/close 12698 (time 1736983022.95 total 45.02 last 269.75) - open/close 15181 (time 1736983032.95 total 55.02 last 248.23) - open/close 17365 (time 1736983042.96 total 65.02 last 218.26) - open/close 19998 (time 1736983052.96 total 75.03 last 263.23) - open/close 22039 (time 1736983062.97 total 85.04 last 203.84) - open/close 23453 (time 1736983072.98 total 95.04 last 141.37) - open/close 24724 (time 1736983082.98 total 105.05 last 127.06) - open/close 26455 (time 1736983092.99 total 115.05 last 172.97) - open/close 28045 (time 1736983102.99 total 125.05 last 158.97) - open/close 29584 (time 1736983112.99 total 135.06 last 153.85) - open/close 30000 (time 1736983115.45 total 137.52 last 169.05) - open/close 31578 (time 1736983125.46 total 147.52 last 157.74) - open/close 33422 (time 1736983135.46 total 157.53 last 184.33) - open/close 35705 (time 1736983145.46 total 167.53 last 228.26) - open/close 37814 (time 1736983155.46 total 177.53 last 210.87) - open/close 39380 (time 1736983165.47 total 187.54 last 156.48) - open/close 40000 (time 1736983169.48 total 191.55 last 154.55) - open/close 41409 (time 1736983179.48 total 201.55 last 140.89) - open/close 43143 (time 1736983189.49 total 211.55 last 173.37) - open/close 45010 (time 1736983199.49 total 221.55 last 186.67) - open/close 46995 (time 1736983209.49 total 231.55 last 198.47) - open/close 48791 (time 1736983219.49 total 241.56 last 179.57) - open/close 50000 (time 1736983226.05 total 248.11 last 184.35) - open/close 51431 (time 1736983236.05 total 258.11 last 143.09) - open/close 53243 (time 1736983246.05 total 268.12 last 181.17) - open/close 55288 (time 1736983256.05 total 278.12 last 204.45) - open/close 57336 (time 1736983266.05 total 288.12 last 204.76) - open/close 59794 (time 1736983276.06 total 298.12 last 245.72) - open/close 62307 (time 1736983286.06 total 308.13 last 251.20) - open/close 64152 (time 1736983296.06 total 318.13 last 184.48) total: 64768 open/close in 321.02 seconds: 201.76 ops/second - /unlink 3963 (time 1736983310.23 total 10.00 last 396.26) - /unlink 7450 (time 1736983320.23 total 20.00 last 348.65) - /unlink 10000 (time 1736983328.02 total 27.79 last 327.45) - /unlink 13208 (time 1736983338.02 total 37.79 last 320.77) - /unlink 17316 (time 1736983348.02 total 47.79 last 410.78) - /unlink 20000 (time 1736983354.42 total 54.19 last 419.29) - /unlink 24930 (time 1736983364.42 total 64.19 last 492.99) - /unlink 29818 (time 1736983374.42 total 74.19 last 488.74) - /unlink 33139 (time 1736983384.43 total 84.20 last 332.04) - /unlink 37671 (time 1736983394.43 total 94.20 last 453.16) - /unlink 40000 (time 1736983399.83 total 99.60 last 431.33) - /unlink 44480 (time 1736983409.83 total 109.60 last 447.96) - /unlink 49511 (time 1736983419.83 total 119.60 last 503.06) - /unlink 54360 (time 1736983429.83 total 129.60 last 484.87) - /unlink 58893 (time 1736983439.83 total 139.60 last 453.27) - /unlink 60000 (time 1736983442.19 total 141.96 last 468.76) - /unlink 64534 (time 1736983452.19 total 151.96 last 453.38) total: 64768 /unlink in 152.58 seconds: 424.48 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2421 (time 1736983480.99 total 10.00 last 242.09) - open/close 4995 (time 1736983491.00 total 20.00 last 257.36) - open/close 7713 (time 1736983501.00 total 30.01 last 271.69) - open/close 10000 (time 1736983509.99 total 39.00 last 254.25) - open/close 12754 (time 1736983520.00 total 49.00 last 275.32) - open/close 15722 (time 1736983530.00 total 59.01 last 296.77) - open/close 18334 (time 1736983540.00 total 69.01 last 261.13) - open/close 20000 (time 1736983545.16 total 74.17 last 322.92) - open/close 23624 (time 1736983555.20 total 84.20 last 361.13) - open/close 26589 (time 1736983565.20 total 94.20 last 296.50) - open/close 29351 (time 1736983575.20 total 104.20 last 276.15) - open/close 30000 (time 1736983577.46 total 106.46 last 287.37) - open/close 32681 (time 1736983587.46 total 116.46 last 268.08) - open/close 34995 (time 1736983597.46 total 126.47 last 231.33) - open/close 37362 (time 1736983607.46 total 136.47 last 236.61) - open/close 39871 (time 1736983617.46 total 146.47 last 250.90) - open/close 42539 (time 1736983627.46 total 156.47 last 266.79) - open/close 45693 (time 1736983637.46 total 166.47 last 315.38) - open/close 48374 (time 1736983647.47 total 176.48 last 267.98) - open/close 50000 (time 1736983652.77 total 181.77 last 306.90) - open/close 53217 (time 1736983662.84 total 191.85 last 319.29) - open/close 56305 (time 1736983672.84 total 201.85 last 308.73) - open/close 58858 (time 1736983682.85 total 211.85 last 255.21) - open/close 60000 (time 1736983687.11 total 216.11 last 268.25) - open/close 63625 (time 1736983697.11 total 226.12 last 362.36) total: 64768 open/close in 228.77 seconds: 283.12 ops/second - /unlink 8278 (time 1736983710.59 total 10.00 last 827.75) - /unlink 10000 (time 1736983712.87 total 12.28 last 756.86) - /unlink 17576 (time 1736983722.87 total 22.28 last 757.53) - /unlink 20000 (time 1736983726.75 total 26.16 last 623.86) - /unlink 27172 (time 1736983736.75 total 36.16 last 717.17) - /unlink 30000 (time 1736983741.99 total 41.40 last 539.96) - /unlink 36840 (time 1736983751.99 total 51.40 last 683.96) - /unlink 40000 (time 1736983756.10 total 55.51 last 769.56) - /unlink 47145 (time 1736983766.10 total 65.51 last 714.46) - /unlink 50000 (time 1736983770.34 total 69.75 last 672.92) - /unlink 57301 (time 1736983780.34 total 79.75 last 730.06) - /unlink 60000 (time 1736983784.78 total 84.19 last 608.13) total: 64768 /unlink in 90.36 seconds: 716.80 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4087 (time 1736983822.48 total 10.00 last 408.60) - open/close 7960 (time 1736983832.48 total 20.00 last 387.26) - open/close 10000 (time 1736983836.59 total 24.11 last 496.40) - open/close 14939 (time 1736983846.59 total 34.11 last 493.87) - open/close 19997 (time 1736983856.60 total 44.12 last 505.65) - open/close 24137 (time 1736983866.60 total 54.12 last 413.92) - open/close 28638 (time 1736983876.60 total 64.12 last 450.08) - open/close 30000 (time 1736983879.35 total 66.87 last 495.67) - open/close 34951 (time 1736983889.35 total 76.87 last 495.01) - open/close 39664 (time 1736983899.35 total 86.87 last 471.26) - open/close 44372 (time 1736983909.35 total 96.87 last 470.73) - open/close 48825 (time 1736983919.35 total 106.87 last 445.30) - open/close 50000 (time 1736983922.45 total 109.97 last 379.19) - open/close 54400 (time 1736983932.45 total 119.97 last 439.93) - open/close 58936 (time 1736983942.45 total 129.97 last 453.60) - open/close 60000 (time 1736983944.66 total 132.18 last 481.89) - open/close 64429 (time 1736983954.66 total 142.18 last 442.84) total: 64768 open/close in 142.96 seconds: 453.05 ops/second - /unlink 8062 (time 1736983966.29 total 10.00 last 806.12) - /unlink 10000 (time 1736983969.04 total 12.76 last 703.11) - /unlink 18878 (time 1736983979.05 total 22.76 last 887.71) - /unlink 28635 (time 1736983989.05 total 32.76 last 975.66) - /unlink 38007 (time 1736983999.05 total 42.76 last 937.17) - /unlink 40000 (time 1736984001.32 total 45.03 last 877.59) - /unlink 48765 (time 1736984011.32 total 55.03 last 876.44) - /unlink 57162 (time 1736984021.32 total 65.03 last 839.70) - /unlink 60000 (time 1736984024.98 total 68.70 last 774.31) total: 64768 /unlink in 74.04 seconds: 874.75 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5256 (time 1736984067.27 total 10.00 last 525.57) - open/close 10000 (time 1736984076.65 total 19.38 last 505.68) - open/close 15315 (time 1736984086.65 total 29.38 last 531.46) - open/close 20000 (time 1736984094.64 total 37.37 last 586.50) - open/close 25878 (time 1736984104.64 total 47.37 last 587.75) - open/close 30000 (time 1736984113.36 total 56.09 last 472.73) - open/close 34014 (time 1736984123.36 total 66.09 last 401.38) - open/close 37193 (time 1736984133.36 total 76.09 last 317.88) - open/close 40000 (time 1736984138.72 total 81.45 last 524.25) - open/close 44958 (time 1736984148.72 total 91.45 last 495.79) - open/close 49038 (time 1736984158.72 total 101.45 last 407.90) - open/close 50000 (time 1736984161.34 total 104.07 last 366.99) - open/close 54639 (time 1736984171.34 total 114.07 last 463.89) - open/close 60000 (time 1736984181.25 total 123.98 last 541.01) total: 64768 open/close in 133.83 seconds: 483.94 ops/second - /unlink 9937 (time 1736984201.90 total 10.00 last 993.68) - /unlink 19480 (time 1736984211.90 total 20.00 last 954.28) - /unlink 29654 (time 1736984221.90 total 30.00 last 1017.37) - /unlink 38580 (time 1736984231.90 total 40.00 last 892.51) - /unlink 47989 (time 1736984241.90 total 50.00 last 940.84) - /unlink 50000 (time 1736984244.17 total 52.27 last 885.61) - /unlink 59240 (time 1736984254.17 total 62.27 last 923.99) total: 64768 /unlink in 68.05 seconds: 951.83 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5005 (time 1736984298.15 total 10.00 last 500.49) - open/close 9809 (time 1736984308.15 total 20.00 last 480.26) - open/close 14177 (time 1736984318.15 total 30.00 last 436.74) - open/close 18534 (time 1736984328.16 total 40.01 last 435.62) - open/close 20000 (time 1736984331.00 total 42.86 last 514.56) - open/close 25017 (time 1736984341.00 total 52.86 last 501.70) - open/close 28326 (time 1736984351.01 total 62.86 last 330.89) - open/close 30000 (time 1736984355.26 total 67.11 last 393.60) - open/close 34265 (time 1736984365.26 total 77.11 last 426.39) - open/close 38503 (time 1736984375.26 total 87.11 last 423.74) - open/close 40000 (time 1736984378.13 total 89.98 last 522.39) - open/close 44231 (time 1736984388.13 total 99.98 last 423.05) - open/close 48909 (time 1736984398.13 total 109.98 last 467.77) - open/close 50000 (time 1736984400.32 total 112.17 last 498.50) - open/close 54629 (time 1736984410.32 total 122.17 last 462.84) - open/close 58338 (time 1736984420.32 total 132.17 last 370.87) - open/close 60000 (time 1736984424.07 total 135.92 last 442.71) - open/close 64376 (time 1736984434.08 total 145.93 last 437.57) total: 64768 open/close in 146.92 seconds: 440.84 ops/second - /unlink 3635 (time 1736984446.01 total 10.00 last 363.41) - /unlink 7213 (time 1736984456.01 total 20.00 last 357.75) - /unlink 10000 (time 1736984460.58 total 24.57 last 610.04) - /unlink 15157 (time 1736984470.58 total 34.57 last 515.63) - /unlink 20000 (time 1736984480.12 total 44.11 last 507.68) - /unlink 24395 (time 1736984490.12 total 54.11 last 439.47) - /unlink 29031 (time 1736984500.12 total 64.12 last 463.54) - /unlink 34685 (time 1736984510.12 total 74.12 last 565.37) - /unlink 40000 (time 1736984519.62 total 83.62 last 559.40) - /unlink 43983 (time 1736984529.62 total 93.62 last 398.29) - /unlink 48238 (time 1736984539.62 total 103.62 last 425.36) - /unlink 50000 (time 1736984544.01 total 108.01 last 401.49) - /unlink 55782 (time 1736984554.02 total 118.01 last 578.02) - /unlink 60000 (time 1736984562.01 total 126.01 last 527.34) total: 64768 /unlink in 134.11 seconds: 482.96 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server LNET ready to unload unloading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local oleg131-server: LNET ready to unload modules unloaded. pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 PASS 106 (1667s) Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1765 sec ========== 18:43:48 (1736984628)