-----============= acceptance-small: conf-sanity ============----- Wed Jan 15 18:47:34 EST 2025 excepting tests: 32newtarball 110 Stopping clients: oleg209-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg209-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg209-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg209-server oleg209-server: oleg209-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: detected 4 online CPUs by sysfs oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg209-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg209-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg209-server: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg209-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:49:11 (1736984951) Stopping clients: oleg209-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg209-client.virtnet /mnt/lustre2 (opts:-f) oleg209-server: oleg209-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: detected 4 online CPUs by sysfs oleg209-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg209-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg209-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock oleg209-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1417 (time 1736985000.11 total 10.00 last 141.67) - open/close 2970 (time 1736985010.11 total 20.00 last 155.27) - open/close 4208 (time 1736985020.12 total 30.01 last 123.72) - open/close 5431 (time 1736985030.12 total 40.01 last 122.28) - open/close 6747 (time 1736985040.12 total 50.01 last 131.56) - open/close 8010 (time 1736985050.12 total 60.02 last 126.29) - open/close 9269 (time 1736985060.13 total 70.02 last 125.86) - open/close 10000 (time 1736985066.35 total 76.24 last 117.46) - open/close 11364 (time 1736985076.35 total 86.25 last 136.35) - open/close 12793 (time 1736985086.36 total 96.25 last 142.86) - open/close 14173 (time 1736985096.36 total 106.25 last 137.99) - open/close 15675 (time 1736985106.36 total 116.25 last 150.12) - open/close 17081 (time 1736985116.37 total 126.26 last 140.57) - open/close 18408 (time 1736985126.37 total 136.26 last 132.64) - open/close 19983 (time 1736985136.37 total 146.26 last 157.49) - open/close 21430 (time 1736985146.37 total 156.27 last 144.65) - open/close 22844 (time 1736985156.37 total 166.27 last 141.39) - open/close 24445 (time 1736985166.38 total 176.27 last 160.06) - open/close 25852 (time 1736985176.38 total 186.27 last 140.68) - open/close 27262 (time 1736985186.38 total 196.27 last 140.98) - open/close 28628 (time 1736985196.39 total 206.28 last 136.51) - open/close 29923 (time 1736985206.39 total 216.29 last 129.39) - open/close 31345 (time 1736985216.40 total 226.29 last 142.16) - open/close 32676 (time 1736985226.40 total 236.29 last 133.05) - open/close 34116 (time 1736985236.41 total 246.30 last 143.92) - open/close 35314 (time 1736985246.41 total 256.30 last 119.79) - open/close 36640 (time 1736985256.42 total 266.31 last 132.51) - open/close 38147 (time 1736985266.42 total 276.31 last 150.67) - open/close 39868 (time 1736985276.42 total 286.31 last 172.06) - open/close 41396 (time 1736985286.42 total 296.31 last 152.77) - open/close 43047 (time 1736985296.43 total 306.32 last 165.02) - open/close 44469 (time 1736985306.43 total 316.32 last 142.14) - open/close 46570 (time 1736985316.43 total 326.32 last 210.03) - open/close 48198 (time 1736985326.43 total 336.33 last 162.79) - open/close 49717 (time 1736985336.44 total 346.33 last 151.83) - open/close 51420 (time 1736985346.44 total 356.34 last 170.20) - open/close 53139 (time 1736985356.45 total 366.34 last 171.85) - open/close 54796 (time 1736985366.45 total 376.34 last 165.61) - open/close 56601 (time 1736985376.45 total 386.35 last 180.47) - open/close 58490 (time 1736985386.45 total 396.35 last 188.89) - open/close 60000 (time 1736985394.78 total 404.68 last 181.28) - open/close 61760 (time 1736985404.79 total 414.68 last 175.95) - open/close 63677 (time 1736985414.79 total 424.69 last 191.56) total: 64768 open/close in 431.57 seconds: 150.08 ops/second - /unlink 2910 (time 1736985433.07 total 10.00 last 290.94) - /unlink 5643 (time 1736985443.08 total 20.00 last 273.28) - /unlink 9319 (time 1736985453.08 total 30.00 last 367.57) - /unlink 10000 (time 1736985455.15 total 32.08 last 328.46) - /unlink 13224 (time 1736985465.15 total 42.08 last 322.40) - /unlink 16678 (time 1736985475.15 total 52.08 last 345.32) - /unlink 19685 (time 1736985485.16 total 62.08 last 300.59) - /unlink 23085 (time 1736985495.16 total 72.08 last 339.95) - /unlink 27078 (time 1736985505.16 total 82.09 last 399.20) - /unlink 30000 (time 1736985512.22 total 89.14 last 414.08) - /unlink 33257 (time 1736985522.22 total 99.14 last 325.69) - /unlink 36432 (time 1736985532.22 total 109.15 last 317.45) - /unlink 39636 (time 1736985542.22 total 119.15 last 320.28) - /unlink 43625 (time 1736985552.22 total 129.15 last 398.90) - /unlink 46802 (time 1736985562.22 total 139.15 last 317.66) - /unlink 50000 (time 1736985571.18 total 148.11 last 357.13) - /unlink 53367 (time 1736985581.18 total 158.11 last 336.58) - /unlink 56948 (time 1736985591.18 total 168.11 last 358.04) - /unlink 60000 (time 1736985597.09 total 174.02 last 516.80) total: 64768 /unlink in 182.30 seconds: 355.29 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2191 (time 1736985638.51 total 10.00 last 219.03) - open/close 4631 (time 1736985648.51 total 20.01 last 243.93) - open/close 6909 (time 1736985658.51 total 30.01 last 227.75) - open/close 9234 (time 1736985668.51 total 40.01 last 232.45) - open/close 10000 (time 1736985671.35 total 42.84 last 270.28) - open/close 12326 (time 1736985681.35 total 52.85 last 232.55) - open/close 15420 (time 1736985691.35 total 62.85 last 309.39) - open/close 18210 (time 1736985701.35 total 72.85 last 278.98) - open/close 20000 (time 1736985707.83 total 79.33 last 276.19) - open/close 22445 (time 1736985717.83 total 89.33 last 244.47) - open/close 24676 (time 1736985727.84 total 99.33 last 223.05) - open/close 27331 (time 1736985737.84 total 109.33 last 265.46) - open/close 29801 (time 1736985747.84 total 119.33 last 246.97) - open/close 32417 (time 1736985757.84 total 129.34 last 261.55) - open/close 35835 (time 1736985767.84 total 139.34 last 341.78) - open/close 38867 (time 1736985777.84 total 149.34 last 303.11) - open/close 40000 (time 1736985781.66 total 153.16 last 296.61) - open/close 42301 (time 1736985791.67 total 163.16 last 230.06) - open/close 45037 (time 1736985801.67 total 173.16 last 273.56) - open/close 47463 (time 1736985811.67 total 183.16 last 242.59) - open/close 50000 (time 1736985819.82 total 191.31 last 311.28) - open/close 53019 (time 1736985829.82 total 201.32 last 301.83) - open/close 56055 (time 1736985839.82 total 211.32 last 303.55) - open/close 58894 (time 1736985849.82 total 221.32 last 283.82) - open/close 60000 (time 1736985854.17 total 225.66 last 254.75) - open/close 63187 (time 1736985864.17 total 235.67 last 318.60) total: 64768 open/close in 240.87 seconds: 268.89 ops/second - /unlink 5904 (time 1736985880.34 total 10.00 last 590.25) - /unlink 10000 (time 1736985888.36 total 18.01 last 511.28) - /unlink 18077 (time 1736985898.36 total 28.01 last 807.65) - /unlink 20000 (time 1736985901.43 total 31.09 last 625.80) - /unlink 26289 (time 1736985911.43 total 41.09 last 628.79) - /unlink 30000 (time 1736985916.47 total 46.13 last 735.99) - /unlink 36396 (time 1736985926.47 total 56.13 last 639.55) - /unlink 40000 (time 1736985931.85 total 61.50 last 670.95) - /unlink 48037 (time 1736985941.85 total 71.50 last 803.65) - /unlink 50000 (time 1736985944.39 total 74.05 last 771.24) - /unlink 58223 (time 1736985954.39 total 84.05 last 822.25) - /unlink 60000 (time 1736985956.66 total 86.32 last 782.84) total: 64768 /unlink in 91.90 seconds: 704.73 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4891 (time 1736985994.07 total 10.00 last 489.07) - open/close 9416 (time 1736986004.08 total 20.00 last 452.42) - open/close 13440 (time 1736986014.08 total 30.00 last 402.37) - open/close 17115 (time 1736986024.08 total 40.00 last 367.44) - open/close 20000 (time 1736986030.44 total 46.37 last 453.56) - open/close 25041 (time 1736986040.44 total 56.37 last 504.06) - open/close 28908 (time 1736986050.44 total 66.37 last 386.68) - open/close 30000 (time 1736986053.19 total 69.11 last 397.59) - open/close 34040 (time 1736986063.19 total 79.11 last 403.97) - open/close 38658 (time 1736986073.19 total 89.12 last 461.72) - open/close 40000 (time 1736986076.10 total 92.03 last 460.93) - open/close 45142 (time 1736986086.10 total 102.03 last 514.15) - open/close 50000 (time 1736986095.91 total 111.83 last 495.51) - open/close 54212 (time 1736986105.91 total 121.83 last 421.18) - open/close 58744 (time 1736986115.91 total 131.83 last 453.12) - open/close 60000 (time 1736986118.46 total 134.38 last 492.88) - open/close 64604 (time 1736986128.46 total 144.38 last 460.32) total: 64768 open/close in 144.79 seconds: 447.34 ops/second - /unlink 6692 (time 1736986139.74 total 10.00 last 669.20) - /unlink 10000 (time 1736986144.28 total 14.54 last 728.21) - /unlink 19226 (time 1736986154.28 total 24.54 last 922.58) - /unlink 28170 (time 1736986164.28 total 34.54 last 894.25) - /unlink 38250 (time 1736986174.28 total 44.55 last 1007.92) - /unlink 40000 (time 1736986176.51 total 46.78 last 784.79) - /unlink 48374 (time 1736986186.51 total 56.78 last 837.37) - /unlink 56732 (time 1736986196.52 total 66.78 last 835.74) - /unlink 60000 (time 1736986200.50 total 70.76 last 821.17) total: 64768 /unlink in 76.22 seconds: 849.70 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4691 (time 1736986244.34 total 10.00 last 468.99) - open/close 9120 (time 1736986254.34 total 20.00 last 442.88) - open/close 13955 (time 1736986264.34 total 30.00 last 483.48) - open/close 18889 (time 1736986274.34 total 40.00 last 493.34) - open/close 23882 (time 1736986284.35 total 50.02 last 498.75) - open/close 28639 (time 1736986294.35 total 60.02 last 475.60) - open/close 30000 (time 1736986296.91 total 62.58 last 532.17) - open/close 35674 (time 1736986306.91 total 72.58 last 567.32) - open/close 40000 (time 1736986315.11 total 80.77 last 527.74) - open/close 45350 (time 1736986325.11 total 90.77 last 534.94) - open/close 50000 (time 1736986333.54 total 99.21 last 551.52) - open/close 55923 (time 1736986343.54 total 109.21 last 592.24) - open/close 60000 (time 1736986351.43 total 117.09 last 517.09) total: 64768 open/close in 125.60 seconds: 515.67 ops/second - /unlink 9669 (time 1736986370.73 total 10.00 last 966.85) - /unlink 19345 (time 1736986380.73 total 20.00 last 967.59) - /unlink 29201 (time 1736986390.73 total 30.00 last 985.56) - /unlink 39633 (time 1736986400.73 total 40.00 last 1043.16) - /unlink 49343 (time 1736986410.73 total 50.00 last 970.94) - /unlink 58063 (time 1736986420.73 total 60.00 last 871.97) - /unlink 60000 (time 1736986422.95 total 62.22 last 873.79) total: 64768 /unlink in 67.71 seconds: 956.62 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5501 (time 1736986474.06 total 10.00 last 550.04) - open/close 10000 (time 1736986482.19 total 18.13 last 553.41) - open/close 15639 (time 1736986492.19 total 28.13 last 563.80) - open/close 20000 (time 1736986500.74 total 36.69 last 509.83) - open/close 24882 (time 1736986510.74 total 46.69 last 488.19) - open/close 30000 (time 1736986520.72 total 56.66 last 513.02) - open/close 35407 (time 1736986530.72 total 66.66 last 540.66) - open/close 40000 (time 1736986540.47 total 76.42 last 470.93) - open/close 45353 (time 1736986550.47 total 86.42 last 535.27) - open/close 50000 (time 1736986558.21 total 94.16 last 600.35) - open/close 55970 (time 1736986568.21 total 104.16 last 596.98) - open/close 60000 (time 1736986576.02 total 111.96 last 516.23) total: 64768 open/close in 120.69 seconds: 536.67 ops/second - /unlink 9685 (time 1736986595.54 total 10.00 last 968.47) - /unlink 19193 (time 1736986605.54 total 20.00 last 950.79) - /unlink 28426 (time 1736986615.54 total 30.00 last 923.28) - /unlink 37583 (time 1736986625.54 total 40.00 last 915.57) - /unlink 40000 (time 1736986628.44 total 42.91 last 831.58) - /unlink 48402 (time 1736986638.45 total 52.91 last 840.09) - /unlink 57662 (time 1736986648.45 total 62.91 last 925.95) - /unlink 60000 (time 1736986651.00 total 65.47 last 914.81) total: 64768 /unlink in 70.54 seconds: 918.23 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server LNET ready to unload unloading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local oleg209-server: LNET ready to unload modules unloaded. pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 PASS 106 (1761s) Stopping clients: oleg209-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg209-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 oleg209-server: oleg209-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: detected 4 online CPUs by sysfs oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg209-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1876 sec ========== 19:18:50 (1736986730)