-----============= acceptance-small: conf-sanity ============----- Fri Jul 19 12:37:05 EDT 2024 excepting tests: 32newtarball 110 Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg131-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg131-server oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg131-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg131-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:38:06 (1721407086) Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg131-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock oleg131-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5647 (time 1721407121.52 total 10.00 last 564.60) - open/close 10000 (time 1721407129.35 total 17.83 last 556.36) - open/close 15507 (time 1721407139.35 total 27.83 last 550.70) - open/close 20000 (time 1721407147.46 total 35.93 last 554.09) - open/close 25406 (time 1721407157.46 total 45.93 last 540.58) - open/close 30000 (time 1721407166.00 total 54.47 last 538.08) - open/close 35530 (time 1721407176.00 total 64.47 last 552.99) - open/close 40000 (time 1721407184.11 total 72.58 last 551.06) - open/close 45150 (time 1721407194.11 total 82.58 last 514.98) - open/close 50000 (time 1721407203.38 total 91.85 last 523.34) - open/close 55189 (time 1721407213.38 total 101.85 last 518.86) - open/close 60000 (time 1721407223.34 total 111.81 last 483.07) total: 64768 open/close in 120.88 seconds: 535.79 ops/second - /unlink 9162 (time 1721407243.24 total 10.00 last 916.14) - /unlink 18423 (time 1721407253.24 total 20.00 last 926.02) - /unlink 27822 (time 1721407263.24 total 30.00 last 939.84) - /unlink 30000 (time 1721407265.53 total 32.29 last 954.05) - /unlink 39299 (time 1721407275.53 total 42.29 last 929.80) - /unlink 48717 (time 1721407285.53 total 52.29 last 941.71) - /unlink 58170 (time 1721407295.53 total 62.29 last 945.23) total: 64768 /unlink in 69.23 seconds: 935.53 ops/second - open/close 5476 (time 1721407313.30 total 10.00 last 547.51) - open/close 10000 (time 1721407321.70 total 18.40 last 538.72) - open/close 15391 (time 1721407331.70 total 28.40 last 539.09) - open/close 20000 (time 1721407340.60 total 37.30 last 518.00) - open/close 25309 (time 1721407350.60 total 47.30 last 530.87) - open/close 30000 (time 1721407359.47 total 56.17 last 528.66) - open/close 35323 (time 1721407369.47 total 66.17 last 532.28) - open/close 40000 (time 1721407378.30 total 75.00 last 529.96) - open/close 45462 (time 1721407388.30 total 85.00 last 546.16) - open/close 50000 (time 1721407396.77 total 93.47 last 535.45) - open/close 55350 (time 1721407406.78 total 103.47 last 534.96) - open/close 60000 (time 1721407415.45 total 112.15 last 535.95) total: 64768 open/close in 120.81 seconds: 536.13 ops/second - /unlink 9313 (time 1721407434.92 total 10.00 last 931.30) - /unlink 18630 (time 1721407444.92 total 20.00 last 931.67) - /unlink 27723 (time 1721407454.92 total 30.00 last 909.21) - /unlink 30000 (time 1721407457.37 total 32.45 last 928.90) - /unlink 39436 (time 1721407467.37 total 42.45 last 943.57) - /unlink 48719 (time 1721407477.37 total 52.45 last 928.30) - /unlink 58117 (time 1721407487.37 total 62.45 last 939.80) total: 64768 /unlink in 69.26 seconds: 935.16 ops/second - open/close 5534 (time 1721407505.04 total 10.00 last 553.36) - open/close 10000 (time 1721407512.65 total 17.61 last 586.65) - open/close 15703 (time 1721407522.66 total 27.61 last 570.22) - open/close 20000 (time 1721407530.50 total 35.46 last 548.05) - open/close 25581 (time 1721407540.50 total 45.46 last 558.01) - open/close 30000 (time 1721407548.74 total 53.70 last 536.33) - open/close 35561 (time 1721407558.74 total 63.70 last 556.04) - open/close 40000 (time 1721407566.78 total 71.74 last 551.95) - open/close 43661 (time 1721407576.78 total 81.74 last 366.05) - open/close 47485 (time 1721407586.78 total 91.74 last 382.30) - open/close 50000 (time 1721407593.38 total 98.34 last 381.14) - open/close 55156 (time 1721407603.38 total 108.34 last 515.58) - open/close 60000 (time 1721407613.11 total 118.06 last 498.26) total: 64768 open/close in 127.49 seconds: 508.03 ops/second - /unlink 7310 (time 1721407633.36 total 10.00 last 730.89) - /unlink 10000 (time 1721407637.47 total 14.12 last 653.59) - /unlink 16825 (time 1721407647.47 total 24.12 last 682.44) - /unlink 20000 (time 1721407651.00 total 27.64 last 900.33) - /unlink 28434 (time 1721407661.00 total 37.64 last 843.39) - /unlink 30000 (time 1721407663.75 total 40.39 last 570.05) - /unlink 37039 (time 1721407673.75 total 50.39 last 703.76) - /unlink 40000 (time 1721407676.97 total 53.62 last 917.82) - /unlink 47860 (time 1721407686.97 total 63.62 last 785.96) - /unlink 50000 (time 1721407689.78 total 66.43 last 762.71) - /unlink 58332 (time 1721407699.78 total 76.43 last 833.17) total: 64768 /unlink in 86.16 seconds: 751.68 ops/second - open/close 3438 (time 1721407720.57 total 10.00 last 343.77) - open/close 8129 (time 1721407730.57 total 20.00 last 469.08) - open/close 10000 (time 1721407734.67 total 24.09 last 457.34) - open/close 14944 (time 1721407744.67 total 34.09 last 494.37) - open/close 19785 (time 1721407754.67 total 44.09 last 484.10) - open/close 23172 (time 1721407764.67 total 54.09 last 338.70) - open/close 26839 (time 1721407774.72 total 64.15 last 364.58) - open/close 30000 (time 1721407782.28 total 71.70 last 418.65) - open/close 34513 (time 1721407792.28 total 81.70 last 451.26) - open/close 39714 (time 1721407802.28 total 91.70 last 520.10) - open/close 44940 (time 1721407812.28 total 101.70 last 522.53) - open/close 50000 (time 1721407822.14 total 111.56 last 513.28) - open/close 55213 (time 1721407832.14 total 121.56 last 521.24) - open/close 60000 (time 1721407841.31 total 130.73 last 521.96) total: 64768 open/close in 139.53 seconds: 464.20 ops/second - /unlink 9342 (time 1721407860.99 total 10.00 last 934.17) - /unlink 18680 (time 1721407870.99 total 20.00 last 933.74) - /unlink 28055 (time 1721407881.00 total 30.00 last 937.43) - /unlink 37545 (time 1721407891.00 total 40.00 last 948.99) - /unlink 40000 (time 1721407893.50 total 42.50 last 982.09) - /unlink 49624 (time 1721407903.50 total 52.50 last 962.31) - /unlink 59363 (time 1721407913.50 total 62.50 last 973.81) total: 64768 /unlink in 67.92 seconds: 953.57 ops/second - open/close 5649 (time 1721407929.76 total 10.00 last 564.80) - open/close 10000 (time 1721407937.48 total 17.73 last 563.31) - open/close 15597 (time 1721407947.48 total 27.73 last 559.61) - open/close 20000 (time 1721407955.71 total 35.95 last 535.19) - open/close 25170 (time 1721407965.71 total 45.96 last 516.95) - open/close 28837 (time 1721407975.71 total 55.96 last 366.65) - open/close 30000 (time 1721407978.59 total 58.83 last 404.22) - open/close 34072 (time 1721407988.59 total 68.84 last 407.12) - open/close 37252 (time 1721407998.60 total 78.84 last 317.90) - open/close 38745 (time 1721408008.60 total 88.84 last 149.27) - open/close 40000 (time 1721408015.89 total 96.13 last 172.11) - open/close 41706 (time 1721408025.89 total 106.14 last 170.54) - open/close 43555 (time 1721408035.90 total 116.14 last 184.78) - open/close 45138 (time 1721408045.90 total 126.14 last 158.28) - open/close 46277 (time 1721408055.90 total 136.14 last 113.89) - open/close 47594 (time 1721408065.90 total 146.15 last 131.66) - open/close 48976 (time 1721408075.91 total 156.15 last 138.18) - open/close 50000 (time 1721408082.34 total 162.59 last 159.04) - open/close 51469 (time 1721408092.35 total 172.59 last 146.81) - open/close 52633 (time 1721408102.36 total 182.60 last 116.32) - open/close 53849 (time 1721408112.36 total 192.60 last 121.58) - open/close 55045 (time 1721408122.36 total 202.60 last 119.58) - open/close 56432 (time 1721408132.36 total 212.61 last 138.65) - open/close 57828 (time 1721408142.37 total 222.62 last 139.48) - open/close 59305 (time 1721408152.37 total 232.62 last 147.67) - open/close 60000 (time 1721408155.64 total 235.88 last 212.77) - open/close 61463 (time 1721408165.67 total 245.91 last 145.95) - open/close 62754 (time 1721408175.67 total 255.91 last 129.05) - open/close 63920 (time 1721408185.67 total 265.91 last 116.57) total: 64768 open/close in 273.04 seconds: 237.21 ops/second - /unlink 2549 (time 1721408204.91 total 10.00 last 254.86) - /unlink 9285 (time 1721408214.91 total 20.01 last 673.20) - /unlink 17734 (time 1721408224.92 total 30.01 last 844.38) - /unlink 20000 (time 1721408228.69 total 33.79 last 600.23) - /unlink 25077 (time 1721408238.69 total 43.79 last 507.63) - /unlink 28186 (time 1721408248.70 total 53.79 last 310.86) - /unlink 30000 (time 1721408257.32 total 62.41 last 210.41) - /unlink 31999 (time 1721408267.32 total 72.41 last 199.86) - /unlink 34194 (time 1721408277.32 total 82.42 last 219.44) - /unlink 36206 (time 1721408287.33 total 92.42 last 201.08) - /unlink 37925 (time 1721408297.33 total 102.43 last 171.85) - /unlink 39435 (time 1721408307.33 total 112.43 last 150.97) - /unlink 40000 (time 1721408310.97 total 116.07 last 155.16) - /unlink 41926 (time 1721408320.98 total 126.07 last 192.55) - /unlink 43910 (time 1721408330.99 total 136.08 last 198.15) - /unlink 45857 (time 1721408340.99 total 146.09 last 194.64) - /unlink 48029 (time 1721408350.99 total 156.09 last 217.19) - /unlink 50000 (time 1721408359.91 total 165.00 last 221.07) - /unlink 52140 (time 1721408369.92 total 175.01 last 213.80) - /unlink 54316 (time 1721408379.92 total 185.02 last 217.52) - /unlink 56651 (time 1721408389.93 total 195.03 last 233.21) - /unlink 59481 (time 1721408399.94 total 205.03 last 282.94) - /unlink 60000 (time 1721408401.95 total 207.04 last 258.36) - /unlink 63509 (time 1721408411.95 total 217.04 last 350.86) total: 64768 /unlink in 220.13 seconds: 294.23 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server LNET ready to unload unloading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local oleg131-server: LNET ready to unload modules unloaded. pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 PASS 106 (1361s) Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1443 sec ========== 13:01:08 (1721408468)