-----============= acceptance-small: conf-sanity ============----- Tue Jan 7 09:49:30 EST 2025 excepting tests: 32newtarball 110 Stopping clients: oleg340-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg340-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg340-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg340-server oleg340-server: oleg340-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg340-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg340-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg340-server: oleg340-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg340-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:50:37 (1736261437) Stopping clients: oleg340-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg340-client.virtnet /mnt/lustre2 (opts:-f) oleg340-server: oleg340-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg340-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg340-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16169 329 15840 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5363 (time 1736261471.24 total 10.00 last 536.23) - open/close 10000 (time 1736261479.69 total 18.45 last 548.59) - open/close 15180 (time 1736261489.70 total 28.45 last 517.95) - open/close 18478 (time 1736261499.70 total 38.46 last 329.74) - open/close 20000 (time 1736261504.18 total 42.94 last 339.65) - open/close 25768 (time 1736261514.18 total 52.94 last 576.79) - open/close 30000 (time 1736261521.81 total 60.57 last 554.58) - open/close 34005 (time 1736261531.81 total 70.57 last 400.46) - open/close 38928 (time 1736261541.81 total 80.57 last 492.29) - open/close 40000 (time 1736261543.84 total 82.60 last 527.96) - open/close 45370 (time 1736261553.84 total 92.60 last 536.92) - open/close 50000 (time 1736261563.26 total 102.02 last 491.78) - open/close 54903 (time 1736261573.26 total 112.02 last 490.25) - open/close 58091 (time 1736261583.26 total 122.02 last 318.78) - open/close 60000 (time 1736261589.60 total 128.36 last 301.00) - open/close 63237 (time 1736261599.60 total 138.36 last 323.68) total: 64768 open/close in 142.67 seconds: 453.98 ops/second - /unlink 5695 (time 1736261614.84 total 10.00 last 569.42) - /unlink 10000 (time 1736261623.59 total 18.75 last 492.01) - /unlink 14008 (time 1736261633.59 total 28.75 last 400.72) - /unlink 17947 (time 1736261643.59 total 38.76 last 393.82) - /unlink 20000 (time 1736261647.91 total 43.07 last 476.01) - /unlink 26228 (time 1736261657.91 total 53.07 last 622.72) - /unlink 30000 (time 1736261663.19 total 58.35 last 713.91) - /unlink 37401 (time 1736261673.19 total 68.35 last 740.06) - /unlink 40000 (time 1736261677.04 total 72.21 last 674.48) - /unlink 46903 (time 1736261687.04 total 82.21 last 690.30) - /unlink 50000 (time 1736261690.81 total 85.98 last 821.72) - /unlink 58355 (time 1736261700.81 total 95.98 last 835.43) total: 64768 /unlink in 103.92 seconds: 623.24 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4999 (time 1736261740.99 total 10.00 last 499.87) - open/close 10000 (time 1736261750.95 total 19.95 last 502.41) - open/close 14756 (time 1736261760.95 total 29.96 last 475.52) - open/close 19501 (time 1736261770.95 total 39.96 last 474.50) - open/close 24570 (time 1736261780.95 total 49.96 last 506.82) - open/close 29363 (time 1736261790.95 total 59.96 last 479.26) - open/close 34211 (time 1736261800.95 total 69.96 last 484.72) - open/close 38694 (time 1736261811.00 total 80.01 last 446.22) - open/close 40000 (time 1736261813.71 total 82.72 last 481.29) - open/close 44161 (time 1736261823.71 total 92.72 last 416.04) - open/close 49131 (time 1736261833.71 total 102.72 last 496.93) - open/close 53538 (time 1736261843.72 total 112.72 last 440.66) - open/close 58015 (time 1736261853.72 total 122.73 last 447.65) - open/close 60000 (time 1736261858.81 total 127.82 last 389.92) - open/close 64633 (time 1736261868.86 total 137.87 last 460.64) total: 64768 open/close in 138.12 seconds: 468.93 ops/second - /unlink 10000 (time 1736261879.69 total 9.80 last 1020.70) - /unlink 19130 (time 1736261889.69 total 19.80 last 913.00) - /unlink 28534 (time 1736261899.69 total 29.80 last 940.33) - /unlink 37382 (time 1736261909.69 total 39.80 last 884.76) - /unlink 40000 (time 1736261912.44 total 42.54 last 954.78) - /unlink 49063 (time 1736261922.44 total 52.54 last 906.25) - /unlink 58421 (time 1736261932.44 total 62.54 last 935.74) total: 64768 /unlink in 68.29 seconds: 948.47 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5952 (time 1736261981.22 total 10.00 last 595.11) - open/close 10000 (time 1736261988.04 total 16.82 last 593.43) - open/close 16074 (time 1736261998.04 total 26.82 last 607.36) - open/close 20000 (time 1736262004.66 total 33.44 last 593.13) - open/close 25935 (time 1736262014.66 total 43.44 last 593.45) - open/close 30000 (time 1736262022.00 total 50.78 last 553.74) - open/close 35497 (time 1736262032.00 total 60.79 last 549.67) - open/close 40000 (time 1736262039.76 total 68.54 last 580.30) - open/close 45523 (time 1736262049.76 total 78.55 last 552.28) - open/close 50000 (time 1736262058.40 total 87.19 last 518.07) - open/close 55433 (time 1736262068.40 total 97.19 last 543.26) - open/close 60000 (time 1736262076.55 total 105.34 last 560.51) total: 64768 open/close in 113.06 seconds: 572.87 ops/second - /unlink 8908 (time 1736262095.05 total 10.00 last 890.78) - /unlink 18014 (time 1736262105.05 total 20.00 last 910.54) - /unlink 20000 (time 1736262107.13 total 22.08 last 957.23) - /unlink 29128 (time 1736262117.13 total 32.08 last 912.80) - /unlink 38405 (time 1736262127.13 total 42.08 last 927.69) - /unlink 47233 (time 1736262137.13 total 52.08 last 882.71) - /unlink 50000 (time 1736262140.26 total 55.21 last 883.37) - /unlink 59149 (time 1736262150.26 total 65.21 last 914.81) total: 64768 /unlink in 71.32 seconds: 908.09 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5357 (time 1736262196.70 total 10.00 last 535.62) - open/close 10000 (time 1736262205.55 total 18.86 last 524.42) - open/close 15230 (time 1736262215.55 total 28.86 last 522.99) - open/close 20000 (time 1736262224.51 total 37.81 last 532.51) - open/close 25187 (time 1736262234.51 total 47.81 last 518.64) - open/close 30000 (time 1736262243.61 total 56.91 last 528.84) - open/close 35149 (time 1736262253.61 total 66.92 last 514.84) - open/close 40000 (time 1736262262.92 total 76.23 last 521.09) - open/close 45217 (time 1736262272.92 total 86.23 last 521.67) - open/close 49008 (time 1736262282.93 total 96.23 last 378.84) - open/close 50000 (time 1736262286.00 total 99.30 last 323.45) - open/close 51982 (time 1736262296.00 total 109.30 last 198.18) - open/close 55974 (time 1736262306.00 total 119.30 last 399.15) - open/close 60000 (time 1736262315.13 total 128.43 last 441.04) - open/close 64170 (time 1736262325.13 total 138.43 last 416.92) total: 64768 open/close in 140.04 seconds: 462.49 ops/second - /unlink 7812 (time 1736262337.59 total 10.00 last 781.15) - /unlink 10000 (time 1736262339.97 total 12.38 last 920.03) - /unlink 19198 (time 1736262349.97 total 22.38 last 919.77) - /unlink 27991 (time 1736262359.97 total 32.38 last 879.28) - /unlink 30000 (time 1736262362.40 total 34.81 last 825.50) - /unlink 39446 (time 1736262372.40 total 44.81 last 944.51) - /unlink 48084 (time 1736262382.40 total 54.81 last 863.76) - /unlink 58285 (time 1736262392.40 total 64.81 last 1020.09) total: 64768 /unlink in 71.27 seconds: 908.81 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5449 (time 1736262437.46 total 10.00 last 544.81) - open/close 10000 (time 1736262445.16 total 17.69 last 591.55) - open/close 15389 (time 1736262455.16 total 27.70 last 538.87) - open/close 20000 (time 1736262464.39 total 36.92 last 499.61) - open/close 25496 (time 1736262474.39 total 46.93 last 549.55) - open/close 30000 (time 1736262483.02 total 55.55 last 522.06) - open/close 35297 (time 1736262493.02 total 65.55 last 529.67) - open/close 40000 (time 1736262501.15 total 73.69 last 578.08) - open/close 45513 (time 1736262511.15 total 83.69 last 551.29) - open/close 50000 (time 1736262519.52 total 92.06 last 536.17) - open/close 55323 (time 1736262529.52 total 102.06 last 532.29) - open/close 60000 (time 1736262539.23 total 111.77 last 481.44) total: 64768 open/close in 120.99 seconds: 535.31 ops/second - /unlink 9444 (time 1736262559.27 total 10.00 last 944.35) - /unlink 18081 (time 1736262569.27 total 20.00 last 863.65) - /unlink 20000 (time 1736262571.51 total 22.23 last 859.25) - /unlink 29283 (time 1736262581.51 total 32.24 last 928.20) - /unlink 38549 (time 1736262591.51 total 42.24 last 926.57) - /unlink 48376 (time 1736262601.51 total 52.24 last 982.67) - /unlink 57992 (time 1736262611.51 total 62.24 last 961.60) - /unlink 60000 (time 1736262613.74 total 64.47 last 898.49) total: 64768 /unlink in 70.09 seconds: 924.06 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server LNET ready to unload unloading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing unload_modules_local oleg340-server: LNET ready to unload modules unloaded. pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 PASS 106 (1229s) Stopping clients: oleg340-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg340-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 oleg340-server: oleg340-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg340-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1313 sec ========== 10:11:23 (1736262683)