-----============= acceptance-small: conf-sanity ============----- Wed Jul 17 08:00:30 EDT 2024 excepting tests: 32newtarball 110 Stopping clients: oleg338-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg338-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg338-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg338-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg338-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg338-server oleg338-server: oleg338-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg338-server' oleg338-server: oleg338-server.virtnet: executing load_modules_local oleg338-server: Loading modules from /home/green/git/lustre-release/lustre oleg338-server: detected 4 online CPUs by sysfs oleg338-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg338-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg338-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg338-server: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg338-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg338-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg338-server stop mds service on oleg338-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg338-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:01:35 (1721217695) Stopping clients: oleg338-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg338-client.virtnet /mnt/lustre2 (opts:-f) oleg338-server: oleg338-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg338-server' oleg338-server: oleg338-server.virtnet: executing load_modules_local oleg338-server: Loading modules from /home/green/git/lustre-release/lustre oleg338-server: detected 4 online CPUs by sysfs oleg338-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg338-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg338-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg338-client.virtnet: -o user_xattr,flock oleg338-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5558 (time 1721217728.23 total 10.00 last 555.76) - open/close 10000 (time 1721217736.53 total 18.31 last 534.88) - open/close 15435 (time 1721217746.53 total 28.31 last 543.43) - open/close 20000 (time 1721217755.04 total 36.81 last 536.95) - open/close 25444 (time 1721217765.04 total 46.81 last 544.32) - open/close 30000 (time 1721217773.26 total 55.03 last 553.96) - open/close 35459 (time 1721217783.26 total 65.04 last 545.80) - open/close 40000 (time 1721217791.81 total 73.58 last 531.37) - open/close 45085 (time 1721217801.81 total 83.58 last 508.44) - open/close 50000 (time 1721217811.26 total 93.03 last 520.10) - open/close 55266 (time 1721217821.26 total 103.03 last 526.52) - open/close 60000 (time 1721217830.38 total 112.15 last 519.09) total: 64768 open/close in 121.27 seconds: 534.07 ops/second - /unlink 8994 (time 1721217850.35 total 10.00 last 899.37) - /unlink 18293 (time 1721217860.35 total 20.00 last 929.79) - /unlink 27497 (time 1721217870.35 total 30.00 last 920.34) - /unlink 30000 (time 1721217873.14 total 32.79 last 896.29) - /unlink 38719 (time 1721217883.14 total 42.80 last 871.87) - /unlink 47738 (time 1721217893.14 total 52.80 last 901.84) - /unlink 50000 (time 1721217895.67 total 55.33 last 893.98) - /unlink 58265 (time 1721217905.67 total 65.33 last 826.41) - /unlink 60000 (time 1721217907.73 total 67.38 last 844.41) total: 64768 /unlink in 73.03 seconds: 886.82 ops/second - open/close 4797 (time 1721217924.34 total 10.00 last 479.70) - open/close 9233 (time 1721217934.34 total 20.00 last 443.53) - open/close 13723 (time 1721217944.34 total 30.00 last 448.92) - open/close 18505 (time 1721217954.34 total 40.00 last 478.13) - open/close 20000 (time 1721217957.22 total 42.88 last 519.93) - open/close 25149 (time 1721217967.22 total 52.88 last 514.84) - open/close 30000 (time 1721217976.58 total 62.25 last 518.05) - open/close 35243 (time 1721217986.58 total 72.25 last 524.25) - open/close 40000 (time 1721217995.67 total 81.33 last 523.70) - open/close 45302 (time 1721218005.67 total 91.33 last 530.12) - open/close 50000 (time 1721218014.61 total 100.27 last 525.75) - open/close 55255 (time 1721218024.61 total 110.27 last 525.48) - open/close 60000 (time 1721218033.63 total 119.29 last 525.67) total: 64768 open/close in 128.34 seconds: 504.65 ops/second - /unlink 9124 (time 1721218053.52 total 10.00 last 912.39) - /unlink 17708 (time 1721218063.52 total 20.00 last 858.36) - /unlink 20000 (time 1721218066.01 total 22.49 last 921.03) - /unlink 29182 (time 1721218076.01 total 32.49 last 918.19) - /unlink 38505 (time 1721218086.01 total 42.49 last 932.24) - /unlink 47751 (time 1721218096.01 total 52.49 last 924.59) - /unlink 50000 (time 1721218098.52 total 55.00 last 897.23) - /unlink 59205 (time 1721218108.52 total 65.00 last 920.50) total: 64768 /unlink in 70.75 seconds: 915.46 ops/second - open/close 4549 (time 1721218125.14 total 10.00 last 454.88) - open/close 9788 (time 1721218135.14 total 20.00 last 523.85) - open/close 14791 (time 1721218145.14 total 30.00 last 500.27) - open/close 19798 (time 1721218155.14 total 40.00 last 500.64) - open/close 22721 (time 1721218165.14 total 50.00 last 292.27) - open/close 26780 (time 1721218175.15 total 60.00 last 405.87) - open/close 30000 (time 1721218184.54 total 69.40 last 342.81) - open/close 34022 (time 1721218194.55 total 79.41 last 401.56) - open/close 37069 (time 1721218204.56 total 89.42 last 304.64) - open/close 40000 (time 1721218212.99 total 97.85 last 347.64) - open/close 43213 (time 1721218222.99 total 107.85 last 321.27) - open/close 46955 (time 1721218232.99 total 117.85 last 374.09) - open/close 50000 (time 1721218242.68 total 127.53 last 314.43) - open/close 53325 (time 1721218252.68 total 137.54 last 332.43) - open/close 56663 (time 1721218262.68 total 147.54 last 333.66) - open/close 59693 (time 1721218272.68 total 157.54 last 302.98) - open/close 63791 (time 1721218282.68 total 167.54 last 409.74) total: 64768 open/close in 169.73 seconds: 381.60 ops/second - /unlink 6801 (time 1721218295.80 total 10.00 last 680.08) - /unlink 10000 (time 1721218299.49 total 13.69 last 867.59) - /unlink 17614 (time 1721218309.49 total 23.69 last 761.32) - /unlink 20000 (time 1721218312.90 total 27.10 last 699.80) - /unlink 26911 (time 1721218322.90 total 37.10 last 691.05) - /unlink 30000 (time 1721218329.45 total 43.65 last 471.40) - /unlink 38147 (time 1721218339.46 total 53.65 last 814.47) - /unlink 40000 (time 1721218341.99 total 56.19 last 730.94) - /unlink 48701 (time 1721218351.99 total 66.19 last 870.03) - /unlink 56150 (time 1721218361.99 total 76.19 last 744.85) - /unlink 60000 (time 1721218368.09 total 82.29 last 631.52) total: 64768 /unlink in 90.70 seconds: 714.13 ops/second - open/close 3484 (time 1721218387.58 total 10.00 last 348.33) - open/close 7246 (time 1721218397.58 total 20.00 last 376.18) - open/close 10000 (time 1721218405.70 total 28.12 last 339.20) - open/close 13274 (time 1721218415.70 total 38.12 last 327.38) - open/close 15788 (time 1721218425.71 total 48.12 last 251.34) - open/close 19748 (time 1721218435.71 total 58.13 last 395.96) - open/close 24519 (time 1721218445.71 total 68.13 last 477.07) - open/close 29177 (time 1721218455.71 total 78.13 last 465.76) - open/close 32935 (time 1721218465.71 total 88.13 last 375.73) - open/close 36217 (time 1721218475.71 total 98.13 last 328.19) - open/close 39816 (time 1721218485.71 total 108.13 last 359.89) - open/close 44607 (time 1721218495.71 total 118.13 last 479.05) - open/close 47902 (time 1721218505.71 total 128.13 last 329.44) - open/close 50000 (time 1721218513.94 total 136.36 last 255.12) - open/close 53189 (time 1721218523.94 total 146.36 last 318.85) - open/close 56576 (time 1721218533.94 total 156.36 last 338.66) - open/close 59466 (time 1721218543.94 total 166.36 last 288.93) - open/close 62326 (time 1721218553.94 total 176.36 last 285.94) total: 64768 open/close in 184.89 seconds: 350.30 ops/second - /unlink 5626 (time 1721218573.70 total 10.00 last 562.47) - /unlink 10000 (time 1721218583.18 total 19.49 last 461.08) - /unlink 14548 (time 1721218593.19 total 29.49 last 454.77) - /unlink 19846 (time 1721218603.19 total 39.49 last 529.74) - /unlink 24936 (time 1721218613.19 total 49.49 last 508.94) - /unlink 29918 (time 1721218623.19 total 59.49 last 498.20) - /unlink 34397 (time 1721218633.19 total 69.49 last 447.83) - /unlink 39001 (time 1721218643.19 total 79.50 last 460.29) - /unlink 40000 (time 1721218645.44 total 81.74 last 444.66) - /unlink 44644 (time 1721218655.44 total 91.74 last 464.35) - /unlink 49399 (time 1721218665.44 total 101.75 last 475.42) - /unlink 54051 (time 1721218675.44 total 111.75 last 465.14) - /unlink 58702 (time 1721218685.44 total 121.75 last 465.03) - /unlink 60000 (time 1721218687.75 total 124.05 last 562.93) total: 64768 /unlink in 133.77 seconds: 484.18 ops/second - open/close 2462 (time 1721218708.82 total 10.00 last 246.13) - open/close 4995 (time 1721218718.82 total 20.00 last 253.25) - open/close 7404 (time 1721218728.82 total 30.01 last 240.82) - open/close 9941 (time 1721218738.82 total 40.01 last 253.66) - open/close 12491 (time 1721218748.83 total 50.01 last 254.91) - open/close 15043 (time 1721218758.83 total 60.01 last 255.17) - open/close 18592 (time 1721218768.83 total 70.01 last 354.88) - open/close 20000 (time 1721218773.49 total 74.67 last 302.24) - open/close 22667 (time 1721218783.49 total 84.68 last 266.65) - open/close 25160 (time 1721218793.49 total 94.68 last 249.27) - open/close 27627 (time 1721218803.49 total 104.68 last 246.68) - open/close 30000 (time 1721218812.31 total 113.50 last 268.98) - open/close 32455 (time 1721218822.32 total 123.50 last 245.48) - open/close 34904 (time 1721218832.32 total 133.50 last 244.88) - open/close 37415 (time 1721218842.32 total 143.50 last 251.01) - open/close 39964 (time 1721218852.32 total 153.51 last 254.88) - open/close 42360 (time 1721218862.32 total 163.51 last 239.56) - open/close 44775 (time 1721218872.32 total 173.51 last 241.49) - open/close 47306 (time 1721218882.33 total 183.51 last 253.04) - open/close 49873 (time 1721218892.33 total 193.51 last 256.66) - open/close 52395 (time 1721218902.33 total 203.52 last 252.11) - open/close 54830 (time 1721218912.33 total 213.52 last 243.45) - open/close 57270 (time 1721218922.34 total 223.52 last 243.89) - open/close 59632 (time 1721218932.34 total 233.52 last 236.17) - open/close 62062 (time 1721218942.34 total 243.53 last 242.94) - open/close 64602 (time 1721218952.34 total 253.53 last 253.97) total: 64768 open/close in 254.24 seconds: 254.76 ops/second - /unlink 4720 (time 1721218964.33 total 10.00 last 471.90) - /unlink 9363 (time 1721218974.33 total 20.00 last 464.26) - /unlink 13960 (time 1721218984.33 total 30.00 last 459.70) - /unlink 18675 (time 1721218994.33 total 40.00 last 471.48) - /unlink 20000 (time 1721218996.87 total 42.55 last 520.58) - /unlink 24666 (time 1721219006.88 total 52.55 last 466.57) - /unlink 29611 (time 1721219016.88 total 62.55 last 494.47) - /unlink 34283 (time 1721219026.88 total 72.55 last 467.16) - /unlink 38975 (time 1721219036.88 total 82.55 last 469.11) - /unlink 40000 (time 1721219039.27 total 84.94 last 429.44) - /unlink 44646 (time 1721219049.27 total 94.94 last 464.56) - /unlink 49261 (time 1721219059.27 total 104.94 last 461.41) - /unlink 54077 (time 1721219069.27 total 114.94 last 481.54) - /unlink 58658 (time 1721219079.27 total 124.94 last 458.06) - /unlink 60000 (time 1721219082.12 total 127.79 last 471.60) total: 64768 /unlink in 137.58 seconds: 470.77 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg338-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg338-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg338-server stop mds service on oleg338-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg338-server LNET ready to unload unloading modules on: 'oleg338-server' oleg338-server: oleg338-server.virtnet: executing unload_modules_local oleg338-server: LNET ready to unload modules unloaded. pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2 PASS 106 (1433s) Stopping clients: oleg338-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg338-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2 oleg338-server: oleg338-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg338-server' oleg338-server: oleg338-server.virtnet: executing load_modules_local oleg338-server: Loading modules from /home/green/git/lustre-release/lustre oleg338-server: detected 4 online CPUs by sysfs oleg338-server: Force libcfs to create 2 CPU partitions oleg338-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg338-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1522 sec ========== 08:25:52 (1721219152)