-----============= acceptance-small: conf-sanity ============----- Mon Apr 15 15:10:40 EDT 2024 excepting tests: 32newtarball 110 41c Stopping clients: oleg223-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg223-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg223-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg223-server oleg223-server: oleg223-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing load_modules_local oleg223-server: Loading modules from /home/green/git/lustre-release/lustre oleg223-server: detected 4 online CPUs by sysfs oleg223-server: Force libcfs to create 2 CPU partitions oleg223-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg223-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg223-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg223-server: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg223-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:11:54 (1713208314) Stopping clients: oleg223-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg223-client.virtnet /mnt/lustre2 (opts:-f) oleg223-server: oleg223-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing load_modules_local oleg223-server: Loading modules from /home/green/git/lustre-release/lustre oleg223-server: detected 4 online CPUs by sysfs oleg223-server: Force libcfs to create 2 CPU partitions oleg223-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg223-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg223-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock oleg223-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5326 (time 1713208347.41 total 10.00 last 532.49) - open/close 10000 (time 1713208356.89 total 19.48 last 493.00) - open/close 15136 (time 1713208366.89 total 29.48 last 513.58) - open/close 19236 (time 1713208376.89 total 39.49 last 409.91) - open/close 23767 (time 1713208386.89 total 49.49 last 453.03) - open/close 29152 (time 1713208396.90 total 59.49 last 538.45) - open/close 34415 (time 1713208406.90 total 69.49 last 526.21) - open/close 39505 (time 1713208416.90 total 79.49 last 508.96) - open/close 45081 (time 1713208426.90 total 89.49 last 557.51) - open/close 49716 (time 1713208436.90 total 99.49 last 463.41) - open/close 54503 (time 1713208446.90 total 109.49 last 478.67) - open/close 58841 (time 1713208456.90 total 119.49 last 433.80) - open/close 60000 (time 1713208459.84 total 122.44 last 393.92) - open/close 64297 (time 1713208469.85 total 132.44 last 429.62) total: 64768 open/close in 133.53 seconds: 485.04 ops/second - /unlink 7364 (time 1713208481.88 total 10.00 last 736.32) - /unlink 10000 (time 1713208485.74 total 13.86 last 682.84) - /unlink 18578 (time 1713208495.74 total 23.86 last 857.74) - /unlink 27634 (time 1713208505.74 total 33.86 last 905.52) - /unlink 30000 (time 1713208508.09 total 36.21 last 1009.05) - /unlink 38704 (time 1713208518.09 total 46.21 last 870.34) - /unlink 47032 (time 1713208528.09 total 56.21 last 832.75) - /unlink 50000 (time 1713208531.67 total 59.79 last 829.41) - /unlink 58611 (time 1713208541.67 total 69.79 last 861.07) total: 64768 /unlink in 76.99 seconds: 841.30 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5310 (time 1713208581.08 total 10.00 last 530.99) - open/close 10000 (time 1713208590.37 total 19.29 last 504.90) - open/close 15127 (time 1713208600.37 total 29.29 last 512.64) - open/close 20000 (time 1713208610.04 total 38.97 last 503.67) - open/close 25101 (time 1713208620.04 total 48.97 last 510.00) - open/close 30000 (time 1713208629.88 total 58.80 last 498.03) - open/close 35035 (time 1713208639.88 total 68.80 last 503.49) - open/close 40000 (time 1713208649.77 total 78.70 last 501.93) - open/close 45094 (time 1713208659.77 total 88.70 last 509.37) - open/close 49924 (time 1713208669.77 total 98.70 last 483.00) - open/close 54497 (time 1713208679.77 total 108.70 last 457.27) - open/close 59315 (time 1713208689.78 total 118.70 last 481.73) - open/close 64251 (time 1713208699.78 total 128.70 last 493.51) total: 64768 open/close in 129.93 seconds: 498.47 ops/second - /unlink 7740 (time 1713208711.91 total 10.00 last 773.96) - /unlink 10000 (time 1713208715.03 total 13.12 last 725.51) - /unlink 17530 (time 1713208725.03 total 23.12 last 752.93) - /unlink 20000 (time 1713208728.11 total 26.20 last 801.62) - /unlink 27865 (time 1713208738.11 total 36.20 last 786.46) - /unlink 30000 (time 1713208740.64 total 38.73 last 843.55) - /unlink 38408 (time 1713208750.64 total 48.73 last 840.72) - /unlink 46676 (time 1713208760.64 total 58.73 last 826.78) - /unlink 50000 (time 1713208764.71 total 62.79 last 817.93) - /unlink 57883 (time 1713208774.71 total 72.79 last 788.28) - /unlink 60000 (time 1713208777.34 total 75.43 last 804.42) total: 64768 /unlink in 81.28 seconds: 796.86 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5192 (time 1713208810.63 total 10.00 last 519.11) - open/close 10000 (time 1713208820.14 total 19.51 last 505.40) - open/close 15926 (time 1713208830.14 total 29.52 last 592.56) - open/close 20000 (time 1713208836.80 total 36.17 last 612.17) - open/close 25470 (time 1713208846.80 total 46.17 last 546.96) - open/close 30000 (time 1713208855.45 total 54.82 last 523.58) - open/close 35029 (time 1713208865.45 total 64.82 last 502.85) - open/close 39757 (time 1713208875.45 total 74.83 last 472.70) - open/close 44627 (time 1713208885.45 total 84.83 last 486.94) - open/close 49197 (time 1713208895.46 total 94.83 last 456.97) - open/close 53931 (time 1713208905.46 total 104.83 last 473.38) - open/close 58704 (time 1713208915.46 total 114.83 last 477.23) - open/close 60000 (time 1713208918.14 total 117.51 last 483.99) total: 64768 open/close in 126.92 seconds: 510.30 ops/second - /unlink 8393 (time 1713208938.43 total 10.00 last 839.21) - /unlink 17026 (time 1713208948.43 total 20.00 last 863.30) - /unlink 20000 (time 1713208951.96 total 23.53 last 842.25) - /unlink 28509 (time 1713208961.96 total 33.53 last 850.84) - /unlink 37128 (time 1713208971.96 total 43.53 last 861.88) - /unlink 40000 (time 1713208975.27 total 46.84 last 869.09) - /unlink 48519 (time 1713208985.27 total 56.84 last 851.87) - /unlink 56873 (time 1713208995.27 total 66.84 last 835.32) - /unlink 60000 (time 1713208998.80 total 70.37 last 886.74) total: 64768 /unlink in 75.69 seconds: 855.69 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5252 (time 1713209036.71 total 10.00 last 525.13) - open/close 7862 (time 1713209046.71 total 20.01 last 260.88) - open/close 10000 (time 1713209056.64 total 29.93 last 215.34) - open/close 14160 (time 1713209066.64 total 39.94 last 415.93) - open/close 19516 (time 1713209076.64 total 49.94 last 535.56) - open/close 23590 (time 1713209086.65 total 59.94 last 407.29) - open/close 28400 (time 1713209096.65 total 69.94 last 480.96) - open/close 30000 (time 1713209100.02 total 73.32 last 473.78) - open/close 35035 (time 1713209110.02 total 83.32 last 503.50) - open/close 39474 (time 1713209120.02 total 93.32 last 443.87) - open/close 44077 (time 1713209130.03 total 103.32 last 460.14) - open/close 46501 (time 1713209140.03 total 113.32 last 242.36) - open/close 49316 (time 1713209150.03 total 123.32 last 281.49) - open/close 50000 (time 1713209152.19 total 125.48 last 317.16) - open/close 52453 (time 1713209162.19 total 135.48 last 245.25) - open/close 55263 (time 1713209172.19 total 145.49 last 280.94) - open/close 57881 (time 1713209182.19 total 155.49 last 261.74) - open/close 60000 (time 1713209191.00 total 164.30 last 240.47) - open/close 63290 (time 1713209201.01 total 174.30 last 328.96) total: 64768 open/close in 178.50 seconds: 362.84 ops/second - /unlink 7267 (time 1713209216.08 total 10.00 last 726.67) - /unlink 10000 (time 1713209219.95 total 13.87 last 707.10) - /unlink 16533 (time 1713209229.95 total 23.87 last 653.21) - /unlink 20000 (time 1713209234.87 total 28.79 last 704.77) - /unlink 27556 (time 1713209244.87 total 38.79 last 755.56) - /unlink 30000 (time 1713209247.81 total 41.73 last 831.07) - /unlink 37569 (time 1713209257.81 total 51.73 last 756.87) - /unlink 40000 (time 1713209261.23 total 55.15 last 709.84) - /unlink 48241 (time 1713209271.24 total 65.15 last 824.08) - /unlink 50000 (time 1713209273.40 total 67.32 last 812.72) - /unlink 57968 (time 1713209283.40 total 77.32 last 796.71) - /unlink 60000 (time 1713209286.32 total 80.24 last 695.88) total: 64768 /unlink in 86.01 seconds: 753.07 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4479 (time 1713209322.26 total 10.00 last 447.89) - open/close 9000 (time 1713209332.26 total 20.00 last 452.01) - open/close 10000 (time 1713209334.26 total 22.01 last 498.68) - open/close 15059 (time 1713209344.26 total 32.01 last 505.83) - open/close 19463 (time 1713209354.26 total 42.01 last 440.35) - open/close 24001 (time 1713209364.27 total 52.01 last 453.77) - open/close 28456 (time 1713209374.27 total 62.01 last 445.42) - open/close 30000 (time 1713209377.72 total 65.46 last 447.65) - open/close 34589 (time 1713209387.72 total 75.46 last 458.80) - open/close 39169 (time 1713209397.72 total 85.46 last 457.94) - open/close 44596 (time 1713209407.72 total 95.47 last 542.64) - open/close 49973 (time 1713209417.72 total 105.47 last 537.66) - open/close 55245 (time 1713209427.72 total 115.47 last 527.18) - open/close 59890 (time 1713209437.72 total 125.47 last 464.44) - open/close 64200 (time 1713209447.73 total 135.47 last 430.91) total: 64768 open/close in 136.77 seconds: 473.57 ops/second - /unlink 8855 (time 1713209459.82 total 10.00 last 885.48) - /unlink 18031 (time 1713209469.82 total 20.00 last 917.55) - /unlink 20000 (time 1713209472.13 total 22.31 last 851.93) - /unlink 28339 (time 1713209482.13 total 32.31 last 833.89) - /unlink 30000 (time 1713209484.46 total 34.64 last 713.05) - /unlink 37117 (time 1713209494.46 total 44.64 last 711.65) - /unlink 40000 (time 1713209497.69 total 47.87 last 893.67) - /unlink 48919 (time 1713209507.69 total 57.87 last 891.90) - /unlink 57007 (time 1713209517.69 total 67.87 last 808.80) - /unlink 60000 (time 1713209521.26 total 71.44 last 838.76) total: 64768 /unlink in 76.66 seconds: 844.85 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server unloading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 PASS 106 (1251s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg223-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg223-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg223-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg223-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 oleg223-server: oleg223-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing load_modules_local oleg223-server: Loading modules from /home/green/git/lustre-release/lustre oleg223-server: detected 4 online CPUs by sysfs oleg223-server: Force libcfs to create 2 CPU partitions oleg223-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg223-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1343 sec ========== 15:33:03 (1713209583) === conf-sanity: start cleanup 15:33:03 (1713209583) === === conf-sanity: finish cleanup 15:33:03 (1713209583) ===