-----============= acceptance-small: conf-sanity ============----- Wed Apr 10 10:58:29 EDT 2024 excepting tests: 32newtarball 110 Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg207-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg207-server oleg207-server: oleg207-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: detected 4 online CPUs by sysfs oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg207-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg207-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:59:43 (1712761183) Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) oleg207-server: oleg207-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: detected 4 online CPUs by sysfs oleg207-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg207-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg207-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock oleg207-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1756 (time 1712761232.29 total 10.00 last 175.52) - open/close 3411 (time 1712761242.29 total 20.01 last 165.47) - open/close 5162 (time 1712761252.29 total 30.01 last 175.02) - open/close 7045 (time 1712761262.29 total 40.01 last 188.26) - open/close 9257 (time 1712761272.30 total 50.01 last 221.17) - open/close 10000 (time 1712761274.85 total 52.57 last 291.01) - open/close 12903 (time 1712761284.85 total 62.57 last 290.23) - open/close 15699 (time 1712761294.85 total 72.57 last 279.59) - open/close 17769 (time 1712761304.86 total 82.58 last 206.90) - open/close 19498 (time 1712761314.86 total 92.58 last 172.88) - open/close 20000 (time 1712761317.94 total 95.66 last 162.78) - open/close 21787 (time 1712761327.94 total 105.66 last 178.66) - open/close 23449 (time 1712761337.94 total 115.66 last 166.17) - open/close 25122 (time 1712761347.95 total 125.67 last 167.25) - open/close 27358 (time 1712761357.95 total 135.67 last 223.54) - open/close 30000 (time 1712761366.01 total 143.73 last 327.79) - open/close 33840 (time 1712761376.01 total 153.73 last 383.92) - open/close 37271 (time 1712761386.01 total 163.73 last 343.02) - open/close 40000 (time 1712761393.85 total 171.57 last 348.12) - open/close 43593 (time 1712761403.85 total 181.57 last 359.25) - open/close 46868 (time 1712761413.86 total 191.58 last 327.42) - open/close 50000 (time 1712761423.29 total 201.01 last 331.95) - open/close 53542 (time 1712761433.29 total 211.01 last 354.16) - open/close 57135 (time 1712761443.29 total 221.01 last 359.28) - open/close 60000 (time 1712761452.82 total 230.54 last 300.72) - open/close 62941 (time 1712761462.82 total 240.54 last 294.08) total: 64768 open/close in 247.52 seconds: 261.67 ops/second - /unlink 4937 (time 1712761480.80 total 10.00 last 493.66) - /unlink 10000 (time 1712761487.46 total 16.66 last 760.16) - /unlink 16425 (time 1712761497.46 total 26.66 last 642.48) - /unlink 20000 (time 1712761503.61 total 32.81 last 581.73) - /unlink 26681 (time 1712761513.61 total 42.81 last 668.03) - /unlink 30000 (time 1712761518.13 total 47.33 last 733.60) - /unlink 37412 (time 1712761528.13 total 57.33 last 741.18) - /unlink 40000 (time 1712761531.72 total 60.92 last 720.94) - /unlink 46596 (time 1712761541.72 total 70.92 last 659.60) - /unlink 50000 (time 1712761546.80 total 76.00 last 670.77) - /unlink 56263 (time 1712761556.80 total 86.00 last 626.26) - /unlink 60000 (time 1712761562.59 total 91.79 last 645.47) total: 64768 /unlink in 98.88 seconds: 654.99 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3456 (time 1712761599.67 total 10.00 last 345.52) - open/close 7292 (time 1712761609.68 total 20.00 last 383.51) - open/close 10000 (time 1712761617.81 total 28.14 last 333.06) - open/close 13646 (time 1712761627.81 total 38.14 last 364.58) - open/close 17997 (time 1712761637.81 total 48.14 last 435.09) - open/close 20000 (time 1712761642.14 total 52.46 last 462.81) - open/close 24721 (time 1712761652.14 total 62.47 last 472.01) - open/close 28945 (time 1712761662.14 total 72.47 last 422.33) - open/close 30000 (time 1712761665.04 total 75.37 last 363.21) - open/close 34267 (time 1712761675.04 total 85.37 last 426.67) - open/close 38848 (time 1712761685.05 total 95.37 last 458.05) - open/close 40000 (time 1712761687.39 total 97.72 last 492.02) - open/close 44745 (time 1712761697.39 total 107.72 last 474.49) - open/close 48840 (time 1712761707.39 total 117.72 last 409.47) - open/close 50000 (time 1712761710.59 total 120.92 last 362.50) - open/close 53550 (time 1712761720.59 total 130.92 last 354.96) - open/close 57510 (time 1712761730.59 total 140.92 last 395.91) - open/close 60000 (time 1712761737.03 total 147.36 last 386.69) - open/close 64069 (time 1712761747.03 total 157.36 last 406.90) total: 64768 open/close in 159.20 seconds: 406.83 ops/second - /unlink 6957 (time 1712761759.76 total 10.00 last 695.68) - /unlink 10000 (time 1712761763.86 total 14.10 last 742.47) - /unlink 18146 (time 1712761773.86 total 24.10 last 814.57) - /unlink 20000 (time 1712761776.19 total 26.43 last 794.47) - /unlink 27744 (time 1712761786.19 total 36.43 last 774.33) - /unlink 30000 (time 1712761789.53 total 39.77 last 676.68) - /unlink 36183 (time 1712761799.53 total 49.77 last 618.26) - /unlink 40000 (time 1712761804.71 total 54.95 last 736.25) - /unlink 47643 (time 1712761814.71 total 64.95 last 764.27) - /unlink 50000 (time 1712761817.14 total 67.38 last 972.06) - /unlink 59381 (time 1712761827.14 total 77.38 last 938.03) total: 64768 /unlink in 84.12 seconds: 769.95 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5141 (time 1712761865.15 total 10.00 last 514.03) - open/close 10000 (time 1712761873.77 total 18.63 last 563.40) - open/close 15366 (time 1712761883.77 total 28.63 last 536.58) - open/close 20000 (time 1712761892.87 total 37.72 last 509.32) - open/close 25343 (time 1712761902.87 total 47.73 last 534.27) - open/close 30000 (time 1712761912.29 total 57.15 last 494.14) - open/close 34730 (time 1712761922.29 total 67.15 last 472.99) - open/close 39157 (time 1712761932.36 total 77.21 last 439.87) - open/close 43940 (time 1712761942.36 total 87.21 last 478.30) - open/close 49173 (time 1712761952.36 total 97.22 last 523.24) - open/close 54030 (time 1712761962.36 total 107.22 last 485.66) - open/close 58758 (time 1712761972.36 total 117.22 last 472.76) - open/close 60000 (time 1712761975.02 total 119.87 last 467.87) total: 64768 open/close in 129.21 seconds: 501.26 ops/second - /unlink 9137 (time 1712761995.16 total 10.00 last 913.69) - /unlink 18224 (time 1712762005.16 total 20.00 last 908.66) - /unlink 20000 (time 1712762007.22 total 22.06 last 863.35) - /unlink 28102 (time 1712762017.22 total 32.06 last 810.19) - /unlink 30000 (time 1712762019.88 total 34.73 last 711.57) - /unlink 37280 (time 1712762029.88 total 44.73 last 727.92) - /unlink 40000 (time 1712762033.49 total 48.33 last 754.82) - /unlink 47422 (time 1712762043.49 total 58.33 last 742.12) - /unlink 50000 (time 1712762046.80 total 61.64 last 778.22) - /unlink 58337 (time 1712762056.80 total 71.64 last 833.63) - /unlink 60000 (time 1712762058.95 total 73.80 last 773.01) total: 64768 /unlink in 79.17 seconds: 818.09 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5398 (time 1712762095.30 total 10.00 last 539.76) - open/close 10000 (time 1712762104.25 total 18.94 last 514.59) - open/close 15229 (time 1712762114.25 total 28.94 last 522.85) - open/close 20000 (time 1712762123.81 total 38.51 last 498.68) - open/close 25412 (time 1712762133.81 total 48.51 last 541.18) - open/close 30000 (time 1712762142.79 total 57.49 last 511.14) - open/close 34344 (time 1712762152.79 total 67.49 last 434.37) - open/close 38892 (time 1712762162.79 total 77.49 last 454.80) - open/close 40000 (time 1712762164.94 total 79.64 last 514.32) - open/close 45575 (time 1712762174.95 total 89.64 last 557.48) - open/close 50000 (time 1712762182.71 total 97.41 last 569.55) - open/close 55249 (time 1712762192.72 total 107.41 last 524.82) - open/close 59941 (time 1712762202.72 total 117.42 last 469.11) total: 64768 open/close in 127.31 seconds: 508.76 ops/second - /unlink 8204 (time 1712762223.47 total 10.00 last 820.37) - /unlink 10000 (time 1712762225.69 total 12.22 last 810.17) - /unlink 18366 (time 1712762235.69 total 22.22 last 836.57) - /unlink 26400 (time 1712762245.69 total 32.22 last 803.33) - /unlink 30000 (time 1712762250.30 total 36.82 last 781.74) - /unlink 38135 (time 1712762260.30 total 46.82 last 813.48) - /unlink 40000 (time 1712762262.50 total 49.03 last 846.08) - /unlink 48271 (time 1712762272.50 total 59.03 last 827.01) - /unlink 50000 (time 1712762274.55 total 61.07 last 845.20) - /unlink 58084 (time 1712762284.55 total 71.07 last 808.39) - /unlink 60000 (time 1712762286.90 total 73.43 last 814.19) total: 64768 /unlink in 79.11 seconds: 818.73 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5034 (time 1712762320.16 total 10.00 last 503.38) - open/close 10000 (time 1712762329.85 total 19.69 last 512.64) - open/close 15012 (time 1712762339.85 total 29.69 last 501.19) - open/close 20000 (time 1712762349.67 total 39.50 last 508.14) - open/close 25120 (time 1712762359.67 total 49.50 last 511.98) - open/close 30000 (time 1712762369.03 total 58.87 last 521.10) - open/close 35092 (time 1712762379.03 total 68.87 last 509.13) - open/close 40000 (time 1712762388.50 total 78.34 last 518.24) - open/close 45166 (time 1712762398.51 total 88.34 last 516.55) - open/close 50000 (time 1712762408.01 total 97.84 last 508.69) - open/close 55011 (time 1712762418.01 total 107.85 last 501.05) - open/close 60000 (time 1712762427.92 total 117.76 last 503.33) - open/close 64509 (time 1712762437.92 total 127.76 last 450.84) total: 64768 open/close in 128.23 seconds: 505.09 ops/second - /unlink 8108 (time 1712762449.46 total 10.00 last 810.75) - /unlink 10000 (time 1712762451.58 total 12.12 last 893.84) - /unlink 18363 (time 1712762461.58 total 22.12 last 836.24) - /unlink 27012 (time 1712762471.58 total 32.12 last 864.85) - /unlink 30000 (time 1712762475.10 total 35.63 last 850.34) - /unlink 38429 (time 1712762485.10 total 45.63 last 842.87) - /unlink 47093 (time 1712762495.10 total 55.63 last 866.35) - /unlink 50000 (time 1712762498.43 total 58.97 last 870.84) - /unlink 58611 (time 1712762508.43 total 68.97 last 861.08) total: 64768 /unlink in 76.09 seconds: 851.15 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server unloading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 PASS 106 (1376s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg207-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg207-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 oleg207-server: oleg207-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: detected 4 online CPUs by sysfs oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg207-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1466 sec ========== 11:22:56 (1712762576) === conf-sanity: start cleanup 11:22:57 (1712762577) === === conf-sanity: finish cleanup 11:22:57 (1712762577) ===