-----============= acceptance-small: conf-sanity ============----- Sun Apr 14 02:45:04 EDT 2024 excepting tests: 32newtarball 110 41c Stopping clients: oleg314-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg314-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg314-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg314-server oleg314-server: oleg314-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing load_modules_local oleg314-server: Loading modules from /home/green/git/lustre-release/lustre oleg314-server: detected 4 online CPUs by sysfs oleg314-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg314-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 02:46:07 (1713077167) Stopping clients: oleg314-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg314-client.virtnet /mnt/lustre2 (opts:-f) oleg314-server: oleg314-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing load_modules_local oleg314-server: Loading modules from /home/green/git/lustre-release/lustre oleg314-server: detected 4 online CPUs by sysfs oleg314-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg314-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4615 (time 1713077199.55 total 10.00 last 461.49) - open/close 9698 (time 1713077209.55 total 20.00 last 508.23) - open/close 15408 (time 1713077219.55 total 30.00 last 570.91) - open/close 20000 (time 1713077228.39 total 38.85 last 519.15) - open/close 24892 (time 1713077238.39 total 48.85 last 489.19) - open/close 29735 (time 1713077248.40 total 58.85 last 484.25) - open/close 34855 (time 1713077258.40 total 68.85 last 511.99) - open/close 39998 (time 1713077268.40 total 78.85 last 514.22) - open/close 45030 (time 1713077278.40 total 88.85 last 503.19) - open/close 49440 (time 1713077288.40 total 98.85 last 440.99) - open/close 54576 (time 1713077298.40 total 108.85 last 513.52) - open/close 59537 (time 1713077308.40 total 118.85 last 496.04) - open/close 64666 (time 1713077318.40 total 128.86 last 512.82) total: 64768 open/close in 129.04 seconds: 501.94 ops/second - /unlink 8646 (time 1713077329.39 total 10.00 last 864.59) - /unlink 17364 (time 1713077339.39 total 20.00 last 871.73) - /unlink 20000 (time 1713077342.40 total 23.02 last 873.86) - /unlink 28399 (time 1713077352.40 total 33.02 last 839.89) - /unlink 30000 (time 1713077354.52 total 35.13 last 757.74) - /unlink 38511 (time 1713077364.52 total 45.13 last 851.03) - /unlink 47219 (time 1713077374.52 total 55.13 last 870.76) - /unlink 50000 (time 1713077377.74 total 58.35 last 864.45) - /unlink 58332 (time 1713077387.74 total 68.35 last 833.19) total: 64768 /unlink in 75.44 seconds: 858.48 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4907 (time 1713077428.38 total 10.00 last 490.70) - open/close 9658 (time 1713077438.38 total 20.00 last 475.03) - open/close 14728 (time 1713077448.38 total 30.00 last 506.96) - open/close 19058 (time 1713077458.38 total 40.00 last 432.93) - open/close 20000 (time 1713077461.75 total 43.37 last 280.08) - open/close 23940 (time 1713077471.75 total 53.37 last 393.93) - open/close 27736 (time 1713077481.75 total 63.37 last 379.43) - open/close 30000 (time 1713077487.20 total 68.82 last 415.34) - open/close 33803 (time 1713077497.20 total 78.83 last 380.23) - open/close 38196 (time 1713077507.21 total 88.83 last 439.18) - open/close 40000 (time 1713077512.52 total 94.14 last 339.48) - open/close 43385 (time 1713077522.52 total 104.15 last 338.44) - open/close 47135 (time 1713077532.52 total 114.15 last 374.95) - open/close 50000 (time 1713077539.94 total 121.56 last 386.56) - open/close 53933 (time 1713077549.94 total 131.56 last 393.30) - open/close 57217 (time 1713077559.94 total 141.56 last 328.34) - open/close 60000 (time 1713077568.97 total 150.59 last 308.18) - open/close 63658 (time 1713077578.97 total 160.59 last 365.76) total: 64768 open/close in 164.14 seconds: 394.59 ops/second - /unlink 5279 (time 1713077593.74 total 10.00 last 527.84) - /unlink 10000 (time 1713077601.48 total 17.74 last 609.81) - /unlink 16462 (time 1713077611.48 total 27.74 last 646.08) - /unlink 20000 (time 1713077616.69 total 32.95 last 679.52) - /unlink 27155 (time 1713077626.69 total 42.95 last 715.41) - /unlink 30000 (time 1713077631.64 total 47.90 last 575.37) - /unlink 36810 (time 1713077641.64 total 57.90 last 680.99) - /unlink 40000 (time 1713077645.42 total 61.68 last 842.34) - /unlink 47076 (time 1713077655.42 total 71.69 last 707.51) - /unlink 50000 (time 1713077658.88 total 75.14 last 845.86) - /unlink 58493 (time 1713077668.88 total 85.14 last 849.27) total: 64768 /unlink in 92.38 seconds: 701.08 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4687 (time 1713077709.42 total 10.00 last 468.61) - open/close 8630 (time 1713077719.42 total 20.00 last 394.25) - open/close 10000 (time 1713077722.91 total 23.50 last 392.23) - open/close 14448 (time 1713077732.91 total 33.50 last 444.76) - open/close 19733 (time 1713077742.92 total 43.50 last 528.45) - open/close 24099 (time 1713077752.92 total 53.50 last 436.60) - open/close 27621 (time 1713077762.92 total 63.50 last 352.20) - open/close 30000 (time 1713077767.46 total 68.04 last 523.37) - open/close 35102 (time 1713077777.46 total 78.04 last 510.20) - open/close 40000 (time 1713077787.07 total 87.65 last 509.70) - open/close 45035 (time 1713077797.07 total 97.65 last 503.49) - open/close 50000 (time 1713077807.01 total 107.59 last 499.67) - open/close 54769 (time 1713077817.01 total 117.59 last 476.83) - open/close 59786 (time 1713077827.01 total 127.59 last 501.62) - open/close 64558 (time 1713077837.01 total 137.59 last 477.17) total: 64768 open/close in 138.07 seconds: 469.09 ops/second - /unlink 8508 (time 1713077848.33 total 10.00 last 850.70) - /unlink 16298 (time 1713077858.33 total 20.00 last 778.97) - /unlink 20000 (time 1713077863.76 total 25.43 last 681.44) - /unlink 26357 (time 1713077873.76 total 35.43 last 635.68) - /unlink 30000 (time 1713077878.11 total 39.78 last 837.64) - /unlink 37652 (time 1713077888.11 total 49.78 last 765.14) - /unlink 40000 (time 1713077892.39 total 54.06 last 549.08) - /unlink 47898 (time 1713077902.39 total 64.06 last 789.78) - /unlink 50000 (time 1713077904.79 total 66.47 last 873.87) - /unlink 57067 (time 1713077914.79 total 76.47 last 706.60) - /unlink 60000 (time 1713077918.98 total 80.65 last 700.58) total: 64768 /unlink in 89.25 seconds: 725.72 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2745 (time 1713077954.53 total 10.00 last 274.43) - open/close 5658 (time 1713077964.53 total 20.00 last 291.24) - open/close 9271 (time 1713077974.53 total 30.01 last 361.25) - open/close 10000 (time 1713077976.72 total 32.19 last 333.59) - open/close 13377 (time 1713077986.72 total 42.19 last 337.66) - open/close 16635 (time 1713077996.73 total 52.20 last 325.59) - open/close 19969 (time 1713078006.73 total 62.20 last 333.21) - open/close 22854 (time 1713078016.74 total 72.21 last 288.43) - open/close 25610 (time 1713078026.74 total 82.21 last 275.59) - open/close 28484 (time 1713078036.74 total 92.21 last 287.33) - open/close 30000 (time 1713078042.70 total 98.17 last 254.24) - open/close 32696 (time 1713078052.70 total 108.17 last 269.59) - open/close 35629 (time 1713078062.70 total 118.17 last 293.30) - open/close 38248 (time 1713078072.70 total 128.18 last 261.84) - open/close 40000 (time 1713078079.60 total 135.07 last 254.13) - open/close 42583 (time 1713078089.60 total 145.07 last 258.24) - open/close 45642 (time 1713078099.66 total 155.13 last 304.13) - open/close 48246 (time 1713078109.66 total 165.13 last 260.37) - open/close 50000 (time 1713078115.43 total 170.90 last 303.91) - open/close 52613 (time 1713078125.43 total 180.91 last 261.22) - open/close 55211 (time 1713078135.43 total 190.91 last 259.80) - open/close 57759 (time 1713078145.43 total 200.91 last 254.78) - open/close 60000 (time 1713078153.96 total 209.43 last 262.83) - open/close 62877 (time 1713078163.96 total 219.43 last 287.65) total: 64768 open/close in 226.99 seconds: 285.33 ops/second - /unlink 4598 (time 1713078182.78 total 10.00 last 459.74) - /unlink 9066 (time 1713078192.78 total 20.00 last 446.77) - /unlink 10000 (time 1713078194.82 total 22.04 last 458.57) - /unlink 14931 (time 1713078204.82 total 32.04 last 493.10) - /unlink 19835 (time 1713078214.82 total 42.04 last 490.35) - /unlink 24580 (time 1713078224.82 total 52.04 last 474.39) - /unlink 30000 (time 1713078234.67 total 61.89 last 550.56) - /unlink 35171 (time 1713078244.67 total 71.89 last 517.00) - /unlink 40000 (time 1713078253.73 total 80.95 last 533.06) - /unlink 45810 (time 1713078263.73 total 90.95 last 580.93) - /unlink 50000 (time 1713078271.25 total 98.47 last 557.03) - /unlink 55386 (time 1713078281.25 total 108.47 last 538.56) - /unlink 60000 (time 1713078289.35 total 116.56 last 570.08) total: 64768 /unlink in 125.83 seconds: 514.71 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3135 (time 1713078325.57 total 10.00 last 313.46) - open/close 6791 (time 1713078335.57 total 20.00 last 365.54) - open/close 9652 (time 1713078345.57 total 30.00 last 286.07) - open/close 12763 (time 1713078355.57 total 40.01 last 310.99) - open/close 15749 (time 1713078365.57 total 50.01 last 298.60) - open/close 18814 (time 1713078375.57 total 60.01 last 306.46) - open/close 20000 (time 1713078379.16 total 63.59 last 330.76) - open/close 24210 (time 1713078389.16 total 73.59 last 420.97) - open/close 28536 (time 1713078399.16 total 83.60 last 432.59) - open/close 30000 (time 1713078402.49 total 86.92 last 439.69) - open/close 34546 (time 1713078412.49 total 96.93 last 454.53) - open/close 38298 (time 1713078422.49 total 106.93 last 375.16) - open/close 40000 (time 1713078428.00 total 112.43 last 309.06) - open/close 42762 (time 1713078438.00 total 122.43 last 276.20) - open/close 46171 (time 1713078448.00 total 132.44 last 340.86) - open/close 49524 (time 1713078458.00 total 142.44 last 335.24) - open/close 54646 (time 1713078468.00 total 152.44 last 512.14) - open/close 58753 (time 1713078478.01 total 162.44 last 410.64) - open/close 60000 (time 1713078480.38 total 164.81 last 526.13) total: 64768 open/close in 174.08 seconds: 372.07 ops/second - /unlink 7878 (time 1713078500.43 total 10.00 last 787.72) - /unlink 10000 (time 1713078503.81 total 13.38 last 627.56) - /unlink 17490 (time 1713078513.81 total 23.38 last 748.99) - /unlink 20000 (time 1713078517.38 total 26.95 last 703.29) - /unlink 26460 (time 1713078527.38 total 36.95 last 645.87) - /unlink 30000 (time 1713078533.42 total 42.99 last 586.05) - /unlink 35276 (time 1713078543.42 total 53.00 last 527.53) - /unlink 40000 (time 1713078552.24 total 61.81 last 535.82) - /unlink 46416 (time 1713078562.24 total 71.81 last 641.51) - /unlink 50000 (time 1713078568.34 total 77.91 last 587.55) - /unlink 55993 (time 1713078578.34 total 87.91 last 599.23) - /unlink 60000 (time 1713078585.75 total 95.32 last 541.26) total: 64768 /unlink in 104.37 seconds: 620.58 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server unloading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 PASS 106 (1464s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg314-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg314-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg314-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg314-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 oleg314-server: oleg314-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing load_modules_local oleg314-server: Loading modules from /home/green/git/lustre-release/lustre oleg314-server: detected 4 online CPUs by sysfs oleg314-server: Force libcfs to create 2 CPU partitions oleg314-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg314-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1548 sec ========== 03:10:52 (1713078652) === conf-sanity: start cleanup 03:10:52 (1713078652) === === conf-sanity: finish cleanup 03:10:52 (1713078652) ===