-----============= acceptance-small: conf-sanity ============----- Tue Jun 4 23:04:23 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg459-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg459-server oleg459-server: oleg459-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: detected 4 online CPUs by sysfs oleg459-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg459-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg459-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg459-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:05:22 (1717556722) Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) oleg459-server: oleg459-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: detected 4 online CPUs by sysfs oleg459-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg459-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg459-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock oleg459-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5640 (time 1717556755.39 total 10.00 last 563.90) - open/close 10000 (time 1717556763.23 total 17.84 last 556.56) - open/close 15499 (time 1717556773.23 total 27.84 last 549.86) - open/close 20000 (time 1717556781.54 total 36.15 last 541.58) - open/close 25479 (time 1717556791.54 total 46.15 last 547.87) - open/close 30000 (time 1717556799.94 total 54.55 last 538.31) - open/close 35355 (time 1717556809.94 total 64.55 last 535.44) - open/close 40000 (time 1717556818.46 total 73.07 last 545.00) - open/close 45343 (time 1717556828.46 total 83.07 last 534.28) - open/close 50000 (time 1717556837.27 total 91.88 last 528.92) - open/close 55281 (time 1717556847.27 total 101.88 last 528.07) - open/close 60000 (time 1717556856.69 total 111.30 last 500.88) total: 64768 open/close in 120.76 seconds: 536.36 ops/second - /unlink 7763 (time 1717556877.10 total 10.00 last 776.25) - /unlink 10000 (time 1717556879.63 total 12.53 last 885.80) - /unlink 18729 (time 1717556889.63 total 22.53 last 872.85) - /unlink 27515 (time 1717556899.63 total 32.53 last 878.51) - /unlink 30000 (time 1717556902.49 total 35.39 last 869.36) - /unlink 38955 (time 1717556912.49 total 45.39 last 895.46) - /unlink 47985 (time 1717556922.49 total 55.39 last 902.98) - /unlink 50000 (time 1717556924.67 total 57.57 last 924.39) - /unlink 59126 (time 1717556934.67 total 67.57 last 912.57) total: 64768 /unlink in 73.62 seconds: 879.79 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4865 (time 1717556974.05 total 10.00 last 486.46) - open/close 10000 (time 1717556983.74 total 19.69 last 529.93) - open/close 14864 (time 1717556993.74 total 29.69 last 486.35) - open/close 20000 (time 1717557003.73 total 39.67 last 514.47) - open/close 25168 (time 1717557013.73 total 49.67 last 516.79) - open/close 30000 (time 1717557023.28 total 59.23 last 505.55) - open/close 35259 (time 1717557033.28 total 69.23 last 525.88) - open/close 40000 (time 1717557042.28 total 78.23 last 527.26) - open/close 45063 (time 1717557052.28 total 88.23 last 506.25) - open/close 49854 (time 1717557062.28 total 98.23 last 479.07) - open/close 54751 (time 1717557072.28 total 108.23 last 489.67) - open/close 59112 (time 1717557082.28 total 118.23 last 436.05) - open/close 64139 (time 1717557092.28 total 128.23 last 502.68) total: 64768 open/close in 129.49 seconds: 500.18 ops/second - /unlink 7403 (time 1717557104.47 total 10.00 last 740.26) - /unlink 10000 (time 1717557107.79 total 13.33 last 780.81) - /unlink 18660 (time 1717557117.79 total 23.33 last 865.93) - /unlink 26911 (time 1717557127.79 total 33.33 last 825.04) - /unlink 30000 (time 1717557131.94 total 37.47 last 745.54) - /unlink 37058 (time 1717557141.94 total 47.47 last 705.74) - /unlink 40000 (time 1717557145.19 total 50.73 last 903.98) - /unlink 48607 (time 1717557155.19 total 60.73 last 860.59) - /unlink 56879 (time 1717557165.20 total 70.73 last 827.12) - /unlink 60000 (time 1717557169.21 total 74.74 last 778.08) total: 64768 /unlink in 80.00 seconds: 809.58 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5449 (time 1717557204.25 total 10.00 last 544.89) - open/close 10000 (time 1717557212.66 total 18.41 last 540.90) - open/close 15162 (time 1717557222.66 total 28.42 last 516.11) - open/close 19673 (time 1717557232.66 total 38.42 last 451.07) - open/close 23948 (time 1717557242.66 total 48.42 last 427.46) - open/close 28789 (time 1717557252.66 total 58.42 last 484.09) - open/close 30000 (time 1717557255.00 total 60.75 last 518.24) - open/close 34919 (time 1717557265.00 total 70.76 last 491.79) - open/close 38537 (time 1717557275.00 total 80.76 last 361.75) - open/close 40000 (time 1717557279.01 total 84.77 last 364.80) - open/close 43362 (time 1717557289.02 total 94.77 last 336.07) - open/close 47691 (time 1717557299.02 total 104.77 last 432.81) - open/close 50000 (time 1717557305.67 total 111.43 last 346.92) - open/close 53401 (time 1717557315.68 total 121.43 last 340.00) - open/close 57447 (time 1717557325.68 total 131.43 last 404.56) - open/close 60000 (time 1717557334.40 total 140.15 last 292.76) - open/close 63008 (time 1717557344.40 total 150.16 last 300.70) total: 64768 open/close in 157.22 seconds: 411.97 ops/second - /unlink 5333 (time 1717557362.52 total 10.00 last 533.27) - /unlink 10000 (time 1717557369.68 total 17.16 last 651.60) - /unlink 15286 (time 1717557379.68 total 27.16 last 528.55) - /unlink 20000 (time 1717557389.05 total 36.54 last 502.99) - /unlink 26385 (time 1717557399.05 total 46.54 last 638.43) - /unlink 30000 (time 1717557404.09 total 51.57 last 718.03) - /unlink 36793 (time 1717557414.09 total 61.57 last 679.14) - /unlink 40000 (time 1717557420.76 total 68.24 last 480.99) - /unlink 45308 (time 1717557430.76 total 78.24 last 530.75) - /unlink 50000 (time 1717557438.78 total 86.26 last 585.01) - /unlink 58375 (time 1717557448.78 total 96.26 last 837.44) total: 64768 /unlink in 104.21 seconds: 621.48 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4444 (time 1717557484.92 total 10.00 last 444.39) - open/close 8167 (time 1717557494.92 total 20.00 last 372.28) - open/close 10000 (time 1717557500.57 total 25.65 last 324.55) - open/close 14077 (time 1717557510.57 total 35.65 last 407.69) - open/close 18097 (time 1717557520.57 total 45.65 last 401.95) - open/close 20000 (time 1717557527.13 total 52.21 last 289.89) - open/close 23850 (time 1717557537.13 total 62.22 last 384.97) - open/close 27536 (time 1717557547.13 total 72.22 last 368.60) - open/close 30000 (time 1717557554.77 total 79.85 last 322.73) - open/close 34249 (time 1717557564.77 total 89.85 last 424.78) - open/close 38427 (time 1717557574.77 total 99.85 last 417.78) - open/close 40000 (time 1717557580.52 total 105.61 last 273.44) - open/close 42964 (time 1717557590.53 total 115.61 last 296.35) - open/close 45643 (time 1717557600.53 total 125.61 last 267.89) - open/close 48528 (time 1717557610.53 total 135.61 last 288.48) - open/close 50000 (time 1717557615.02 total 140.10 last 327.84) - open/close 54003 (time 1717557625.02 total 150.10 last 400.24) - open/close 57057 (time 1717557635.02 total 160.10 last 305.32) - open/close 59606 (time 1717557645.02 total 170.11 last 254.83) - open/close 62310 (time 1717557655.02 total 180.11 last 270.40) total: 64768 open/close in 187.88 seconds: 344.73 ops/second - /unlink 6033 (time 1717557674.04 total 10.00 last 603.28) - /unlink 10000 (time 1717557681.87 total 17.84 last 506.17) - /unlink 14825 (time 1717557691.87 total 27.84 last 482.49) - /unlink 19895 (time 1717557701.87 total 37.84 last 506.99) - /unlink 24832 (time 1717557711.87 total 47.84 last 493.68) - /unlink 29637 (time 1717557721.87 total 57.84 last 480.43) - /unlink 34661 (time 1717557731.88 total 67.84 last 502.39) - /unlink 39983 (time 1717557741.88 total 77.84 last 532.16) - /unlink 44960 (time 1717557751.88 total 87.84 last 497.62) - /unlink 49895 (time 1717557761.88 total 97.84 last 493.46) - /unlink 54965 (time 1717557771.88 total 107.84 last 506.95) - /unlink 59891 (time 1717557781.88 total 117.85 last 492.50) total: 64768 /unlink in 127.67 seconds: 507.32 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2638 (time 1717557820.01 total 10.00 last 263.79) - open/close 5272 (time 1717557830.01 total 20.00 last 263.33) - open/close 7823 (time 1717557840.01 total 30.01 last 255.03) - open/close 10000 (time 1717557846.92 total 36.91 last 315.24) - open/close 12492 (time 1717557856.92 total 46.92 last 249.11) - open/close 15372 (time 1717557866.92 total 56.92 last 287.92) - open/close 19307 (time 1717557876.92 total 66.92 last 393.48) - open/close 20000 (time 1717557879.43 total 69.42 last 276.61) - open/close 22511 (time 1717557889.43 total 79.43 last 251.06) - open/close 24978 (time 1717557899.43 total 89.43 last 246.62) - open/close 27445 (time 1717557909.44 total 99.43 last 246.62) - open/close 29922 (time 1717557919.44 total 109.43 last 247.64) - open/close 32478 (time 1717557929.44 total 119.44 last 255.52) - open/close 34999 (time 1717557939.44 total 129.44 last 252.09) - open/close 37570 (time 1717557949.44 total 139.44 last 257.05) - open/close 40000 (time 1717557959.15 total 149.15 last 250.28) - open/close 42548 (time 1717557969.16 total 159.15 last 254.70) - open/close 45063 (time 1717557979.16 total 169.15 last 251.48) - open/close 47511 (time 1717557989.16 total 179.16 last 244.74) - open/close 50000 (time 1717557998.74 total 188.74 last 259.83) - open/close 52583 (time 1717558008.74 total 198.74 last 258.29) - open/close 55045 (time 1717558018.74 total 208.74 last 246.20) - open/close 57572 (time 1717558028.74 total 218.74 last 252.63) - open/close 60000 (time 1717558038.66 total 228.65 last 244.88) - open/close 62478 (time 1717558048.66 total 238.66 last 247.74) total: 64768 open/close in 247.22 seconds: 261.98 ops/second - /unlink 4951 (time 1717558068.50 total 10.00 last 494.99) - /unlink 9872 (time 1717558078.50 total 20.00 last 492.03) - /unlink 14735 (time 1717558088.50 total 30.00 last 486.25) - /unlink 19479 (time 1717558098.51 total 40.01 last 474.35) - /unlink 24234 (time 1717558108.51 total 50.01 last 475.48) - /unlink 28958 (time 1717558118.51 total 60.01 last 472.36) - /unlink 34207 (time 1717558128.51 total 70.01 last 524.85) - /unlink 39377 (time 1717558138.51 total 80.01 last 516.97) - /unlink 45157 (time 1717558148.51 total 90.01 last 577.99) - /unlink 50000 (time 1717558157.54 total 99.04 last 536.04) - /unlink 54826 (time 1717558167.55 total 109.05 last 482.50) - /unlink 60000 (time 1717558175.43 total 116.93 last 656.52) total: 64768 /unlink in 122.21 seconds: 529.96 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server unloading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 PASS 106 (1499s) Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 oleg459-server: oleg459-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: detected 4 online CPUs by sysfs oleg459-server: Force libcfs to create 2 CPU partitions oleg459-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg459-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1581 sec ========== 23:30:44 (1717558244) === conf-sanity: start cleanup 23:30:45 (1717558245) === === conf-sanity: finish cleanup 23:30:45 (1717558245) ===