-----============= acceptance-small: conf-sanity ============----- Thu May 1 23:12:15 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg349-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg349-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg349-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg349-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg349-server oleg349-server: oleg349-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing load_modules_local oleg349-server: Loading modules from /home/green/git/lustre-release/lustre oleg349-server: detected 4 online CPUs by sysfs oleg349-server: MODOPTS_LIBCFS= oleg349-server: Force libcfs to create 2 CPU partitions oleg349-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg349-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg349-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg349-server: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg349-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:13:41 (1746155621) Stopping clients: oleg349-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg349-client.virtnet /mnt/lustre2 (opts:-f) oleg349-server: oleg349-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing load_modules_local oleg349-server: Loading modules from /home/green/git/lustre-release/lustre oleg349-server: detected 4 online CPUs by sysfs oleg349-server: MODOPTS_LIBCFS= oleg349-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg349-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg349-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1147 (time 1746155677.61 total 10.00 last 114.65) - open/close 2295 (time 1746155687.62 total 20.01 last 114.73) - open/close 3508 (time 1746155697.62 total 30.01 last 121.29) - open/close 4675 (time 1746155707.62 total 40.02 last 116.63) - open/close 5958 (time 1746155717.62 total 50.02 last 128.28) - open/close 7175 (time 1746155727.63 total 60.02 last 121.65) - open/close 8324 (time 1746155737.63 total 70.03 last 114.83) - open/close 9467 (time 1746155747.65 total 80.04 last 114.15) - open/close 10000 (time 1746155752.93 total 85.33 last 100.81) - open/close 11024 (time 1746155762.94 total 95.34 last 102.33) - open/close 12120 (time 1746155772.94 total 105.34 last 109.59) - open/close 13388 (time 1746155782.95 total 115.34 last 126.75) - open/close 14510 (time 1746155792.96 total 125.35 last 112.10) - open/close 15597 (time 1746155802.96 total 135.35 last 108.65) - open/close 16683 (time 1746155812.97 total 145.36 last 108.50) - open/close 17585 (time 1746155822.97 total 155.37 last 90.17) - open/close 18607 (time 1746155832.97 total 165.37 last 102.19) - open/close 19645 (time 1746155842.98 total 175.37 last 103.73) - open/close 20000 (time 1746155846.78 total 179.18 last 93.33) - open/close 20930 (time 1746155856.79 total 189.19 last 92.93) - open/close 22012 (time 1746155866.80 total 199.19 last 108.14) - open/close 23128 (time 1746155876.80 total 209.19 last 111.57) - open/close 24230 (time 1746155886.81 total 219.20 last 110.11) - open/close 25308 (time 1746155896.81 total 229.21 last 107.77) - open/close 26417 (time 1746155906.82 total 239.21 last 110.82) - open/close 27463 (time 1746155916.82 total 249.22 last 104.54) - open/close 28515 (time 1746155926.84 total 259.23 last 105.05) - open/close 29766 (time 1746155936.84 total 269.24 last 125.02) - open/close 30887 (time 1746155946.85 total 279.24 last 112.04) - open/close 32097 (time 1746155956.86 total 289.25 last 120.87) - open/close 33156 (time 1746155966.87 total 299.26 last 105.83) - open/close 34218 (time 1746155976.88 total 309.28 last 106.04) - open/close 35349 (time 1746155986.89 total 319.28 last 113.05) - open/close 36550 (time 1746155996.89 total 329.29 last 120.04) - open/close 37734 (time 1746156006.90 total 339.29 last 118.31) - open/close 39035 (time 1746156016.90 total 349.30 last 130.05) - open/close 40000 (time 1746156024.06 total 356.45 last 134.82) - open/close 41120 (time 1746156034.07 total 366.46 last 111.91) - open/close 42114 (time 1746156044.08 total 376.48 last 99.25) - open/close 43127 (time 1746156054.09 total 386.49 last 101.18) - open/close 44188 (time 1746156064.10 total 396.50 last 106.03) - open/close 45409 (time 1746156074.10 total 406.50 last 122.09) - open/close 46502 (time 1746156084.11 total 416.50 last 109.24) - open/close 47563 (time 1746156094.12 total 426.51 last 105.99) - open/close 48527 (time 1746156104.12 total 436.52 last 96.36) - open/close 49608 (time 1746156114.13 total 446.52 last 108.06) - open/close 50000 (time 1746156117.93 total 450.33 last 103.03) - open/close 51058 (time 1746156127.94 total 460.33 last 105.74) - open/close 52074 (time 1746156137.95 total 470.35 last 101.46) - open/close 53021 (time 1746156147.96 total 480.35 last 94.64) - open/close 54235 (time 1746156157.96 total 490.35 last 121.37) - open/close 55344 (time 1746156167.96 total 500.35 last 110.90) - open/close 56674 (time 1746156177.96 total 510.36 last 132.97) - open/close 57876 (time 1746156187.97 total 520.36 last 120.13) - open/close 58937 (time 1746156197.97 total 530.36 last 106.10) - open/close 60000 (time 1746156205.83 total 538.23 last 135.15) - open/close 60978 (time 1746156215.84 total 548.23 last 97.76) - open/close 62141 (time 1746156225.84 total 558.23 last 116.28) - open/close 63179 (time 1746156235.84 total 568.24 last 103.77) - open/close 64287 (time 1746156245.85 total 578.24 last 110.75) total: 64768 open/close in 582.13 seconds: 111.26 ops/second - /unlink 1756 (time 1746156261.52 total 10.00 last 175.55) - /unlink 3443 (time 1746156271.52 total 20.00 last 168.67) - /unlink 5088 (time 1746156281.53 total 30.01 last 164.43) - /unlink 7107 (time 1746156291.53 total 40.01 last 201.84) - /unlink 9275 (time 1746156301.53 total 50.01 last 216.73) - /unlink 10000 (time 1746156304.66 total 53.14 last 231.70) - /unlink 11612 (time 1746156314.67 total 63.15 last 161.17) - /unlink 13216 (time 1746156324.67 total 73.15 last 160.36) - /unlink 15096 (time 1746156334.67 total 83.15 last 187.95) - /unlink 16926 (time 1746156344.67 total 93.15 last 182.95) - /unlink 19152 (time 1746156354.68 total 103.16 last 222.57) - /unlink 20000 (time 1746156358.04 total 106.52 last 252.38) - /unlink 22010 (time 1746156368.04 total 116.52 last 200.93) - /unlink 24094 (time 1746156378.04 total 126.52 last 208.35) - /unlink 26223 (time 1746156388.04 total 136.52 last 212.82) - /unlink 28357 (time 1746156398.06 total 146.54 last 213.17) - /unlink 30000 (time 1746156406.99 total 155.47 last 183.82) - /unlink 31956 (time 1746156417.00 total 165.48 last 195.50) - /unlink 33851 (time 1746156427.00 total 175.48 last 189.44) - /unlink 35814 (time 1746156437.00 total 185.48 last 196.27) - /unlink 37981 (time 1746156447.01 total 195.49 last 216.63) - /unlink 40000 (time 1746156455.90 total 204.38 last 226.93) - /unlink 42322 (time 1746156465.90 total 214.38 last 232.19) - /unlink 44173 (time 1746156475.90 total 224.38 last 185.08) - /unlink 46272 (time 1746156485.91 total 234.39 last 209.88) - /unlink 48182 (time 1746156495.91 total 244.39 last 190.96) - /unlink 50000 (time 1746156504.08 total 252.56 last 222.60) - /unlink 53259 (time 1746156514.08 total 262.56 last 325.89) - /unlink 55948 (time 1746156524.08 total 272.56 last 268.83) - /unlink 58475 (time 1746156534.08 total 282.56 last 252.66) - /unlink 60000 (time 1746156538.91 total 287.39 last 315.60) - /unlink 63605 (time 1746156548.91 total 297.39 last 360.48) total: 64768 /unlink in 301.98 seconds: 214.48 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1824 (time 1746156583.94 total 10.01 last 182.28) - open/close 3414 (time 1746156593.94 total 20.01 last 159.00) - open/close 5100 (time 1746156603.95 total 30.01 last 168.55) - open/close 6401 (time 1746156613.95 total 40.01 last 130.06) - open/close 7787 (time 1746156623.95 total 50.02 last 138.58) - open/close 9278 (time 1746156633.95 total 60.02 last 149.10) - open/close 10000 (time 1746156638.96 total 65.02 last 144.18) - open/close 11401 (time 1746156648.96 total 75.03 last 140.04) - open/close 12609 (time 1746156658.97 total 85.03 last 120.78) - open/close 13773 (time 1746156668.97 total 95.03 last 116.36) - open/close 14916 (time 1746156678.97 total 105.03 last 114.29) - open/close 16085 (time 1746156688.98 total 115.04 last 116.85) - open/close 17219 (time 1746156698.98 total 125.04 last 113.35) - open/close 18494 (time 1746156708.99 total 135.05 last 127.42) - open/close 19742 (time 1746156718.99 total 145.05 last 124.77) - open/close 20000 (time 1746156721.25 total 147.32 last 113.85) - open/close 21184 (time 1746156731.26 total 157.32 last 118.39) - open/close 22413 (time 1746156741.26 total 167.32 last 122.87) - open/close 23655 (time 1746156751.26 total 177.32 last 124.17) - open/close 24841 (time 1746156761.26 total 187.33 last 118.57) - open/close 26186 (time 1746156771.27 total 197.33 last 134.47) - open/close 27425 (time 1746156781.27 total 207.33 last 123.87) - open/close 28819 (time 1746156791.28 total 217.34 last 139.29) - open/close 30000 (time 1746156800.26 total 226.32 last 131.51) - open/close 31336 (time 1746156810.26 total 236.32 last 133.56) - open/close 32704 (time 1746156820.26 total 246.33 last 136.76) - open/close 33902 (time 1746156830.27 total 256.33 last 119.70) - open/close 34971 (time 1746156840.27 total 266.34 last 106.87) - open/close 36051 (time 1746156850.28 total 276.34 last 107.98) - open/close 37191 (time 1746156860.28 total 286.34 last 113.94) - open/close 38304 (time 1746156870.29 total 296.36 last 111.14) - open/close 39382 (time 1746156880.30 total 306.36 last 107.79) - open/close 40000 (time 1746156885.99 total 312.05 last 108.61) - open/close 41174 (time 1746156896.00 total 322.06 last 117.28) - open/close 42353 (time 1746156906.00 total 332.06 last 117.88) - open/close 43338 (time 1746156916.00 total 342.06 last 98.48) - open/close 44428 (time 1746156926.00 total 352.07 last 108.96) - open/close 45468 (time 1746156936.01 total 362.07 last 103.98) - open/close 46484 (time 1746156946.01 total 372.07 last 101.55) - open/close 47814 (time 1746156956.01 total 382.07 last 132.99) - open/close 48948 (time 1746156966.02 total 392.09 last 113.27) - open/close 49921 (time 1746156976.02 total 402.09 last 97.29) - open/close 51171 (time 1746156986.03 total 412.10 last 124.90) - open/close 52379 (time 1746156996.03 total 422.10 last 120.78) - open/close 53470 (time 1746157006.04 total 432.10 last 109.04) - open/close 54466 (time 1746157016.04 total 442.10 last 99.60) - open/close 55504 (time 1746157026.04 total 452.10 last 103.79) - open/close 56459 (time 1746157036.05 total 462.11 last 95.45) - open/close 57443 (time 1746157046.05 total 472.11 last 98.37) - open/close 58610 (time 1746157056.05 total 482.11 last 116.66) - open/close 59698 (time 1746157066.06 total 492.12 last 108.75) - open/close 60000 (time 1746157068.98 total 495.04 last 103.38) - open/close 61197 (time 1746157078.98 total 505.04 last 119.67) - open/close 62187 (time 1746157088.99 total 515.05 last 98.92) - open/close 63139 (time 1746157099.00 total 525.06 last 95.11) - open/close 64099 (time 1746157109.00 total 535.06 last 95.99) total: 64768 open/close in 542.23 seconds: 119.45 ops/second - /unlink 1630 (time 1746157128.52 total 10.00 last 162.98) - /unlink 3218 (time 1746157138.52 total 20.00 last 158.76) - /unlink 5238 (time 1746157148.52 total 30.01 last 201.96) - /unlink 6989 (time 1746157158.52 total 40.01 last 175.10) - /unlink 8695 (time 1746157168.53 total 50.01 last 170.49) - /unlink 10000 (time 1746157176.91 total 58.40 last 155.63) - /unlink 11745 (time 1746157186.92 total 68.40 last 174.44) - /unlink 13831 (time 1746157196.92 total 78.40 last 208.59) - /unlink 16131 (time 1746157206.92 total 88.40 last 230.00) - /unlink 18548 (time 1746157216.92 total 98.40 last 241.65) - /unlink 20000 (time 1746157223.21 total 104.70 last 230.80) - /unlink 22394 (time 1746157233.21 total 114.70 last 239.39) - /unlink 24983 (time 1746157243.21 total 124.70 last 258.86) - /unlink 27175 (time 1746157253.22 total 134.70 last 219.06) - /unlink 29290 (time 1746157263.22 total 144.71 last 211.47) - /unlink 30000 (time 1746157266.80 total 148.28 last 198.40) - /unlink 32047 (time 1746157276.81 total 158.29 last 204.60) - /unlink 34605 (time 1746157286.81 total 168.29 last 255.79) - /unlink 37250 (time 1746157296.81 total 178.29 last 264.38) - /unlink 39565 (time 1746157306.81 total 188.30 last 231.47) - /unlink 40000 (time 1746157308.89 total 190.37 last 209.33) - /unlink 41884 (time 1746157318.89 total 200.38 last 188.35) - /unlink 43850 (time 1746157328.89 total 210.38 last 196.56) - /unlink 46220 (time 1746157338.90 total 220.38 last 236.92) - /unlink 48553 (time 1746157348.90 total 230.38 last 233.23) - /unlink 50000 (time 1746157356.27 total 237.76 last 196.30) - /unlink 52341 (time 1746157366.27 total 247.76 last 234.10) - /unlink 54319 (time 1746157376.27 total 257.76 last 197.78) - /unlink 56559 (time 1746157386.28 total 267.76 last 223.87) - /unlink 58974 (time 1746157396.28 total 277.77 last 241.43) - /unlink 60000 (time 1746157399.75 total 281.23 last 296.28) - /unlink 62822 (time 1746157409.75 total 291.23 last 282.18) total: 64768 /unlink in 299.57 seconds: 216.20 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1538 (time 1746157446.02 total 10.00 last 153.74) - open/close 3584 (time 1746157456.02 total 20.01 last 204.53) - open/close 5373 (time 1746157466.03 total 30.01 last 178.80) - open/close 7077 (time 1746157476.03 total 40.01 last 170.37) - open/close 8600 (time 1746157486.04 total 50.02 last 152.17) - open/close 10000 (time 1746157493.71 total 57.70 last 182.42) - open/close 11838 (time 1746157503.71 total 67.70 last 183.79) - open/close 13720 (time 1746157513.71 total 77.70 last 188.20) - open/close 15594 (time 1746157523.71 total 87.70 last 187.39) - open/close 17223 (time 1746157533.72 total 97.70 last 162.82) - open/close 18600 (time 1746157543.72 total 107.70 last 137.69) - open/close 20000 (time 1746157553.08 total 117.06 last 149.57) - open/close 21491 (time 1746157563.08 total 127.07 last 149.06) - open/close 23156 (time 1746157573.08 total 137.07 last 166.48) - open/close 24805 (time 1746157583.09 total 147.08 last 164.74) - open/close 26512 (time 1746157593.10 total 157.08 last 170.66) - open/close 27963 (time 1746157603.10 total 167.08 last 145.08) - open/close 29420 (time 1746157613.10 total 177.08 last 145.68) - open/close 30000 (time 1746157617.49 total 181.47 last 132.14) - open/close 31175 (time 1746157627.50 total 191.48 last 117.39) - open/close 32460 (time 1746157637.50 total 201.48 last 128.48) - open/close 33876 (time 1746157647.50 total 211.49 last 141.54) - open/close 35455 (time 1746157657.51 total 221.49 last 157.83) - open/close 36881 (time 1746157667.52 total 231.50 last 142.48) - open/close 38364 (time 1746157677.52 total 241.50 last 148.25) - open/close 39905 (time 1746157687.52 total 251.51 last 154.04) - open/close 41305 (time 1746157697.53 total 261.51 last 139.94) - open/close 42762 (time 1746157707.53 total 271.52 last 145.62) - open/close 44296 (time 1746157717.53 total 281.52 last 153.36) - open/close 45628 (time 1746157727.54 total 291.52 last 133.18) - open/close 46921 (time 1746157737.54 total 301.52 last 129.27) - open/close 48246 (time 1746157747.54 total 311.53 last 132.44) - open/close 49421 (time 1746157757.55 total 321.53 last 117.45) - open/close 50000 (time 1746157762.83 total 326.82 last 109.57) - open/close 51322 (time 1746157772.83 total 336.82 last 132.16) - open/close 52649 (time 1746157782.84 total 346.82 last 132.68) - open/close 53839 (time 1746157792.84 total 356.82 last 119.00) - open/close 55217 (time 1746157802.84 total 366.82 last 137.75) - open/close 56362 (time 1746157812.85 total 376.83 last 114.42) - open/close 57527 (time 1746157822.85 total 386.84 last 116.44) - open/close 58646 (time 1746157832.85 total 396.84 last 111.89) - open/close 59732 (time 1746157842.86 total 406.85 last 108.49) - open/close 60000 (time 1746157845.10 total 409.09 last 119.54) - open/close 61025 (time 1746157855.11 total 419.09 last 102.49) - open/close 62071 (time 1746157865.11 total 429.09 last 104.55) - open/close 63299 (time 1746157875.12 total 439.10 last 122.74) - open/close 64554 (time 1746157885.12 total 449.11 last 125.42) total: 64768 open/close in 450.75 seconds: 143.69 ops/second - /unlink 1907 (time 1746157898.26 total 10.00 last 190.65) - /unlink 3791 (time 1746157908.26 total 20.00 last 188.40) - /unlink 5881 (time 1746157918.27 total 30.01 last 208.92) - /unlink 8416 (time 1746157928.27 total 40.01 last 253.45) - /unlink 10000 (time 1746157935.59 total 47.33 last 216.30) - /unlink 12249 (time 1746157945.59 total 57.33 last 224.85) - /unlink 14379 (time 1746157955.60 total 67.34 last 212.94) - /unlink 16334 (time 1746157965.60 total 77.34 last 195.44) - /unlink 18255 (time 1746157975.60 total 87.34 last 192.09) - /unlink 20000 (time 1746157983.93 total 95.67 last 209.54) - /unlink 22076 (time 1746157993.93 total 105.67 last 207.57) - /unlink 24291 (time 1746158003.93 total 115.67 last 221.45) - /unlink 26621 (time 1746158013.93 total 125.68 last 232.93) - /unlink 28894 (time 1746158023.94 total 135.68 last 227.20) - /unlink 30000 (time 1746158029.78 total 141.52 last 189.38) - /unlink 32108 (time 1746158039.79 total 151.53 last 210.62) - /unlink 34288 (time 1746158049.79 total 161.53 last 217.97) - /unlink 36426 (time 1746158059.79 total 171.53 last 213.78) - /unlink 38311 (time 1746158069.79 total 181.53 last 188.45) - /unlink 40000 (time 1746158079.56 total 191.31 last 172.83) - /unlink 42050 (time 1746158089.57 total 201.31 last 204.99) - /unlink 44968 (time 1746158099.57 total 211.31 last 291.72) - /unlink 48858 (time 1746158109.57 total 221.31 last 388.99) - /unlink 50000 (time 1746158112.01 total 223.75 last 468.31) - /unlink 52929 (time 1746158122.01 total 233.75 last 292.89) - /unlink 55241 (time 1746158132.01 total 243.75 last 231.15) - /unlink 57491 (time 1746158142.01 total 253.75 last 224.94) - /unlink 59402 (time 1746158152.01 total 263.76 last 191.05) - /unlink 60000 (time 1746158154.86 total 266.60 last 210.17) - /unlink 61951 (time 1746158164.86 total 276.60 last 195.04) - /unlink 64036 (time 1746158174.87 total 286.61 last 208.44) total: 64768 /unlink in 289.62 seconds: 223.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1762 (time 1746158207.30 total 10.01 last 176.09) - open/close 3173 (time 1746158217.30 total 20.01 last 141.06) - open/close 4637 (time 1746158227.31 total 30.02 last 146.31) - open/close 6165 (time 1746158237.31 total 40.02 last 152.72) - open/close 7981 (time 1746158247.32 total 50.03 last 181.52) - open/close 9375 (time 1746158257.32 total 60.03 last 139.34) - open/close 10000 (time 1746158261.73 total 64.44 last 141.71) - open/close 11811 (time 1746158271.74 total 74.45 last 181.01) - open/close 13095 (time 1746158281.74 total 84.45 last 128.38) - open/close 14634 (time 1746158291.75 total 94.45 last 153.79) - open/close 16537 (time 1746158301.75 total 104.46 last 190.23) - open/close 19077 (time 1746158311.75 total 114.46 last 253.98) - open/close 20000 (time 1746158315.61 total 118.31 last 239.39) - open/close 22134 (time 1746158325.61 total 128.32 last 213.37) - open/close 23805 (time 1746158335.61 total 138.32 last 167.07) - open/close 25357 (time 1746158345.61 total 148.32 last 155.15) - open/close 26989 (time 1746158355.62 total 158.33 last 163.12) - open/close 28561 (time 1746158365.62 total 168.33 last 157.16) - open/close 30000 (time 1746158370.47 total 173.17 last 296.90) - open/close 32055 (time 1746158380.47 total 183.17 last 205.49) - open/close 34278 (time 1746158390.47 total 193.18 last 222.30) - open/close 35829 (time 1746158400.47 total 203.18 last 154.98) - open/close 37334 (time 1746158410.48 total 213.18 last 150.48) - open/close 39295 (time 1746158420.48 total 223.18 last 196.10) - open/close 40000 (time 1746158425.48 total 228.18 last 141.04) - open/close 41389 (time 1746158435.48 total 238.19 last 138.83) - open/close 42896 (time 1746158445.49 total 248.20 last 150.56) - open/close 44644 (time 1746158455.49 total 258.20 last 174.78) - open/close 46219 (time 1746158465.49 total 268.20 last 157.46) - open/close 48052 (time 1746158475.50 total 278.20 last 183.25) - open/close 49434 (time 1746158485.50 total 288.21 last 138.13) - open/close 50000 (time 1746158489.09 total 291.80 last 157.81) - open/close 51688 (time 1746158499.09 total 301.80 last 168.77) - open/close 53723 (time 1746158509.09 total 311.80 last 203.45) - open/close 55955 (time 1746158519.10 total 321.81 last 223.04) - open/close 58002 (time 1746158529.10 total 331.81 last 204.61) - open/close 60000 (time 1746158538.76 total 341.47 last 206.86) - open/close 62201 (time 1746158548.76 total 351.47 last 220.07) - open/close 63607 (time 1746158558.77 total 361.47 last 140.57) total: 64768 open/close in 368.87 seconds: 175.58 ops/second - /unlink 3600 (time 1746158577.28 total 10.00 last 359.90) - /unlink 6052 (time 1746158587.28 total 20.00 last 245.20) - /unlink 8774 (time 1746158597.28 total 30.00 last 272.19) - /unlink 10000 (time 1746158601.66 total 34.39 last 279.63) - /unlink 12767 (time 1746158611.66 total 44.39 last 276.64) - /unlink 15529 (time 1746158621.67 total 54.39 last 276.07) - /unlink 17946 (time 1746158631.67 total 64.40 last 241.64) - /unlink 20000 (time 1746158640.70 total 73.43 last 227.39) - /unlink 22529 (time 1746158650.71 total 83.43 last 252.84) - /unlink 25019 (time 1746158660.71 total 93.43 last 248.99) - /unlink 27472 (time 1746158670.71 total 103.43 last 245.25) - /unlink 29766 (time 1746158680.71 total 113.44 last 229.38) - /unlink 31879 (time 1746158690.71 total 123.44 last 211.28) - /unlink 33745 (time 1746158700.72 total 133.44 last 186.46) - /unlink 35714 (time 1746158710.73 total 143.45 last 196.73) - /unlink 37870 (time 1746158720.73 total 153.45 last 215.56) - /unlink 40000 (time 1746158729.29 total 162.02 last 248.73) - /unlink 42386 (time 1746158739.29 total 172.02 last 238.57) - /unlink 44605 (time 1746158749.29 total 182.02 last 221.88) - /unlink 47887 (time 1746158759.30 total 192.02 last 328.15) - /unlink 50000 (time 1746158765.40 total 198.13 last 346.03) - /unlink 53277 (time 1746158775.40 total 208.13 last 327.65) - /unlink 55561 (time 1746158785.41 total 218.13 last 228.33) - /unlink 58315 (time 1746158795.41 total 228.13 last 275.39) - /unlink 60000 (time 1746158802.41 total 235.14 last 240.60) - /unlink 62852 (time 1746158812.41 total 245.14 last 285.11) total: 64768 /unlink in 249.90 seconds: 259.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1369 (time 1746158843.78 total 10.00 last 136.87) - open/close 3006 (time 1746158853.81 total 20.03 last 163.27) - open/close 4631 (time 1746158863.81 total 30.03 last 162.41) - open/close 5913 (time 1746158873.82 total 40.04 last 128.16) - open/close 7122 (time 1746158883.82 total 50.04 last 120.84) - open/close 8428 (time 1746158893.83 total 60.05 last 130.49) - open/close 9634 (time 1746158903.83 total 70.05 last 120.55) - open/close 10000 (time 1746158906.66 total 72.88 last 129.48) - open/close 11168 (time 1746158916.67 total 82.89 last 116.73) - open/close 12466 (time 1746158926.67 total 92.89 last 129.70) - open/close 13699 (time 1746158936.68 total 102.90 last 123.24) - open/close 14899 (time 1746158946.68 total 112.90 last 119.95) - open/close 16247 (time 1746158956.69 total 122.91 last 134.66) - open/close 17475 (time 1746158966.69 total 132.92 last 122.77) - open/close 18586 (time 1746158976.70 total 142.92 last 111.03) - open/close 19756 (time 1746158986.70 total 152.92 last 116.99) - open/close 20000 (time 1746158989.13 total 155.35 last 100.47) - open/close 21136 (time 1746158999.14 total 165.36 last 113.46) - open/close 22363 (time 1746159009.15 total 175.37 last 122.64) - open/close 23769 (time 1746159019.15 total 185.37 last 140.56) - open/close 25086 (time 1746159029.16 total 195.38 last 131.61) - open/close 26358 (time 1746159039.16 total 205.38 last 127.14) - open/close 27645 (time 1746159049.16 total 215.39 last 128.66) - open/close 28911 (time 1746159059.17 total 225.39 last 126.57) - open/close 30000 (time 1746159067.84 total 234.06 last 125.56) - open/close 31341 (time 1746159077.84 total 244.06 last 134.10) - open/close 32556 (time 1746159087.84 total 254.07 last 121.46) - open/close 33591 (time 1746159097.84 total 264.07 last 103.50) - open/close 34667 (time 1746159107.85 total 274.07 last 107.58) - open/close 35894 (time 1746159117.85 total 284.07 last 122.66) - open/close 37162 (time 1746159127.85 total 294.08 last 126.74) - open/close 38379 (time 1746159137.86 total 304.08 last 121.68) - open/close 39680 (time 1746159147.86 total 314.08 last 130.06) - open/close 40000 (time 1746159150.29 total 316.51 last 131.56) - open/close 41375 (time 1746159160.29 total 326.51 last 137.47) - open/close 42697 (time 1746159170.29 total 336.51 last 132.20) - open/close 44199 (time 1746159180.30 total 346.52 last 150.16) - open/close 45492 (time 1746159190.30 total 356.52 last 129.22) - open/close 46768 (time 1746159200.32 total 366.54 last 127.40) - open/close 48035 (time 1746159210.32 total 376.54 last 126.69) - open/close 49149 (time 1746159220.32 total 386.54 last 111.34) - open/close 50000 (time 1746159227.74 total 393.96 last 114.70) - open/close 51109 (time 1746159237.75 total 403.97 last 110.84) - open/close 52216 (time 1746159247.75 total 413.97 last 110.66) - open/close 53459 (time 1746159257.75 total 423.97 last 124.28) - open/close 54866 (time 1746159267.75 total 433.98 last 140.68) - open/close 56291 (time 1746159277.76 total 443.98 last 142.47) - open/close 57568 (time 1746159287.76 total 453.98 last 127.68) - open/close 58983 (time 1746159297.76 total 463.98 last 141.47) - open/close 60000 (time 1746159305.41 total 471.63 last 132.99) - open/close 61171 (time 1746159315.41 total 481.63 last 117.04) - open/close 62386 (time 1746159325.42 total 491.64 last 121.44) - open/close 63478 (time 1746159335.42 total 501.64 last 109.15) - open/close 64766 (time 1746159345.42 total 511.65 last 128.77) total: 64768 open/close in 511.65 seconds: 126.59 ops/second - /unlink 1895 (time 1746159357.44 total 10.00 last 189.45) - /unlink 3623 (time 1746159367.44 total 20.00 last 172.77) - /unlink 5287 (time 1746159377.45 total 30.01 last 166.25) - /unlink 7057 (time 1746159387.45 total 40.02 last 176.96) - /unlink 9238 (time 1746159397.45 total 50.02 last 218.01) - /unlink 10000 (time 1746159400.26 total 52.83 last 271.52) - /unlink 11975 (time 1746159410.26 total 62.83 last 197.44) - /unlink 13970 (time 1746159420.26 total 72.83 last 199.50) - /unlink 16112 (time 1746159430.26 total 82.83 last 214.18) - /unlink 18118 (time 1746159440.27 total 92.84 last 200.50) - /unlink 19862 (time 1746159450.27 total 102.84 last 174.33) - /unlink 21605 (time 1746159460.28 total 112.85 last 174.11) - /unlink 23491 (time 1746159470.29 total 122.85 last 188.56) - /unlink 25530 (time 1746159480.29 total 132.85 last 203.87) - /unlink 27389 (time 1746159490.29 total 142.86 last 185.86) - /unlink 29308 (time 1746159500.29 total 152.86 last 191.87) - /unlink 30000 (time 1746159503.93 total 156.50 last 190.23) - /unlink 31827 (time 1746159513.93 total 166.50 last 182.70) - /unlink 33538 (time 1746159523.94 total 176.51 last 170.93) - /unlink 35044 (time 1746159533.97 total 186.54 last 150.08) - /unlink 36858 (time 1746159543.98 total 196.55 last 181.30) - /unlink 38860 (time 1746159553.98 total 206.55 last 200.15) - /unlink 40000 (time 1746159559.95 total 212.52 last 191.02) - /unlink 41835 (time 1746159569.95 total 222.52 last 183.45) - /unlink 43763 (time 1746159579.95 total 232.52 last 192.79) - /unlink 45714 (time 1746159589.96 total 242.52 last 194.99) - /unlink 47598 (time 1746159599.96 total 252.53 last 188.32) - /unlink 50000 (time 1746159609.76 total 262.32 last 245.22) - /unlink 52330 (time 1746159619.76 total 272.33 last 232.95) - /unlink 54843 (time 1746159629.76 total 282.33 last 251.28) - /unlink 57919 (time 1746159639.76 total 292.33 last 307.58) - /unlink 60000 (time 1746159648.15 total 300.72 last 248.08) - /unlink 62820 (time 1746159658.15 total 310.72 last 281.95) total: 64768 /unlink in 317.40 seconds: 204.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg349-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server unloading modules via unload_modules_local on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2 PASS 106 (4078s) Stopping clients: oleg349-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg349-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2 oleg349-server: oleg349-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing load_modules_local oleg349-server: Loading modules from /home/green/git/lustre-release/lustre oleg349-server: detected 4 online CPUs by sysfs oleg349-server: MODOPTS_LIBCFS= oleg349-server: Force libcfs to create 2 CPU partitions oleg349-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg349-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg349-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4196 sec ========== 00:22:12 (1746159732) === conf-sanity: start cleanup 00:22:13 (1746159733) === === conf-sanity: finish cleanup 00:22:14 (1746159734) ===