-----============= acceptance-small: conf-sanity ============----- Mon Dec 22 09:02:12 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg235-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball Stopping clients: oleg235-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg235-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg235-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg235-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg235-server oleg235-server: oleg235-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing load_modules_local oleg235-server: Loading modules from /home/green/git/lustre-release/lustre oleg235-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg235-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg235-server: detected 4 online CPUs by sysfs oleg235-server: MODOPTS_LIBCFS= oleg235-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg235-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg235-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg235-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg235-server: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg235-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg235-server: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg235-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg235-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:05:42 (1766412342) Stopping clients: oleg235-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg235-client.virtnet /mnt/lustre2 (opts:-f) oleg235-server: oleg235-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing load_modules_local oleg235-server: Loading modules from /home/green/git/lustre-release/lustre oleg235-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg235-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg235-server: detected 4 online CPUs by sysfs oleg235-server: MODOPTS_LIBCFS= oleg235-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg235-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg235-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg235-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 692 (time 1766412470.64 total 10.00 last 69.17) - open/close 1385 (time 1766412480.65 total 20.01 last 69.25) - open/close 2090 (time 1766412490.65 total 30.01 last 70.48) - open/close 2796 (time 1766412500.66 total 40.02 last 70.59) - open/close 3489 (time 1766412510.66 total 50.02 last 69.29) - open/close 4205 (time 1766412520.66 total 60.02 last 71.58) - open/close 4937 (time 1766412530.66 total 70.02 last 73.16) - open/close 5719 (time 1766412540.67 total 80.03 last 78.17) - open/close 6405 (time 1766412550.68 total 90.04 last 68.54) - open/close 7163 (time 1766412560.69 total 100.05 last 75.74) - open/close 7802 (time 1766412570.70 total 110.06 last 63.82) - open/close 8473 (time 1766412580.70 total 120.06 last 67.08) - open/close 9182 (time 1766412590.72 total 130.08 last 70.77) - open/close 9982 (time 1766412600.72 total 140.08 last 79.99) - open/close 10669 (time 1766412610.73 total 150.10 last 68.61) - open/close 11384 (time 1766412620.74 total 160.10 last 71.49) - open/close 12045 (time 1766412630.74 total 170.10 last 66.06) - open/close 12772 (time 1766412640.75 total 180.11 last 72.64) - open/close 13533 (time 1766412650.77 total 190.13 last 75.98) - open/close 14296 (time 1766412660.78 total 200.14 last 76.21) - open/close 14999 (time 1766412670.78 total 210.14 last 70.26) - open/close 15600 (time 1766412680.79 total 220.15 last 60.08) - open/close 16266 (time 1766412690.79 total 230.15 last 66.57) - open/close 17001 (time 1766412700.80 total 240.16 last 73.44) - open/close 17743 (time 1766412710.80 total 250.16 last 74.19) - open/close 18426 (time 1766412720.82 total 260.18 last 68.21) - open/close 19075 (time 1766412730.83 total 270.19 last 64.78) - open/close 19702 (time 1766412740.84 total 280.20 last 62.65) - open/close 20000 (time 1766412744.98 total 284.34 last 71.94) - open/close 20682 (time 1766412754.99 total 294.35 last 68.16) - open/close 21376 (time 1766412765.00 total 304.36 last 69.35) - open/close 21983 (time 1766412775.00 total 314.36 last 60.66) - open/close 22586 (time 1766412785.02 total 324.38 last 60.18) - open/close 23261 (time 1766412795.03 total 334.39 last 67.48) - open/close 23963 (time 1766412805.03 total 344.39 last 70.17) - open/close 24621 (time 1766412815.04 total 354.40 last 65.75) - open/close 25355 (time 1766412825.05 total 364.41 last 73.29) - open/close 26027 (time 1766412835.06 total 374.42 last 67.18) - open/close 26708 (time 1766412845.07 total 384.43 last 68.03) - open/close 27472 (time 1766412855.08 total 394.44 last 76.28) - open/close 28236 (time 1766412865.09 total 404.45 last 76.35) - open/close 29005 (time 1766412875.11 total 414.47 last 76.77) - open/close 29633 (time 1766412885.11 total 424.47 last 62.79) - open/close 30000 (time 1766412890.42 total 429.78 last 69.14) - open/close 30713 (time 1766412900.43 total 439.79 last 71.22) - open/close 31378 (time 1766412910.43 total 449.79 last 66.49) - open/close 32051 (time 1766412920.43 total 459.79 last 67.30) - open/close 32603 (time 1766412930.44 total 469.80 last 55.14) - open/close 33272 (time 1766412940.44 total 479.80 last 66.88) - open/close 34017 (time 1766412950.46 total 489.82 last 74.41) - open/close 34778 (time 1766412960.47 total 499.83 last 75.97) - open/close 35467 (time 1766412970.48 total 509.84 last 68.87) - open/close 36173 (time 1766412980.48 total 519.84 last 70.55) - open/close 36937 (time 1766412990.49 total 529.85 last 76.37) - open/close 37734 (time 1766413000.49 total 539.85 last 79.66) - open/close 38520 (time 1766413010.51 total 549.87 last 78.48) - open/close 39292 (time 1766413020.51 total 559.87 last 77.16) - open/close 40000 (time 1766413029.84 total 569.20 last 75.89) - open/close 40732 (time 1766413039.85 total 579.21 last 73.17) - open/close 41406 (time 1766413049.85 total 589.21 last 67.38) - open/close 42045 (time 1766413059.85 total 599.21 last 63.89) - open/close 42677 (time 1766413069.85 total 609.21 last 63.19) - open/close 43377 (time 1766413079.86 total 619.22 last 69.95) - open/close 43984 (time 1766413089.88 total 629.24 last 60.59) - open/close 44583 (time 1766413099.88 total 639.24 last 59.88) - open/close 45234 (time 1766413109.88 total 649.24 last 65.09) - open/close 45937 (time 1766413119.89 total 659.25 last 70.25) - open/close 46651 (time 1766413129.92 total 669.28 last 71.15) - open/close 47374 (time 1766413139.95 total 679.31 last 72.11) - open/close 48149 (time 1766413149.95 total 689.31 last 77.50) - open/close 48880 (time 1766413159.95 total 699.31 last 73.08) - open/close 49561 (time 1766413169.97 total 709.33 last 68.01) - open/close 50000 (time 1766413176.31 total 715.67 last 69.20) - open/close 50698 (time 1766413186.32 total 725.68 last 69.76) - open/close 51381 (time 1766413196.33 total 735.69 last 68.24) - open/close 52092 (time 1766413206.33 total 745.69 last 71.09) - open/close 52779 (time 1766413216.34 total 755.70 last 68.61) - open/close 53323 (time 1766413226.36 total 765.72 last 54.27) - open/close 53957 (time 1766413236.37 total 775.73 last 63.37) - open/close 54675 (time 1766413246.38 total 785.74 last 71.75) - open/close 55348 (time 1766413256.39 total 795.75 last 67.19) - open/close 56010 (time 1766413266.39 total 805.75 last 66.20) - open/close 56687 (time 1766413276.41 total 815.78 last 67.55) - open/close 57302 (time 1766413286.43 total 825.79 last 61.44) - open/close 57903 (time 1766413296.43 total 835.79 last 60.06) - open/close 58586 (time 1766413306.44 total 845.80 last 68.23) - open/close 59267 (time 1766413316.46 total 855.82 last 67.96) - open/close 59874 (time 1766413326.48 total 865.84 last 60.62) - open/close 60000 (time 1766413328.70 total 868.06 last 56.59) - open/close 60709 (time 1766413338.71 total 878.07 last 70.88) - open/close 61477 (time 1766413348.71 total 888.07 last 76.73) - open/close 62234 (time 1766413358.72 total 898.08 last 75.65) - open/close 62872 (time 1766413368.73 total 908.09 last 63.75) - open/close 63512 (time 1766413378.74 total 918.10 last 63.95) - open/close 64181 (time 1766413388.74 total 928.10 last 66.88) total: 64768 open/close in 936.50 seconds: 69.16 ops/second - /unlink 744 (time 1766413411.68 total 10.01 last 74.32) - /unlink 1433 (time 1766413421.68 total 20.01 last 68.88) - /unlink 2083 (time 1766413431.70 total 30.03 last 64.92) - /unlink 2651 (time 1766413441.70 total 40.03 last 56.76) - /unlink 3282 (time 1766413451.71 total 50.04 last 63.05) - /unlink 3931 (time 1766413461.71 total 60.04 last 64.88) - /unlink 4503 (time 1766413471.73 total 70.07 last 57.08) - /unlink 5185 (time 1766413481.74 total 80.07 last 68.14) - /unlink 5970 (time 1766413491.76 total 90.09 last 78.34) - /unlink 6706 (time 1766413501.77 total 100.10 last 73.55) - /unlink 7351 (time 1766413511.79 total 110.12 last 64.38) - /unlink 8035 (time 1766413521.79 total 120.12 last 68.40) - /unlink 8780 (time 1766413531.80 total 130.13 last 74.43) - /unlink 9511 (time 1766413541.80 total 140.13 last 73.09) - /unlink 10000 (time 1766413548.58 total 146.91 last 72.11) - /unlink 10764 (time 1766413558.59 total 156.92 last 76.33) - /unlink 11498 (time 1766413568.61 total 166.94 last 73.28) - /unlink 12245 (time 1766413578.62 total 176.95 last 74.64) - /unlink 12993 (time 1766413588.63 total 186.96 last 74.69) - /unlink 13691 (time 1766413598.63 total 196.96 last 69.78) - /unlink 14397 (time 1766413608.64 total 206.97 last 70.56) - /unlink 15077 (time 1766413618.64 total 216.98 last 67.97) - /unlink 15755 (time 1766413628.65 total 226.98 last 67.77) - /unlink 16406 (time 1766413638.67 total 237.00 last 65.00) - /unlink 17111 (time 1766413648.67 total 247.00 last 70.48) - /unlink 17849 (time 1766413658.68 total 257.01 last 73.70) - /unlink 18581 (time 1766413668.70 total 267.03 last 73.11) - /unlink 19281 (time 1766413678.70 total 277.03 last 69.95) - /unlink 19973 (time 1766413688.72 total 287.05 last 69.09) - /unlink 20589 (time 1766413698.72 total 297.05 last 61.59) - /unlink 21258 (time 1766413708.74 total 307.07 last 66.78) - /unlink 21988 (time 1766413718.74 total 317.07 last 73.00) - /unlink 22710 (time 1766413728.75 total 327.08 last 72.12) - /unlink 23414 (time 1766413738.76 total 337.09 last 70.33) - /unlink 24142 (time 1766413748.76 total 347.09 last 72.77) - /unlink 24791 (time 1766413758.77 total 357.10 last 64.88) - /unlink 25446 (time 1766413768.78 total 367.11 last 65.40) - /unlink 26092 (time 1766413778.79 total 377.13 last 64.52) - /unlink 26741 (time 1766413788.81 total 387.14 last 64.79) - /unlink 27367 (time 1766413798.81 total 397.14 last 62.59) - /unlink 27988 (time 1766413808.83 total 407.17 last 61.97) - /unlink 28646 (time 1766413818.84 total 417.17 last 65.80) - /unlink 29287 (time 1766413828.86 total 427.19 last 63.96) - /unlink 29929 (time 1766413838.87 total 437.20 last 64.12) - /unlink 30519 (time 1766413848.88 total 447.21 last 58.94) - /unlink 31212 (time 1766413858.90 total 457.23 last 69.15) - /unlink 31898 (time 1766413868.91 total 467.24 last 68.56) - /unlink 32575 (time 1766413878.92 total 477.25 last 67.62) - /unlink 33311 (time 1766413888.94 total 487.27 last 73.49) - /unlink 33952 (time 1766413898.95 total 497.28 last 64.04) - /unlink 34569 (time 1766413908.96 total 507.29 last 61.61) - /unlink 35224 (time 1766413918.96 total 517.30 last 65.47) - /unlink 35834 (time 1766413928.97 total 527.30 last 60.95) - /unlink 36344 (time 1766413938.99 total 537.32 last 50.93) - /unlink 36943 (time 1766413949.00 total 547.33 last 59.82) - /unlink 37628 (time 1766413959.00 total 557.33 last 68.50) - /unlink 38339 (time 1766413969.02 total 567.35 last 70.98) - /unlink 39083 (time 1766413979.04 total 577.37 last 74.21) - /unlink 39792 (time 1766413989.05 total 587.39 last 70.83) - /unlink 40000 (time 1766413991.95 total 590.28 last 71.77) - /unlink 40704 (time 1766414001.96 total 600.29 last 70.37) - /unlink 41443 (time 1766414011.97 total 610.30 last 73.82) - /unlink 42184 (time 1766414021.98 total 620.31 last 74.02) - /unlink 42905 (time 1766414031.99 total 630.32 last 72.00) - /unlink 43672 (time 1766414042.00 total 640.33 last 76.67) - /unlink 44422 (time 1766414052.01 total 650.34 last 74.88) - /unlink 45146 (time 1766414062.02 total 660.35 last 72.36) - /unlink 45869 (time 1766414072.03 total 670.36 last 72.24) - /unlink 46616 (time 1766414082.03 total 680.36 last 74.69) - /unlink 47320 (time 1766414092.04 total 690.37 last 70.29) - /unlink 48054 (time 1766414102.06 total 700.39 last 73.24) - /unlink 48768 (time 1766414112.07 total 710.40 last 71.33) - /unlink 49517 (time 1766414122.08 total 720.41 last 74.86) - /unlink 50000 (time 1766414128.89 total 727.22 last 70.93) - /unlink 50720 (time 1766414138.89 total 737.22 last 71.99) - /unlink 51374 (time 1766414148.90 total 747.23 last 65.36) - /unlink 52128 (time 1766414158.90 total 757.23 last 75.33) - /unlink 52865 (time 1766414168.91 total 767.24 last 73.65) - /unlink 53561 (time 1766414178.92 total 777.25 last 69.55) - /unlink 54255 (time 1766414188.93 total 787.26 last 69.30) - /unlink 54947 (time 1766414198.96 total 797.29 last 69.03) - /unlink 55674 (time 1766414208.97 total 807.30 last 72.64) - /unlink 56472 (time 1766414218.97 total 817.30 last 79.79) - /unlink 57239 (time 1766414228.99 total 827.32 last 76.56) - /unlink 58020 (time 1766414238.99 total 837.32 last 78.04) - /unlink 58785 (time 1766414249.01 total 847.34 last 76.40) - /unlink 59522 (time 1766414259.01 total 857.34 last 73.64) - /unlink 60000 (time 1766414265.18 total 863.51 last 77.53) - /unlink 60780 (time 1766414275.19 total 873.52 last 77.90) - /unlink 61541 (time 1766414285.20 total 883.53 last 76.06) - /unlink 62264 (time 1766414295.21 total 893.54 last 72.24) - /unlink 62971 (time 1766414305.22 total 903.56 last 70.57) - /unlink 63691 (time 1766414315.23 total 913.56 last 71.96) - /unlink 64465 (time 1766414325.24 total 923.57 last 77.36) total: 64768 /unlink in 927.70 seconds: 69.82 ops/second Waiting for MDT destroys to complete oleg235-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 - open/close 738 (time 1766414354.44 total 10.00 last 73.79) - open/close 1487 (time 1766414364.44 total 20.01 last 74.84) - open/close 2244 (time 1766414374.45 total 30.02 last 75.65) - open/close 2962 (time 1766414384.45 total 40.02 last 71.78) - open/close 3768 (time 1766414394.46 total 50.02 last 80.57) - open/close 4464 (time 1766414404.47 total 60.04 last 69.50) - open/close 5197 (time 1766414414.47 total 70.04 last 73.29) - open/close 5959 (time 1766414424.49 total 80.06 last 76.04) - open/close 6707 (time 1766414434.50 total 90.06 last 74.77) - open/close 7403 (time 1766414444.50 total 100.07 last 69.58) - open/close 8137 (time 1766414454.50 total 110.07 last 73.39) - open/close 8903 (time 1766414464.51 total 120.08 last 76.53) - open/close 9594 (time 1766414474.51 total 130.08 last 69.09) - open/close 10000 (time 1766414479.64 total 135.21 last 79.09) - open/close 10730 (time 1766414489.65 total 145.22 last 72.93) - open/close 11511 (time 1766414499.66 total 155.23 last 78.02) - open/close 12196 (time 1766414509.67 total 165.24 last 68.46) - open/close 12867 (time 1766414519.68 total 175.25 last 67.04) - open/close 13514 (time 1766414529.69 total 185.25 last 64.65) - open/close 14129 (time 1766414539.70 total 195.26 last 61.43) - open/close 14792 (time 1766414549.70 total 205.27 last 66.28) - open/close 15420 (time 1766414559.71 total 215.27 last 62.76) - open/close 16139 (time 1766414569.71 total 225.28 last 71.88) - open/close 16953 (time 1766414579.72 total 235.29 last 81.30) - open/close 17752 (time 1766414589.72 total 245.29 last 79.88) - open/close 18617 (time 1766414599.73 total 255.29 last 86.47) - open/close 19458 (time 1766414609.73 total 265.30 last 84.08) - open/close 20000 (time 1766414616.42 total 271.99 last 81.00) - open/close 20789 (time 1766414626.42 total 281.99 last 78.88) - open/close 21584 (time 1766414636.43 total 292.00 last 79.43) - open/close 22372 (time 1766414646.44 total 302.00 last 78.78) - open/close 23136 (time 1766414656.45 total 312.02 last 76.27) - open/close 23822 (time 1766414666.46 total 322.03 last 68.54) - open/close 24582 (time 1766414676.47 total 332.03 last 75.95) - open/close 25333 (time 1766414686.48 total 342.04 last 75.04) - open/close 26080 (time 1766414696.48 total 352.05 last 74.66) - open/close 26866 (time 1766414706.49 total 362.06 last 78.53) - open/close 27715 (time 1766414716.50 total 372.06 last 84.82) - open/close 28504 (time 1766414726.50 total 382.07 last 78.87) - open/close 29161 (time 1766414736.52 total 392.08 last 65.60) - open/close 29891 (time 1766414746.52 total 402.09 last 72.97) - open/close 30684 (time 1766414756.52 total 412.09 last 79.27) - open/close 31328 (time 1766414766.54 total 422.11 last 64.28) - open/close 31892 (time 1766414776.55 total 432.11 last 56.37) - open/close 32572 (time 1766414786.55 total 442.11 last 67.99) - open/close 33318 (time 1766414796.56 total 452.13 last 74.51) - open/close 34175 (time 1766414806.57 total 462.13 last 85.65) - open/close 34939 (time 1766414816.58 total 472.15 last 76.31) - open/close 35749 (time 1766414826.59 total 482.15 last 80.95) - open/close 36538 (time 1766414836.59 total 492.16 last 78.87) - open/close 37238 (time 1766414846.59 total 502.16 last 69.96) - open/close 37981 (time 1766414856.61 total 512.17 last 74.22) - open/close 38750 (time 1766414866.61 total 522.18 last 76.86) - open/close 39567 (time 1766414876.61 total 532.18 last 81.69) - open/close 40000 (time 1766414882.35 total 537.92 last 75.43) - open/close 40825 (time 1766414892.37 total 547.93 last 82.39) - open/close 41625 (time 1766414902.37 total 557.94 last 79.94) - open/close 42429 (time 1766414912.38 total 567.94 last 80.38) - open/close 43214 (time 1766414922.39 total 577.95 last 78.43) - open/close 43973 (time 1766414932.39 total 587.95 last 75.89) - open/close 44621 (time 1766414942.40 total 597.96 last 64.72) - open/close 45381 (time 1766414952.40 total 607.96 last 76.00) - open/close 46232 (time 1766414962.40 total 617.97 last 85.09) - open/close 46936 (time 1766414972.41 total 627.97 last 70.33) - open/close 47683 (time 1766414982.42 total 637.98 last 74.64) - open/close 48374 (time 1766414992.43 total 648.00 last 69.00) - open/close 49117 (time 1766415002.44 total 658.00 last 74.26) - open/close 49953 (time 1766415012.45 total 668.01 last 83.52) - open/close 50782 (time 1766415022.45 total 678.02 last 82.86) - open/close 51625 (time 1766415032.47 total 688.03 last 84.18) - open/close 52472 (time 1766415042.47 total 698.03 last 84.69) - open/close 53296 (time 1766415052.48 total 708.05 last 82.29) - open/close 54037 (time 1766415062.49 total 718.06 last 74.01) - open/close 54866 (time 1766415072.49 total 728.06 last 82.89) - open/close 55724 (time 1766415082.50 total 738.07 last 85.74) - open/close 56467 (time 1766415092.50 total 748.07 last 74.28) - open/close 57228 (time 1766415102.52 total 758.08 last 76.00) - open/close 58001 (time 1766415112.54 total 768.10 last 77.14) - open/close 58741 (time 1766415122.54 total 778.11 last 73.98) - open/close 59580 (time 1766415132.55 total 788.11 last 83.82) - open/close 60000 (time 1766415137.40 total 792.96 last 86.59) - open/close 60752 (time 1766415147.40 total 802.97 last 75.19) - open/close 61509 (time 1766415157.42 total 812.99 last 75.51) - open/close 62298 (time 1766415167.43 total 823.00 last 78.86) - open/close 63151 (time 1766415177.43 total 833.00 last 85.30) - open/close 63940 (time 1766415187.44 total 843.01 last 78.83) - open/close 64748 (time 1766415197.45 total 853.01 last 80.73) total: 64768 open/close in 853.21 seconds: 75.91 ops/second - /unlink 794 (time 1766415211.20 total 10.02 last 79.22) - /unlink 1616 (time 1766415221.21 total 20.04 last 82.09) - /unlink 2371 (time 1766415231.23 total 30.05 last 75.40) - /unlink 3146 (time 1766415241.23 total 40.06 last 77.44) - /unlink 3914 (time 1766415251.24 total 50.06 last 76.78) - /unlink 4702 (time 1766415261.24 total 60.06 last 78.77) - /unlink 5513 (time 1766415271.24 total 70.06 last 81.10) - /unlink 6260 (time 1766415281.25 total 80.07 last 74.64) - /unlink 7024 (time 1766415291.26 total 90.08 last 76.30) - /unlink 7811 (time 1766415301.26 total 100.09 last 78.67) - /unlink 8685 (time 1766415311.27 total 110.09 last 87.35) - /unlink 9492 (time 1766415321.27 total 120.10 last 80.69) - /unlink 10000 (time 1766415327.28 total 126.11 last 84.50) - /unlink 10908 (time 1766415337.29 total 136.11 last 90.78) - /unlink 11803 (time 1766415347.29 total 146.11 last 89.50) - /unlink 12685 (time 1766415357.30 total 156.12 last 88.10) - /unlink 13486 (time 1766415367.31 total 166.13 last 80.00) - /unlink 14234 (time 1766415377.32 total 176.15 last 74.72) - /unlink 15051 (time 1766415387.32 total 186.15 last 81.69) - /unlink 15836 (time 1766415397.33 total 196.15 last 78.44) - /unlink 16649 (time 1766415407.34 total 206.16 last 81.21) - /unlink 17436 (time 1766415417.35 total 216.17 last 78.63) - /unlink 18235 (time 1766415427.35 total 226.17 last 79.90) - /unlink 19076 (time 1766415437.36 total 236.18 last 84.02) - /unlink 19906 (time 1766415447.36 total 246.19 last 82.98) - /unlink 20718 (time 1766415457.37 total 256.19 last 81.15) - /unlink 21516 (time 1766415467.37 total 266.20 last 79.77) - /unlink 22401 (time 1766415477.38 total 276.20 last 88.47) - /unlink 23253 (time 1766415487.38 total 286.20 last 85.15) - /unlink 24142 (time 1766415497.39 total 296.21 last 88.84) - /unlink 25167 (time 1766415507.39 total 306.21 last 102.49) - /unlink 26029 (time 1766415517.39 total 316.22 last 86.16) - /unlink 26911 (time 1766415527.40 total 326.22 last 88.17) - /unlink 27900 (time 1766415537.40 total 336.22 last 98.86) - /unlink 28866 (time 1766415547.40 total 346.23 last 96.58) - /unlink 29808 (time 1766415557.41 total 356.23 last 94.15) - /unlink 30000 (time 1766415559.49 total 358.31 last 92.27) - /unlink 31099 (time 1766415569.49 total 368.32 last 109.85) - /unlink 32125 (time 1766415579.50 total 378.33 last 102.48) - /unlink 32942 (time 1766415589.51 total 388.34 last 81.63) - /unlink 33781 (time 1766415599.51 total 398.34 last 83.89) - /unlink 34753 (time 1766415609.52 total 408.34 last 97.15) - /unlink 35627 (time 1766415619.52 total 418.34 last 87.39) - /unlink 36504 (time 1766415629.52 total 428.35 last 87.69) - /unlink 37388 (time 1766415639.53 total 438.35 last 88.34) - /unlink 38419 (time 1766415649.53 total 448.36 last 103.04) - /unlink 39407 (time 1766415659.54 total 458.36 last 98.75) - /unlink 40000 (time 1766415666.32 total 465.14 last 87.48) - /unlink 40928 (time 1766415676.32 total 475.14 last 92.79) - /unlink 42109 (time 1766415686.33 total 485.15 last 118.00) - /unlink 43128 (time 1766415696.33 total 495.16 last 101.86) - /unlink 44066 (time 1766415706.33 total 505.16 last 93.78) - /unlink 44958 (time 1766415716.34 total 515.16 last 89.15) - /unlink 45958 (time 1766415726.35 total 525.17 last 99.95) - /unlink 46969 (time 1766415736.35 total 535.18 last 101.01) - /unlink 48117 (time 1766415746.36 total 545.18 last 114.76) - /unlink 49282 (time 1766415756.36 total 555.18 last 116.48) - /unlink 50000 (time 1766415762.67 total 561.49 last 113.79) - /unlink 51192 (time 1766415772.67 total 571.50 last 119.18) - /unlink 52132 (time 1766415782.68 total 581.50 last 93.97) - /unlink 53178 (time 1766415792.68 total 591.50 last 104.57) - /unlink 54168 (time 1766415802.68 total 601.50 last 98.98) - /unlink 55258 (time 1766415812.68 total 611.51 last 108.97) - /unlink 56561 (time 1766415822.69 total 621.51 last 130.25) - /unlink 57807 (time 1766415832.69 total 631.52 last 124.52) - /unlink 58962 (time 1766415842.70 total 641.53 last 115.37) - /unlink 59858 (time 1766415852.71 total 651.54 last 89.54) - /unlink 60818 (time 1766415862.72 total 661.55 last 95.88) - /unlink 61777 (time 1766415872.73 total 671.55 last 95.86) - /unlink 62945 (time 1766415882.74 total 681.56 last 116.72) - /unlink 64549 (time 1766415892.74 total 691.56 last 160.37) total: 64768 /unlink in 692.92 seconds: 93.47 ops/second Waiting for MDT destroys to complete oleg235-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 - open/close 1355 (time 1766415920.23 total 10.00 last 135.50) - open/close 2691 (time 1766415930.23 total 20.01 last 133.51) - open/close 4328 (time 1766415940.23 total 30.01 last 163.68) - open/close 5815 (time 1766415950.23 total 40.01 last 148.69) - open/close 7148 (time 1766415960.24 total 50.02 last 133.20) - open/close 8945 (time 1766415970.24 total 60.02 last 179.69) - open/close 10000 (time 1766415974.51 total 64.28 last 247.40) - open/close 11950 (time 1766415984.51 total 74.28 last 195.00) - open/close 13155 (time 1766415994.51 total 84.28 last 120.49) - open/close 14416 (time 1766416004.52 total 94.29 last 125.99) - open/close 15614 (time 1766416014.52 total 104.29 last 119.79) - open/close 17377 (time 1766416024.52 total 114.30 last 176.16) - open/close 19802 (time 1766416034.53 total 124.30 last 242.49) - open/close 22484 (time 1766416044.53 total 134.30 last 268.14) - open/close 24539 (time 1766416054.53 total 144.31 last 205.38) - open/close 26531 (time 1766416064.53 total 154.31 last 199.18) - open/close 28171 (time 1766416074.54 total 164.31 last 163.92) - open/close 30000 (time 1766416084.04 total 173.82 last 192.46) - open/close 32363 (time 1766416094.04 total 183.82 last 236.27) - open/close 35082 (time 1766416104.04 total 193.82 last 271.88) - open/close 37512 (time 1766416114.04 total 203.82 last 242.98) - open/close 39790 (time 1766416124.05 total 213.82 last 227.68) - open/close 41508 (time 1766416134.05 total 223.83 last 171.77) - open/close 43605 (time 1766416144.05 total 233.83 last 209.66) - open/close 45634 (time 1766416154.06 total 243.84 last 202.76) - open/close 48164 (time 1766416164.06 total 253.84 last 253.00) - open/close 50000 (time 1766416171.83 total 261.61 last 236.26) - open/close 52271 (time 1766416181.84 total 271.62 last 226.83) - open/close 53805 (time 1766416191.85 total 281.63 last 153.27) - open/close 55240 (time 1766416201.86 total 291.63 last 143.46) - open/close 56743 (time 1766416211.86 total 301.63 last 150.30) - open/close 58193 (time 1766416221.86 total 311.64 last 144.91) - open/close 60000 (time 1766416229.92 total 319.69 last 224.30) - open/close 61746 (time 1766416239.92 total 329.69 last 174.57) - open/close 63486 (time 1766416249.92 total 339.70 last 173.97) total: 64768 open/close in 346.91 seconds: 186.70 ops/second - /unlink 3676 (time 1766416268.89 total 10.00 last 367.53) - /unlink 7783 (time 1766416278.89 total 20.00 last 410.66) - /unlink 10000 (time 1766416284.26 total 25.37 last 413.34) - /unlink 13454 (time 1766416294.26 total 35.37 last 345.39) - /unlink 16440 (time 1766416304.26 total 45.37 last 298.55) - /unlink 19854 (time 1766416314.26 total 55.37 last 341.29) - /unlink 22907 (time 1766416324.26 total 65.37 last 305.23) - /unlink 25708 (time 1766416334.27 total 75.38 last 280.05) - /unlink 30000 (time 1766416344.14 total 85.25 last 434.56) - /unlink 33596 (time 1766416354.14 total 95.25 last 359.57) - /unlink 37797 (time 1766416364.14 total 105.25 last 420.05) - /unlink 40000 (time 1766416370.06 total 111.17 last 372.45) - /unlink 43892 (time 1766416380.06 total 121.17 last 389.14) - /unlink 47558 (time 1766416390.06 total 131.17 last 366.59) - /unlink 50000 (time 1766416396.70 total 137.81 last 367.62) - /unlink 53760 (time 1766416406.70 total 147.81 last 375.97) - /unlink 58021 (time 1766416416.71 total 157.82 last 426.04) - /unlink 60000 (time 1766416422.05 total 163.16 last 370.15) - /unlink 63744 (time 1766416432.05 total 173.17 last 374.30) total: 64768 /unlink in 176.47 seconds: 367.02 ops/second Waiting for MDT destroys to complete oleg235-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 - open/close 2812 (time 1766416454.27 total 10.00 last 281.15) - open/close 5858 (time 1766416464.28 total 20.00 last 304.52) - open/close 8921 (time 1766416474.28 total 30.01 last 306.27) - open/close 10000 (time 1766416477.33 total 33.06 last 353.73) - open/close 13128 (time 1766416487.33 total 43.06 last 312.74) - open/close 16021 (time 1766416497.33 total 53.06 last 289.24) - open/close 18873 (time 1766416507.33 total 63.06 last 285.18) - open/close 20000 (time 1766416511.92 total 67.64 last 245.87) - open/close 22327 (time 1766416521.92 total 77.65 last 232.64) - open/close 24637 (time 1766416531.92 total 87.65 last 230.98) - open/close 26958 (time 1766416541.92 total 97.65 last 232.04) - open/close 29976 (time 1766416551.92 total 107.65 last 301.78) - open/close 33205 (time 1766416561.92 total 117.65 last 322.90) - open/close 36252 (time 1766416571.92 total 127.65 last 304.62) - open/close 39546 (time 1766416581.93 total 137.65 last 329.36) - open/close 42392 (time 1766416591.93 total 147.66 last 284.53) - open/close 45243 (time 1766416601.93 total 157.66 last 285.02) - open/close 47831 (time 1766416611.93 total 167.66 last 258.71) - open/close 50000 (time 1766416619.57 total 175.30 last 283.97) - open/close 53934 (time 1766416629.57 total 185.30 last 393.35) - open/close 56912 (time 1766416639.57 total 195.30 last 297.80) - open/close 59495 (time 1766416649.58 total 205.31 last 258.21) - open/close 62305 (time 1766416659.58 total 215.31 last 280.95) total: 64768 open/close in 220.91 seconds: 293.19 ops/second - /unlink 6621 (time 1766416676.65 total 10.00 last 662.05) - /unlink 10000 (time 1766416682.56 total 15.92 last 570.91) - /unlink 17875 (time 1766416692.57 total 25.92 last 787.37) - /unlink 20000 (time 1766416695.25 total 28.61 last 790.50) - /unlink 27746 (time 1766416705.25 total 38.61 last 774.54) - /unlink 30000 (time 1766416708.25 total 41.61 last 752.28) - /unlink 35476 (time 1766416718.25 total 51.61 last 547.59) - /unlink 40000 (time 1766416724.94 total 58.29 last 676.61) - /unlink 47172 (time 1766416734.94 total 68.29 last 717.13) - /unlink 50000 (time 1766416738.80 total 72.16 last 731.57) - /unlink 57761 (time 1766416748.81 total 82.16 last 776.01) - /unlink 60000 (time 1766416752.14 total 85.49 last 671.52) total: 64768 /unlink in 91.20 seconds: 710.17 ops/second Waiting for MDT destroys to complete oleg235-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 - open/close 3611 (time 1766416824.92 total 10.00 last 361.02) - open/close 6849 (time 1766416834.92 total 20.00 last 323.72) - open/close 10000 (time 1766416842.82 total 27.91 last 398.81) - open/close 14426 (time 1766416852.82 total 37.91 last 442.58) - open/close 18322 (time 1766416862.82 total 47.91 last 389.60) - open/close 20000 (time 1766416867.17 total 52.25 last 386.22) - open/close 23798 (time 1766416877.17 total 62.25 last 379.72) - open/close 27707 (time 1766416887.17 total 72.25 last 390.83) - open/close 30000 (time 1766416892.88 total 77.97 last 401.44) - open/close 33454 (time 1766416902.89 total 87.97 last 345.33) - open/close 36639 (time 1766416912.89 total 97.97 last 318.45) - open/close 39489 (time 1766416922.89 total 107.97 last 284.90) - open/close 42338 (time 1766416932.89 total 117.98 last 284.82) - open/close 45398 (time 1766416942.90 total 127.98 last 305.95) - open/close 48928 (time 1766416952.90 total 137.98 last 352.95) - open/close 50000 (time 1766416955.75 total 140.83 last 376.46) - open/close 52837 (time 1766416965.75 total 150.83 last 283.68) - open/close 56864 (time 1766416975.75 total 160.83 last 402.62) - open/close 60000 (time 1766416984.10 total 169.19 last 375.33) - open/close 63791 (time 1766416994.11 total 179.19 last 379.01) total: 64768 open/close in 182.23 seconds: 355.42 ops/second - /unlink 6732 (time 1766417008.61 total 10.00 last 673.11) - /unlink 10000 (time 1766417015.85 total 17.24 last 451.34) - /unlink 14222 (time 1766417025.85 total 27.24 last 422.13) - /unlink 18467 (time 1766417035.85 total 37.24 last 424.50) - /unlink 20000 (time 1766417038.54 total 39.93 last 571.09) - /unlink 25605 (time 1766417048.54 total 49.93 last 560.37) - /unlink 29657 (time 1766417058.54 total 59.93 last 405.13) - /unlink 33818 (time 1766417068.54 total 69.93 last 416.03) - /unlink 38000 (time 1766417078.54 total 79.93 last 418.16) - /unlink 40000 (time 1766417082.56 total 83.95 last 498.51) - /unlink 45982 (time 1766417092.56 total 93.95 last 598.10) - /unlink 50000 (time 1766417098.15 total 99.54 last 718.08) - /unlink 57620 (time 1766417108.15 total 109.54 last 761.97) - /unlink 60000 (time 1766417110.90 total 112.29 last 866.54) total: 64768 /unlink in 118.49 seconds: 546.59 ops/second Waiting for MDT destroys to complete oleg235-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg235-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg235-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing unload_modules_local oleg235-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg235-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg235-client: oleg235-client: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-client: ssh exited with exit code 2 PASS 106 (4817s) Stopping clients: oleg235-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg235-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2 oleg235-server: oleg235-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing load_modules_local oleg235-server: Loading modules from /home/green/git/lustre-release/lustre oleg235-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg235-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg235-server: detected 4 online CPUs by sysfs oleg235-server: MODOPTS_LIBCFS= oleg235-server: Force libcfs to create 2 CPU partitions oleg235-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg235-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 5049 sec ========== 10:26:23 (1766417183) === conf-sanity: start cleanup 10:26:24 (1766417184) === === conf-sanity: finish cleanup 10:26:25 (1766417185) ===