-----============= acceptance-small: conf-sanity ============----- Mon May 11 17:28:04 EDT 2026 oleg158-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg158-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg158-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg158-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg158-server oleg158-server: oleg158-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing load_modules_local oleg158-server: Loading modules from /home/green/git/lustre-release/lustre oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg158-server: detected 4 online CPUs by sysfs oleg158-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg158-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg158-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg158-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg158-server: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg158-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg158-server: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg158-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:31:33 (1778535093) Stopping clients: oleg158-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg158-client.virtnet /mnt/lustre2 (opts:-f) oleg158-server: oleg158-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing load_modules_local oleg158-server: Loading modules from /home/green/git/lustre-release/lustre oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg158-server: detected 4 online CPUs by sysfs oleg158-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg158-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg158-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg158-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg158-client.virtnet: -o user_xattr,flock oleg158-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16232 424 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 699 (time 1778535238.28 total 10.01 last 69.84) - open/close 1462 (time 1778535248.28 total 20.01 last 76.27) - open/close 2306 (time 1778535258.30 total 30.03 last 84.29) - open/close 3079 (time 1778535268.31 total 40.03 last 77.24) - open/close 3888 (time 1778535278.31 total 50.03 last 80.89) - open/close 4666 (time 1778535288.31 total 60.04 last 77.74) - open/close 5466 (time 1778535298.33 total 70.06 last 79.85) - open/close 6185 (time 1778535308.35 total 80.07 last 71.81) - open/close 6835 (time 1778535318.35 total 90.08 last 64.99) - open/close 7431 (time 1778535328.35 total 100.08 last 59.58) - open/close 8095 (time 1778535338.35 total 110.08 last 66.40) - open/close 8787 (time 1778535348.36 total 120.09 last 69.14) - open/close 9540 (time 1778535358.38 total 130.10 last 75.18) - open/close 10000 (time 1778535364.11 total 135.83 last 80.28) - open/close 10841 (time 1778535374.11 total 145.84 last 84.05) - open/close 11657 (time 1778535384.15 total 155.88 last 81.31) - open/close 12509 (time 1778535394.15 total 165.88 last 85.18) - open/close 13341 (time 1778535404.15 total 175.88 last 83.18) - open/close 14168 (time 1778535414.16 total 185.89 last 82.63) - open/close 15077 (time 1778535424.16 total 195.89 last 90.87) - open/close 16045 (time 1778535434.17 total 205.89 last 96.78) - open/close 16860 (time 1778535444.17 total 215.90 last 81.48) - open/close 17768 (time 1778535454.17 total 225.90 last 90.77) - open/close 18686 (time 1778535464.17 total 235.90 last 91.79) - open/close 19613 (time 1778535474.18 total 245.91 last 92.64) - open/close 20000 (time 1778535477.88 total 249.61 last 104.56) - open/close 20998 (time 1778535487.90 total 259.62 last 99.64) - open/close 22000 (time 1778535497.90 total 269.62 last 100.19) - open/close 22998 (time 1778535507.90 total 279.63 last 99.76) - open/close 23932 (time 1778535517.91 total 289.64 last 93.33) - open/close 24839 (time 1778535527.92 total 299.65 last 90.62) - open/close 25699 (time 1778535537.92 total 309.65 last 85.95) - open/close 26573 (time 1778535547.93 total 319.66 last 87.33) - open/close 27441 (time 1778535557.93 total 329.66 last 86.80) - open/close 28323 (time 1778535567.93 total 339.66 last 88.17) - open/close 29313 (time 1778535577.93 total 349.66 last 99.00) - open/close 30000 (time 1778535585.85 total 357.58 last 86.76) - open/close 31008 (time 1778535595.85 total 367.58 last 100.79) - open/close 31988 (time 1778535605.87 total 377.60 last 97.81) - open/close 32958 (time 1778535615.88 total 387.60 last 96.98) - open/close 33890 (time 1778535625.88 total 397.60 last 93.20) - open/close 34821 (time 1778535635.90 total 407.63 last 92.90) - open/close 35771 (time 1778535645.90 total 417.63 last 95.00) - open/close 36674 (time 1778535655.90 total 427.63 last 90.25) - open/close 37570 (time 1778535665.91 total 437.63 last 89.56) - open/close 38407 (time 1778535675.91 total 447.64 last 83.69) - open/close 39308 (time 1778535685.91 total 457.64 last 90.05) - open/close 40000 (time 1778535694.01 total 465.74 last 85.44) - open/close 40863 (time 1778535704.02 total 475.75 last 86.26) - open/close 41852 (time 1778535714.04 total 485.77 last 98.63) - open/close 42795 (time 1778535724.05 total 495.78 last 94.27) - open/close 43660 (time 1778535734.05 total 505.78 last 86.48) - open/close 44555 (time 1778535744.06 total 515.79 last 89.42) - open/close 45421 (time 1778535754.09 total 525.81 last 86.36) - open/close 46371 (time 1778535764.09 total 535.82 last 94.95) - open/close 47408 (time 1778535774.09 total 545.82 last 103.69) - open/close 48357 (time 1778535784.10 total 555.83 last 94.81) - open/close 49401 (time 1778535794.11 total 565.84 last 104.35) - open/close 50000 (time 1778535800.28 total 572.01 last 97.00) - open/close 50849 (time 1778535810.28 total 582.01 last 84.90) - open/close 51788 (time 1778535820.29 total 592.02 last 93.80) - open/close 52958 (time 1778535830.30 total 602.02 last 116.98) - open/close 53980 (time 1778535840.30 total 612.03 last 102.16) - open/close 55037 (time 1778535850.30 total 622.03 last 105.64) - open/close 56009 (time 1778535860.34 total 632.07 last 96.84) - open/close 56922 (time 1778535870.34 total 642.07 last 91.28) - open/close 57860 (time 1778535880.35 total 652.07 last 93.79) - open/close 58795 (time 1778535890.35 total 662.08 last 93.45) - open/close 59889 (time 1778535900.35 total 672.08 last 109.39) - open/close 60990 (time 1778535910.35 total 682.08 last 110.09) - open/close 62000 (time 1778535920.36 total 692.09 last 100.93) - open/close 63073 (time 1778535930.36 total 702.09 last 107.28) - open/close 64123 (time 1778535940.37 total 712.10 last 104.89) total: 64768 open/close in 718.09 seconds: 90.19 ops/second - /unlink 2101 (time 1778535958.89 total 10.00 last 210.02) - /unlink 4311 (time 1778535968.89 total 20.00 last 220.98) - /unlink 6352 (time 1778535978.89 total 30.01 last 204.09) - /unlink 8574 (time 1778535988.89 total 40.01 last 222.20) - /unlink 10000 (time 1778535994.89 total 46.01 last 237.66) - /unlink 11986 (time 1778536004.90 total 56.01 last 198.56) - /unlink 13851 (time 1778536014.90 total 66.01 last 186.45) - /unlink 16058 (time 1778536024.90 total 76.01 last 220.66) - /unlink 18275 (time 1778536034.90 total 86.02 last 221.59) - /unlink 20000 (time 1778536043.32 total 94.44 last 204.87) - /unlink 21962 (time 1778536053.33 total 104.44 last 196.13) - /unlink 23917 (time 1778536063.33 total 114.45 last 195.41) - /unlink 25885 (time 1778536073.34 total 124.45 last 196.72) - /unlink 28025 (time 1778536083.34 total 134.45 last 213.92) - /unlink 29876 (time 1778536093.34 total 144.46 last 185.04) - /unlink 31694 (time 1778536103.34 total 154.46 last 181.79) - /unlink 33906 (time 1778536113.35 total 164.46 last 221.14) - /unlink 35932 (time 1778536123.35 total 174.46 last 202.56) - /unlink 38051 (time 1778536133.35 total 184.46 last 211.88) - /unlink 40000 (time 1778536142.92 total 194.03 last 203.76) - /unlink 42102 (time 1778536152.92 total 204.03 last 210.18) - /unlink 44385 (time 1778536162.92 total 214.03 last 228.30) - /unlink 46655 (time 1778536172.92 total 224.03 last 226.94) - /unlink 48760 (time 1778536182.92 total 234.03 last 210.45) - /unlink 50000 (time 1778536188.64 total 239.75 last 216.93) - /unlink 52161 (time 1778536198.64 total 249.76 last 216.00) - /unlink 54094 (time 1778536208.65 total 259.76 last 193.21) - /unlink 56125 (time 1778536218.66 total 269.77 last 202.94) - /unlink 58380 (time 1778536228.66 total 279.77 last 225.44) - /unlink 60000 (time 1778536236.57 total 287.68 last 204.81) - /unlink 62437 (time 1778536246.57 total 297.68 last 243.68) - /unlink 64643 (time 1778536256.57 total 307.68 last 220.59) total: 64768 /unlink in 308.21 seconds: 210.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1243 (time 1778536294.36 total 10.00 last 124.28) - open/close 2386 (time 1778536304.36 total 20.00 last 114.27) - open/close 3701 (time 1778536314.36 total 30.00 last 131.50) - open/close 4979 (time 1778536324.37 total 40.01 last 127.71) - open/close 6305 (time 1778536334.38 total 50.02 last 132.53) - open/close 7514 (time 1778536344.38 total 60.02 last 120.81) - open/close 8199 (time 1778536354.38 total 70.02 last 68.50) - open/close 8773 (time 1778536364.39 total 80.03 last 57.36) - open/close 9414 (time 1778536374.40 total 90.05 last 64.01) - open/close 9977 (time 1778536384.54 total 100.18 last 55.55) - open/close 10606 (time 1778536394.54 total 110.18 last 62.89) - open/close 11261 (time 1778536404.54 total 120.18 last 65.49) - open/close 12087 (time 1778536414.56 total 130.20 last 82.48) - open/close 12969 (time 1778536424.56 total 140.20 last 88.19) - open/close 13819 (time 1778536434.57 total 150.21 last 84.91) - open/close 14780 (time 1778536444.57 total 160.21 last 96.08) - open/close 15756 (time 1778536454.57 total 170.22 last 97.58) - open/close 16673 (time 1778536464.58 total 180.22 last 91.65) - open/close 17568 (time 1778536474.58 total 190.22 last 89.47) - open/close 18470 (time 1778536484.59 total 200.23 last 90.16) - open/close 19398 (time 1778536494.59 total 210.24 last 92.73) - open/close 20000 (time 1778536501.58 total 217.22 last 86.19) - open/close 20895 (time 1778536511.59 total 227.23 last 89.41) - open/close 21818 (time 1778536521.59 total 237.23 last 92.28) - open/close 22675 (time 1778536531.60 total 247.24 last 85.65) - open/close 23591 (time 1778536541.60 total 257.25 last 91.53) - open/close 24491 (time 1778536551.60 total 267.25 last 90.00) - open/close 25330 (time 1778536561.62 total 277.26 last 83.79) - open/close 26236 (time 1778536571.63 total 287.27 last 90.51) - open/close 27103 (time 1778536581.64 total 297.28 last 86.61) - open/close 27982 (time 1778536591.64 total 307.28 last 87.87) - open/close 28803 (time 1778536601.64 total 317.29 last 82.08) - open/close 29676 (time 1778536611.64 total 327.29 last 87.30) - open/close 30000 (time 1778536616.17 total 331.81 last 71.61) - open/close 30868 (time 1778536626.18 total 341.82 last 86.70) - open/close 31693 (time 1778536636.19 total 351.84 last 82.39) - open/close 32497 (time 1778536646.20 total 361.84 last 80.35) - open/close 33388 (time 1778536656.20 total 371.85 last 89.06) - open/close 34249 (time 1778536666.21 total 381.85 last 86.10) - open/close 35129 (time 1778536676.21 total 391.85 last 87.94) - open/close 36041 (time 1778536686.22 total 401.86 last 91.15) - open/close 36928 (time 1778536696.23 total 411.87 last 88.58) - open/close 37869 (time 1778536706.24 total 421.88 last 94.02) - open/close 38739 (time 1778536716.25 total 431.89 last 86.92) - open/close 39605 (time 1778536726.25 total 441.89 last 86.57) - open/close 40000 (time 1778536730.45 total 446.09 last 94.13) - open/close 40831 (time 1778536740.45 total 456.09 last 83.09) - open/close 41658 (time 1778536750.46 total 466.10 last 82.66) - open/close 42525 (time 1778536760.46 total 476.10 last 86.65) - open/close 43400 (time 1778536770.47 total 486.11 last 87.38) - open/close 44199 (time 1778536780.48 total 496.12 last 79.87) - open/close 45032 (time 1778536790.49 total 506.13 last 83.22) - open/close 45902 (time 1778536800.50 total 516.14 last 86.91) - open/close 46606 (time 1778536810.51 total 526.15 last 70.32) - open/close 47454 (time 1778536820.52 total 536.16 last 84.74) - open/close 48385 (time 1778536830.52 total 546.16 last 93.07) - open/close 49362 (time 1778536840.52 total 556.16 last 97.68) - open/close 50000 (time 1778536846.94 total 562.58 last 99.41) - open/close 50971 (time 1778536856.94 total 572.58 last 97.06) - open/close 51845 (time 1778536866.96 total 582.60 last 87.29) - open/close 52833 (time 1778536876.98 total 592.62 last 98.60) - open/close 53762 (time 1778536886.98 total 602.62 last 92.86) - open/close 54668 (time 1778536896.99 total 612.63 last 90.50) - open/close 55565 (time 1778536906.99 total 622.63 last 89.69) - open/close 56453 (time 1778536917.00 total 632.64 last 88.75) - open/close 57307 (time 1778536927.00 total 642.64 last 85.37) - open/close 58220 (time 1778536937.01 total 652.65 last 91.23) - open/close 59011 (time 1778536947.01 total 662.65 last 79.09) - open/close 59996 (time 1778536957.02 total 672.66 last 98.41) - open/close 60901 (time 1778536967.02 total 682.66 last 90.49) - open/close 61742 (time 1778536977.03 total 692.67 last 84.03) - open/close 62657 (time 1778536987.03 total 702.67 last 91.45) - open/close 63515 (time 1778536997.04 total 712.68 last 85.75) - open/close 64315 (time 1778537007.05 total 722.69 last 79.91) total: 64768 open/close in 727.72 seconds: 89.00 ops/second - /unlink 1496 (time 1778537025.22 total 10.00 last 149.53) - /unlink 2955 (time 1778537035.23 total 20.01 last 145.84) - /unlink 4675 (time 1778537045.23 total 30.01 last 171.99) - /unlink 6511 (time 1778537055.23 total 40.01 last 183.59) - /unlink 8080 (time 1778537065.23 total 50.01 last 156.85) - /unlink 9819 (time 1778537075.24 total 60.02 last 173.81) - /unlink 11569 (time 1778537085.24 total 70.02 last 174.97) - /unlink 13142 (time 1778537095.24 total 80.02 last 157.30) - /unlink 14806 (time 1778537105.24 total 90.02 last 166.34) - /unlink 16348 (time 1778537115.25 total 100.03 last 154.11) - /unlink 18020 (time 1778537125.26 total 110.05 last 166.95) - /unlink 19573 (time 1778537135.26 total 120.05 last 155.30) - /unlink 20000 (time 1778537138.08 total 122.86 last 151.59) - /unlink 21638 (time 1778537148.09 total 132.87 last 163.71) - /unlink 23205 (time 1778537158.09 total 142.87 last 156.70) - /unlink 24603 (time 1778537168.09 total 152.87 last 139.76) - /unlink 26107 (time 1778537178.10 total 162.88 last 150.30) - /unlink 27680 (time 1778537188.10 total 172.88 last 157.30) - /unlink 29290 (time 1778537198.10 total 182.88 last 160.98) - /unlink 30000 (time 1778537203.20 total 187.98 last 139.20) - /unlink 31752 (time 1778537213.20 total 197.98 last 175.20) - /unlink 33484 (time 1778537223.20 total 207.98 last 173.20) - /unlink 35223 (time 1778537233.20 total 217.98 last 173.86) - /unlink 37014 (time 1778537243.23 total 228.01 last 178.57) - /unlink 39007 (time 1778537253.23 total 238.02 last 199.22) - /unlink 40000 (time 1778537258.82 total 243.60 last 177.71) - /unlink 41876 (time 1778537268.83 total 253.61 last 187.53) - /unlink 43742 (time 1778537278.83 total 263.61 last 186.58) - /unlink 45582 (time 1778537288.83 total 273.61 last 183.94) - /unlink 47291 (time 1778537298.83 total 283.61 last 170.89) - /unlink 49186 (time 1778537308.83 total 293.62 last 189.46) - /unlink 50000 (time 1778537313.80 total 298.58 last 163.87) - /unlink 51803 (time 1778537323.81 total 308.59 last 180.22) - /unlink 53753 (time 1778537333.81 total 318.59 last 194.87) - /unlink 56042 (time 1778537343.81 total 328.59 last 228.89) - /unlink 58138 (time 1778537353.82 total 338.60 last 209.53) - /unlink 60000 (time 1778537363.02 total 347.80 last 202.30) - /unlink 61958 (time 1778537373.02 total 357.81 last 195.73) - /unlink 64032 (time 1778537383.03 total 367.81 last 207.33) total: 64768 /unlink in 371.72 seconds: 174.24 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1158 (time 1778537415.74 total 10.01 last 115.68) - open/close 2274 (time 1778537425.75 total 20.02 last 111.55) - open/close 3370 (time 1778537435.75 total 30.02 last 109.59) - open/close 4588 (time 1778537445.75 total 40.02 last 121.77) - open/close 5719 (time 1778537455.75 total 50.02 last 113.08) - open/close 6845 (time 1778537465.75 total 60.02 last 112.57) - open/close 7918 (time 1778537475.76 total 70.03 last 107.23) - open/close 8989 (time 1778537485.76 total 80.03 last 107.06) - open/close 9997 (time 1778537495.78 total 90.05 last 100.67) - open/close 11077 (time 1778537505.78 total 100.05 last 107.94) - open/close 12085 (time 1778537515.79 total 110.05 last 100.78) - open/close 13221 (time 1778537525.79 total 120.06 last 113.56) - open/close 14401 (time 1778537535.79 total 130.06 last 117.98) - open/close 15659 (time 1778537545.80 total 140.07 last 125.68) - open/close 16780 (time 1778537555.81 total 150.08 last 112.03) - open/close 17791 (time 1778537565.81 total 160.08 last 101.08) - open/close 18753 (time 1778537575.81 total 170.08 last 96.16) - open/close 19837 (time 1778537585.82 total 180.09 last 108.29) - open/close 20897 (time 1778537595.82 total 190.09 last 105.99) - open/close 21928 (time 1778537605.83 total 200.10 last 102.99) - open/close 23023 (time 1778537615.83 total 210.10 last 109.50) - open/close 24100 (time 1778537625.84 total 220.11 last 107.64) - open/close 25177 (time 1778537635.84 total 230.11 last 107.67) - open/close 26309 (time 1778537645.84 total 240.11 last 113.20) - open/close 27536 (time 1778537655.86 total 250.13 last 122.52) - open/close 28632 (time 1778537665.86 total 260.13 last 109.57) - open/close 29730 (time 1778537675.88 total 270.15 last 109.60) - open/close 30000 (time 1778537678.57 total 272.84 last 100.43) - open/close 31078 (time 1778537688.58 total 282.84 last 107.72) - open/close 32142 (time 1778537698.58 total 292.85 last 106.38) - open/close 33256 (time 1778537708.60 total 302.87 last 111.16) - open/close 34233 (time 1778537718.60 total 312.87 last 97.70) - open/close 35319 (time 1778537728.60 total 322.87 last 108.59) - open/close 36519 (time 1778537738.61 total 332.88 last 119.91) - open/close 37665 (time 1778537748.62 total 342.89 last 114.44) - open/close 38740 (time 1778537758.62 total 352.89 last 107.47) - open/close 39893 (time 1778537768.63 total 362.90 last 115.18) - open/close 41045 (time 1778537778.64 total 372.90 last 115.19) - open/close 42228 (time 1778537788.64 total 382.91 last 118.23) - open/close 43293 (time 1778537798.65 total 392.91 last 106.46) - open/close 44511 (time 1778537808.65 total 402.92 last 121.69) - open/close 45678 (time 1778537818.67 total 412.93 last 116.57) - open/close 46789 (time 1778537828.67 total 422.94 last 111.08) - open/close 47890 (time 1778537838.68 total 432.94 last 110.01) - open/close 49046 (time 1778537848.68 total 442.95 last 115.57) - open/close 50000 (time 1778537856.99 total 451.26 last 114.72) - open/close 51047 (time 1778537867.00 total 461.27 last 104.66) - open/close 52187 (time 1778537877.00 total 471.27 last 113.95) - open/close 53384 (time 1778537887.00 total 481.27 last 119.67) - open/close 54446 (time 1778537897.01 total 491.28 last 106.17) - open/close 55574 (time 1778537907.01 total 501.28 last 112.74) - open/close 56640 (time 1778537917.03 total 511.30 last 106.41) - open/close 57757 (time 1778537927.03 total 521.30 last 111.69) - open/close 58887 (time 1778537937.03 total 531.30 last 112.98) - open/close 59944 (time 1778537947.04 total 541.31 last 105.61) - open/close 61054 (time 1778537957.05 total 551.32 last 110.93) - open/close 62227 (time 1778537967.05 total 561.32 last 117.28) - open/close 63312 (time 1778537977.05 total 571.32 last 108.45) - open/close 64470 (time 1778537987.06 total 581.32 last 115.79) total: 64768 open/close in 584.07 seconds: 110.89 ops/second - /unlink 2278 (time 1778538002.11 total 10.00 last 227.73) - /unlink 4369 (time 1778538012.12 total 20.01 last 209.04) - /unlink 6514 (time 1778538022.12 total 30.01 last 214.46) - /unlink 8646 (time 1778538032.13 total 40.02 last 212.99) - /unlink 10000 (time 1778538038.52 total 46.41 last 211.98) - /unlink 12002 (time 1778538048.52 total 56.41 last 200.10) - /unlink 14046 (time 1778538058.53 total 66.41 last 204.31) - /unlink 16508 (time 1778538068.53 total 76.41 last 246.19) - /unlink 18958 (time 1778538078.53 total 86.42 last 244.96) - /unlink 20000 (time 1778538083.03 total 90.92 last 231.44) - /unlink 22166 (time 1778538093.03 total 100.92 last 216.56) - /unlink 24768 (time 1778538103.03 total 110.92 last 260.14) - /unlink 27254 (time 1778538113.04 total 120.93 last 248.52) - /unlink 29882 (time 1778538123.04 total 130.93 last 262.73) - /unlink 32310 (time 1778538133.04 total 140.93 last 242.78) - /unlink 34775 (time 1778538143.04 total 150.93 last 246.45) - /unlink 37455 (time 1778538153.04 total 160.93 last 267.96) - /unlink 40000 (time 1778538163.03 total 170.92 last 254.90) - /unlink 42694 (time 1778538173.03 total 180.92 last 269.33) - /unlink 45253 (time 1778538183.03 total 190.92 last 255.80) - /unlink 47848 (time 1778538193.03 total 200.92 last 259.49) - /unlink 50000 (time 1778538200.68 total 208.57 last 281.35) - /unlink 52657 (time 1778538210.69 total 218.58 last 265.63) - /unlink 55608 (time 1778538220.69 total 228.58 last 295.04) - /unlink 58305 (time 1778538230.69 total 238.58 last 269.66) - /unlink 60000 (time 1778538236.94 total 244.83 last 271.18) - /unlink 62886 (time 1778538246.94 total 254.83 last 288.53) total: 64768 /unlink in 261.02 seconds: 248.13 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1713 (time 1778538285.43 total 10.00 last 171.23) - open/close 3474 (time 1778538295.43 total 20.00 last 176.10) - open/close 5282 (time 1778538305.43 total 30.00 last 180.79) - open/close 7193 (time 1778538315.43 total 40.01 last 191.03) - open/close 9162 (time 1778538325.44 total 50.02 last 196.67) - open/close 10000 (time 1778538329.88 total 54.45 last 189.00) - open/close 11877 (time 1778538339.88 total 64.46 last 187.63) - open/close 14671 (time 1778538349.88 total 74.46 last 279.35) - open/close 17293 (time 1778538359.89 total 84.46 last 262.10) - open/close 19565 (time 1778538369.89 total 94.47 last 227.13) - open/close 21530 (time 1778538379.89 total 104.47 last 196.41) - open/close 23939 (time 1778538389.90 total 114.47 last 240.83) - open/close 26543 (time 1778538399.90 total 124.48 last 260.34) - open/close 29083 (time 1778538409.90 total 134.48 last 253.97) - open/close 30000 (time 1778538413.27 total 137.85 last 272.13) - open/close 32302 (time 1778538423.27 total 147.85 last 230.13) - open/close 35172 (time 1778538433.27 total 157.85 last 286.95) - open/close 37824 (time 1778538443.27 total 167.85 last 265.19) - open/close 40000 (time 1778538452.85 total 177.42 last 227.34) - open/close 42228 (time 1778538462.85 total 187.43 last 222.74) - open/close 44197 (time 1778538472.85 total 197.43 last 196.85) - open/close 46371 (time 1778538482.85 total 207.43 last 217.38) - open/close 48622 (time 1778538492.85 total 217.43 last 225.05) - open/close 50000 (time 1778538499.96 total 224.54 last 193.96) - open/close 51873 (time 1778538509.96 total 234.54 last 187.27) - open/close 53857 (time 1778538519.96 total 244.54 last 198.39) - open/close 55788 (time 1778538529.97 total 254.55 last 192.90) - open/close 57654 (time 1778538539.97 total 264.55 last 186.56) - open/close 59996 (time 1778538549.98 total 274.55 last 234.11) - open/close 62092 (time 1778538559.98 total 284.56 last 209.54) - open/close 63931 (time 1778538569.98 total 294.56 last 183.86) total: 64768 open/close in 299.23 seconds: 216.45 ops/second - /unlink 3501 (time 1778538586.30 total 10.00 last 349.98) - /unlink 6919 (time 1778538596.30 total 20.00 last 341.80) - /unlink 10000 (time 1778538604.29 total 28.00 last 385.23) - /unlink 14283 (time 1778538614.30 total 38.00 last 428.21) - /unlink 18648 (time 1778538624.30 total 48.00 last 436.50) - /unlink 20000 (time 1778538627.87 total 51.58 last 378.18) - /unlink 23664 (time 1778538637.87 total 61.58 last 366.38) - /unlink 27439 (time 1778538647.87 total 71.58 last 377.45) - /unlink 30000 (time 1778538654.77 total 78.47 last 371.49) - /unlink 33153 (time 1778538664.77 total 88.48 last 315.25) - /unlink 36410 (time 1778538674.77 total 98.48 last 325.61) - /unlink 40000 (time 1778538684.43 total 108.14 last 371.51) - /unlink 44056 (time 1778538694.43 total 118.14 last 405.58) - /unlink 49008 (time 1778538704.44 total 128.14 last 495.13) - /unlink 50000 (time 1778538706.78 total 130.49 last 422.36) - /unlink 54169 (time 1778538716.78 total 140.49 last 416.88) - /unlink 59905 (time 1778538726.78 total 150.49 last 573.60) total: 64768 /unlink in 159.16 seconds: 406.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2703 (time 1778538771.42 total 10.00 last 270.20) - open/close 5256 (time 1778538781.42 total 20.01 last 255.24) - open/close 7803 (time 1778538791.43 total 30.01 last 254.60) - open/close 9911 (time 1778538801.43 total 40.01 last 210.76) - open/close 13055 (time 1778538811.43 total 50.01 last 314.38) - open/close 16157 (time 1778538821.43 total 60.01 last 310.19) - open/close 19046 (time 1778538831.43 total 70.01 last 288.85) - open/close 20000 (time 1778538834.18 total 72.76 last 346.96) - open/close 23521 (time 1778538844.18 total 82.77 last 352.05) - open/close 26547 (time 1778538854.18 total 92.77 last 302.55) - open/close 29524 (time 1778538864.19 total 102.77 last 297.60) - open/close 32903 (time 1778538874.19 total 112.77 last 337.88) - open/close 36228 (time 1778538884.19 total 122.77 last 332.46) - open/close 39515 (time 1778538894.19 total 132.77 last 328.63) - open/close 42723 (time 1778538904.19 total 142.78 last 320.72) - open/close 46083 (time 1778538914.20 total 152.78 last 335.92) - open/close 49521 (time 1778538924.20 total 162.78 last 343.76) - open/close 53279 (time 1778538934.20 total 172.78 last 375.68) - open/close 57103 (time 1778538944.20 total 182.78 last 382.36) - open/close 60000 (time 1778538951.80 total 190.38 last 381.16) - open/close 63873 (time 1778538961.80 total 200.39 last 387.20) total: 64768 open/close in 202.82 seconds: 319.34 ops/second - /unlink 6813 (time 1778538975.60 total 10.00 last 681.27) - /unlink 10000 (time 1778538980.03 total 14.44 last 718.39) - /unlink 15853 (time 1778538990.04 total 24.44 last 585.26) - /unlink 20000 (time 1778538996.26 total 30.67 last 665.84) - /unlink 25770 (time 1778539006.26 total 40.67 last 576.96) - /unlink 30000 (time 1778539012.84 total 47.25 last 642.97) - /unlink 36578 (time 1778539022.84 total 57.25 last 657.72) - /unlink 40000 (time 1778539028.35 total 62.75 last 621.90) - /unlink 46839 (time 1778539038.35 total 72.75 last 683.86) - /unlink 50000 (time 1778539042.92 total 77.32 last 691.89) - /unlink 57092 (time 1778539052.92 total 87.32 last 709.16) - /unlink 60000 (time 1778539057.32 total 91.72 last 660.45) total: 64768 /unlink in 98.02 seconds: 660.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg158-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing unload_modules_local oleg158-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg158-server: LNET ready to unload oleg158-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2 PASS 106 (4020s) Stopping clients: oleg158-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg158-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2 oleg158-server: oleg158-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing load_modules_local oleg158-server: Loading modules from /home/green/git/lustre-release/lustre oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg158-server: detected 4 online CPUs by sysfs oleg158-server: Force libcfs to create 2 CPU partitions oleg158-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg158-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4256 sec ========== 18:39:01 (1778539141)