-----============= acceptance-small: conf-sanity ============----- Fri Apr 5 18:46:30 EDT 2024 excepting tests: 32newtarball 110 Stopping clients: oleg256-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg256-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg256-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg256-server oleg256-server: oleg256-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: detected 4 online CPUs by sysfs oleg256-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg256-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg256-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg256-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:48:01 (1712357281) Stopping clients: oleg256-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg256-client.virtnet /mnt/lustre2 (opts:-f) oleg256-server: oleg256-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: detected 4 online CPUs by sysfs oleg256-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg256-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg256-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg256-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock oleg256-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1838 (time 1712357329.79 total 10.00 last 183.74) - open/close 3520 (time 1712357339.79 total 20.01 last 168.17) - open/close 5228 (time 1712357349.79 total 30.01 last 170.80) - open/close 6970 (time 1712357359.79 total 40.01 last 174.18) - open/close 8434 (time 1712357369.79 total 50.01 last 146.36) - open/close 9577 (time 1712357379.79 total 60.01 last 114.30) - open/close 10000 (time 1712357383.96 total 64.18 last 101.54) - open/close 11086 (time 1712357393.96 total 74.18 last 108.55) - open/close 12213 (time 1712357403.97 total 84.19 last 112.63) - open/close 13422 (time 1712357413.97 total 94.19 last 120.89) - open/close 14617 (time 1712357423.97 total 104.19 last 119.46) - open/close 15691 (time 1712357433.97 total 114.19 last 107.38) - open/close 16874 (time 1712357443.98 total 124.19 last 118.28) - open/close 17949 (time 1712357453.98 total 134.20 last 107.42) - open/close 18992 (time 1712357463.99 total 144.21 last 104.24) - open/close 19804 (time 1712357473.99 total 154.21 last 81.17) - open/close 20000 (time 1712357476.60 total 156.82 last 75.09) - open/close 20768 (time 1712357486.61 total 166.83 last 76.73) - open/close 21583 (time 1712357496.62 total 176.84 last 81.43) - open/close 22496 (time 1712357506.63 total 186.85 last 91.23) - open/close 23267 (time 1712357516.63 total 196.85 last 77.09) - open/close 24000 (time 1712357526.64 total 206.86 last 73.24) - open/close 24729 (time 1712357536.64 total 216.86 last 72.89) - open/close 25404 (time 1712357546.65 total 226.87 last 67.43) - open/close 26228 (time 1712357556.66 total 236.87 last 82.33) - open/close 27036 (time 1712357566.67 total 246.88 last 80.73) - open/close 27782 (time 1712357576.68 total 256.90 last 74.50) - open/close 28494 (time 1712357586.69 total 266.91 last 71.12) - open/close 29254 (time 1712357596.70 total 276.92 last 75.92) - open/close 30000 (time 1712357606.18 total 286.40 last 78.68) - open/close 30776 (time 1712357616.19 total 296.41 last 77.54) - open/close 31511 (time 1712357626.19 total 306.41 last 73.48) - open/close 32192 (time 1712357636.20 total 316.42 last 68.04) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 322.08 seconds: 101.09 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 600 (time 1712357652.78 total 10.01 last 59.91) - /unlink 1213 (time 1712357662.79 total 20.02 last 61.25) - /unlink 1892 (time 1712357672.79 total 30.02 last 67.89) - /unlink 2549 (time 1712357682.80 total 40.03 last 65.64) - /unlink 3145 (time 1712357692.81 total 50.04 last 59.58) - /unlink 3920 (time 1712357702.82 total 60.05 last 77.42) - /unlink 4807 (time 1712357712.82 total 70.05 last 88.65) - /unlink 5673 (time 1712357722.83 total 80.06 last 86.54) - /unlink 6230 (time 1712357732.83 total 90.06 last 55.68) - /unlink 6916 (time 1712357742.85 total 100.08 last 68.45) - /unlink 7712 (time 1712357752.85 total 110.09 last 79.59) - /unlink 8702 (time 1712357762.86 total 120.09 last 98.92) - /unlink 9687 (time 1712357772.86 total 130.10 last 98.48) - /unlink 10000 (time 1712357775.72 total 132.96 last 109.46) - /unlink 11110 (time 1712357785.73 total 142.96 last 110.98) - /unlink 12188 (time 1712357795.73 total 152.96 last 107.77) - /unlink 13581 (time 1712357805.74 total 162.97 last 139.21) - /unlink 15102 (time 1712357815.74 total 172.97 last 152.01) - /unlink 16672 (time 1712357825.75 total 182.98 last 156.93) - /unlink 18351 (time 1712357835.75 total 192.98 last 167.89) - /unlink 20000 (time 1712357844.48 total 201.71 last 188.82) - /unlink 22607 (time 1712357854.48 total 211.71 last 260.62) - /unlink 25810 (time 1712357864.48 total 221.71 last 320.30) - /unlink 30000 (time 1712357874.16 total 231.39 last 432.93) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 235.78 seconds: 138.09 ops/second Waiting for MDT destroys to complete - open/close 1358 (time 1712357897.08 total 10.01 last 135.73) - open/close 3121 (time 1712357907.08 total 20.01 last 176.28) - open/close 5387 (time 1712357917.08 total 30.01 last 226.55) - open/close 7163 (time 1712357927.08 total 40.01 last 177.50) - open/close 8547 (time 1712357937.10 total 50.03 last 138.23) - open/close 9972 (time 1712357947.10 total 60.03 last 142.47) - open/close 11316 (time 1712357957.10 total 70.03 last 134.38) - open/close 12902 (time 1712357967.10 total 80.03 last 158.59) - open/close 14218 (time 1712357977.11 total 90.04 last 131.53) - open/close 15384 (time 1712357987.12 total 100.05 last 116.47) - open/close 16328 (time 1712357997.12 total 110.05 last 94.33) - open/close 17272 (time 1712358007.13 total 120.05 last 94.39) - open/close 18074 (time 1712358017.13 total 130.06 last 80.17) - open/close 18839 (time 1712358027.13 total 140.06 last 76.47) - open/close 19594 (time 1712358037.14 total 150.07 last 75.43) - open/close 20000 (time 1712358042.56 total 155.49 last 74.94) - open/close 20748 (time 1712358052.57 total 165.50 last 74.75) - open/close 21741 (time 1712358062.57 total 175.50 last 99.24) - open/close 22737 (time 1712358072.58 total 185.51 last 99.53) - open/close 23650 (time 1712358082.59 total 195.52 last 91.23) - open/close 24589 (time 1712358092.60 total 205.53 last 93.82) - open/close 25497 (time 1712358102.61 total 215.54 last 90.67) - open/close 26282 (time 1712358112.62 total 225.55 last 78.44) - open/close 27016 (time 1712358122.62 total 235.55 last 73.36) - open/close 27759 (time 1712358132.63 total 245.56 last 74.26) - open/close 28479 (time 1712358142.63 total 255.56 last 71.98) - open/close 29187 (time 1712358152.64 total 265.56 last 70.77) - open/close 29870 (time 1712358162.65 total 275.58 last 68.21) - open/close 30586 (time 1712358172.66 total 285.59 last 71.52) - open/close 31258 (time 1712358182.67 total 295.60 last 67.14) - open/close 31899 (time 1712358192.68 total 305.61 last 64.04) - open/close 32511 (time 1712358202.69 total 315.62 last 61.14) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 316.45 seconds: 102.89 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 513 (time 1712358214.48 total 10.01 last 51.23) - /unlink 1018 (time 1712358224.50 total 20.03 last 50.42) - /unlink 1517 (time 1712358234.52 total 30.05 last 49.80) - /unlink 2017 (time 1712358244.53 total 40.06 last 49.95) - /unlink 2479 (time 1712358254.54 total 50.07 last 46.15) - /unlink 2967 (time 1712358264.54 total 60.07 last 48.78) - /unlink 3447 (time 1712358274.55 total 70.08 last 47.96) - /unlink 3948 (time 1712358284.56 total 80.09 last 50.06) - /unlink 4460 (time 1712358294.57 total 90.10 last 51.16) - /unlink 5048 (time 1712358304.57 total 100.10 last 58.80) - /unlink 5741 (time 1712358314.57 total 110.10 last 69.26) - /unlink 6465 (time 1712358324.58 total 120.11 last 72.32) - /unlink 7128 (time 1712358334.59 total 130.12 last 66.28) - /unlink 7953 (time 1712358344.59 total 140.12 last 82.47) - /unlink 8800 (time 1712358354.59 total 150.12 last 84.66) - /unlink 9792 (time 1712358364.60 total 160.13 last 99.17) - /unlink 10000 (time 1712358366.84 total 162.37 last 92.76) - /unlink 10983 (time 1712358376.85 total 172.38 last 98.23) - /unlink 12227 (time 1712358386.85 total 182.38 last 124.33) - /unlink 13620 (time 1712358396.85 total 192.38 last 139.28) - /unlink 15083 (time 1712358406.86 total 202.39 last 146.25) - /unlink 16742 (time 1712358416.86 total 212.39 last 165.84) - /unlink 18558 (time 1712358426.86 total 222.39 last 181.58) - /unlink 20000 (time 1712358434.28 total 229.81 last 194.49) - /unlink 22293 (time 1712358444.28 total 239.81 last 229.23) - /unlink 24830 (time 1712358454.28 total 249.81 last 253.62) - /unlink 28418 (time 1712358464.28 total 259.81 last 358.77) - /unlink 30000 (time 1712358466.94 total 262.47 last 595.06) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 265.47 seconds: 122.65 ops/second Waiting for MDT destroys to complete - open/close 4368 (time 1712358487.10 total 10.00 last 436.67) - open/close 7199 (time 1712358497.11 total 20.01 last 283.01) - open/close 9300 (time 1712358507.11 total 30.01 last 210.08) - open/close 10000 (time 1712358510.79 total 33.69 last 190.13) - open/close 11895 (time 1712358520.79 total 43.69 last 189.43) - open/close 13466 (time 1712358530.80 total 53.70 last 157.03) - open/close 15006 (time 1712358540.80 total 63.70 last 153.91) - open/close 16415 (time 1712358550.81 total 73.71 last 140.86) - open/close 17661 (time 1712358560.81 total 83.71 last 124.52) - open/close 18785 (time 1712358570.82 total 93.72 last 112.32) - open/close 19921 (time 1712358580.82 total 103.72 last 113.55) - open/close 21007 (time 1712358590.83 total 113.73 last 108.56) - open/close 22141 (time 1712358600.84 total 123.74 last 113.24) - open/close 23116 (time 1712358610.84 total 133.74 last 97.49) - open/close 24110 (time 1712358620.85 total 143.75 last 99.32) - open/close 25135 (time 1712358630.86 total 153.76 last 102.42) - open/close 26142 (time 1712358640.86 total 163.76 last 100.68) - open/close 27066 (time 1712358650.87 total 173.77 last 92.32) - open/close 27917 (time 1712358660.87 total 183.77 last 85.07) - open/close 28707 (time 1712358670.88 total 193.78 last 78.93) - open/close 29520 (time 1712358680.89 total 203.79 last 81.22) - open/close 30000 (time 1712358686.69 total 209.59 last 82.78) - open/close 30802 (time 1712358696.70 total 219.59 last 80.18) - open/close 31525 (time 1712358706.70 total 229.60 last 72.25) - open/close 32218 (time 1712358716.72 total 239.62 last 69.18) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 244.72 seconds: 133.05 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 689 (time 1712358732.77 total 10.01 last 68.86) - /unlink 1403 (time 1712358742.79 total 20.02 last 71.28) - /unlink 2125 (time 1712358752.79 total 30.02 last 72.18) - /unlink 2773 (time 1712358762.80 total 40.03 last 64.77) - /unlink 3569 (time 1712358772.81 total 50.04 last 79.51) - /unlink 4470 (time 1712358782.82 total 60.05 last 90.01) - /unlink 5216 (time 1712358792.83 total 70.06 last 74.53) - /unlink 6169 (time 1712358802.84 total 80.07 last 95.20) - /unlink 7152 (time 1712358812.84 total 90.07 last 98.29) - /unlink 8210 (time 1712358822.85 total 100.08 last 105.71) - /unlink 9364 (time 1712358832.85 total 110.09 last 115.31) - /unlink 10000 (time 1712358838.25 total 115.49 last 117.82) - /unlink 11222 (time 1712358848.25 total 125.49 last 122.18) - /unlink 12712 (time 1712358858.26 total 135.49 last 148.93) - /unlink 14198 (time 1712358868.26 total 145.49 last 148.59) - /unlink 15933 (time 1712358878.26 total 155.49 last 173.49) - /unlink 17920 (time 1712358888.26 total 165.50 last 198.67) - /unlink 20000 (time 1712358897.70 total 174.93 last 220.41) - /unlink 22876 (time 1712358907.70 total 184.93 last 287.57) - /unlink 26077 (time 1712358917.70 total 194.93 last 320.09) - /unlink 30000 (time 1712358925.32 total 202.55 last 514.89) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 205.73 seconds: 158.26 ops/second Waiting for MDT destroys to complete - open/close 4414 (time 1712358953.02 total 10.00 last 441.36) - open/close 7311 (time 1712358963.02 total 20.00 last 289.63) - open/close 9837 (time 1712358973.02 total 30.00 last 252.58) - open/close 11647 (time 1712358983.02 total 40.01 last 180.92) - open/close 13710 (time 1712358993.03 total 50.01 last 206.24) - open/close 15080 (time 1712359003.03 total 60.02 last 136.89) - open/close 16595 (time 1712359013.04 total 70.02 last 151.44) - open/close 18034 (time 1712359023.05 total 80.03 last 143.81) - open/close 19456 (time 1712359033.05 total 90.03 last 142.16) - open/close 20000 (time 1712359035.80 total 92.79 last 197.63) - open/close 21372 (time 1712359045.80 total 102.79 last 137.19) - open/close 22958 (time 1712359055.80 total 112.79 last 158.56) - open/close 24223 (time 1712359065.81 total 122.80 last 126.40) - open/close 25267 (time 1712359075.82 total 132.80 last 104.37) - open/close 26246 (time 1712359085.82 total 142.81 last 97.82) - open/close 27361 (time 1712359095.83 total 152.82 last 111.42) - open/close 28639 (time 1712359105.83 total 162.82 last 127.75) - open/close 29942 (time 1712359115.84 total 172.82 last 130.25) - open/close 31101 (time 1712359125.84 total 182.83 last 115.88) - open/close 32126 (time 1712359135.85 total 192.83 last 102.41) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 197.07 seconds: 165.22 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32782 47210 41% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33037 33037 0 100% /mnt/lustre - /unlink 1159 (time 1712359150.96 total 10.00 last 115.85) - /unlink 2342 (time 1712359160.96 total 20.01 last 118.26) - /unlink 3485 (time 1712359170.97 total 30.02 last 114.20) - /unlink 4843 (time 1712359180.98 total 40.02 last 135.75) - /unlink 6181 (time 1712359190.98 total 50.03 last 133.73) - /unlink 7590 (time 1712359200.98 total 60.03 last 140.90) - /unlink 9030 (time 1712359210.99 total 70.03 last 143.94) - /unlink 10000 (time 1712359217.41 total 76.45 last 151.04) - /unlink 11696 (time 1712359227.41 total 86.45 last 169.60) - /unlink 13460 (time 1712359237.41 total 96.45 last 176.36) - /unlink 15611 (time 1712359247.41 total 106.45 last 215.09) - /unlink 17798 (time 1712359257.41 total 116.46 last 218.63) - /unlink 20000 (time 1712359266.87 total 125.91 last 232.90) - /unlink 23058 (time 1712359276.87 total 135.91 last 305.75) - /unlink 27184 (time 1712359286.87 total 145.91 last 412.58) - /unlink 30000 (time 1712359292.10 total 151.14 last 538.42) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 154.30 seconds: 211.02 ops/second Waiting for MDT destroys to complete - open/close 4474 (time 1712359318.27 total 10.00 last 447.33) - open/close 7882 (time 1712359328.27 total 20.00 last 340.78) - open/close 10000 (time 1712359335.23 total 26.96 last 304.24) - open/close 12014 (time 1712359345.24 total 36.97 last 201.32) - open/close 14210 (time 1712359355.24 total 46.97 last 219.59) - open/close 16299 (time 1712359365.24 total 56.97 last 208.87) - open/close 17993 (time 1712359375.25 total 66.98 last 169.28) - open/close 19432 (time 1712359385.25 total 76.98 last 143.87) - open/close 20000 (time 1712359389.73 total 81.46 last 126.62) - open/close 21483 (time 1712359399.74 total 91.47 last 148.23) - open/close 22792 (time 1712359409.74 total 101.47 last 130.86) - open/close 23946 (time 1712359419.75 total 111.48 last 115.31) - open/close 25116 (time 1712359429.76 total 121.49 last 116.88) - open/close 26211 (time 1712359439.77 total 131.50 last 109.44) - open/close 27402 (time 1712359449.77 total 141.50 last 119.07) - open/close 28699 (time 1712359459.77 total 151.50 last 129.63) - open/close 29674 (time 1712359469.78 total 161.51 last 97.47) - open/close 30000 (time 1712359472.92 total 164.65 last 103.78) - open/close 30983 (time 1712359482.92 total 174.65 last 98.25) - open/close 32026 (time 1712359492.93 total 184.66 last 104.26) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 190.33 seconds: 171.07 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1135 (time 1712359509.48 total 10.00 last 113.49) - /unlink 2369 (time 1712359519.48 total 20.00 last 123.39) - /unlink 3583 (time 1712359529.49 total 30.01 last 121.35) - /unlink 4730 (time 1712359539.49 total 40.01 last 114.64) - /unlink 5834 (time 1712359549.50 total 50.02 last 110.35) - /unlink 6961 (time 1712359559.50 total 60.02 last 112.68) - /unlink 8033 (time 1712359569.50 total 70.02 last 107.18) - /unlink 9512 (time 1712359579.50 total 80.02 last 147.89) - /unlink 10000 (time 1712359582.51 total 83.03 last 162.22) - /unlink 11175 (time 1712359592.51 total 93.03 last 117.49) - /unlink 12968 (time 1712359602.51 total 103.03 last 179.25) - /unlink 14449 (time 1712359612.51 total 113.03 last 148.10) - /unlink 16032 (time 1712359622.52 total 123.04 last 158.25) - /unlink 18403 (time 1712359632.52 total 133.04 last 237.08) - /unlink 20000 (time 1712359639.35 total 139.87 last 233.76) - /unlink 22693 (time 1712359649.35 total 149.87 last 269.22) - /unlink 26627 (time 1712359659.35 total 159.87 last 393.33) - /unlink 30000 (time 1712359664.95 total 165.47 last 602.50) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 168.55 seconds: 193.18 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server unloading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 PASS 106 (2414s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg256-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg256-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg256-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg256-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 oleg256-server: oleg256-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: detected 4 online CPUs by sysfs oleg256-server: Force libcfs to create 2 CPU partitions oleg256-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg256-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 2524 sec ========== 19:28:34 (1712359714) === conf-sanity: start cleanup 19:28:34 (1712359714) === === conf-sanity: finish cleanup 19:28:34 (1712359714) ===