== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:42:53 (1774518173) Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:-f) oleg132-server: oleg132-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg132-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg132-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock oleg132-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 772 (time 1774518330.75 total 10.00 last 77.17) - open/close 1420 (time 1774518340.76 total 20.01 last 64.73) - open/close 2129 (time 1774518350.77 total 30.02 last 70.86) - open/close 2862 (time 1774518360.77 total 40.02 last 73.27) - open/close 3571 (time 1774518370.78 total 50.04 last 70.79) - open/close 4354 (time 1774518380.79 total 60.04 last 78.27) - open/close 5141 (time 1774518390.80 total 70.05 last 78.63) - open/close 5890 (time 1774518400.81 total 80.06 last 74.84) - open/close 6648 (time 1774518410.81 total 90.07 last 75.74) - open/close 7294 (time 1774518420.82 total 100.08 last 64.53) - open/close 7900 (time 1774518430.83 total 110.08 last 60.57) - open/close 8486 (time 1774518440.87 total 120.13 last 58.33) - open/close 9189 (time 1774518450.88 total 130.13 last 70.28) - open/close 9841 (time 1774518460.89 total 140.15 last 65.10) - open/close 10636 (time 1774518470.90 total 150.16 last 79.44) - open/close 11239 (time 1774518480.90 total 160.16 last 60.29) - open/close 12068 (time 1774518490.91 total 170.16 last 82.88) - open/close 12795 (time 1774518500.91 total 180.17 last 72.64) - open/close 13537 (time 1774518510.91 total 190.17 last 74.20) - open/close 14270 (time 1774518520.92 total 200.17 last 73.27) - open/close 15022 (time 1774518530.94 total 210.19 last 75.04) - open/close 15801 (time 1774518540.94 total 220.20 last 77.89) - open/close 16540 (time 1774518550.95 total 230.21 last 73.83) - open/close 17283 (time 1774518560.95 total 240.21 last 74.27) - open/close 18081 (time 1774518570.96 total 250.21 last 79.78) - open/close 18923 (time 1774518580.96 total 260.21 last 84.20) - open/close 19768 (time 1774518590.96 total 270.21 last 84.49) - open/close 20000 (time 1774518593.59 total 272.84 last 88.16) - open/close 20860 (time 1774518603.59 total 282.85 last 85.96) - open/close 21719 (time 1774518613.60 total 292.86 last 85.83) - open/close 22650 (time 1774518623.61 total 302.87 last 93.03) - open/close 23535 (time 1774518633.61 total 312.87 last 88.48) - open/close 24439 (time 1774518643.62 total 322.87 last 90.35) - open/close 25327 (time 1774518653.63 total 332.88 last 88.73) - open/close 26301 (time 1774518663.63 total 342.89 last 97.32) - open/close 27200 (time 1774518673.64 total 352.90 last 89.85) - open/close 28173 (time 1774518683.64 total 362.90 last 97.25) - open/close 29146 (time 1774518693.66 total 372.92 last 97.15) - open/close 30000 (time 1774518701.92 total 381.18 last 103.37) - open/close 31119 (time 1774518711.93 total 391.18 last 111.83) - open/close 32180 (time 1774518721.93 total 401.19 last 106.04) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 405.41 seconds: 80.40 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1924 (time 1774518738.99 total 10.00 last 192.32) - /unlink 3770 (time 1774518748.99 total 20.00 last 184.59) - /unlink 6255 (time 1774518758.99 total 30.01 last 248.44) - /unlink 8299 (time 1774518768.99 total 40.01 last 204.38) - /unlink 10000 (time 1774518777.50 total 48.52 last 199.91) - /unlink 11958 (time 1774518787.50 total 58.52 last 195.77) - /unlink 13859 (time 1774518797.51 total 68.52 last 190.06) - /unlink 15734 (time 1774518807.51 total 78.52 last 187.50) - /unlink 17534 (time 1774518817.51 total 88.52 last 179.98) - /unlink 19691 (time 1774518827.51 total 98.53 last 215.61) - /unlink 21561 (time 1774518837.51 total 108.53 last 186.97) - /unlink 23781 (time 1774518847.52 total 118.53 last 221.93) - /unlink 25747 (time 1774518857.53 total 128.54 last 196.43) - /unlink 27685 (time 1774518867.53 total 138.54 last 193.78) - /unlink 30000 (time 1774518877.42 total 148.43 last 234.07) - /unlink 32199 (time 1774518887.42 total 158.43 last 219.89) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 160.04 seconds: 203.66 ops/second Waiting for MDT destroys to complete - open/close 1356 (time 1774518912.00 total 10.01 last 135.52) - open/close 2400 (time 1774518922.00 total 20.01 last 104.39) - open/close 3501 (time 1774518932.00 total 30.01 last 110.08) - open/close 4696 (time 1774518942.00 total 40.01 last 119.50) - open/close 5731 (time 1774518952.00 total 50.01 last 103.48) - open/close 6776 (time 1774518962.01 total 60.02 last 104.45) - open/close 7870 (time 1774518972.03 total 70.04 last 109.18) - open/close 9001 (time 1774518982.03 total 80.04 last 113.10) - open/close 10000 (time 1774518990.31 total 88.32 last 120.58) - open/close 11109 (time 1774519000.32 total 98.33 last 110.80) - open/close 12256 (time 1774519010.33 total 108.34 last 114.64) - open/close 13470 (time 1774519020.34 total 118.34 last 121.31) - open/close 14608 (time 1774519030.34 total 128.35 last 113.76) - open/close 15854 (time 1774519040.35 total 138.36 last 124.48) - open/close 17000 (time 1774519050.35 total 148.36 last 114.60) - open/close 18061 (time 1774519060.36 total 158.37 last 105.97) - open/close 19214 (time 1774519070.37 total 168.37 last 115.24) - open/close 20000 (time 1774519076.47 total 174.48 last 128.83) - open/close 21448 (time 1774519086.48 total 184.48 last 144.70) - open/close 23120 (time 1774519096.48 total 194.49 last 167.12) - open/close 24532 (time 1774519106.48 total 204.49 last 141.17) - open/close 26100 (time 1774519116.49 total 214.50 last 156.69) - open/close 27412 (time 1774519126.49 total 224.50 last 131.16) - open/close 28779 (time 1774519136.49 total 234.50 last 136.69) - open/close 30000 (time 1774519144.03 total 242.03 last 162.07) - open/close 31327 (time 1774519154.03 total 252.04 last 132.64) - open/close 32469 (time 1774519164.04 total 262.05 last 114.13) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 263.13 seconds: 123.87 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 2220 (time 1774519177.70 total 10.00 last 221.91) - /unlink 4481 (time 1774519187.70 total 20.01 last 225.99) - /unlink 7250 (time 1774519197.70 total 30.01 last 276.83) - /unlink 9734 (time 1774519207.71 total 40.01 last 248.33) - /unlink 12295 (time 1774519217.72 total 50.03 last 255.73) - /unlink 14908 (time 1774519227.73 total 60.04 last 261.05) - /unlink 18178 (time 1774519237.73 total 70.04 last 326.99) - /unlink 20000 (time 1774519245.17 total 77.48 last 244.92) - /unlink 22673 (time 1774519255.17 total 87.48 last 267.24) - /unlink 25259 (time 1774519265.17 total 97.48 last 258.59) - /unlink 27590 (time 1774519275.17 total 107.48 last 233.09) - /unlink 30000 (time 1774519284.77 total 117.08 last 251.08) - /unlink 32300 (time 1774519294.77 total 127.08 last 229.96) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 128.33 seconds: 253.98 ops/second Waiting for MDT destroys to complete - open/close 1443 (time 1774519315.05 total 10.01 last 144.18) - open/close 2668 (time 1774519325.06 total 20.01 last 122.42) - open/close 3874 (time 1774519335.06 total 30.02 last 120.53) - open/close 5215 (time 1774519345.07 total 40.02 last 134.09) - open/close 6567 (time 1774519355.07 total 50.02 last 135.15) - open/close 7909 (time 1774519365.08 total 60.03 last 134.09) - open/close 9188 (time 1774519375.08 total 70.03 last 127.88) - open/close 10000 (time 1774519380.77 total 75.73 last 142.65) - open/close 11349 (time 1774519390.77 total 85.73 last 134.89) - open/close 12757 (time 1774519400.77 total 95.73 last 140.78) - open/close 14086 (time 1774519410.78 total 105.73 last 132.85) - open/close 15636 (time 1774519420.78 total 115.73 last 154.96) - open/close 17418 (time 1774519430.78 total 125.74 last 178.13) - open/close 18926 (time 1774519440.79 total 135.74 last 150.71) - open/close 20000 (time 1774519448.25 total 143.21 last 143.88) - open/close 21435 (time 1774519458.26 total 153.21 last 143.42) - open/close 23252 (time 1774519468.26 total 163.22 last 181.69) - open/close 24651 (time 1774519478.27 total 173.22 last 139.81) - open/close 26021 (time 1774519488.27 total 183.23 last 136.94) - open/close 27646 (time 1774519498.28 total 193.23 last 162.42) - open/close 29289 (time 1774519508.28 total 203.23 last 164.28) - open/close 30000 (time 1774519512.51 total 207.46 last 168.05) - open/close 31581 (time 1774519522.51 total 217.46 last 158.06) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 223.56 seconds: 145.80 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 2921 (time 1774519540.52 total 10.00 last 292.04) - /unlink 5671 (time 1774519550.52 total 20.00 last 274.99) - /unlink 8227 (time 1774519560.52 total 30.00 last 255.59) - /unlink 10000 (time 1774519567.19 total 36.67 last 265.83) - /unlink 12806 (time 1774519577.19 total 46.68 last 280.49) - /unlink 15370 (time 1774519587.19 total 56.68 last 256.39) - /unlink 18269 (time 1774519597.19 total 66.68 last 289.89) - /unlink 20000 (time 1774519603.25 total 72.73 last 285.89) - /unlink 22833 (time 1774519613.25 total 82.73 last 283.26) - /unlink 26191 (time 1774519623.25 total 92.74 last 335.70) - /unlink 29329 (time 1774519633.25 total 102.74 last 313.76) - /unlink 30000 (time 1774519636.25 total 105.73 last 223.87) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 114.40 seconds: 284.90 ops/second Waiting for MDT destroys to complete - open/close 2146 (time 1774519666.54 total 10.00 last 214.53) - open/close 3777 (time 1774519676.55 total 20.01 last 162.99) - open/close 6183 (time 1774519686.56 total 30.02 last 240.34) - open/close 7962 (time 1774519696.57 total 40.03 last 177.78) - open/close 9614 (time 1774519706.57 total 50.03 last 165.18) - open/close 10000 (time 1774519708.77 total 52.23 last 175.28) - open/close 12206 (time 1774519718.77 total 62.23 last 220.59) - open/close 14978 (time 1774519728.77 total 72.23 last 277.19) - open/close 17408 (time 1774519738.77 total 82.23 last 242.96) - open/close 19495 (time 1774519748.78 total 92.24 last 208.66) - open/close 20000 (time 1774519751.43 total 94.89 last 190.07) - open/close 21840 (time 1774519761.43 total 104.89 last 183.97) - open/close 23505 (time 1774519771.43 total 114.89 last 166.49) - open/close 25575 (time 1774519781.44 total 124.90 last 206.94) - open/close 27872 (time 1774519791.44 total 134.90 last 229.60) - open/close 30000 (time 1774519799.78 total 143.24 last 255.14) - open/close 32524 (time 1774519809.79 total 153.25 last 252.26) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 153.56 seconds: 212.26 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 3597 (time 1774519821.97 total 10.00 last 359.64) - /unlink 7084 (time 1774519831.97 total 20.00 last 348.62) - /unlink 10000 (time 1774519840.52 total 28.55 last 341.24) - /unlink 13313 (time 1774519850.52 total 38.55 last 331.28) - /unlink 16781 (time 1774519860.52 total 48.55 last 346.79) - /unlink 20000 (time 1774519868.51 total 56.54 last 402.65) - /unlink 23961 (time 1774519878.51 total 66.55 last 396.09) - /unlink 27064 (time 1774519888.52 total 76.55 last 310.21) - /unlink 30000 (time 1774519896.03 total 84.06 last 390.74) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 90.94 seconds: 358.41 ops/second Waiting for MDT destroys to complete - open/close 1773 (time 1774519927.04 total 10.00 last 177.26) - open/close 4122 (time 1774519937.04 total 20.00 last 234.87) - open/close 6739 (time 1774519947.04 total 30.01 last 261.65) - open/close 8787 (time 1774519957.04 total 40.01 last 204.79) - open/close 10000 (time 1774519962.24 total 45.20 last 233.38) - open/close 12019 (time 1774519972.24 total 55.21 last 201.85) - open/close 13891 (time 1774519982.25 total 65.21 last 187.15) - open/close 16213 (time 1774519992.25 total 75.21 last 232.18) - open/close 18110 (time 1774520002.25 total 85.21 last 189.59) - open/close 19883 (time 1774520012.25 total 95.22 last 177.29) - open/close 21464 (time 1774520022.26 total 105.22 last 158.02) - open/close 23771 (time 1774520032.26 total 115.22 last 230.59) - open/close 25927 (time 1774520042.27 total 125.23 last 215.55) - open/close 28292 (time 1774520052.27 total 135.23 last 236.41) - open/close 30000 (time 1774520058.24 total 141.21 last 285.88) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 149.61 seconds: 217.86 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4859 (time 1774520078.49 total 10.00 last 485.84) - /unlink 10000 (time 1774520087.40 total 18.91 last 576.82) - /unlink 16059 (time 1774520097.40 total 28.91 last 605.86) - /unlink 20000 (time 1774520103.19 total 34.70 last 680.84) - /unlink 27348 (time 1774520113.19 total 44.70 last 734.79) - /unlink 30000 (time 1774520116.67 total 48.19 last 761.03) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 51.62 seconds: 631.46 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local oleg132-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg132-server: LNET ready to unload oleg132-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2