== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:13:11 (1755695591) Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) oleg159-server: oleg159-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg159-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2823 (time 1755695642.60 total 10.00 last 282.21) - open/close 5986 (time 1755695652.60 total 20.00 last 316.26) - open/close 9255 (time 1755695662.60 total 30.00 last 326.89) - open/close 10000 (time 1755695665.32 total 32.72 last 274.41) - open/close 12529 (time 1755695675.32 total 42.72 last 252.83) - open/close 15656 (time 1755695685.32 total 52.73 last 312.57) - open/close 18371 (time 1755695695.33 total 62.73 last 271.47) - open/close 20000 (time 1755695700.20 total 67.60 last 334.29) - open/close 23754 (time 1755695710.20 total 77.60 last 375.39) - open/close 26704 (time 1755695720.20 total 87.60 last 294.94) - open/close 29242 (time 1755695730.20 total 97.61 last 253.71) - open/close 30000 (time 1755695733.25 total 100.65 last 249.17) - open/close 32558 (time 1755695743.25 total 110.65 last 255.75) - open/close 34847 (time 1755695753.25 total 120.65 last 228.81) - open/close 37146 (time 1755695763.26 total 130.66 last 229.81) - open/close 39488 (time 1755695773.26 total 140.66 last 234.20) - open/close 40000 (time 1755695775.31 total 142.71 last 249.76) - open/close 42619 (time 1755695785.31 total 152.71 last 261.86) - open/close 44877 (time 1755695795.31 total 162.71 last 225.75) - open/close 46992 (time 1755695805.31 total 172.72 last 211.40) - open/close 49709 (time 1755695815.31 total 182.72 last 271.69) - open/close 52508 (time 1755695825.32 total 192.72 last 279.77) - open/close 55140 (time 1755695835.32 total 202.72 last 263.14) - open/close 57817 (time 1755695845.32 total 212.73 last 267.62) - open/close 60000 (time 1755695855.00 total 222.40 last 225.65) - open/close 62678 (time 1755695865.00 total 232.41 last 267.68) total: 64768 open/close in 239.46 seconds: 270.47 ops/second - /unlink 6157 (time 1755695883.68 total 10.00 last 615.66) - /unlink 10000 (time 1755695891.08 total 17.40 last 519.06) - /unlink 13749 (time 1755695901.08 total 27.40 last 374.88) - /unlink 18479 (time 1755695911.08 total 37.41 last 472.96) - /unlink 20000 (time 1755695913.33 total 39.65 last 676.75) - /unlink 25961 (time 1755695923.33 total 49.66 last 595.88) - /unlink 30000 (time 1755695930.92 total 57.24 last 532.47) - /unlink 36644 (time 1755695940.92 total 67.24 last 664.25) - /unlink 40000 (time 1755695946.91 total 73.24 last 559.93) - /unlink 45582 (time 1755695956.92 total 83.24 last 558.10) - /unlink 50000 (time 1755695963.54 total 89.86 last 666.90) - /unlink 57119 (time 1755695973.54 total 99.87 last 711.78) - /unlink 60000 (time 1755695977.75 total 104.07 last 685.05) total: 64768 /unlink in 111.51 seconds: 580.80 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2765 (time 1755696014.62 total 10.00 last 276.48) - open/close 5659 (time 1755696024.62 total 20.00 last 289.33) - open/close 8481 (time 1755696034.62 total 30.01 last 282.12) - open/close 10000 (time 1755696041.25 total 36.63 last 229.26) - open/close 12532 (time 1755696051.25 total 46.63 last 253.18) - open/close 15631 (time 1755696061.25 total 56.63 last 309.87) - open/close 18188 (time 1755696071.25 total 66.63 last 255.69) - open/close 20000 (time 1755696078.36 total 73.75 last 254.80) - open/close 22587 (time 1755696088.37 total 83.75 last 258.66) - open/close 26018 (time 1755696098.37 total 93.75 last 343.10) - open/close 29262 (time 1755696108.37 total 103.75 last 324.37) - open/close 30000 (time 1755696111.11 total 106.49 last 269.47) - open/close 32867 (time 1755696121.11 total 116.49 last 286.64) - open/close 35149 (time 1755696131.11 total 126.49 last 228.16) - open/close 38089 (time 1755696141.11 total 136.49 last 293.89) - open/close 40000 (time 1755696147.77 total 143.15 last 287.12) - open/close 42749 (time 1755696157.77 total 153.15 last 274.79) - open/close 45360 (time 1755696167.77 total 163.16 last 261.06) - open/close 48320 (time 1755696177.78 total 173.16 last 295.97) - open/close 50000 (time 1755696183.62 total 179.00 last 287.42) - open/close 52564 (time 1755696193.62 total 189.00 last 256.34) - open/close 54860 (time 1755696203.63 total 199.01 last 229.53) - open/close 57236 (time 1755696213.63 total 209.01 last 237.60) - open/close 60000 (time 1755696221.82 total 217.20 last 337.47) - open/close 63066 (time 1755696231.82 total 227.20 last 306.55) total: 64768 open/close in 232.53 seconds: 278.53 ops/second - /unlink 5997 (time 1755696248.40 total 10.00 last 599.67) - /unlink 10000 (time 1755696255.57 total 17.17 last 558.25) - /unlink 15755 (time 1755696265.57 total 27.17 last 575.44) - /unlink 20000 (time 1755696272.27 total 33.87 last 633.32) - /unlink 26083 (time 1755696282.28 total 43.88 last 608.24) - /unlink 30000 (time 1755696289.81 total 51.41 last 519.63) - /unlink 36215 (time 1755696299.82 total 61.42 last 621.19) - /unlink 40000 (time 1755696306.21 total 67.81 last 592.41) - /unlink 45025 (time 1755696316.21 total 77.81 last 502.43) - /unlink 50000 (time 1755696324.09 total 85.69 last 631.09) - /unlink 54805 (time 1755696334.09 total 95.69 last 480.46) - /unlink 60000 (time 1755696342.26 total 103.86 last 635.98) total: 64768 /unlink in 112.60 seconds: 575.21 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3934 (time 1755696385.78 total 10.00 last 393.27) - open/close 8556 (time 1755696395.78 total 20.00 last 462.19) - open/close 10000 (time 1755696398.83 total 23.05 last 473.31) - open/close 14415 (time 1755696408.84 total 33.06 last 441.40) - open/close 17765 (time 1755696418.84 total 43.06 last 334.94) - open/close 20000 (time 1755696424.56 total 48.78 last 390.63) - open/close 23783 (time 1755696434.56 total 58.78 last 378.23) - open/close 26948 (time 1755696444.56 total 68.79 last 316.40) - open/close 30000 (time 1755696452.86 total 77.08 last 367.92) - open/close 33536 (time 1755696462.86 total 87.08 last 353.58) - open/close 37088 (time 1755696472.86 total 97.08 last 355.14) - open/close 40000 (time 1755696482.63 total 106.85 last 298.25) - open/close 42633 (time 1755696492.63 total 116.85 last 263.18) - open/close 46174 (time 1755696502.63 total 126.85 last 354.04) - open/close 49981 (time 1755696512.63 total 136.85 last 380.66) - open/close 53359 (time 1755696522.63 total 146.86 last 337.75) - open/close 56551 (time 1755696532.64 total 156.86 last 319.06) - open/close 60000 (time 1755696542.48 total 166.71 last 350.29) - open/close 62846 (time 1755696552.49 total 176.71 last 284.52) total: 64768 open/close in 183.28 seconds: 353.38 ops/second - /unlink 6191 (time 1755696570.69 total 10.00 last 619.07) - /unlink 10000 (time 1755696575.83 total 15.14 last 740.96) - /unlink 17707 (time 1755696585.83 total 25.14 last 770.65) - /unlink 20000 (time 1755696588.43 total 27.75 last 880.73) - /unlink 25995 (time 1755696598.44 total 37.75 last 599.16) - /unlink 30000 (time 1755696605.09 total 44.41 last 601.83) - /unlink 34801 (time 1755696615.10 total 54.41 last 480.04) - /unlink 39693 (time 1755696625.10 total 64.41 last 489.16) - /unlink 45053 (time 1755696635.12 total 74.43 last 534.84) - /unlink 50000 (time 1755696643.11 total 82.43 last 618.71) - /unlink 57115 (time 1755696653.11 total 92.43 last 711.43) - /unlink 60000 (time 1755696656.85 total 96.16 last 773.09) total: 64768 /unlink in 103.89 seconds: 623.41 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2863 (time 1755696696.06 total 10.00 last 286.27) - open/close 5995 (time 1755696706.06 total 20.00 last 313.13) - open/close 10000 (time 1755696715.78 total 29.73 last 411.77) - open/close 13776 (time 1755696725.79 total 39.73 last 377.49) - open/close 16800 (time 1755696735.79 total 49.74 last 302.27) - open/close 20000 (time 1755696744.31 total 58.25 last 375.87) - open/close 22978 (time 1755696754.31 total 68.25 last 297.76) - open/close 26138 (time 1755696764.31 total 78.25 last 315.94) - open/close 29075 (time 1755696774.31 total 88.25 last 293.68) - open/close 30000 (time 1755696776.41 total 90.35 last 441.09) - open/close 33772 (time 1755696786.41 total 100.35 last 377.20) - open/close 37517 (time 1755696796.41 total 110.35 last 374.43) - open/close 39017 (time 1755696806.41 total 120.36 last 149.95) - open/close 40000 (time 1755696813.77 total 127.71 last 133.64) - open/close 43890 (time 1755696823.77 total 137.71 last 388.97) - open/close 48256 (time 1755696833.77 total 147.71 last 436.55) - open/close 50000 (time 1755696837.82 total 151.76 last 430.71) - open/close 54365 (time 1755696847.82 total 161.77 last 436.41) - open/close 58637 (time 1755696857.82 total 171.77 last 427.11) - open/close 60000 (time 1755696860.95 total 174.89 last 435.95) - open/close 64447 (time 1755696870.95 total 184.90 last 444.63) total: 64768 open/close in 185.69 seconds: 348.79 ops/second - /unlink 7112 (time 1755696883.04 total 10.00 last 711.12) - /unlink 10000 (time 1755696886.67 total 13.63 last 795.66) - /unlink 17330 (time 1755696896.67 total 23.63 last 732.96) - /unlink 20000 (time 1755696900.44 total 27.40 last 708.49) - /unlink 27237 (time 1755696910.44 total 37.40 last 723.62) - /unlink 30000 (time 1755696914.44 total 41.40 last 691.69) - /unlink 37276 (time 1755696924.44 total 51.40 last 727.60) - /unlink 40000 (time 1755696927.70 total 54.66 last 835.48) - /unlink 47248 (time 1755696937.70 total 64.66 last 724.76) - /unlink 50000 (time 1755696941.52 total 68.47 last 720.87) - /unlink 57169 (time 1755696951.52 total 78.47 last 716.84) - /unlink 60000 (time 1755696955.47 total 82.43 last 716.65) total: 64768 /unlink in 88.82 seconds: 729.23 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3632 (time 1755696992.07 total 10.00 last 363.12) - open/close 7980 (time 1755697002.07 total 20.00 last 434.80) - open/close 10000 (time 1755697007.87 total 25.80 last 348.33) - open/close 14009 (time 1755697017.87 total 35.80 last 400.85) - open/close 17851 (time 1755697027.87 total 45.80 last 384.17) - open/close 20000 (time 1755697033.81 total 51.74 last 362.27) - open/close 23951 (time 1755697043.81 total 61.74 last 395.07) - open/close 27268 (time 1755697053.81 total 71.74 last 331.66) - open/close 30000 (time 1755697061.79 total 79.72 last 342.06) - open/close 33412 (time 1755697071.80 total 89.73 last 341.15) - open/close 36784 (time 1755697081.80 total 99.73 last 337.08) - open/close 40000 (time 1755697090.58 total 108.51 last 366.20) - open/close 43030 (time 1755697100.58 total 118.51 last 302.95) - open/close 46420 (time 1755697110.58 total 128.51 last 338.97) - open/close 48524 (time 1755697120.59 total 138.52 last 210.37) - open/close 50000 (time 1755697127.29 total 145.22 last 220.21) - open/close 51601 (time 1755697137.29 total 155.22 last 160.03) - open/close 53313 (time 1755697147.30 total 165.23 last 171.10) - open/close 55581 (time 1755697157.30 total 175.23 last 226.78) - open/close 59091 (time 1755697167.30 total 185.23 last 350.96) - open/close 60000 (time 1755697169.47 total 187.40 last 418.30) total: 64768 open/close in 197.29 seconds: 328.29 ops/second - /unlink 6964 (time 1755697190.62 total 10.00 last 696.27) - /unlink 10000 (time 1755697195.88 total 15.26 last 576.85) - /unlink 16733 (time 1755697205.88 total 25.27 last 673.23) - /unlink 20000 (time 1755697210.10 total 29.49 last 773.76) - /unlink 27678 (time 1755697220.11 total 39.49 last 767.75) - /unlink 30000 (time 1755697222.82 total 42.20 last 856.43) - /unlink 37451 (time 1755697232.82 total 52.20 last 745.04) - /unlink 40000 (time 1755697237.05 total 56.43 last 602.71) - /unlink 47841 (time 1755697247.05 total 66.43 last 783.83) - /unlink 50000 (time 1755697250.49 total 69.87 last 627.76) - /unlink 56962 (time 1755697260.49 total 79.87 last 696.18) - /unlink 60000 (time 1755697265.30 total 84.68 last 631.74) total: 64768 /unlink in 93.39 seconds: 693.55 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2