== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:18:53 (1785421133) Stopping clients: oleg104-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg104-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg104-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg104-server oleg104-server: oleg104-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg104-server: detected 4 online CPUs by sysfs oleg104-server: MODOPTS_LIBCFS= oleg104-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg104-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg104-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg104-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 606 (time 1785421382.97 total 10.01 last 60.54) - open/close 1148 (time 1785421392.97 total 20.01 last 54.19) - open/close 1940 (time 1785421402.99 total 30.04 last 79.00) - open/close 2830 (time 1785421413.00 total 40.04 last 88.96) - open/close 3590 (time 1785421423.00 total 50.04 last 75.99) - open/close 4250 (time 1785421433.01 total 60.05 last 65.95) total: 4500 open/close in 62.92 seconds: 71.52 ops/second - /unlink 1262 (time 1785421451.08 total 10.01 last 126.13) - /unlink 2199 (time 1785421461.10 total 20.02 last 93.56) - /unlink 2897 (time 1785421471.56 total 30.49 last 66.69) - /unlink 4262 (time 1785421481.57 total 40.50 last 136.35) total: 4500 /unlink in 42.98 seconds: 104.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 507 (time 1785421501.03 total 10.00 last 50.69) - open/close 1185 (time 1785421511.04 total 20.02 last 67.69) - open/close 1940 (time 1785421521.04 total 30.02 last 75.49) - open/close 2539 (time 1785421531.05 total 40.02 last 59.89) - open/close 3078 (time 1785421541.05 total 50.03 last 53.86) - open/close 3586 (time 1785421551.06 total 60.04 last 50.76) - open/close 4040 (time 1785421561.07 total 70.04 last 45.36) total: 4500 open/close in 79.60 seconds: 56.53 ops/second - /unlink 1337 (time 1785421588.35 total 10.00 last 133.68) - /unlink 2580 (time 1785421598.36 total 20.01 last 124.25) - /unlink 3957 (time 1785421608.36 total 30.01 last 137.66) total: 4500 /unlink in 33.54 seconds: 134.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 546 (time 1785421626.58 total 10.02 last 54.50) - open/close 1217 (time 1785421636.59 total 20.03 last 67.04) - open/close 1806 (time 1785421646.60 total 30.03 last 58.87) - open/close 2254 (time 1785421656.60 total 40.03 last 44.79) - open/close 2718 (time 1785421666.60 total 50.04 last 46.39) - open/close 3256 (time 1785421676.61 total 60.05 last 53.73) - open/close 3699 (time 1785421686.63 total 70.06 last 44.25) - open/close 4052 (time 1785421696.63 total 80.06 last 35.28) total: 4500 open/close in 89.42 seconds: 50.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1068 (time 1785421723.05 total 10.00 last 106.76) - /unlink 2064 (time 1785421733.05 total 20.01 last 99.56) - /unlink 3092 (time 1785421743.05 total 30.01 last 102.78) - /unlink 4044 (time 1785421753.05 total 40.01 last 95.19) total: 4500 /unlink in 45.95 seconds: 97.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 609 (time 1785421776.27 total 10.03 last 60.73) - open/close 1204 (time 1785421786.28 total 20.04 last 59.44) - open/close 1792 (time 1785421796.29 total 30.04 last 58.77) - open/close 2593 (time 1785421806.30 total 40.05 last 80.01) - open/close 3318 (time 1785421816.30 total 50.06 last 72.47) - open/close 3926 (time 1785421826.31 total 60.07 last 60.75) total: 4500 open/close in 67.75 seconds: 66.42 ops/second - /unlink 1404 (time 1785421848.92 total 10.00 last 140.34) - /unlink 2777 (time 1785421858.93 total 20.01 last 137.20) - /unlink 3673 (time 1785421868.95 total 30.03 last 89.44) total: 4500 /unlink in 36.94 seconds: 121.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 466 (time 1785421892.86 total 10.01 last 46.58) - open/close 1002 (time 1785421902.88 total 20.03 last 53.49) - open/close 1715 (time 1785421912.89 total 30.03 last 71.24) - open/close 2358 (time 1785421922.89 total 40.04 last 64.28) - open/close 2829 (time 1785421932.90 total 50.05 last 47.06) - open/close 3531 (time 1785421942.91 total 60.05 last 70.15) - open/close 4240 (time 1785421952.91 total 70.06 last 70.83) total: 4500 open/close in 74.35 seconds: 60.52 ops/second - /unlink 1460 (time 1785421972.91 total 10.00 last 145.99) - /unlink 2364 (time 1785421982.92 total 20.01 last 90.35) - /unlink 3624 (time 1785421992.92 total 30.01 last 125.92) total: 4500 /unlink in 38.52 seconds: 116.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 638 (time 1785422017.73 total 10.01 last 63.75) - open/close 1302 (time 1785422027.74 total 20.02 last 66.30) - open/close 1903 (time 1785422037.75 total 30.03 last 60.06) - open/close 2326 (time 1785422047.75 total 40.03 last 42.29) - open/close 2749 (time 1785422057.76 total 50.04 last 42.28) - open/close 3258 (time 1785422067.76 total 60.04 last 50.89) - open/close 3709 (time 1785422077.77 total 70.05 last 45.04) - open/close 4353 (time 1785422087.79 total 80.07 last 64.30) total: 4500 open/close in 82.49 seconds: 54.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1024 (time 1785422106.63 total 10.01 last 102.27) - /unlink 1718 (time 1785422116.64 total 20.03 last 69.28) - /unlink 2352 (time 1785422126.68 total 30.07 last 63.16) - /unlink 2904 (time 1785422136.68 total 40.07 last 55.19) - /unlink 3713 (time 1785422146.69 total 50.08 last 80.83) total: 4500 /unlink in 57.97 seconds: 77.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 492 (time 1785422171.09 total 10.00 last 49.19) - open/close 1007 (time 1785422181.09 total 20.01 last 51.48) - open/close 1610 (time 1785422191.12 total 30.04 last 60.11) - open/close 2421 (time 1785422201.13 total 40.04 last 81.06) - open/close 3058 (time 1785422211.14 total 50.06 last 63.62) - open/close 3658 (time 1785422221.16 total 60.07 last 59.89) - open/close 4190 (time 1785422231.17 total 70.09 last 53.13) total: 4500 open/close in 75.64 seconds: 59.49 ops/second - /unlink 1197 (time 1785422252.60 total 10.02 last 119.50) - /unlink 2342 (time 1785422262.61 total 20.03 last 114.39) - /unlink 3200 (time 1785422272.61 total 30.03 last 85.79) - /unlink 4106 (time 1785422282.61 total 40.03 last 90.58) total: 4500 /unlink in 44.29 seconds: 101.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 594 (time 1785422303.02 total 10.01 last 59.37) - open/close 1370 (time 1785422313.03 total 20.02 last 77.51) - open/close 2139 (time 1785422323.03 total 30.02 last 76.88) - open/close 2698 (time 1785422333.03 total 40.02 last 55.89) - open/close 3172 (time 1785422343.04 total 50.03 last 47.35) - open/close 3812 (time 1785422353.07 total 60.06 last 63.82) - open/close 4475 (time 1785422363.07 total 70.06 last 66.28) total: 4500 open/close in 70.41 seconds: 63.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1130 (time 1785422377.93 total 10.00 last 112.96) - /unlink 2352 (time 1785422387.93 total 20.01 last 122.12) - /unlink 3624 (time 1785422397.94 total 30.02 last 127.07) total: 4500 /unlink in 39.29 seconds: 114.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 510 (time 1785422422.97 total 10.00 last 50.99) - open/close 1121 (time 1785422432.99 total 20.02 last 60.98) - open/close 1567 (time 1785422443.01 total 30.04 last 44.52) - open/close 2147 (time 1785422453.08 total 40.10 last 57.64) - open/close 2997 (time 1785422463.10 total 50.13 last 84.77) - open/close 3643 (time 1785422473.11 total 60.13 last 64.58) - open/close 4384 (time 1785422483.12 total 70.15 last 73.96) total: 4500 open/close in 72.32 seconds: 62.22 ops/second - /unlink 1074 (time 1785422501.34 total 10.01 last 107.33) - /unlink 2281 (time 1785422511.35 total 20.01 last 120.67) - /unlink 3231 (time 1785422521.35 total 30.02 last 94.93) - /unlink 4374 (time 1785422531.36 total 40.02 last 114.25) total: 4500 /unlink in 41.47 seconds: 108.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 554 (time 1785422548.65 total 10.02 last 55.29) - open/close 1164 (time 1785422558.83 total 20.20 last 59.92) - open/close 1745 (time 1785422568.84 total 30.21 last 58.03) - open/close 2458 (time 1785422578.86 total 40.22 last 71.21) - open/close 3042 (time 1785422588.87 total 50.23 last 58.35) - open/close 3656 (time 1785422598.87 total 60.23 last 61.38) - open/close 4215 (time 1785422608.88 total 70.24 last 55.85) total: 4500 open/close in 73.89 seconds: 60.90 ops/second - /unlink 1191 (time 1785422627.57 total 10.01 last 119.01) - /unlink 2097 (time 1785422637.57 total 20.01 last 90.60) - /unlink 3326 (time 1785422647.58 total 30.01 last 122.85) - /unlink 4272 (time 1785422657.58 total 40.02 last 94.53) total: 4500 /unlink in 43.26 seconds: 104.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 633 (time 1785422676.60 total 10.01 last 63.23) - open/close 1239 (time 1785422686.62 total 20.03 last 60.50) - open/close 1860 (time 1785422696.62 total 30.03 last 62.08) - open/close 2550 (time 1785422706.62 total 40.03 last 68.99) - open/close 3054 (time 1785422716.63 total 50.04 last 50.36) - open/close 3557 (time 1785422726.65 total 60.05 last 50.23) - open/close 4093 (time 1785422736.65 total 70.06 last 53.58) total: 4500 open/close in 75.45 seconds: 59.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 713 (time 1785422757.64 total 10.00 last 71.29) - /unlink 1911 (time 1785422767.64 total 20.01 last 119.71) - /unlink 2762 (time 1785422777.65 total 30.01 last 85.07) - /unlink 3594 (time 1785422787.65 total 40.01 last 83.18) - /unlink 4358 (time 1785422797.66 total 50.02 last 76.36) total: 4500 /unlink in 52.19 seconds: 86.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 649 (time 1785422816.07 total 10.00 last 64.90) - open/close 1366 (time 1785422826.07 total 20.01 last 71.67) - open/close 2063 (time 1785422836.08 total 30.01 last 69.68) - open/close 2738 (time 1785422846.09 total 40.02 last 67.44) - open/close 3382 (time 1785422856.09 total 50.02 last 64.35) - open/close 4241 (time 1785422866.12 total 60.05 last 85.66) total: 4500 open/close in 63.10 seconds: 71.32 ops/second - /unlink 1154 (time 1785422884.76 total 10.00 last 115.40) - /unlink 2398 (time 1785422894.76 total 20.00 last 124.36) - /unlink 3377 (time 1785422904.76 total 30.00 last 97.89) - /unlink 4387 (time 1785422914.77 total 40.01 last 100.96) total: 4500 /unlink in 40.88 seconds: 110.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 774 (time 1785422931.55 total 10.01 last 77.32) - open/close 1629 (time 1785422941.56 total 20.01 last 85.48) - open/close 2350 (time 1785422951.57 total 30.03 last 71.99) - open/close 2953 (time 1785422961.57 total 40.03 last 60.29) - open/close 3623 (time 1785422971.59 total 50.05 last 66.88) - open/close 4170 (time 1785422981.63 total 60.09 last 54.47) total: 4500 open/close in 64.91 seconds: 69.33 ops/second - /unlink 1106 (time 1785423003.09 total 10.00 last 110.60) - /unlink 2045 (time 1785423013.10 total 20.01 last 93.82) - /unlink 2717 (time 1785423023.11 total 30.02 last 67.14) - /unlink 3378 (time 1785423033.12 total 40.02 last 66.05) - /unlink 4021 (time 1785423043.12 total 50.03 last 64.26) total: 4500 /unlink in 54.21 seconds: 83.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 668 (time 1785423063.54 total 10.01 last 66.75) - open/close 1279 (time 1785423073.55 total 20.02 last 61.03) - open/close 2066 (time 1785423083.57 total 30.04 last 78.54) - open/close 2696 (time 1785423093.58 total 40.05 last 62.95) - open/close 3377 (time 1785423103.60 total 50.06 last 67.98) - open/close 3891 (time 1785423113.60 total 60.07 last 51.39) - open/close 4411 (time 1785423123.60 total 70.07 last 51.99) total: 4500 open/close in 71.65 seconds: 62.80 ops/second - /unlink 1200 (time 1785423140.66 total 10.00 last 119.96) - /unlink 2382 (time 1785423150.67 total 20.01 last 118.12) - /unlink 3526 (time 1785423160.67 total 30.01 last 114.36) total: 4500 /unlink in 39.67 seconds: 113.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 578 (time 1785423185.70 total 10.01 last 57.75) - open/close 1265 (time 1785423195.70 total 20.01 last 68.68) - open/close 1957 (time 1785423205.70 total 30.01 last 69.19) - open/close 2512 (time 1785423215.74 total 40.05 last 55.32) - open/close 3317 (time 1785423225.75 total 50.06 last 80.38) - open/close 4188 (time 1785423235.76 total 60.07 last 86.99) total: 4500 open/close in 64.66 seconds: 69.60 ops/second - /unlink 1305 (time 1785423255.37 total 10.01 last 130.32) - /unlink 2499 (time 1785423265.37 total 20.02 last 119.36) - /unlink 3903 (time 1785423275.38 total 30.02 last 140.36) total: 4500 /unlink in 35.68 seconds: 126.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 665 (time 1785423296.21 total 10.02 last 66.37) - open/close 1442 (time 1785423306.21 total 20.02 last 77.68) - open/close 2216 (time 1785423316.22 total 30.03 last 77.37) - open/close 2942 (time 1785423326.23 total 40.04 last 72.49) - open/close 3796 (time 1785423336.25 total 50.05 last 85.29) - open/close 4367 (time 1785423346.25 total 60.06 last 57.05) total: 4500 open/close in 61.68 seconds: 72.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1147 (time 1785423363.42 total 10.02 last 114.46) - /unlink 2325 (time 1785423373.42 total 20.03 last 117.74) - /unlink 3208 (time 1785423383.43 total 30.03 last 88.25) - /unlink 4226 (time 1785423393.44 total 40.05 last 101.64) total: 4500 /unlink in 43.09 seconds: 104.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 547 (time 1785423411.86 total 10.02 last 54.58) - open/close 1024 (time 1785423421.86 total 20.02 last 47.68) - open/close 1708 (time 1785423431.87 total 30.03 last 68.36) - open/close 2343 (time 1785423441.87 total 40.04 last 63.46) - open/close 2865 (time 1785423451.88 total 50.04 last 52.18) - open/close 3316 (time 1785423461.92 total 60.08 last 44.91) - open/close 3848 (time 1785423471.92 total 70.09 last 53.18) total: 4500 open/close in 79.56 seconds: 56.56 ops/second - /unlink 1386 (time 1785423496.54 total 10.01 last 138.48) - /unlink 2676 (time 1785423506.54 total 20.01 last 128.99) - /unlink 4036 (time 1785423516.54 total 30.01 last 135.97) total: 4500 /unlink in 34.22 seconds: 131.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 582 (time 1785423536.58 total 10.02 last 58.07) - open/close 1206 (time 1785423546.60 total 20.04 last 62.28) - open/close 1560 (time 1785423556.60 total 30.05 last 35.37) - open/close 2022 (time 1785423566.61 total 40.06 last 46.17) - open/close 2558 (time 1785423576.62 total 50.06 last 53.57) - open/close 3052 (time 1785423586.63 total 60.08 last 49.34) - open/close 3544 (time 1785423596.65 total 70.10 last 49.10) - open/close 4062 (time 1785423606.68 total 80.12 last 51.66) total: 4500 open/close in 87.12 seconds: 51.65 ops/second - /unlink 1326 (time 1785423628.28 total 10.00 last 132.56) - /unlink 2763 (time 1785423638.28 total 20.01 last 143.66) - /unlink 4286 (time 1785423648.28 total 30.01 last 152.29) total: 4500 /unlink in 32.70 seconds: 137.63 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11297: 95494 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg104-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg104-server unloading modules via unload_modules_local on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing unload_modules_local oleg104-server: modules unloaded.