== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:28:27 (1786966107) Stopping clients: oleg146-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg146-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg146-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg146-server oleg146-server: oleg146-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing load_modules_local oleg146-server: Loading modules from /home/green/git/lustre-release/lustre oleg146-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg146-server: detected 4 online CPUs by sysfs oleg146-server: MODOPTS_LIBCFS= oleg146-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg146-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg146-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg146-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 725 (time 1786966340.68 total 10.02 last 72.34) - open/close 1434 (time 1786966350.68 total 20.02 last 70.88) - open/close 2130 (time 1786966360.69 total 30.03 last 69.54) - open/close 2770 (time 1786966370.71 total 40.05 last 63.90) - open/close 3591 (time 1786966380.71 total 50.06 last 82.04) - open/close 4233 (time 1786966390.72 total 60.06 last 64.16) total: 4500 open/close in 64.51 seconds: 69.76 ops/second - /unlink 1256 (time 1786966410.89 total 10.01 last 125.52) - /unlink 2328 (time 1786966420.89 total 20.01 last 107.17) - /unlink 3465 (time 1786966430.89 total 30.01 last 113.70) total: 4500 /unlink in 37.26 seconds: 120.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 692 (time 1786966454.14 total 10.01 last 69.16) - open/close 1373 (time 1786966464.15 total 20.02 last 68.04) - open/close 2065 (time 1786966474.16 total 30.02 last 69.14) - open/close 2750 (time 1786966484.17 total 40.03 last 68.46) - open/close 3450 (time 1786966494.18 total 50.04 last 69.94) - open/close 4174 (time 1786966504.19 total 60.05 last 72.33) total: 4500 open/close in 65.86 seconds: 68.32 ops/second - /unlink 1261 (time 1786966525.62 total 10.00 last 126.04) - /unlink 2428 (time 1786966535.63 total 20.01 last 116.64) - /unlink 3866 (time 1786966545.63 total 30.01 last 143.79) total: 4500 /unlink in 33.99 seconds: 132.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 619 (time 1786966565.14 total 10.01 last 61.83) - open/close 1420 (time 1786966575.15 total 20.01 last 80.08) - open/close 2100 (time 1786966585.16 total 30.03 last 67.91) - open/close 2755 (time 1786966595.17 total 40.04 last 65.41) - open/close 3427 (time 1786966605.18 total 50.05 last 67.15) - open/close 4131 (time 1786966615.20 total 60.07 last 70.26) total: 4500 open/close in 65.12 seconds: 69.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1483 (time 1786966634.98 total 10.03 last 147.91) - /unlink 2610 (time 1786966644.98 total 20.03 last 112.63) - /unlink 4055 (time 1786966654.99 total 30.03 last 144.48) total: 4500 /unlink in 33.84 seconds: 132.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 677 (time 1786966674.49 total 10.01 last 67.61) - open/close 1343 (time 1786966684.50 total 20.03 last 66.52) - open/close 2032 (time 1786966694.50 total 30.03 last 68.90) - open/close 2797 (time 1786966704.50 total 40.03 last 76.47) - open/close 3576 (time 1786966714.51 total 50.04 last 77.84) - open/close 4307 (time 1786966724.52 total 60.05 last 73.01) total: 4500 open/close in 62.34 seconds: 72.19 ops/second - /unlink 1158 (time 1786966742.17 total 10.00 last 115.78) - /unlink 2479 (time 1786966752.17 total 20.00 last 132.08) - /unlink 3721 (time 1786966762.17 total 30.01 last 124.15) total: 4500 /unlink in 36.25 seconds: 124.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 612 (time 1786966783.52 total 10.03 last 61.03) - open/close 1100 (time 1786966793.56 total 20.07 last 48.61) - open/close 1685 (time 1786966803.56 total 30.07 last 58.49) - open/close 2402 (time 1786966813.56 total 40.08 last 71.65) - open/close 2945 (time 1786966823.59 total 50.10 last 54.17) - open/close 3575 (time 1786966833.60 total 60.11 last 62.95) - open/close 4358 (time 1786966843.62 total 70.14 last 78.09) total: 4500 open/close in 73.27 seconds: 61.42 ops/second - /unlink 836 (time 1786966862.73 total 10.01 last 83.49) - /unlink 1893 (time 1786966872.73 total 20.01 last 105.69) - /unlink 2965 (time 1786966882.73 total 30.02 last 107.18) - /unlink 4134 (time 1786966892.74 total 40.03 last 116.79) total: 4500 /unlink in 43.76 seconds: 102.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 442 (time 1786966913.38 total 10.01 last 44.17) - open/close 923 (time 1786966923.40 total 20.02 last 48.02) - open/close 1514 (time 1786966933.42 total 30.05 last 58.97) - open/close 2104 (time 1786966943.43 total 40.05 last 58.95) - open/close 2795 (time 1786966953.44 total 50.06 last 69.04) - open/close 3434 (time 1786966963.46 total 60.09 last 63.75) - open/close 4071 (time 1786966973.47 total 70.10 last 63.63) total: 4500 open/close in 76.81 seconds: 58.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1280 (time 1786966996.04 total 10.00 last 127.94) - /unlink 2404 (time 1786967006.04 total 20.01 last 112.36) - /unlink 3651 (time 1786967016.05 total 30.02 last 124.59) total: 4500 /unlink in 39.35 seconds: 114.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 471 (time 1786967042.01 total 10.01 last 47.04) - open/close 1242 (time 1786967052.03 total 20.04 last 76.91) - open/close 1984 (time 1786967062.04 total 30.05 last 74.11) - open/close 2806 (time 1786967072.05 total 40.05 last 82.18) - open/close 3599 (time 1786967082.05 total 50.05 last 79.28) - open/close 4429 (time 1786967092.05 total 60.05 last 83.00) total: 4500 open/close in 60.90 seconds: 73.89 ops/second - /unlink 1463 (time 1786967107.27 total 10.00 last 146.29) - /unlink 2706 (time 1786967117.27 total 20.00 last 124.27) - /unlink 4050 (time 1786967127.27 total 30.01 last 134.34) total: 4500 /unlink in 33.11 seconds: 135.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 730 (time 1786967145.78 total 10.01 last 72.92) - open/close 1432 (time 1786967155.79 total 20.02 last 70.12) - open/close 2155 (time 1786967165.79 total 30.02 last 72.29) - open/close 2956 (time 1786967175.81 total 40.05 last 79.92) - open/close 3663 (time 1786967185.81 total 50.05 last 70.69) - open/close 4407 (time 1786967195.83 total 60.06 last 74.30) total: 4500 open/close in 61.42 seconds: 73.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1354 (time 1786967212.02 total 10.00 last 135.38) - /unlink 2706 (time 1786967222.02 total 20.00 last 135.18) - /unlink 4250 (time 1786967232.03 total 30.01 last 154.33) total: 4500 /unlink in 31.79 seconds: 141.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 737 (time 1786967248.86 total 10.01 last 73.65) - open/close 1453 (time 1786967258.87 total 20.02 last 71.51) - open/close 2096 (time 1786967268.89 total 30.04 last 64.16) - open/close 2771 (time 1786967278.89 total 40.04 last 67.50) - open/close 3513 (time 1786967288.91 total 50.06 last 74.10) - open/close 4351 (time 1786967298.93 total 60.08 last 83.59) total: 4500 open/close in 61.80 seconds: 72.82 ops/second - /unlink 1315 (time 1786967315.02 total 10.02 last 131.28) - /unlink 2328 (time 1786967325.02 total 20.02 last 101.26) - /unlink 3405 (time 1786967335.03 total 30.03 last 107.65) total: 4500 /unlink in 39.46 seconds: 114.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 540 (time 1786967360.57 total 10.02 last 53.90) - open/close 1251 (time 1786967370.58 total 20.03 last 71.03) - open/close 1976 (time 1786967380.58 total 30.03 last 72.48) - open/close 2668 (time 1786967390.59 total 40.04 last 69.15) - open/close 3305 (time 1786967400.60 total 50.05 last 63.63) - open/close 3884 (time 1786967410.61 total 60.06 last 57.84) total: 4500 open/close in 68.83 seconds: 65.38 ops/second - /unlink 1154 (time 1786967434.44 total 10.01 last 115.32) - /unlink 2318 (time 1786967444.44 total 20.01 last 116.36) - /unlink 3483 (time 1786967454.45 total 30.01 last 116.48) total: 4500 /unlink in 38.73 seconds: 116.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 737 (time 1786967479.17 total 10.01 last 73.60) - open/close 1365 (time 1786967489.21 total 20.05 last 62.60) - open/close 2132 (time 1786967499.22 total 30.06 last 76.62) - open/close 2929 (time 1786967509.23 total 40.06 last 79.63) - open/close 3576 (time 1786967519.24 total 50.08 last 64.62) - open/close 4281 (time 1786967529.25 total 60.09 last 70.39) total: 4500 open/close in 63.15 seconds: 71.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1205 (time 1786967546.53 total 10.00 last 120.48) - /unlink 2293 (time 1786967556.53 total 20.01 last 108.75) - /unlink 3338 (time 1786967566.53 total 30.01 last 104.48) total: 4500 /unlink in 39.87 seconds: 112.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 640 (time 1786967591.52 total 10.01 last 63.93) - open/close 1266 (time 1786967601.53 total 20.02 last 62.55) - open/close 1945 (time 1786967611.54 total 30.03 last 67.84) - open/close 2542 (time 1786967621.55 total 40.04 last 59.64) - open/close 3185 (time 1786967631.56 total 50.05 last 64.24) - open/close 3899 (time 1786967641.57 total 60.06 last 71.31) total: 4500 open/close in 68.12 seconds: 66.06 ops/second - /unlink 1174 (time 1786967665.09 total 10.00 last 117.35) - /unlink 2545 (time 1786967675.09 total 20.01 last 137.09) - /unlink 3771 (time 1786967685.10 total 30.01 last 122.55) total: 4500 /unlink in 35.88 seconds: 125.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 722 (time 1786967706.19 total 10.01 last 72.13) - open/close 1359 (time 1786967716.19 total 20.01 last 63.68) - open/close 1998 (time 1786967726.20 total 30.02 last 63.86) - open/close 2780 (time 1786967736.20 total 40.02 last 78.18) - open/close 3473 (time 1786967746.21 total 50.03 last 69.23) - open/close 4156 (time 1786967756.22 total 60.04 last 68.25) total: 4500 open/close in 65.88 seconds: 68.31 ops/second - /unlink 1388 (time 1786967776.62 total 10.01 last 138.71) - /unlink 2581 (time 1786967786.63 total 20.01 last 119.22) - /unlink 3964 (time 1786967796.63 total 30.01 last 138.28) total: 4500 /unlink in 34.33 seconds: 131.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 842 (time 1786967815.30 total 10.00 last 84.17) - open/close 1712 (time 1786967825.31 total 20.01 last 86.92) - open/close 2482 (time 1786967835.32 total 30.02 last 76.94) - open/close 3157 (time 1786967845.33 total 40.03 last 67.45) - open/close 3862 (time 1786967855.33 total 50.03 last 70.48) total: 4500 open/close in 58.41 seconds: 77.04 ops/second - /unlink 1279 (time 1786967878.20 total 10.01 last 127.83) - /unlink 2718 (time 1786967888.20 total 20.01 last 143.84) - /unlink 4044 (time 1786967898.20 total 30.01 last 132.55) total: 4500 /unlink in 33.76 seconds: 133.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 629 (time 1786967916.93 total 10.01 last 62.84) - open/close 1354 (time 1786967926.93 total 20.01 last 72.49) - open/close 2065 (time 1786967936.93 total 30.01 last 71.08) - open/close 2834 (time 1786967946.94 total 40.02 last 76.84) - open/close 3533 (time 1786967956.94 total 50.02 last 69.90) - open/close 4301 (time 1786967966.96 total 60.04 last 76.65) total: 4500 open/close in 63.34 seconds: 71.04 ops/second - /unlink 1332 (time 1786967985.04 total 10.00 last 133.20) - /unlink 2531 (time 1786967995.04 total 20.00 last 119.90) - /unlink 3693 (time 1786968005.04 total 30.01 last 116.14) total: 4500 /unlink in 36.94 seconds: 121.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 732 (time 1786968027.32 total 10.00 last 73.18) - open/close 1395 (time 1786968037.33 total 20.01 last 66.24) - open/close 2123 (time 1786968047.34 total 30.03 last 72.69) - open/close 2940 (time 1786968057.37 total 40.05 last 81.50) - open/close 3688 (time 1786968067.37 total 50.06 last 74.77) total: 4500 open/close in 59.83 seconds: 75.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1485 (time 1786968092.17 total 10.01 last 148.38) - /unlink 2772 (time 1786968102.17 total 20.01 last 128.66) - /unlink 4343 (time 1786968112.17 total 30.01 last 157.05) total: 4500 /unlink in 31.23 seconds: 144.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 741 (time 1786968129.03 total 10.02 last 73.93) - open/close 1523 (time 1786968139.04 total 20.03 last 78.17) - open/close 2319 (time 1786968149.04 total 30.03 last 79.58) - open/close 3089 (time 1786968159.04 total 40.03 last 76.96) - open/close 3863 (time 1786968169.06 total 50.05 last 77.27) total: 4500 open/close in 57.68 seconds: 78.02 ops/second - /unlink 1207 (time 1786968191.45 total 10.00 last 120.69) - /unlink 2548 (time 1786968201.46 total 20.01 last 133.98) - /unlink 3942 (time 1786968211.47 total 30.02 last 139.33) total: 4500 /unlink in 33.58 seconds: 134.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 802 (time 1786968230.82 total 10.00 last 80.20) - open/close 1416 (time 1786968240.83 total 20.01 last 61.33) - open/close 1924 (time 1786968250.83 total 30.02 last 50.77) - open/close 2558 (time 1786968260.84 total 40.02 last 63.39) - open/close 3156 (time 1786968270.84 total 50.02 last 59.77) - open/close 3828 (time 1786968280.86 total 60.04 last 67.09) total: 4500 open/close in 69.77 seconds: 64.50 ops/second - /unlink 1183 (time 1786968306.20 total 10.00 last 118.25) - /unlink 2381 (time 1786968316.20 total 20.01 last 119.78) - /unlink 3811 (time 1786968326.20 total 30.01 last 142.94) total: 4500 /unlink in 36.72 seconds: 122.56 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95279 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg146-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg146-server unloading modules via unload_modules_local on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing unload_modules_local oleg146-server: modules unloaded.