== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:04:53 (1788581093) Stopping clients: oleg252-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg252-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg252-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg252-server oleg252-server: oleg252-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing load_modules_local oleg252-server: kmemleak disabled oleg252-server: Loading modules from /home/green/git/lustre-release/lustre oleg252-server: detected 4 online CPUs by sysfs oleg252-server: MODOPTS_LIBCFS= oleg252-server: Force libcfs to create 2 CPU partitions oleg252-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg252-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg252-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg252-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 734 (time 1788581309.24 total 10.00 last 73.39) - open/close 1539 (time 1788581319.25 total 20.01 last 80.44) - open/close 2403 (time 1788581329.26 total 30.02 last 86.31) - open/close 3239 (time 1788581339.27 total 40.03 last 83.55) - open/close 4050 (time 1788581349.27 total 50.03 last 81.09) total: 4500 open/close in 56.29 seconds: 79.94 ops/second - /unlink 1471 (time 1788581370.07 total 10.00 last 147.06) - /unlink 3011 (time 1788581380.07 total 20.01 last 153.94) - /unlink 4296 (time 1788581390.07 total 30.01 last 128.48) total: 4500 /unlink in 31.89 seconds: 141.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 613 (time 1788581407.46 total 10.00 last 61.27) - open/close 1384 (time 1788581417.46 total 20.01 last 77.08) - open/close 2238 (time 1788581427.46 total 30.01 last 85.39) - open/close 3044 (time 1788581437.47 total 40.01 last 80.56) - open/close 3933 (time 1788581447.47 total 50.02 last 88.83) total: 4500 open/close in 56.75 seconds: 79.30 ops/second - /unlink 1425 (time 1788581468.98 total 10.01 last 142.41) - /unlink 2966 (time 1788581478.98 total 20.01 last 154.00) - /unlink 4372 (time 1788581488.99 total 30.01 last 140.57) total: 4500 /unlink in 30.98 seconds: 145.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 772 (time 1788581504.74 total 10.01 last 77.09) - open/close 1474 (time 1788581514.76 total 20.03 last 70.08) - open/close 2156 (time 1788581524.76 total 30.03 last 68.17) - open/close 2883 (time 1788581534.76 total 40.04 last 72.69) - open/close 3592 (time 1788581544.77 total 50.04 last 70.83) - open/close 4396 (time 1788581554.79 total 60.06 last 80.28) total: 4500 open/close in 61.46 seconds: 73.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1523 (time 1788581570.64 total 10.01 last 152.20) - /unlink 2781 (time 1788581580.64 total 20.01 last 125.80) - /unlink 4208 (time 1788581590.64 total 30.01 last 142.69) total: 4500 /unlink in 31.59 seconds: 142.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 780 (time 1788581606.60 total 10.00 last 77.96) - open/close 1597 (time 1788581616.61 total 20.01 last 81.66) - open/close 2253 (time 1788581626.62 total 30.02 last 65.55) - open/close 2998 (time 1788581636.63 total 40.03 last 74.39) - open/close 3908 (time 1788581646.64 total 50.04 last 90.95) total: 4500 open/close in 57.67 seconds: 78.03 ops/second - /unlink 1364 (time 1788581669.49 total 10.00 last 136.38) - /unlink 2912 (time 1788581679.49 total 20.00 last 154.78) - /unlink 4439 (time 1788581689.50 total 30.00 last 152.69) total: 4500 /unlink in 30.64 seconds: 146.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 816 (time 1788581704.80 total 10.01 last 81.52) - open/close 1726 (time 1788581714.80 total 20.01 last 90.98) - open/close 2560 (time 1788581724.80 total 30.01 last 83.39) - open/close 3348 (time 1788581734.81 total 40.02 last 78.73) - open/close 4121 (time 1788581744.82 total 50.03 last 77.25) total: 4500 open/close in 55.32 seconds: 81.35 ops/second - /unlink 1493 (time 1788581764.93 total 10.02 last 149.07) - /unlink 2957 (time 1788581774.93 total 20.02 last 146.33) - /unlink 4403 (time 1788581784.94 total 30.03 last 144.47) total: 4500 /unlink in 31.35 seconds: 143.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 865 (time 1788581800.99 total 10.01 last 86.39) - open/close 1486 (time 1788581811.00 total 20.02 last 62.05) - open/close 2195 (time 1788581821.02 total 30.04 last 70.75) - open/close 2915 (time 1788581831.02 total 40.05 last 71.96) - open/close 3624 (time 1788581841.03 total 50.05 last 70.86) - open/close 4350 (time 1788581851.04 total 60.07 last 72.48) total: 4500 open/close in 62.67 seconds: 71.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1334 (time 1788581869.17 total 10.00 last 133.37) - /unlink 2963 (time 1788581879.17 total 20.01 last 162.85) total: 4500 /unlink in 29.80 seconds: 151.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 653 (time 1788581904.33 total 10.01 last 65.24) - open/close 1407 (time 1788581914.33 total 20.01 last 75.39) - open/close 2187 (time 1788581924.33 total 30.01 last 77.96) - open/close 2881 (time 1788581934.34 total 40.03 last 69.32) - open/close 3657 (time 1788581944.35 total 50.03 last 77.55) - open/close 4464 (time 1788581954.37 total 60.06 last 80.51) total: 4500 open/close in 60.37 seconds: 74.54 ops/second - /unlink 1599 (time 1788581970.54 total 10.02 last 159.65) - /unlink 3326 (time 1788581980.54 total 20.02 last 172.69) total: 4500 /unlink in 27.07 seconds: 166.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 821 (time 1788582003.26 total 10.01 last 82.04) - open/close 1536 (time 1788582013.27 total 20.01 last 71.46) - open/close 2360 (time 1788582023.28 total 30.02 last 82.34) - open/close 3116 (time 1788582033.29 total 40.03 last 75.52) - open/close 3860 (time 1788582043.30 total 50.04 last 74.31) total: 4500 open/close in 58.44 seconds: 77.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1564 (time 1788582066.59 total 10.01 last 156.32) - /unlink 3097 (time 1788582076.59 total 20.01 last 153.26) total: 4500 /unlink in 28.16 seconds: 159.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 868 (time 1788582099.98 total 10.01 last 86.75) - open/close 1712 (time 1788582109.99 total 20.01 last 84.33) - open/close 2449 (time 1788582120.01 total 30.03 last 73.54) - open/close 3216 (time 1788582130.01 total 40.04 last 76.70) - open/close 4088 (time 1788582140.02 total 50.04 last 87.13) total: 4500 open/close in 54.24 seconds: 82.97 ops/second - /unlink 1538 (time 1788582158.40 total 10.00 last 153.72) - /unlink 3060 (time 1788582168.40 total 20.01 last 152.12) total: 4500 /unlink in 29.26 seconds: 153.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 799 (time 1788582192.45 total 10.00 last 79.88) - open/close 1534 (time 1788582202.46 total 20.01 last 73.43) - open/close 2282 (time 1788582212.47 total 30.02 last 74.75) - open/close 3125 (time 1788582222.47 total 40.03 last 84.23) - open/close 4060 (time 1788582232.48 total 50.03 last 93.44) total: 4500 open/close in 54.76 seconds: 82.18 ops/second - /unlink 1736 (time 1788582251.73 total 10.00 last 173.60) - /unlink 3534 (time 1788582261.73 total 20.00 last 179.74) total: 4500 /unlink in 25.76 seconds: 174.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 919 (time 1788582281.45 total 10.00 last 91.88) - open/close 1582 (time 1788582291.45 total 20.00 last 66.29) - open/close 2239 (time 1788582301.45 total 30.01 last 65.67) - open/close 3097 (time 1788582311.46 total 40.02 last 85.73) - open/close 3870 (time 1788582321.46 total 50.02 last 77.28) total: 4500 open/close in 57.50 seconds: 78.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1613 (time 1788582343.54 total 10.01 last 161.14) - /unlink 3234 (time 1788582353.55 total 20.01 last 162.02) total: 4500 /unlink in 27.84 seconds: 161.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 983 (time 1788582375.06 total 10.00 last 98.27) - open/close 1895 (time 1788582385.08 total 20.02 last 91.05) - open/close 2757 (time 1788582395.08 total 30.02 last 86.16) - open/close 3552 (time 1788582405.08 total 40.03 last 79.48) - open/close 4443 (time 1788582415.09 total 50.03 last 89.05) total: 4500 open/close in 50.58 seconds: 88.96 ops/second - /unlink 1855 (time 1788582429.59 total 10.00 last 185.49) - /unlink 3320 (time 1788582439.59 total 20.00 last 146.50) total: 4500 /unlink in 27.88 seconds: 161.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 866 (time 1788582461.83 total 10.00 last 86.58) - open/close 1715 (time 1788582471.83 total 20.00 last 84.89) - open/close 2498 (time 1788582481.84 total 30.01 last 78.26) - open/close 3102 (time 1788582491.84 total 40.01 last 60.37) - open/close 3939 (time 1788582501.85 total 50.02 last 83.63) total: 4500 open/close in 56.56 seconds: 79.55 ops/second - /unlink 1631 (time 1788582522.74 total 10.00 last 163.04) - /unlink 3187 (time 1788582532.74 total 20.01 last 155.56) total: 4500 /unlink in 27.81 seconds: 161.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 774 (time 1788582555.45 total 10.01 last 77.35) - open/close 1651 (time 1788582565.45 total 20.01 last 87.68) - open/close 2619 (time 1788582575.46 total 30.02 last 96.70) - open/close 3464 (time 1788582585.47 total 40.03 last 84.44) - open/close 4264 (time 1788582595.47 total 50.03 last 80.00) total: 4500 open/close in 53.06 seconds: 84.80 ops/second - /unlink 1872 (time 1788582612.78 total 10.00 last 187.18) - /unlink 3444 (time 1788582622.78 total 20.01 last 157.12) total: 4500 /unlink in 26.46 seconds: 170.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 783 (time 1788582644.18 total 10.00 last 78.27) - open/close 1597 (time 1788582654.19 total 20.01 last 81.36) - open/close 2498 (time 1788582664.19 total 30.02 last 90.03) - open/close 3377 (time 1788582674.21 total 40.03 last 87.77) - open/close 4161 (time 1788582684.22 total 50.04 last 78.32) total: 4500 open/close in 54.75 seconds: 82.19 ops/second - /unlink 1666 (time 1788582703.60 total 10.00 last 166.56) - /unlink 3131 (time 1788582713.60 total 20.00 last 146.48) total: 4500 /unlink in 27.33 seconds: 164.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 765 (time 1788582734.64 total 10.02 last 76.31) - open/close 1255 (time 1788582744.65 total 20.03 last 48.99) - open/close 1951 (time 1788582754.65 total 30.03 last 69.60) - open/close 2670 (time 1788582764.65 total 40.03 last 71.89) - open/close 3504 (time 1788582774.65 total 50.03 last 83.39) - open/close 4272 (time 1788582784.65 total 60.03 last 76.78) total: 4500 open/close in 62.30 seconds: 72.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1564 (time 1788582801.81 total 10.00 last 156.40) - /unlink 3199 (time 1788582811.82 total 20.00 last 163.45) total: 4500 /unlink in 28.12 seconds: 160.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 926 (time 1788582835.30 total 10.03 last 92.33) - open/close 1692 (time 1788582845.31 total 20.03 last 76.56) - open/close 2476 (time 1788582855.32 total 30.04 last 78.34) - open/close 3232 (time 1788582865.32 total 40.04 last 75.59) - open/close 3953 (time 1788582875.32 total 50.04 last 72.10) total: 4500 open/close in 58.07 seconds: 77.49 ops/second - /unlink 1670 (time 1788582898.11 total 10.01 last 166.90) - /unlink 3106 (time 1788582908.12 total 20.01 last 143.48) total: 4500 /unlink in 29.70 seconds: 151.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 812 (time 1788582932.34 total 10.03 last 80.98) - open/close 1564 (time 1788582942.34 total 20.03 last 75.18) - open/close 2454 (time 1788582952.34 total 30.03 last 88.96) - open/close 3272 (time 1788582962.35 total 40.04 last 81.74) - open/close 4179 (time 1788582972.36 total 50.05 last 90.65) total: 4500 open/close in 52.70 seconds: 85.38 ops/second - /unlink 1724 (time 1788582988.61 total 10.01 last 172.23) - /unlink 3059 (time 1788582998.62 total 20.02 last 133.39) - /unlink 4497 (time 1788583008.63 total 30.03 last 143.67) total: 4500 /unlink in 30.04 seconds: 149.80 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95147 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg252-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg252-server unloading modules via unload_modules_local on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing unload_modules_local oleg252-server: kmemleak disabled modules unloaded.