== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:35:35 (1788453335) Stopping clients: oleg246-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg246-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg246-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg246-server oleg246-server: oleg246-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: kmemleak disabled oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: detected 4 online CPUs by sysfs oleg246-server: MODOPTS_LIBCFS= oleg246-server: Force libcfs to create 2 CPU partitions oleg246-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg246-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg246-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg246-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 1005 (time 1788453536.68 total 10.00 last 100.48) - open/close 2062 (time 1788453546.69 total 20.02 last 105.53) - open/close 2986 (time 1788453556.69 total 30.02 last 92.37) - open/close 3871 (time 1788453566.70 total 40.03 last 88.46) total: 4500 open/close in 47.04 seconds: 95.66 ops/second - /unlink 1572 (time 1788453588.17 total 10.01 last 157.12) - /unlink 2879 (time 1788453598.17 total 20.01 last 130.63) - /unlink 4128 (time 1788453608.18 total 30.01 last 124.87) total: 4500 /unlink in 32.17 seconds: 139.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 810 (time 1788453626.06 total 10.01 last 80.95) - open/close 1697 (time 1788453636.06 total 20.01 last 88.65) - open/close 2536 (time 1788453646.07 total 30.02 last 83.87) - open/close 3290 (time 1788453656.08 total 40.03 last 75.28) - open/close 3946 (time 1788453666.09 total 50.04 last 65.55) total: 4500 open/close in 57.53 seconds: 78.22 ops/second - /unlink 1629 (time 1788453687.49 total 10.00 last 162.89) - /unlink 3240 (time 1788453697.49 total 20.00 last 161.06) total: 4500 /unlink in 27.42 seconds: 164.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 879 (time 1788453719.84 total 10.01 last 87.84) - open/close 1652 (time 1788453729.84 total 20.01 last 77.26) - open/close 2520 (time 1788453739.85 total 30.02 last 86.77) - open/close 3466 (time 1788453749.85 total 40.02 last 94.54) total: 4500 open/close in 49.68 seconds: 90.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1294 (time 1788453774.95 total 10.01 last 129.25) - /unlink 2762 (time 1788453784.96 total 20.02 last 146.72) - /unlink 4447 (time 1788453794.97 total 30.03 last 168.31) total: 4500 /unlink in 30.47 seconds: 147.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 745 (time 1788453810.69 total 10.00 last 74.48) - open/close 1625 (time 1788453820.69 total 20.01 last 87.96) - open/close 2524 (time 1788453830.70 total 30.01 last 89.88) - open/close 3477 (time 1788453840.70 total 40.01 last 95.29) - open/close 4323 (time 1788453850.70 total 50.01 last 84.59) total: 4500 open/close in 51.78 seconds: 86.90 ops/second - /unlink 1835 (time 1788453866.44 total 10.00 last 183.45) - /unlink 3369 (time 1788453876.44 total 20.00 last 153.37) total: 4500 /unlink in 27.80 seconds: 161.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 889 (time 1788453899.08 total 10.00 last 88.87) - open/close 1843 (time 1788453909.09 total 20.01 last 95.35) - open/close 2816 (time 1788453919.10 total 30.02 last 97.22) - open/close 3655 (time 1788453929.11 total 40.03 last 83.80) total: 4500 open/close in 49.73 seconds: 90.48 ops/second - /unlink 1711 (time 1788453952.95 total 10.01 last 170.99) - /unlink 2811 (time 1788453962.95 total 20.01 last 109.96) - /unlink 4268 (time 1788453972.96 total 30.02 last 145.56) total: 4500 /unlink in 31.88 seconds: 141.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 746 (time 1788453990.04 total 10.00 last 74.59) - open/close 1450 (time 1788454000.05 total 20.01 last 70.34) - open/close 2242 (time 1788454010.05 total 30.01 last 79.18) - open/close 2980 (time 1788454020.05 total 40.02 last 73.75) - open/close 3609 (time 1788454030.08 total 50.04 last 62.76) - open/close 4270 (time 1788454040.11 total 60.07 last 65.90) total: 4500 open/close in 62.43 seconds: 72.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1690 (time 1788454056.76 total 10.01 last 168.85) - /unlink 3312 (time 1788454066.76 total 20.02 last 162.08) total: 4500 /unlink in 28.18 seconds: 159.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 777 (time 1788454089.83 total 10.01 last 77.66) - open/close 1514 (time 1788454099.88 total 20.05 last 73.36) - open/close 2400 (time 1788454109.88 total 30.06 last 88.55) - open/close 3199 (time 1788454119.89 total 40.06 last 79.87) - open/close 4011 (time 1788454129.89 total 50.06 last 81.20) total: 4500 open/close in 55.95 seconds: 80.43 ops/second - /unlink 1687 (time 1788454150.08 total 10.00 last 168.67) - /unlink 3185 (time 1788454160.08 total 20.00 last 149.80) total: 4500 /unlink in 27.98 seconds: 160.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 897 (time 1788454183.22 total 10.00 last 89.69) - open/close 1688 (time 1788454193.25 total 20.03 last 78.90) - open/close 2476 (time 1788454203.25 total 30.03 last 78.79) - open/close 3273 (time 1788454213.25 total 40.03 last 79.69) - open/close 4150 (time 1788454223.25 total 50.03 last 87.67) total: 4500 open/close in 54.85 seconds: 82.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1598 (time 1788454241.99 total 10.00 last 159.79) - /unlink 3094 (time 1788454251.99 total 20.00 last 149.58) - /unlink 4366 (time 1788454261.99 total 30.00 last 127.19) total: 4500 /unlink in 31.12 seconds: 144.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 889 (time 1788454278.28 total 10.00 last 88.87) - open/close 1721 (time 1788454288.28 total 20.01 last 83.17) - open/close 2612 (time 1788454298.28 total 30.01 last 89.09) - open/close 3280 (time 1788454308.29 total 40.02 last 66.74) - open/close 4193 (time 1788454318.32 total 50.05 last 91.04) total: 4500 open/close in 53.00 seconds: 84.90 ops/second - /unlink 1947 (time 1788454335.57 total 10.00 last 194.60) - /unlink 3640 (time 1788454345.57 total 20.01 last 169.26) total: 4500 /unlink in 24.89 seconds: 180.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 916 (time 1788454365.01 total 10.00 last 91.56) - open/close 1758 (time 1788454375.02 total 20.02 last 84.10) - open/close 2375 (time 1788454385.03 total 30.02 last 61.69) - open/close 3047 (time 1788454395.03 total 40.02 last 67.15) - open/close 3702 (time 1788454405.06 total 50.05 last 65.35) - open/close 4390 (time 1788454415.06 total 60.05 last 68.75) total: 4500 open/close in 61.71 seconds: 72.92 ops/second - /unlink 1622 (time 1788454431.61 total 10.01 last 161.98) - /unlink 3385 (time 1788454441.61 total 20.02 last 176.24) total: 4500 /unlink in 27.11 seconds: 165.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 841 (time 1788454463.14 total 10.00 last 84.08) - open/close 1673 (time 1788454473.15 total 20.02 last 83.09) - open/close 2552 (time 1788454483.15 total 30.02 last 87.87) - open/close 3379 (time 1788454493.15 total 40.02 last 82.70) - open/close 4118 (time 1788454503.16 total 50.02 last 73.86) total: 4500 open/close in 55.00 seconds: 81.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 971 (time 1788454524.15 total 10.01 last 97.04) - /unlink 2279 (time 1788454534.15 total 20.01 last 130.78) - /unlink 3934 (time 1788454544.15 total 30.01 last 165.42) total: 4500 /unlink in 33.60 seconds: 133.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 935 (time 1788454563.06 total 10.00 last 93.50) - open/close 1786 (time 1788454573.09 total 20.03 last 84.89) - open/close 2666 (time 1788454583.09 total 30.03 last 87.99) - open/close 3620 (time 1788454593.09 total 40.03 last 95.36) total: 4500 open/close in 49.67 seconds: 90.59 ops/second - /unlink 1848 (time 1788454616.99 total 10.01 last 184.68) - /unlink 3372 (time 1788454627.00 total 20.01 last 152.39) total: 4500 /unlink in 27.54 seconds: 163.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 763 (time 1788454650.91 total 10.01 last 76.24) - open/close 1599 (time 1788454660.94 total 20.04 last 83.37) - open/close 2491 (time 1788454670.95 total 30.04 last 89.14) - open/close 3090 (time 1788454680.96 total 40.06 last 59.82) - open/close 3716 (time 1788454690.96 total 50.06 last 62.58) - open/close 4462 (time 1788454700.98 total 60.08 last 74.45) total: 4500 open/close in 60.57 seconds: 74.29 ops/second - /unlink 1755 (time 1788454716.01 total 10.00 last 175.42) - /unlink 3522 (time 1788454726.01 total 20.01 last 176.69) total: 4500 /unlink in 28.02 seconds: 160.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 776 (time 1788454750.80 total 10.01 last 77.55) - open/close 1606 (time 1788454760.80 total 20.01 last 82.97) - open/close 2184 (time 1788454770.82 total 30.03 last 57.69) - open/close 2910 (time 1788454780.82 total 40.03 last 72.60) - open/close 3668 (time 1788454790.84 total 50.05 last 75.64) - open/close 4434 (time 1788454800.84 total 60.05 last 76.59) total: 4500 open/close in 60.88 seconds: 73.92 ops/second - /unlink 1564 (time 1788454816.73 total 10.01 last 156.31) - /unlink 3346 (time 1788454826.73 total 20.01 last 178.13) total: 4500 /unlink in 28.30 seconds: 159.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 921 (time 1788454850.50 total 10.01 last 91.97) - open/close 1659 (time 1788454860.50 total 20.02 last 73.77) - open/close 2315 (time 1788454870.52 total 30.04 last 65.46) - open/close 3156 (time 1788454880.53 total 40.05 last 84.02) - open/close 3886 (time 1788454890.54 total 50.06 last 72.95) total: 4500 open/close in 58.57 seconds: 76.83 ops/second - /unlink 1539 (time 1788454914.34 total 10.00 last 153.86) - /unlink 2771 (time 1788454924.36 total 20.02 last 123.02) - /unlink 4036 (time 1788454934.36 total 30.03 last 126.40) total: 4500 /unlink in 33.29 seconds: 135.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 758 (time 1788454953.42 total 10.01 last 75.74) - open/close 1504 (time 1788454963.43 total 20.02 last 74.53) - open/close 2236 (time 1788454973.44 total 30.02 last 73.14) - open/close 2922 (time 1788454983.45 total 40.04 last 68.52) - open/close 3695 (time 1788454993.46 total 50.04 last 77.25) total: 4500 open/close in 59.12 seconds: 76.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1034 (time 1788455017.35 total 10.00 last 103.35) - /unlink 2275 (time 1788455027.35 total 20.01 last 124.02) - /unlink 3587 (time 1788455037.36 total 30.02 last 131.07) total: 4500 /unlink in 36.56 seconds: 123.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 571 (time 1788455059.70 total 10.02 last 57.00) - open/close 1099 (time 1788455069.74 total 20.05 last 52.62) - open/close 1734 (time 1788455079.77 total 30.09 last 63.26) - open/close 2467 (time 1788455089.78 total 40.10 last 73.24) - open/close 3023 (time 1788455099.78 total 50.10 last 55.59) - open/close 3584 (time 1788455109.79 total 60.11 last 56.04) - open/close 4314 (time 1788455119.81 total 70.12 last 72.91) total: 4500 open/close in 72.67 seconds: 61.93 ops/second - /unlink 1589 (time 1788455136.42 total 10.00 last 158.84) - /unlink 2929 (time 1788455146.42 total 20.00 last 133.99) - /unlink 4035 (time 1788455156.43 total 30.01 last 110.53) total: 4500 /unlink in 33.25 seconds: 135.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 703 (time 1788455175.78 total 10.02 last 70.16) - open/close 1170 (time 1788455185.79 total 20.03 last 46.64) - open/close 1600 (time 1788455195.80 total 30.04 last 42.96) - open/close 2305 (time 1788455205.81 total 40.05 last 70.42) - open/close 3119 (time 1788455215.82 total 50.06 last 81.36) - open/close 3965 (time 1788455225.82 total 60.06 last 84.57) total: 4500 open/close in 66.69 seconds: 67.47 ops/second - /unlink 1469 (time 1788455247.82 total 10.00 last 146.87) - /unlink 3172 (time 1788455257.83 total 20.01 last 170.20) total: 4500 /unlink in 29.04 seconds: 154.97 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95182 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg246-server unloading modules via unload_modules_local on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing unload_modules_local oleg246-server: kmemleak disabled modules unloaded.