== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:11:18 (1790950278) Stopping clients: oleg636-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg636-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg636-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg636-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg636-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg636-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg636-server oleg636-server: oleg636-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg636-server' oleg636-server: oleg636-server.virtnet: executing load_modules_local oleg636-server: kmemleak disabled oleg636-server: Loading modules from /home/green/git/lustre-release/lustre oleg636-server: detected 4 online CPUs by sysfs oleg636-server: MODOPTS_LIBCFS= oleg636-server: Force libcfs to create 2 CPU partitions oleg636-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg636-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg636-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg636-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg636-client.virtnet: -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 872 (time 1790950469.96 total 10.01 last 87.11) - open/close 1781 (time 1790950479.97 total 20.02 last 90.85) - open/close 2721 (time 1790950489.98 total 30.02 last 93.92) - open/close 3520 (time 1790950499.98 total 40.03 last 79.88) - open/close 4282 (time 1790950509.98 total 50.03 last 76.17) total: 4500 open/close in 52.55 seconds: 85.64 ops/second - /unlink 1388 (time 1790950527.23 total 10.00 last 138.78) - /unlink 2834 (time 1790950537.24 total 20.01 last 144.52) - /unlink 4283 (time 1790950547.26 total 30.03 last 144.59) total: 4500 /unlink in 31.70 seconds: 141.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 868 (time 1790950564.16 total 10.00 last 86.79) - open/close 1753 (time 1790950574.17 total 20.01 last 88.41) - open/close 2637 (time 1790950584.18 total 30.02 last 88.31) - open/close 3541 (time 1790950594.19 total 40.03 last 90.33) - open/close 4363 (time 1790950604.20 total 50.04 last 82.14) total: 4500 open/close in 51.75 seconds: 86.96 ops/second - /unlink 1628 (time 1790950620.09 total 10.00 last 162.73) - /unlink 3207 (time 1790950630.09 total 20.01 last 157.88) total: 4500 /unlink in 28.20 seconds: 159.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 900 (time 1790950652.99 total 10.01 last 89.94) - open/close 1789 (time 1790950663.00 total 20.02 last 88.77) - open/close 2690 (time 1790950673.01 total 30.03 last 90.02) - open/close 3560 (time 1790950683.02 total 40.04 last 86.90) - open/close 4331 (time 1790950693.03 total 50.05 last 77.07) total: 4500 open/close in 51.96 seconds: 86.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1549 (time 1790950708.98 total 10.01 last 154.79) - /unlink 3131 (time 1790950718.98 total 20.01 last 158.11) total: 4500 /unlink in 28.66 seconds: 157.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 865 (time 1790950741.93 total 10.01 last 86.38) - open/close 1695 (time 1790950751.94 total 20.02 last 82.98) - open/close 2460 (time 1790950761.94 total 30.02 last 76.47) - open/close 3362 (time 1790950771.95 total 40.03 last 90.09) - open/close 4196 (time 1790950781.95 total 50.04 last 83.37) total: 4500 open/close in 53.16 seconds: 84.65 ops/second - /unlink 1765 (time 1790950799.06 total 10.01 last 176.40) - /unlink 3394 (time 1790950809.06 total 20.01 last 162.84) total: 4500 /unlink in 26.05 seconds: 172.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 792 (time 1790950829.58 total 10.02 last 79.05) - open/close 1607 (time 1790950839.59 total 20.03 last 81.40) - open/close 2358 (time 1790950849.60 total 30.04 last 74.99) - open/close 3192 (time 1790950859.61 total 40.05 last 83.32) - open/close 4041 (time 1790950869.62 total 50.06 last 84.84) total: 4500 open/close in 55.36 seconds: 81.29 ops/second - /unlink 1682 (time 1790950888.53 total 10.01 last 168.10) - /unlink 3309 (time 1790950898.53 total 20.01 last 162.59) total: 4500 /unlink in 28.47 seconds: 158.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 807 (time 1790950921.85 total 10.00 last 80.68) - open/close 1640 (time 1790950931.86 total 20.01 last 83.26) - open/close 2438 (time 1790950941.86 total 30.01 last 79.78) - open/close 3331 (time 1790950951.87 total 40.02 last 89.21) - open/close 4191 (time 1790950961.87 total 50.02 last 86.00) total: 4500 open/close in 54.10 seconds: 83.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1704 (time 1790950979.81 total 10.00 last 170.32) - /unlink 3146 (time 1790950989.81 total 20.01 last 144.19) total: 4500 /unlink in 28.49 seconds: 157.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 826 (time 1790951012.90 total 10.01 last 82.56) - open/close 1739 (time 1790951022.90 total 20.01 last 91.25) - open/close 2586 (time 1790951032.91 total 30.02 last 84.63) - open/close 3468 (time 1790951042.92 total 40.03 last 88.13) - open/close 4343 (time 1790951052.92 total 50.03 last 87.50) total: 4500 open/close in 51.69 seconds: 87.05 ops/second - /unlink 1631 (time 1790951068.35 total 10.01 last 163.01) - /unlink 3222 (time 1790951078.36 total 20.02 last 158.82) total: 4500 /unlink in 27.72 seconds: 162.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 906 (time 1790951100.36 total 10.00 last 90.56) - open/close 1739 (time 1790951110.36 total 20.01 last 83.27) - open/close 2559 (time 1790951120.36 total 30.01 last 81.99) - open/close 3450 (time 1790951130.37 total 40.02 last 89.05) - open/close 4324 (time 1790951140.37 total 50.02 last 87.38) total: 4500 open/close in 52.29 seconds: 86.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1610 (time 1790951157.23 total 10.01 last 160.81) - /unlink 3315 (time 1790951167.24 total 20.01 last 170.47) total: 4500 /unlink in 27.93 seconds: 161.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 837 (time 1790951189.66 total 10.00 last 83.67) - open/close 1601 (time 1790951199.67 total 20.02 last 76.29) - open/close 2508 (time 1790951209.68 total 30.02 last 90.65) - open/close 3371 (time 1790951219.70 total 40.05 last 86.10) - open/close 4218 (time 1790951229.71 total 50.05 last 84.67) total: 4500 open/close in 53.34 seconds: 84.36 ops/second - /unlink 1716 (time 1790951247.17 total 10.00 last 171.58) - /unlink 3233 (time 1790951257.17 total 20.00 last 151.68) total: 4500 /unlink in 28.13 seconds: 159.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 859 (time 1790951280.97 total 10.01 last 85.83) - open/close 1586 (time 1790951290.97 total 20.01 last 72.65) - open/close 2326 (time 1790951300.98 total 30.01 last 73.99) - open/close 3131 (time 1790951310.98 total 40.02 last 80.44) - open/close 3963 (time 1790951320.99 total 50.03 last 83.14) total: 4500 open/close in 57.16 seconds: 78.73 ops/second - /unlink 1589 (time 1790951343.36 total 10.00 last 158.83) - /unlink 2706 (time 1790951353.36 total 20.01 last 111.66) - /unlink 4111 (time 1790951363.37 total 30.02 last 140.39) total: 4500 /unlink in 33.11 seconds: 135.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 733 (time 1790951381.71 total 10.01 last 73.23) - open/close 1580 (time 1790951391.71 total 20.01 last 84.68) - open/close 2562 (time 1790951401.72 total 30.02 last 98.15) - open/close 3409 (time 1790951411.72 total 40.02 last 84.68) - open/close 4338 (time 1790951421.72 total 50.02 last 92.86) total: 4500 open/close in 51.96 seconds: 86.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1530 (time 1790951438.05 total 10.00 last 152.98) - /unlink 3160 (time 1790951448.05 total 20.01 last 162.90) total: 4500 /unlink in 28.06 seconds: 160.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 893 (time 1790951470.95 total 10.00 last 89.27) - open/close 1682 (time 1790951480.95 total 20.01 last 78.86) - open/close 2551 (time 1790951490.95 total 30.01 last 86.89) - open/close 3479 (time 1790951500.96 total 40.01 last 92.76) - open/close 4301 (time 1790951510.98 total 50.03 last 82.06) total: 4500 open/close in 52.40 seconds: 85.88 ops/second - /unlink 1752 (time 1790951527.36 total 10.00 last 175.19) - /unlink 2985 (time 1790951537.37 total 20.01 last 123.19) - /unlink 4174 (time 1790951547.38 total 30.02 last 118.77) total: 4500 /unlink in 33.20 seconds: 135.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 800 (time 1790951565.32 total 10.02 last 79.84) - open/close 1586 (time 1790951575.32 total 20.03 last 78.53) - open/close 2187 (time 1790951585.33 total 30.03 last 60.08) - open/close 3016 (time 1790951595.33 total 40.03 last 82.89) - open/close 3723 (time 1790951605.34 total 50.05 last 70.59) - open/close 4374 (time 1790951615.35 total 60.06 last 65.06) total: 4500 open/close in 61.98 seconds: 72.60 ops/second - /unlink 1453 (time 1790951632.90 total 10.00 last 145.25) - /unlink 2784 (time 1790951642.90 total 20.01 last 133.06) - /unlink 4392 (time 1790951652.90 total 30.01 last 160.80) total: 4500 /unlink in 30.86 seconds: 145.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 847 (time 1790951668.13 total 10.01 last 84.65) - open/close 1648 (time 1790951678.14 total 20.02 last 80.03) - open/close 2256 (time 1790951688.15 total 30.03 last 60.71) - open/close 2978 (time 1790951698.15 total 40.03 last 72.18) - open/close 3728 (time 1790951708.18 total 50.06 last 74.78) - open/close 4494 (time 1790951718.19 total 60.06 last 76.57) total: 4500 open/close in 60.13 seconds: 74.84 ops/second - /unlink 1532 (time 1790951733.10 total 10.01 last 153.08) - /unlink 3077 (time 1790951743.10 total 20.01 last 154.49) total: 4500 /unlink in 29.23 seconds: 153.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 859 (time 1790951767.46 total 10.01 last 85.84) - open/close 1664 (time 1790951777.46 total 20.01 last 80.49) - open/close 2460 (time 1790951787.46 total 30.01 last 79.58) - open/close 3401 (time 1790951797.47 total 40.02 last 94.04) - open/close 4309 (time 1790951807.47 total 50.02 last 90.73) total: 4500 open/close in 52.00 seconds: 86.54 ops/second - /unlink 1522 (time 1790951823.24 total 10.00 last 152.16) - /unlink 3095 (time 1790951833.24 total 20.00 last 157.30) total: 4500 /unlink in 29.24 seconds: 153.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 937 (time 1790951857.17 total 10.01 last 93.62) - open/close 1834 (time 1790951867.18 total 20.02 last 89.58) - open/close 2708 (time 1790951877.19 total 30.03 last 87.29) - open/close 3644 (time 1790951887.20 total 40.04 last 93.54) total: 4500 open/close in 49.65 seconds: 90.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1750 (time 1790951911.53 total 10.01 last 174.89) - /unlink 3249 (time 1790951921.53 total 20.01 last 149.86) total: 4500 /unlink in 27.82 seconds: 161.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 751 (time 1790951945.27 total 10.02 last 74.98) - open/close 1436 (time 1790951955.28 total 20.03 last 68.42) - open/close 2047 (time 1790951965.31 total 30.06 last 60.93) - open/close 2975 (time 1790951975.33 total 40.08 last 92.63) - open/close 3829 (time 1790951985.33 total 50.08 last 85.37) total: 4500 open/close in 57.40 seconds: 78.40 ops/second - /unlink 1653 (time 1790952006.45 total 10.00 last 165.27) - /unlink 3218 (time 1790952016.46 total 20.01 last 156.42) total: 4500 /unlink in 28.22 seconds: 159.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 910 (time 1790952038.61 total 10.00 last 90.99) - open/close 1790 (time 1790952048.61 total 20.00 last 87.97) - open/close 2613 (time 1790952058.66 total 30.05 last 81.89) - open/close 3481 (time 1790952068.66 total 40.06 last 86.78) - open/close 4214 (time 1790952078.66 total 50.06 last 73.28) total: 4500 open/close in 54.50 seconds: 82.56 ops/second - /unlink 1571 (time 1790952097.33 total 10.00 last 157.02) - /unlink 3121 (time 1790952107.34 total 20.02 last 154.77) total: 4500 /unlink in 29.32 seconds: 153.46 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11358: 95346 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg636-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg636-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg636-server stop mds service on oleg636-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg636-server stop mds service on oleg636-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg636-server unloading modules via unload_modules_local on: 'oleg636-server' oleg636-server: oleg636-server.virtnet: executing unload_modules_local oleg636-server: kmemleak disabled modules unloaded.