== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:21:31 (1790119291) Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg207-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg207-server oleg207-server: oleg207-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: kmemleak disabled oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: detected 4 online CPUs by sysfs oleg207-server: MODOPTS_LIBCFS= oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg207-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg207-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 496 (time 1790119535.91 total 10.01 last 49.55) - open/close 958 (time 1790119545.91 total 20.01 last 46.20) - open/close 1506 (time 1790119555.93 total 30.03 last 54.70) - open/close 1949 (time 1790119565.93 total 40.03 last 44.28) - open/close 2467 (time 1790119575.96 total 50.06 last 51.64) - open/close 2877 (time 1790119585.99 total 60.09 last 40.89) - open/close 3404 (time 1790119595.99 total 70.09 last 52.69) - open/close 3898 (time 1790119606.03 total 80.13 last 49.20) - open/close 4366 (time 1790119616.06 total 90.16 last 46.68) total: 4500 open/close in 93.18 seconds: 48.29 ops/second - /unlink 1308 (time 1790119635.42 total 10.00 last 130.77) - /unlink 2644 (time 1790119645.42 total 20.00 last 133.56) - /unlink 3978 (time 1790119655.42 total 30.01 last 133.38) total: 4500 /unlink in 34.12 seconds: 131.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 756 (time 1790119675.13 total 10.02 last 75.42) - open/close 1544 (time 1790119685.13 total 20.03 last 78.79) - open/close 2305 (time 1790119695.14 total 30.04 last 76.03) - open/close 2865 (time 1790119705.15 total 40.04 last 55.95) - open/close 3518 (time 1790119715.15 total 50.05 last 65.29) - open/close 4414 (time 1790119725.15 total 60.05 last 89.59) total: 4500 open/close in 60.98 seconds: 73.79 ops/second - /unlink 1185 (time 1790119740.62 total 10.00 last 118.50) - /unlink 2612 (time 1790119750.62 total 20.00 last 142.68) - /unlink 3676 (time 1790119760.62 total 30.00 last 106.37) total: 4500 /unlink in 35.81 seconds: 125.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 762 (time 1790119781.77 total 10.01 last 76.13) - open/close 1415 (time 1790119791.78 total 20.02 last 65.24) - open/close 2165 (time 1790119801.79 total 30.03 last 74.95) - open/close 2933 (time 1790119811.79 total 40.03 last 76.79) - open/close 3681 (time 1790119821.79 total 50.03 last 74.78) - open/close 4413 (time 1790119831.80 total 60.04 last 73.10) total: 4500 open/close in 61.20 seconds: 73.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1018 (time 1790119849.21 total 10.00 last 101.78) - /unlink 2001 (time 1790119859.21 total 20.01 last 98.24) - /unlink 3545 (time 1790119869.22 total 30.01 last 154.36) total: 4500 /unlink in 37.10 seconds: 121.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 626 (time 1790119892.55 total 10.00 last 62.59) - open/close 1397 (time 1790119902.56 total 20.00 last 77.08) - open/close 2018 (time 1790119912.56 total 30.01 last 62.10) - open/close 2768 (time 1790119922.61 total 40.06 last 74.63) - open/close 3435 (time 1790119932.63 total 50.08 last 66.54) - open/close 4009 (time 1790119942.64 total 60.09 last 57.36) total: 4500 open/close in 66.76 seconds: 67.41 ops/second - /unlink 1268 (time 1790119964.27 total 10.00 last 126.78) - /unlink 2704 (time 1790119974.27 total 20.00 last 143.57) - /unlink 3906 (time 1790119984.28 total 30.01 last 120.17) total: 4500 /unlink in 35.37 seconds: 127.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 685 (time 1790120004.79 total 10.00 last 68.48) - open/close 1325 (time 1790120014.80 total 20.01 last 63.96) - open/close 2115 (time 1790120024.80 total 30.01 last 78.99) - open/close 2800 (time 1790120034.82 total 40.03 last 68.36) - open/close 3403 (time 1790120044.84 total 50.05 last 60.17) - open/close 4023 (time 1790120054.85 total 60.06 last 61.96) total: 4500 open/close in 67.27 seconds: 66.90 ops/second - /unlink 896 (time 1790120078.42 total 10.00 last 89.56) - /unlink 1815 (time 1790120088.43 total 20.01 last 91.83) - /unlink 3013 (time 1790120098.44 total 30.02 last 119.70) - /unlink 4133 (time 1790120108.44 total 40.02 last 111.99) total: 4500 /unlink in 43.14 seconds: 104.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 612 (time 1790120126.59 total 10.00 last 61.19) - open/close 1235 (time 1790120136.60 total 20.01 last 62.24) - open/close 2005 (time 1790120146.60 total 30.01 last 77.00) - open/close 2710 (time 1790120156.61 total 40.02 last 70.42) - open/close 3398 (time 1790120166.62 total 50.03 last 68.75) - open/close 4170 (time 1790120176.63 total 60.04 last 77.11) total: 4500 open/close in 65.53 seconds: 68.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1424 (time 1790120197.00 total 10.00 last 142.37) - /unlink 2850 (time 1790120207.00 total 20.00 last 142.60) - /unlink 4133 (time 1790120217.00 total 30.00 last 128.29) total: 4500 /unlink in 32.78 seconds: 137.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 755 (time 1790120235.83 total 10.00 last 75.50) - open/close 1400 (time 1790120245.84 total 20.01 last 64.43) - open/close 2153 (time 1790120255.85 total 30.02 last 75.23) - open/close 2829 (time 1790120265.86 total 40.03 last 67.56) - open/close 3514 (time 1790120275.87 total 50.04 last 68.43) - open/close 4227 (time 1790120285.88 total 60.04 last 71.24) total: 4500 open/close in 63.86 seconds: 70.47 ops/second - /unlink 1268 (time 1790120305.00 total 10.02 last 126.60) - /unlink 2247 (time 1790120315.01 total 20.02 last 97.82) - /unlink 3318 (time 1790120325.01 total 30.03 last 107.08) total: 4500 /unlink in 39.83 seconds: 112.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 718 (time 1790120352.01 total 10.00 last 71.79) - open/close 1437 (time 1790120362.02 total 20.01 last 71.84) - open/close 2013 (time 1790120372.03 total 30.01 last 57.57) - open/close 2661 (time 1790120382.05 total 40.03 last 64.67) - open/close 3220 (time 1790120392.07 total 50.05 last 55.79) - open/close 3722 (time 1790120402.07 total 60.06 last 50.16) - open/close 4329 (time 1790120412.09 total 70.08 last 60.59) total: 4500 open/close in 72.75 seconds: 61.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1192 (time 1790120429.45 total 10.01 last 119.03) - /unlink 2442 (time 1790120439.45 total 20.02 last 124.98) - /unlink 3839 (time 1790120449.46 total 30.02 last 139.65) total: 4500 /unlink in 34.56 seconds: 130.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 680 (time 1790120469.21 total 10.01 last 67.96) - open/close 1297 (time 1790120479.22 total 20.01 last 61.68) - open/close 1808 (time 1790120489.23 total 30.03 last 51.01) - open/close 2463 (time 1790120499.28 total 40.07 last 65.22) - open/close 3165 (time 1790120509.29 total 50.09 last 70.08) - open/close 3894 (time 1790120519.31 total 60.10 last 72.80) total: 4500 open/close in 68.47 seconds: 65.72 ops/second - /unlink 1349 (time 1790120542.80 total 10.00 last 134.87) - /unlink 2654 (time 1790120552.80 total 20.01 last 130.44) - /unlink 3972 (time 1790120562.80 total 30.01 last 131.78) total: 4500 /unlink in 34.41 seconds: 130.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 637 (time 1790120582.85 total 10.01 last 63.62) - open/close 1299 (time 1790120592.85 total 20.01 last 66.18) - open/close 1956 (time 1790120602.86 total 30.02 last 65.64) - open/close 2644 (time 1790120612.87 total 40.03 last 68.74) - open/close 3425 (time 1790120622.88 total 50.04 last 78.07) - open/close 4139 (time 1790120632.90 total 60.06 last 71.26) total: 4500 open/close in 65.29 seconds: 68.93 ops/second - /unlink 1182 (time 1790120653.23 total 10.00 last 118.19) - /unlink 2426 (time 1790120663.23 total 20.00 last 124.36) - /unlink 3712 (time 1790120673.23 total 30.00 last 128.60) total: 4500 /unlink in 37.47 seconds: 120.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 618 (time 1790120696.64 total 10.02 last 61.65) - open/close 1108 (time 1790120706.65 total 20.03 last 48.95) - open/close 1733 (time 1790120716.66 total 30.05 last 62.43) - open/close 2391 (time 1790120726.66 total 40.05 last 65.80) - open/close 3022 (time 1790120736.67 total 50.06 last 63.03) - open/close 3835 (time 1790120746.68 total 60.06 last 81.27) - open/close 4465 (time 1790120756.68 total 70.06 last 62.98) total: 4500 open/close in 70.71 seconds: 63.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1203 (time 1790120773.13 total 10.01 last 120.22) - /unlink 2491 (time 1790120783.15 total 20.03 last 128.53) - /unlink 3819 (time 1790120793.15 total 30.03 last 132.76) total: 4500 /unlink in 34.52 seconds: 130.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 732 (time 1790120813.37 total 10.00 last 73.17) - open/close 1425 (time 1790120823.37 total 20.01 last 69.27) - open/close 2122 (time 1790120833.37 total 30.01 last 69.68) - open/close 2740 (time 1790120843.41 total 40.05 last 61.58) - open/close 3466 (time 1790120853.42 total 50.06 last 72.54) - open/close 4178 (time 1790120863.43 total 60.07 last 71.10) total: 4500 open/close in 65.50 seconds: 68.70 ops/second - /unlink 1309 (time 1790120884.35 total 10.00 last 130.85) - /unlink 2476 (time 1790120894.36 total 20.01 last 116.63) - /unlink 3752 (time 1790120904.36 total 30.01 last 127.56) total: 4500 /unlink in 35.72 seconds: 125.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 705 (time 1790120925.21 total 10.01 last 70.41) - open/close 1447 (time 1790120935.23 total 20.03 last 74.06) - open/close 2220 (time 1790120945.25 total 30.05 last 77.12) - open/close 3065 (time 1790120955.26 total 40.06 last 84.48) - open/close 3874 (time 1790120965.26 total 50.06 last 80.90) total: 4500 open/close in 57.93 seconds: 77.67 ops/second - /unlink 1513 (time 1790120988.75 total 10.00 last 151.25) - /unlink 2969 (time 1790120998.76 total 20.01 last 145.50) - /unlink 4242 (time 1790121008.77 total 30.02 last 127.20) total: 4500 /unlink in 31.92 seconds: 140.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 680 (time 1790121026.71 total 10.00 last 67.98) - open/close 1251 (time 1790121036.72 total 20.01 last 57.03) - open/close 1853 (time 1790121046.74 total 30.03 last 60.12) - open/close 2348 (time 1790121056.75 total 40.04 last 49.44) - open/close 2977 (time 1790121066.75 total 50.04 last 62.89) - open/close 3678 (time 1790121076.75 total 60.05 last 70.07) - open/close 4412 (time 1790121086.76 total 70.06 last 73.33) total: 4500 open/close in 71.22 seconds: 63.18 ops/second - /unlink 1271 (time 1790121103.38 total 10.01 last 126.95) - /unlink 2480 (time 1790121113.38 total 20.01 last 120.90) - /unlink 3659 (time 1790121123.39 total 30.02 last 117.87) total: 4500 /unlink in 37.35 seconds: 120.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 681 (time 1790121146.54 total 10.01 last 68.06) - open/close 1403 (time 1790121156.55 total 20.02 last 72.11) - open/close 2063 (time 1790121166.56 total 30.03 last 65.92) - open/close 2737 (time 1790121176.56 total 40.03 last 67.40) - open/close 3551 (time 1790121186.57 total 50.04 last 81.36) - open/close 4191 (time 1790121196.59 total 60.06 last 63.83) total: 4500 open/close in 65.44 seconds: 68.76 ops/second - /unlink 1379 (time 1790121217.41 total 10.01 last 137.74) - /unlink 2542 (time 1790121227.41 total 20.01 last 116.29) - /unlink 3661 (time 1790121237.42 total 30.02 last 111.82) total: 4500 /unlink in 38.64 seconds: 116.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 621 (time 1790121262.74 total 10.02 last 62.00) - open/close 1298 (time 1790121272.76 total 20.03 last 67.60) - open/close 1993 (time 1790121282.78 total 30.05 last 69.35) - open/close 2715 (time 1790121292.78 total 40.05 last 72.19) - open/close 3526 (time 1790121302.79 total 50.06 last 81.05) - open/close 4218 (time 1790121312.80 total 60.08 last 69.09) total: 4500 open/close in 64.44 seconds: 69.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1354 (time 1790121332.26 total 10.00 last 135.37) - /unlink 2605 (time 1790121342.26 total 20.00 last 125.07) - /unlink 3945 (time 1790121352.26 total 30.01 last 133.96) total: 4500 /unlink in 34.68 seconds: 129.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 583 (time 1790121373.56 total 10.00 last 58.28) - open/close 1195 (time 1790121383.57 total 20.01 last 61.14) - open/close 2100 (time 1790121393.59 total 30.03 last 90.36) - open/close 2721 (time 1790121403.61 total 40.05 last 61.95) - open/close 3619 (time 1790121413.62 total 50.06 last 89.74) - open/close 4208 (time 1790121423.62 total 60.06 last 58.89) total: 4500 open/close in 63.75 seconds: 70.59 ops/second - /unlink 1363 (time 1790121442.98 total 10.00 last 136.24) - /unlink 2646 (time 1790121452.98 total 20.00 last 128.30) - /unlink 3977 (time 1790121462.99 total 30.01 last 132.99) total: 4500 /unlink in 34.59 seconds: 130.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 581 (time 1790121483.05 total 10.01 last 58.05) - open/close 1245 (time 1790121493.06 total 20.02 last 66.32) - open/close 1936 (time 1790121503.09 total 30.05 last 68.91) - open/close 2560 (time 1790121513.09 total 40.05 last 62.40) - open/close 3107 (time 1790121523.09 total 50.05 last 54.70) - open/close 3510 (time 1790121533.10 total 60.06 last 40.25) - open/close 3965 (time 1790121543.11 total 70.07 last 45.47) total: 4500 open/close in 79.09 seconds: 56.90 ops/second - /unlink 1162 (time 1790121566.46 total 10.00 last 116.17) - /unlink 2537 (time 1790121576.46 total 20.00 last 137.49) - /unlink 3756 (time 1790121586.46 total 30.00 last 121.89) total: 4500 /unlink in 36.66 seconds: 122.75 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11390: 95648 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg207-server unloading modules via unload_modules_local on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local oleg207-server: kmemleak disabled modules unloaded.