== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:00:51 (1788609651) Stopping clients: oleg314-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg314-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg314-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg314-server oleg314-server: oleg314-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing load_modules_local oleg314-server: kmemleak disabled oleg314-server: Loading modules from /home/green/git/lustre-release/lustre oleg314-server: detected 4 online CPUs by sysfs oleg314-server: MODOPTS_LIBCFS= oleg314-server: Force libcfs to create 2 CPU partitions oleg314-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg314-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg314-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg314-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 629 (time 1788609861.32 total 10.01 last 62.87) - open/close 1330 (time 1788609871.33 total 20.02 last 70.02) - open/close 2010 (time 1788609881.34 total 30.02 last 67.96) - open/close 2743 (time 1788609891.34 total 40.03 last 73.27) - open/close 3419 (time 1788609901.35 total 50.03 last 67.57) - open/close 4070 (time 1788609911.35 total 60.03 last 65.10) total: 4500 open/close in 66.86 seconds: 67.31 ops/second - /unlink 1310 (time 1788609932.95 total 10.01 last 130.92) - /unlink 2534 (time 1788609942.95 total 20.01 last 122.34) - /unlink 3611 (time 1788609952.96 total 30.01 last 107.67) total: 4500 /unlink in 36.67 seconds: 122.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 617 (time 1788609974.42 total 10.01 last 61.63) - open/close 1272 (time 1788609984.42 total 20.02 last 65.45) - open/close 1851 (time 1788609994.43 total 30.02 last 57.89) - open/close 2500 (time 1788610004.43 total 40.03 last 64.86) - open/close 3161 (time 1788610014.44 total 50.03 last 66.08) - open/close 3840 (time 1788610024.44 total 60.04 last 67.87) - open/close 4482 (time 1788610034.45 total 70.04 last 64.15) total: 4500 open/close in 70.32 seconds: 63.99 ops/second - /unlink 1273 (time 1788610050.06 total 10.01 last 127.19) - /unlink 2534 (time 1788610060.08 total 20.02 last 125.92) - /unlink 3722 (time 1788610070.08 total 30.03 last 118.69) total: 4500 /unlink in 36.44 seconds: 123.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 644 (time 1788610092.20 total 10.01 last 64.34) - open/close 1302 (time 1788610102.21 total 20.02 last 65.72) - open/close 1983 (time 1788610112.22 total 30.03 last 68.04) - open/close 2673 (time 1788610122.23 total 40.04 last 68.95) - open/close 3334 (time 1788610132.23 total 50.04 last 66.07) - open/close 3967 (time 1788610142.26 total 60.07 last 63.15) total: 4500 open/close in 68.31 seconds: 65.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1240 (time 1788610165.84 total 10.01 last 123.91) - /unlink 2439 (time 1788610175.84 total 20.01 last 119.90) - /unlink 3656 (time 1788610185.85 total 30.01 last 121.68) total: 4500 /unlink in 37.01 seconds: 121.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 620 (time 1788610207.80 total 10.00 last 61.98) - open/close 1240 (time 1788610217.81 total 20.01 last 61.97) - open/close 1911 (time 1788610227.81 total 30.01 last 67.05) - open/close 2615 (time 1788610237.82 total 40.02 last 70.33) - open/close 3307 (time 1788610247.83 total 50.03 last 69.15) - open/close 3973 (time 1788610257.83 total 60.03 last 66.60) total: 4500 open/close in 67.88 seconds: 66.29 ops/second - /unlink 1317 (time 1788610280.12 total 10.00 last 131.68) - /unlink 2602 (time 1788610290.13 total 20.01 last 128.45) - /unlink 3896 (time 1788610300.13 total 30.01 last 129.34) total: 4500 /unlink in 35.02 seconds: 128.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 575 (time 1788610320.13 total 10.01 last 57.44) - open/close 1208 (time 1788610330.13 total 20.01 last 63.29) - open/close 1832 (time 1788610340.14 total 30.02 last 62.36) - open/close 2449 (time 1788610350.14 total 40.02 last 61.69) - open/close 3134 (time 1788610360.17 total 50.05 last 68.32) - open/close 3784 (time 1788610370.17 total 60.05 last 64.97) - open/close 4480 (time 1788610380.18 total 70.06 last 69.56) total: 4500 open/close in 70.28 seconds: 64.03 ops/second - /unlink 1127 (time 1788610395.79 total 10.00 last 112.67) - /unlink 2326 (time 1788610405.79 total 20.01 last 119.86) - /unlink 3517 (time 1788610415.80 total 30.01 last 119.04) total: 4500 /unlink in 38.38 seconds: 117.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 638 (time 1788610439.46 total 10.01 last 63.74) - open/close 1322 (time 1788610449.48 total 20.03 last 68.29) - open/close 2004 (time 1788610459.48 total 30.03 last 68.13) - open/close 2664 (time 1788610469.49 total 40.04 last 65.97) - open/close 3383 (time 1788610479.49 total 50.04 last 71.88) - open/close 3966 (time 1788610489.49 total 60.04 last 58.29) total: 4500 open/close in 68.05 seconds: 66.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1264 (time 1788610512.52 total 10.01 last 126.28) - /unlink 2462 (time 1788610522.53 total 20.01 last 119.76) - /unlink 3714 (time 1788610532.53 total 30.02 last 125.10) total: 4500 /unlink in 37.03 seconds: 121.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 578 (time 1788610555.26 total 10.00 last 57.78) - open/close 1268 (time 1788610565.26 total 20.01 last 68.96) - open/close 1824 (time 1788610575.28 total 30.03 last 55.51) - open/close 2431 (time 1788610585.28 total 40.03 last 60.69) - open/close 2967 (time 1788610595.30 total 50.04 last 53.52) - open/close 3561 (time 1788610605.30 total 60.05 last 59.38) - open/close 4132 (time 1788610615.30 total 70.05 last 57.10) total: 4500 open/close in 75.57 seconds: 59.55 ops/second - /unlink 1270 (time 1788610635.30 total 10.01 last 126.93) - /unlink 2444 (time 1788610645.30 total 20.01 last 117.39) - /unlink 3646 (time 1788610655.30 total 30.01 last 120.16) total: 4500 /unlink in 36.65 seconds: 122.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 737 (time 1788610676.34 total 10.01 last 73.63) - open/close 1473 (time 1788610686.34 total 20.01 last 73.59) - open/close 2190 (time 1788610696.34 total 30.01 last 71.69) - open/close 2858 (time 1788610706.35 total 40.02 last 66.75) - open/close 3577 (time 1788610716.36 total 50.03 last 71.80) - open/close 4322 (time 1788610726.37 total 60.04 last 74.44) total: 4500 open/close in 62.04 seconds: 72.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1286 (time 1788610742.08 total 10.01 last 128.53) - /unlink 2715 (time 1788610752.08 total 20.01 last 142.81) - /unlink 4099 (time 1788610762.09 total 30.01 last 138.38) total: 4500 /unlink in 33.10 seconds: 135.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 627 (time 1788610780.60 total 10.01 last 62.66) - open/close 1222 (time 1788610790.61 total 20.01 last 59.47) - open/close 1892 (time 1788610800.61 total 30.01 last 66.98) - open/close 2610 (time 1788610810.64 total 40.04 last 71.58) - open/close 3228 (time 1788610820.65 total 50.05 last 61.75) - open/close 4000 (time 1788610830.66 total 60.07 last 77.09) total: 4500 open/close in 66.89 seconds: 67.27 ops/second - /unlink 1313 (time 1788610851.95 total 10.01 last 131.23) - /unlink 2583 (time 1788610861.95 total 20.01 last 126.95) - /unlink 3868 (time 1788610871.96 total 30.01 last 128.47) total: 4500 /unlink in 36.44 seconds: 123.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 511 (time 1788610895.54 total 10.02 last 51.02) - open/close 1111 (time 1788610905.54 total 20.02 last 59.98) - open/close 1705 (time 1788610915.55 total 30.02 last 59.39) - open/close 2308 (time 1788610925.55 total 40.03 last 60.26) - open/close 2956 (time 1788610935.56 total 50.03 last 64.77) - open/close 3585 (time 1788610945.56 total 60.04 last 62.87) - open/close 4233 (time 1788610955.58 total 70.05 last 64.70) total: 4500 open/close in 73.98 seconds: 60.82 ops/second - /unlink 1282 (time 1788610974.47 total 10.00 last 128.14) - /unlink 2583 (time 1788610984.48 total 20.01 last 130.05) - /unlink 3892 (time 1788610994.48 total 30.01 last 130.89) total: 4500 /unlink in 34.96 seconds: 128.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 649 (time 1788611014.67 total 10.00 last 64.88) - open/close 1300 (time 1788611024.67 total 20.00 last 65.10) - open/close 1939 (time 1788611034.70 total 30.03 last 63.72) - open/close 2605 (time 1788611044.70 total 40.03 last 66.58) - open/close 3270 (time 1788611054.71 total 50.04 last 66.44) - open/close 3904 (time 1788611064.71 total 60.05 last 63.37) total: 4500 open/close in 69.03 seconds: 65.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1297 (time 1788611088.01 total 10.01 last 129.62) - /unlink 2544 (time 1788611098.02 total 20.02 last 124.58) - /unlink 3725 (time 1788611108.03 total 30.03 last 117.94) total: 4500 /unlink in 36.80 seconds: 122.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 672 (time 1788611130.10 total 10.00 last 67.17) - open/close 1361 (time 1788611140.10 total 20.01 last 68.85) - open/close 2009 (time 1788611150.11 total 30.02 last 64.75) - open/close 2563 (time 1788611160.11 total 40.02 last 55.39) - open/close 3233 (time 1788611170.11 total 50.02 last 66.99) - open/close 3877 (time 1788611180.12 total 60.03 last 64.37) total: 4500 open/close in 70.00 seconds: 64.29 ops/second - /unlink 1352 (time 1788611204.25 total 10.00 last 135.18) - /unlink 2665 (time 1788611214.25 total 20.00 last 131.29) - /unlink 3911 (time 1788611224.25 total 30.00 last 124.59) total: 4500 /unlink in 34.80 seconds: 129.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 648 (time 1788611243.93 total 10.02 last 64.67) - open/close 1348 (time 1788611253.93 total 20.03 last 69.96) - open/close 2041 (time 1788611263.97 total 30.06 last 69.07) - open/close 2784 (time 1788611273.97 total 40.06 last 74.29) - open/close 3505 (time 1788611283.97 total 50.07 last 72.07) - open/close 4216 (time 1788611293.98 total 60.08 last 71.03) total: 4500 open/close in 64.01 seconds: 70.30 ops/second - /unlink 1263 (time 1788611313.01 total 10.01 last 126.22) - /unlink 2558 (time 1788611323.02 total 20.01 last 129.40) - /unlink 3806 (time 1788611333.02 total 30.02 last 124.75) total: 4500 /unlink in 35.67 seconds: 126.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 642 (time 1788611352.99 total 10.02 last 64.08) - open/close 1309 (time 1788611362.99 total 20.02 last 66.67) - open/close 1984 (time 1788611373.03 total 30.06 last 67.27) - open/close 2642 (time 1788611383.04 total 40.07 last 65.72) - open/close 3303 (time 1788611393.05 total 50.08 last 66.03) - open/close 3877 (time 1788611403.07 total 60.10 last 57.30) total: 4500 open/close in 69.14 seconds: 65.08 ops/second - /unlink 1269 (time 1788611426.80 total 10.00 last 126.84) - /unlink 2430 (time 1788611436.80 total 20.01 last 116.02) - /unlink 3637 (time 1788611446.81 total 30.02 last 120.63) total: 4500 /unlink in 37.24 seconds: 120.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 519 (time 1788611470.69 total 10.04 last 51.71) - open/close 1110 (time 1788611480.70 total 20.04 last 59.06) - open/close 1692 (time 1788611490.72 total 30.06 last 58.10) - open/close 2350 (time 1788611500.72 total 40.06 last 65.78) - open/close 2989 (time 1788611510.73 total 50.08 last 63.81) - open/close 3656 (time 1788611520.74 total 60.09 last 66.63) - open/close 4273 (time 1788611530.76 total 70.10 last 61.61) total: 4500 open/close in 74.01 seconds: 60.80 ops/second - /unlink 1153 (time 1788611549.31 total 10.01 last 115.21) - /unlink 2152 (time 1788611559.32 total 20.01 last 99.85) - /unlink 3440 (time 1788611569.32 total 30.02 last 128.74) total: 4500 /unlink in 38.64 seconds: 116.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 645 (time 1788611592.40 total 10.02 last 64.39) - open/close 1265 (time 1788611602.40 total 20.02 last 61.98) - open/close 1919 (time 1788611612.42 total 30.04 last 65.29) - open/close 2593 (time 1788611622.43 total 40.05 last 67.30) - open/close 3158 (time 1788611632.47 total 50.09 last 56.27) - open/close 3780 (time 1788611642.50 total 60.12 last 62.05) - open/close 4444 (time 1788611652.51 total 70.13 last 66.33) total: 4500 open/close in 71.08 seconds: 63.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1289 (time 1788611667.68 total 10.01 last 128.81) - /unlink 2456 (time 1788611677.69 total 20.01 last 116.63) - /unlink 3412 (time 1788611687.69 total 30.02 last 95.57) total: 4500 /unlink in 39.71 seconds: 113.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 634 (time 1788611712.34 total 10.01 last 63.36) - open/close 1251 (time 1788611722.35 total 20.02 last 61.64) - open/close 1844 (time 1788611732.35 total 30.02 last 59.29) - open/close 2450 (time 1788611742.35 total 40.02 last 60.57) - open/close 2902 (time 1788611752.38 total 50.05 last 45.10) - open/close 3422 (time 1788611762.38 total 60.05 last 51.99) - open/close 4072 (time 1788611772.41 total 70.08 last 64.81) total: 4500 open/close in 76.22 seconds: 59.04 ops/second - /unlink 1364 (time 1788611792.54 total 10.00 last 136.38) - /unlink 2628 (time 1788611802.55 total 20.01 last 126.34) - /unlink 3873 (time 1788611812.55 total 30.01 last 124.44) total: 4500 /unlink in 35.81 seconds: 125.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 691 (time 1788611832.74 total 10.00 last 69.09) - open/close 1394 (time 1788611842.75 total 20.01 last 70.25) - open/close 2074 (time 1788611852.75 total 30.01 last 67.97) - open/close 2723 (time 1788611862.77 total 40.02 last 64.82) - open/close 3405 (time 1788611872.77 total 50.03 last 68.15) - open/close 4087 (time 1788611882.77 total 60.03 last 68.19) total: 4500 open/close in 66.33 seconds: 67.85 ops/second - /unlink 1282 (time 1788611904.05 total 10.01 last 128.10) - /unlink 2462 (time 1788611914.05 total 20.01 last 117.98) - /unlink 3888 (time 1788611924.06 total 30.02 last 142.45) total: 4500 /unlink in 34.67 seconds: 129.79 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95263 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg314-server unloading modules via unload_modules_local on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing unload_modules_local oleg314-server: kmemleak disabled modules unloaded.