== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:18:32 (1788841112) Stopping clients: oleg416-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg416-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg416-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg416-server oleg416-server: oleg416-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: kmemleak disabled oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: MODOPTS_LIBCFS= oleg416-server: Force libcfs to create 2 CPU partitions oleg416-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg416-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg416-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg416-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg416-client.virtnet: -o user_xattr,flock 192.168.204.116@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 781 (time 1788841342.77 total 10.01 last 78.05) - open/close 1575 (time 1788841352.78 total 20.02 last 79.29) - open/close 2412 (time 1788841362.79 total 30.03 last 83.59) - open/close 3272 (time 1788841372.81 total 40.04 last 85.90) - open/close 4006 (time 1788841382.81 total 50.05 last 73.37) total: 4500 open/close in 56.50 seconds: 79.64 ops/second - /unlink 1454 (time 1788841403.75 total 10.00 last 145.40) - /unlink 2940 (time 1788841413.76 total 20.00 last 148.60) - /unlink 4397 (time 1788841423.76 total 30.00 last 145.67) total: 4500 /unlink in 31.19 seconds: 144.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 686 (time 1788841439.52 total 10.00 last 68.57) - open/close 1508 (time 1788841449.52 total 20.01 last 82.18) - open/close 2225 (time 1788841459.53 total 30.01 last 71.66) - open/close 2904 (time 1788841469.53 total 40.01 last 67.89) - open/close 3637 (time 1788841479.53 total 50.02 last 73.27) - open/close 4430 (time 1788841489.54 total 60.02 last 79.27) total: 4500 open/close in 61.03 seconds: 73.74 ops/second - /unlink 1453 (time 1788841505.20 total 10.00 last 145.30) - /unlink 2598 (time 1788841515.21 total 20.00 last 114.47) - /unlink 3815 (time 1788841525.21 total 30.01 last 121.65) total: 4500 /unlink in 34.36 seconds: 130.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 690 (time 1788841544.55 total 10.01 last 68.94) - open/close 1482 (time 1788841554.56 total 20.02 last 79.13) - open/close 2309 (time 1788841564.56 total 30.03 last 82.62) - open/close 3086 (time 1788841574.58 total 40.04 last 77.62) - open/close 3936 (time 1788841584.58 total 50.04 last 84.95) total: 4500 open/close in 56.80 seconds: 79.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1496 (time 1788841605.54 total 10.00 last 149.58) - /unlink 2930 (time 1788841615.55 total 20.01 last 143.33) - /unlink 4399 (time 1788841625.55 total 30.01 last 146.88) total: 4500 /unlink in 30.86 seconds: 145.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 827 (time 1788841640.58 total 10.01 last 82.61) - open/close 1608 (time 1788841650.59 total 20.02 last 78.03) - open/close 2416 (time 1788841660.59 total 30.02 last 80.79) - open/close 3142 (time 1788841670.60 total 40.03 last 72.56) - open/close 3856 (time 1788841680.61 total 50.03 last 71.34) total: 4500 open/close in 58.60 seconds: 76.80 ops/second - /unlink 1559 (time 1788841704.09 total 10.00 last 155.86) - /unlink 3106 (time 1788841714.10 total 20.01 last 154.54) total: 4500 /unlink in 29.46 seconds: 152.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 850 (time 1788841739.43 total 10.00 last 84.99) - open/close 1700 (time 1788841749.43 total 20.00 last 84.97) - open/close 2543 (time 1788841759.44 total 30.01 last 84.22) - open/close 3323 (time 1788841769.45 total 40.02 last 77.94) - open/close 4069 (time 1788841779.45 total 50.02 last 74.59) total: 4500 open/close in 55.86 seconds: 80.56 ops/second - /unlink 1396 (time 1788841800.28 total 10.00 last 139.58) - /unlink 2847 (time 1788841810.29 total 20.01 last 145.03) - /unlink 4338 (time 1788841820.29 total 30.01 last 149.05) total: 4500 /unlink in 31.42 seconds: 143.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 824 (time 1788841835.79 total 10.00 last 82.38) - open/close 1617 (time 1788841845.80 total 20.01 last 79.20) - open/close 2333 (time 1788841855.82 total 30.03 last 71.48) - open/close 3207 (time 1788841865.83 total 40.04 last 87.33) - open/close 4057 (time 1788841875.83 total 50.05 last 84.95) total: 4500 open/close in 55.83 seconds: 80.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1461 (time 1788841896.30 total 10.01 last 145.98) - /unlink 2915 (time 1788841906.31 total 20.02 last 145.29) - /unlink 4328 (time 1788841916.31 total 30.02 last 141.23) total: 4500 /unlink in 31.74 seconds: 141.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 611 (time 1788841933.02 total 10.01 last 61.05) - open/close 1254 (time 1788841943.02 total 20.01 last 64.27) - open/close 1871 (time 1788841953.04 total 30.03 last 61.62) - open/close 2800 (time 1788841963.05 total 40.03 last 92.82) - open/close 3614 (time 1788841973.06 total 50.05 last 81.29) - open/close 4370 (time 1788841983.06 total 60.05 last 75.56) total: 4500 open/close in 61.86 seconds: 72.74 ops/second - /unlink 1440 (time 1788841999.26 total 10.01 last 143.93) - /unlink 2710 (time 1788842009.26 total 20.01 last 126.98) - /unlink 4153 (time 1788842019.26 total 30.01 last 144.29) total: 4500 /unlink in 32.51 seconds: 138.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 795 (time 1788842036.37 total 10.03 last 79.29) - open/close 1652 (time 1788842046.37 total 20.03 last 85.68) - open/close 2445 (time 1788842056.38 total 30.04 last 79.23) - open/close 3241 (time 1788842066.39 total 40.05 last 79.55) - open/close 4062 (time 1788842076.41 total 50.06 last 81.96) total: 4500 open/close in 55.23 seconds: 81.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1401 (time 1788842096.74 total 10.00 last 140.08) - /unlink 2768 (time 1788842106.74 total 20.00 last 136.68) - /unlink 4103 (time 1788842116.74 total 30.01 last 133.45) total: 4500 /unlink in 33.33 seconds: 135.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 761 (time 1788842134.54 total 10.00 last 76.10) - open/close 1543 (time 1788842144.54 total 20.01 last 78.15) - open/close 2305 (time 1788842154.55 total 30.01 last 76.13) - open/close 3119 (time 1788842164.56 total 40.02 last 81.33) - open/close 3869 (time 1788842174.56 total 50.02 last 75.00) total: 4500 open/close in 58.34 seconds: 77.13 ops/second - /unlink 1586 (time 1788842197.18 total 10.01 last 158.46) - /unlink 2971 (time 1788842207.18 total 20.01 last 138.48) - /unlink 4477 (time 1788842217.20 total 30.03 last 150.26) total: 4500 /unlink in 30.28 seconds: 148.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 660 (time 1788842231.91 total 10.02 last 65.90) - open/close 1345 (time 1788842241.92 total 20.02 last 68.45) - open/close 2129 (time 1788842251.92 total 30.02 last 78.39) - open/close 2894 (time 1788842261.92 total 40.03 last 76.46) - open/close 3643 (time 1788842271.93 total 50.04 last 74.85) - open/close 4390 (time 1788842281.93 total 60.04 last 74.68) total: 4500 open/close in 61.45 seconds: 73.23 ops/second - /unlink 1295 (time 1788842297.62 total 10.00 last 129.49) - /unlink 2740 (time 1788842307.63 total 20.01 last 144.42) - /unlink 4278 (time 1788842317.63 total 30.01 last 153.76) total: 4500 /unlink in 31.56 seconds: 142.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 610 (time 1788842333.51 total 10.01 last 60.92) - open/close 1305 (time 1788842343.51 total 20.02 last 69.44) - open/close 1947 (time 1788842353.51 total 30.02 last 64.20) - open/close 2535 (time 1788842363.51 total 40.02 last 58.80) - open/close 3199 (time 1788842373.52 total 50.03 last 66.36) - open/close 4022 (time 1788842383.53 total 60.03 last 82.25) total: 4500 open/close in 65.87 seconds: 68.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1521 (time 1788842403.14 total 10.00 last 152.03) - /unlink 2965 (time 1788842413.14 total 20.01 last 144.35) - /unlink 4316 (time 1788842423.14 total 30.01 last 135.04) total: 4500 /unlink in 31.19 seconds: 144.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 780 (time 1788842438.78 total 10.01 last 77.94) - open/close 1599 (time 1788842448.80 total 20.02 last 81.80) - open/close 2373 (time 1788842458.81 total 30.03 last 77.29) - open/close 3199 (time 1788842468.81 total 40.04 last 82.58) - open/close 3986 (time 1788842478.83 total 50.05 last 78.58) total: 4500 open/close in 56.22 seconds: 80.04 ops/second - /unlink 1773 (time 1788842500.11 total 10.00 last 177.26) - /unlink 3127 (time 1788842510.11 total 20.00 last 135.38) - /unlink 4460 (time 1788842520.11 total 30.01 last 133.24) total: 4500 /unlink in 30.45 seconds: 147.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 749 (time 1788842535.63 total 10.04 last 74.56) - open/close 1541 (time 1788842545.63 total 20.05 last 79.17) - open/close 2198 (time 1788842555.64 total 30.06 last 65.65) - open/close 2871 (time 1788842565.64 total 40.06 last 67.29) - open/close 3568 (time 1788842575.64 total 50.06 last 69.68) - open/close 4325 (time 1788842585.65 total 60.07 last 75.62) total: 4500 open/close in 62.72 seconds: 71.75 ops/second - /unlink 1539 (time 1788842603.29 total 10.01 last 153.70) - /unlink 2993 (time 1788842613.30 total 20.03 last 145.19) - /unlink 4230 (time 1788842623.32 total 30.04 last 123.52) total: 4500 /unlink in 32.46 seconds: 138.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 834 (time 1788842641.56 total 10.00 last 83.36) - open/close 1615 (time 1788842651.58 total 20.02 last 77.95) - open/close 2523 (time 1788842661.58 total 30.03 last 90.77) - open/close 3375 (time 1788842671.59 total 40.03 last 85.16) - open/close 4167 (time 1788842681.59 total 50.04 last 79.18) total: 4500 open/close in 55.12 seconds: 81.65 ops/second - /unlink 1317 (time 1788842701.13 total 10.00 last 131.69) - /unlink 2445 (time 1788842711.15 total 20.02 last 112.56) - /unlink 3688 (time 1788842721.15 total 30.03 last 124.24) total: 4500 /unlink in 35.11 seconds: 128.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 676 (time 1788842741.31 total 10.01 last 67.56) - open/close 1419 (time 1788842751.32 total 20.02 last 74.22) - open/close 2208 (time 1788842761.33 total 30.02 last 78.86) - open/close 3018 (time 1788842771.35 total 40.04 last 80.83) - open/close 3777 (time 1788842781.35 total 50.05 last 75.89) total: 4500 open/close in 59.33 seconds: 75.85 ops/second - /unlink 1607 (time 1788842805.43 total 10.00 last 160.65) - /unlink 2863 (time 1788842815.43 total 20.01 last 125.55) - /unlink 4323 (time 1788842825.44 total 30.02 last 145.87) total: 4500 /unlink in 31.36 seconds: 143.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 847 (time 1788842840.68 total 10.01 last 84.65) - open/close 1616 (time 1788842850.69 total 20.01 last 76.85) - open/close 2417 (time 1788842860.70 total 30.02 last 80.00) - open/close 3272 (time 1788842870.70 total 40.03 last 85.48) - open/close 4004 (time 1788842880.72 total 50.05 last 73.03) total: 4500 open/close in 57.93 seconds: 77.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1111 (time 1788842904.97 total 10.01 last 111.00) - /unlink 2062 (time 1788842914.99 total 20.02 last 94.96) - /unlink 3104 (time 1788842925.01 total 30.04 last 104.01) total: 4500 /unlink in 39.98 seconds: 112.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 737 (time 1788842949.93 total 10.00 last 73.67) - open/close 1586 (time 1788842959.93 total 20.01 last 84.84) - open/close 2358 (time 1788842969.94 total 30.02 last 77.17) - open/close 3154 (time 1788842979.95 total 40.03 last 79.47) - open/close 3884 (time 1788842989.96 total 50.04 last 72.92) total: 4500 open/close in 58.71 seconds: 76.65 ops/second - /unlink 1485 (time 1788843013.76 total 10.01 last 148.41) - /unlink 2904 (time 1788843023.77 total 20.01 last 141.83) - /unlink 4385 (time 1788843033.77 total 30.01 last 148.06) total: 4500 /unlink in 30.88 seconds: 145.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 658 (time 1788843050.79 total 10.01 last 65.74) - open/close 1450 (time 1788843060.79 total 20.02 last 79.14) - open/close 2084 (time 1788843070.80 total 30.02 last 63.39) - open/close 2711 (time 1788843080.83 total 40.05 last 62.51) - open/close 3263 (time 1788843090.83 total 50.06 last 55.15) - open/close 3901 (time 1788843100.84 total 60.06 last 63.77) total: 4500 open/close in 68.82 seconds: 65.39 ops/second - /unlink 1323 (time 1788843123.65 total 10.00 last 132.28) - /unlink 2765 (time 1788843133.66 total 20.00 last 144.16) - /unlink 4173 (time 1788843143.66 total 30.01 last 140.77) total: 4500 /unlink in 32.51 seconds: 138.44 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95281 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg416-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg416-server unloading modules via unload_modules_local on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing unload_modules_local oleg416-server: kmemleak disabled modules unloaded.