== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:15:18 (1790100918) Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg351-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg351-server oleg351-server: oleg351-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: kmemleak disabled oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg351-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg351-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg351-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 620 (time 1790101138.24 total 10.02 last 61.90) - open/close 1211 (time 1790101148.25 total 20.02 last 59.08) - open/close 1785 (time 1790101158.26 total 30.03 last 57.34) - open/close 2401 (time 1790101168.26 total 40.04 last 61.55) - open/close 2941 (time 1790101178.27 total 50.04 last 53.97) - open/close 3558 (time 1790101188.28 total 60.05 last 61.66) - open/close 4155 (time 1790101198.29 total 70.06 last 59.63) total: 4500 open/close in 76.63 seconds: 58.72 ops/second - /unlink 1185 (time 1790101220.64 total 10.01 last 118.40) - /unlink 2360 (time 1790101230.64 total 20.01 last 117.48) - /unlink 3481 (time 1790101240.65 total 30.02 last 112.02) total: 4500 /unlink in 39.06 seconds: 115.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 707 (time 1790101264.74 total 10.01 last 70.66) - open/close 1362 (time 1790101274.75 total 20.01 last 65.46) - open/close 1992 (time 1790101284.76 total 30.03 last 62.89) - open/close 2647 (time 1790101294.77 total 40.03 last 65.47) - open/close 3374 (time 1790101304.78 total 50.04 last 72.62) - open/close 4071 (time 1790101314.79 total 60.06 last 69.62) total: 4500 open/close in 66.25 seconds: 67.92 ops/second - /unlink 1319 (time 1790101336.17 total 10.00 last 131.85) - /unlink 2685 (time 1790101346.17 total 20.00 last 136.60) - /unlink 4012 (time 1790101356.17 total 30.00 last 132.69) total: 4500 /unlink in 34.02 seconds: 132.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 712 (time 1790101374.60 total 10.02 last 71.03) - open/close 1377 (time 1790101384.60 total 20.03 last 66.48) - open/close 1973 (time 1790101394.62 total 30.04 last 59.50) - open/close 2567 (time 1790101404.62 total 40.05 last 59.38) - open/close 3212 (time 1790101414.63 total 50.06 last 64.44) - open/close 3847 (time 1790101424.66 total 60.08 last 63.36) total: 4500 open/close in 69.48 seconds: 64.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1251 (time 1790101449.21 total 10.00 last 125.08) - /unlink 2416 (time 1790101459.22 total 20.01 last 116.42) - /unlink 3623 (time 1790101469.23 total 30.01 last 120.63) total: 4500 /unlink in 36.76 seconds: 122.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 669 (time 1790101490.69 total 10.02 last 66.76) - open/close 1346 (time 1790101500.69 total 20.02 last 67.68) - open/close 2011 (time 1790101510.70 total 30.04 last 66.40) - open/close 2694 (time 1790101520.71 total 40.05 last 68.25) - open/close 3413 (time 1790101530.72 total 50.05 last 71.84) - open/close 4055 (time 1790101540.73 total 60.07 last 64.11) total: 4500 open/close in 67.30 seconds: 66.87 ops/second - /unlink 1344 (time 1790101562.66 total 10.01 last 134.32) - /unlink 2669 (time 1790101572.67 total 20.01 last 132.49) - /unlink 3986 (time 1790101582.67 total 30.01 last 131.64) total: 4500 /unlink in 33.64 seconds: 133.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 654 (time 1790101600.59 total 10.00 last 65.37) - open/close 1354 (time 1790101610.63 total 20.05 last 69.69) - open/close 2078 (time 1790101620.64 total 30.06 last 72.35) - open/close 2769 (time 1790101630.64 total 40.06 last 69.06) - open/close 3497 (time 1790101640.65 total 50.07 last 72.73) - open/close 4256 (time 1790101650.66 total 60.08 last 75.84) total: 4500 open/close in 63.15 seconds: 71.26 ops/second - /unlink 1294 (time 1790101667.86 total 10.00 last 129.37) - /unlink 2516 (time 1790101677.88 total 20.02 last 122.02) - /unlink 3743 (time 1790101687.88 total 30.02 last 122.68) total: 4500 /unlink in 35.88 seconds: 125.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 671 (time 1790101709.52 total 10.00 last 67.08) - open/close 1257 (time 1790101719.54 total 20.02 last 58.48) - open/close 1881 (time 1790101729.54 total 30.03 last 62.36) - open/close 2468 (time 1790101739.55 total 40.04 last 58.66) - open/close 3013 (time 1790101749.55 total 50.04 last 54.49) - open/close 3534 (time 1790101759.55 total 60.04 last 52.09) - open/close 4124 (time 1790101769.56 total 70.05 last 58.94) total: 4500 open/close in 77.01 seconds: 58.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1183 (time 1790101792.15 total 10.01 last 118.23) - /unlink 2370 (time 1790101802.16 total 20.02 last 118.51) - /unlink 3577 (time 1790101812.17 total 30.03 last 120.57) total: 4500 /unlink in 38.42 seconds: 117.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 713 (time 1790101836.04 total 10.01 last 71.19) - open/close 1363 (time 1790101846.05 total 20.02 last 64.94) - open/close 2029 (time 1790101856.05 total 30.03 last 66.57) - open/close 2734 (time 1790101866.06 total 40.04 last 70.41) - open/close 3435 (time 1790101876.07 total 50.05 last 70.04) - open/close 4116 (time 1790101886.08 total 60.05 last 68.07) total: 4500 open/close in 65.53 seconds: 68.68 ops/second - /unlink 1265 (time 1790101905.95 total 10.00 last 126.45) - /unlink 2511 (time 1790101915.96 total 20.01 last 124.46) - /unlink 3727 (time 1790101925.96 total 30.02 last 121.55) total: 4500 /unlink in 36.45 seconds: 123.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 619 (time 1790101947.54 total 10.00 last 61.88) - open/close 1303 (time 1790101957.54 total 20.01 last 68.38) - open/close 1966 (time 1790101967.55 total 30.01 last 66.28) - open/close 2624 (time 1790101977.55 total 40.01 last 65.78) - open/close 3296 (time 1790101987.56 total 50.02 last 67.17) - open/close 3974 (time 1790101997.56 total 60.03 last 67.75) total: 4500 open/close in 67.35 seconds: 66.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1260 (time 1790102019.44 total 10.00 last 125.97) - /unlink 2484 (time 1790102029.44 total 20.00 last 122.39) - /unlink 3674 (time 1790102039.45 total 30.01 last 118.89) total: 4500 /unlink in 38.10 seconds: 118.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 629 (time 1790102063.92 total 10.01 last 62.84) - open/close 1249 (time 1790102073.94 total 20.03 last 61.87) - open/close 1887 (time 1790102083.96 total 30.05 last 63.69) - open/close 2526 (time 1790102093.98 total 40.07 last 63.78) - open/close 3224 (time 1790102103.98 total 50.07 last 69.78) - open/close 3909 (time 1790102113.98 total 60.07 last 68.49) total: 4500 open/close in 68.62 seconds: 65.57 ops/second - /unlink 1350 (time 1790102136.79 total 10.00 last 134.96) - /unlink 2466 (time 1790102146.79 total 20.01 last 111.57) - /unlink 3667 (time 1790102156.79 total 30.01 last 120.08) total: 4500 /unlink in 36.69 seconds: 122.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 705 (time 1790102178.55 total 10.00 last 70.47) - open/close 1386 (time 1790102188.56 total 20.02 last 68.00) - open/close 2068 (time 1790102198.58 total 30.03 last 68.10) - open/close 2701 (time 1790102208.59 total 40.05 last 63.23) - open/close 3374 (time 1790102218.60 total 50.06 last 67.23) - open/close 4184 (time 1790102228.61 total 60.07 last 80.90) total: 4500 open/close in 65.06 seconds: 69.17 ops/second - /unlink 1269 (time 1790102249.13 total 10.01 last 126.77) - /unlink 2503 (time 1790102259.13 total 20.01 last 123.36) - /unlink 3668 (time 1790102269.14 total 30.02 last 116.41) total: 4500 /unlink in 36.64 seconds: 122.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 707 (time 1790102290.56 total 10.03 last 70.50) - open/close 1315 (time 1790102300.57 total 20.04 last 60.73) - open/close 1934 (time 1790102310.61 total 30.08 last 61.66) - open/close 2433 (time 1790102320.65 total 40.12 last 49.69) - open/close 2958 (time 1790102330.66 total 50.13 last 52.44) - open/close 3595 (time 1790102340.67 total 60.14 last 63.64) - open/close 4221 (time 1790102350.70 total 70.17 last 62.44) total: 4500 open/close in 74.00 seconds: 60.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1222 (time 1790102369.05 total 10.00 last 122.15) - /unlink 2445 (time 1790102379.05 total 20.01 last 122.25) - /unlink 3783 (time 1790102389.06 total 30.01 last 133.75) total: 4500 /unlink in 35.78 seconds: 125.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 629 (time 1790102409.21 total 10.01 last 62.83) - open/close 1324 (time 1790102419.23 total 20.03 last 69.40) - open/close 1971 (time 1790102429.24 total 30.04 last 64.63) - open/close 2668 (time 1790102439.24 total 40.04 last 69.69) - open/close 3325 (time 1790102449.26 total 50.06 last 65.58) - open/close 3945 (time 1790102459.28 total 60.08 last 61.85) total: 4500 open/close in 68.47 seconds: 65.72 ops/second - /unlink 1300 (time 1790102482.94 total 10.00 last 130.00) - /unlink 2473 (time 1790102492.95 total 20.01 last 117.21) - /unlink 3779 (time 1790102502.96 total 30.02 last 130.46) total: 4500 /unlink in 36.47 seconds: 123.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 660 (time 1790102524.46 total 10.01 last 65.95) - open/close 1307 (time 1790102534.46 total 20.01 last 64.66) - open/close 1929 (time 1790102544.48 total 30.03 last 62.11) - open/close 2598 (time 1790102554.49 total 40.04 last 66.84) - open/close 3175 (time 1790102564.49 total 50.04 last 57.67) - open/close 3750 (time 1790102574.50 total 60.05 last 57.48) - open/close 4358 (time 1790102584.52 total 70.07 last 60.68) total: 4500 open/close in 72.95 seconds: 61.69 ops/second - /unlink 1093 (time 1790102602.06 total 10.00 last 109.29) - /unlink 2215 (time 1790102612.06 total 20.01 last 112.13) - /unlink 3381 (time 1790102622.06 total 30.01 last 116.59) total: 4500 /unlink in 38.50 seconds: 116.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 580 (time 1790102646.10 total 10.03 last 57.84) - open/close 1246 (time 1790102656.13 total 20.06 last 66.39) - open/close 1862 (time 1790102666.14 total 30.07 last 61.53) - open/close 2466 (time 1790102676.14 total 40.07 last 60.39) - open/close 3135 (time 1790102686.15 total 50.08 last 66.85) - open/close 3904 (time 1790102696.16 total 60.09 last 76.86) total: 4500 open/close in 69.57 seconds: 64.68 ops/second - /unlink 1388 (time 1790102721.87 total 10.01 last 138.72) - /unlink 2528 (time 1790102731.88 total 20.01 last 113.94) - /unlink 3754 (time 1790102741.89 total 30.02 last 122.52) total: 4500 /unlink in 36.23 seconds: 124.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 583 (time 1790102763.66 total 10.00 last 58.30) - open/close 1281 (time 1790102773.67 total 20.01 last 69.74) - open/close 1960 (time 1790102783.68 total 30.02 last 67.81) - open/close 2653 (time 1790102793.69 total 40.03 last 69.23) - open/close 3298 (time 1790102803.70 total 50.04 last 64.45) - open/close 3947 (time 1790102813.71 total 60.05 last 64.84) total: 4500 open/close in 69.75 seconds: 64.52 ops/second - /unlink 1293 (time 1790102838.11 total 10.00 last 129.25) - /unlink 2712 (time 1790102848.11 total 20.00 last 141.90) - /unlink 4083 (time 1790102858.11 total 30.00 last 137.09) total: 4500 /unlink in 34.18 seconds: 131.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 667 (time 1790102877.85 total 10.01 last 66.63) - open/close 1236 (time 1790102887.87 total 20.03 last 56.77) - open/close 1888 (time 1790102897.88 total 30.04 last 65.14) - open/close 2613 (time 1790102907.90 total 40.06 last 72.35) - open/close 3219 (time 1790102917.91 total 50.08 last 60.53) - open/close 3791 (time 1790102927.92 total 60.08 last 57.18) - open/close 4401 (time 1790102937.93 total 70.09 last 60.96) total: 4500 open/close in 71.32 seconds: 63.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1107 (time 1790102954.43 total 10.01 last 110.59) - /unlink 2085 (time 1790102964.44 total 20.02 last 97.75) - /unlink 3119 (time 1790102974.45 total 30.02 last 103.32) - /unlink 4278 (time 1790102984.45 total 40.03 last 115.86) total: 4500 /unlink in 42.47 seconds: 105.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 696 (time 1790103002.81 total 10.00 last 69.59) - open/close 1276 (time 1790103012.82 total 20.01 last 57.97) - open/close 1925 (time 1790103022.82 total 30.01 last 64.89) - open/close 2478 (time 1790103032.83 total 40.02 last 55.22) - open/close 2961 (time 1790103042.85 total 50.04 last 48.23) - open/close 3481 (time 1790103052.86 total 60.05 last 51.91) - open/close 4102 (time 1790103062.87 total 70.06 last 62.05) total: 4500 open/close in 76.08 seconds: 59.15 ops/second - /unlink 1139 (time 1790103084.37 total 10.01 last 113.78) - /unlink 2308 (time 1790103094.37 total 20.01 last 116.88) - /unlink 3656 (time 1790103104.37 total 30.02 last 134.76) total: 4500 /unlink in 36.74 seconds: 122.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 704 (time 1790103126.40 total 10.01 last 70.35) - open/close 1363 (time 1790103136.41 total 20.01 last 65.89) - open/close 1914 (time 1790103146.43 total 30.03 last 54.97) - open/close 2515 (time 1790103156.43 total 40.04 last 60.07) - open/close 3074 (time 1790103166.45 total 50.05 last 55.82) - open/close 3688 (time 1790103176.49 total 60.09 last 61.18) - open/close 4316 (time 1790103186.50 total 70.10 last 62.73) total: 4500 open/close in 72.61 seconds: 61.97 ops/second - /unlink 1147 (time 1790103203.86 total 10.01 last 114.62) - /unlink 2383 (time 1790103213.87 total 20.01 last 123.59) - /unlink 3625 (time 1790103223.88 total 30.02 last 124.07) total: 4500 /unlink in 37.29 seconds: 120.69 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11390: 95106 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg351-server unloading modules via unload_modules_local on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local oleg351-server: kmemleak disabled modules unloaded.