== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:21:56 (1790900516) Stopping clients: oleg246-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg246-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg246-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg246-server oleg246-server: oleg246-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: kmemleak disabled oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: detected 4 online CPUs by sysfs oleg246-server: MODOPTS_LIBCFS= oleg246-server: Force libcfs to create 2 CPU partitions oleg246-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg246-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg246-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg246-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 576 (time 1790900721.64 total 10.00 last 57.59) - open/close 1197 (time 1790900731.64 total 20.00 last 62.10) - open/close 1937 (time 1790900741.64 total 30.01 last 73.96) - open/close 2822 (time 1790900751.65 total 40.02 last 88.42) - open/close 3659 (time 1790900761.65 total 50.02 last 83.70) - open/close 4487 (time 1790900771.65 total 60.02 last 82.79) total: 4500 open/close in 60.15 seconds: 74.82 ops/second - /unlink 1430 (time 1790900785.99 total 10.00 last 142.99) - /unlink 2758 (time 1790900795.99 total 20.00 last 132.77) - /unlink 4066 (time 1790900805.99 total 30.01 last 130.76) total: 4500 /unlink in 33.12 seconds: 135.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 755 (time 1790900823.55 total 10.00 last 75.47) - open/close 1575 (time 1790900833.56 total 20.02 last 81.90) - open/close 2415 (time 1790900843.57 total 30.02 last 83.94) - open/close 3203 (time 1790900853.57 total 40.03 last 78.79) - open/close 3948 (time 1790900863.58 total 50.03 last 74.45) total: 4500 open/close in 57.21 seconds: 78.65 ops/second - /unlink 1234 (time 1790900886.57 total 10.00 last 123.36) - /unlink 2417 (time 1790900896.58 total 20.01 last 118.22) - /unlink 3704 (time 1790900906.58 total 30.01 last 128.67) total: 4500 /unlink in 36.14 seconds: 124.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 609 (time 1790900928.15 total 10.01 last 60.82) - open/close 1246 (time 1790900938.16 total 20.02 last 63.65) - open/close 1918 (time 1790900948.17 total 30.03 last 67.17) - open/close 2490 (time 1790900958.17 total 40.03 last 57.17) - open/close 3281 (time 1790900968.17 total 50.03 last 79.09) - open/close 3899 (time 1790900978.18 total 60.04 last 61.77) total: 4500 open/close in 68.70 seconds: 65.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1197 (time 1790901001.27 total 10.00 last 119.64) - /unlink 2550 (time 1790901011.27 total 20.01 last 135.22) - /unlink 3824 (time 1790901021.28 total 30.02 last 127.34) total: 4500 /unlink in 35.15 seconds: 128.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 661 (time 1790901041.50 total 10.03 last 65.93) - open/close 1398 (time 1790901051.51 total 20.03 last 73.68) - open/close 2188 (time 1790901061.51 total 30.03 last 78.99) - open/close 3011 (time 1790901071.52 total 40.04 last 82.24) - open/close 3698 (time 1790901081.53 total 50.05 last 68.58) - open/close 4415 (time 1790901091.54 total 60.06 last 71.65) total: 4500 open/close in 61.14 seconds: 73.60 ops/second - /unlink 1216 (time 1790901107.67 total 10.00 last 121.56) - /unlink 2418 (time 1790901117.67 total 20.00 last 120.20) - /unlink 3528 (time 1790901127.68 total 30.01 last 110.96) total: 4500 /unlink in 38.09 seconds: 118.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 639 (time 1790901152.83 total 10.02 last 63.79) - open/close 1242 (time 1790901162.83 total 20.02 last 60.26) - open/close 1994 (time 1790901172.84 total 30.03 last 75.13) - open/close 2723 (time 1790901182.84 total 40.03 last 72.90) - open/close 3484 (time 1790901192.85 total 50.04 last 76.08) - open/close 4337 (time 1790901202.85 total 60.04 last 85.27) total: 4500 open/close in 61.91 seconds: 72.69 ops/second - /unlink 1297 (time 1790901219.30 total 10.00 last 129.67) - /unlink 2576 (time 1790901229.30 total 20.00 last 127.90) - /unlink 3892 (time 1790901239.31 total 30.01 last 131.49) total: 4500 /unlink in 34.82 seconds: 129.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 672 (time 1790901259.67 total 10.03 last 66.98) - open/close 1433 (time 1790901269.68 total 20.04 last 76.06) - open/close 2135 (time 1790901279.68 total 30.04 last 70.19) - open/close 2949 (time 1790901289.69 total 40.04 last 81.36) - open/close 3727 (time 1790901299.69 total 50.05 last 77.74) - open/close 4440 (time 1790901309.70 total 60.06 last 71.26) total: 4500 open/close in 60.76 seconds: 74.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 881 (time 1790901326.10 total 10.00 last 88.07) - /unlink 1954 (time 1790901336.12 total 20.02 last 107.16) - /unlink 3187 (time 1790901346.12 total 30.02 last 123.29) total: 4500 /unlink in 39.49 seconds: 113.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 690 (time 1790901370.89 total 10.01 last 68.96) - open/close 1329 (time 1790901380.90 total 20.01 last 63.86) - open/close 1964 (time 1790901390.91 total 30.02 last 63.46) - open/close 2473 (time 1790901400.91 total 40.02 last 50.87) - open/close 3095 (time 1790901410.92 total 50.03 last 62.18) - open/close 3774 (time 1790901420.92 total 60.04 last 67.84) - open/close 4198 (time 1790901430.93 total 70.04 last 42.36) total: 4500 open/close in 76.80 seconds: 58.59 ops/second - /unlink 679 (time 1790901454.55 total 10.00 last 67.87) - /unlink 1215 (time 1790901464.57 total 20.02 last 53.53) - /unlink 2225 (time 1790901474.58 total 30.03 last 100.85) - /unlink 3275 (time 1790901484.59 total 40.04 last 104.87) - /unlink 4317 (time 1790901494.61 total 50.06 last 103.99) total: 4500 /unlink in 52.04 seconds: 86.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 536 (time 1790901512.75 total 10.00 last 53.58) - open/close 1093 (time 1790901522.76 total 20.02 last 55.62) - open/close 1714 (time 1790901532.76 total 30.02 last 62.09) - open/close 2462 (time 1790901542.77 total 40.02 last 74.79) - open/close 3234 (time 1790901552.78 total 50.03 last 77.11) - open/close 3844 (time 1790901562.80 total 60.05 last 60.89) - open/close 4450 (time 1790901572.80 total 70.05 last 60.58) total: 4500 open/close in 70.90 seconds: 63.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1098 (time 1790901589.54 total 10.00 last 109.76) - /unlink 2289 (time 1790901599.56 total 20.03 last 118.83) - /unlink 3394 (time 1790901609.56 total 30.03 last 110.50) total: 4500 /unlink in 39.57 seconds: 113.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 703 (time 1790901634.75 total 10.01 last 70.24) - open/close 1406 (time 1790901644.75 total 20.01 last 70.29) - open/close 2091 (time 1790901654.76 total 30.01 last 68.47) - open/close 2781 (time 1790901664.77 total 40.03 last 68.90) - open/close 3529 (time 1790901674.80 total 50.06 last 74.60) - open/close 4139 (time 1790901684.80 total 60.06 last 61.00) total: 4500 open/close in 64.78 seconds: 69.47 ops/second - /unlink 1456 (time 1790901703.74 total 10.00 last 145.53) - /unlink 2827 (time 1790901713.74 total 20.01 last 137.04) - /unlink 4139 (time 1790901723.75 total 30.02 last 131.07) total: 4500 /unlink in 32.80 seconds: 137.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 539 (time 1790901742.86 total 10.00 last 53.88) - open/close 1110 (time 1790901752.88 total 20.02 last 57.01) - open/close 1646 (time 1790901762.89 total 30.03 last 53.56) - open/close 2341 (time 1790901772.89 total 40.03 last 69.48) - open/close 2958 (time 1790901782.89 total 50.03 last 61.70) - open/close 3639 (time 1790901792.89 total 60.04 last 68.06) - open/close 4222 (time 1790901802.90 total 70.04 last 58.29) total: 4500 open/close in 74.59 seconds: 60.33 ops/second - /unlink 1310 (time 1790901823.32 total 10.00 last 130.97) - /unlink 2581 (time 1790901833.32 total 20.01 last 127.04) - /unlink 3875 (time 1790901843.33 total 30.01 last 129.31) total: 4500 /unlink in 36.26 seconds: 124.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 636 (time 1790901865.39 total 10.01 last 63.55) - open/close 1251 (time 1790901875.40 total 20.02 last 61.42) - open/close 1817 (time 1790901885.43 total 30.05 last 56.41) - open/close 2391 (time 1790901895.45 total 40.07 last 57.33) - open/close 2973 (time 1790901905.45 total 50.07 last 58.17) - open/close 3619 (time 1790901915.45 total 60.07 last 64.60) - open/close 4162 (time 1790901925.45 total 70.07 last 54.29) total: 4500 open/close in 74.96 seconds: 60.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1235 (time 1790901946.79 total 10.01 last 123.40) - /unlink 2569 (time 1790901956.80 total 20.01 last 133.38) - /unlink 3747 (time 1790901966.80 total 30.01 last 117.80) total: 4500 /unlink in 37.22 seconds: 120.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 659 (time 1790901989.40 total 10.00 last 65.90) - open/close 1258 (time 1790901999.41 total 20.01 last 59.84) - open/close 1913 (time 1790902009.45 total 30.05 last 65.27) - open/close 2554 (time 1790902019.45 total 40.05 last 64.08) - open/close 3232 (time 1790902029.46 total 50.05 last 67.77) - open/close 3941 (time 1790902039.47 total 60.06 last 70.83) total: 4500 open/close in 67.89 seconds: 66.29 ops/second - /unlink 1307 (time 1790902062.15 total 10.01 last 130.63) - /unlink 2462 (time 1790902072.15 total 20.01 last 115.45) - /unlink 3699 (time 1790902082.15 total 30.01 last 123.68) total: 4500 /unlink in 36.56 seconds: 123.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 691 (time 1790902103.89 total 10.01 last 69.07) - open/close 1411 (time 1790902113.91 total 20.02 last 71.92) - open/close 2011 (time 1790902123.93 total 30.04 last 59.89) - open/close 2752 (time 1790902133.93 total 40.04 last 74.08) - open/close 3607 (time 1790902143.93 total 50.04 last 85.48) - open/close 4356 (time 1790902153.93 total 60.04 last 74.90) total: 4500 open/close in 61.75 seconds: 72.88 ops/second - /unlink 1440 (time 1790902170.70 total 10.00 last 143.97) - /unlink 2922 (time 1790902180.71 total 20.00 last 148.16) - /unlink 4218 (time 1790902190.71 total 30.01 last 129.57) total: 4500 /unlink in 32.18 seconds: 139.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 683 (time 1790902207.80 total 10.00 last 68.29) - open/close 1465 (time 1790902217.82 total 20.02 last 78.07) - open/close 2065 (time 1790902227.86 total 30.05 last 59.78) - open/close 2508 (time 1790902237.88 total 40.08 last 44.18) - open/close 3007 (time 1790902247.89 total 50.09 last 49.86) - open/close 3755 (time 1790902257.91 total 60.11 last 74.64) total: 4500 open/close in 69.78 seconds: 64.49 ops/second - /unlink 1377 (time 1790902282.33 total 10.00 last 137.69) - /unlink 3056 (time 1790902292.34 total 20.01 last 167.76) - /unlink 4402 (time 1790902302.35 total 30.02 last 134.48) total: 4500 /unlink in 30.72 seconds: 146.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 725 (time 1790902317.46 total 10.00 last 72.48) - open/close 1450 (time 1790902327.47 total 20.01 last 72.43) - open/close 2039 (time 1790902337.47 total 30.01 last 58.90) - open/close 2615 (time 1790902347.49 total 40.03 last 57.48) - open/close 3126 (time 1790902357.51 total 50.05 last 51.03) - open/close 3689 (time 1790902367.52 total 60.07 last 56.21) - open/close 4423 (time 1790902377.53 total 70.07 last 73.36) total: 4500 open/close in 71.00 seconds: 63.38 ops/second - /unlink 1270 (time 1790902393.51 total 10.01 last 126.93) - /unlink 2682 (time 1790902403.51 total 20.01 last 141.12) - /unlink 3909 (time 1790902413.52 total 30.02 last 122.64) total: 4500 /unlink in 34.94 seconds: 128.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 653 (time 1790902433.98 total 10.00 last 65.27) - open/close 1247 (time 1790902443.98 total 20.01 last 59.39) - open/close 1976 (time 1790902453.99 total 30.01 last 72.86) - open/close 2741 (time 1790902463.99 total 40.01 last 76.49) - open/close 3515 (time 1790902473.99 total 50.02 last 77.35) - open/close 4317 (time 1790902484.00 total 60.02 last 80.18) total: 4500 open/close in 62.47 seconds: 72.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1368 (time 1790902501.30 total 10.02 last 136.54) - /unlink 2780 (time 1790902511.31 total 20.03 last 141.11) - /unlink 4323 (time 1790902521.31 total 30.03 last 154.28) total: 4500 /unlink in 31.29 seconds: 143.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 865 (time 1790902536.84 total 10.00 last 86.50) - open/close 1516 (time 1790902546.84 total 20.01 last 65.05) - open/close 2078 (time 1790902556.85 total 30.01 last 56.17) - open/close 2735 (time 1790902566.86 total 40.02 last 65.66) - open/close 3377 (time 1790902576.88 total 50.04 last 64.05) - open/close 3990 (time 1790902586.92 total 60.08 last 61.07) total: 4500 open/close in 69.11 seconds: 65.12 ops/second - /unlink 1123 (time 1790902611.89 total 10.00 last 112.26) - /unlink 2320 (time 1790902621.89 total 20.00 last 119.69) - /unlink 3548 (time 1790902631.90 total 30.01 last 122.69) total: 4500 /unlink in 38.19 seconds: 117.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 682 (time 1790902655.55 total 10.03 last 68.01) - open/close 1306 (time 1790902665.56 total 20.04 last 62.32) - open/close 1892 (time 1790902675.57 total 30.05 last 58.53) - open/close 2516 (time 1790902685.58 total 40.06 last 62.35) - open/close 3189 (time 1790902695.59 total 50.07 last 67.25) - open/close 3876 (time 1790902705.60 total 60.07 last 68.67) total: 4500 open/close in 68.47 seconds: 65.72 ops/second - /unlink 1539 (time 1790902729.40 total 10.01 last 153.71) - /unlink 2834 (time 1790902739.40 total 20.02 last 129.45) - /unlink 4167 (time 1790902749.40 total 30.02 last 133.28) total: 4500 /unlink in 32.65 seconds: 137.84 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11390: 95468 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg246-server unloading modules via unload_modules_local on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing unload_modules_local oleg246-server: kmemleak disabled modules unloaded.