== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:30:14 (1790775014) Stopping clients: oleg613-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg613-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg613-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg613-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg613-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg613-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg613-server oleg613-server: oleg613-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg613-server' oleg613-server: oleg613-server.virtnet: executing load_modules_local oleg613-server: kmemleak disabled oleg613-server: Loading modules from /home/green/git/lustre-release/lustre oleg613-server: detected 4 online CPUs by sysfs oleg613-server: MODOPTS_LIBCFS= oleg613-server: Force libcfs to create 2 CPU partitions oleg613-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg613-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg613-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg613-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 641 (time 1790775209.74 total 10.01 last 64.06) - open/close 1383 (time 1790775219.74 total 20.01 last 74.18) - open/close 2127 (time 1790775229.76 total 30.02 last 74.29) - open/close 2863 (time 1790775239.76 total 40.03 last 73.58) - open/close 3568 (time 1790775249.77 total 50.03 last 70.45) - open/close 4295 (time 1790775259.77 total 60.04 last 72.68) total: 4500 open/close in 63.33 seconds: 71.06 ops/second - /unlink 1421 (time 1790775277.24 total 10.00 last 142.04) - /unlink 2781 (time 1790775287.24 total 20.00 last 136.00) - /unlink 4110 (time 1790775297.25 total 30.01 last 132.88) total: 4500 /unlink in 33.46 seconds: 134.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 697 (time 1790775315.40 total 10.01 last 69.65) - open/close 1287 (time 1790775325.41 total 20.01 last 58.97) - open/close 1923 (time 1790775335.42 total 30.02 last 63.54) - open/close 2562 (time 1790775345.43 total 40.03 last 63.83) - open/close 3189 (time 1790775355.44 total 50.04 last 62.65) - open/close 3835 (time 1790775365.45 total 60.05 last 64.51) total: 4500 open/close in 69.65 seconds: 64.61 ops/second - /unlink 1337 (time 1790775389.18 total 10.01 last 133.61) - /unlink 2645 (time 1790775399.19 total 20.02 last 130.68) - /unlink 4074 (time 1790775409.19 total 30.02 last 142.86) total: 4500 /unlink in 32.87 seconds: 136.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 669 (time 1790775426.77 total 10.00 last 66.88) - open/close 1301 (time 1790775436.77 total 20.00 last 63.19) - open/close 2039 (time 1790775446.79 total 30.02 last 73.69) - open/close 2751 (time 1790775456.79 total 40.02 last 71.16) - open/close 3496 (time 1790775466.81 total 50.04 last 74.38) - open/close 4186 (time 1790775476.81 total 60.04 last 68.99) total: 4500 open/close in 64.12 seconds: 70.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1478 (time 1790775495.33 total 10.01 last 147.71) - /unlink 2791 (time 1790775505.33 total 20.01 last 131.24) - /unlink 4144 (time 1790775515.33 total 30.01 last 135.28) total: 4500 /unlink in 32.63 seconds: 137.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 701 (time 1790775532.93 total 10.01 last 70.03) - open/close 1438 (time 1790775542.93 total 20.01 last 73.69) - open/close 2029 (time 1790775552.93 total 30.01 last 59.09) - open/close 2529 (time 1790775562.94 total 40.02 last 49.97) - open/close 3121 (time 1790775572.95 total 50.03 last 59.16) - open/close 3913 (time 1790775582.95 total 60.03 last 79.20) total: 4500 open/close in 68.41 seconds: 65.78 ops/second - /unlink 1367 (time 1790775606.12 total 10.00 last 136.65) - /unlink 2730 (time 1790775616.12 total 20.01 last 136.24) - /unlink 4128 (time 1790775626.13 total 30.01 last 139.73) total: 4500 /unlink in 32.62 seconds: 137.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 755 (time 1790775643.42 total 10.01 last 75.42) - open/close 1456 (time 1790775653.43 total 20.02 last 70.06) - open/close 2212 (time 1790775663.43 total 30.02 last 75.57) - open/close 2935 (time 1790775673.43 total 40.02 last 72.28) - open/close 3706 (time 1790775683.45 total 50.04 last 77.00) - open/close 4386 (time 1790775693.45 total 60.04 last 67.96) total: 4500 open/close in 61.77 seconds: 72.85 ops/second - /unlink 1329 (time 1790775710.18 total 10.00 last 132.89) - /unlink 2754 (time 1790775720.18 total 20.00 last 142.49) - /unlink 4037 (time 1790775730.18 total 30.00 last 128.26) total: 4500 /unlink in 33.82 seconds: 133.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 583 (time 1790775750.11 total 10.01 last 58.25) - open/close 1293 (time 1790775760.12 total 20.02 last 70.93) - open/close 2034 (time 1790775770.12 total 30.02 last 74.05) - open/close 2693 (time 1790775780.14 total 40.04 last 65.80) - open/close 3454 (time 1790775790.14 total 50.04 last 76.10) - open/close 4164 (time 1790775800.15 total 60.05 last 70.95) total: 4500 open/close in 65.18 seconds: 69.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1351 (time 1790775820.02 total 10.00 last 135.08) - /unlink 2552 (time 1790775830.02 total 20.00 last 120.06) - /unlink 3656 (time 1790775840.03 total 30.01 last 110.31) total: 4500 /unlink in 38.27 seconds: 117.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 632 (time 1790775864.20 total 10.02 last 63.10) - open/close 1360 (time 1790775874.21 total 20.02 last 72.78) - open/close 2211 (time 1790775884.21 total 30.02 last 85.08) - open/close 2930 (time 1790775894.22 total 40.03 last 71.85) - open/close 3603 (time 1790775904.22 total 50.03 last 67.28) - open/close 4301 (time 1790775914.27 total 60.08 last 69.46) total: 4500 open/close in 62.60 seconds: 71.88 ops/second - /unlink 1549 (time 1790775931.49 total 10.00 last 154.84) - /unlink 2790 (time 1790775941.50 total 20.01 last 124.02) - /unlink 4206 (time 1790775951.50 total 30.02 last 141.47) total: 4500 /unlink in 32.60 seconds: 138.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 715 (time 1790775970.51 total 10.00 last 71.49) - open/close 1427 (time 1790775980.51 total 20.01 last 71.17) - open/close 2191 (time 1790775990.51 total 30.01 last 76.39) - open/close 2935 (time 1790776000.52 total 40.01 last 74.35) - open/close 3648 (time 1790776010.52 total 50.02 last 71.28) - open/close 4426 (time 1790776020.55 total 60.04 last 77.61) total: 4500 open/close in 61.28 seconds: 73.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1365 (time 1790776036.17 total 10.00 last 136.46) - /unlink 2368 (time 1790776046.17 total 20.01 last 100.27) - /unlink 3464 (time 1790776056.18 total 30.01 last 109.57) total: 4500 /unlink in 38.27 seconds: 117.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 760 (time 1790776079.69 total 10.00 last 75.99) - open/close 1373 (time 1790776089.70 total 20.01 last 61.25) - open/close 2104 (time 1790776099.70 total 30.01 last 73.08) - open/close 2853 (time 1790776109.71 total 40.02 last 74.82) - open/close 3624 (time 1790776119.71 total 50.03 last 77.06) - open/close 4377 (time 1790776129.72 total 60.03 last 75.25) total: 4500 open/close in 61.92 seconds: 72.68 ops/second - /unlink 1476 (time 1790776146.25 total 10.02 last 147.38) - /unlink 2818 (time 1790776156.25 total 20.02 last 134.20) - /unlink 3916 (time 1790776166.26 total 30.03 last 109.67) total: 4500 /unlink in 35.81 seconds: 125.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 718 (time 1790776188.90 total 10.00 last 71.80) - open/close 1384 (time 1790776198.91 total 20.01 last 66.53) - open/close 2021 (time 1790776208.92 total 30.02 last 63.66) - open/close 2609 (time 1790776218.93 total 40.03 last 58.71) - open/close 3243 (time 1790776228.94 total 50.03 last 63.38) - open/close 3963 (time 1790776238.94 total 60.04 last 71.95) total: 4500 open/close in 67.31 seconds: 66.86 ops/second - /unlink 1364 (time 1790776260.75 total 10.00 last 136.36) - /unlink 2660 (time 1790776270.76 total 20.01 last 129.45) - /unlink 4129 (time 1790776280.77 total 30.02 last 146.82) total: 4500 /unlink in 32.40 seconds: 138.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 732 (time 1790776298.29 total 10.01 last 73.13) - open/close 1428 (time 1790776308.29 total 20.02 last 69.56) - open/close 2221 (time 1790776318.29 total 30.02 last 79.28) - open/close 2896 (time 1790776328.30 total 40.02 last 67.49) - open/close 3661 (time 1790776338.30 total 50.02 last 76.50) - open/close 4339 (time 1790776348.32 total 60.04 last 67.64) total: 4500 open/close in 61.76 seconds: 72.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1209 (time 1790776365.25 total 10.01 last 120.81) - /unlink 2403 (time 1790776375.25 total 20.01 last 119.39) - /unlink 3630 (time 1790776385.25 total 30.01 last 122.69) total: 4500 /unlink in 36.30 seconds: 123.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 651 (time 1790776406.27 total 10.01 last 65.01) - open/close 1400 (time 1790776416.27 total 20.02 last 74.86) - open/close 2135 (time 1790776426.29 total 30.03 last 73.41) - open/close 2920 (time 1790776436.29 total 40.04 last 78.48) - open/close 3668 (time 1790776446.29 total 50.04 last 74.76) - open/close 4426 (time 1790776456.30 total 60.04 last 75.77) total: 4500 open/close in 61.13 seconds: 73.62 ops/second - /unlink 1363 (time 1790776472.57 total 10.01 last 136.21) - /unlink 2656 (time 1790776482.58 total 20.01 last 129.25) - /unlink 4008 (time 1790776492.58 total 30.02 last 135.08) total: 4500 /unlink in 34.24 seconds: 131.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 647 (time 1790776513.21 total 10.01 last 64.67) - open/close 1319 (time 1790776523.22 total 20.01 last 67.18) - open/close 1897 (time 1790776533.23 total 30.02 last 57.73) - open/close 2521 (time 1790776543.23 total 40.03 last 62.37) - open/close 3110 (time 1790776553.25 total 50.04 last 58.80) - open/close 3870 (time 1790776563.26 total 60.05 last 75.93) - open/close 4436 (time 1790776573.27 total 70.06 last 56.54) total: 4500 open/close in 71.20 seconds: 63.21 ops/second - /unlink 1240 (time 1790776589.76 total 10.00 last 123.96) - /unlink 2487 (time 1790776599.77 total 20.01 last 124.66) - /unlink 3820 (time 1790776609.77 total 30.01 last 133.26) total: 4500 /unlink in 35.36 seconds: 127.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 716 (time 1790776629.70 total 10.01 last 71.55) - open/close 1443 (time 1790776639.71 total 20.02 last 72.59) - open/close 2119 (time 1790776649.72 total 30.03 last 67.56) - open/close 2719 (time 1790776659.72 total 40.03 last 60.00) - open/close 3400 (time 1790776669.73 total 50.04 last 68.00) - open/close 3990 (time 1790776679.74 total 60.05 last 58.94) total: 4500 open/close in 68.46 seconds: 65.73 ops/second - /unlink 1292 (time 1790776702.95 total 10.00 last 129.14) - /unlink 2319 (time 1790776712.96 total 20.01 last 102.65) - /unlink 3412 (time 1790776722.96 total 30.01 last 109.29) total: 4500 /unlink in 38.08 seconds: 118.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 632 (time 1790776746.68 total 10.01 last 63.12) - open/close 1325 (time 1790776756.70 total 20.03 last 69.17) - open/close 2004 (time 1790776766.71 total 30.04 last 67.84) - open/close 2553 (time 1790776776.73 total 40.06 last 54.79) - open/close 3242 (time 1790776786.73 total 50.06 last 68.90) - open/close 3993 (time 1790776796.74 total 60.06 last 75.08) total: 4500 open/close in 67.43 seconds: 66.73 ops/second - /unlink 1302 (time 1790776818.57 total 10.00 last 130.17) - /unlink 2590 (time 1790776828.57 total 20.00 last 128.77) - /unlink 3853 (time 1790776838.57 total 30.00 last 126.29) total: 4500 /unlink in 35.35 seconds: 127.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 630 (time 1790776858.93 total 10.00 last 62.99) - open/close 1240 (time 1790776868.94 total 20.01 last 60.95) - open/close 1900 (time 1790776878.95 total 30.02 last 65.95) - open/close 2614 (time 1790776888.98 total 40.05 last 71.20) - open/close 3271 (time 1790776898.99 total 50.06 last 65.61) - open/close 3950 (time 1790776909.01 total 60.07 last 67.80) total: 4500 open/close in 67.68 seconds: 66.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1312 (time 1790776931.15 total 10.01 last 131.11) - /unlink 2588 (time 1790776941.16 total 20.02 last 127.48) - /unlink 3753 (time 1790776951.17 total 30.03 last 116.36) total: 4500 /unlink in 36.17 seconds: 124.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 751 (time 1790776972.93 total 10.01 last 75.00) - open/close 1460 (time 1790776982.93 total 20.02 last 70.88) - open/close 2165 (time 1790776992.95 total 30.03 last 70.36) - open/close 2844 (time 1790777002.96 total 40.04 last 67.87) - open/close 3470 (time 1790777012.97 total 50.05 last 62.53) - open/close 4193 (time 1790777022.99 total 60.07 last 72.16) total: 4500 open/close in 64.77 seconds: 69.47 ops/second - /unlink 1402 (time 1790777041.34 total 10.00 last 140.17) - /unlink 2627 (time 1790777051.35 total 20.01 last 122.40) - /unlink 3930 (time 1790777061.36 total 30.01 last 130.27) total: 4500 /unlink in 34.66 seconds: 129.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 657 (time 1790777080.63 total 10.01 last 65.65) - open/close 1290 (time 1790777090.64 total 20.02 last 63.24) - open/close 1944 (time 1790777100.64 total 30.02 last 65.37) - open/close 2574 (time 1790777110.65 total 40.03 last 62.97) - open/close 3169 (time 1790777120.65 total 50.03 last 59.49) - open/close 3842 (time 1790777130.65 total 60.03 last 67.30) total: 4500 open/close in 69.31 seconds: 64.93 ops/second - /unlink 1279 (time 1790777154.58 total 10.00 last 127.85) - /unlink 2547 (time 1790777164.59 total 20.01 last 126.77) - /unlink 3808 (time 1790777174.59 total 30.01 last 126.03) total: 4500 /unlink in 35.85 seconds: 125.52 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11358: 95524 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg613-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg613-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg613-server stop mds service on oleg613-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg613-server stop mds service on oleg613-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg613-server unloading modules via unload_modules_local on: 'oleg613-server' oleg613-server: oleg613-server.virtnet: executing unload_modules_local oleg613-server: kmemleak disabled modules unloaded.