== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:56:14 (1788533774) Stopping clients: oleg615-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg615-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg615-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg615-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg615-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg615-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg615-server oleg615-server: oleg615-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg615-server' oleg615-server: oleg615-server.virtnet: executing load_modules_local oleg615-server: kmemleak disabled oleg615-server: Loading modules from /home/green/git/lustre-release/lustre oleg615-server: detected 4 online CPUs by sysfs oleg615-server: MODOPTS_LIBCFS= oleg615-server: Force libcfs to create 2 CPU partitions oleg615-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg615-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg615-server: oleg615-server.virtnet: executing set_default_debug -1 all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg615-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg615-server: oleg615-server.virtnet: executing set_default_debug -1 all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg615-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg615-server: oleg615-server.virtnet: executing set_default_debug -1 all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg615-client.virtnet: -o user_xattr,flock 192.168.206.115@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 731 (time 1788533996.46 total 10.00 last 73.07) - open/close 1481 (time 1788534006.47 total 20.01 last 74.95) - open/close 2143 (time 1788534016.47 total 30.01 last 66.19) - open/close 2855 (time 1788534026.48 total 40.02 last 71.13) - open/close 3565 (time 1788534036.49 total 50.03 last 70.93) - open/close 4309 (time 1788534046.49 total 60.03 last 74.39) total: 4500 open/close in 62.66 seconds: 71.82 ops/second - /unlink 1318 (time 1788534064.28 total 10.00 last 131.77) - /unlink 2663 (time 1788534074.28 total 20.01 last 134.46) - /unlink 4099 (time 1788534084.28 total 30.01 last 143.58) total: 4500 /unlink in 33.33 seconds: 135.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 648 (time 1788534103.67 total 10.00 last 64.77) - open/close 1290 (time 1788534113.68 total 20.01 last 64.15) - open/close 2051 (time 1788534123.68 total 30.02 last 76.08) - open/close 2737 (time 1788534133.69 total 40.03 last 68.53) - open/close 3312 (time 1788534143.70 total 50.03 last 57.46) - open/close 4004 (time 1788534153.71 total 60.04 last 69.12) total: 4500 open/close in 66.28 seconds: 67.89 ops/second - /unlink 1498 (time 1788534174.81 total 10.00 last 149.76) - /unlink 2989 (time 1788534184.82 total 20.01 last 149.04) total: 4500 /unlink in 29.95 seconds: 150.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 702 (time 1788534209.92 total 10.03 last 69.99) - open/close 1344 (time 1788534219.93 total 20.04 last 64.16) - open/close 2095 (time 1788534229.93 total 30.04 last 75.08) - open/close 2862 (time 1788534239.95 total 40.05 last 76.60) - open/close 3609 (time 1788534249.95 total 50.05 last 74.70) - open/close 4364 (time 1788534259.95 total 60.06 last 75.46) total: 4500 open/close in 61.97 seconds: 72.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1567 (time 1788534276.79 total 10.00 last 156.63) - /unlink 2899 (time 1788534286.79 total 20.01 last 133.16) - /unlink 4288 (time 1788534296.80 total 30.01 last 138.83) total: 4500 /unlink in 31.50 seconds: 142.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 767 (time 1788534312.68 total 10.00 last 76.66) - open/close 1394 (time 1788534322.69 total 20.01 last 62.67) - open/close 1978 (time 1788534332.69 total 30.01 last 58.39) - open/close 2691 (time 1788534342.69 total 40.02 last 71.27) - open/close 3517 (time 1788534352.71 total 50.03 last 82.47) - open/close 4267 (time 1788534362.72 total 60.04 last 74.93) total: 4500 open/close in 63.33 seconds: 71.06 ops/second - /unlink 1368 (time 1788534380.45 total 10.02 last 136.56) - /unlink 2601 (time 1788534390.46 total 20.02 last 123.24) - /unlink 3947 (time 1788534400.47 total 30.03 last 134.46) total: 4500 /unlink in 34.04 seconds: 132.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 860 (time 1788534418.87 total 10.00 last 85.99) - open/close 1642 (time 1788534428.87 total 20.00 last 78.18) - open/close 2298 (time 1788534438.87 total 30.01 last 65.56) - open/close 3017 (time 1788534448.89 total 40.03 last 71.78) - open/close 3727 (time 1788534458.91 total 50.04 last 70.90) - open/close 4325 (time 1788534468.92 total 60.05 last 59.73) total: 4500 open/close in 62.86 seconds: 71.59 ops/second - /unlink 1418 (time 1788534487.11 total 10.00 last 141.73) - /unlink 2779 (time 1788534497.11 total 20.01 last 136.08) - /unlink 4034 (time 1788534507.11 total 30.01 last 125.42) total: 4500 /unlink in 33.42 seconds: 134.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 715 (time 1788534525.88 total 10.00 last 71.47) - open/close 1436 (time 1788534535.89 total 20.01 last 72.06) - open/close 2136 (time 1788534545.90 total 30.03 last 69.90) - open/close 2776 (time 1788534555.91 total 40.03 last 63.97) - open/close 3459 (time 1788534565.91 total 50.04 last 68.25) - open/close 4165 (time 1788534575.92 total 60.04 last 70.57) total: 4500 open/close in 64.40 seconds: 69.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1341 (time 1788534595.12 total 10.00 last 134.08) - /unlink 2469 (time 1788534605.13 total 20.01 last 112.75) - /unlink 3592 (time 1788534615.14 total 30.02 last 112.17) total: 4500 /unlink in 37.47 seconds: 120.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 701 (time 1788534638.47 total 10.01 last 70.05) - open/close 1380 (time 1788534648.48 total 20.01 last 67.87) - open/close 2146 (time 1788534658.49 total 30.02 last 76.51) - open/close 2862 (time 1788534668.50 total 40.03 last 71.56) - open/close 3623 (time 1788534678.50 total 50.03 last 76.07) - open/close 4378 (time 1788534688.51 total 60.04 last 75.42) total: 4500 open/close in 61.83 seconds: 72.78 ops/second - /unlink 1445 (time 1788534704.83 total 10.01 last 144.42) - /unlink 2837 (time 1788534714.85 total 20.02 last 138.96) - /unlink 4066 (time 1788534724.85 total 30.03 last 122.84) total: 4500 /unlink in 33.60 seconds: 133.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 624 (time 1788534744.51 total 10.00 last 62.37) - open/close 1265 (time 1788534754.52 total 20.02 last 64.03) - open/close 1947 (time 1788534764.52 total 30.02 last 68.19) - open/close 2620 (time 1788534774.53 total 40.02 last 67.27) - open/close 3158 (time 1788534784.55 total 50.04 last 53.70) - open/close 3666 (time 1788534794.55 total 60.05 last 50.77) - open/close 4317 (time 1788534804.55 total 70.05 last 65.08) total: 4500 open/close in 73.41 seconds: 61.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1405 (time 1788534821.98 total 10.00 last 140.48) - /unlink 2773 (time 1788534831.99 total 20.01 last 136.71) - /unlink 4123 (time 1788534841.99 total 30.01 last 134.95) total: 4500 /unlink in 33.26 seconds: 135.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 713 (time 1788534860.48 total 10.00 last 71.28) - open/close 1288 (time 1788534870.49 total 20.01 last 57.45) - open/close 1967 (time 1788534880.50 total 30.02 last 67.85) - open/close 2708 (time 1788534890.50 total 40.02 last 74.09) - open/close 3487 (time 1788534900.50 total 50.02 last 77.87) - open/close 4219 (time 1788534910.50 total 60.02 last 73.19) total: 4500 open/close in 64.07 seconds: 70.24 ops/second - /unlink 1330 (time 1788534929.00 total 10.00 last 132.95) - /unlink 2646 (time 1788534939.00 total 20.00 last 131.60) - /unlink 3976 (time 1788534949.00 total 30.01 last 132.96) total: 4500 /unlink in 33.83 seconds: 133.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 668 (time 1788534968.57 total 10.02 last 66.67) - open/close 1346 (time 1788534978.58 total 20.03 last 67.75) - open/close 2089 (time 1788534988.58 total 30.04 last 74.24) - open/close 2811 (time 1788534998.58 total 40.04 last 72.20) - open/close 3502 (time 1788535008.59 total 50.04 last 69.07) - open/close 4220 (time 1788535018.60 total 60.05 last 71.74) total: 4500 open/close in 64.24 seconds: 70.05 ops/second - /unlink 1259 (time 1788535038.06 total 10.01 last 125.81) - /unlink 2618 (time 1788535048.06 total 20.01 last 135.87) - /unlink 3850 (time 1788535058.08 total 30.03 last 122.93) total: 4500 /unlink in 34.95 seconds: 128.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 656 (time 1788535078.09 total 10.00 last 65.59) - open/close 1276 (time 1788535088.09 total 20.00 last 62.00) - open/close 1920 (time 1788535098.09 total 30.01 last 64.36) - open/close 2601 (time 1788535108.10 total 40.01 last 68.06) - open/close 3286 (time 1788535118.11 total 50.03 last 68.41) - open/close 3979 (time 1788535128.11 total 60.03 last 69.29) total: 4500 open/close in 67.59 seconds: 66.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1304 (time 1788535150.24 total 10.01 last 130.32) - /unlink 2668 (time 1788535160.25 total 20.01 last 136.32) - /unlink 4074 (time 1788535170.26 total 30.02 last 140.49) total: 4500 /unlink in 33.26 seconds: 135.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 722 (time 1788535188.75 total 10.02 last 72.05) - open/close 1337 (time 1788535198.79 total 20.06 last 61.25) - open/close 1910 (time 1788535208.79 total 30.06 last 57.29) - open/close 2541 (time 1788535218.80 total 40.07 last 63.06) - open/close 3162 (time 1788535228.80 total 50.08 last 62.05) - open/close 3800 (time 1788535238.81 total 60.09 last 63.75) total: 4500 open/close in 70.03 seconds: 64.26 ops/second - /unlink 1282 (time 1788535263.48 total 10.00 last 128.17) - /unlink 2599 (time 1788535273.50 total 20.03 last 131.38) - /unlink 4019 (time 1788535283.51 total 30.03 last 141.90) total: 4500 /unlink in 33.09 seconds: 135.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 627 (time 1788535301.84 total 10.00 last 62.68) - open/close 1326 (time 1788535311.84 total 20.00 last 69.90) - open/close 2094 (time 1788535321.84 total 30.01 last 76.77) - open/close 2688 (time 1788535331.85 total 40.01 last 59.39) - open/close 3288 (time 1788535341.85 total 50.01 last 59.98) - open/close 3879 (time 1788535351.87 total 60.03 last 58.97) - open/close 4406 (time 1788535361.88 total 70.04 last 52.67) total: 4500 open/close in 71.41 seconds: 63.02 ops/second - /unlink 1081 (time 1788535378.17 total 10.00 last 108.06) - /unlink 2303 (time 1788535388.17 total 20.01 last 122.16) - /unlink 3589 (time 1788535398.18 total 30.01 last 128.54) total: 4500 /unlink in 37.42 seconds: 120.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 659 (time 1788535420.86 total 10.00 last 65.90) - open/close 1400 (time 1788535430.88 total 20.01 last 74.00) - open/close 2178 (time 1788535440.89 total 30.03 last 77.68) - open/close 2949 (time 1788535450.90 total 40.04 last 77.02) - open/close 3702 (time 1788535460.91 total 50.05 last 75.21) - open/close 4380 (time 1788535470.97 total 60.11 last 67.43) total: 4500 open/close in 62.05 seconds: 72.52 ops/second - /unlink 1421 (time 1788535487.39 total 10.00 last 142.07) - /unlink 2952 (time 1788535497.39 total 20.00 last 153.07) - /unlink 4330 (time 1788535507.40 total 30.01 last 137.73) total: 4500 /unlink in 31.17 seconds: 144.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 802 (time 1788535524.07 total 10.01 last 80.15) - open/close 1589 (time 1788535534.08 total 20.02 last 78.62) - open/close 2360 (time 1788535544.10 total 30.04 last 76.95) - open/close 3070 (time 1788535554.15 total 40.08 last 70.68) - open/close 3844 (time 1788535564.15 total 50.08 last 77.37) total: 4500 open/close in 59.69 seconds: 75.39 ops/second - /unlink 1437 (time 1788535589.16 total 10.01 last 143.59) - /unlink 2787 (time 1788535599.16 total 20.01 last 134.97) - /unlink 4256 (time 1788535609.16 total 30.01 last 146.89) total: 4500 /unlink in 31.57 seconds: 142.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 771 (time 1788535625.50 total 10.00 last 77.07) - open/close 1455 (time 1788535635.51 total 20.02 last 68.31) - open/close 2167 (time 1788535645.52 total 30.03 last 71.12) - open/close 2912 (time 1788535655.52 total 40.03 last 74.46) - open/close 3636 (time 1788535665.54 total 50.04 last 72.32) - open/close 4391 (time 1788535675.54 total 60.05 last 75.46) total: 4500 open/close in 61.73 seconds: 72.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1293 (time 1788535691.95 total 10.00 last 129.24) - /unlink 2383 (time 1788535701.95 total 20.01 last 108.97) - /unlink 3531 (time 1788535711.95 total 30.01 last 114.77) total: 4500 /unlink in 38.77 seconds: 116.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 712 (time 1788535735.96 total 10.02 last 71.08) - open/close 1499 (time 1788535745.96 total 20.02 last 78.69) - open/close 2183 (time 1788535755.97 total 30.02 last 68.35) - open/close 2810 (time 1788535765.99 total 40.04 last 62.61) - open/close 3444 (time 1788535776.00 total 50.05 last 63.34) - open/close 4106 (time 1788535786.01 total 60.06 last 66.13) total: 4500 open/close in 66.15 seconds: 68.02 ops/second - /unlink 1288 (time 1788535806.87 total 10.00 last 128.76) - /unlink 2501 (time 1788535816.87 total 20.01 last 121.25) - /unlink 3866 (time 1788535826.88 total 30.02 last 136.37) total: 4500 /unlink in 35.04 seconds: 128.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 601 (time 1788535846.53 total 10.01 last 60.05) - open/close 1305 (time 1788535856.53 total 20.01 last 70.37) - open/close 2015 (time 1788535866.54 total 30.01 last 70.98) - open/close 2627 (time 1788535876.55 total 40.03 last 61.11) - open/close 3346 (time 1788535886.55 total 50.03 last 71.90) - open/close 4138 (time 1788535896.55 total 60.03 last 79.20) total: 4500 open/close in 65.35 seconds: 68.86 ops/second - /unlink 1419 (time 1788535916.68 total 10.00 last 141.87) - /unlink 2886 (time 1788535926.68 total 20.00 last 146.70) - /unlink 4283 (time 1788535936.69 total 30.00 last 139.67) total: 4500 /unlink in 31.82 seconds: 141.41 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95256 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg615-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg615-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg615-server stop mds service on oleg615-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg615-server stop mds service on oleg615-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg615-server unloading modules via unload_modules_local on: 'oleg615-server' oleg615-server: oleg615-server.virtnet: executing unload_modules_local oleg615-server: kmemleak disabled modules unloaded.