== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:54:16 (1788594856) Stopping clients: oleg228-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg228-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg228-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg228-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg228-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg228-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg228-server oleg228-server: oleg228-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: kmemleak disabled oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions oleg228-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg228-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg228-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg228-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 629 (time 1788595077.38 total 10.01 last 62.85) - open/close 1170 (time 1788595087.38 total 20.01 last 54.08) - open/close 1802 (time 1788595097.38 total 30.01 last 63.18) - open/close 2448 (time 1788595107.39 total 40.02 last 64.54) - open/close 3231 (time 1788595117.40 total 50.03 last 78.26) - open/close 3880 (time 1788595127.40 total 60.03 last 64.89) total: 4500 open/close in 69.24 seconds: 64.99 ops/second - /unlink 1414 (time 1788595151.90 total 10.00 last 141.39) - /unlink 2657 (time 1788595161.90 total 20.00 last 124.27) - /unlink 3844 (time 1788595171.91 total 30.01 last 118.66) total: 4500 /unlink in 35.27 seconds: 127.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 563 (time 1788595192.13 total 10.01 last 56.26) - open/close 1253 (time 1788595202.13 total 20.01 last 68.94) - open/close 1922 (time 1788595212.15 total 30.03 last 66.81) - open/close 2719 (time 1788595222.16 total 40.04 last 79.63) - open/close 3387 (time 1788595232.17 total 50.05 last 66.69) - open/close 4160 (time 1788595242.19 total 60.07 last 77.20) total: 4500 open/close in 65.03 seconds: 69.20 ops/second - /unlink 1350 (time 1788595262.53 total 10.00 last 134.96) - /unlink 2653 (time 1788595272.53 total 20.00 last 130.29) - /unlink 3816 (time 1788595282.53 total 30.01 last 116.24) total: 4500 /unlink in 37.05 seconds: 121.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 644 (time 1788595306.51 total 10.05 last 64.11) - open/close 1253 (time 1788595316.52 total 20.05 last 60.85) - open/close 1903 (time 1788595326.54 total 30.08 last 64.86) - open/close 2634 (time 1788595336.56 total 40.09 last 73.01) - open/close 3374 (time 1788595346.56 total 50.09 last 73.99) - open/close 4050 (time 1788595356.56 total 60.09 last 67.59) total: 4500 open/close in 66.94 seconds: 67.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1114 (time 1788595379.02 total 10.00 last 111.38) - /unlink 2488 (time 1788595389.02 total 20.01 last 137.35) - /unlink 3475 (time 1788595399.03 total 30.01 last 98.62) total: 4500 /unlink in 36.99 seconds: 121.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 706 (time 1788595421.12 total 10.02 last 70.48) - open/close 1371 (time 1788595431.12 total 20.02 last 66.46) - open/close 2040 (time 1788595441.13 total 30.03 last 66.88) - open/close 2805 (time 1788595451.13 total 40.03 last 76.47) - open/close 3475 (time 1788595461.13 total 50.03 last 66.98) - open/close 4218 (time 1788595471.14 total 60.04 last 74.26) total: 4500 open/close in 63.52 seconds: 70.85 ops/second - /unlink 1433 (time 1788595489.35 total 10.00 last 143.28) - /unlink 2731 (time 1788595499.35 total 20.00 last 129.79) - /unlink 4035 (time 1788595509.35 total 30.01 last 130.35) total: 4500 /unlink in 33.54 seconds: 134.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 725 (time 1788595527.66 total 10.02 last 72.34) - open/close 1515 (time 1788595537.66 total 20.02 last 78.99) - open/close 2044 (time 1788595547.67 total 30.04 last 52.82) - open/close 2808 (time 1788595557.67 total 40.04 last 76.39) - open/close 3482 (time 1788595567.70 total 50.06 last 67.24) - open/close 4132 (time 1788595577.71 total 60.08 last 64.92) total: 4500 open/close in 66.53 seconds: 67.64 ops/second - /unlink 1355 (time 1788595599.53 total 10.00 last 135.47) - /unlink 2570 (time 1788595609.53 total 20.01 last 121.44) - /unlink 3760 (time 1788595619.54 total 30.01 last 118.94) total: 4500 /unlink in 36.48 seconds: 123.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 618 (time 1788595641.06 total 10.00 last 61.79) - open/close 1217 (time 1788595651.07 total 20.00 last 59.89) - open/close 1777 (time 1788595661.08 total 30.02 last 55.92) - open/close 2413 (time 1788595671.11 total 40.05 last 63.41) - open/close 3069 (time 1788595681.11 total 50.05 last 65.60) - open/close 3729 (time 1788595691.12 total 60.06 last 65.95) - open/close 4379 (time 1788595701.12 total 70.06 last 64.97) total: 4500 open/close in 71.50 seconds: 62.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1245 (time 1788595717.29 total 10.00 last 124.50) - /unlink 2396 (time 1788595727.30 total 20.01 last 115.02) - /unlink 3641 (time 1788595737.30 total 30.01 last 124.47) total: 4500 /unlink in 36.85 seconds: 122.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 585 (time 1788595759.12 total 10.01 last 58.42) - open/close 1178 (time 1788595769.13 total 20.02 last 59.27) - open/close 1848 (time 1788595779.13 total 30.02 last 66.98) - open/close 2477 (time 1788595789.13 total 40.02 last 62.90) - open/close 3083 (time 1788595799.15 total 50.04 last 60.50) - open/close 3820 (time 1788595809.15 total 60.04 last 73.68) total: 4500 open/close in 68.89 seconds: 65.32 ops/second - /unlink 1464 (time 1788595833.11 total 10.00 last 146.38) - /unlink 2886 (time 1788595843.12 total 20.00 last 142.16) - /unlink 4435 (time 1788595853.12 total 30.01 last 154.83) total: 4500 /unlink in 30.59 seconds: 147.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 710 (time 1788595869.48 total 10.00 last 70.98) - open/close 1247 (time 1788595879.48 total 20.01 last 53.66) - open/close 1861 (time 1788595889.49 total 30.02 last 61.35) - open/close 2493 (time 1788595899.50 total 40.02 last 63.17) - open/close 3089 (time 1788595909.52 total 50.04 last 59.47) - open/close 3857 (time 1788595919.52 total 60.05 last 76.76) total: 4500 open/close in 69.05 seconds: 65.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1389 (time 1788595942.88 total 10.00 last 138.86) - /unlink 2710 (time 1788595952.89 total 20.01 last 132.02) - /unlink 4037 (time 1788595962.89 total 30.02 last 132.59) total: 4500 /unlink in 34.17 seconds: 131.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 708 (time 1788595981.58 total 10.01 last 70.76) - open/close 1356 (time 1788595991.58 total 20.01 last 64.78) - open/close 2223 (time 1788596001.58 total 30.01 last 86.67) - open/close 2963 (time 1788596011.59 total 40.01 last 73.99) - open/close 3810 (time 1788596021.60 total 50.03 last 84.60) - open/close 4484 (time 1788596031.61 total 60.04 last 67.33) total: 4500 open/close in 60.26 seconds: 74.68 ops/second - /unlink 1403 (time 1788596047.17 total 10.00 last 140.26) - /unlink 2706 (time 1788596057.18 total 20.00 last 130.28) - /unlink 3847 (time 1788596067.19 total 30.01 last 113.98) total: 4500 /unlink in 34.87 seconds: 129.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 753 (time 1788596087.84 total 10.03 last 75.08) - open/close 1395 (time 1788596097.86 total 20.04 last 64.10) - open/close 2223 (time 1788596107.86 total 30.05 last 82.77) - open/close 3015 (time 1788596117.86 total 40.05 last 79.19) - open/close 3765 (time 1788596127.87 total 50.05 last 74.99) total: 4500 open/close in 58.94 seconds: 76.35 ops/second - /unlink 1661 (time 1788596151.49 total 10.01 last 165.95) - /unlink 3229 (time 1788596161.50 total 20.01 last 156.77) total: 4500 /unlink in 28.00 seconds: 160.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 791 (time 1788596184.44 total 10.01 last 78.99) - open/close 1371 (time 1788596194.45 total 20.02 last 57.96) - open/close 2153 (time 1788596204.46 total 30.03 last 78.17) - open/close 2895 (time 1788596214.47 total 40.04 last 74.11) - open/close 3554 (time 1788596224.48 total 50.05 last 65.84) - open/close 4460 (time 1788596234.48 total 60.05 last 90.54) total: 4500 open/close in 60.40 seconds: 74.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1464 (time 1788596249.89 total 10.01 last 146.31) - /unlink 2789 (time 1788596259.90 total 20.02 last 132.37) - /unlink 4176 (time 1788596269.91 total 30.02 last 138.62) total: 4500 /unlink in 32.36 seconds: 139.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 819 (time 1788596286.98 total 10.00 last 81.89) - open/close 1657 (time 1788596296.99 total 20.01 last 83.75) - open/close 2444 (time 1788596307.00 total 30.02 last 78.62) - open/close 3082 (time 1788596317.01 total 40.03 last 63.71) - open/close 3761 (time 1788596327.02 total 50.03 last 67.87) - open/close 4355 (time 1788596337.03 total 60.05 last 59.33) total: 4500 open/close in 62.71 seconds: 71.76 ops/second - /unlink 1706 (time 1788596354.25 total 10.00 last 170.55) - /unlink 3402 (time 1788596364.25 total 20.01 last 169.56) total: 4500 /unlink in 26.24 seconds: 171.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 859 (time 1788596385.22 total 10.01 last 85.85) - open/close 1796 (time 1788596395.22 total 20.01 last 93.66) - open/close 2568 (time 1788596405.23 total 30.01 last 77.17) - open/close 3416 (time 1788596415.23 total 40.02 last 84.76) - open/close 4279 (time 1788596425.23 total 50.02 last 86.28) total: 4500 open/close in 52.52 seconds: 85.67 ops/second - /unlink 1840 (time 1788596442.13 total 10.00 last 183.99) - /unlink 3353 (time 1788596452.14 total 20.00 last 151.26) total: 4500 /unlink in 26.31 seconds: 171.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 836 (time 1788596472.49 total 10.00 last 83.56) - open/close 1718 (time 1788596482.51 total 20.02 last 88.07) - open/close 2460 (time 1788596492.52 total 30.03 last 74.12) - open/close 3128 (time 1788596502.53 total 40.04 last 66.72) - open/close 3880 (time 1788596512.54 total 50.05 last 75.17) total: 4500 open/close in 57.93 seconds: 77.68 ops/second - /unlink 1391 (time 1788596534.63 total 10.00 last 139.04) - /unlink 2935 (time 1788596544.63 total 20.01 last 154.39) - /unlink 4357 (time 1788596554.64 total 30.01 last 142.18) total: 4500 /unlink in 30.74 seconds: 146.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 857 (time 1788596571.01 total 10.01 last 85.62) - open/close 1758 (time 1788596581.01 total 20.01 last 90.07) - open/close 2489 (time 1788596591.02 total 30.02 last 73.02) - open/close 3341 (time 1788596601.02 total 40.03 last 85.17) - open/close 4214 (time 1788596611.03 total 50.03 last 87.28) total: 4500 open/close in 53.45 seconds: 84.19 ops/second - /unlink 1642 (time 1788596628.75 total 10.00 last 164.16) - /unlink 3106 (time 1788596638.75 total 20.00 last 146.39) total: 4500 /unlink in 28.99 seconds: 155.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 815 (time 1788596662.52 total 10.01 last 81.45) - open/close 1580 (time 1788596672.53 total 20.01 last 76.46) - open/close 2443 (time 1788596682.53 total 30.01 last 86.27) - open/close 3399 (time 1788596692.53 total 40.01 last 95.60) - open/close 4301 (time 1788596702.54 total 50.02 last 90.14) total: 4500 open/close in 53.12 seconds: 84.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1606 (time 1788596720.96 total 10.00 last 160.57) - /unlink 2806 (time 1788596730.96 total 20.00 last 119.99) - /unlink 4160 (time 1788596740.98 total 30.02 last 135.15) total: 4500 /unlink in 32.17 seconds: 139.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 876 (time 1788596758.12 total 10.01 last 87.54) - open/close 1848 (time 1788596768.13 total 20.02 last 97.11) - open/close 2598 (time 1788596778.13 total 30.02 last 74.97) - open/close 3324 (time 1788596788.16 total 40.04 last 72.45) - open/close 3956 (time 1788596798.16 total 50.04 last 63.19) total: 4500 open/close in 58.86 seconds: 76.45 ops/second - /unlink 1783 (time 1788596821.91 total 10.00 last 178.28) - /unlink 3356 (time 1788596831.92 total 20.00 last 157.24) total: 4500 /unlink in 25.87 seconds: 173.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 772 (time 1788596852.24 total 10.00 last 77.18) - open/close 1613 (time 1788596862.27 total 20.04 last 83.82) - open/close 2571 (time 1788596872.27 total 30.04 last 95.76) - open/close 3536 (time 1788596882.30 total 40.06 last 96.29) - open/close 4476 (time 1788596892.30 total 50.06 last 94.00) total: 4500 open/close in 50.34 seconds: 89.39 ops/second - /unlink 1509 (time 1788596906.98 total 10.00 last 150.89) - /unlink 2859 (time 1788596916.98 total 20.00 last 134.96) - /unlink 4273 (time 1788596926.98 total 30.00 last 141.40) total: 4500 /unlink in 31.84 seconds: 141.35 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95289 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg228-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg228-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg228-server stop mds service on oleg228-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg228-server stop mds service on oleg228-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg228-server unloading modules via unload_modules_local on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing unload_modules_local oleg228-server: kmemleak disabled modules unloaded.