== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:01:20 (1788624080) Stopping clients: oleg601-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg601-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg601-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg601-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg601-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg601-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg601-server oleg601-server: oleg601-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg601-server' oleg601-server: oleg601-server.virtnet: executing load_modules_local oleg601-server: kmemleak disabled oleg601-server: Loading modules from /home/green/git/lustre-release/lustre oleg601-server: detected 4 online CPUs by sysfs oleg601-server: MODOPTS_LIBCFS= oleg601-server: Force libcfs to create 2 CPU partitions oleg601-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg601-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg601-server: oleg601-server.virtnet: executing set_default_debug -1 all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg601-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg601-server: oleg601-server.virtnet: executing set_default_debug -1 all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg601-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug -1 all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 962 (time 1788624274.10 total 10.01 last 96.12) - open/close 1874 (time 1788624284.11 total 20.01 last 91.14) - open/close 2820 (time 1788624294.11 total 30.02 last 94.57) - open/close 3835 (time 1788624304.12 total 40.02 last 101.47) total: 4500 open/close in 47.61 seconds: 94.52 ops/second - /unlink 1983 (time 1788624326.59 total 10.00 last 198.22) - /unlink 3491 (time 1788624336.59 total 20.00 last 150.79) total: 4500 /unlink in 26.41 seconds: 170.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 639 (time 1788624357.15 total 10.01 last 63.83) - open/close 1485 (time 1788624367.15 total 20.02 last 84.56) - open/close 2501 (time 1788624377.15 total 30.02 last 101.58) - open/close 3392 (time 1788624387.15 total 40.02 last 89.09) - open/close 4260 (time 1788624397.16 total 50.02 last 86.76) total: 4500 open/close in 52.98 seconds: 84.93 ops/second - /unlink 1739 (time 1788624415.04 total 10.01 last 173.80) - /unlink 3557 (time 1788624425.04 total 20.01 last 181.78) total: 4500 /unlink in 25.47 seconds: 176.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 922 (time 1788624444.28 total 10.01 last 92.14) - open/close 1897 (time 1788624454.28 total 20.01 last 97.47) - open/close 2832 (time 1788624464.28 total 30.01 last 93.46) - open/close 3667 (time 1788624474.29 total 40.02 last 83.45) total: 4500 open/close in 49.78 seconds: 90.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1711 (time 1788624497.72 total 10.00 last 171.02) - /unlink 3063 (time 1788624507.73 total 20.01 last 135.12) total: 4500 /unlink in 28.28 seconds: 159.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 841 (time 1788624530.42 total 10.02 last 83.97) - open/close 1744 (time 1788624540.42 total 20.02 last 90.29) - open/close 2638 (time 1788624550.42 total 30.02 last 89.33) - open/close 3486 (time 1788624560.43 total 40.03 last 84.76) - open/close 4380 (time 1788624570.44 total 50.04 last 89.33) total: 4500 open/close in 51.17 seconds: 87.95 ops/second - /unlink 1611 (time 1788624585.66 total 10.00 last 161.07) - /unlink 3081 (time 1788624595.66 total 20.00 last 146.98) total: 4500 /unlink in 29.31 seconds: 153.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 749 (time 1788624619.39 total 10.02 last 74.76) - open/close 1550 (time 1788624629.39 total 20.03 last 80.05) - open/close 2471 (time 1788624639.39 total 30.03 last 92.09) - open/close 3281 (time 1788624649.40 total 40.03 last 80.95) - open/close 3989 (time 1788624659.41 total 50.04 last 70.76) total: 4500 open/close in 55.55 seconds: 81.00 ops/second - /unlink 1938 (time 1788624679.39 total 10.00 last 193.78) - /unlink 3605 (time 1788624689.40 total 20.01 last 166.60) total: 4500 /unlink in 25.44 seconds: 176.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 908 (time 1788624708.69 total 10.01 last 90.74) - open/close 1798 (time 1788624718.70 total 20.01 last 88.96) - open/close 2722 (time 1788624728.71 total 30.02 last 92.30) - open/close 3568 (time 1788624738.71 total 40.03 last 84.55) - open/close 4491 (time 1788624748.71 total 50.03 last 92.30) total: 4500 open/close in 50.10 seconds: 89.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1862 (time 1788624762.51 total 10.01 last 186.05) - /unlink 3489 (time 1788624772.52 total 20.01 last 162.67) total: 4500 /unlink in 26.64 seconds: 168.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 950 (time 1788624793.88 total 10.03 last 94.68) - open/close 1830 (time 1788624803.89 total 20.04 last 87.90) - open/close 2758 (time 1788624813.90 total 30.05 last 92.71) - open/close 3715 (time 1788624823.90 total 40.06 last 95.69) total: 4500 open/close in 48.55 seconds: 92.70 ops/second - /unlink 1564 (time 1788624846.53 total 10.00 last 156.35) - /unlink 3201 (time 1788624856.53 total 20.01 last 163.65) total: 4500 /unlink in 28.02 seconds: 160.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 939 (time 1788624879.19 total 10.00 last 93.86) - open/close 1820 (time 1788624889.20 total 20.01 last 88.05) - open/close 2687 (time 1788624899.20 total 30.01 last 86.70) - open/close 3551 (time 1788624909.20 total 40.01 last 86.39) - open/close 4407 (time 1788624919.20 total 50.01 last 85.58) total: 4500 open/close in 51.12 seconds: 88.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1462 (time 1788624934.64 total 10.00 last 146.15) - /unlink 2958 (time 1788624944.64 total 20.01 last 149.56) total: 4500 /unlink in 29.51 seconds: 152.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 865 (time 1788624968.53 total 10.00 last 86.48) - open/close 1702 (time 1788624978.53 total 20.01 last 83.67) - open/close 2467 (time 1788624988.54 total 30.01 last 76.47) - open/close 3270 (time 1788624998.54 total 40.01 last 80.28) - open/close 4080 (time 1788625008.54 total 50.02 last 80.97) total: 4500 open/close in 54.70 seconds: 82.26 ops/second - /unlink 1387 (time 1788625028.57 total 10.02 last 138.48) - /unlink 2884 (time 1788625038.57 total 20.02 last 149.69) total: 4500 /unlink in 29.85 seconds: 150.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 907 (time 1788625062.16 total 10.01 last 90.60) - open/close 1822 (time 1788625072.16 total 20.01 last 91.48) - open/close 2673 (time 1788625082.18 total 30.03 last 84.93) - open/close 3561 (time 1788625092.18 total 40.04 last 88.77) - open/close 4442 (time 1788625102.18 total 50.04 last 88.10) total: 4500 open/close in 50.61 seconds: 88.92 ops/second - /unlink 1649 (time 1788625116.72 total 10.00 last 164.83) - /unlink 3377 (time 1788625126.73 total 20.01 last 172.62) total: 4500 /unlink in 27.28 seconds: 164.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 763 (time 1788625149.26 total 10.00 last 76.27) - open/close 1588 (time 1788625159.27 total 20.02 last 82.40) - open/close 2401 (time 1788625169.28 total 30.03 last 81.20) - open/close 3257 (time 1788625179.29 total 40.04 last 85.49) - open/close 4198 (time 1788625189.31 total 50.06 last 93.96) total: 4500 open/close in 53.06 seconds: 84.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1730 (time 1788625206.30 total 10.00 last 172.97) - /unlink 3395 (time 1788625216.31 total 20.01 last 166.35) total: 4500 /unlink in 26.58 seconds: 169.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 969 (time 1788625236.96 total 10.01 last 96.83) - open/close 1853 (time 1788625246.98 total 20.03 last 88.23) - open/close 2662 (time 1788625256.98 total 30.03 last 80.86) - open/close 3484 (time 1788625266.99 total 40.04 last 82.17) - open/close 4370 (time 1788625277.00 total 50.04 last 88.54) total: 4500 open/close in 51.36 seconds: 87.62 ops/second - /unlink 1731 (time 1788625292.40 total 10.00 last 173.06) - /unlink 3332 (time 1788625302.40 total 20.01 last 160.03) total: 4500 /unlink in 26.96 seconds: 166.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 750 (time 1788625324.08 total 10.00 last 74.98) - open/close 1672 (time 1788625334.09 total 20.01 last 92.14) - open/close 2543 (time 1788625344.10 total 30.01 last 87.05) - open/close 3566 (time 1788625354.11 total 40.03 last 102.17) - open/close 4477 (time 1788625364.12 total 50.03 last 91.03) total: 4500 open/close in 50.23 seconds: 89.58 ops/second - /unlink 1583 (time 1788625378.60 total 10.01 last 158.12) - /unlink 3298 (time 1788625388.60 total 20.01 last 171.49) total: 4500 /unlink in 27.33 seconds: 164.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 909 (time 1788625410.57 total 10.01 last 90.81) - open/close 1857 (time 1788625420.58 total 20.02 last 94.75) - open/close 2751 (time 1788625430.58 total 30.02 last 89.33) - open/close 3677 (time 1788625440.59 total 40.03 last 92.53) total: 4500 open/close in 48.73 seconds: 92.34 ops/second - /unlink 1780 (time 1788625462.70 total 10.00 last 177.92) - /unlink 3313 (time 1788625472.70 total 20.01 last 153.23) total: 4500 /unlink in 27.73 seconds: 162.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 902 (time 1788625495.06 total 10.00 last 90.18) - open/close 1786 (time 1788625505.06 total 20.01 last 88.36) - open/close 2680 (time 1788625515.06 total 30.01 last 89.39) - open/close 3553 (time 1788625525.06 total 40.01 last 87.29) - open/close 4481 (time 1788625535.07 total 50.02 last 92.72) total: 4500 open/close in 50.22 seconds: 89.61 ops/second - /unlink 1712 (time 1788625549.57 total 10.01 last 171.11) - /unlink 3259 (time 1788625559.58 total 20.01 last 154.66) total: 4500 /unlink in 27.90 seconds: 161.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 802 (time 1788625581.70 total 10.01 last 80.16) - open/close 1615 (time 1788625591.71 total 20.01 last 81.24) - open/close 2459 (time 1788625601.72 total 30.02 last 84.34) - open/close 3318 (time 1788625611.72 total 40.03 last 85.83) - open/close 4140 (time 1788625621.74 total 50.04 last 82.07) total: 4500 open/close in 55.07 seconds: 81.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1412 (time 1788625640.90 total 10.00 last 141.14) - /unlink 2988 (time 1788625650.90 total 20.00 last 157.59) total: 4500 /unlink in 29.90 seconds: 150.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 831 (time 1788625675.44 total 10.01 last 83.03) - open/close 1721 (time 1788625685.45 total 20.02 last 88.93) - open/close 2649 (time 1788625695.45 total 30.02 last 92.79) - open/close 3582 (time 1788625705.45 total 40.02 last 93.28) - open/close 4336 (time 1788625715.45 total 50.02 last 75.40) total: 4500 open/close in 52.25 seconds: 86.12 ops/second - /unlink 1514 (time 1788625731.68 total 10.01 last 151.29) - /unlink 2935 (time 1788625741.68 total 20.01 last 142.08) - /unlink 4375 (time 1788625751.68 total 30.01 last 143.95) total: 4500 /unlink in 31.01 seconds: 145.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 885 (time 1788625768.63 total 10.00 last 88.48) - open/close 1578 (time 1788625778.64 total 20.01 last 69.25) - open/close 2339 (time 1788625788.64 total 30.01 last 76.06) - open/close 3107 (time 1788625798.65 total 40.02 last 76.75) - open/close 3916 (time 1788625808.66 total 50.03 last 80.83) total: 4500 open/close in 57.24 seconds: 78.62 ops/second - /unlink 1531 (time 1788625829.89 total 10.00 last 153.06) - /unlink 3082 (time 1788625839.91 total 20.02 last 154.83) total: 4500 /unlink in 28.90 seconds: 155.71 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95149 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg601-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg601-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg601-server stop mds service on oleg601-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg601-server stop mds service on oleg601-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg601-server unloading modules via unload_modules_local on: 'oleg601-server' oleg601-server: oleg601-server.virtnet: executing unload_modules_local oleg601-server: kmemleak disabled modules unloaded.