== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:13:07 (1789045987) Stopping clients: oleg302-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg302-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg302-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg302-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg302-server oleg302-server: oleg302-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing load_modules_local oleg302-server: kmemleak disabled oleg302-server: Loading modules from /home/green/git/lustre-release/lustre oleg302-server: detected 4 online CPUs by sysfs oleg302-server: MODOPTS_LIBCFS= oleg302-server: Force libcfs to create 2 CPU partitions oleg302-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg302-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg302-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg302-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 744 (time 1789046193.08 total 10.01 last 74.35) - open/close 1379 (time 1789046203.08 total 20.01 last 63.48) - open/close 1980 (time 1789046213.09 total 30.01 last 60.09) - open/close 2571 (time 1789046223.09 total 40.02 last 59.08) - open/close 3196 (time 1789046233.09 total 50.02 last 62.49) - open/close 3875 (time 1789046243.09 total 60.02 last 67.88) total: 4500 open/close in 69.20 seconds: 65.03 ops/second - /unlink 1282 (time 1789046266.74 total 10.00 last 128.14) - /unlink 2487 (time 1789046276.74 total 20.01 last 120.47) - /unlink 3836 (time 1789046286.75 total 30.01 last 134.87) total: 4500 /unlink in 35.51 seconds: 126.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 656 (time 1789046306.71 total 10.00 last 65.60) - open/close 1411 (time 1789046316.72 total 20.01 last 75.43) - open/close 2105 (time 1789046326.73 total 30.03 last 69.28) - open/close 2821 (time 1789046336.74 total 40.04 last 71.54) - open/close 3479 (time 1789046346.74 total 50.04 last 65.79) - open/close 4164 (time 1789046356.75 total 60.04 last 68.46) total: 4500 open/close in 64.90 seconds: 69.34 ops/second - /unlink 1270 (time 1789046376.11 total 10.00 last 126.97) - /unlink 2566 (time 1789046386.11 total 20.00 last 129.58) - /unlink 3890 (time 1789046396.11 total 30.01 last 132.38) total: 4500 /unlink in 34.49 seconds: 130.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 761 (time 1789046414.71 total 10.01 last 76.01) - open/close 1438 (time 1789046424.72 total 20.01 last 67.69) - open/close 2189 (time 1789046434.73 total 30.02 last 75.02) - open/close 2905 (time 1789046444.73 total 40.03 last 71.57) - open/close 3553 (time 1789046454.76 total 50.05 last 64.63) - open/close 4202 (time 1789046464.76 total 60.06 last 64.85) total: 4500 open/close in 64.83 seconds: 69.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1312 (time 1789046484.48 total 10.00 last 131.18) - /unlink 2432 (time 1789046494.50 total 20.02 last 111.83) - /unlink 3689 (time 1789046504.51 total 30.02 last 125.60) total: 4500 /unlink in 36.92 seconds: 121.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 625 (time 1789046526.09 total 10.01 last 62.44) - open/close 1289 (time 1789046536.10 total 20.02 last 66.36) - open/close 1955 (time 1789046546.11 total 30.03 last 66.53) - open/close 2610 (time 1789046556.14 total 40.06 last 65.30) - open/close 3208 (time 1789046566.14 total 50.06 last 59.76) - open/close 3936 (time 1789046576.16 total 60.08 last 72.66) total: 4500 open/close in 68.26 seconds: 65.92 ops/second - /unlink 1233 (time 1789046598.58 total 10.00 last 123.26) - /unlink 2489 (time 1789046608.59 total 20.01 last 125.55) - /unlink 3564 (time 1789046618.60 total 30.02 last 107.39) total: 4500 /unlink in 38.55 seconds: 116.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 738 (time 1789046642.92 total 10.00 last 73.79) - open/close 1423 (time 1789046652.92 total 20.01 last 68.46) - open/close 2110 (time 1789046662.92 total 30.01 last 68.69) - open/close 2844 (time 1789046672.92 total 40.01 last 73.40) - open/close 3449 (time 1789046682.96 total 50.05 last 60.27) - open/close 4129 (time 1789046692.97 total 60.05 last 67.97) total: 4500 open/close in 65.78 seconds: 68.41 ops/second - /unlink 1319 (time 1789046712.96 total 10.00 last 131.87) - /unlink 2556 (time 1789046722.98 total 20.02 last 123.53) - /unlink 3910 (time 1789046732.98 total 30.02 last 135.34) total: 4500 /unlink in 34.44 seconds: 130.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 718 (time 1789046752.02 total 10.01 last 71.72) - open/close 1443 (time 1789046762.03 total 20.02 last 72.42) - open/close 2170 (time 1789046772.05 total 30.04 last 72.59) - open/close 2882 (time 1789046782.05 total 40.04 last 71.20) - open/close 3586 (time 1789046792.05 total 50.04 last 70.37) - open/close 4240 (time 1789046802.08 total 60.07 last 65.23) total: 4500 open/close in 64.29 seconds: 70.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1215 (time 1789046820.94 total 10.01 last 121.39) - /unlink 2482 (time 1789046830.94 total 20.01 last 126.70) - /unlink 3716 (time 1789046840.95 total 30.02 last 123.33) total: 4500 /unlink in 36.16 seconds: 124.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 690 (time 1789046860.93 total 10.00 last 68.97) - open/close 1385 (time 1789046870.94 total 20.02 last 69.42) - open/close 2103 (time 1789046880.94 total 30.02 last 71.80) - open/close 2745 (time 1789046890.94 total 40.02 last 64.20) - open/close 3441 (time 1789046900.94 total 50.02 last 69.57) - open/close 4174 (time 1789046910.95 total 60.03 last 73.27) total: 4500 open/close in 64.63 seconds: 69.62 ops/second - /unlink 1301 (time 1789046929.20 total 10.00 last 130.06) - /unlink 2666 (time 1789046939.20 total 20.01 last 136.44) - /unlink 3993 (time 1789046949.21 total 30.01 last 132.62) total: 4500 /unlink in 33.73 seconds: 133.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 710 (time 1789046966.97 total 10.01 last 70.95) - open/close 1393 (time 1789046976.99 total 20.02 last 68.18) - open/close 2063 (time 1789046987.01 total 30.04 last 66.89) - open/close 2719 (time 1789046997.01 total 40.04 last 65.58) - open/close 3283 (time 1789047007.02 total 50.05 last 56.35) - open/close 3922 (time 1789047017.03 total 60.06 last 63.86) total: 4500 open/close in 69.83 seconds: 64.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1292 (time 1789047040.40 total 10.00 last 129.18) - /unlink 2457 (time 1789047050.40 total 20.01 last 116.44) - /unlink 3767 (time 1789047060.41 total 30.01 last 130.95) total: 4500 /unlink in 35.76 seconds: 125.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 746 (time 1789047080.73 total 10.01 last 74.56) - open/close 1504 (time 1789047090.73 total 20.01 last 75.77) - open/close 2127 (time 1789047100.74 total 30.01 last 62.28) - open/close 2744 (time 1789047110.74 total 40.02 last 61.69) - open/close 3423 (time 1789047120.75 total 50.03 last 67.79) - open/close 4054 (time 1789047130.76 total 60.03 last 63.09) total: 4500 open/close in 66.99 seconds: 67.18 ops/second - /unlink 1366 (time 1789047152.70 total 10.00 last 136.57) - /unlink 2749 (time 1789047162.70 total 20.00 last 138.29) - /unlink 3981 (time 1789047172.72 total 30.02 last 123.03) total: 4500 /unlink in 34.13 seconds: 131.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 646 (time 1789047191.87 total 10.02 last 64.46) - open/close 1336 (time 1789047201.88 total 20.03 last 68.94) - open/close 1965 (time 1789047211.89 total 30.04 last 62.83) - open/close 2642 (time 1789047221.89 total 40.04 last 67.70) - open/close 3256 (time 1789047231.90 total 50.05 last 61.37) - open/close 4002 (time 1789047241.90 total 60.05 last 74.59) total: 4500 open/close in 67.06 seconds: 67.11 ops/second - /unlink 1254 (time 1789047263.44 total 10.00 last 125.37) - /unlink 2487 (time 1789047273.44 total 20.00 last 123.29) - /unlink 3780 (time 1789047283.47 total 30.03 last 128.93) total: 4500 /unlink in 35.78 seconds: 125.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 649 (time 1789047304.63 total 10.00 last 64.89) - open/close 1273 (time 1789047314.63 total 20.00 last 62.40) - open/close 1988 (time 1789047324.63 total 30.00 last 71.48) - open/close 2762 (time 1789047334.64 total 40.01 last 77.33) - open/close 3423 (time 1789047344.64 total 50.02 last 66.08) - open/close 4014 (time 1789047354.64 total 60.02 last 59.09) total: 4500 open/close in 68.51 seconds: 65.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1274 (time 1789047378.34 total 10.01 last 127.32) - /unlink 2440 (time 1789047388.34 total 20.01 last 116.55) - /unlink 3720 (time 1789047398.34 total 30.01 last 127.99) total: 4500 /unlink in 35.74 seconds: 125.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 700 (time 1789047418.82 total 10.01 last 69.93) - open/close 1367 (time 1789047428.84 total 20.02 last 66.60) - open/close 2059 (time 1789047438.84 total 30.03 last 69.18) - open/close 2769 (time 1789047448.85 total 40.03 last 70.95) - open/close 3471 (time 1789047458.86 total 50.05 last 70.12) - open/close 4194 (time 1789047468.86 total 60.05 last 72.26) total: 4500 open/close in 64.59 seconds: 69.68 ops/second - /unlink 1342 (time 1789047488.34 total 10.00 last 134.17) - /unlink 2612 (time 1789047498.35 total 20.01 last 126.93) - /unlink 3796 (time 1789047508.35 total 30.01 last 118.38) total: 4500 /unlink in 34.89 seconds: 128.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 683 (time 1789047528.32 total 10.01 last 68.22) - open/close 1319 (time 1789047538.34 total 20.03 last 63.50) - open/close 1917 (time 1789047548.34 total 30.03 last 59.79) - open/close 2640 (time 1789047558.34 total 40.03 last 72.27) - open/close 3297 (time 1789047568.35 total 50.04 last 65.65) - open/close 4036 (time 1789047578.35 total 60.05 last 73.85) total: 4500 open/close in 66.64 seconds: 67.53 ops/second - /unlink 1332 (time 1789047599.06 total 10.00 last 133.18) - /unlink 2637 (time 1789047609.07 total 20.01 last 130.43) - /unlink 3994 (time 1789047619.07 total 30.01 last 135.67) total: 4500 /unlink in 33.68 seconds: 133.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 711 (time 1789047637.60 total 10.01 last 71.03) - open/close 1376 (time 1789047647.61 total 20.02 last 66.42) - open/close 2034 (time 1789047657.61 total 30.02 last 65.79) - open/close 2749 (time 1789047667.62 total 40.03 last 71.47) - open/close 3454 (time 1789047677.63 total 50.04 last 70.42) - open/close 4150 (time 1789047687.64 total 60.05 last 69.53) total: 4500 open/close in 65.58 seconds: 68.62 ops/second - /unlink 1348 (time 1789047708.26 total 10.01 last 134.67) - /unlink 2674 (time 1789047718.27 total 20.01 last 132.56) - /unlink 3971 (time 1789047728.27 total 30.02 last 129.64) total: 4500 /unlink in 34.09 seconds: 132.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 714 (time 1789047747.15 total 10.00 last 71.39) - open/close 1353 (time 1789047757.16 total 20.01 last 63.86) - open/close 2049 (time 1789047767.17 total 30.02 last 69.53) - open/close 2760 (time 1789047777.18 total 40.03 last 71.05) - open/close 3318 (time 1789047787.19 total 50.03 last 55.75) - open/close 3866 (time 1789047797.22 total 60.06 last 54.64) - open/close 4378 (time 1789047807.24 total 70.09 last 51.08) total: 4500 open/close in 72.98 seconds: 61.66 ops/second - /unlink 1181 (time 1789047826.84 total 10.00 last 118.09) - /unlink 2492 (time 1789047836.84 total 20.00 last 131.10) - /unlink 3831 (time 1789047846.84 total 30.00 last 133.87) total: 4500 /unlink in 35.42 seconds: 127.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 692 (time 1789047866.84 total 10.00 last 69.20) - open/close 1330 (time 1789047876.85 total 20.00 last 63.77) - open/close 2050 (time 1789047886.85 total 30.01 last 71.97) - open/close 2714 (time 1789047896.86 total 40.02 last 66.31) - open/close 3397 (time 1789047906.87 total 50.03 last 68.26) - open/close 4045 (time 1789047916.88 total 60.04 last 64.74) total: 4500 open/close in 66.96 seconds: 67.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1183 (time 1789047938.96 total 10.01 last 118.17) - /unlink 2339 (time 1789047948.96 total 20.01 last 115.59) - /unlink 3618 (time 1789047958.97 total 30.02 last 127.82) total: 4500 /unlink in 37.05 seconds: 121.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 632 (time 1789047980.08 total 10.01 last 63.12) - open/close 1298 (time 1789047990.09 total 20.02 last 66.57) - open/close 1975 (time 1789048000.12 total 30.05 last 67.50) - open/close 2562 (time 1789048010.12 total 40.05 last 58.69) - open/close 3158 (time 1789048020.12 total 50.05 last 59.57) - open/close 3944 (time 1789048030.13 total 60.06 last 78.55) total: 4500 open/close in 69.32 seconds: 64.92 ops/second - /unlink 1358 (time 1789048054.29 total 10.01 last 135.65) - /unlink 2658 (time 1789048064.30 total 20.01 last 129.96) - /unlink 3932 (time 1789048074.30 total 30.01 last 127.40) total: 4500 /unlink in 34.33 seconds: 131.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 709 (time 1789048093.60 total 10.01 last 70.86) - open/close 1388 (time 1789048103.60 total 20.01 last 67.87) - open/close 2134 (time 1789048113.61 total 30.01 last 74.58) - open/close 2888 (time 1789048123.62 total 40.03 last 75.27) - open/close 3620 (time 1789048133.63 total 50.04 last 73.11) - open/close 4314 (time 1789048143.64 total 60.05 last 69.37) total: 4500 open/close in 62.95 seconds: 71.49 ops/second - /unlink 1386 (time 1789048160.08 total 10.01 last 138.52) - /unlink 2708 (time 1789048170.09 total 20.01 last 132.17) - /unlink 4063 (time 1789048180.09 total 30.01 last 135.46) total: 4500 /unlink in 33.64 seconds: 133.78 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95274 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg302-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg302-server unloading modules via unload_modules_local on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing unload_modules_local oleg302-server: kmemleak disabled modules unloaded.