== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:34:48 (1788514488) Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg403-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg403-server oleg403-server: oleg403-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: kmemleak disabled oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg403-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg403-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg403-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 530 (time 1788514703.92 total 10.01 last 52.93) - open/close 1013 (time 1788514713.93 total 20.03 last 48.24) - open/close 1469 (time 1788514723.94 total 30.04 last 45.53) - open/close 2016 (time 1788514733.95 total 40.05 last 54.67) - open/close 2608 (time 1788514743.96 total 50.06 last 59.12) - open/close 3230 (time 1788514753.96 total 60.06 last 62.20) - open/close 3856 (time 1788514763.97 total 70.07 last 62.56) - open/close 4423 (time 1788514773.98 total 80.08 last 56.63) total: 4500 open/close in 81.54 seconds: 55.19 ops/second - /unlink 1110 (time 1788514790.91 total 10.00 last 110.99) - /unlink 2121 (time 1788514800.93 total 20.02 last 100.90) - /unlink 3205 (time 1788514810.93 total 30.02 last 108.40) - /unlink 4368 (time 1788514820.93 total 40.02 last 116.29) total: 4500 /unlink in 41.33 seconds: 108.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 593 (time 1788514837.33 total 10.01 last 59.24) - open/close 1136 (time 1788514847.33 total 20.01 last 54.29) - open/close 1706 (time 1788514857.35 total 30.03 last 56.92) - open/close 2259 (time 1788514867.35 total 40.03 last 55.30) - open/close 2850 (time 1788514877.35 total 50.03 last 59.09) - open/close 3395 (time 1788514887.36 total 60.04 last 54.45) - open/close 3969 (time 1788514897.36 total 70.04 last 57.38) total: 4500 open/close in 78.81 seconds: 57.10 ops/second - /unlink 1128 (time 1788514921.05 total 10.00 last 112.78) - /unlink 2278 (time 1788514931.05 total 20.01 last 114.95) - /unlink 3453 (time 1788514941.06 total 30.01 last 117.42) total: 4500 /unlink in 39.93 seconds: 112.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 599 (time 1788514965.95 total 10.01 last 59.86) - open/close 1184 (time 1788514975.96 total 20.02 last 58.45) - open/close 1822 (time 1788514985.96 total 30.02 last 63.79) - open/close 2576 (time 1788514995.96 total 40.02 last 75.39) - open/close 3143 (time 1788515005.97 total 50.03 last 56.65) - open/close 3775 (time 1788515015.98 total 60.04 last 63.14) - open/close 4254 (time 1788515025.99 total 70.05 last 47.83) total: 4500 open/close in 74.54 seconds: 60.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 881 (time 1788515046.33 total 10.00 last 88.07) - /unlink 2127 (time 1788515056.34 total 20.01 last 124.52) - /unlink 3354 (time 1788515066.35 total 30.02 last 122.57) total: 4500 /unlink in 39.36 seconds: 114.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 637 (time 1788515090.36 total 10.01 last 63.62) - open/close 1286 (time 1788515100.37 total 20.01 last 64.89) - open/close 1869 (time 1788515110.37 total 30.01 last 58.29) - open/close 2441 (time 1788515120.37 total 40.02 last 57.19) - open/close 3032 (time 1788515130.39 total 50.03 last 58.99) - open/close 3653 (time 1788515140.41 total 60.05 last 61.99) - open/close 4318 (time 1788515150.41 total 70.06 last 66.45) total: 4500 open/close in 72.83 seconds: 61.79 ops/second - /unlink 1219 (time 1788515168.18 total 10.00 last 121.85) - /unlink 2471 (time 1788515178.19 total 20.01 last 125.17) - /unlink 3728 (time 1788515188.19 total 30.01 last 125.64) total: 4500 /unlink in 36.85 seconds: 122.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 529 (time 1788515210.40 total 10.02 last 52.79) - open/close 1077 (time 1788515220.41 total 20.02 last 54.79) - open/close 1620 (time 1788515230.41 total 30.03 last 54.27) - open/close 2256 (time 1788515240.43 total 40.05 last 63.47) - open/close 2882 (time 1788515250.44 total 50.05 last 62.57) - open/close 3557 (time 1788515260.44 total 60.06 last 67.45) - open/close 4258 (time 1788515270.46 total 70.08 last 69.98) total: 4500 open/close in 73.38 seconds: 61.32 ops/second - /unlink 1199 (time 1788515288.67 total 10.00 last 119.88) - /unlink 2377 (time 1788515298.67 total 20.00 last 117.76) - /unlink 3520 (time 1788515308.69 total 30.02 last 114.17) total: 4500 /unlink in 38.57 seconds: 116.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 501 (time 1788515332.98 total 10.03 last 49.96) - open/close 1058 (time 1788515342.99 total 20.04 last 55.65) - open/close 1568 (time 1788515353.01 total 30.05 last 50.93) - open/close 2215 (time 1788515363.01 total 40.05 last 64.68) - open/close 2880 (time 1788515373.02 total 50.07 last 66.41) - open/close 3530 (time 1788515383.05 total 60.09 last 64.86) - open/close 4225 (time 1788515393.06 total 70.10 last 69.43) total: 4500 open/close in 74.12 seconds: 60.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1250 (time 1788515412.21 total 10.00 last 124.96) - /unlink 2477 (time 1788515422.22 total 20.01 last 122.65) - /unlink 3602 (time 1788515432.23 total 30.02 last 112.40) total: 4500 /unlink in 38.02 seconds: 118.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 543 (time 1788515456.00 total 10.02 last 54.20) - open/close 1135 (time 1788515466.01 total 20.03 last 59.14) - open/close 1724 (time 1788515476.02 total 30.04 last 58.85) - open/close 2394 (time 1788515486.03 total 40.05 last 66.92) - open/close 3022 (time 1788515496.04 total 50.06 last 62.75) - open/close 3706 (time 1788515506.04 total 60.06 last 68.37) - open/close 4380 (time 1788515516.05 total 70.06 last 67.38) total: 4500 open/close in 71.59 seconds: 62.86 ops/second - /unlink 1404 (time 1788515532.14 total 10.00 last 140.39) - /unlink 2571 (time 1788515542.15 total 20.01 last 116.60) - /unlink 3519 (time 1788515552.16 total 30.01 last 94.76) - /unlink 4376 (time 1788515562.16 total 40.02 last 85.68) total: 4500 /unlink in 41.12 seconds: 109.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 576 (time 1788515580.05 total 10.01 last 57.52) - open/close 1160 (time 1788515590.05 total 20.01 last 58.40) - open/close 1750 (time 1788515600.05 total 30.01 last 59.00) - open/close 2408 (time 1788515610.07 total 40.03 last 65.67) - open/close 3011 (time 1788515620.08 total 50.04 last 60.25) - open/close 3647 (time 1788515630.08 total 60.04 last 63.59) - open/close 4316 (time 1788515640.10 total 70.07 last 66.76) total: 4500 open/close in 73.36 seconds: 61.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1273 (time 1788515658.65 total 10.00 last 127.24) - /unlink 2413 (time 1788515668.66 total 20.01 last 113.91) - /unlink 3531 (time 1788515678.66 total 30.02 last 111.76) total: 4500 /unlink in 38.36 seconds: 117.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 618 (time 1788515702.88 total 10.01 last 61.74) - open/close 1324 (time 1788515712.89 total 20.02 last 70.52) - open/close 1998 (time 1788515722.90 total 30.03 last 67.36) - open/close 2768 (time 1788515732.91 total 40.04 last 76.92) - open/close 3430 (time 1788515742.93 total 50.06 last 66.07) - open/close 4062 (time 1788515752.95 total 60.08 last 63.08) total: 4500 open/close in 66.10 seconds: 68.07 ops/second - /unlink 1223 (time 1788515773.68 total 10.00 last 122.24) - /unlink 2551 (time 1788515783.68 total 20.01 last 132.78) - /unlink 3753 (time 1788515793.68 total 30.01 last 120.16) total: 4500 /unlink in 36.36 seconds: 123.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 762 (time 1788515815.21 total 10.00 last 76.17) - open/close 1484 (time 1788515825.21 total 20.01 last 72.18) - open/close 2210 (time 1788515835.25 total 30.05 last 72.30) - open/close 2839 (time 1788515845.27 total 40.07 last 62.79) - open/close 3561 (time 1788515855.28 total 50.07 last 72.16) - open/close 4278 (time 1788515865.28 total 60.07 last 71.68) total: 4500 open/close in 62.99 seconds: 71.44 ops/second - /unlink 1302 (time 1788515883.15 total 10.00 last 130.18) - /unlink 2600 (time 1788515893.15 total 20.00 last 129.77) - /unlink 3875 (time 1788515903.15 total 30.01 last 127.46) total: 4500 /unlink in 34.53 seconds: 130.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 617 (time 1788515922.62 total 10.01 last 61.64) - open/close 1184 (time 1788515932.65 total 20.04 last 56.54) - open/close 1894 (time 1788515942.65 total 30.04 last 70.98) - open/close 2608 (time 1788515952.66 total 40.05 last 71.34) - open/close 3314 (time 1788515962.67 total 50.05 last 70.57) - open/close 4003 (time 1788515972.68 total 60.06 last 68.83) total: 4500 open/close in 67.12 seconds: 67.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1183 (time 1788515995.23 total 10.02 last 118.12) - /unlink 2429 (time 1788516005.23 total 20.02 last 124.60) - /unlink 3669 (time 1788516015.24 total 30.02 last 123.96) total: 4500 /unlink in 37.01 seconds: 121.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 616 (time 1788516037.56 total 10.01 last 61.55) - open/close 1112 (time 1788516047.57 total 20.01 last 49.56) - open/close 1617 (time 1788516057.59 total 30.04 last 50.37) - open/close 2013 (time 1788516067.62 total 40.06 last 39.52) - open/close 2539 (time 1788516077.62 total 50.06 last 52.59) - open/close 3238 (time 1788516087.62 total 60.07 last 69.88) - open/close 3866 (time 1788516097.63 total 70.08 last 62.73) total: 4500 open/close in 79.53 seconds: 56.58 ops/second - /unlink 1198 (time 1788516122.05 total 10.00 last 119.75) - /unlink 2394 (time 1788516132.06 total 20.01 last 119.48) - /unlink 3685 (time 1788516142.06 total 30.02 last 129.08) total: 4500 /unlink in 36.35 seconds: 123.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 652 (time 1788516162.90 total 10.00 last 65.19) - open/close 1260 (time 1788516172.91 total 20.02 last 60.71) - open/close 1944 (time 1788516182.93 total 30.03 last 68.28) - open/close 2638 (time 1788516192.94 total 40.04 last 69.33) - open/close 3350 (time 1788516202.95 total 50.05 last 71.15) - open/close 4017 (time 1788516212.96 total 60.06 last 66.66) total: 4500 open/close in 67.02 seconds: 67.14 ops/second - /unlink 1257 (time 1788516234.24 total 10.00 last 125.69) - /unlink 2536 (time 1788516244.24 total 20.00 last 127.88) - /unlink 3869 (time 1788516254.25 total 30.01 last 133.23) total: 4500 /unlink in 35.25 seconds: 127.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 729 (time 1788516274.63 total 10.01 last 72.83) - open/close 1393 (time 1788516284.64 total 20.02 last 66.31) - open/close 2080 (time 1788516294.65 total 30.03 last 68.67) - open/close 2750 (time 1788516304.67 total 40.05 last 66.87) - open/close 3444 (time 1788516314.68 total 50.06 last 69.34) - open/close 4137 (time 1788516324.69 total 60.07 last 69.24) total: 4500 open/close in 65.30 seconds: 68.91 ops/second - /unlink 1315 (time 1788516344.94 total 10.01 last 131.43) - /unlink 2518 (time 1788516354.96 total 20.03 last 120.06) - /unlink 3600 (time 1788516364.97 total 30.03 last 108.14) total: 4500 /unlink in 38.84 seconds: 115.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 608 (time 1788516390.07 total 10.02 last 60.68) - open/close 1273 (time 1788516400.08 total 20.03 last 66.44) - open/close 2000 (time 1788516410.08 total 30.03 last 72.69) - open/close 2751 (time 1788516420.09 total 40.04 last 75.04) - open/close 3322 (time 1788516430.09 total 50.04 last 57.10) - open/close 3957 (time 1788516440.10 total 60.04 last 63.45) total: 4500 open/close in 67.96 seconds: 66.21 ops/second - /unlink 1327 (time 1788516462.40 total 10.00 last 132.69) - /unlink 2461 (time 1788516472.41 total 20.01 last 113.30) - /unlink 3619 (time 1788516482.42 total 30.01 last 115.76) total: 4500 /unlink in 38.54 seconds: 116.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 647 (time 1788516505.83 total 10.01 last 64.65) - open/close 1181 (time 1788516515.86 total 20.04 last 53.24) - open/close 1615 (time 1788516525.87 total 30.04 last 43.39) - open/close 2246 (time 1788516535.87 total 40.04 last 63.08) - open/close 2825 (time 1788516545.88 total 50.06 last 57.82) - open/close 3396 (time 1788516555.93 total 60.10 last 56.85) - open/close 4015 (time 1788516565.94 total 70.11 last 61.81) total: 4500 open/close in 78.60 seconds: 57.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1175 (time 1788516590.10 total 10.00 last 117.49) - /unlink 2427 (time 1788516600.10 total 20.01 last 125.15) - /unlink 3607 (time 1788516610.10 total 30.01 last 117.98) total: 4500 /unlink in 37.93 seconds: 118.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 620 (time 1788516634.10 total 10.01 last 61.92) - open/close 1245 (time 1788516644.11 total 20.02 last 62.44) - open/close 1882 (time 1788516654.11 total 30.02 last 63.70) - open/close 2548 (time 1788516664.11 total 40.02 last 66.60) - open/close 3172 (time 1788516674.12 total 50.03 last 62.35) - open/close 3812 (time 1788516684.13 total 60.04 last 63.96) - open/close 4494 (time 1788516694.13 total 70.04 last 68.18) total: 4500 open/close in 70.12 seconds: 64.17 ops/second - /unlink 1264 (time 1788516709.34 total 10.00 last 126.40) - /unlink 2517 (time 1788516719.34 total 20.00 last 125.30) - /unlink 3819 (time 1788516729.35 total 30.01 last 130.14) total: 4500 /unlink in 35.93 seconds: 125.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 603 (time 1788516751.34 total 10.00 last 60.29) - open/close 1237 (time 1788516761.35 total 20.02 last 63.29) - open/close 1812 (time 1788516771.37 total 30.03 last 57.41) - open/close 2326 (time 1788516781.37 total 40.04 last 51.38) - open/close 2877 (time 1788516791.40 total 50.07 last 54.93) - open/close 3511 (time 1788516801.41 total 60.07 last 63.38) - open/close 4245 (time 1788516811.41 total 70.08 last 73.36) total: 4500 open/close in 73.29 seconds: 61.40 ops/second - /unlink 1184 (time 1788516829.02 total 10.00 last 118.38) - /unlink 2403 (time 1788516839.03 total 20.01 last 121.85) - /unlink 3690 (time 1788516849.03 total 30.01 last 128.65) total: 4500 /unlink in 36.07 seconds: 124.77 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95488 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server unloading modules via unload_modules_local on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local oleg403-server: kmemleak disabled modules unloaded.