== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:41:02 (1788655262) Stopping clients: oleg433-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg433-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg433-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg433-server oleg433-server: oleg433-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: kmemleak disabled oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: detected 4 online CPUs by sysfs oleg433-server: MODOPTS_LIBCFS= oleg433-server: Force libcfs to create 2 CPU partitions oleg433-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg433-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg433-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 713 (time 1788655478.28 total 10.01 last 71.21) - open/close 1316 (time 1788655488.29 total 20.01 last 60.29) - open/close 1964 (time 1788655498.29 total 30.02 last 64.75) - open/close 2592 (time 1788655508.29 total 40.02 last 62.80) - open/close 3181 (time 1788655518.30 total 50.03 last 58.87) - open/close 3803 (time 1788655528.32 total 60.05 last 62.08) - open/close 4414 (time 1788655538.32 total 70.05 last 61.08) total: 4500 open/close in 71.07 seconds: 63.32 ops/second - /unlink 1289 (time 1788655554.21 total 10.00 last 128.90) - /unlink 2531 (time 1788655564.21 total 20.01 last 124.14) - /unlink 3729 (time 1788655574.21 total 30.01 last 119.76) total: 4500 /unlink in 35.94 seconds: 125.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 659 (time 1788655594.41 total 10.00 last 65.87) - open/close 1299 (time 1788655604.41 total 20.00 last 64.00) - open/close 1912 (time 1788655614.42 total 30.01 last 61.24) - open/close 2538 (time 1788655624.44 total 40.04 last 62.44) - open/close 3179 (time 1788655634.44 total 50.04 last 64.09) - open/close 3851 (time 1788655644.45 total 60.05 last 67.15) total: 4500 open/close in 69.45 seconds: 64.80 ops/second - /unlink 1188 (time 1788655668.85 total 10.01 last 118.71) - /unlink 2292 (time 1788655678.86 total 20.02 last 110.28) - /unlink 3369 (time 1788655688.87 total 30.03 last 107.61) total: 4500 /unlink in 39.18 seconds: 114.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 710 (time 1788655712.86 total 10.01 last 70.94) - open/close 1392 (time 1788655722.87 total 20.01 last 68.17) - open/close 2026 (time 1788655732.87 total 30.02 last 63.37) - open/close 2676 (time 1788655742.88 total 40.03 last 64.93) - open/close 3341 (time 1788655752.88 total 50.03 last 66.50) - open/close 4012 (time 1788655762.89 total 60.03 last 67.08) total: 4500 open/close in 67.45 seconds: 66.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1278 (time 1788655784.71 total 10.01 last 127.69) - /unlink 2634 (time 1788655794.72 total 20.02 last 135.45) - /unlink 3929 (time 1788655804.72 total 30.02 last 129.49) total: 4500 /unlink in 34.24 seconds: 131.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 684 (time 1788655823.06 total 10.01 last 68.34) - open/close 1391 (time 1788655833.08 total 20.02 last 70.61) - open/close 2044 (time 1788655843.08 total 30.03 last 65.25) - open/close 2702 (time 1788655853.09 total 40.03 last 65.78) - open/close 3371 (time 1788655863.09 total 50.04 last 66.86) - open/close 3658 (time 1788655873.10 total 60.05 last 28.67) - open/close 3996 (time 1788655883.12 total 70.06 last 33.75) - open/close 4443 (time 1788655893.14 total 80.09 last 44.57) total: 4500 open/close in 81.05 seconds: 55.52 ops/second - /unlink 1046 (time 1788655911.78 total 10.01 last 104.49) - /unlink 1886 (time 1788655921.78 total 20.02 last 83.96) - /unlink 3102 (time 1788655931.79 total 30.02 last 121.51) - /unlink 4282 (time 1788655941.80 total 40.04 last 117.86) total: 4500 /unlink in 42.05 seconds: 107.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 683 (time 1788655958.52 total 10.01 last 68.25) - open/close 1342 (time 1788655968.55 total 20.04 last 65.69) - open/close 2007 (time 1788655978.56 total 30.05 last 66.44) - open/close 2660 (time 1788655988.57 total 40.06 last 65.25) - open/close 3338 (time 1788655998.58 total 50.07 last 67.70) - open/close 3995 (time 1788656008.58 total 60.07 last 65.70) total: 4500 open/close in 67.29 seconds: 66.88 ops/second - /unlink 1252 (time 1788656030.52 total 10.01 last 125.02) - /unlink 2473 (time 1788656040.53 total 20.03 last 121.97) - /unlink 3581 (time 1788656050.54 total 30.03 last 110.77) total: 4500 /unlink in 38.63 seconds: 116.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 720 (time 1788656074.42 total 10.00 last 71.96) - open/close 1383 (time 1788656084.43 total 20.02 last 66.18) - open/close 2019 (time 1788656094.44 total 30.03 last 63.57) - open/close 2649 (time 1788656104.45 total 40.03 last 62.95) - open/close 3314 (time 1788656114.45 total 50.03 last 66.50) - open/close 3961 (time 1788656124.46 total 60.05 last 64.63) total: 4500 open/close in 67.82 seconds: 66.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1102 (time 1788656147.10 total 10.01 last 110.13) - /unlink 2295 (time 1788656157.11 total 20.01 last 119.25) - /unlink 3440 (time 1788656167.11 total 30.01 last 114.50) total: 4500 /unlink in 39.08 seconds: 115.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 666 (time 1788656191.88 total 10.01 last 66.54) - open/close 1302 (time 1788656201.89 total 20.01 last 63.57) - open/close 1941 (time 1788656211.89 total 30.01 last 63.89) - open/close 2560 (time 1788656221.90 total 40.02 last 61.85) - open/close 3199 (time 1788656231.90 total 50.02 last 63.90) - open/close 3926 (time 1788656241.91 total 60.03 last 72.63) total: 4500 open/close in 68.91 seconds: 65.30 ops/second - /unlink 1290 (time 1788656265.73 total 10.00 last 128.97) - /unlink 2514 (time 1788656275.73 total 20.00 last 122.38) - /unlink 3652 (time 1788656285.73 total 30.01 last 113.78) total: 4500 /unlink in 36.44 seconds: 123.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 660 (time 1788656307.37 total 10.01 last 65.96) - open/close 1324 (time 1788656317.39 total 20.02 last 66.33) - open/close 1951 (time 1788656327.39 total 30.02 last 62.66) - open/close 2632 (time 1788656337.40 total 40.03 last 68.05) - open/close 3308 (time 1788656347.40 total 50.03 last 67.58) - open/close 3939 (time 1788656357.41 total 60.04 last 63.05) total: 4500 open/close in 68.86 seconds: 65.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1282 (time 1788656381.38 total 10.00 last 128.14) - /unlink 2562 (time 1788656391.38 total 20.01 last 127.98) - /unlink 3714 (time 1788656401.40 total 30.02 last 115.04) total: 4500 /unlink in 36.65 seconds: 122.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 617 (time 1788656422.38 total 10.01 last 61.63) - open/close 1241 (time 1788656432.39 total 20.02 last 62.36) - open/close 1846 (time 1788656442.41 total 30.04 last 60.40) - open/close 2466 (time 1788656452.41 total 40.04 last 61.99) - open/close 3079 (time 1788656462.41 total 50.04 last 61.29) - open/close 3691 (time 1788656472.43 total 60.06 last 61.10) - open/close 4338 (time 1788656482.44 total 70.06 last 64.64) total: 4500 open/close in 72.68 seconds: 61.91 ops/second - /unlink 1244 (time 1788656500.01 total 10.00 last 124.36) - /unlink 2395 (time 1788656510.01 total 20.00 last 115.09) - /unlink 3593 (time 1788656520.01 total 30.01 last 119.78) total: 4500 /unlink in 36.90 seconds: 121.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 567 (time 1788656542.27 total 10.01 last 56.64) - open/close 1204 (time 1788656552.29 total 20.03 last 63.59) - open/close 1800 (time 1788656562.29 total 30.03 last 59.58) - open/close 2459 (time 1788656572.29 total 40.03 last 65.89) - open/close 3019 (time 1788656582.32 total 50.06 last 55.83) - open/close 3533 (time 1788656592.36 total 60.10 last 51.19) - open/close 4138 (time 1788656602.37 total 70.11 last 60.46) total: 4500 open/close in 76.35 seconds: 58.94 ops/second - /unlink 1281 (time 1788656623.42 total 10.00 last 128.06) - /unlink 2368 (time 1788656633.43 total 20.01 last 108.66) - /unlink 3573 (time 1788656643.43 total 30.01 last 120.44) total: 4500 /unlink in 38.41 seconds: 117.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 587 (time 1788656667.73 total 10.01 last 58.66) - open/close 1146 (time 1788656677.74 total 20.01 last 55.87) - open/close 1797 (time 1788656687.74 total 30.02 last 65.05) - open/close 2461 (time 1788656697.75 total 40.03 last 66.36) - open/close 3074 (time 1788656707.75 total 50.03 last 61.27) - open/close 3717 (time 1788656717.76 total 60.04 last 64.24) - open/close 4373 (time 1788656727.77 total 70.04 last 65.60) total: 4500 open/close in 71.61 seconds: 62.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1230 (time 1788656744.07 total 10.00 last 123.00) - /unlink 2509 (time 1788656754.07 total 20.00 last 127.89) - /unlink 3733 (time 1788656764.07 total 30.00 last 122.39) total: 4500 /unlink in 35.78 seconds: 125.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 658 (time 1788656784.41 total 10.00 last 65.80) - open/close 1303 (time 1788656794.41 total 20.00 last 64.49) - open/close 1986 (time 1788656804.42 total 30.01 last 68.23) - open/close 2636 (time 1788656814.42 total 40.02 last 64.97) - open/close 3280 (time 1788656824.43 total 50.02 last 64.37) - open/close 3967 (time 1788656834.45 total 60.04 last 68.56) total: 4500 open/close in 68.31 seconds: 65.88 ops/second - /unlink 1248 (time 1788656857.09 total 10.00 last 124.74) - /unlink 2527 (time 1788656867.09 total 20.01 last 127.88) - /unlink 3837 (time 1788656877.09 total 30.01 last 130.93) total: 4500 /unlink in 36.10 seconds: 124.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 701 (time 1788656898.18 total 10.01 last 70.01) - open/close 1387 (time 1788656908.19 total 20.02 last 68.55) - open/close 2076 (time 1788656918.22 total 30.04 last 68.74) - open/close 2725 (time 1788656928.22 total 40.04 last 64.89) - open/close 3478 (time 1788656938.22 total 50.05 last 75.27) - open/close 4196 (time 1788656948.22 total 60.05 last 71.76) total: 4500 open/close in 64.55 seconds: 69.71 ops/second - /unlink 1163 (time 1788656966.89 total 10.00 last 116.27) - /unlink 2309 (time 1788656976.89 total 20.00 last 114.60) - /unlink 3495 (time 1788656986.89 total 30.01 last 118.57) total: 4500 /unlink in 37.93 seconds: 118.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 721 (time 1788657010.21 total 10.01 last 72.03) - open/close 1425 (time 1788657020.21 total 20.01 last 70.39) - open/close 2041 (time 1788657030.24 total 30.04 last 61.45) - open/close 2726 (time 1788657040.25 total 40.05 last 68.39) - open/close 3425 (time 1788657050.26 total 50.05 last 69.88) - open/close 4096 (time 1788657060.26 total 60.06 last 67.08) total: 4500 open/close in 66.44 seconds: 67.73 ops/second - /unlink 1200 (time 1788657081.83 total 10.00 last 119.96) - /unlink 2434 (time 1788657091.84 total 20.01 last 123.32) - /unlink 3688 (time 1788657101.84 total 30.01 last 125.40) total: 4500 /unlink in 36.42 seconds: 123.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 663 (time 1788657122.73 total 10.02 last 66.20) - open/close 1275 (time 1788657132.73 total 20.02 last 61.18) - open/close 2001 (time 1788657142.73 total 30.02 last 72.56) - open/close 2683 (time 1788657152.74 total 40.03 last 68.18) - open/close 3249 (time 1788657162.77 total 50.06 last 56.40) - open/close 3811 (time 1788657172.82 total 60.11 last 55.91) - open/close 4384 (time 1788657182.83 total 70.12 last 57.28) total: 4500 open/close in 71.66 seconds: 62.80 ops/second - /unlink 1390 (time 1788657198.88 total 10.00 last 138.99) - /unlink 2672 (time 1788657208.89 total 20.01 last 128.14) - /unlink 3993 (time 1788657218.89 total 30.01 last 132.00) total: 4500 /unlink in 33.77 seconds: 133.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 723 (time 1788657237.16 total 10.00 last 72.30) - open/close 1409 (time 1788657247.17 total 20.01 last 68.54) - open/close 2139 (time 1788657257.18 total 30.01 last 72.97) - open/close 2857 (time 1788657267.18 total 40.02 last 71.77) - open/close 3577 (time 1788657277.18 total 50.02 last 71.97) - open/close 4204 (time 1788657287.19 total 60.03 last 62.64) total: 4500 open/close in 63.93 seconds: 70.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1138 (time 1788657306.03 total 10.00 last 113.76) - /unlink 2329 (time 1788657316.04 total 20.01 last 118.97) - /unlink 3506 (time 1788657326.05 total 30.02 last 117.60) total: 4500 /unlink in 38.42 seconds: 117.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 653 (time 1788657349.68 total 10.01 last 65.25) - open/close 1378 (time 1788657359.69 total 20.02 last 72.44) - open/close 2079 (time 1788657369.70 total 30.02 last 70.04) - open/close 2760 (time 1788657379.71 total 40.03 last 68.05) - open/close 3421 (time 1788657389.72 total 50.04 last 66.05) - open/close 4085 (time 1788657399.72 total 60.04 last 66.40) total: 4500 open/close in 66.56 seconds: 67.61 ops/second - /unlink 1169 (time 1788657421.63 total 10.01 last 116.83) - /unlink 2378 (time 1788657431.64 total 20.02 last 120.79) - /unlink 3611 (time 1788657441.64 total 30.02 last 123.30) total: 4500 /unlink in 37.95 seconds: 118.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 577 (time 1788657465.44 total 10.01 last 57.64) - open/close 1128 (time 1788657475.47 total 20.04 last 54.93) - open/close 1797 (time 1788657485.49 total 30.06 last 66.79) - open/close 2451 (time 1788657495.49 total 40.07 last 65.34) - open/close 3066 (time 1788657505.50 total 50.07 last 61.48) - open/close 3730 (time 1788657515.52 total 60.09 last 66.25) - open/close 4486 (time 1788657525.53 total 70.10 last 75.52) total: 4500 open/close in 70.30 seconds: 64.01 ops/second - /unlink 1282 (time 1788657540.29 total 10.00 last 128.20) - /unlink 2546 (time 1788657550.30 total 20.00 last 126.36) - /unlink 3948 (time 1788657560.30 total 30.01 last 140.17) total: 4500 /unlink in 34.22 seconds: 131.48 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95354 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg433-server unloading modules via unload_modules_local on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing unload_modules_local oleg433-server: kmemleak disabled modules unloaded.