== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:56:18 (1777575378) Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg357-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg357-server oleg357-server: oleg357-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg357-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg357-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 676 (time 1777575596.31 total 10.00 last 67.59) - open/close 1257 (time 1777575606.31 total 20.01 last 58.08) - open/close 1891 (time 1777575616.33 total 30.02 last 63.30) - open/close 2579 (time 1777575626.33 total 40.02 last 68.78) - open/close 3176 (time 1777575636.34 total 50.03 last 59.67) - open/close 3782 (time 1777575646.34 total 60.03 last 60.59) - open/close 4388 (time 1777575656.34 total 70.03 last 60.59) total: 4500 open/close in 71.88 seconds: 62.60 ops/second - /unlink 1107 (time 1777575673.76 total 10.01 last 110.62) - /unlink 2187 (time 1777575683.76 total 20.01 last 107.96) - /unlink 3237 (time 1777575693.77 total 30.02 last 104.91) - /unlink 4421 (time 1777575703.77 total 40.02 last 118.39) total: 4500 /unlink in 40.73 seconds: 110.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 696 (time 1777575718.99 total 10.00 last 69.59) - open/close 1388 (time 1777575729.01 total 20.03 last 69.02) - open/close 2086 (time 1777575739.02 total 30.03 last 69.75) - open/close 2659 (time 1777575749.03 total 40.04 last 57.24) - open/close 3282 (time 1777575759.04 total 50.06 last 62.23) - open/close 3870 (time 1777575769.05 total 60.06 last 58.78) total: 4500 open/close in 69.45 seconds: 64.80 ops/second - /unlink 1001 (time 1777575792.86 total 10.00 last 100.09) - /unlink 2097 (time 1777575802.87 total 20.01 last 109.48) - /unlink 3094 (time 1777575812.88 total 30.02 last 99.65) - /unlink 4092 (time 1777575822.89 total 40.02 last 99.73) total: 4500 /unlink in 44.06 seconds: 102.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 595 (time 1777575842.88 total 10.01 last 59.41) - open/close 1194 (time 1777575852.89 total 20.02 last 59.87) - open/close 1780 (time 1777575862.89 total 30.02 last 58.59) - open/close 2352 (time 1777575872.89 total 40.03 last 57.18) - open/close 2934 (time 1777575882.91 total 50.04 last 58.12) - open/close 3603 (time 1777575892.91 total 60.04 last 66.86) - open/close 4238 (time 1777575902.93 total 70.06 last 63.40) total: 4500 open/close in 74.19 seconds: 60.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1236 (time 1777575922.34 total 10.00 last 123.59) - /unlink 2311 (time 1777575932.35 total 20.01 last 107.40) - /unlink 3353 (time 1777575942.36 total 30.02 last 104.09) - /unlink 4453 (time 1777575952.36 total 40.03 last 109.93) total: 4500 /unlink in 40.55 seconds: 110.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 598 (time 1777575967.33 total 10.01 last 59.71) - open/close 1172 (time 1777575977.33 total 20.02 last 57.40) - open/close 1760 (time 1777575987.34 total 30.02 last 58.79) - open/close 2311 (time 1777575997.37 total 40.05 last 54.90) - open/close 2794 (time 1777576007.38 total 50.06 last 48.26) - open/close 3441 (time 1777576017.39 total 60.07 last 64.64) - open/close 4021 (time 1777576027.41 total 70.09 last 57.91) total: 4500 open/close in 78.29 seconds: 57.48 ops/second - /unlink 1085 (time 1777576050.50 total 10.00 last 108.49) - /unlink 2198 (time 1777576060.50 total 20.00 last 111.27) - /unlink 3328 (time 1777576070.52 total 30.02 last 112.80) - /unlink 4476 (time 1777576080.53 total 40.03 last 114.71) total: 4500 /unlink in 40.36 seconds: 111.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 609 (time 1777576096.18 total 10.01 last 60.83) - open/close 1225 (time 1777576106.19 total 20.02 last 61.55) - open/close 1850 (time 1777576116.21 total 30.04 last 62.37) - open/close 2489 (time 1777576126.25 total 40.07 last 63.68) - open/close 3102 (time 1777576136.25 total 50.08 last 61.28) - open/close 3753 (time 1777576146.26 total 60.09 last 65.05) - open/close 4407 (time 1777576156.26 total 70.09 last 65.36) total: 4500 open/close in 71.67 seconds: 62.78 ops/second - /unlink 1189 (time 1777576172.32 total 10.01 last 118.83) - /unlink 2451 (time 1777576182.37 total 20.06 last 125.58) - /unlink 3544 (time 1777576192.37 total 30.06 last 109.30) total: 4500 /unlink in 37.61 seconds: 119.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 584 (time 1777576215.81 total 10.01 last 58.34) - open/close 1215 (time 1777576225.81 total 20.01 last 63.08) - open/close 1884 (time 1777576235.81 total 30.02 last 66.88) - open/close 2503 (time 1777576245.81 total 40.02 last 61.88) - open/close 3151 (time 1777576255.83 total 50.03 last 64.71) - open/close 3696 (time 1777576265.85 total 60.05 last 54.39) - open/close 4222 (time 1777576275.86 total 70.07 last 52.53) total: 4500 open/close in 74.31 seconds: 60.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1103 (time 1777576295.48 total 10.00 last 110.29) - /unlink 2169 (time 1777576305.48 total 20.01 last 106.51) - /unlink 3324 (time 1777576315.50 total 30.02 last 115.35) total: 4500 /unlink in 39.70 seconds: 113.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 607 (time 1777576340.29 total 10.01 last 60.61) - open/close 1168 (time 1777576350.30 total 20.03 last 56.04) - open/close 1812 (time 1777576360.31 total 30.03 last 64.39) - open/close 2444 (time 1777576370.32 total 40.04 last 63.11) - open/close 3091 (time 1777576380.33 total 50.05 last 64.66) - open/close 3741 (time 1777576390.34 total 60.06 last 64.91) - open/close 4396 (time 1777576400.35 total 70.07 last 65.44) total: 4500 open/close in 71.75 seconds: 62.72 ops/second - /unlink 1224 (time 1777576416.99 total 10.00 last 122.37) - /unlink 2354 (time 1777576426.99 total 20.00 last 112.98) - /unlink 3444 (time 1777576436.99 total 30.01 last 108.96) total: 4500 /unlink in 39.21 seconds: 114.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 643 (time 1777576461.15 total 10.01 last 64.27) - open/close 1235 (time 1777576471.16 total 20.01 last 59.16) - open/close 1799 (time 1777576481.19 total 30.04 last 56.24) - open/close 2382 (time 1777576491.19 total 40.05 last 58.28) - open/close 3024 (time 1777576501.20 total 50.05 last 64.14) - open/close 3677 (time 1777576511.20 total 60.06 last 65.29) - open/close 4304 (time 1777576521.22 total 70.08 last 62.58) total: 4500 open/close in 72.72 seconds: 61.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1274 (time 1777576538.43 total 10.01 last 127.28) - /unlink 2405 (time 1777576548.43 total 20.01 last 113.04) - /unlink 3590 (time 1777576558.43 total 30.02 last 118.49) total: 4500 /unlink in 37.67 seconds: 119.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 707 (time 1777576582.56 total 10.00 last 70.68) - open/close 1341 (time 1777576592.57 total 20.01 last 63.35) - open/close 1924 (time 1777576602.59 total 30.03 last 58.20) - open/close 2507 (time 1777576612.61 total 40.04 last 58.20) - open/close 3117 (time 1777576622.62 total 50.05 last 60.94) - open/close 3701 (time 1777576632.62 total 60.06 last 58.35) - open/close 4246 (time 1777576642.63 total 70.07 last 54.46) total: 4500 open/close in 74.60 seconds: 60.32 ops/second - /unlink 1077 (time 1777576662.12 total 10.01 last 107.63) - /unlink 2134 (time 1777576672.12 total 20.01 last 105.67) - /unlink 3278 (time 1777576682.12 total 30.01 last 114.38) total: 4500 /unlink in 39.98 seconds: 112.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 628 (time 1777576706.58 total 10.01 last 62.72) - open/close 1245 (time 1777576716.58 total 20.02 last 61.67) - open/close 1880 (time 1777576726.59 total 30.02 last 63.49) - open/close 2519 (time 1777576736.59 total 40.02 last 63.89) - open/close 3164 (time 1777576746.60 total 50.03 last 64.45) - open/close 3840 (time 1777576756.60 total 60.03 last 67.58) - open/close 4491 (time 1777576766.61 total 70.04 last 65.04) total: 4500 open/close in 70.16 seconds: 64.14 ops/second - /unlink 1208 (time 1777576781.36 total 10.00 last 120.76) - /unlink 2365 (time 1777576791.36 total 20.01 last 115.67) - /unlink 3440 (time 1777576801.37 total 30.02 last 107.39) total: 4500 /unlink in 39.26 seconds: 114.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 624 (time 1777576825.33 total 10.00 last 62.39) - open/close 1179 (time 1777576835.36 total 20.03 last 55.34) - open/close 1761 (time 1777576845.36 total 30.04 last 58.15) - open/close 2302 (time 1777576855.38 total 40.05 last 54.04) - open/close 2932 (time 1777576865.38 total 50.05 last 62.98) - open/close 3562 (time 1777576875.42 total 60.09 last 62.76) - open/close 4202 (time 1777576885.44 total 70.11 last 63.88) total: 4500 open/close in 74.54 seconds: 60.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1108 (time 1777576904.39 total 10.00 last 110.75) - /unlink 2289 (time 1777576914.40 total 20.01 last 118.07) - /unlink 3418 (time 1777576924.41 total 30.02 last 112.74) - /unlink 4445 (time 1777576934.43 total 40.04 last 102.48) total: 4500 /unlink in 40.62 seconds: 110.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 594 (time 1777576951.18 total 10.01 last 59.35) - open/close 1191 (time 1777576961.21 total 20.03 last 59.55) - open/close 1717 (time 1777576971.23 total 30.06 last 52.48) - open/close 2288 (time 1777576981.24 total 40.07 last 57.04) - open/close 2793 (time 1777576991.27 total 50.09 last 50.37) - open/close 3399 (time 1777577001.28 total 60.11 last 60.52) - open/close 4056 (time 1777577011.31 total 70.13 last 65.53) total: 4500 open/close in 77.90 seconds: 57.77 ops/second - /unlink 1160 (time 1777577033.72 total 10.00 last 115.99) - /unlink 2361 (time 1777577043.73 total 20.01 last 119.98) - /unlink 3511 (time 1777577053.75 total 30.03 last 114.77) total: 4500 /unlink in 38.03 seconds: 118.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 655 (time 1777577076.30 total 10.02 last 65.38) - open/close 1308 (time 1777577086.31 total 20.03 last 65.22) - open/close 1986 (time 1777577096.31 total 30.03 last 67.79) - open/close 2661 (time 1777577106.32 total 40.05 last 67.40) - open/close 3305 (time 1777577116.33 total 50.05 last 64.35) - open/close 3893 (time 1777577126.34 total 60.06 last 58.77) - open/close 4498 (time 1777577136.34 total 70.06 last 60.50) total: 4500 open/close in 70.07 seconds: 64.22 ops/second - /unlink 1165 (time 1777577150.63 total 10.00 last 116.44) - /unlink 2263 (time 1777577160.64 total 20.01 last 109.72) - /unlink 3316 (time 1777577170.66 total 30.03 last 105.08) total: 4500 /unlink in 39.71 seconds: 113.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 453 (time 1777577197.65 total 10.02 last 45.23) - open/close 940 (time 1777577207.65 total 20.02 last 48.66) - open/close 1561 (time 1777577217.67 total 30.04 last 62.00) - open/close 2163 (time 1777577227.68 total 40.05 last 60.12) - open/close 2823 (time 1777577237.68 total 50.06 last 66.00) - open/close 3509 (time 1777577247.70 total 60.07 last 68.52) - open/close 4168 (time 1777577257.70 total 70.07 last 65.88) total: 4500 open/close in 74.77 seconds: 60.18 ops/second - /unlink 1148 (time 1777577277.19 total 10.01 last 114.69) - /unlink 2393 (time 1777577287.20 total 20.02 last 124.43) - /unlink 3605 (time 1777577297.20 total 30.02 last 121.11) total: 4500 /unlink in 37.52 seconds: 119.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 692 (time 1777577319.87 total 10.00 last 69.17) - open/close 1384 (time 1777577329.88 total 20.02 last 69.08) - open/close 1996 (time 1777577339.90 total 30.04 last 61.11) - open/close 2581 (time 1777577349.91 total 40.05 last 58.43) - open/close 3248 (time 1777577359.91 total 50.05 last 66.70) - open/close 3920 (time 1777577369.92 total 60.06 last 67.15) - open/close 4490 (time 1777577379.93 total 70.07 last 56.95) total: 4500 open/close in 70.17 seconds: 64.13 ops/second - /unlink 1073 (time 1777577395.45 total 10.01 last 107.24) - /unlink 2210 (time 1777577405.45 total 20.01 last 113.68) - /unlink 3355 (time 1777577415.48 total 30.03 last 114.25) - /unlink 4421 (time 1777577425.49 total 40.04 last 106.48) total: 4500 /unlink in 40.94 seconds: 109.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 580 (time 1777577442.38 total 10.01 last 57.94) - open/close 1123 (time 1777577452.39 total 20.02 last 54.25) - open/close 1747 (time 1777577462.40 total 30.03 last 62.34) - open/close 2289 (time 1777577472.40 total 40.04 last 54.16) - open/close 2889 (time 1777577482.41 total 50.04 last 59.98) - open/close 3475 (time 1777577492.41 total 60.04 last 58.60) - open/close 4119 (time 1777577502.41 total 70.05 last 64.34) total: 4500 open/close in 76.83 seconds: 58.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1031 (time 1777577523.34 total 10.01 last 103.03) - /unlink 2133 (time 1777577533.34 total 20.01 last 110.18) - /unlink 3279 (time 1777577543.34 total 30.01 last 114.54) - /unlink 4484 (time 1777577553.35 total 40.01 last 120.48) total: 4500 /unlink in 40.15 seconds: 112.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 652 (time 1777577568.24 total 10.01 last 65.16) - open/close 1322 (time 1777577578.25 total 20.02 last 66.93) - open/close 1976 (time 1777577588.25 total 30.02 last 65.38) - open/close 2662 (time 1777577598.27 total 40.04 last 68.46) - open/close 3292 (time 1777577608.28 total 50.05 last 62.92) - open/close 3801 (time 1777577618.31 total 60.08 last 50.79) - open/close 4230 (time 1777577628.33 total 70.10 last 42.81) total: 4500 open/close in 75.03 seconds: 59.98 ops/second - /unlink 1070 (time 1777577648.44 total 10.01 last 106.88) - /unlink 2277 (time 1777577658.45 total 20.02 last 120.59) - /unlink 3451 (time 1777577668.46 total 30.03 last 117.27) total: 4500 /unlink in 39.69 seconds: 113.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 639 (time 1777577692.65 total 10.01 last 63.81) - open/close 1290 (time 1777577702.68 total 20.04 last 64.96) - open/close 1983 (time 1777577712.70 total 30.06 last 69.16) - open/close 2625 (time 1777577722.70 total 40.06 last 64.19) - open/close 3255 (time 1777577732.70 total 50.06 last 62.96) - open/close 3925 (time 1777577742.71 total 60.07 last 66.96) total: 4500 open/close in 68.65 seconds: 65.55 ops/second - /unlink 1192 (time 1777577765.86 total 10.00 last 119.17) - /unlink 2409 (time 1777577775.86 total 20.01 last 121.64) - /unlink 3633 (time 1777577785.87 total 30.01 last 122.36) total: 4500 /unlink in 38.02 seconds: 118.35 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 93441 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server unloading modules via unload_modules_local on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local oleg357-server: modules unloaded.