== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:27:13 (1787851633) Stopping clients: oleg409-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg409-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg409-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg409-server oleg409-server: oleg409-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing load_modules_local oleg409-server: Loading modules from /home/green/git/lustre-release/lustre oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg409-server: detected 4 online CPUs by sysfs oleg409-server: MODOPTS_LIBCFS= oleg409-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg409-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg409-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg409-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 636 (time 1787851867.77 total 10.02 last 63.47) - open/close 1177 (time 1787851877.78 total 20.04 last 54.02) - open/close 1807 (time 1787851887.82 total 30.07 last 62.77) - open/close 2396 (time 1787851897.83 total 40.08 last 58.84) - open/close 2974 (time 1787851907.84 total 50.10 last 57.72) - open/close 3579 (time 1787851917.85 total 60.11 last 60.43) - open/close 4124 (time 1787851927.86 total 70.11 last 54.49) total: 4500 open/close in 77.71 seconds: 57.91 ops/second - /unlink 846 (time 1787851951.69 total 10.03 last 84.38) - /unlink 1792 (time 1787851961.70 total 20.04 last 94.51) - /unlink 2789 (time 1787851971.71 total 30.04 last 99.64) - /unlink 3811 (time 1787851981.71 total 40.05 last 102.15) total: 4500 /unlink in 46.36 seconds: 97.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 571 (time 1787852004.67 total 10.01 last 57.07) - open/close 1103 (time 1787852014.68 total 20.02 last 53.11) - open/close 1579 (time 1787852024.70 total 30.04 last 47.52) - open/close 2163 (time 1787852034.70 total 40.04 last 58.40) - open/close 2784 (time 1787852044.71 total 50.05 last 62.04) - open/close 3403 (time 1787852054.75 total 60.09 last 61.65) - open/close 3932 (time 1787852064.76 total 70.10 last 52.84) - open/close 4456 (time 1787852074.77 total 80.11 last 52.38) total: 4500 open/close in 80.80 seconds: 55.70 ops/second - /unlink 1089 (time 1787852090.81 total 10.00 last 108.87) - /unlink 2171 (time 1787852100.81 total 20.01 last 108.16) - /unlink 3230 (time 1787852110.81 total 30.01 last 105.88) - /unlink 4456 (time 1787852120.81 total 40.01 last 122.59) total: 4500 /unlink in 40.53 seconds: 111.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 540 (time 1787852136.79 total 10.00 last 53.97) - open/close 1114 (time 1787852146.80 total 20.02 last 57.33) - open/close 1516 (time 1787852156.87 total 30.08 last 39.95) - open/close 1850 (time 1787852166.87 total 40.09 last 33.38) - open/close 2178 (time 1787852176.88 total 50.10 last 32.77) - open/close 2572 (time 1787852186.89 total 60.10 last 39.37) - open/close 2969 (time 1787852196.90 total 70.11 last 39.68) - open/close 3421 (time 1787852206.91 total 80.12 last 45.14) - open/close 3972 (time 1787852216.92 total 90.14 last 55.01) - open/close 4498 (time 1787852226.93 total 100.14 last 52.58) total: 4500 open/close in 100.16 seconds: 44.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1054 (time 1787852243.42 total 10.00 last 105.36) - /unlink 2068 (time 1787852253.42 total 20.01 last 101.38) - /unlink 3102 (time 1787852263.42 total 30.01 last 103.37) - /unlink 4033 (time 1787852273.42 total 40.01 last 93.08) total: 4500 /unlink in 44.71 seconds: 100.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 558 (time 1787852294.78 total 10.00 last 55.78) - open/close 1120 (time 1787852304.79 total 20.02 last 56.12) - open/close 1660 (time 1787852314.81 total 30.04 last 53.88) - open/close 2282 (time 1787852324.82 total 40.04 last 62.17) - open/close 2788 (time 1787852334.84 total 50.07 last 50.48) - open/close 3336 (time 1787852344.84 total 60.07 last 54.80) - open/close 3881 (time 1787852354.85 total 70.08 last 54.43) - open/close 4399 (time 1787852364.86 total 80.08 last 51.79) total: 4500 open/close in 82.04 seconds: 54.85 ops/second - /unlink 1233 (time 1787852381.65 total 10.00 last 123.25) - /unlink 2227 (time 1787852391.66 total 20.02 last 99.26) - /unlink 3205 (time 1787852401.66 total 30.02 last 97.78) - /unlink 4193 (time 1787852411.67 total 40.02 last 98.77) total: 4500 /unlink in 44.18 seconds: 101.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 340 (time 1787852432.44 total 10.00 last 34.00) - open/close 732 (time 1787852442.46 total 20.02 last 39.12) - open/close 1303 (time 1787852452.48 total 30.04 last 57.00) - open/close 1867 (time 1787852462.49 total 40.05 last 56.35) - open/close 2453 (time 1787852472.50 total 50.05 last 58.57) - open/close 2954 (time 1787852482.50 total 60.05 last 50.09) - open/close 3466 (time 1787852492.50 total 70.05 last 51.20) - open/close 3928 (time 1787852502.51 total 80.07 last 46.13) total: 4500 open/close in 89.80 seconds: 50.11 ops/second - /unlink 1159 (time 1787852527.32 total 10.02 last 115.66) - /unlink 2332 (time 1787852537.32 total 20.02 last 117.29) - /unlink 3468 (time 1787852547.33 total 30.03 last 113.53) total: 4500 /unlink in 39.63 seconds: 113.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 537 (time 1787852572.80 total 10.01 last 53.66) - open/close 1121 (time 1787852582.83 total 20.04 last 58.23) - open/close 1530 (time 1787852592.85 total 30.05 last 40.84) - open/close 1925 (time 1787852602.87 total 40.08 last 39.40) - open/close 2238 (time 1787852612.88 total 50.08 last 31.28) - open/close 2640 (time 1787852622.88 total 60.09 last 40.18) - open/close 3267 (time 1787852632.90 total 70.10 last 62.62) - open/close 3889 (time 1787852642.91 total 80.11 last 62.14) - open/close 4470 (time 1787852652.93 total 90.13 last 57.98) total: 4500 open/close in 90.57 seconds: 49.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1151 (time 1787852669.04 total 10.01 last 115.01) - /unlink 2227 (time 1787852679.04 total 20.01 last 107.56) - /unlink 3104 (time 1787852689.05 total 30.02 last 87.67) - /unlink 4163 (time 1787852699.06 total 40.02 last 105.82) total: 4500 /unlink in 43.76 seconds: 102.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 373 (time 1787852720.86 total 10.06 last 37.06) - open/close 763 (time 1787852730.89 total 20.09 last 38.91) - open/close 1205 (time 1787852740.92 total 30.13 last 44.04) - open/close 1717 (time 1787852750.94 total 40.14 last 51.12) - open/close 2331 (time 1787852760.95 total 50.15 last 61.36) - open/close 2881 (time 1787852770.96 total 60.16 last 54.95) - open/close 3351 (time 1787852780.97 total 70.17 last 46.94) - open/close 3726 (time 1787852790.97 total 80.17 last 37.50) - open/close 4153 (time 1787852800.99 total 90.19 last 42.62) total: 4500 open/close in 98.12 seconds: 45.86 ops/second - /unlink 1014 (time 1787852824.22 total 10.02 last 101.24) - /unlink 1940 (time 1787852834.22 total 20.02 last 92.60) - /unlink 2800 (time 1787852844.23 total 30.02 last 85.97) - /unlink 3677 (time 1787852854.23 total 40.03 last 87.63) total: 4500 /unlink in 48.47 seconds: 92.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 624 (time 1787852877.39 total 10.01 last 62.34) - open/close 1116 (time 1787852887.39 total 20.01 last 49.20) - open/close 1612 (time 1787852897.40 total 30.02 last 49.56) - open/close 2062 (time 1787852907.42 total 40.03 last 44.92) - open/close 2572 (time 1787852917.43 total 50.04 last 50.96) - open/close 3109 (time 1787852927.43 total 60.05 last 53.67) - open/close 3739 (time 1787852937.44 total 70.06 last 62.95) - open/close 4232 (time 1787852947.46 total 80.08 last 49.20) total: 4500 open/close in 85.73 seconds: 52.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 958 (time 1787852969.82 total 10.01 last 95.66) - /unlink 1858 (time 1787852979.83 total 20.02 last 89.91) - /unlink 2896 (time 1787852989.83 total 30.03 last 103.78) - /unlink 3943 (time 1787852999.83 total 40.03 last 104.69) total: 4500 /unlink in 45.14 seconds: 99.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 520 (time 1787853021.73 total 10.02 last 51.87) - open/close 1034 (time 1787853031.75 total 20.04 last 51.31) - open/close 1590 (time 1787853041.76 total 30.05 last 55.56) - open/close 2136 (time 1787853051.79 total 40.08 last 54.41) - open/close 2771 (time 1787853061.81 total 50.10 last 63.41) - open/close 3402 (time 1787853071.81 total 60.10 last 63.08) - open/close 3918 (time 1787853081.82 total 70.11 last 51.56) total: 4500 open/close in 79.97 seconds: 56.27 ops/second - /unlink 1093 (time 1787853108.18 total 10.00 last 109.26) - /unlink 2154 (time 1787853118.18 total 20.01 last 106.07) - /unlink 3324 (time 1787853128.18 total 30.01 last 116.98) - /unlink 4402 (time 1787853138.18 total 40.01 last 107.79) total: 4500 /unlink in 41.70 seconds: 107.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 563 (time 1787853157.05 total 10.02 last 56.17) - open/close 1116 (time 1787853167.06 total 20.03 last 55.24) - open/close 1777 (time 1787853177.06 total 30.04 last 66.09) - open/close 2382 (time 1787853187.08 total 40.05 last 60.40) - open/close 3001 (time 1787853197.08 total 50.06 last 61.85) - open/close 3606 (time 1787853207.09 total 60.06 last 60.48) - open/close 4201 (time 1787853217.09 total 70.07 last 59.49) total: 4500 open/close in 74.55 seconds: 60.36 ops/second - /unlink 1239 (time 1787853236.48 total 10.00 last 123.88) - /unlink 2385 (time 1787853246.50 total 20.02 last 114.35) - /unlink 3482 (time 1787853256.50 total 30.03 last 109.64) total: 4500 /unlink in 38.95 seconds: 115.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 537 (time 1787853281.02 total 10.03 last 53.52) - open/close 936 (time 1787853291.04 total 20.06 last 39.81) - open/close 1528 (time 1787853301.04 total 30.06 last 59.20) - open/close 2200 (time 1787853311.06 total 40.08 last 67.05) - open/close 2817 (time 1787853321.07 total 50.09 last 61.66) - open/close 3425 (time 1787853331.08 total 60.10 last 60.74) - open/close 4038 (time 1787853341.08 total 70.10 last 61.28) total: 4500 open/close in 77.10 seconds: 58.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1196 (time 1787853363.67 total 10.00 last 119.58) - /unlink 2326 (time 1787853373.68 total 20.01 last 112.95) - /unlink 3471 (time 1787853383.68 total 30.01 last 114.46) total: 4500 /unlink in 39.17 seconds: 114.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 663 (time 1787853408.53 total 10.02 last 66.17) - open/close 1246 (time 1787853418.53 total 20.02 last 58.27) - open/close 1868 (time 1787853428.54 total 30.03 last 62.14) - open/close 2559 (time 1787853438.55 total 40.04 last 69.07) - open/close 3155 (time 1787853448.55 total 50.04 last 59.57) - open/close 3737 (time 1787853458.56 total 60.06 last 58.13) - open/close 4355 (time 1787853468.56 total 70.06 last 61.80) total: 4500 open/close in 72.46 seconds: 62.10 ops/second - /unlink 1075 (time 1787853486.26 total 10.01 last 107.42) - /unlink 1929 (time 1787853496.27 total 20.01 last 85.34) - /unlink 2539 (time 1787853506.27 total 30.02 last 60.96) - /unlink 3373 (time 1787853516.28 total 40.02 last 83.37) - /unlink 4392 (time 1787853526.28 total 50.02 last 101.89) total: 4500 /unlink in 51.27 seconds: 87.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 548 (time 1787853542.07 total 10.01 last 54.75) - open/close 1036 (time 1787853552.07 total 20.01 last 48.80) - open/close 1511 (time 1787853562.07 total 30.01 last 47.47) - open/close 2105 (time 1787853572.10 total 40.04 last 59.26) - open/close 2657 (time 1787853582.11 total 50.05 last 55.13) - open/close 3181 (time 1787853592.13 total 60.07 last 52.29) - open/close 3801 (time 1787853602.14 total 70.08 last 61.94) - open/close 4399 (time 1787853612.14 total 80.09 last 59.77) total: 4500 open/close in 81.61 seconds: 55.14 ops/second - /unlink 1243 (time 1787853629.19 total 10.01 last 124.20) - /unlink 2376 (time 1787853639.20 total 20.01 last 113.26) - /unlink 3459 (time 1787853649.20 total 30.02 last 108.26) total: 4500 /unlink in 39.28 seconds: 114.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 654 (time 1787853674.62 total 10.00 last 65.40) - open/close 1202 (time 1787853684.64 total 20.02 last 54.70) - open/close 1784 (time 1787853694.64 total 30.02 last 58.20) - open/close 2416 (time 1787853704.66 total 40.04 last 63.07) - open/close 3068 (time 1787853714.68 total 50.06 last 65.09) - open/close 3603 (time 1787853724.68 total 60.06 last 53.48) - open/close 4108 (time 1787853734.69 total 70.07 last 50.44) total: 4500 open/close in 76.97 seconds: 58.47 ops/second - /unlink 1021 (time 1787853756.88 total 10.02 last 101.87) - /unlink 2116 (time 1787853766.88 total 20.02 last 109.49) - /unlink 3246 (time 1787853776.89 total 30.04 last 112.86) - /unlink 4481 (time 1787853786.89 total 40.04 last 123.48) total: 4500 /unlink in 40.16 seconds: 112.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 613 (time 1787853802.72 total 10.01 last 61.24) - open/close 1220 (time 1787853812.73 total 20.01 last 60.68) - open/close 1824 (time 1787853822.73 total 30.02 last 60.35) - open/close 2431 (time 1787853832.74 total 40.03 last 60.65) - open/close 2971 (time 1787853842.75 total 50.03 last 53.98) - open/close 3515 (time 1787853852.76 total 60.04 last 54.33) - open/close 4133 (time 1787853862.78 total 70.06 last 61.67) total: 4500 open/close in 75.78 seconds: 59.39 ops/second - /unlink 1125 (time 1787853884.07 total 10.01 last 112.44) - /unlink 2225 (time 1787853894.07 total 20.01 last 109.95) - /unlink 3422 (time 1787853904.08 total 30.01 last 119.65) total: 4500 /unlink in 38.60 seconds: 116.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 614 (time 1787853928.53 total 10.01 last 61.36) - open/close 1305 (time 1787853938.53 total 20.01 last 69.06) - open/close 2008 (time 1787853948.54 total 30.02 last 70.22) - open/close 2673 (time 1787853958.55 total 40.03 last 66.45) - open/close 3334 (time 1787853968.57 total 50.05 last 66.00) - open/close 3987 (time 1787853978.58 total 60.06 last 65.21) total: 4500 open/close in 67.63 seconds: 66.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1276 (time 1787854001.20 total 10.01 last 127.53) - /unlink 2596 (time 1787854011.22 total 20.02 last 131.81) - /unlink 3883 (time 1787854021.23 total 30.03 last 128.55) total: 4500 /unlink in 35.87 seconds: 125.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 693 (time 1787854041.37 total 10.01 last 69.21) - open/close 1351 (time 1787854051.37 total 20.02 last 65.76) - open/close 1987 (time 1787854061.37 total 30.02 last 63.59) - open/close 2622 (time 1787854071.38 total 40.02 last 63.48) - open/close 3350 (time 1787854081.39 total 50.04 last 72.67) - open/close 4066 (time 1787854091.40 total 60.05 last 71.56) total: 4500 open/close in 66.70 seconds: 67.47 ops/second - /unlink 1272 (time 1787854113.12 total 10.01 last 127.11) - /unlink 2522 (time 1787854123.13 total 20.01 last 124.98) - /unlink 3728 (time 1787854133.14 total 30.02 last 120.48) total: 4500 /unlink in 36.17 seconds: 124.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 629 (time 1787854154.86 total 10.01 last 62.84) - open/close 1249 (time 1787854164.87 total 20.02 last 61.96) - open/close 1934 (time 1787854174.88 total 30.02 last 68.45) - open/close 2621 (time 1787854184.89 total 40.03 last 68.64) - open/close 3260 (time 1787854194.89 total 50.04 last 63.87) - open/close 3934 (time 1787854204.89 total 60.04 last 67.39) total: 4500 open/close in 68.14 seconds: 66.04 ops/second - /unlink 1250 (time 1787854227.99 total 10.01 last 124.94) - /unlink 2536 (time 1787854238.00 total 20.01 last 128.56) - /unlink 3835 (time 1787854248.00 total 30.02 last 129.79) total: 4500 /unlink in 35.46 seconds: 126.91 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11369: 95319 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg409-server unloading modules via unload_modules_local on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing unload_modules_local oleg409-server: modules unloaded.