== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:02:55 (1785402175) Stopping clients: oleg117-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg117-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg117-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg117-server oleg117-server: oleg117-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing load_modules_local oleg117-server: Loading modules from /home/green/git/lustre-release/lustre oleg117-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg117-server: detected 4 online CPUs by sysfs oleg117-server: MODOPTS_LIBCFS= oleg117-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg117-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg117-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg117-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 670 (time 1785402448.68 total 10.01 last 66.96) - open/close 1116 (time 1785402458.71 total 20.03 last 44.50) - open/close 1744 (time 1785402468.73 total 30.05 last 62.67) - open/close 2340 (time 1785402478.73 total 40.05 last 59.58) - open/close 2957 (time 1785402488.75 total 50.07 last 61.58) - open/close 3532 (time 1785402498.76 total 60.08 last 57.46) - open/close 4030 (time 1785402508.76 total 70.09 last 49.76) total: 4500 open/close in 75.91 seconds: 59.28 ops/second - /unlink 1372 (time 1785402530.67 total 10.01 last 137.08) - /unlink 2217 (time 1785402540.68 total 20.02 last 84.43) - /unlink 3100 (time 1785402550.69 total 30.02 last 88.24) - /unlink 3929 (time 1785402560.69 total 40.03 last 82.84) total: 4500 /unlink in 47.29 seconds: 95.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 394 (time 1785402585.94 total 10.01 last 39.37) - open/close 985 (time 1785402595.94 total 20.01 last 59.08) - open/close 1526 (time 1785402605.95 total 30.02 last 54.06) - open/close 1984 (time 1785402615.98 total 40.05 last 45.67) - open/close 2469 (time 1785402625.99 total 50.06 last 48.44) - open/close 3010 (time 1785402636.01 total 60.08 last 53.98) - open/close 3489 (time 1785402646.02 total 70.09 last 47.85) - open/close 3973 (time 1785402656.02 total 80.10 last 48.38) total: 4500 open/close in 85.31 seconds: 52.75 ops/second - /unlink 1070 (time 1785402675.98 total 10.01 last 106.94) - /unlink 2242 (time 1785402685.99 total 20.02 last 117.07) - /unlink 3611 (time 1785402695.99 total 30.02 last 136.89) total: 4500 /unlink in 38.55 seconds: 116.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 603 (time 1785402720.73 total 10.00 last 60.30) - open/close 1014 (time 1785402730.74 total 20.01 last 41.06) - open/close 1623 (time 1785402740.75 total 30.02 last 60.85) - open/close 2194 (time 1785402750.75 total 40.02 last 57.08) - open/close 2670 (time 1785402760.75 total 50.03 last 47.59) - open/close 3175 (time 1785402770.77 total 60.04 last 50.41) - open/close 3764 (time 1785402780.77 total 70.05 last 58.88) - open/close 4225 (time 1785402790.79 total 80.06 last 46.03) total: 4500 open/close in 85.17 seconds: 52.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1433 (time 1785402811.60 total 10.00 last 143.25) - /unlink 2462 (time 1785402821.62 total 20.02 last 102.71) - /unlink 3301 (time 1785402831.63 total 30.03 last 83.83) - /unlink 4095 (time 1785402841.63 total 40.04 last 79.34) total: 4500 /unlink in 44.03 seconds: 102.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 594 (time 1785402863.48 total 10.01 last 59.37) - open/close 1391 (time 1785402873.48 total 20.01 last 79.66) - open/close 2250 (time 1785402883.49 total 30.02 last 85.82) - open/close 2642 (time 1785402893.51 total 40.04 last 39.12) - open/close 2984 (time 1785402903.61 total 50.14 last 33.86) - open/close 3378 (time 1785402913.62 total 60.15 last 39.38) - open/close 3895 (time 1785402923.63 total 70.16 last 51.63) - open/close 4334 (time 1785402933.65 total 80.18 last 43.82) total: 4500 open/close in 83.35 seconds: 53.99 ops/second - /unlink 1167 (time 1785402952.42 total 10.01 last 116.61) - /unlink 2253 (time 1785402962.42 total 20.01 last 108.56) - /unlink 3313 (time 1785402972.43 total 30.01 last 105.96) - /unlink 4158 (time 1785402982.43 total 40.02 last 84.46) total: 4500 /unlink in 43.30 seconds: 103.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 486 (time 1785403002.01 total 10.01 last 48.55) - open/close 891 (time 1785403012.02 total 20.02 last 40.48) - open/close 1485 (time 1785403022.03 total 30.02 last 59.35) - open/close 2148 (time 1785403032.05 total 40.05 last 66.13) - open/close 2690 (time 1785403042.06 total 50.05 last 54.17) - open/close 3303 (time 1785403052.07 total 60.06 last 61.25) - open/close 3875 (time 1785403062.07 total 70.07 last 57.17) total: 4500 open/close in 78.30 seconds: 57.47 ops/second - /unlink 1094 (time 1785403087.59 total 10.00 last 109.38) - /unlink 1915 (time 1785403097.60 total 20.01 last 82.01) - /unlink 2702 (time 1785403107.61 total 30.03 last 78.57) - /unlink 3653 (time 1785403117.65 total 40.07 last 94.75) total: 4500 /unlink in 46.98 seconds: 95.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 538 (time 1785403141.69 total 10.00 last 53.80) - open/close 889 (time 1785403151.70 total 20.01 last 35.08) - open/close 1352 (time 1785403161.72 total 30.02 last 46.23) - open/close 1941 (time 1785403171.73 total 40.04 last 58.80) - open/close 2369 (time 1785403181.73 total 50.04 last 42.79) - open/close 2749 (time 1785403191.75 total 60.06 last 37.94) - open/close 3382 (time 1785403201.75 total 70.06 last 63.28) - open/close 3908 (time 1785403211.76 total 80.07 last 52.56) total: 4500 open/close in 90.04 seconds: 49.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 865 (time 1785403237.83 total 10.00 last 86.48) - /unlink 2229 (time 1785403247.83 total 20.00 last 136.37) - /unlink 3742 (time 1785403257.83 total 30.01 last 151.29) total: 4500 /unlink in 37.17 seconds: 121.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 497 (time 1785403281.70 total 10.02 last 49.60) - open/close 884 (time 1785403291.72 total 20.03 last 38.64) - open/close 1306 (time 1785403301.75 total 30.06 last 42.09) - open/close 1716 (time 1785403311.75 total 40.07 last 40.97) - open/close 2348 (time 1785403321.77 total 50.09 last 63.08) - open/close 3032 (time 1785403331.78 total 60.09 last 68.36) - open/close 3594 (time 1785403341.78 total 70.10 last 56.17) - open/close 4048 (time 1785403351.81 total 80.12 last 45.30) - open/close 4487 (time 1785403361.81 total 90.13 last 43.89) total: 4500 open/close in 90.44 seconds: 49.76 ops/second - /unlink 975 (time 1785403377.96 total 10.00 last 97.47) - /unlink 1965 (time 1785403387.99 total 20.04 last 98.68) - /unlink 3001 (time 1785403397.99 total 30.04 last 103.59) - /unlink 4309 (time 1785403408.00 total 40.04 last 130.75) total: 4500 /unlink in 42.26 seconds: 106.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 497 (time 1785403428.23 total 10.01 last 49.67) - open/close 1101 (time 1785403438.25 total 20.03 last 60.26) - open/close 1588 (time 1785403448.27 total 30.04 last 48.65) - open/close 2290 (time 1785403458.29 total 40.07 last 70.03) - open/close 2860 (time 1785403468.31 total 50.08 last 56.90) - open/close 3332 (time 1785403478.35 total 60.13 last 46.99) - open/close 3837 (time 1785403488.35 total 70.13 last 50.50) total: 4500 open/close in 79.62 seconds: 56.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1202 (time 1785403513.16 total 10.01 last 120.13) - /unlink 2520 (time 1785403523.16 total 20.01 last 131.76) - /unlink 4012 (time 1785403533.17 total 30.02 last 149.09) total: 4500 /unlink in 34.99 seconds: 128.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 519 (time 1785403553.72 total 10.01 last 51.85) - open/close 1066 (time 1785403563.75 total 20.04 last 54.54) - open/close 1469 (time 1785403573.76 total 30.05 last 40.26) - open/close 2033 (time 1785403583.78 total 40.07 last 56.32) - open/close 2598 (time 1785403593.80 total 50.09 last 56.36) - open/close 3119 (time 1785403603.83 total 60.12 last 51.94) - open/close 3814 (time 1785403613.83 total 70.12 last 69.49) - open/close 4469 (time 1785403623.86 total 80.15 last 65.31) total: 4500 open/close in 81.06 seconds: 55.51 ops/second - /unlink 1311 (time 1785403639.71 total 10.01 last 131.02) - /unlink 2235 (time 1785403649.72 total 20.01 last 92.35) - /unlink 3236 (time 1785403659.73 total 30.02 last 100.03) - /unlink 4421 (time 1785403669.73 total 40.03 last 118.42) total: 4500 /unlink in 41.63 seconds: 108.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 646 (time 1785403687.47 total 10.01 last 64.52) - open/close 1595 (time 1785403697.49 total 20.03 last 94.76) - open/close 2098 (time 1785403707.50 total 30.03 last 50.27) - open/close 2541 (time 1785403717.50 total 40.04 last 44.29) - open/close 2854 (time 1785403727.51 total 50.05 last 31.25) - open/close 3153 (time 1785403737.52 total 60.06 last 29.87) - open/close 3661 (time 1785403747.53 total 70.07 last 50.78) - open/close 4269 (time 1785403757.53 total 80.07 last 60.79) total: 4500 open/close in 83.71 seconds: 53.76 ops/second - /unlink 1438 (time 1785403776.73 total 10.00 last 143.79) - /unlink 2740 (time 1785403786.74 total 20.01 last 130.09) - /unlink 3930 (time 1785403796.75 total 30.02 last 118.88) total: 4500 /unlink in 37.39 seconds: 120.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 623 (time 1785403822.82 total 10.01 last 62.26) - open/close 1144 (time 1785403832.85 total 20.04 last 51.94) - open/close 1720 (time 1785403842.85 total 30.04 last 57.56) - open/close 2331 (time 1785403852.86 total 40.05 last 61.09) - open/close 3014 (time 1785403862.87 total 50.06 last 68.17) - open/close 3675 (time 1785403872.89 total 60.08 last 66.00) - open/close 4173 (time 1785403882.98 total 70.17 last 49.36) total: 4500 open/close in 74.45 seconds: 60.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1117 (time 1785403902.82 total 10.00 last 111.66) - /unlink 2513 (time 1785403912.83 total 20.01 last 139.48) - /unlink 3523 (time 1785403922.83 total 30.02 last 100.97) total: 4500 /unlink in 39.39 seconds: 114.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 442 (time 1785403947.97 total 10.02 last 44.12) - open/close 970 (time 1785403958.00 total 20.04 last 52.66) - open/close 1447 (time 1785403968.01 total 30.05 last 47.65) - open/close 1813 (time 1785403978.01 total 40.06 last 36.58) - open/close 2516 (time 1785403988.02 total 50.07 last 70.26) - open/close 3019 (time 1785403998.03 total 60.08 last 50.22) - open/close 3545 (time 1785404008.04 total 70.08 last 52.59) - open/close 3978 (time 1785404018.05 total 80.10 last 43.23) - open/close 4478 (time 1785404028.07 total 90.12 last 49.91) total: 4500 open/close in 90.85 seconds: 49.53 ops/second - /unlink 808 (time 1785404046.00 total 10.00 last 80.79) - /unlink 1791 (time 1785404056.00 total 20.01 last 98.23) - /unlink 2898 (time 1785404066.00 total 30.01 last 110.68) - /unlink 4230 (time 1785404076.01 total 40.02 last 133.13) total: 4500 /unlink in 41.95 seconds: 107.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 634 (time 1785404092.94 total 10.03 last 63.20) - open/close 1148 (time 1785404102.95 total 20.04 last 51.36) - open/close 1806 (time 1785404112.95 total 30.05 last 65.76) - open/close 2240 (time 1785404122.99 total 40.08 last 43.24) - open/close 2740 (time 1785404132.99 total 50.08 last 50.00) - open/close 3232 (time 1785404143.00 total 60.09 last 49.15) - open/close 3773 (time 1785404153.03 total 70.12 last 53.96) - open/close 4272 (time 1785404163.03 total 80.12 last 49.88) total: 4500 open/close in 85.19 seconds: 52.82 ops/second - /unlink 1047 (time 1785404186.74 total 10.02 last 104.48) - /unlink 1924 (time 1785404196.76 total 20.03 last 87.58) - /unlink 3555 (time 1785404206.76 total 30.03 last 163.09) total: 4500 /unlink in 33.92 seconds: 132.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 586 (time 1785404226.91 total 10.01 last 58.57) - open/close 1461 (time 1785404236.92 total 20.01 last 87.45) - open/close 2180 (time 1785404246.92 total 30.02 last 71.86) - open/close 2701 (time 1785404256.96 total 40.05 last 51.93) - open/close 3134 (time 1785404266.98 total 50.07 last 43.20) - open/close 3730 (time 1785404276.98 total 60.07 last 59.60) total: 4500 open/close in 69.82 seconds: 64.45 ops/second - /unlink 1076 (time 1785404302.16 total 10.01 last 107.52) - /unlink 2013 (time 1785404312.17 total 20.02 last 93.55) - /unlink 3070 (time 1785404322.18 total 30.03 last 105.66) - /unlink 3744 (time 1785404332.20 total 40.05 last 67.23) total: 4500 /unlink in 48.74 seconds: 92.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 422 (time 1785404358.19 total 10.01 last 42.15) - open/close 963 (time 1785404368.19 total 20.01 last 54.08) - open/close 1461 (time 1785404378.20 total 30.03 last 49.73) - open/close 1853 (time 1785404388.20 total 40.03 last 39.20) - open/close 2252 (time 1785404398.25 total 50.07 last 39.72) - open/close 2745 (time 1785404408.27 total 60.09 last 49.22) - open/close 3489 (time 1785404418.27 total 70.10 last 74.33) - open/close 4248 (time 1785404428.30 total 80.12 last 75.72) total: 4500 open/close in 83.36 seconds: 53.98 ops/second - /unlink 885 (time 1785404447.68 total 10.01 last 88.43) - /unlink 1563 (time 1785404457.69 total 20.02 last 67.75) - /unlink 2253 (time 1785404467.69 total 30.02 last 69.00) - /unlink 3042 (time 1785404477.70 total 40.03 last 78.79) - /unlink 3901 (time 1785404487.73 total 50.05 last 85.72) total: 4500 /unlink in 54.32 seconds: 82.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 753 (time 1785404509.03 total 10.01 last 75.23) - open/close 1247 (time 1785404519.04 total 20.02 last 49.37) - open/close 1828 (time 1785404529.05 total 30.02 last 58.05) - open/close 2362 (time 1785404539.08 total 40.06 last 53.22) - open/close 2979 (time 1785404549.08 total 50.06 last 61.69) - open/close 3618 (time 1785404559.08 total 60.06 last 63.89) - open/close 4168 (time 1785404569.09 total 70.07 last 54.95) total: 4500 open/close in 77.26 seconds: 58.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 829 (time 1785404592.61 total 10.03 last 82.68) - /unlink 1514 (time 1785404602.62 total 20.04 last 68.43) - /unlink 2577 (time 1785404612.63 total 30.04 last 106.26) - /unlink 3812 (time 1785404622.63 total 40.04 last 123.49) total: 4500 /unlink in 44.62 seconds: 100.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 937 (time 1785404642.22 total 10.02 last 93.54) - open/close 1402 (time 1785404652.22 total 20.02 last 46.49) - open/close 1955 (time 1785404662.37 total 30.17 last 54.46) - open/close 2425 (time 1785404672.41 total 40.22 last 46.80) - open/close 2970 (time 1785404682.43 total 50.23 last 54.44) - open/close 3421 (time 1785404692.43 total 60.23 last 45.07) - open/close 4036 (time 1785404702.43 total 70.23 last 61.50) total: 4500 open/close in 77.63 seconds: 57.96 ops/second - /unlink 839 (time 1785404727.71 total 10.01 last 83.84) - /unlink 1774 (time 1785404737.74 total 20.03 last 93.26) - /unlink 2406 (time 1785404747.74 total 30.03 last 63.19) - /unlink 3264 (time 1785404757.74 total 40.04 last 85.77) - /unlink 4246 (time 1785404767.75 total 50.04 last 98.12) total: 4500 /unlink in 52.85 seconds: 85.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 388 (time 1785404787.79 total 10.02 last 38.72) - open/close 838 (time 1785404797.79 total 20.02 last 44.99) - open/close 1278 (time 1785404807.79 total 30.03 last 43.99) - open/close 1694 (time 1785404817.80 total 40.03 last 41.59) - open/close 2184 (time 1785404827.82 total 50.05 last 48.90) - open/close 2614 (time 1785404837.82 total 60.05 last 43.00) - open/close 3040 (time 1785404847.85 total 70.08 last 42.46) - open/close 3689 (time 1785404857.87 total 80.10 last 64.78) - open/close 4203 (time 1785404867.91 total 90.14 last 51.21) total: 4500 open/close in 94.56 seconds: 47.59 ops/second - /unlink 901 (time 1785404887.56 total 10.01 last 90.05) - /unlink 1935 (time 1785404897.56 total 20.01 last 103.40) - /unlink 3123 (time 1785404907.57 total 30.01 last 118.77) - /unlink 4084 (time 1785404917.57 total 40.01 last 96.07) total: 4500 /unlink in 44.14 seconds: 101.95 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11297: 95533 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server unloading modules via unload_modules_local on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing unload_modules_local oleg117-server: modules unloaded.