== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:20:24 (1786706424) Stopping clients: oleg139-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg139-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg139-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg139-server oleg139-server: oleg139-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing load_modules_local oleg139-server: Loading modules from /home/green/git/lustre-release/lustre oleg139-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg139-server: detected 4 online CPUs by sysfs oleg139-server: MODOPTS_LIBCFS= oleg139-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg139-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg139-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg139-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 612 (time 1786706645.77 total 10.02 last 61.09) - open/close 1379 (time 1786706655.77 total 20.02 last 76.68) - open/close 2125 (time 1786706665.78 total 30.03 last 74.56) - open/close 2742 (time 1786706675.78 total 40.03 last 61.68) - open/close 3452 (time 1786706685.80 total 50.05 last 70.87) - open/close 4083 (time 1786706695.81 total 60.06 last 63.03) total: 4500 open/close in 67.49 seconds: 66.67 ops/second - /unlink 1058 (time 1786706718.99 total 10.00 last 105.78) - /unlink 2246 (time 1786706728.99 total 20.00 last 118.79) - /unlink 3561 (time 1786706738.99 total 30.01 last 131.44) total: 4500 /unlink in 39.01 seconds: 115.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 541 (time 1786706764.21 total 10.01 last 54.04) - open/close 1158 (time 1786706774.22 total 20.02 last 61.66) - open/close 1843 (time 1786706784.22 total 30.02 last 68.46) - open/close 2516 (time 1786706794.23 total 40.04 last 67.22) - open/close 3132 (time 1786706804.24 total 50.04 last 61.57) - open/close 3826 (time 1786706814.28 total 60.08 last 69.14) - open/close 4451 (time 1786706824.30 total 70.10 last 62.35) total: 4500 open/close in 70.92 seconds: 63.45 ops/second - /unlink 1445 (time 1786706840.78 total 10.00 last 144.50) - /unlink 2738 (time 1786706850.79 total 20.01 last 129.21) - /unlink 3864 (time 1786706860.79 total 30.01 last 112.56) total: 4500 /unlink in 35.34 seconds: 127.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 701 (time 1786706880.59 total 10.01 last 70.03) - open/close 1445 (time 1786706890.59 total 20.02 last 74.34) - open/close 2278 (time 1786706900.60 total 30.03 last 83.21) - open/close 2918 (time 1786706910.61 total 40.03 last 63.97) - open/close 3566 (time 1786706920.61 total 50.04 last 64.79) - open/close 4299 (time 1786706930.63 total 60.06 last 73.14) total: 4500 open/close in 62.98 seconds: 71.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1237 (time 1786706948.87 total 10.01 last 123.62) - /unlink 2569 (time 1786706958.87 total 20.01 last 133.20) - /unlink 3828 (time 1786706968.88 total 30.01 last 125.86) total: 4500 /unlink in 35.08 seconds: 128.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 685 (time 1786706989.30 total 10.01 last 68.46) - open/close 1419 (time 1786706999.31 total 20.01 last 73.36) - open/close 2157 (time 1786707009.31 total 30.01 last 73.79) - open/close 2959 (time 1786707019.32 total 40.03 last 80.05) - open/close 3659 (time 1786707029.33 total 50.03 last 69.98) - open/close 4455 (time 1786707039.33 total 60.03 last 79.59) total: 4500 open/close in 60.75 seconds: 74.07 ops/second - /unlink 1294 (time 1786707054.76 total 10.01 last 129.31) - /unlink 2505 (time 1786707064.77 total 20.02 last 120.97) - /unlink 3586 (time 1786707074.78 total 30.03 last 107.95) total: 4500 /unlink in 38.47 seconds: 116.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 639 (time 1786707099.61 total 10.01 last 63.86) - open/close 1280 (time 1786707109.62 total 20.02 last 64.02) - open/close 1949 (time 1786707119.63 total 30.03 last 66.80) - open/close 2589 (time 1786707129.64 total 40.04 last 63.97) - open/close 3292 (time 1786707139.64 total 50.04 last 70.29) - open/close 4023 (time 1786707149.67 total 60.07 last 72.91) total: 4500 open/close in 68.23 seconds: 65.95 ops/second - /unlink 1089 (time 1786707175.53 total 10.00 last 108.86) - /unlink 2097 (time 1786707185.54 total 20.02 last 100.69) - /unlink 3298 (time 1786707195.54 total 30.02 last 120.04) - /unlink 4273 (time 1786707205.55 total 40.02 last 97.47) total: 4500 /unlink in 41.63 seconds: 108.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 739 (time 1786707221.86 total 10.02 last 73.77) - open/close 1393 (time 1786707231.87 total 20.03 last 65.32) - open/close 2187 (time 1786707241.90 total 30.05 last 79.21) - open/close 2938 (time 1786707251.90 total 40.06 last 75.04) - open/close 3602 (time 1786707261.90 total 50.06 last 66.39) - open/close 4322 (time 1786707271.91 total 60.07 last 71.98) total: 4500 open/close in 62.58 seconds: 71.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1438 (time 1786707289.73 total 10.00 last 143.77) - /unlink 2627 (time 1786707299.74 total 20.01 last 118.79) - /unlink 3787 (time 1786707309.74 total 30.01 last 115.98) total: 4500 /unlink in 36.20 seconds: 124.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 693 (time 1786707330.64 total 10.01 last 69.23) - open/close 1316 (time 1786707340.66 total 20.03 last 62.18) - open/close 1865 (time 1786707350.68 total 30.05 last 54.81) - open/close 2478 (time 1786707360.69 total 40.05 last 61.27) - open/close 3077 (time 1786707370.69 total 50.06 last 59.86) - open/close 3766 (time 1786707380.70 total 60.06 last 68.85) total: 4500 open/close in 69.15 seconds: 65.08 ops/second - /unlink 1380 (time 1786707404.87 total 10.00 last 137.97) - /unlink 2875 (time 1786707414.88 total 20.01 last 149.38) - /unlink 4236 (time 1786707424.89 total 30.01 last 136.05) total: 4500 /unlink in 31.89 seconds: 141.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 752 (time 1786707441.80 total 10.00 last 75.17) - open/close 1440 (time 1786707451.81 total 20.02 last 68.72) - open/close 2078 (time 1786707461.81 total 30.02 last 63.77) - open/close 2756 (time 1786707471.82 total 40.03 last 67.73) - open/close 3418 (time 1786707481.82 total 50.03 last 66.19) - open/close 4027 (time 1786707491.83 total 60.04 last 60.84) total: 4500 open/close in 67.65 seconds: 66.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1141 (time 1786707515.88 total 10.00 last 114.08) - /unlink 2266 (time 1786707525.88 total 20.01 last 112.46) - /unlink 3429 (time 1786707535.88 total 30.01 last 116.29) total: 4500 /unlink in 38.90 seconds: 115.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 593 (time 1786707560.16 total 10.00 last 59.27) - open/close 1161 (time 1786707570.16 total 20.01 last 56.75) - open/close 1845 (time 1786707580.17 total 30.02 last 68.35) - open/close 2575 (time 1786707590.18 total 40.03 last 72.95) - open/close 3183 (time 1786707600.18 total 50.03 last 60.76) - open/close 4032 (time 1786707610.19 total 60.04 last 84.87) total: 4500 open/close in 65.37 seconds: 68.83 ops/second - /unlink 1180 (time 1786707631.72 total 10.00 last 117.95) - /unlink 2489 (time 1786707641.72 total 20.01 last 130.89) - /unlink 3854 (time 1786707651.73 total 30.01 last 136.42) total: 4500 /unlink in 34.75 seconds: 129.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 640 (time 1786707671.69 total 10.01 last 63.96) - open/close 1161 (time 1786707681.70 total 20.02 last 52.06) - open/close 1633 (time 1786707691.72 total 30.03 last 47.11) - open/close 2198 (time 1786707701.72 total 40.04 last 56.49) - open/close 2655 (time 1786707711.75 total 50.06 last 45.57) - open/close 3268 (time 1786707721.81 total 60.12 last 60.94) - open/close 3966 (time 1786707731.81 total 70.13 last 69.77) total: 4500 open/close in 77.69 seconds: 57.92 ops/second - /unlink 1368 (time 1786707755.41 total 10.01 last 136.73) - /unlink 2905 (time 1786707765.41 total 20.01 last 153.66) - /unlink 4096 (time 1786707775.42 total 30.01 last 119.08) total: 4500 /unlink in 32.91 seconds: 136.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 749 (time 1786707794.04 total 10.04 last 74.60) - open/close 1568 (time 1786707804.05 total 20.05 last 81.84) - open/close 2383 (time 1786707814.06 total 30.05 last 81.44) - open/close 3084 (time 1786707824.07 total 40.07 last 70.01) - open/close 3734 (time 1786707834.07 total 50.07 last 64.99) - open/close 4373 (time 1786707844.07 total 60.07 last 63.89) total: 4500 open/close in 62.27 seconds: 72.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1253 (time 1786707861.12 total 10.01 last 125.18) - /unlink 2540 (time 1786707871.12 total 20.01 last 128.65) - /unlink 3799 (time 1786707881.13 total 30.02 last 125.84) total: 4500 /unlink in 35.75 seconds: 125.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 759 (time 1786707902.23 total 10.01 last 75.79) - open/close 1434 (time 1786707912.23 total 20.02 last 67.47) - open/close 2144 (time 1786707922.25 total 30.04 last 70.87) - open/close 2894 (time 1786707932.26 total 40.04 last 74.97) - open/close 3587 (time 1786707942.26 total 50.05 last 69.24) - open/close 4248 (time 1786707952.27 total 60.06 last 66.05) total: 4500 open/close in 64.66 seconds: 69.59 ops/second - /unlink 1227 (time 1786707971.84 total 10.01 last 122.59) - /unlink 2371 (time 1786707981.85 total 20.01 last 114.34) - /unlink 3810 (time 1786707991.85 total 30.02 last 143.82) total: 4500 /unlink in 34.64 seconds: 129.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 681 (time 1786708011.99 total 10.04 last 67.86) - open/close 1320 (time 1786708022.00 total 20.04 last 63.86) - open/close 2040 (time 1786708032.01 total 30.05 last 71.92) - open/close 2687 (time 1786708042.02 total 40.07 last 64.61) - open/close 3393 (time 1786708052.03 total 50.08 last 70.51) - open/close 4072 (time 1786708062.04 total 60.09 last 67.83) total: 4500 open/close in 69.50 seconds: 64.75 ops/second - /unlink 1203 (time 1786708087.99 total 10.00 last 120.26) - /unlink 2382 (time 1786708098.00 total 20.02 last 117.71) - /unlink 3640 (time 1786708108.01 total 30.02 last 125.76) total: 4500 /unlink in 36.03 seconds: 124.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 657 (time 1786708129.27 total 10.01 last 65.67) - open/close 1369 (time 1786708139.27 total 20.01 last 71.18) - open/close 2017 (time 1786708149.30 total 30.03 last 64.64) - open/close 2665 (time 1786708159.32 total 40.06 last 64.64) - open/close 3351 (time 1786708169.35 total 50.08 last 68.44) - open/close 4065 (time 1786708179.35 total 60.08 last 71.39) total: 4500 open/close in 66.14 seconds: 68.03 ops/second - /unlink 1155 (time 1786708200.98 total 10.00 last 115.48) - /unlink 2412 (time 1786708210.99 total 20.01 last 125.63) - /unlink 3518 (time 1786708220.99 total 30.01 last 110.54) - /unlink 4437 (time 1786708231.00 total 40.02 last 91.80) total: 4500 /unlink in 40.95 seconds: 109.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 750 (time 1786708249.48 total 10.00 last 74.98) - open/close 1383 (time 1786708259.49 total 20.02 last 63.19) - open/close 2065 (time 1786708269.51 total 30.03 last 68.10) - open/close 2775 (time 1786708279.51 total 40.04 last 70.99) - open/close 3428 (time 1786708289.53 total 50.05 last 65.19) - open/close 3952 (time 1786708299.56 total 60.08 last 52.24) - open/close 4327 (time 1786708309.58 total 70.11 last 37.40) total: 4500 open/close in 73.44 seconds: 61.27 ops/second - /unlink 1154 (time 1786708328.97 total 10.00 last 115.40) - /unlink 2514 (time 1786708338.99 total 20.02 last 135.73) - /unlink 3793 (time 1786708348.99 total 30.03 last 127.83) total: 4500 /unlink in 36.44 seconds: 123.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 782 (time 1786708371.10 total 10.01 last 78.15) - open/close 1464 (time 1786708381.11 total 20.01 last 68.17) - open/close 2217 (time 1786708391.11 total 30.02 last 75.25) - open/close 3015 (time 1786708401.11 total 40.02 last 79.80) - open/close 3729 (time 1786708411.11 total 50.02 last 71.39) - open/close 4363 (time 1786708421.12 total 60.03 last 63.34) total: 4500 open/close in 61.93 seconds: 72.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1277 (time 1786708437.72 total 10.00 last 127.68) - /unlink 2547 (time 1786708447.73 total 20.01 last 126.86) - /unlink 3721 (time 1786708457.74 total 30.02 last 117.27) total: 4500 /unlink in 36.53 seconds: 123.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 551 (time 1786708480.47 total 10.01 last 55.03) - open/close 964 (time 1786708490.48 total 20.02 last 41.27) - open/close 1443 (time 1786708500.49 total 30.03 last 47.85) - open/close 1999 (time 1786708510.50 total 40.04 last 55.57) - open/close 2600 (time 1786708520.51 total 50.04 last 60.05) - open/close 3320 (time 1786708530.51 total 60.05 last 71.95) - open/close 4051 (time 1786708540.51 total 70.05 last 73.10) total: 4500 open/close in 77.45 seconds: 58.10 ops/second - /unlink 1433 (time 1786708563.36 total 10.00 last 143.27) - /unlink 2526 (time 1786708573.37 total 20.00 last 109.27) - /unlink 3755 (time 1786708583.37 total 30.01 last 122.90) total: 4500 /unlink in 35.83 seconds: 125.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 506 (time 1786708604.78 total 10.00 last 50.59) - open/close 1143 (time 1786708614.78 total 20.00 last 63.70) - open/close 1930 (time 1786708624.78 total 30.01 last 78.66) - open/close 2574 (time 1786708634.79 total 40.02 last 64.34) - open/close 3223 (time 1786708644.79 total 50.02 last 64.89) - open/close 3942 (time 1786708654.80 total 60.02 last 71.86) total: 4500 open/close in 68.65 seconds: 65.55 ops/second - /unlink 1218 (time 1786708678.50 total 10.02 last 121.59) - /unlink 2447 (time 1786708688.51 total 20.02 last 122.87) - /unlink 3656 (time 1786708698.51 total 30.02 last 120.88) total: 4500 /unlink in 37.50 seconds: 120.01 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95559 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg139-server unloading modules via unload_modules_local on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing unload_modules_local oleg139-server: modules unloaded.