== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:36:58 (1786750618) Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg132-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg132-server oleg132-server: oleg132-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 737 (time 1786750834.47 total 10.01 last 73.65) - open/close 1523 (time 1786750844.48 total 20.02 last 78.52) - open/close 2271 (time 1786750854.48 total 30.02 last 74.79) - open/close 3067 (time 1786750864.48 total 40.02 last 79.57) - open/close 3882 (time 1786750874.50 total 50.03 last 81.40) total: 4500 open/close in 59.00 seconds: 76.28 ops/second - /unlink 1322 (time 1786750898.19 total 10.00 last 132.18) - /unlink 2670 (time 1786750908.19 total 20.01 last 134.74) - /unlink 4037 (time 1786750918.20 total 30.01 last 136.61) total: 4500 /unlink in 34.01 seconds: 132.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 747 (time 1786750938.05 total 10.01 last 74.62) - open/close 1548 (time 1786750948.05 total 20.02 last 80.05) - open/close 2282 (time 1786750958.06 total 30.03 last 73.32) - open/close 2898 (time 1786750968.07 total 40.04 last 61.55) - open/close 3430 (time 1786750978.08 total 50.04 last 53.17) - open/close 3979 (time 1786750988.08 total 60.04 last 54.89) total: 4500 open/close in 67.61 seconds: 66.56 ops/second - /unlink 1382 (time 1786751010.70 total 10.00 last 138.19) - /unlink 2629 (time 1786751020.70 total 20.00 last 124.69) - /unlink 4065 (time 1786751030.71 total 30.02 last 143.41) total: 4500 /unlink in 32.95 seconds: 136.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 704 (time 1786751049.63 total 10.00 last 70.40) - open/close 1423 (time 1786751059.64 total 20.01 last 71.82) - open/close 2130 (time 1786751069.64 total 30.01 last 70.69) - open/close 2923 (time 1786751079.66 total 40.03 last 79.19) - open/close 3610 (time 1786751089.68 total 50.05 last 68.57) - open/close 4260 (time 1786751099.68 total 60.05 last 64.97) total: 4500 open/close in 62.34 seconds: 72.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1119 (time 1786751117.12 total 10.00 last 111.88) - /unlink 2240 (time 1786751127.12 total 20.00 last 112.07) - /unlink 3592 (time 1786751137.12 total 30.01 last 135.19) total: 4500 /unlink in 36.93 seconds: 121.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 772 (time 1786751160.42 total 10.01 last 77.14) - open/close 1511 (time 1786751170.43 total 20.01 last 73.86) - open/close 2288 (time 1786751180.44 total 30.03 last 77.61) - open/close 3046 (time 1786751190.45 total 40.03 last 75.74) - open/close 3789 (time 1786751200.46 total 50.04 last 74.24) total: 4500 open/close in 59.40 seconds: 75.76 ops/second - /unlink 1412 (time 1786751224.14 total 10.00 last 141.13) - /unlink 2811 (time 1786751234.14 total 20.01 last 139.86) - /unlink 4157 (time 1786751244.15 total 30.01 last 134.54) total: 4500 /unlink in 32.43 seconds: 138.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 690 (time 1786751261.96 total 10.01 last 68.96) - open/close 1585 (time 1786751271.96 total 20.01 last 89.46) - open/close 2323 (time 1786751281.97 total 30.01 last 73.77) - open/close 2978 (time 1786751291.97 total 40.02 last 65.46) - open/close 3600 (time 1786751301.98 total 50.02 last 62.17) - open/close 4248 (time 1786751312.00 total 60.05 last 64.66) total: 4500 open/close in 63.52 seconds: 70.84 ops/second - /unlink 1249 (time 1786751330.51 total 10.00 last 124.87) - /unlink 2500 (time 1786751340.52 total 20.00 last 125.08) - /unlink 3785 (time 1786751350.53 total 30.02 last 128.31) total: 4500 /unlink in 35.16 seconds: 128.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 702 (time 1786751370.69 total 10.00 last 70.20) - open/close 1374 (time 1786751380.70 total 20.01 last 67.12) - open/close 2107 (time 1786751390.71 total 30.02 last 73.25) - open/close 2780 (time 1786751400.71 total 40.03 last 67.26) - open/close 3491 (time 1786751410.74 total 50.06 last 70.89) - open/close 4065 (time 1786751420.79 total 60.10 last 57.15) total: 4500 open/close in 65.81 seconds: 68.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1163 (time 1786751441.05 total 10.00 last 116.27) - /unlink 2458 (time 1786751451.05 total 20.01 last 129.45) - /unlink 3827 (time 1786751461.06 total 30.01 last 136.83) total: 4500 /unlink in 34.88 seconds: 129.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 721 (time 1786751481.33 total 10.01 last 72.04) - open/close 1324 (time 1786751491.34 total 20.02 last 60.24) - open/close 1880 (time 1786751501.35 total 30.03 last 55.54) - open/close 2504 (time 1786751511.37 total 40.05 last 62.30) - open/close 3044 (time 1786751521.37 total 50.05 last 53.96) - open/close 3592 (time 1786751531.38 total 60.06 last 54.78) - open/close 4339 (time 1786751541.39 total 70.07 last 74.62) total: 4500 open/close in 72.21 seconds: 62.32 ops/second - /unlink 1209 (time 1786751559.89 total 10.02 last 120.66) - /unlink 2432 (time 1786751569.89 total 20.02 last 122.28) - /unlink 3750 (time 1786751579.90 total 30.03 last 131.68) total: 4500 /unlink in 36.23 seconds: 124.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 479 (time 1786751602.86 total 10.02 last 47.82) - open/close 1005 (time 1786751612.87 total 20.02 last 52.56) - open/close 1649 (time 1786751622.87 total 30.03 last 64.39) - open/close 2343 (time 1786751632.87 total 40.03 last 69.36) - open/close 3029 (time 1786751642.89 total 50.05 last 68.48) - open/close 3672 (time 1786751652.90 total 60.06 last 64.22) - open/close 4472 (time 1786751662.92 total 70.08 last 79.86) total: 4500 open/close in 70.74 seconds: 63.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1292 (time 1786751678.82 total 10.01 last 129.06) - /unlink 2795 (time 1786751688.83 total 20.02 last 150.19) - /unlink 3885 (time 1786751698.84 total 30.03 last 108.88) total: 4500 /unlink in 36.01 seconds: 124.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 689 (time 1786751720.91 total 10.00 last 68.90) - open/close 1259 (time 1786751730.91 total 20.01 last 56.97) - open/close 1913 (time 1786751740.92 total 30.02 last 65.33) - open/close 2682 (time 1786751750.93 total 40.03 last 76.82) - open/close 3450 (time 1786751760.94 total 50.03 last 76.76) - open/close 4224 (time 1786751770.94 total 60.04 last 77.36) total: 4500 open/close in 63.46 seconds: 70.92 ops/second - /unlink 1534 (time 1786751789.27 total 10.00 last 153.36) - /unlink 2969 (time 1786751799.27 total 20.00 last 143.49) - /unlink 4286 (time 1786751809.28 total 30.01 last 131.60) total: 4500 /unlink in 31.99 seconds: 140.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 601 (time 1786751827.27 total 10.01 last 60.07) - open/close 1303 (time 1786751837.28 total 20.01 last 70.18) - open/close 2128 (time 1786751847.28 total 30.01 last 82.49) - open/close 2872 (time 1786751857.28 total 40.01 last 74.40) - open/close 3567 (time 1786751867.28 total 50.01 last 69.48) - open/close 4220 (time 1786751877.29 total 60.03 last 65.21) total: 4500 open/close in 64.06 seconds: 70.25 ops/second - /unlink 1417 (time 1786751896.84 total 10.01 last 141.63) - /unlink 2541 (time 1786751906.85 total 20.01 last 112.33) - /unlink 3950 (time 1786751916.85 total 30.01 last 140.87) total: 4500 /unlink in 33.46 seconds: 134.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 712 (time 1786751936.67 total 10.02 last 71.02) - open/close 1381 (time 1786751946.67 total 20.03 last 66.87) - open/close 2145 (time 1786751956.67 total 30.03 last 76.38) - open/close 2854 (time 1786751966.68 total 40.04 last 70.84) - open/close 3529 (time 1786751976.69 total 50.04 last 67.48) - open/close 4255 (time 1786751986.69 total 60.05 last 72.58) total: 4500 open/close in 62.70 seconds: 71.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1457 (time 1786752004.26 total 10.00 last 145.67) - /unlink 2644 (time 1786752014.26 total 20.00 last 118.67) - /unlink 4060 (time 1786752024.27 total 30.01 last 141.50) total: 4500 /unlink in 33.32 seconds: 135.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 798 (time 1786752042.82 total 10.02 last 79.67) - open/close 1483 (time 1786752052.82 total 20.02 last 68.50) - open/close 1940 (time 1786752062.83 total 30.03 last 45.66) - open/close 2692 (time 1786752072.83 total 40.03 last 75.19) - open/close 3432 (time 1786752082.84 total 50.03 last 73.99) - open/close 4078 (time 1786752092.84 total 60.04 last 64.56) total: 4500 open/close in 65.83 seconds: 68.36 ops/second - /unlink 1406 (time 1786752114.57 total 10.00 last 140.56) - /unlink 2832 (time 1786752124.57 total 20.00 last 142.58) - /unlink 4248 (time 1786752134.57 total 30.01 last 141.53) total: 4500 /unlink in 32.21 seconds: 139.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 685 (time 1786752152.58 total 10.02 last 68.39) - open/close 1444 (time 1786752162.59 total 20.03 last 75.81) - open/close 2102 (time 1786752172.60 total 30.03 last 65.76) - open/close 2935 (time 1786752182.60 total 40.04 last 83.28) - open/close 3692 (time 1786752192.61 total 50.04 last 75.65) - open/close 4478 (time 1786752202.61 total 60.05 last 78.55) total: 4500 open/close in 60.33 seconds: 74.59 ops/second - /unlink 1391 (time 1786752216.90 total 10.01 last 138.98) - /unlink 2772 (time 1786752226.90 total 20.01 last 138.07) - /unlink 4148 (time 1786752236.91 total 30.02 last 137.48) total: 4500 /unlink in 32.76 seconds: 137.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 780 (time 1786752255.93 total 10.00 last 77.98) - open/close 1572 (time 1786752265.93 total 20.01 last 79.17) - open/close 2272 (time 1786752275.94 total 30.02 last 69.93) - open/close 3005 (time 1786752285.95 total 40.02 last 73.27) - open/close 3861 (time 1786752295.95 total 50.02 last 85.56) total: 4500 open/close in 59.97 seconds: 75.04 ops/second - /unlink 1352 (time 1786752320.90 total 10.00 last 135.19) - /unlink 2709 (time 1786752330.90 total 20.00 last 135.68) - /unlink 4107 (time 1786752340.90 total 30.00 last 139.79) total: 4500 /unlink in 32.52 seconds: 138.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 782 (time 1786752357.21 total 10.01 last 78.12) - open/close 1620 (time 1786752367.22 total 20.02 last 83.74) - open/close 2224 (time 1786752377.23 total 30.02 last 60.37) - open/close 2874 (time 1786752387.23 total 40.03 last 64.96) - open/close 3553 (time 1786752397.24 total 50.04 last 67.85) - open/close 4219 (time 1786752407.24 total 60.04 last 66.59) total: 4500 open/close in 63.80 seconds: 70.53 ops/second - /unlink 1336 (time 1786752425.75 total 10.00 last 133.57) - /unlink 2732 (time 1786752435.75 total 20.00 last 139.56) - /unlink 4350 (time 1786752445.75 total 30.01 last 161.75) total: 4500 /unlink in 31.78 seconds: 141.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 733 (time 1786752462.39 total 10.01 last 73.22) - open/close 1422 (time 1786752472.39 total 20.01 last 68.87) - open/close 2107 (time 1786752482.39 total 30.01 last 68.50) - open/close 2774 (time 1786752492.40 total 40.03 last 66.62) - open/close 3563 (time 1786752502.41 total 50.04 last 78.82) - open/close 4244 (time 1786752512.43 total 60.05 last 68.01) total: 4500 open/close in 64.65 seconds: 69.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1317 (time 1786752532.14 total 10.01 last 131.60) - /unlink 2646 (time 1786752542.15 total 20.02 last 132.73) - /unlink 3627 (time 1786752552.15 total 30.02 last 98.06) total: 4500 /unlink in 36.61 seconds: 122.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 717 (time 1786752574.10 total 10.00 last 71.69) - open/close 1434 (time 1786752584.12 total 20.02 last 71.54) - open/close 2114 (time 1786752594.15 total 30.05 last 67.83) - open/close 2782 (time 1786752604.16 total 40.06 last 66.74) - open/close 3559 (time 1786752614.17 total 50.07 last 77.62) - open/close 4131 (time 1786752624.17 total 60.07 last 57.16) total: 4500 open/close in 67.16 seconds: 67.00 ops/second - /unlink 1537 (time 1786752646.23 total 10.00 last 153.65) - /unlink 2874 (time 1786752656.23 total 20.01 last 133.66) - /unlink 4343 (time 1786752666.24 total 30.01 last 146.85) total: 4500 /unlink in 31.15 seconds: 144.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 854 (time 1786752681.61 total 10.01 last 85.35) - open/close 1659 (time 1786752691.62 total 20.02 last 80.38) - open/close 2482 (time 1786752701.63 total 30.03 last 82.25) - open/close 3221 (time 1786752711.64 total 40.04 last 73.83) - open/close 3966 (time 1786752721.64 total 50.04 last 74.44) total: 4500 open/close in 56.32 seconds: 79.90 ops/second - /unlink 1561 (time 1786752741.30 total 10.01 last 155.93) - /unlink 3013 (time 1786752751.31 total 20.02 last 145.05) total: 4500 /unlink in 29.57 seconds: 152.20 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95283 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server unloading modules via unload_modules_local on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local oleg132-server: modules unloaded.