== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:34:13 (1786750453) Stopping clients: oleg103-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg103-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg103-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg103-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg103-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg103-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg103-server oleg103-server: oleg103-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing load_modules_local oleg103-server: Loading modules from /home/green/git/lustre-release/lustre oleg103-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg103-server: detected 4 online CPUs by sysfs oleg103-server: MODOPTS_LIBCFS= oleg103-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg103-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg103-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg103-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 766 (time 1786750685.27 total 10.01 last 76.54) - open/close 1424 (time 1786750695.30 total 20.03 last 65.63) - open/close 2145 (time 1786750705.32 total 30.05 last 71.95) - open/close 2832 (time 1786750715.32 total 40.06 last 68.68) - open/close 3578 (time 1786750725.33 total 50.07 last 74.51) - open/close 4218 (time 1786750735.34 total 60.08 last 63.94) total: 4500 open/close in 64.53 seconds: 69.73 ops/second - /unlink 1198 (time 1786750754.74 total 10.00 last 119.77) - /unlink 2239 (time 1786750764.74 total 20.01 last 104.04) - /unlink 3633 (time 1786750774.74 total 30.01 last 139.38) total: 4500 /unlink in 36.36 seconds: 123.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 647 (time 1786750797.37 total 10.04 last 64.42) - open/close 1208 (time 1786750807.41 total 20.08 last 55.88) - open/close 1874 (time 1786750817.41 total 30.09 last 66.59) - open/close 2599 (time 1786750827.42 total 40.09 last 72.48) - open/close 3306 (time 1786750837.43 total 50.10 last 70.62) - open/close 4016 (time 1786750847.46 total 60.13 last 70.78) total: 4500 open/close in 68.07 seconds: 66.11 ops/second - /unlink 1102 (time 1786750870.83 total 10.00 last 110.18) - /unlink 2432 (time 1786750880.83 total 20.00 last 133.00) - /unlink 3705 (time 1786750890.83 total 30.01 last 127.26) total: 4500 /unlink in 35.03 seconds: 128.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 692 (time 1786750910.87 total 10.01 last 69.13) - open/close 1358 (time 1786750920.88 total 20.02 last 66.56) - open/close 2028 (time 1786750930.90 total 30.03 last 66.88) - open/close 2862 (time 1786750940.91 total 40.05 last 83.26) - open/close 3662 (time 1786750950.94 total 50.08 last 79.75) - open/close 4377 (time 1786750960.96 total 60.09 last 71.41) total: 4500 open/close in 61.79 seconds: 72.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 815 (time 1786750977.98 total 10.00 last 81.48) - /unlink 1867 (time 1786750987.98 total 20.00 last 105.20) - /unlink 3149 (time 1786750997.98 total 30.00 last 128.19) - /unlink 4430 (time 1786751007.98 total 40.00 last 128.08) total: 4500 /unlink in 40.60 seconds: 110.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 669 (time 1786751023.29 total 10.01 last 66.86) - open/close 1380 (time 1786751033.30 total 20.02 last 70.98) - open/close 2017 (time 1786751043.31 total 30.03 last 63.65) - open/close 2742 (time 1786751053.33 total 40.05 last 72.37) - open/close 3346 (time 1786751063.34 total 50.06 last 60.35) - open/close 4066 (time 1786751073.35 total 60.07 last 71.93) total: 4500 open/close in 66.81 seconds: 67.35 ops/second - /unlink 1159 (time 1786751094.84 total 10.01 last 115.82) - /unlink 2436 (time 1786751104.84 total 20.01 last 127.69) - /unlink 3698 (time 1786751114.84 total 30.01 last 126.19) total: 4500 /unlink in 37.56 seconds: 119.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 601 (time 1786751138.06 total 10.01 last 60.06) - open/close 1227 (time 1786751148.06 total 20.01 last 62.56) - open/close 1849 (time 1786751158.07 total 30.02 last 62.15) - open/close 2557 (time 1786751168.08 total 40.03 last 70.72) - open/close 3299 (time 1786751178.09 total 50.04 last 74.13) - open/close 4058 (time 1786751188.10 total 60.05 last 75.83) total: 4500 open/close in 66.71 seconds: 67.46 ops/second - /unlink 1299 (time 1786751209.29 total 10.00 last 129.86) - /unlink 2390 (time 1786751219.30 total 20.01 last 109.06) - /unlink 3514 (time 1786751229.30 total 30.01 last 112.39) total: 4500 /unlink in 37.94 seconds: 118.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 744 (time 1786751253.37 total 10.00 last 74.39) - open/close 1568 (time 1786751263.37 total 20.01 last 82.36) - open/close 2187 (time 1786751273.39 total 30.02 last 61.82) - open/close 2821 (time 1786751283.39 total 40.02 last 63.38) - open/close 3447 (time 1786751293.41 total 50.04 last 62.48) - open/close 3977 (time 1786751303.43 total 60.06 last 52.92) total: 4500 open/close in 68.23 seconds: 65.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1305 (time 1786751327.21 total 10.01 last 130.41) - /unlink 2645 (time 1786751337.21 total 20.01 last 133.99) - /unlink 3981 (time 1786751347.21 total 30.01 last 133.53) total: 4500 /unlink in 33.90 seconds: 132.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 774 (time 1786751366.28 total 10.00 last 77.38) - open/close 1514 (time 1786751376.29 total 20.02 last 73.91) - open/close 2034 (time 1786751386.30 total 30.03 last 51.94) - open/close 2713 (time 1786751396.31 total 40.03 last 67.85) - open/close 3397 (time 1786751406.31 total 50.03 last 68.40) - open/close 3949 (time 1786751416.32 total 60.04 last 55.16) total: 4500 open/close in 68.40 seconds: 65.79 ops/second - /unlink 1279 (time 1786751441.13 total 10.00 last 127.87) - /unlink 2638 (time 1786751451.14 total 20.01 last 135.84) - /unlink 3909 (time 1786751461.15 total 30.02 last 126.98) total: 4500 /unlink in 34.43 seconds: 130.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 643 (time 1786751480.82 total 10.02 last 64.19) - open/close 1262 (time 1786751490.86 total 20.06 last 61.66) - open/close 1848 (time 1786751500.87 total 30.06 last 58.59) - open/close 2522 (time 1786751510.87 total 40.07 last 67.34) - open/close 3084 (time 1786751520.88 total 50.07 last 56.16) - open/close 3595 (time 1786751530.89 total 60.08 last 51.06) - open/close 4193 (time 1786751540.89 total 70.09 last 59.76) total: 4500 open/close in 75.81 seconds: 59.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1237 (time 1786751563.18 total 10.01 last 123.61) - /unlink 2222 (time 1786751573.18 total 20.01 last 98.46) - /unlink 3595 (time 1786751583.19 total 30.01 last 137.25) total: 4500 /unlink in 38.74 seconds: 116.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 540 (time 1786751609.74 total 10.01 last 53.94) - open/close 1144 (time 1786751619.75 total 20.02 last 60.36) - open/close 1811 (time 1786751629.76 total 30.03 last 66.65) - open/close 2461 (time 1786751639.76 total 40.03 last 64.99) - open/close 3125 (time 1786751649.77 total 50.04 last 66.33) - open/close 3869 (time 1786751659.77 total 60.04 last 74.38) - open/close 4365 (time 1786751669.78 total 70.05 last 49.53) total: 4500 open/close in 72.29 seconds: 62.25 ops/second - /unlink 1256 (time 1786751688.19 total 10.01 last 125.54) - /unlink 2255 (time 1786751698.19 total 20.01 last 99.86) - /unlink 3470 (time 1786751708.20 total 30.01 last 121.46) total: 4500 /unlink in 37.71 seconds: 119.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 498 (time 1786751732.47 total 10.01 last 49.75) - open/close 1132 (time 1786751742.48 total 20.02 last 63.34) - open/close 1843 (time 1786751752.49 total 30.03 last 71.05) - open/close 2613 (time 1786751762.50 total 40.03 last 76.95) - open/close 3318 (time 1786751772.51 total 50.05 last 70.41) - open/close 4048 (time 1786751782.51 total 60.05 last 72.99) total: 4500 open/close in 65.15 seconds: 69.07 ops/second - /unlink 1192 (time 1786751802.85 total 10.00 last 119.17) - /unlink 2417 (time 1786751812.85 total 20.00 last 122.50) - /unlink 3737 (time 1786751822.85 total 30.01 last 131.96) total: 4500 /unlink in 35.71 seconds: 126.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 726 (time 1786751843.11 total 10.01 last 72.52) - open/close 1449 (time 1786751853.12 total 20.02 last 72.23) - open/close 2105 (time 1786751863.13 total 30.03 last 65.55) - open/close 2666 (time 1786751873.15 total 40.05 last 55.96) - open/close 3400 (time 1786751883.16 total 50.06 last 73.33) - open/close 4103 (time 1786751893.18 total 60.08 last 70.16) total: 4500 open/close in 65.54 seconds: 68.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1094 (time 1786751914.17 total 10.00 last 109.37) - /unlink 2445 (time 1786751924.17 total 20.01 last 135.02) - /unlink 3649 (time 1786751934.19 total 30.02 last 120.24) total: 4500 /unlink in 37.92 seconds: 118.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 634 (time 1786751957.91 total 10.03 last 63.23) - open/close 1366 (time 1786751967.91 total 20.03 last 73.16) - open/close 2135 (time 1786751977.92 total 30.04 last 76.88) - open/close 2888 (time 1786751987.92 total 40.04 last 75.29) - open/close 3623 (time 1786751997.93 total 50.05 last 73.43) - open/close 4266 (time 1786752007.93 total 60.05 last 64.27) total: 4500 open/close in 63.52 seconds: 70.84 ops/second - /unlink 1399 (time 1786752027.13 total 10.01 last 139.83) - /unlink 2526 (time 1786752037.13 total 20.01 last 112.66) - /unlink 3692 (time 1786752047.14 total 30.01 last 116.55) total: 4500 /unlink in 36.34 seconds: 123.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 591 (time 1786752069.76 total 10.01 last 59.01) - open/close 1342 (time 1786752079.77 total 20.03 last 74.97) - open/close 2099 (time 1786752089.78 total 30.04 last 75.65) - open/close 2906 (time 1786752099.79 total 40.05 last 80.58) - open/close 3530 (time 1786752109.80 total 50.06 last 62.34) - open/close 4283 (time 1786752119.81 total 60.06 last 75.30) total: 4500 open/close in 62.82 seconds: 71.63 ops/second - /unlink 992 (time 1786752137.85 total 10.01 last 99.13) - /unlink 2103 (time 1786752147.85 total 20.01 last 111.09) - /unlink 3209 (time 1786752157.87 total 30.02 last 110.41) - /unlink 4221 (time 1786752167.87 total 40.03 last 101.16) total: 4500 /unlink in 42.45 seconds: 106.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 781 (time 1786752185.62 total 10.01 last 78.05) - open/close 1538 (time 1786752195.63 total 20.01 last 75.64) - open/close 2338 (time 1786752205.63 total 30.01 last 80.00) - open/close 3143 (time 1786752215.64 total 40.03 last 80.38) - open/close 3801 (time 1786752225.68 total 50.07 last 65.52) total: 4500 open/close in 59.19 seconds: 76.03 ops/second - /unlink 1461 (time 1786752248.97 total 10.00 last 146.04) - /unlink 2886 (time 1786752258.97 total 20.01 last 142.47) - /unlink 4371 (time 1786752268.97 total 30.01 last 148.42) total: 4500 /unlink in 30.99 seconds: 145.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 704 (time 1786752284.39 total 10.01 last 70.30) - open/close 1610 (time 1786752294.40 total 20.02 last 90.56) - open/close 2472 (time 1786752304.40 total 30.02 last 86.16) - open/close 3206 (time 1786752314.40 total 40.03 last 73.39) - open/close 3955 (time 1786752324.41 total 50.03 last 74.86) total: 4500 open/close in 57.78 seconds: 77.88 ops/second - /unlink 1507 (time 1786752346.13 total 10.01 last 150.61) - /unlink 2951 (time 1786752356.14 total 20.01 last 144.35) - /unlink 4223 (time 1786752366.14 total 30.01 last 127.16) total: 4500 /unlink in 32.67 seconds: 137.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 778 (time 1786752383.30 total 10.02 last 77.63) - open/close 1531 (time 1786752393.30 total 20.02 last 75.29) - open/close 2243 (time 1786752403.31 total 30.03 last 71.16) - open/close 2725 (time 1786752413.31 total 40.03 last 48.18) - open/close 3318 (time 1786752423.33 total 50.05 last 59.20) - open/close 4047 (time 1786752433.34 total 60.07 last 72.77) total: 4500 open/close in 66.14 seconds: 68.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1424 (time 1786752453.97 total 10.00 last 142.34) - /unlink 2839 (time 1786752463.98 total 20.02 last 141.33) - /unlink 4004 (time 1786752473.99 total 30.02 last 116.44) total: 4500 /unlink in 35.07 seconds: 128.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 708 (time 1786752494.11 total 10.00 last 70.78) - open/close 1412 (time 1786752504.12 total 20.01 last 70.32) - open/close 2036 (time 1786752514.13 total 30.02 last 62.38) - open/close 2764 (time 1786752524.14 total 40.03 last 72.70) - open/close 3376 (time 1786752534.15 total 50.04 last 61.16) - open/close 4050 (time 1786752544.16 total 60.06 last 67.28) total: 4500 open/close in 67.74 seconds: 66.43 ops/second - /unlink 1302 (time 1786752566.55 total 10.00 last 130.14) - /unlink 2622 (time 1786752576.56 total 20.01 last 131.97) - /unlink 3903 (time 1786752586.56 total 30.01 last 128.07) total: 4500 /unlink in 35.30 seconds: 127.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 552 (time 1786752606.93 total 10.01 last 55.16) - open/close 1330 (time 1786752616.93 total 20.01 last 77.75) - open/close 1822 (time 1786752626.94 total 30.02 last 49.17) - open/close 2474 (time 1786752636.94 total 40.03 last 65.16) - open/close 3092 (time 1786752646.95 total 50.03 last 61.78) - open/close 3838 (time 1786752656.97 total 60.05 last 74.46) total: 4500 open/close in 67.89 seconds: 66.29 ops/second - /unlink 1383 (time 1786752678.73 total 10.00 last 138.25) - /unlink 2563 (time 1786752688.73 total 20.00 last 117.98) - /unlink 3906 (time 1786752698.73 total 30.01 last 134.21) total: 4500 /unlink in 34.71 seconds: 129.65 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11297: 95300 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg103-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg103-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg103-server stop mds service on oleg103-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg103-server stop mds service on oleg103-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg103-server unloading modules via unload_modules_local on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing unload_modules_local oleg103-server: modules unloaded.