== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:46:59 (1786405619) Stopping clients: oleg428-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg428-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg428-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg428-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg428-server oleg428-server: oleg428-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing load_modules_local oleg428-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg428-server: Loading modules from /home/green/git/lustre-release/lustre oleg428-server: detected 4 online CPUs by sysfs oleg428-server: MODOPTS_LIBCFS= oleg428-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg428-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg428-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg428-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 683 (time 1786405856.23 total 10.01 last 68.22) - open/close 1384 (time 1786405866.25 total 20.04 last 69.94) - open/close 2075 (time 1786405876.26 total 30.04 last 69.03) - open/close 2769 (time 1786405886.27 total 40.05 last 69.34) - open/close 3432 (time 1786405896.28 total 50.06 last 66.24) - open/close 4161 (time 1786405906.29 total 60.07 last 72.83) total: 4500 open/close in 65.08 seconds: 69.15 ops/second - /unlink 1272 (time 1786405927.19 total 10.00 last 127.14) - /unlink 2510 (time 1786405937.20 total 20.01 last 123.75) - /unlink 3620 (time 1786405947.20 total 30.01 last 110.94) total: 4500 /unlink in 36.89 seconds: 121.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 780 (time 1786405969.86 total 10.01 last 77.95) - open/close 1453 (time 1786405979.86 total 20.01 last 67.28) - open/close 2158 (time 1786405989.87 total 30.02 last 70.44) - open/close 2903 (time 1786405999.89 total 40.03 last 74.38) - open/close 3748 (time 1786406009.89 total 50.04 last 84.48) total: 4500 open/close in 59.85 seconds: 75.19 ops/second - /unlink 1314 (time 1786406035.01 total 10.00 last 131.39) - /unlink 2542 (time 1786406045.01 total 20.00 last 122.78) - /unlink 3794 (time 1786406055.01 total 30.01 last 125.15) total: 4500 /unlink in 35.89 seconds: 125.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 784 (time 1786406077.47 total 10.02 last 78.28) - open/close 1500 (time 1786406087.49 total 20.03 last 71.50) - open/close 2221 (time 1786406097.49 total 30.03 last 72.07) - open/close 2943 (time 1786406107.50 total 40.04 last 72.16) - open/close 3663 (time 1786406117.51 total 50.05 last 71.94) - open/close 4369 (time 1786406127.52 total 60.06 last 70.54) total: 4500 open/close in 61.90 seconds: 72.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1222 (time 1786406145.85 total 10.01 last 122.12) - /unlink 2470 (time 1786406155.86 total 20.02 last 124.67) - /unlink 3619 (time 1786406165.86 total 30.02 last 114.85) total: 4500 /unlink in 37.43 seconds: 120.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 747 (time 1786406190.48 total 10.01 last 74.60) - open/close 1423 (time 1786406200.50 total 20.03 last 67.48) - open/close 1996 (time 1786406210.50 total 30.03 last 57.29) - open/close 2646 (time 1786406220.52 total 40.05 last 64.87) - open/close 3344 (time 1786406230.53 total 50.06 last 69.75) - open/close 4050 (time 1786406240.53 total 60.06 last 70.59) total: 4500 open/close in 66.39 seconds: 67.78 ops/second - /unlink 1302 (time 1786406261.26 total 10.00 last 130.16) - /unlink 2374 (time 1786406271.27 total 20.01 last 107.13) - /unlink 3472 (time 1786406281.27 total 30.01 last 109.78) - /unlink 4492 (time 1786406291.27 total 40.01 last 101.98) total: 4500 /unlink in 40.07 seconds: 112.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 660 (time 1786406308.13 total 10.00 last 65.98) - open/close 1314 (time 1786406318.13 total 20.00 last 65.40) - open/close 2050 (time 1786406328.14 total 30.01 last 73.57) - open/close 2759 (time 1786406338.15 total 40.02 last 70.83) - open/close 3503 (time 1786406348.16 total 50.03 last 74.30) - open/close 4247 (time 1786406358.18 total 60.05 last 74.29) total: 4500 open/close in 63.30 seconds: 71.09 ops/second - /unlink 1166 (time 1786406376.81 total 10.01 last 116.53) - /unlink 2307 (time 1786406386.81 total 20.01 last 114.09) - /unlink 3534 (time 1786406396.81 total 30.01 last 122.69) total: 4500 /unlink in 38.58 seconds: 116.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 708 (time 1786406421.81 total 10.01 last 70.75) - open/close 1443 (time 1786406431.84 total 20.04 last 73.28) - open/close 2189 (time 1786406441.85 total 30.05 last 74.54) - open/close 2997 (time 1786406451.85 total 40.05 last 80.77) - open/close 3788 (time 1786406461.85 total 50.05 last 79.09) total: 4500 open/close in 59.79 seconds: 75.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1299 (time 1786406487.43 total 10.01 last 129.82) - /unlink 2450 (time 1786406497.44 total 20.01 last 115.03) - /unlink 3713 (time 1786406507.44 total 30.02 last 126.25) total: 4500 /unlink in 36.54 seconds: 123.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 741 (time 1786406529.86 total 10.01 last 74.06) - open/close 1441 (time 1786406539.86 total 20.01 last 70.00) - open/close 2106 (time 1786406549.88 total 30.02 last 66.42) - open/close 2845 (time 1786406559.89 total 40.03 last 73.81) - open/close 3539 (time 1786406569.90 total 50.04 last 69.34) - open/close 4248 (time 1786406579.91 total 60.05 last 70.80) total: 4500 open/close in 63.61 seconds: 70.74 ops/second - /unlink 1384 (time 1786406598.98 total 10.00 last 138.36) - /unlink 2533 (time 1786406608.99 total 20.01 last 114.85) - /unlink 3761 (time 1786406618.99 total 30.01 last 122.77) total: 4500 /unlink in 36.69 seconds: 122.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 698 (time 1786406641.88 total 10.00 last 69.79) - open/close 1336 (time 1786406651.88 total 20.00 last 63.78) - open/close 1995 (time 1786406661.89 total 30.02 last 65.81) - open/close 2735 (time 1786406671.90 total 40.03 last 73.93) - open/close 3497 (time 1786406681.91 total 50.04 last 76.13) - open/close 4161 (time 1786406691.93 total 60.06 last 66.27) total: 4500 open/close in 65.22 seconds: 69.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1271 (time 1786406714.94 total 10.00 last 127.08) - /unlink 2439 (time 1786406724.94 total 20.00 last 116.78) - /unlink 3786 (time 1786406734.94 total 30.01 last 134.67) total: 4500 /unlink in 35.95 seconds: 125.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 704 (time 1786406755.46 total 10.00 last 70.37) - open/close 1402 (time 1786406765.47 total 20.02 last 69.71) - open/close 2105 (time 1786406775.47 total 30.02 last 70.28) - open/close 2845 (time 1786406785.48 total 40.03 last 73.93) - open/close 3660 (time 1786406795.49 total 50.04 last 81.41) - open/close 4424 (time 1786406805.51 total 60.05 last 76.32) total: 4500 open/close in 61.02 seconds: 73.74 ops/second - /unlink 1276 (time 1786406821.52 total 10.01 last 127.48) - /unlink 2487 (time 1786406831.52 total 20.01 last 121.06) - /unlink 3671 (time 1786406841.52 total 30.01 last 118.39) total: 4500 /unlink in 36.37 seconds: 123.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 765 (time 1786406864.12 total 10.02 last 76.34) - open/close 1549 (time 1786406874.14 total 20.03 last 78.30) - open/close 2384 (time 1786406884.15 total 30.05 last 83.40) - open/close 3228 (time 1786406894.16 total 40.05 last 84.33) - open/close 3923 (time 1786406904.16 total 50.06 last 69.45) total: 4500 open/close in 57.92 seconds: 77.69 ops/second - /unlink 1390 (time 1786406927.95 total 10.00 last 139.00) - /unlink 2471 (time 1786406937.96 total 20.01 last 108.01) - /unlink 3635 (time 1786406947.97 total 30.01 last 116.34) total: 4500 /unlink in 36.29 seconds: 124.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 762 (time 1786406970.34 total 10.01 last 76.15) - open/close 1458 (time 1786406980.35 total 20.01 last 69.55) - open/close 2274 (time 1786406990.38 total 30.05 last 81.35) - open/close 2966 (time 1786407000.39 total 40.06 last 69.12) - open/close 3767 (time 1786407010.40 total 50.07 last 79.99) total: 4500 open/close in 59.33 seconds: 75.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1294 (time 1786407034.91 total 10.00 last 129.36) - /unlink 2403 (time 1786407044.91 total 20.01 last 110.87) - /unlink 3624 (time 1786407054.92 total 30.01 last 122.01) total: 4500 /unlink in 36.40 seconds: 123.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 831 (time 1786407077.75 total 10.00 last 83.09) - open/close 1563 (time 1786407087.75 total 20.00 last 73.19) - open/close 2260 (time 1786407097.78 total 30.04 last 69.47) - open/close 2967 (time 1786407107.79 total 40.04 last 70.67) - open/close 3693 (time 1786407117.79 total 50.04 last 72.57) - open/close 4469 (time 1786407127.80 total 60.05 last 77.54) total: 4500 open/close in 60.57 seconds: 74.29 ops/second - /unlink 1108 (time 1786407144.87 total 10.00 last 110.76) - /unlink 2004 (time 1786407154.87 total 20.01 last 89.57) - /unlink 3018 (time 1786407164.88 total 30.01 last 101.34) - /unlink 4052 (time 1786407174.88 total 40.01 last 103.40) total: 4500 /unlink in 44.58 seconds: 100.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 634 (time 1786407197.05 total 10.00 last 63.37) - open/close 1366 (time 1786407207.05 total 20.01 last 73.18) - open/close 2079 (time 1786407217.09 total 30.05 last 71.01) - open/close 2787 (time 1786407227.09 total 40.05 last 70.79) - open/close 3550 (time 1786407237.10 total 50.06 last 76.25) - open/close 4292 (time 1786407247.11 total 60.07 last 74.11) total: 4500 open/close in 63.28 seconds: 71.12 ops/second - /unlink 1292 (time 1786407265.87 total 10.00 last 129.16) - /unlink 2583 (time 1786407275.88 total 20.01 last 128.96) - /unlink 3953 (time 1786407285.88 total 30.02 last 136.98) total: 4500 /unlink in 34.27 seconds: 131.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 705 (time 1786407306.16 total 10.00 last 70.47) - open/close 1513 (time 1786407316.16 total 20.01 last 80.78) - open/close 2302 (time 1786407326.16 total 30.01 last 78.88) - open/close 3107 (time 1786407336.17 total 40.01 last 80.45) - open/close 3906 (time 1786407346.18 total 50.03 last 79.78) total: 4500 open/close in 57.46 seconds: 78.31 ops/second - /unlink 1243 (time 1786407369.30 total 10.00 last 124.26) - /unlink 2688 (time 1786407379.30 total 20.01 last 144.44) - /unlink 3932 (time 1786407389.30 total 30.01 last 124.38) total: 4500 /unlink in 33.98 seconds: 132.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 748 (time 1786407409.02 total 10.00 last 74.80) - open/close 1390 (time 1786407419.03 total 20.01 last 64.13) - open/close 2137 (time 1786407429.03 total 30.01 last 74.68) - open/close 2932 (time 1786407439.04 total 40.02 last 79.43) - open/close 3656 (time 1786407449.05 total 50.03 last 72.36) - open/close 4412 (time 1786407459.06 total 60.04 last 75.51) total: 4500 open/close in 61.13 seconds: 73.61 ops/second - /unlink 1315 (time 1786407475.63 total 10.01 last 131.36) - /unlink 2566 (time 1786407485.63 total 20.01 last 125.06) - /unlink 3942 (time 1786407495.63 total 30.02 last 137.53) total: 4500 /unlink in 34.19 seconds: 131.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 683 (time 1786407517.94 total 10.01 last 68.22) - open/close 1299 (time 1786407527.95 total 20.02 last 61.52) - open/close 2011 (time 1786407537.95 total 30.03 last 71.20) - open/close 2697 (time 1786407547.96 total 40.03 last 68.56) - open/close 3438 (time 1786407557.97 total 50.04 last 74.03) - open/close 4110 (time 1786407567.97 total 60.05 last 67.17) total: 4500 open/close in 64.78 seconds: 69.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1318 (time 1786407587.72 total 10.00 last 131.74) - /unlink 2630 (time 1786407597.72 total 20.01 last 131.19) - /unlink 3934 (time 1786407607.72 total 30.01 last 130.30) total: 4500 /unlink in 34.27 seconds: 131.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 788 (time 1786407626.67 total 10.00 last 78.79) - open/close 1540 (time 1786407636.68 total 20.01 last 75.14) - open/close 2335 (time 1786407646.69 total 30.02 last 79.41) - open/close 3125 (time 1786407656.71 total 40.04 last 78.87) - open/close 3941 (time 1786407666.71 total 50.04 last 81.58) total: 4500 open/close in 56.95 seconds: 79.02 ops/second - /unlink 1507 (time 1786407688.50 total 10.00 last 150.65) - /unlink 2893 (time 1786407698.51 total 20.01 last 138.54) - /unlink 4145 (time 1786407708.51 total 30.01 last 125.13) total: 4500 /unlink in 32.60 seconds: 138.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 732 (time 1786407727.58 total 10.01 last 73.13) - open/close 1482 (time 1786407737.59 total 20.01 last 74.96) - open/close 2229 (time 1786407747.61 total 30.04 last 74.54) - open/close 3023 (time 1786407757.62 total 40.04 last 79.34) - open/close 3842 (time 1786407767.62 total 50.05 last 81.86) total: 4500 open/close in 59.10 seconds: 76.14 ops/second - /unlink 1305 (time 1786407791.35 total 10.00 last 130.44) - /unlink 2672 (time 1786407801.35 total 20.01 last 136.68) - /unlink 3936 (time 1786407811.35 total 30.01 last 126.36) total: 4500 /unlink in 34.04 seconds: 132.20 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94990 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user 'cl1': success lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user 'cl1': success Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg428-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg428-server unloading modules via unload_modules_local on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing unload_modules_local oleg428-server: modules unloaded.