== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:03:12 (1785456192) Stopping clients: oleg655-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg655-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg655-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg655-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg655-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg655-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg655-server oleg655-server: oleg655-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg655-server' oleg655-server: oleg655-server.virtnet: executing load_modules_local oleg655-server: Loading modules from /home/green/git/lustre-release/lustre oleg655-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg655-server: detected 4 online CPUs by sysfs oleg655-server: MODOPTS_LIBCFS= oleg655-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg655-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg655-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg655-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg655-client.virtnet: -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 691 (time 1785456403.32 total 10.03 last 68.90) - open/close 1420 (time 1785456413.33 total 20.04 last 72.83) - open/close 2194 (time 1785456423.35 total 30.06 last 77.26) - open/close 2908 (time 1785456433.35 total 40.06 last 71.38) - open/close 3672 (time 1785456443.35 total 50.06 last 76.36) - open/close 4430 (time 1785456453.37 total 60.08 last 75.69) total: 4500 open/close in 61.10 seconds: 73.65 ops/second - /unlink 1406 (time 1785456469.00 total 10.01 last 140.47) - /unlink 2895 (time 1785456479.00 total 20.01 last 148.83) - /unlink 4355 (time 1785456489.00 total 30.02 last 145.98) total: 4500 /unlink in 30.98 seconds: 145.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 801 (time 1785456503.88 total 10.01 last 80.04) - open/close 1559 (time 1785456513.89 total 20.02 last 75.70) - open/close 2293 (time 1785456523.89 total 30.02 last 73.38) - open/close 3064 (time 1785456533.90 total 40.03 last 77.04) - open/close 3789 (time 1785456543.91 total 50.04 last 72.47) total: 4500 open/close in 59.23 seconds: 75.98 ops/second - /unlink 1386 (time 1785456566.99 total 10.01 last 138.53) - /unlink 2447 (time 1785456577.01 total 20.02 last 105.91) - /unlink 3520 (time 1785456587.03 total 30.04 last 107.09) total: 4500 /unlink in 39.33 seconds: 114.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 538 (time 1785456612.26 total 10.01 last 53.77) - open/close 1059 (time 1785456622.27 total 20.02 last 52.05) - open/close 1722 (time 1785456632.27 total 30.02 last 66.26) - open/close 2454 (time 1785456642.28 total 40.02 last 73.18) - open/close 3147 (time 1785456652.29 total 50.04 last 69.18) - open/close 3876 (time 1785456662.30 total 60.04 last 72.87) total: 4500 open/close in 68.08 seconds: 66.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1433 (time 1785456684.39 total 10.00 last 143.28) - /unlink 2779 (time 1785456694.39 total 20.00 last 134.59) - /unlink 4160 (time 1785456704.39 total 30.01 last 138.04) total: 4500 /unlink in 32.37 seconds: 139.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 702 (time 1785456721.85 total 10.00 last 70.19) - open/close 1427 (time 1785456731.87 total 20.01 last 72.40) - open/close 2226 (time 1785456741.87 total 30.02 last 79.88) - open/close 2958 (time 1785456751.88 total 40.03 last 73.14) - open/close 3673 (time 1785456761.88 total 50.03 last 71.47) - open/close 4427 (time 1785456771.89 total 60.04 last 75.32) total: 4500 open/close in 61.05 seconds: 73.71 ops/second - /unlink 1421 (time 1785456787.06 total 10.00 last 142.06) - /unlink 2784 (time 1785456797.06 total 20.00 last 136.29) - /unlink 4128 (time 1785456807.06 total 30.00 last 134.39) total: 4500 /unlink in 32.66 seconds: 137.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 787 (time 1785456824.31 total 10.00 last 78.68) - open/close 1527 (time 1785456834.32 total 20.01 last 73.95) - open/close 2230 (time 1785456844.32 total 30.01 last 70.27) - open/close 2901 (time 1785456854.34 total 40.03 last 66.99) - open/close 3493 (time 1785456864.35 total 50.04 last 59.15) - open/close 4121 (time 1785456874.35 total 60.04 last 62.77) total: 4500 open/close in 66.03 seconds: 68.15 ops/second - /unlink 1278 (time 1785456894.59 total 10.00 last 127.75) - /unlink 2543 (time 1785456904.60 total 20.01 last 126.45) - /unlink 3862 (time 1785456914.62 total 30.03 last 131.64) total: 4500 /unlink in 35.32 seconds: 127.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 666 (time 1785456934.55 total 10.01 last 66.52) - open/close 1359 (time 1785456944.56 total 20.02 last 69.26) - open/close 2047 (time 1785456954.56 total 30.03 last 68.74) - open/close 2719 (time 1785456964.57 total 40.03 last 67.18) - open/close 3438 (time 1785456974.57 total 50.03 last 71.90) - open/close 4202 (time 1785456984.59 total 60.05 last 76.26) total: 4500 open/close in 64.32 seconds: 69.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1334 (time 1785457003.61 total 10.00 last 133.38) - /unlink 2730 (time 1785457013.61 total 20.00 last 139.58) - /unlink 4150 (time 1785457023.62 total 30.01 last 141.95) total: 4500 /unlink in 32.83 seconds: 137.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 729 (time 1785457040.93 total 10.00 last 72.87) - open/close 1512 (time 1785457050.93 total 20.01 last 78.27) - open/close 2262 (time 1785457060.93 total 30.01 last 74.98) - open/close 2963 (time 1785457070.94 total 40.02 last 70.05) - open/close 3702 (time 1785457080.95 total 50.03 last 73.84) - open/close 4430 (time 1785457090.97 total 60.04 last 72.69) total: 4500 open/close in 61.32 seconds: 73.39 ops/second - /unlink 1340 (time 1785457107.03 total 10.01 last 133.81) - /unlink 2683 (time 1785457117.03 total 20.02 last 134.23) - /unlink 3962 (time 1785457127.04 total 30.02 last 127.83) total: 4500 /unlink in 34.22 seconds: 131.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 752 (time 1785457146.11 total 10.01 last 75.15) - open/close 1460 (time 1785457156.11 total 20.02 last 70.74) - open/close 2069 (time 1785457166.11 total 30.02 last 60.90) - open/close 2816 (time 1785457176.12 total 40.02 last 74.69) - open/close 3545 (time 1785457186.12 total 50.02 last 72.85) - open/close 4254 (time 1785457196.13 total 60.03 last 70.88) total: 4500 open/close in 63.84 seconds: 70.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1083 (time 1785457215.73 total 10.01 last 108.20) - /unlink 2056 (time 1785457225.73 total 20.01 last 97.26) - /unlink 3172 (time 1785457235.74 total 30.02 last 111.48) - /unlink 4322 (time 1785457245.74 total 40.03 last 114.97) total: 4500 /unlink in 41.51 seconds: 108.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 760 (time 1785457262.93 total 10.00 last 75.96) - open/close 1509 (time 1785457272.93 total 20.01 last 74.84) - open/close 2182 (time 1785457282.95 total 30.03 last 67.18) - open/close 2842 (time 1785457292.96 total 40.04 last 65.95) - open/close 3568 (time 1785457302.97 total 50.05 last 72.54) - open/close 4131 (time 1785457312.98 total 60.06 last 56.25) total: 4500 open/close in 66.56 seconds: 67.60 ops/second - /unlink 1107 (time 1785457336.04 total 10.00 last 110.68) - /unlink 2177 (time 1785457346.05 total 20.00 last 106.97) - /unlink 3166 (time 1785457356.05 total 30.01 last 98.85) - /unlink 4450 (time 1785457366.05 total 40.01 last 128.38) total: 4500 /unlink in 40.74 seconds: 110.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 611 (time 1785457382.94 total 10.00 last 61.08) - open/close 1167 (time 1785457392.95 total 20.02 last 55.50) - open/close 1724 (time 1785457402.96 total 30.03 last 55.65) - open/close 2332 (time 1785457412.97 total 40.03 last 60.77) - open/close 2981 (time 1785457422.97 total 50.03 last 64.89) - open/close 3663 (time 1785457432.99 total 60.05 last 68.07) - open/close 4344 (time 1785457442.99 total 70.06 last 68.08) total: 4500 open/close in 72.36 seconds: 62.19 ops/second - /unlink 1305 (time 1785457460.35 total 10.01 last 130.38) - /unlink 2425 (time 1785457470.35 total 20.01 last 111.99) - /unlink 3768 (time 1785457480.36 total 30.02 last 134.21) total: 4500 /unlink in 35.38 seconds: 127.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 713 (time 1785457500.67 total 10.01 last 71.22) - open/close 1371 (time 1785457510.69 total 20.03 last 65.68) - open/close 1916 (time 1785457520.70 total 30.04 last 54.42) - open/close 2583 (time 1785457530.72 total 40.06 last 66.60) - open/close 3222 (time 1785457540.74 total 50.08 last 63.78) - open/close 3960 (time 1785457550.75 total 60.09 last 73.74) total: 4500 open/close in 68.00 seconds: 66.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1316 (time 1785457573.83 total 10.00 last 131.58) - /unlink 2456 (time 1785457583.83 total 20.01 last 113.90) - /unlink 3604 (time 1785457593.84 total 30.02 last 114.73) total: 4500 /unlink in 39.38 seconds: 114.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 669 (time 1785457619.49 total 10.01 last 66.86) - open/close 1334 (time 1785457629.50 total 20.02 last 66.43) - open/close 2044 (time 1785457639.50 total 30.02 last 70.94) - open/close 2796 (time 1785457649.51 total 40.03 last 75.19) - open/close 3432 (time 1785457659.51 total 50.03 last 63.58) - open/close 3998 (time 1785457669.51 total 60.03 last 56.58) total: 4500 open/close in 67.74 seconds: 66.43 ops/second - /unlink 1108 (time 1785457693.36 total 10.00 last 110.76) - /unlink 2302 (time 1785457703.37 total 20.01 last 119.32) - /unlink 3438 (time 1785457713.38 total 30.01 last 113.55) total: 4500 /unlink in 38.51 seconds: 116.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 624 (time 1785457737.36 total 10.00 last 62.37) - open/close 1271 (time 1785457747.36 total 20.01 last 64.66) - open/close 1908 (time 1785457757.39 total 30.04 last 63.53) - open/close 2628 (time 1785457767.39 total 40.04 last 71.99) - open/close 3318 (time 1785457777.40 total 50.04 last 68.97) - open/close 3958 (time 1785457787.41 total 60.05 last 63.93) total: 4500 open/close in 68.56 seconds: 65.63 ops/second - /unlink 1249 (time 1785457811.19 total 10.01 last 124.83) - /unlink 2340 (time 1785457821.19 total 20.01 last 109.09) - /unlink 3357 (time 1785457831.19 total 30.01 last 101.69) - /unlink 4096 (time 1785457841.22 total 40.04 last 73.70) total: 4500 /unlink in 45.02 seconds: 99.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 555 (time 1785457862.72 total 10.00 last 55.49) - open/close 1152 (time 1785457872.73 total 20.01 last 59.62) - open/close 1775 (time 1785457882.73 total 30.02 last 62.28) - open/close 2371 (time 1785457892.73 total 40.02 last 59.59) - open/close 2969 (time 1785457902.74 total 50.02 last 59.77) - open/close 3555 (time 1785457912.75 total 60.03 last 58.55) - open/close 4214 (time 1785457922.75 total 70.03 last 65.89) total: 4500 open/close in 74.28 seconds: 60.58 ops/second - /unlink 1150 (time 1785457942.16 total 10.00 last 114.98) - /unlink 2446 (time 1785457952.16 total 20.00 last 129.58) - /unlink 3430 (time 1785457962.17 total 30.01 last 98.35) total: 4500 /unlink in 39.69 seconds: 113.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 568 (time 1785457987.88 total 10.02 last 56.71) - open/close 1259 (time 1785457997.88 total 20.02 last 69.07) - open/close 1855 (time 1785458007.90 total 30.04 last 59.49) - open/close 2627 (time 1785458017.91 total 40.05 last 77.09) - open/close 3330 (time 1785458027.92 total 50.06 last 70.24) - open/close 3910 (time 1785458037.95 total 60.09 last 57.85) - open/close 4429 (time 1785458047.96 total 70.10 last 51.85) total: 4500 open/close in 71.58 seconds: 62.87 ops/second - /unlink 1122 (time 1785458065.16 total 10.00 last 112.19) - /unlink 2300 (time 1785458075.17 total 20.01 last 117.71) - /unlink 3552 (time 1785458085.18 total 30.02 last 125.10) total: 4500 /unlink in 37.62 seconds: 119.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 672 (time 1785458108.60 total 10.00 last 67.19) - open/close 1462 (time 1785458118.61 total 20.02 last 78.87) - open/close 2073 (time 1785458128.63 total 30.03 last 61.02) - open/close 2802 (time 1785458138.64 total 40.04 last 72.81) - open/close 3418 (time 1785458148.65 total 50.05 last 61.54) - open/close 4137 (time 1785458158.65 total 60.05 last 71.90) total: 4500 open/close in 64.33 seconds: 69.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1410 (time 1785458177.15 total 10.00 last 140.94) - /unlink 2651 (time 1785458187.16 total 20.01 last 124.08) - /unlink 3899 (time 1785458197.16 total 30.01 last 124.78) total: 4500 /unlink in 34.63 seconds: 129.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 679 (time 1785458217.55 total 10.01 last 67.82) - open/close 1492 (time 1785458227.57 total 20.03 last 81.15) - open/close 2251 (time 1785458237.58 total 30.04 last 75.81) - open/close 2949 (time 1785458247.58 total 40.04 last 69.78) - open/close 3707 (time 1785458257.58 total 50.05 last 75.79) - open/close 4467 (time 1785458267.59 total 60.05 last 75.95) total: 4500 open/close in 60.53 seconds: 74.35 ops/second - /unlink 1503 (time 1785458283.21 total 10.00 last 150.24) - /unlink 2845 (time 1785458293.21 total 20.01 last 134.17) - /unlink 4132 (time 1785458303.21 total 30.01 last 128.68) total: 4500 /unlink in 32.77 seconds: 137.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 677 (time 1785458322.09 total 10.00 last 67.67) - open/close 1344 (time 1785458332.10 total 20.01 last 66.65) - open/close 2078 (time 1785458342.11 total 30.03 last 73.28) - open/close 2704 (time 1785458352.12 total 40.04 last 62.53) - open/close 3541 (time 1785458362.12 total 50.04 last 83.70) - open/close 4243 (time 1785458372.13 total 60.05 last 70.16) total: 4500 open/close in 63.61 seconds: 70.74 ops/second - /unlink 1359 (time 1785458390.72 total 10.00 last 135.90) - /unlink 2524 (time 1785458400.74 total 20.02 last 116.31) - /unlink 3793 (time 1785458410.75 total 30.02 last 126.84) total: 4500 /unlink in 35.52 seconds: 126.67 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11297: 95188 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg655-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg655-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg655-server stop mds service on oleg655-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg655-server stop mds service on oleg655-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg655-server unloading modules via unload_modules_local on: 'oleg655-server' oleg655-server: oleg655-server.virtnet: executing unload_modules_local oleg655-server: modules unloaded.