== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:15:03 (1775870103) Stopping clients: oleg214-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg214-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg214-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg214-server oleg214-server: oleg214-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg214-server: detected 4 online CPUs by sysfs oleg214-server: MODOPTS_LIBCFS= oleg214-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg214-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg214-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg214-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 689 (time 1775870326.81 total 10.01 last 68.84) - open/close 1282 (time 1775870336.83 total 20.03 last 59.16) - open/close 1974 (time 1775870346.83 total 30.03 last 69.19) - open/close 2732 (time 1775870356.87 total 40.07 last 75.54) - open/close 3259 (time 1775870366.87 total 50.07 last 52.68) - open/close 3851 (time 1775870376.90 total 60.11 last 59.01) - open/close 4380 (time 1775870386.93 total 70.13 last 52.74) total: 4500 open/close in 72.10 seconds: 62.42 ops/second - /unlink 1307 (time 1775870404.77 total 10.00 last 130.66) - /unlink 2472 (time 1775870414.77 total 20.00 last 116.49) - /unlink 3751 (time 1775870424.78 total 30.01 last 127.82) total: 4500 /unlink in 36.50 seconds: 123.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 792 (time 1775870447.00 total 10.01 last 79.12) - open/close 1482 (time 1775870457.00 total 20.02 last 68.96) - open/close 2121 (time 1775870467.01 total 30.02 last 63.89) - open/close 2703 (time 1775870477.01 total 40.02 last 58.18) - open/close 3419 (time 1775870487.01 total 50.03 last 71.56) - open/close 4171 (time 1775870497.03 total 60.04 last 75.10) total: 4500 open/close in 64.48 seconds: 69.79 ops/second - /unlink 1277 (time 1775870516.85 total 10.00 last 127.64) - /unlink 2398 (time 1775870526.86 total 20.01 last 112.05) - /unlink 3847 (time 1775870536.86 total 30.01 last 144.83) total: 4500 /unlink in 33.89 seconds: 132.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 832 (time 1775870556.24 total 10.01 last 83.10) - open/close 1587 (time 1775870566.24 total 20.01 last 75.49) - open/close 2309 (time 1775870576.24 total 30.01 last 72.20) - open/close 3048 (time 1775870586.25 total 40.02 last 73.82) - open/close 3724 (time 1775870596.26 total 50.03 last 67.53) - open/close 4389 (time 1775870606.28 total 60.05 last 66.40) total: 4500 open/close in 61.53 seconds: 73.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1270 (time 1775870622.74 total 10.00 last 127.00) - /unlink 2475 (time 1775870632.76 total 20.02 last 120.27) - /unlink 3452 (time 1775870642.76 total 30.02 last 97.67) total: 4500 /unlink in 37.65 seconds: 119.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 683 (time 1775870666.91 total 10.01 last 68.22) - open/close 1318 (time 1775870676.93 total 20.02 last 63.43) - open/close 1858 (time 1775870686.95 total 30.05 last 53.84) - open/close 2410 (time 1775870696.96 total 40.05 last 55.19) - open/close 3081 (time 1775870706.96 total 50.05 last 67.09) - open/close 3761 (time 1775870716.96 total 60.06 last 67.99) total: 4500 open/close in 69.97 seconds: 64.31 ops/second - /unlink 1320 (time 1775870741.66 total 10.00 last 132.00) - /unlink 2697 (time 1775870751.66 total 20.00 last 137.68) - /unlink 4109 (time 1775870761.67 total 30.00 last 141.16) total: 4500 /unlink in 33.37 seconds: 134.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 681 (time 1775870780.46 total 10.02 last 67.98) - open/close 1362 (time 1775870790.46 total 20.02 last 68.06) - open/close 2098 (time 1775870800.47 total 30.03 last 73.58) - open/close 2780 (time 1775870810.47 total 40.03 last 68.18) - open/close 3429 (time 1775870820.48 total 50.04 last 64.83) - open/close 4171 (time 1775870830.49 total 60.05 last 74.11) total: 4500 open/close in 64.93 seconds: 69.31 ops/second - /unlink 1423 (time 1775870850.46 total 10.02 last 142.00) - /unlink 2721 (time 1775870860.47 total 20.04 last 129.58) - /unlink 4073 (time 1775870870.48 total 30.04 last 135.19) total: 4500 /unlink in 33.12 seconds: 135.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 668 (time 1775870889.37 total 10.03 last 66.58) - open/close 1300 (time 1775870899.38 total 20.04 last 63.16) - open/close 1991 (time 1775870909.38 total 30.05 last 69.04) - open/close 2802 (time 1775870919.39 total 40.05 last 81.05) - open/close 3439 (time 1775870929.40 total 50.06 last 63.67) - open/close 4104 (time 1775870939.41 total 60.07 last 66.42) total: 4500 open/close in 65.54 seconds: 68.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1436 (time 1775870960.00 total 10.00 last 143.58) - /unlink 2765 (time 1775870970.00 total 20.00 last 132.89) - /unlink 4351 (time 1775870980.01 total 30.01 last 158.51) total: 4500 /unlink in 31.42 seconds: 143.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 711 (time 1775870995.95 total 10.01 last 71.05) - open/close 1360 (time 1775871005.96 total 20.03 last 64.78) - open/close 1930 (time 1775871015.98 total 30.04 last 56.91) - open/close 2494 (time 1775871025.98 total 40.04 last 56.38) - open/close 3191 (time 1775871035.99 total 50.05 last 69.64) - open/close 3890 (time 1775871046.00 total 60.06 last 69.88) - open/close 4453 (time 1775871056.00 total 70.06 last 56.27) total: 4500 open/close in 70.81 seconds: 63.55 ops/second - /unlink 1327 (time 1775871071.59 total 10.00 last 132.66) - /unlink 2979 (time 1775871081.60 total 20.01 last 165.07) - /unlink 4349 (time 1775871091.60 total 30.02 last 136.93) total: 4500 /unlink in 32.31 seconds: 139.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 684 (time 1775871110.21 total 10.01 last 68.34) - open/close 1419 (time 1775871120.22 total 20.01 last 73.48) - open/close 2098 (time 1775871130.24 total 30.03 last 67.78) - open/close 2536 (time 1775871140.25 total 40.04 last 43.74) - open/close 3164 (time 1775871150.26 total 50.05 last 62.74) - open/close 3907 (time 1775871160.26 total 60.06 last 74.28) total: 4500 open/close in 68.77 seconds: 65.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1613 (time 1775871184.04 total 10.00 last 161.25) - /unlink 3184 (time 1775871194.04 total 20.01 last 157.06) total: 4500 /unlink in 29.70 seconds: 151.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 749 (time 1775871218.38 total 10.01 last 74.86) - open/close 1429 (time 1775871228.39 total 20.01 last 67.96) - open/close 2120 (time 1775871238.39 total 30.01 last 69.09) - open/close 2835 (time 1775871248.41 total 40.03 last 71.38) - open/close 3419 (time 1775871258.43 total 50.05 last 58.29) - open/close 4138 (time 1775871268.43 total 60.05 last 71.86) total: 4500 open/close in 64.35 seconds: 69.93 ops/second - /unlink 1635 (time 1775871286.72 total 10.01 last 163.41) - /unlink 2986 (time 1775871296.72 total 20.01 last 135.06) - /unlink 4025 (time 1775871306.73 total 30.01 last 103.86) total: 4500 /unlink in 33.40 seconds: 134.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 805 (time 1775871325.71 total 10.01 last 80.45) - open/close 1680 (time 1775871335.72 total 20.01 last 87.47) - open/close 2250 (time 1775871345.72 total 30.02 last 56.96) - open/close 2999 (time 1775871355.73 total 40.03 last 74.82) - open/close 3538 (time 1775871365.74 total 50.03 last 53.86) - open/close 4218 (time 1775871375.75 total 60.04 last 67.96) total: 4500 open/close in 64.19 seconds: 70.10 ops/second - /unlink 1515 (time 1775871394.91 total 10.01 last 151.41) - /unlink 3112 (time 1775871404.91 total 20.01 last 159.69) - /unlink 4317 (time 1775871414.91 total 30.01 last 120.50) total: 4500 /unlink in 31.61 seconds: 142.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 737 (time 1775871430.90 total 10.01 last 73.63) - open/close 1422 (time 1775871440.91 total 20.01 last 68.48) - open/close 2235 (time 1775871450.91 total 30.01 last 81.29) - open/close 3103 (time 1775871460.92 total 40.03 last 86.69) - open/close 3855 (time 1775871470.93 total 50.03 last 75.15) total: 4500 open/close in 57.24 seconds: 78.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1431 (time 1775871492.62 total 10.00 last 143.05) - /unlink 2919 (time 1775871502.64 total 20.02 last 148.61) - /unlink 4185 (time 1775871512.64 total 30.02 last 126.52) total: 4500 /unlink in 32.26 seconds: 139.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 698 (time 1775871530.26 total 10.00 last 69.77) - open/close 1474 (time 1775871540.26 total 20.00 last 77.60) - open/close 2341 (time 1775871550.27 total 30.01 last 86.66) - open/close 3155 (time 1775871560.28 total 40.02 last 81.31) - open/close 3891 (time 1775871570.28 total 50.03 last 73.56) total: 4500 open/close in 57.83 seconds: 77.81 ops/second - /unlink 1458 (time 1775871592.71 total 10.00 last 145.77) - /unlink 2842 (time 1775871602.71 total 20.01 last 138.33) - /unlink 4270 (time 1775871612.71 total 30.01 last 142.79) total: 4500 /unlink in 31.26 seconds: 143.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 682 (time 1775871628.83 total 10.02 last 68.09) - open/close 1349 (time 1775871638.84 total 20.02 last 66.65) - open/close 2160 (time 1775871648.89 total 30.07 last 80.72) - open/close 2988 (time 1775871658.90 total 40.08 last 82.74) - open/close 3841 (time 1775871668.92 total 50.10 last 85.11) - open/close 4464 (time 1775871678.93 total 60.11 last 62.21) total: 4500 open/close in 60.55 seconds: 74.32 ops/second - /unlink 1555 (time 1775871694.83 total 10.01 last 155.32) - /unlink 2966 (time 1775871704.84 total 20.02 last 140.91) - /unlink 4191 (time 1775871714.85 total 30.03 last 122.42) total: 4500 /unlink in 32.03 seconds: 140.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 686 (time 1775871732.37 total 10.01 last 68.56) - open/close 1226 (time 1775871742.37 total 20.01 last 53.98) - open/close 1964 (time 1775871752.39 total 30.03 last 73.65) - open/close 2732 (time 1775871762.40 total 40.04 last 76.75) - open/close 3514 (time 1775871772.40 total 50.04 last 78.18) - open/close 4312 (time 1775871782.40 total 60.04 last 79.78) total: 4500 open/close in 62.42 seconds: 72.09 ops/second - /unlink 1202 (time 1775871799.70 total 10.00 last 120.19) - /unlink 2424 (time 1775871809.70 total 20.00 last 122.19) - /unlink 3617 (time 1775871819.70 total 30.01 last 119.21) total: 4500 /unlink in 37.53 seconds: 119.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 739 (time 1775871842.57 total 10.00 last 73.87) - open/close 1415 (time 1775871852.58 total 20.01 last 67.55) - open/close 2098 (time 1775871862.58 total 30.01 last 68.29) - open/close 2763 (time 1775871872.58 total 40.01 last 66.49) - open/close 3497 (time 1775871882.59 total 50.02 last 73.38) - open/close 4181 (time 1775871892.61 total 60.04 last 68.25) total: 4500 open/close in 64.29 seconds: 70.00 ops/second - /unlink 1239 (time 1775871912.34 total 10.00 last 123.87) - /unlink 2713 (time 1775871922.35 total 20.01 last 147.28) - /unlink 4132 (time 1775871932.35 total 30.01 last 141.86) total: 4500 /unlink in 32.53 seconds: 138.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 688 (time 1775871950.22 total 10.01 last 68.73) - open/close 1445 (time 1775871960.23 total 20.02 last 75.62) - open/close 2297 (time 1775871970.23 total 30.02 last 85.20) - open/close 3114 (time 1775871980.23 total 40.02 last 81.67) - open/close 3949 (time 1775871990.24 total 50.03 last 83.46) total: 4500 open/close in 56.96 seconds: 79.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1346 (time 1775872011.82 total 10.00 last 134.53) - /unlink 2725 (time 1775872021.82 total 20.01 last 137.87) - /unlink 3876 (time 1775872031.82 total 30.01 last 115.07) total: 4500 /unlink in 35.21 seconds: 127.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 606 (time 1775872052.41 total 10.01 last 60.52) - open/close 1104 (time 1775872062.41 total 20.02 last 49.77) - open/close 1600 (time 1775872072.41 total 30.02 last 49.60) - open/close 2140 (time 1775872082.42 total 40.03 last 53.97) - open/close 2785 (time 1775872092.43 total 50.04 last 64.44) - open/close 3378 (time 1775872102.43 total 60.04 last 59.30) - open/close 4130 (time 1775872112.44 total 70.05 last 75.09) total: 4500 open/close in 74.88 seconds: 60.10 ops/second - /unlink 1370 (time 1775872131.85 total 10.02 last 136.75) - /unlink 2868 (time 1775872141.86 total 20.03 last 149.68) - /unlink 4139 (time 1775872151.88 total 30.05 last 126.81) total: 4500 /unlink in 33.40 seconds: 134.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 695 (time 1775872170.75 total 10.01 last 69.46) - open/close 1251 (time 1775872180.75 total 20.01 last 55.59) - open/close 1861 (time 1775872190.75 total 30.01 last 60.99) - open/close 2626 (time 1775872200.76 total 40.01 last 76.47) - open/close 3327 (time 1775872210.77 total 50.03 last 70.02) - open/close 3970 (time 1775872220.79 total 60.04 last 64.18) total: 4500 open/close in 69.43 seconds: 64.81 ops/second - /unlink 1231 (time 1775872245.52 total 10.00 last 123.10) - /unlink 2204 (time 1775872255.53 total 20.01 last 97.18) - /unlink 3407 (time 1775872265.54 total 30.02 last 120.26) - /unlink 4390 (time 1775872275.54 total 40.02 last 98.24) total: 4500 /unlink in 41.96 seconds: 107.25 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11187: 94021 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg214-server unloading modules via unload_modules_local on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local oleg214-server: modules unloaded.