== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:40:02 (1788795602) Stopping clients: oleg252-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg252-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg252-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg252-server oleg252-server: oleg252-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing load_modules_local oleg252-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg252-server: Loading modules from /home/green/git/lustre-release/lustre oleg252-server: detected 4 online CPUs by sysfs oleg252-server: MODOPTS_LIBCFS= oleg252-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg252-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg252-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg252-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 572 (time 1788795835.81 total 10.00 last 57.19) - open/close 1161 (time 1788795845.81 total 20.00 last 58.89) - open/close 1877 (time 1788795855.82 total 30.01 last 71.57) - open/close 2446 (time 1788795865.84 total 40.03 last 56.77) - open/close 3002 (time 1788795875.84 total 50.03 last 55.59) - open/close 3606 (time 1788795885.84 total 60.03 last 60.39) - open/close 4276 (time 1788795895.86 total 70.05 last 66.92) total: 4500 open/close in 73.48 seconds: 61.24 ops/second - /unlink 1315 (time 1788795914.22 total 10.00 last 131.50) - /unlink 2683 (time 1788795924.22 total 20.01 last 136.69) - /unlink 3944 (time 1788795934.23 total 30.01 last 126.08) total: 4500 /unlink in 34.76 seconds: 129.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 662 (time 1788795953.75 total 10.02 last 66.06) - open/close 1372 (time 1788795963.77 total 20.04 last 70.90) - open/close 2038 (time 1788795973.77 total 30.04 last 66.56) - open/close 2596 (time 1788795983.78 total 40.05 last 55.77) - open/close 3195 (time 1788795993.78 total 50.05 last 59.87) - open/close 3833 (time 1788796003.80 total 60.07 last 63.71) - open/close 4429 (time 1788796013.80 total 70.07 last 59.59) total: 4500 open/close in 71.12 seconds: 63.28 ops/second - /unlink 1116 (time 1788796030.42 total 10.01 last 111.50) - /unlink 2289 (time 1788796040.42 total 20.01 last 117.29) - /unlink 3520 (time 1788796050.43 total 30.01 last 123.05) total: 4500 /unlink in 37.22 seconds: 120.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 699 (time 1788796072.57 total 10.00 last 69.90) - open/close 1456 (time 1788796082.57 total 20.00 last 75.70) - open/close 2124 (time 1788796092.57 total 30.00 last 66.79) - open/close 2707 (time 1788796102.58 total 40.01 last 58.24) - open/close 3216 (time 1788796112.60 total 50.03 last 50.82) - open/close 3586 (time 1788796122.62 total 60.05 last 36.93) - open/close 4135 (time 1788796132.63 total 70.06 last 54.84) total: 4500 open/close in 77.49 seconds: 58.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 794 (time 1788796157.13 total 10.01 last 79.35) - /unlink 1625 (time 1788796167.15 total 20.02 last 82.96) - /unlink 2488 (time 1788796177.15 total 30.03 last 86.25) - /unlink 3548 (time 1788796187.17 total 40.04 last 105.87) total: 4500 /unlink in 47.60 seconds: 94.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 701 (time 1788796209.79 total 10.01 last 70.02) - open/close 1323 (time 1788796219.79 total 20.01 last 62.20) - open/close 1922 (time 1788796229.79 total 30.02 last 59.87) - open/close 2432 (time 1788796239.81 total 40.04 last 50.89) - open/close 2986 (time 1788796249.81 total 50.04 last 55.40) - open/close 3594 (time 1788796259.82 total 60.05 last 60.75) - open/close 4129 (time 1788796269.82 total 70.05 last 53.50) total: 4500 open/close in 75.70 seconds: 59.45 ops/second - /unlink 1055 (time 1788796291.29 total 10.00 last 105.49) - /unlink 1994 (time 1788796301.29 total 20.00 last 93.86) - /unlink 3130 (time 1788796311.29 total 30.01 last 113.56) - /unlink 4384 (time 1788796321.30 total 40.02 last 125.31) total: 4500 /unlink in 41.20 seconds: 109.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 684 (time 1788796337.59 total 10.00 last 68.37) - open/close 1398 (time 1788796347.59 total 20.01 last 71.36) - open/close 2091 (time 1788796357.60 total 30.02 last 69.26) - open/close 2860 (time 1788796367.60 total 40.02 last 76.88) - open/close 3596 (time 1788796377.61 total 50.03 last 73.53) - open/close 4330 (time 1788796387.62 total 60.04 last 73.33) total: 4500 open/close in 62.03 seconds: 72.55 ops/second - /unlink 1339 (time 1788796404.62 total 10.01 last 133.81) - /unlink 2438 (time 1788796414.63 total 20.01 last 109.86) - /unlink 3706 (time 1788796424.63 total 30.02 last 126.75) total: 4500 /unlink in 35.72 seconds: 125.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 625 (time 1788796444.95 total 10.01 last 62.42) - open/close 1349 (time 1788796454.96 total 20.02 last 72.34) - open/close 1993 (time 1788796464.97 total 30.03 last 64.33) - open/close 2686 (time 1788796474.98 total 40.05 last 69.18) - open/close 3377 (time 1788796484.99 total 50.05 last 69.09) - open/close 3989 (time 1788796495.00 total 60.07 last 61.11) total: 4500 open/close in 69.87 seconds: 64.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 991 (time 1788796521.08 total 10.00 last 99.05) - /unlink 1928 (time 1788796531.13 total 20.06 last 93.23) - /unlink 2875 (time 1788796541.14 total 30.06 last 94.65) - /unlink 3730 (time 1788796551.14 total 40.06 last 85.47) total: 4500 /unlink in 45.24 seconds: 99.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 648 (time 1788796571.67 total 10.00 last 64.77) - open/close 1219 (time 1788796581.68 total 20.01 last 57.05) - open/close 1822 (time 1788796591.68 total 30.02 last 60.27) - open/close 2506 (time 1788796601.70 total 40.03 last 68.32) - open/close 3119 (time 1788796611.70 total 50.03 last 61.27) - open/close 3824 (time 1788796621.71 total 60.04 last 70.45) total: 4500 open/close in 69.97 seconds: 64.31 ops/second - /unlink 1320 (time 1788796647.77 total 10.00 last 131.98) - /unlink 2450 (time 1788796657.77 total 20.00 last 112.98) - /unlink 3688 (time 1788796667.77 total 30.01 last 123.77) total: 4500 /unlink in 38.53 seconds: 116.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 380 (time 1788796693.61 total 10.01 last 37.97) - open/close 889 (time 1788796703.61 total 20.01 last 50.88) - open/close 1314 (time 1788796713.62 total 30.02 last 42.48) - open/close 1743 (time 1788796723.63 total 40.03 last 42.86) - open/close 2320 (time 1788796733.63 total 50.03 last 57.68) - open/close 3020 (time 1788796743.63 total 60.03 last 69.98) - open/close 3677 (time 1788796753.63 total 70.03 last 65.69) - open/close 4332 (time 1788796763.64 total 80.04 last 65.49) total: 4500 open/close in 82.31 seconds: 54.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1283 (time 1788796780.43 total 10.01 last 128.22) - /unlink 2429 (time 1788796790.43 total 20.01 last 114.59) - /unlink 3655 (time 1788796800.43 total 30.01 last 122.59) total: 4500 /unlink in 35.93 seconds: 125.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 620 (time 1788796820.79 total 10.02 last 61.86) - open/close 1323 (time 1788796830.79 total 20.03 last 70.24) - open/close 1963 (time 1788796840.82 total 30.06 last 63.82) - open/close 2718 (time 1788796850.84 total 40.07 last 75.39) - open/close 3530 (time 1788796860.84 total 50.07 last 81.20) - open/close 4196 (time 1788796870.84 total 60.08 last 66.56) total: 4500 open/close in 64.12 seconds: 70.18 ops/second - /unlink 1047 (time 1788796891.94 total 10.01 last 104.62) - /unlink 2002 (time 1788796901.94 total 20.01 last 95.48) - /unlink 3043 (time 1788796911.95 total 30.02 last 104.04) - /unlink 4355 (time 1788796921.95 total 40.02 last 131.17) total: 4500 /unlink in 41.16 seconds: 109.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 577 (time 1788796937.56 total 10.00 last 57.69) - open/close 1225 (time 1788796947.56 total 20.00 last 64.78) - open/close 2007 (time 1788796957.57 total 30.02 last 78.10) - open/close 2777 (time 1788796967.58 total 40.02 last 76.96) - open/close 3554 (time 1788796977.59 total 50.03 last 77.65) - open/close 4212 (time 1788796987.60 total 60.04 last 65.73) total: 4500 open/close in 64.12 seconds: 70.18 ops/second - /unlink 1215 (time 1788797007.57 total 10.00 last 121.44) - /unlink 2380 (time 1788797017.57 total 20.01 last 116.47) - /unlink 3710 (time 1788797027.58 total 30.01 last 132.93) total: 4500 /unlink in 36.46 seconds: 123.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 572 (time 1788797050.16 total 10.01 last 57.16) - open/close 1087 (time 1788797060.16 total 20.02 last 51.45) - open/close 1720 (time 1788797070.20 total 30.05 last 63.10) - open/close 2458 (time 1788797080.21 total 40.06 last 73.72) - open/close 3150 (time 1788797090.21 total 50.07 last 69.15) - open/close 3875 (time 1788797100.22 total 60.07 last 72.46) - open/close 4456 (time 1788797110.22 total 70.07 last 58.09) total: 4500 open/close in 70.73 seconds: 63.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1230 (time 1788797126.50 total 10.01 last 122.92) - /unlink 2343 (time 1788797136.51 total 20.01 last 111.25) - /unlink 3358 (time 1788797146.51 total 30.01 last 101.46) total: 4500 /unlink in 39.45 seconds: 114.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 548 (time 1788797171.49 total 10.01 last 54.77) - open/close 1279 (time 1788797181.49 total 20.01 last 73.07) - open/close 1873 (time 1788797191.49 total 30.01 last 59.39) - open/close 2400 (time 1788797201.51 total 40.03 last 52.61) - open/close 2959 (time 1788797211.51 total 50.03 last 55.88) - open/close 3575 (time 1788797221.53 total 60.05 last 61.51) - open/close 4206 (time 1788797231.54 total 70.06 last 63.06) total: 4500 open/close in 74.92 seconds: 60.07 ops/second - /unlink 1259 (time 1788797251.66 total 10.00 last 125.87) - /unlink 2462 (time 1788797261.66 total 20.00 last 120.27) - /unlink 3463 (time 1788797271.67 total 30.02 last 99.99) total: 4500 /unlink in 38.82 seconds: 115.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 698 (time 1788797296.12 total 10.01 last 69.73) - open/close 1355 (time 1788797306.13 total 20.02 last 65.61) - open/close 2006 (time 1788797316.14 total 30.03 last 65.07) - open/close 2623 (time 1788797326.15 total 40.04 last 61.62) - open/close 3164 (time 1788797336.16 total 50.05 last 54.07) - open/close 3756 (time 1788797346.17 total 60.06 last 59.11) - open/close 4297 (time 1788797356.19 total 70.08 last 54.00) total: 4500 open/close in 73.33 seconds: 61.36 ops/second - /unlink 1231 (time 1788797375.21 total 10.01 last 123.03) - /unlink 2401 (time 1788797385.21 total 20.01 last 116.96) - /unlink 3586 (time 1788797395.21 total 30.01 last 118.50) total: 4500 /unlink in 39.81 seconds: 113.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 612 (time 1788797420.67 total 10.01 last 61.16) - open/close 1238 (time 1788797430.68 total 20.02 last 62.54) - open/close 1674 (time 1788797440.70 total 30.03 last 43.52) - open/close 2236 (time 1788797450.70 total 40.04 last 56.17) - open/close 2789 (time 1788797460.72 total 50.06 last 55.20) - open/close 3350 (time 1788797470.73 total 60.07 last 56.02) - open/close 3875 (time 1788797480.73 total 70.07 last 52.50) total: 4500 open/close in 79.82 seconds: 56.38 ops/second - /unlink 1450 (time 1788797506.73 total 10.00 last 144.98) - /unlink 2442 (time 1788797516.73 total 20.00 last 99.18) - /unlink 3548 (time 1788797526.74 total 30.01 last 110.51) total: 4500 /unlink in 38.13 seconds: 118.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 561 (time 1788797551.74 total 10.00 last 56.09) - open/close 1004 (time 1788797561.75 total 20.01 last 44.24) - open/close 1563 (time 1788797571.77 total 30.03 last 55.82) - open/close 2023 (time 1788797581.77 total 40.03 last 45.99) - open/close 2654 (time 1788797591.77 total 50.03 last 63.08) - open/close 3304 (time 1788797601.78 total 60.04 last 64.98) - open/close 3836 (time 1788797611.79 total 70.05 last 53.13) - open/close 4399 (time 1788797621.80 total 80.06 last 56.26) total: 4500 open/close in 81.80 seconds: 55.01 ops/second - /unlink 1491 (time 1788797639.08 total 10.01 last 148.96) - /unlink 2691 (time 1788797649.08 total 20.01 last 119.93) - /unlink 3549 (time 1788797659.09 total 30.02 last 85.73) - /unlink 4351 (time 1788797669.10 total 40.03 last 80.13) total: 4500 /unlink in 42.18 seconds: 106.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 513 (time 1788797688.00 total 10.01 last 51.27) - open/close 1016 (time 1788797698.01 total 20.02 last 50.25) - open/close 1665 (time 1788797708.03 total 30.04 last 64.76) - open/close 2210 (time 1788797718.04 total 40.04 last 54.48) - open/close 2769 (time 1788797728.05 total 50.06 last 55.81) - open/close 3323 (time 1788797738.06 total 60.07 last 55.35) - open/close 3905 (time 1788797748.07 total 70.07 last 58.15) total: 4500 open/close in 79.81 seconds: 56.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1015 (time 1788797774.05 total 10.00 last 101.49) - /unlink 2134 (time 1788797784.05 total 20.01 last 111.83) - /unlink 3212 (time 1788797794.05 total 30.01 last 107.79) - /unlink 4256 (time 1788797804.06 total 40.02 last 104.31) total: 4500 /unlink in 42.23 seconds: 106.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 635 (time 1788797822.08 total 10.01 last 63.41) - open/close 1228 (time 1788797832.08 total 20.02 last 59.25) - open/close 1910 (time 1788797842.10 total 30.04 last 68.07) - open/close 2550 (time 1788797852.12 total 40.06 last 63.91) - open/close 3276 (time 1788797862.14 total 50.08 last 72.44) - open/close 3913 (time 1788797872.15 total 60.08 last 63.66) total: 4500 open/close in 69.52 seconds: 64.73 ops/second - /unlink 1112 (time 1788797897.34 total 10.00 last 111.16) - /unlink 2189 (time 1788797907.34 total 20.01 last 107.67) - /unlink 3098 (time 1788797917.34 total 30.01 last 90.87) - /unlink 3825 (time 1788797927.36 total 40.02 last 72.63) total: 4500 /unlink in 47.89 seconds: 93.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 556 (time 1788797952.03 total 10.02 last 55.47) - open/close 1147 (time 1788797962.04 total 20.03 last 59.07) - open/close 1772 (time 1788797972.04 total 30.03 last 62.48) - open/close 2368 (time 1788797982.06 total 40.05 last 59.49) - open/close 2991 (time 1788797992.09 total 50.08 last 62.10) - open/close 3548 (time 1788798002.10 total 60.10 last 55.63) - open/close 4233 (time 1788798012.15 total 70.15 last 68.16) total: 4500 open/close in 74.23 seconds: 60.62 ops/second - /unlink 1163 (time 1788798031.48 total 10.01 last 116.16) - /unlink 2243 (time 1788798041.49 total 20.01 last 107.98) - /unlink 3331 (time 1788798051.49 total 30.02 last 108.78) - /unlink 4482 (time 1788798061.49 total 40.02 last 115.08) total: 4500 /unlink in 40.14 seconds: 112.10 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95347 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg252-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg252-server unloading modules via unload_modules_local on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing unload_modules_local oleg252-server: modules unloaded.