== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:17:24 (1781291844) Stopping clients: oleg458-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg458-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg458-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg458-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg458-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg458-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg458-server oleg458-server: oleg458-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing load_modules_local oleg458-server: Loading modules from /home/green/git/lustre-release/lustre oleg458-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg458-server: detected 4 online CPUs by sysfs oleg458-server: MODOPTS_LIBCFS= oleg458-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg458-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg458-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg458-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 550 (time 1781292073.03 total 10.00 last 54.98) - open/close 1051 (time 1781292083.03 total 20.00 last 50.10) - open/close 1528 (time 1781292093.04 total 30.01 last 47.68) - open/close 2007 (time 1781292103.06 total 40.03 last 47.79) - open/close 2518 (time 1781292113.07 total 50.05 last 51.03) - open/close 3059 (time 1781292123.09 total 60.06 last 54.01) - open/close 3536 (time 1781292133.11 total 70.08 last 47.61) - open/close 4011 (time 1781292143.12 total 80.09 last 47.44) - open/close 4467 (time 1781292153.15 total 90.12 last 45.48) total: 4500 open/close in 90.97 seconds: 49.47 ops/second - /unlink 1041 (time 1781292170.10 total 10.01 last 103.99) - /unlink 2125 (time 1781292180.11 total 20.01 last 108.40) - /unlink 3142 (time 1781292190.12 total 30.02 last 101.60) - /unlink 4164 (time 1781292200.12 total 40.02 last 102.18) total: 4500 /unlink in 44.06 seconds: 102.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 589 (time 1781292220.57 total 10.01 last 58.87) - open/close 1242 (time 1781292230.59 total 20.03 last 65.16) - open/close 1767 (time 1781292240.62 total 30.05 last 52.38) - open/close 2395 (time 1781292250.63 total 40.06 last 62.73) - open/close 3045 (time 1781292260.63 total 50.06 last 64.99) - open/close 3597 (time 1781292270.63 total 60.06 last 55.20) - open/close 4198 (time 1781292280.65 total 70.08 last 59.98) total: 4500 open/close in 74.74 seconds: 60.21 ops/second - /unlink 1168 (time 1781292300.39 total 10.01 last 116.72) - /unlink 2301 (time 1781292310.39 total 20.01 last 113.26) - /unlink 3433 (time 1781292320.39 total 30.01 last 113.16) total: 4500 /unlink in 39.43 seconds: 114.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 594 (time 1781292345.30 total 10.00 last 59.38) - open/close 1245 (time 1781292355.30 total 20.01 last 65.07) - open/close 1904 (time 1781292365.31 total 30.01 last 65.87) - open/close 2408 (time 1781292375.33 total 40.03 last 50.31) - open/close 2914 (time 1781292385.34 total 50.04 last 50.55) - open/close 3379 (time 1781292395.35 total 60.06 last 46.42) - open/close 3927 (time 1781292405.36 total 70.07 last 54.75) - open/close 4390 (time 1781292415.37 total 80.07 last 46.28) total: 4500 open/close in 83.22 seconds: 54.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1021 (time 1781292433.86 total 10.01 last 102.04) - /unlink 2124 (time 1781292443.86 total 20.01 last 110.24) - /unlink 3261 (time 1781292453.86 total 30.01 last 113.67) - /unlink 4389 (time 1781292463.87 total 40.02 last 112.77) total: 4500 /unlink in 41.46 seconds: 108.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 496 (time 1781292481.38 total 10.01 last 49.57) - open/close 918 (time 1781292491.38 total 20.01 last 42.19) - open/close 1340 (time 1781292501.39 total 30.01 last 42.18) - open/close 1945 (time 1781292511.39 total 40.01 last 60.49) - open/close 2463 (time 1781292521.40 total 50.03 last 51.73) - open/close 2954 (time 1781292531.44 total 60.06 last 48.92) - open/close 3518 (time 1781292541.45 total 70.07 last 56.34) - open/close 4122 (time 1781292551.47 total 80.09 last 60.28) total: 4500 open/close in 85.32 seconds: 52.74 ops/second - /unlink 1009 (time 1781292571.98 total 10.02 last 100.75) - /unlink 1967 (time 1781292581.99 total 20.02 last 95.75) - /unlink 3145 (time 1781292591.99 total 30.02 last 117.75) - /unlink 4224 (time 1781292601.99 total 40.03 last 107.88) total: 4500 /unlink in 42.92 seconds: 104.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 483 (time 1781292621.42 total 10.02 last 48.23) - open/close 906 (time 1781292631.42 total 20.02 last 42.28) - open/close 1410 (time 1781292641.44 total 30.04 last 50.30) - open/close 1911 (time 1781292651.44 total 40.04 last 50.08) - open/close 2548 (time 1781292661.45 total 50.05 last 63.65) - open/close 3141 (time 1781292671.45 total 60.05 last 59.29) - open/close 3769 (time 1781292681.45 total 70.05 last 62.79) - open/close 4312 (time 1781292691.46 total 80.06 last 54.26) total: 4500 open/close in 83.33 seconds: 54.00 ops/second - /unlink 1151 (time 1781292709.87 total 10.01 last 115.04) - /unlink 2207 (time 1781292719.88 total 20.02 last 105.49) - /unlink 3212 (time 1781292729.88 total 30.02 last 100.48) - /unlink 4310 (time 1781292739.89 total 40.03 last 109.66) total: 4500 /unlink in 41.98 seconds: 107.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 610 (time 1781292757.27 total 10.00 last 60.98) - open/close 1191 (time 1781292767.28 total 20.01 last 58.05) - open/close 1814 (time 1781292777.28 total 30.02 last 62.25) - open/close 2477 (time 1781292787.29 total 40.03 last 66.25) - open/close 3064 (time 1781292797.31 total 50.04 last 58.61) - open/close 3604 (time 1781292807.31 total 60.05 last 53.98) - open/close 4209 (time 1781292817.32 total 70.05 last 60.47) total: 4500 open/close in 74.72 seconds: 60.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 912 (time 1781292837.62 total 10.00 last 91.19) - /unlink 1877 (time 1781292847.63 total 20.00 last 96.48) - /unlink 3024 (time 1781292857.63 total 30.01 last 114.67) - /unlink 4178 (time 1781292867.64 total 40.02 last 115.27) total: 4500 /unlink in 42.99 seconds: 104.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 603 (time 1781292886.10 total 10.02 last 60.21) - open/close 1166 (time 1781292896.12 total 20.03 last 56.22) - open/close 1843 (time 1781292906.13 total 30.05 last 67.60) - open/close 2402 (time 1781292916.16 total 40.07 last 55.75) - open/close 2908 (time 1781292926.16 total 50.07 last 50.60) - open/close 3561 (time 1781292936.17 total 60.08 last 65.25) - open/close 4194 (time 1781292946.17 total 70.09 last 63.25) total: 4500 open/close in 74.69 seconds: 60.25 ops/second - /unlink 1254 (time 1781292965.70 total 10.01 last 125.22) - /unlink 2299 (time 1781292975.71 total 20.02 last 104.43) - /unlink 3309 (time 1781292985.71 total 30.03 last 100.96) - /unlink 4443 (time 1781292995.72 total 40.03 last 113.31) total: 4500 /unlink in 40.86 seconds: 110.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 622 (time 1781293012.80 total 10.01 last 62.15) - open/close 1238 (time 1781293022.80 total 20.01 last 61.56) - open/close 1841 (time 1781293032.81 total 30.02 last 60.26) - open/close 2477 (time 1781293042.82 total 40.04 last 63.51) - open/close 3102 (time 1781293052.86 total 50.07 last 62.30) - open/close 3646 (time 1781293062.88 total 60.09 last 54.29) - open/close 4215 (time 1781293072.88 total 70.09 last 56.88) total: 4500 open/close in 74.59 seconds: 60.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1206 (time 1781293092.67 total 10.00 last 120.56) - /unlink 2335 (time 1781293102.67 total 20.01 last 112.88) - /unlink 3512 (time 1781293112.67 total 30.01 last 117.68) total: 4500 /unlink in 39.80 seconds: 113.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 577 (time 1781293139.14 total 10.02 last 57.57) - open/close 1192 (time 1781293149.15 total 20.03 last 61.43) - open/close 1786 (time 1781293159.16 total 30.04 last 59.33) - open/close 2458 (time 1781293169.16 total 40.05 last 67.17) - open/close 3048 (time 1781293179.18 total 50.06 last 58.92) - open/close 3653 (time 1781293189.20 total 60.08 last 60.37) - open/close 4195 (time 1781293199.21 total 70.09 last 54.16) total: 4500 open/close in 75.95 seconds: 59.25 ops/second - /unlink 1080 (time 1781293221.34 total 10.02 last 107.78) - /unlink 2198 (time 1781293231.34 total 20.02 last 111.79) - /unlink 3334 (time 1781293241.34 total 30.02 last 113.59) - /unlink 4319 (time 1781293251.35 total 40.03 last 98.46) total: 4500 /unlink in 42.26 seconds: 106.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 641 (time 1781293270.84 total 10.00 last 64.09) - open/close 1245 (time 1781293280.84 total 20.01 last 60.36) - open/close 1854 (time 1781293290.86 total 30.02 last 60.81) - open/close 2409 (time 1781293300.86 total 40.03 last 55.48) - open/close 3044 (time 1781293310.86 total 50.03 last 63.49) - open/close 3630 (time 1781293320.88 total 60.05 last 58.50) - open/close 4293 (time 1781293330.91 total 70.08 last 66.10) total: 4500 open/close in 73.13 seconds: 61.53 ops/second - /unlink 1234 (time 1781293348.68 total 10.00 last 123.36) - /unlink 2314 (time 1781293358.69 total 20.01 last 107.93) - /unlink 3404 (time 1781293368.69 total 30.01 last 108.99) total: 4500 /unlink in 39.39 seconds: 114.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 584 (time 1781293393.09 total 10.01 last 58.33) - open/close 1184 (time 1781293403.14 total 20.07 last 59.66) - open/close 1872 (time 1781293413.15 total 30.07 last 68.78) - open/close 2490 (time 1781293423.16 total 40.09 last 61.72) - open/close 3155 (time 1781293433.17 total 50.09 last 66.45) - open/close 3776 (time 1781293443.18 total 60.11 last 62.02) - open/close 4376 (time 1781293453.19 total 70.11 last 59.96) total: 4500 open/close in 71.69 seconds: 62.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1184 (time 1781293469.40 total 10.00 last 118.35) - /unlink 2245 (time 1781293479.40 total 20.00 last 106.10) - /unlink 3347 (time 1781293489.42 total 30.02 last 110.05) - /unlink 4496 (time 1781293499.42 total 40.02 last 114.82) total: 4500 /unlink in 40.05 seconds: 112.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 565 (time 1781293514.65 total 10.00 last 56.50) - open/close 1221 (time 1781293524.66 total 20.01 last 65.57) - open/close 1831 (time 1781293534.69 total 30.03 last 60.82) - open/close 2407 (time 1781293544.69 total 40.03 last 57.59) - open/close 3037 (time 1781293554.70 total 50.05 last 62.90) - open/close 3590 (time 1781293564.71 total 60.05 last 55.28) - open/close 4170 (time 1781293574.72 total 70.07 last 57.92) total: 4500 open/close in 75.34 seconds: 59.73 ops/second - /unlink 1112 (time 1781293595.23 total 10.01 last 111.11) - /unlink 2308 (time 1781293605.24 total 20.01 last 119.53) - /unlink 3473 (time 1781293615.24 total 30.02 last 116.45) total: 4500 /unlink in 39.30 seconds: 114.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 600 (time 1781293640.64 total 10.01 last 59.96) - open/close 1181 (time 1781293650.64 total 20.01 last 58.10) - open/close 1789 (time 1781293660.67 total 30.03 last 60.67) - open/close 2432 (time 1781293670.67 total 40.03 last 64.26) - open/close 3074 (time 1781293680.68 total 50.04 last 64.15) - open/close 3688 (time 1781293690.69 total 60.05 last 61.33) - open/close 4320 (time 1781293700.70 total 70.06 last 63.16) total: 4500 open/close in 72.45 seconds: 62.11 ops/second - /unlink 1166 (time 1781293717.70 total 10.01 last 116.45) - /unlink 2375 (time 1781293727.70 total 20.02 last 120.86) - /unlink 3550 (time 1781293737.70 total 30.02 last 117.47) total: 4500 /unlink in 37.81 seconds: 119.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 572 (time 1781293759.85 total 10.00 last 57.19) - open/close 1210 (time 1781293769.85 total 20.01 last 63.77) - open/close 1824 (time 1781293779.86 total 30.01 last 61.38) - open/close 2442 (time 1781293789.86 total 40.01 last 61.80) - open/close 3077 (time 1781293799.86 total 50.02 last 63.45) - open/close 3712 (time 1781293809.87 total 60.02 last 63.49) - open/close 4288 (time 1781293819.87 total 70.02 last 57.59) total: 4500 open/close in 73.79 seconds: 60.99 ops/second - /unlink 1189 (time 1781293838.29 total 10.01 last 118.83) - /unlink 2254 (time 1781293848.29 total 20.01 last 106.47) - /unlink 3055 (time 1781293858.30 total 30.01 last 80.07) - /unlink 3936 (time 1781293868.31 total 40.02 last 88.03) total: 4500 /unlink in 46.00 seconds: 97.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 601 (time 1781293889.75 total 10.01 last 60.05) - open/close 1235 (time 1781293899.76 total 20.01 last 63.38) - open/close 1873 (time 1781293909.76 total 30.02 last 63.77) - open/close 2463 (time 1781293919.79 total 40.04 last 58.85) - open/close 3008 (time 1781293929.80 total 50.05 last 54.45) - open/close 3618 (time 1781293939.80 total 60.05 last 60.97) - open/close 4194 (time 1781293949.81 total 70.06 last 57.57) total: 4500 open/close in 75.72 seconds: 59.43 ops/second - /unlink 1131 (time 1781293970.42 total 10.01 last 113.04) - /unlink 2229 (time 1781293980.43 total 20.01 last 109.74) - /unlink 3421 (time 1781293990.43 total 30.01 last 119.17) total: 4500 /unlink in 39.89 seconds: 112.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 625 (time 1781294015.69 total 10.00 last 62.48) - open/close 1212 (time 1781294025.70 total 20.01 last 58.63) - open/close 1857 (time 1781294035.70 total 30.02 last 64.48) - open/close 2510 (time 1781294045.71 total 40.03 last 65.25) - open/close 3205 (time 1781294055.72 total 50.04 last 69.42) - open/close 3789 (time 1781294065.72 total 60.04 last 58.39) - open/close 4404 (time 1781294075.73 total 70.04 last 61.46) total: 4500 open/close in 71.73 seconds: 62.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1265 (time 1781294091.87 total 10.01 last 126.42) - /unlink 2435 (time 1781294101.87 total 20.01 last 116.96) - /unlink 3593 (time 1781294111.88 total 30.01 last 115.76) total: 4500 /unlink in 37.44 seconds: 120.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 657 (time 1781294134.21 total 10.01 last 65.63) - open/close 1261 (time 1781294144.22 total 20.02 last 60.35) - open/close 1941 (time 1781294154.22 total 30.02 last 67.98) - open/close 2539 (time 1781294164.25 total 40.04 last 59.66) - open/close 3098 (time 1781294174.25 total 50.05 last 55.89) - open/close 3743 (time 1781294184.25 total 60.05 last 64.48) - open/close 4355 (time 1781294194.27 total 70.06 last 61.12) total: 4500 open/close in 72.32 seconds: 62.22 ops/second - /unlink 1106 (time 1781294211.38 total 10.01 last 110.51) - /unlink 2324 (time 1781294221.39 total 20.01 last 121.76) - /unlink 3551 (time 1781294231.39 total 30.01 last 122.70) total: 4500 /unlink in 37.98 seconds: 118.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 586 (time 1781294254.33 total 10.01 last 58.55) - open/close 1235 (time 1781294264.34 total 20.02 last 64.82) - open/close 1887 (time 1781294274.35 total 30.03 last 65.15) - open/close 2555 (time 1781294284.36 total 40.04 last 66.72) - open/close 3187 (time 1781294294.39 total 50.07 last 63.04) - open/close 3820 (time 1781294304.39 total 60.07 last 63.30) - open/close 4420 (time 1781294314.39 total 70.07 last 59.97) total: 4500 open/close in 71.16 seconds: 63.24 ops/second - /unlink 1117 (time 1781294330.57 total 10.02 last 111.52) - /unlink 2131 (time 1781294340.57 total 20.02 last 101.36) - /unlink 3155 (time 1781294350.58 total 30.03 last 102.29) - /unlink 4254 (time 1781294360.59 total 40.04 last 109.80) total: 4500 /unlink in 42.10 seconds: 106.88 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11221: 94840 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg458-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg458-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg458-server stop mds service on oleg458-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg458-server stop mds service on oleg458-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg458-server unloading modules via unload_modules_local on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing unload_modules_local oleg458-server: modules unloaded.