== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:42:38 (1788640958) Stopping clients: oleg412-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg412-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg412-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg412-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg412-server oleg412-server: oleg412-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing load_modules_local oleg412-server: kmemleak disabled oleg412-server: Loading modules from /home/green/git/lustre-release/lustre oleg412-server: detected 4 online CPUs by sysfs oleg412-server: MODOPTS_LIBCFS= oleg412-server: Force libcfs to create 2 CPU partitions oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg412-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg412-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg412-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 639 (time 1788641179.92 total 10.01 last 63.85) - open/close 1271 (time 1788641189.92 total 20.01 last 63.19) - open/close 1960 (time 1788641199.93 total 30.01 last 68.89) - open/close 2608 (time 1788641209.94 total 40.02 last 64.72) - open/close 3214 (time 1788641219.94 total 50.02 last 60.59) - open/close 3836 (time 1788641229.94 total 60.03 last 62.18) - open/close 4398 (time 1788641239.96 total 70.04 last 56.12) total: 4500 open/close in 71.68 seconds: 62.78 ops/second - /unlink 1194 (time 1788641256.83 total 10.00 last 119.36) - /unlink 2261 (time 1788641266.84 total 20.01 last 106.58) - /unlink 3473 (time 1788641276.84 total 30.02 last 121.11) total: 4500 /unlink in 38.34 seconds: 117.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 685 (time 1788641300.26 total 10.00 last 68.50) - open/close 1357 (time 1788641310.26 total 20.00 last 67.18) - open/close 2062 (time 1788641320.27 total 30.00 last 70.50) - open/close 2722 (time 1788641330.28 total 40.02 last 65.88) - open/close 3360 (time 1788641340.30 total 50.04 last 63.68) - open/close 4009 (time 1788641350.31 total 60.05 last 64.83) total: 4500 open/close in 67.98 seconds: 66.20 ops/second - /unlink 1171 (time 1788641373.36 total 10.00 last 117.05) - /unlink 2217 (time 1788641383.40 total 20.04 last 104.25) - /unlink 3307 (time 1788641393.40 total 30.04 last 108.93) - /unlink 4373 (time 1788641403.41 total 40.05 last 106.56) total: 4500 /unlink in 41.08 seconds: 109.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 634 (time 1788641419.61 total 10.02 last 63.30) - open/close 1317 (time 1788641429.63 total 20.03 last 68.22) - open/close 1924 (time 1788641439.66 total 30.06 last 60.48) - open/close 2511 (time 1788641449.69 total 40.09 last 58.53) - open/close 3139 (time 1788641459.70 total 50.10 last 62.77) - open/close 3705 (time 1788641469.71 total 60.11 last 56.52) - open/close 4248 (time 1788641479.73 total 70.13 last 54.21) total: 4500 open/close in 74.83 seconds: 60.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1207 (time 1788641499.65 total 10.00 last 120.67) - /unlink 2428 (time 1788641509.65 total 20.01 last 122.04) - /unlink 3674 (time 1788641519.65 total 30.01 last 124.58) total: 4500 /unlink in 37.36 seconds: 120.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 640 (time 1788641541.91 total 10.01 last 63.94) - open/close 1315 (time 1788641551.93 total 20.02 last 67.40) - open/close 1986 (time 1788641561.94 total 30.03 last 67.05) - open/close 2664 (time 1788641571.95 total 40.05 last 67.70) - open/close 3382 (time 1788641581.97 total 50.06 last 71.68) - open/close 4016 (time 1788641591.97 total 60.07 last 63.38) total: 4500 open/close in 67.84 seconds: 66.33 ops/second - /unlink 1304 (time 1788641614.58 total 10.00 last 130.34) - /unlink 2506 (time 1788641624.58 total 20.00 last 120.20) - /unlink 3826 (time 1788641634.59 total 30.01 last 131.96) total: 4500 /unlink in 35.52 seconds: 126.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 664 (time 1788641655.10 total 10.01 last 66.35) - open/close 1344 (time 1788641665.11 total 20.01 last 67.96) - open/close 2026 (time 1788641675.11 total 30.02 last 68.18) - open/close 2703 (time 1788641685.12 total 40.03 last 67.62) - open/close 3371 (time 1788641695.13 total 50.03 last 66.75) - open/close 4056 (time 1788641705.15 total 60.05 last 68.39) total: 4500 open/close in 67.07 seconds: 67.09 ops/second - /unlink 1146 (time 1788641726.64 total 10.02 last 114.41) - /unlink 2308 (time 1788641736.64 total 20.02 last 116.17) - /unlink 3604 (time 1788641746.64 total 30.02 last 129.59) total: 4500 /unlink in 37.94 seconds: 118.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 713 (time 1788641770.51 total 10.01 last 71.24) - open/close 1295 (time 1788641780.52 total 20.02 last 58.15) - open/close 1842 (time 1788641790.54 total 30.04 last 54.61) - open/close 2466 (time 1788641800.57 total 40.07 last 62.18) - open/close 3081 (time 1788641810.57 total 50.07 last 61.50) - open/close 3738 (time 1788641820.61 total 60.11 last 65.46) - open/close 4332 (time 1788641830.62 total 70.11 last 59.37) total: 4500 open/close in 73.04 seconds: 61.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1213 (time 1788641848.63 total 10.00 last 121.26) - /unlink 2378 (time 1788641858.65 total 20.02 last 116.29) - /unlink 3483 (time 1788641868.65 total 30.03 last 110.44) total: 4500 /unlink in 39.52 seconds: 113.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 565 (time 1788641894.82 total 10.03 last 56.35) - open/close 1160 (time 1788641904.83 total 20.04 last 59.40) - open/close 1841 (time 1788641914.84 total 30.05 last 68.03) - open/close 2498 (time 1788641924.85 total 40.06 last 65.67) - open/close 3170 (time 1788641934.86 total 50.07 last 67.11) - open/close 3811 (time 1788641944.88 total 60.09 last 64.02) - open/close 4447 (time 1788641954.89 total 70.10 last 63.52) total: 4500 open/close in 71.00 seconds: 63.38 ops/second - /unlink 1111 (time 1788641970.72 total 10.01 last 110.97) - /unlink 2251 (time 1788641980.72 total 20.02 last 113.94) - /unlink 3395 (time 1788641990.73 total 30.02 last 114.38) total: 4500 /unlink in 38.41 seconds: 117.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 654 (time 1788642014.56 total 10.00 last 65.40) - open/close 1306 (time 1788642024.56 total 20.00 last 65.20) - open/close 1923 (time 1788642034.57 total 30.01 last 61.66) - open/close 2561 (time 1788642044.57 total 40.01 last 63.79) - open/close 3232 (time 1788642054.57 total 50.01 last 67.05) - open/close 3856 (time 1788642064.58 total 60.02 last 62.36) total: 4500 open/close in 69.83 seconds: 64.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1217 (time 1788642089.08 total 10.00 last 121.65) - /unlink 2425 (time 1788642099.09 total 20.01 last 120.78) - /unlink 3637 (time 1788642109.09 total 30.01 last 121.10) total: 4500 /unlink in 37.37 seconds: 120.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 643 (time 1788642131.19 total 10.00 last 64.28) - open/close 1277 (time 1788642141.19 total 20.00 last 63.40) - open/close 1969 (time 1788642151.20 total 30.01 last 69.17) - open/close 2636 (time 1788642161.22 total 40.03 last 66.57) - open/close 3309 (time 1788642171.22 total 50.03 last 67.27) - open/close 3921 (time 1788642181.24 total 60.05 last 61.09) total: 4500 open/close in 68.98 seconds: 65.24 ops/second - /unlink 1252 (time 1788642205.22 total 10.00 last 125.20) - /unlink 2504 (time 1788642215.23 total 20.00 last 125.15) - /unlink 3780 (time 1788642225.23 total 30.01 last 127.57) total: 4500 /unlink in 36.26 seconds: 124.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 601 (time 1788642247.24 total 10.00 last 60.07) - open/close 1202 (time 1788642257.24 total 20.00 last 60.10) - open/close 1715 (time 1788642267.26 total 30.02 last 51.23) - open/close 2364 (time 1788642277.27 total 40.04 last 64.79) - open/close 2905 (time 1788642287.28 total 50.04 last 54.09) - open/close 3546 (time 1788642297.28 total 60.04 last 64.05) - open/close 4183 (time 1788642307.30 total 70.06 last 63.58) total: 4500 open/close in 75.40 seconds: 59.69 ops/second - /unlink 1234 (time 1788642328.35 total 10.00 last 123.37) - /unlink 2479 (time 1788642338.36 total 20.02 last 124.32) - /unlink 3592 (time 1788642348.36 total 30.02 last 111.30) total: 4500 /unlink in 36.79 seconds: 122.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 596 (time 1788642369.39 total 10.01 last 59.54) - open/close 1215 (time 1788642379.40 total 20.02 last 61.85) - open/close 1754 (time 1788642389.40 total 30.02 last 53.88) - open/close 2357 (time 1788642399.41 total 40.03 last 60.25) - open/close 3064 (time 1788642409.42 total 50.04 last 70.62) - open/close 3727 (time 1788642419.42 total 60.04 last 66.28) - open/close 4394 (time 1788642429.44 total 70.06 last 66.59) total: 4500 open/close in 71.42 seconds: 63.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1071 (time 1788642445.34 total 10.00 last 107.06) - /unlink 2223 (time 1788642455.35 total 20.01 last 115.15) - /unlink 3398 (time 1788642465.35 total 30.01 last 117.50) - /unlink 4497 (time 1788642475.35 total 40.01 last 109.83) total: 4500 /unlink in 40.03 seconds: 112.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 622 (time 1788642490.73 total 10.00 last 62.19) - open/close 1303 (time 1788642500.74 total 20.01 last 68.04) - open/close 1951 (time 1788642510.75 total 30.02 last 64.73) - open/close 2637 (time 1788642520.76 total 40.03 last 68.54) - open/close 3279 (time 1788642530.76 total 50.03 last 64.20) - open/close 3896 (time 1788642540.77 total 60.04 last 61.65) total: 4500 open/close in 69.16 seconds: 65.06 ops/second - /unlink 1286 (time 1788642564.42 total 10.02 last 128.36) - /unlink 2567 (time 1788642574.42 total 20.02 last 128.09) - /unlink 3802 (time 1788642584.42 total 30.02 last 123.46) total: 4500 /unlink in 36.45 seconds: 123.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 615 (time 1788642605.67 total 10.01 last 61.44) - open/close 1242 (time 1788642615.69 total 20.04 last 62.53) - open/close 1929 (time 1788642625.70 total 30.04 last 68.69) - open/close 2619 (time 1788642635.70 total 40.04 last 68.95) - open/close 3195 (time 1788642645.70 total 50.05 last 57.59) - open/close 3712 (time 1788642655.70 total 60.05 last 51.69) - open/close 4201 (time 1788642665.72 total 70.06 last 48.84) total: 4500 open/close in 75.20 seconds: 59.84 ops/second - /unlink 1167 (time 1788642685.79 total 10.01 last 116.56) - /unlink 2297 (time 1788642695.79 total 20.01 last 112.98) - /unlink 3491 (time 1788642705.80 total 30.02 last 119.38) total: 4500 /unlink in 37.48 seconds: 120.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 599 (time 1788642729.11 total 10.00 last 59.89) - open/close 1260 (time 1788642739.12 total 20.01 last 66.06) - open/close 2002 (time 1788642749.12 total 30.01 last 74.20) - open/close 2673 (time 1788642759.12 total 40.01 last 67.09) - open/close 3330 (time 1788642769.13 total 50.02 last 65.62) - open/close 3908 (time 1788642779.14 total 60.03 last 57.78) - open/close 4432 (time 1788642789.14 total 70.03 last 52.39) total: 4500 open/close in 70.99 seconds: 63.39 ops/second - /unlink 1248 (time 1788642805.23 total 10.01 last 124.71) - /unlink 2406 (time 1788642815.23 total 20.01 last 115.74) - /unlink 3629 (time 1788642825.24 total 30.02 last 122.18) total: 4500 /unlink in 37.70 seconds: 119.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 623 (time 1788642848.35 total 10.02 last 62.19) - open/close 1192 (time 1788642858.35 total 20.02 last 56.88) - open/close 1806 (time 1788642868.36 total 30.03 last 61.37) - open/close 2411 (time 1788642878.37 total 40.04 last 60.42) - open/close 2942 (time 1788642888.38 total 50.05 last 53.06) - open/close 3390 (time 1788642898.42 total 60.08 last 44.63) - open/close 3941 (time 1788642908.43 total 70.10 last 55.04) total: 4500 open/close in 80.05 seconds: 56.22 ops/second - /unlink 1285 (time 1788642933.34 total 10.01 last 128.41) - /unlink 2413 (time 1788642943.35 total 20.01 last 112.74) - /unlink 3597 (time 1788642953.36 total 30.02 last 118.25) total: 4500 /unlink in 38.32 seconds: 117.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 553 (time 1788642976.84 total 10.00 last 55.29) - open/close 1179 (time 1788642986.87 total 20.02 last 62.47) - open/close 1825 (time 1788642996.87 total 30.02 last 64.59) - open/close 2534 (time 1788643006.87 total 40.02 last 70.90) - open/close 3147 (time 1788643016.87 total 50.03 last 61.29) - open/close 3827 (time 1788643026.87 total 60.03 last 68.00) - open/close 4487 (time 1788643036.87 total 70.03 last 65.98) total: 4500 open/close in 70.17 seconds: 64.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1201 (time 1788643051.72 total 10.01 last 120.01) - /unlink 2464 (time 1788643061.72 total 20.01 last 126.30) - /unlink 3735 (time 1788643071.73 total 30.02 last 127.01) total: 4500 /unlink in 37.09 seconds: 121.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 629 (time 1788643093.98 total 10.01 last 62.83) - open/close 1259 (time 1788643104.00 total 20.03 last 62.88) - open/close 1865 (time 1788643114.00 total 30.04 last 60.55) - open/close 2457 (time 1788643124.01 total 40.04 last 59.17) - open/close 3096 (time 1788643134.01 total 50.05 last 63.87) - open/close 3688 (time 1788643144.03 total 60.06 last 59.12) - open/close 4351 (time 1788643154.06 total 70.09 last 66.08) total: 4500 open/close in 72.68 seconds: 61.92 ops/second - /unlink 1149 (time 1788643171.54 total 10.00 last 114.88) - /unlink 2377 (time 1788643181.54 total 20.00 last 122.77) - /unlink 3668 (time 1788643191.54 total 30.00 last 129.10) total: 4500 /unlink in 36.70 seconds: 122.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 619 (time 1788643212.34 total 10.00 last 61.87) - open/close 1249 (time 1788643222.34 total 20.01 last 62.98) - open/close 1871 (time 1788643232.35 total 30.01 last 62.18) - open/close 2579 (time 1788643242.35 total 40.02 last 70.74) - open/close 3274 (time 1788643252.36 total 50.02 last 69.48) - open/close 3926 (time 1788643262.37 total 60.04 last 65.10) total: 4500 open/close in 68.41 seconds: 65.78 ops/second - /unlink 1251 (time 1788643285.45 total 10.00 last 125.06) - /unlink 2474 (time 1788643295.46 total 20.01 last 122.17) - /unlink 3757 (time 1788643305.47 total 30.02 last 128.26) total: 4500 /unlink in 35.59 seconds: 126.42 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95340 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg412-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg412-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg412-server unloading modules via unload_modules_local on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing unload_modules_local oleg412-server: kmemleak disabled modules unloaded.