== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:17:31 (1788643051) Stopping clients: oleg442-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg442-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg442-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg442-server oleg442-server: oleg442-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: kmemleak disabled oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg442-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg442-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg442-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 717 (time 1788643282.97 total 10.01 last 71.63) - open/close 1409 (time 1788643293.01 total 20.05 last 68.93) - open/close 2075 (time 1788643303.03 total 30.07 last 66.44) - open/close 2735 (time 1788643313.05 total 40.09 last 65.92) - open/close 3359 (time 1788643323.05 total 50.09 last 62.39) - open/close 4111 (time 1788643333.06 total 60.10 last 75.14) total: 4500 open/close in 66.00 seconds: 68.18 ops/second - /unlink 1264 (time 1788643353.22 total 10.01 last 126.27) - /unlink 2513 (time 1788643363.22 total 20.01 last 124.87) - /unlink 3800 (time 1788643373.22 total 30.01 last 128.67) total: 4500 /unlink in 35.86 seconds: 125.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 547 (time 1788643394.91 total 10.01 last 54.64) - open/close 1124 (time 1788643404.92 total 20.02 last 57.66) - open/close 1693 (time 1788643414.93 total 30.03 last 56.82) - open/close 2327 (time 1788643424.94 total 40.04 last 63.33) - open/close 3000 (time 1788643434.95 total 50.05 last 67.28) - open/close 3638 (time 1788643444.95 total 60.05 last 63.80) - open/close 4292 (time 1788643454.96 total 70.06 last 65.33) total: 4500 open/close in 72.80 seconds: 61.81 ops/second - /unlink 1289 (time 1788643471.92 total 10.01 last 128.74) - /unlink 2478 (time 1788643481.92 total 20.01 last 118.89) - /unlink 3671 (time 1788643491.92 total 30.01 last 119.30) total: 4500 /unlink in 37.14 seconds: 121.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 520 (time 1788643514.84 total 10.00 last 51.99) - open/close 1002 (time 1788643524.87 total 20.03 last 48.05) - open/close 1519 (time 1788643534.88 total 30.04 last 51.65) - open/close 2073 (time 1788643544.89 total 40.06 last 55.33) - open/close 2803 (time 1788643554.90 total 50.07 last 72.92) - open/close 3478 (time 1788643564.91 total 60.07 last 67.50) - open/close 4210 (time 1788643574.91 total 70.07 last 73.14) total: 4500 open/close in 75.13 seconds: 59.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1201 (time 1788643595.05 total 10.00 last 120.05) - /unlink 2440 (time 1788643605.05 total 20.01 last 123.84) - /unlink 3540 (time 1788643615.06 total 30.01 last 109.99) total: 4500 /unlink in 37.68 seconds: 119.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 660 (time 1788643637.41 total 10.01 last 65.91) - open/close 1352 (time 1788643647.42 total 20.03 last 69.11) - open/close 1988 (time 1788643657.43 total 30.03 last 63.56) - open/close 2618 (time 1788643667.44 total 40.04 last 62.93) - open/close 3301 (time 1788643677.46 total 50.06 last 68.17) - open/close 3945 (time 1788643687.47 total 60.07 last 64.36) total: 4500 open/close in 67.79 seconds: 66.39 ops/second - /unlink 1163 (time 1788643710.55 total 10.00 last 116.27) - /unlink 2414 (time 1788643720.56 total 20.01 last 124.98) - /unlink 3671 (time 1788643730.57 total 30.02 last 125.60) total: 4500 /unlink in 36.53 seconds: 123.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 695 (time 1788643751.15 total 10.00 last 69.49) - open/close 1340 (time 1788643761.16 total 20.01 last 64.45) - open/close 1999 (time 1788643771.16 total 30.01 last 65.89) - open/close 2671 (time 1788643781.16 total 40.01 last 67.19) - open/close 3277 (time 1788643791.17 total 50.02 last 60.57) - open/close 3956 (time 1788643801.19 total 60.03 last 67.80) total: 4500 open/close in 67.56 seconds: 66.61 ops/second - /unlink 1263 (time 1788643823.22 total 10.00 last 126.26) - /unlink 2518 (time 1788643833.23 total 20.01 last 125.39) - /unlink 3763 (time 1788643843.23 total 30.02 last 124.46) total: 4500 /unlink in 36.30 seconds: 123.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 622 (time 1788643864.98 total 10.01 last 62.14) - open/close 1367 (time 1788643874.99 total 20.02 last 74.45) - open/close 2050 (time 1788643884.99 total 30.02 last 68.28) - open/close 2658 (time 1788643894.99 total 40.02 last 60.76) - open/close 3298 (time 1788643905.00 total 50.03 last 63.94) - open/close 3959 (time 1788643915.01 total 60.04 last 66.04) total: 4500 open/close in 67.85 seconds: 66.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1316 (time 1788643937.46 total 10.00 last 131.57) - /unlink 2609 (time 1788643947.46 total 20.00 last 129.29) - /unlink 3887 (time 1788643957.48 total 30.02 last 127.61) total: 4500 /unlink in 34.92 seconds: 128.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 664 (time 1788643977.12 total 10.00 last 66.37) - open/close 1306 (time 1788643987.12 total 20.00 last 64.19) - open/close 1985 (time 1788643997.12 total 30.01 last 67.89) - open/close 2624 (time 1788644007.14 total 40.02 last 63.79) - open/close 3281 (time 1788644017.14 total 50.03 last 65.68) - open/close 4004 (time 1788644027.16 total 60.04 last 72.18) total: 4500 open/close in 69.06 seconds: 65.16 ops/second - /unlink 995 (time 1788644052.00 total 10.00 last 99.48) - /unlink 2176 (time 1788644062.01 total 20.01 last 118.02) - /unlink 3380 (time 1788644072.01 total 30.01 last 120.37) total: 4500 /unlink in 39.52 seconds: 113.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 627 (time 1788644096.27 total 10.00 last 62.68) - open/close 1275 (time 1788644106.29 total 20.02 last 64.71) - open/close 1901 (time 1788644116.29 total 30.02 last 62.58) - open/close 2619 (time 1788644126.30 total 40.03 last 71.75) - open/close 3328 (time 1788644136.30 total 50.04 last 70.85) - open/close 4061 (time 1788644146.31 total 60.04 last 73.28) total: 4500 open/close in 66.66 seconds: 67.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1187 (time 1788644167.30 total 10.00 last 118.68) - /unlink 2398 (time 1788644177.31 total 20.00 last 121.07) - /unlink 3620 (time 1788644187.31 total 30.01 last 122.17) total: 4500 /unlink in 36.72 seconds: 122.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 614 (time 1788644207.96 total 10.00 last 61.38) - open/close 1284 (time 1788644217.97 total 20.01 last 66.99) - open/close 1855 (time 1788644227.98 total 30.02 last 57.02) - open/close 2435 (time 1788644237.99 total 40.03 last 57.93) - open/close 2999 (time 1788644248.00 total 50.04 last 56.34) - open/close 3556 (time 1788644258.01 total 60.05 last 55.63) - open/close 4117 (time 1788644268.02 total 70.06 last 56.04) total: 4500 open/close in 76.67 seconds: 58.69 ops/second - /unlink 1186 (time 1788644290.46 total 10.00 last 118.57) - /unlink 2213 (time 1788644300.46 total 20.00 last 102.70) - /unlink 3252 (time 1788644310.47 total 30.01 last 103.84) - /unlink 4402 (time 1788644320.48 total 40.02 last 114.88) total: 4500 /unlink in 41.17 seconds: 109.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 618 (time 1788644337.22 total 10.01 last 61.75) - open/close 1280 (time 1788644347.22 total 20.01 last 66.16) - open/close 1908 (time 1788644357.25 total 30.04 last 62.63) - open/close 2414 (time 1788644367.27 total 40.06 last 50.48) - open/close 3027 (time 1788644377.27 total 50.07 last 61.29) - open/close 3634 (time 1788644387.30 total 60.09 last 60.53) - open/close 4225 (time 1788644397.34 total 70.13 last 58.89) total: 4500 open/close in 74.25 seconds: 60.61 ops/second - /unlink 1202 (time 1788644417.07 total 10.01 last 120.13) - /unlink 2327 (time 1788644427.07 total 20.01 last 112.50) - /unlink 3460 (time 1788644437.07 total 30.01 last 113.25) total: 4500 /unlink in 38.05 seconds: 118.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 603 (time 1788644460.75 total 10.01 last 60.22) - open/close 1233 (time 1788644470.77 total 20.03 last 62.88) - open/close 1886 (time 1788644480.77 total 30.03 last 65.29) - open/close 2464 (time 1788644490.77 total 40.04 last 57.79) - open/close 3138 (time 1788644500.77 total 50.04 last 67.38) - open/close 3771 (time 1788644510.79 total 60.05 last 63.23) - open/close 4476 (time 1788644520.79 total 70.06 last 70.46) total: 4500 open/close in 70.72 seconds: 63.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1155 (time 1788644536.63 total 10.00 last 115.50) - /unlink 2401 (time 1788644546.64 total 20.00 last 124.57) - /unlink 3576 (time 1788644556.64 total 30.01 last 117.45) total: 4500 /unlink in 37.32 seconds: 120.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 601 (time 1788644579.24 total 10.03 last 59.91) - open/close 1246 (time 1788644589.25 total 20.04 last 64.47) - open/close 1875 (time 1788644599.26 total 30.05 last 62.79) - open/close 2460 (time 1788644609.27 total 40.06 last 58.44) - open/close 3018 (time 1788644619.31 total 50.10 last 55.61) - open/close 3612 (time 1788644629.31 total 60.10 last 59.38) - open/close 4301 (time 1788644639.31 total 70.10 last 68.90) total: 4500 open/close in 73.25 seconds: 61.43 ops/second - /unlink 1326 (time 1788644657.74 total 10.00 last 132.56) - /unlink 2472 (time 1788644667.75 total 20.01 last 114.54) - /unlink 3711 (time 1788644677.76 total 30.01 last 123.82) total: 4500 /unlink in 36.03 seconds: 124.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 677 (time 1788644698.82 total 10.01 last 67.66) - open/close 1433 (time 1788644708.83 total 20.01 last 75.60) - open/close 2117 (time 1788644718.85 total 30.03 last 68.21) - open/close 2713 (time 1788644728.88 total 40.06 last 59.46) - open/close 3296 (time 1788644738.89 total 50.07 last 58.24) - open/close 3979 (time 1788644748.89 total 60.08 last 68.24) total: 4500 open/close in 68.25 seconds: 65.93 ops/second - /unlink 1014 (time 1788644771.73 total 10.02 last 101.23) - /unlink 2098 (time 1788644781.73 total 20.02 last 108.37) - /unlink 3192 (time 1788644791.73 total 30.02 last 109.38) - /unlink 4362 (time 1788644801.74 total 40.03 last 116.94) total: 4500 /unlink in 41.63 seconds: 108.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 638 (time 1788644818.72 total 10.00 last 63.80) - open/close 1230 (time 1788644828.72 total 20.00 last 59.18) - open/close 1885 (time 1788644838.72 total 30.00 last 65.50) - open/close 2545 (time 1788644848.73 total 40.01 last 65.93) - open/close 3204 (time 1788644858.74 total 50.02 last 65.84) - open/close 3875 (time 1788644868.74 total 60.03 last 67.09) total: 4500 open/close in 69.30 seconds: 64.94 ops/second - /unlink 1227 (time 1788644892.85 total 10.01 last 122.63) - /unlink 2450 (time 1788644902.86 total 20.02 last 122.17) - /unlink 3571 (time 1788644912.86 total 30.02 last 112.10) total: 4500 /unlink in 38.54 seconds: 116.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 610 (time 1788644937.19 total 10.00 last 60.97) - open/close 1236 (time 1788644947.20 total 20.02 last 62.51) - open/close 1864 (time 1788644957.22 total 30.04 last 62.70) - open/close 2457 (time 1788644967.23 total 40.05 last 59.24) - open/close 3084 (time 1788644977.23 total 50.05 last 62.68) - open/close 3608 (time 1788644987.25 total 60.07 last 52.30) - open/close 4107 (time 1788644997.26 total 70.08 last 49.86) total: 4500 open/close in 77.57 seconds: 58.01 ops/second - /unlink 1045 (time 1788645020.71 total 10.01 last 104.41) - /unlink 2032 (time 1788645030.71 total 20.01 last 98.66) - /unlink 2996 (time 1788645040.71 total 30.01 last 96.39) - /unlink 3869 (time 1788645050.72 total 40.02 last 87.25) total: 4500 /unlink in 46.37 seconds: 97.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 625 (time 1788645072.05 total 10.01 last 62.44) - open/close 1225 (time 1788645082.05 total 20.01 last 59.99) - open/close 1836 (time 1788645092.05 total 30.01 last 61.09) - open/close 2384 (time 1788645102.06 total 40.02 last 54.75) - open/close 2964 (time 1788645112.07 total 50.03 last 57.96) - open/close 3566 (time 1788645122.07 total 60.03 last 60.19) - open/close 4163 (time 1788645132.10 total 70.06 last 59.52) total: 4500 open/close in 75.21 seconds: 59.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1102 (time 1788645152.50 total 10.01 last 110.14) - /unlink 2289 (time 1788645162.50 total 20.01 last 118.68) - /unlink 3469 (time 1788645172.50 total 30.01 last 118.00) total: 4500 /unlink in 39.42 seconds: 114.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 618 (time 1788645197.04 total 10.00 last 61.78) - open/close 1209 (time 1788645207.04 total 20.01 last 59.08) - open/close 1808 (time 1788645217.06 total 30.02 last 59.81) - open/close 2352 (time 1788645227.06 total 40.02 last 54.39) - open/close 2940 (time 1788645237.06 total 50.02 last 58.80) - open/close 3516 (time 1788645247.07 total 60.03 last 57.57) - open/close 4093 (time 1788645257.07 total 70.03 last 57.70) total: 4500 open/close in 76.67 seconds: 58.69 ops/second - /unlink 1141 (time 1788645279.75 total 10.00 last 114.06) - /unlink 2275 (time 1788645289.75 total 20.01 last 113.37) - /unlink 3357 (time 1788645299.76 total 30.02 last 108.11) total: 4500 /unlink in 39.99 seconds: 112.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 550 (time 1788645324.46 total 10.01 last 54.96) - open/close 1177 (time 1788645334.47 total 20.01 last 62.66) - open/close 1785 (time 1788645344.48 total 30.02 last 60.75) - open/close 2423 (time 1788645354.49 total 40.04 last 63.71) - open/close 3062 (time 1788645364.50 total 50.05 last 63.83) - open/close 3639 (time 1788645374.51 total 60.05 last 57.67) - open/close 4238 (time 1788645384.52 total 70.07 last 59.81) total: 4500 open/close in 74.15 seconds: 60.69 ops/second - /unlink 1199 (time 1788645403.49 total 10.00 last 119.86) - /unlink 2355 (time 1788645413.49 total 20.01 last 115.57) - /unlink 3485 (time 1788645423.49 total 30.01 last 113.00) total: 4500 /unlink in 38.70 seconds: 116.28 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95253 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg442-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg442-server unloading modules via unload_modules_local on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing unload_modules_local oleg442-server: kmemleak disabled modules unloaded.