== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:40:40 (1776998440) Stopping clients: oleg411-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg411-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg411-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg411-server oleg411-server: oleg411-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg411-server: detected 4 online CPUs by sysfs oleg411-server: MODOPTS_LIBCFS= oleg411-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg411-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg411-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg411-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 724 (time 1776998625.11 total 10.01 last 72.30) - open/close 1328 (time 1776998635.11 total 20.02 last 60.39) - open/close 2047 (time 1776998645.12 total 30.03 last 71.83) - open/close 2757 (time 1776998655.13 total 40.03 last 70.97) - open/close 3472 (time 1776998665.13 total 50.03 last 71.47) - open/close 4079 (time 1776998675.15 total 60.05 last 60.59) total: 4500 open/close in 66.40 seconds: 67.77 ops/second - /unlink 1396 (time 1776998696.31 total 10.00 last 139.58) - /unlink 2833 (time 1776998706.31 total 20.00 last 143.68) - /unlink 4379 (time 1776998716.33 total 30.02 last 154.37) total: 4500 /unlink in 30.95 seconds: 145.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 815 (time 1776998730.61 total 10.01 last 81.43) - open/close 1550 (time 1776998740.62 total 20.02 last 73.44) - open/close 2310 (time 1776998750.63 total 30.02 last 75.94) - open/close 3010 (time 1776998760.65 total 40.05 last 69.85) - open/close 3749 (time 1776998770.66 total 50.06 last 73.83) - open/close 4348 (time 1776998780.68 total 60.08 last 59.75) total: 4500 open/close in 62.72 seconds: 71.75 ops/second - /unlink 1190 (time 1776998799.52 total 10.01 last 118.92) - /unlink 2311 (time 1776998809.52 total 20.01 last 112.05) - /unlink 3600 (time 1776998819.53 total 30.02 last 128.81) total: 4500 /unlink in 37.04 seconds: 121.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 690 (time 1776998840.97 total 10.00 last 68.99) - open/close 1365 (time 1776998850.98 total 20.01 last 67.42) - open/close 2044 (time 1776998860.99 total 30.01 last 67.90) - open/close 2704 (time 1776998871.00 total 40.03 last 65.90) - open/close 3393 (time 1776998881.02 total 50.05 last 68.75) - open/close 4035 (time 1776998891.03 total 60.06 last 64.16) total: 4500 open/close in 66.74 seconds: 67.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1283 (time 1776998912.53 total 10.01 last 128.17) - /unlink 2527 (time 1776998922.54 total 20.01 last 124.38) - /unlink 3469 (time 1776998932.54 total 30.01 last 94.19) - /unlink 4479 (time 1776998942.54 total 40.01 last 101.00) total: 4500 /unlink in 40.19 seconds: 111.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 591 (time 1776998957.46 total 10.03 last 58.91) - open/close 1286 (time 1776998967.47 total 20.04 last 69.47) - open/close 1888 (time 1776998977.47 total 30.04 last 60.19) - open/close 2518 (time 1776998987.48 total 40.04 last 62.97) - open/close 3232 (time 1776998997.48 total 50.05 last 71.36) - open/close 3923 (time 1776999007.49 total 60.06 last 69.01) total: 4500 open/close in 68.05 seconds: 66.13 ops/second - /unlink 1377 (time 1776999029.37 total 10.00 last 137.68) - /unlink 2682 (time 1776999039.38 total 20.01 last 130.41) - /unlink 3821 (time 1776999049.38 total 30.01 last 113.83) total: 4500 /unlink in 35.68 seconds: 126.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 640 (time 1776999070.15 total 10.00 last 64.00) - open/close 1323 (time 1776999080.16 total 20.00 last 68.29) - open/close 2048 (time 1776999090.18 total 30.02 last 72.35) - open/close 2760 (time 1776999100.19 total 40.03 last 71.12) - open/close 3493 (time 1776999110.19 total 50.04 last 73.26) - open/close 4157 (time 1776999120.21 total 60.05 last 66.33) total: 4500 open/close in 65.42 seconds: 68.79 ops/second - /unlink 1248 (time 1776999140.51 total 10.03 last 124.47) - /unlink 2598 (time 1776999150.52 total 20.04 last 134.80) - /unlink 3759 (time 1776999160.53 total 30.05 last 116.02) total: 4500 /unlink in 35.06 seconds: 128.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 654 (time 1776999180.13 total 10.01 last 65.32) - open/close 1309 (time 1776999190.14 total 20.01 last 65.49) - open/close 2047 (time 1776999200.14 total 30.02 last 73.78) - open/close 2819 (time 1776999210.14 total 40.02 last 77.19) - open/close 3490 (time 1776999220.15 total 50.02 last 67.06) - open/close 4162 (time 1776999230.16 total 60.04 last 67.11) total: 4500 open/close in 64.97 seconds: 69.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1172 (time 1776999249.88 total 10.01 last 117.12) - /unlink 2166 (time 1776999259.88 total 20.01 last 99.40) - /unlink 3470 (time 1776999269.88 total 30.01 last 130.40) total: 4500 /unlink in 38.37 seconds: 117.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 669 (time 1776999293.43 total 10.01 last 66.83) - open/close 1387 (time 1776999303.44 total 20.02 last 71.74) - open/close 2108 (time 1776999313.45 total 30.02 last 72.07) - open/close 2871 (time 1776999323.45 total 40.03 last 76.23) - open/close 3627 (time 1776999333.46 total 50.04 last 75.57) - open/close 4334 (time 1776999343.46 total 60.04 last 70.67) total: 4500 open/close in 62.93 seconds: 71.51 ops/second - /unlink 1324 (time 1776999361.11 total 10.01 last 132.31) - /unlink 2699 (time 1776999371.12 total 20.01 last 137.46) - /unlink 3953 (time 1776999381.13 total 30.02 last 125.28) total: 4500 /unlink in 34.35 seconds: 131.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 721 (time 1776999399.88 total 10.00 last 72.09) - open/close 1442 (time 1776999409.89 total 20.01 last 72.05) - open/close 2231 (time 1776999419.90 total 30.02 last 78.82) - open/close 2863 (time 1776999429.90 total 40.02 last 63.20) - open/close 3602 (time 1776999439.91 total 50.03 last 73.80) - open/close 4359 (time 1776999449.92 total 60.05 last 75.58) total: 4500 open/close in 62.15 seconds: 72.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1324 (time 1776999466.87 total 10.00 last 132.34) - /unlink 2554 (time 1776999476.88 total 20.01 last 122.93) - /unlink 3915 (time 1776999486.88 total 30.01 last 136.05) total: 4500 /unlink in 34.53 seconds: 130.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 719 (time 1776999506.13 total 10.00 last 71.90) - open/close 1427 (time 1776999516.13 total 20.00 last 70.79) - open/close 2189 (time 1776999526.13 total 30.01 last 76.17) - open/close 2893 (time 1776999536.14 total 40.01 last 70.39) - open/close 3573 (time 1776999546.15 total 50.02 last 67.93) - open/close 4318 (time 1776999556.15 total 60.03 last 74.44) total: 4500 open/close in 62.43 seconds: 72.08 ops/second - /unlink 1199 (time 1776999572.84 total 10.00 last 119.88) - /unlink 2341 (time 1776999582.85 total 20.01 last 114.07) - /unlink 3608 (time 1776999592.88 total 30.04 last 126.40) total: 4500 /unlink in 36.72 seconds: 122.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 641 (time 1776999615.16 total 10.02 last 63.94) - open/close 1306 (time 1776999625.16 total 20.03 last 66.49) - open/close 2041 (time 1776999635.16 total 30.03 last 73.49) - open/close 2695 (time 1776999645.18 total 40.04 last 65.30) - open/close 3398 (time 1776999655.18 total 50.04 last 70.29) - open/close 4070 (time 1776999665.19 total 60.06 last 67.10) total: 4500 open/close in 66.04 seconds: 68.14 ops/second - /unlink 1280 (time 1776999685.17 total 10.00 last 127.99) - /unlink 2643 (time 1776999695.17 total 20.00 last 136.26) - /unlink 3975 (time 1776999705.18 total 30.01 last 133.11) total: 4500 /unlink in 34.01 seconds: 132.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 709 (time 1776999723.93 total 10.00 last 70.87) - open/close 1421 (time 1776999733.95 total 20.02 last 71.08) - open/close 2122 (time 1776999743.96 total 30.03 last 70.05) - open/close 2820 (time 1776999753.96 total 40.03 last 69.75) - open/close 3515 (time 1776999763.97 total 50.04 last 69.44) - open/close 4303 (time 1776999773.98 total 60.05 last 78.76) total: 4500 open/close in 62.65 seconds: 71.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1308 (time 1776999790.84 total 10.00 last 130.75) - /unlink 2414 (time 1776999800.84 total 20.01 last 110.55) - /unlink 3857 (time 1776999810.84 total 30.01 last 144.29) total: 4500 /unlink in 34.22 seconds: 131.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 768 (time 1776999828.85 total 10.01 last 76.76) - open/close 1453 (time 1776999838.87 total 20.02 last 68.40) - open/close 2191 (time 1776999848.88 total 30.03 last 73.73) - open/close 2955 (time 1776999858.88 total 40.04 last 76.33) - open/close 3739 (time 1776999868.89 total 50.04 last 78.35) - open/close 4482 (time 1776999878.89 total 60.05 last 74.28) total: 4500 open/close in 60.28 seconds: 74.66 ops/second - /unlink 1369 (time 1776999893.22 total 10.01 last 136.82) - /unlink 2702 (time 1776999903.23 total 20.01 last 133.19) - /unlink 4006 (time 1776999913.23 total 30.01 last 130.40) total: 4500 /unlink in 33.91 seconds: 132.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 749 (time 1776999932.30 total 10.00 last 74.87) - open/close 1479 (time 1776999942.32 total 20.03 last 72.84) - open/close 2195 (time 1776999952.33 total 30.03 last 71.54) - open/close 2966 (time 1776999962.34 total 40.04 last 77.04) - open/close 3646 (time 1776999972.34 total 50.04 last 67.99) - open/close 4406 (time 1776999982.34 total 60.04 last 75.99) total: 4500 open/close in 61.55 seconds: 73.11 ops/second - /unlink 1451 (time 1776999997.99 total 10.00 last 145.07) - /unlink 2769 (time 1777000007.99 total 20.00 last 131.77) - /unlink 4181 (time 1777000017.99 total 30.01 last 141.19) total: 4500 /unlink in 32.27 seconds: 139.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 764 (time 1777000034.68 total 10.00 last 76.36) - open/close 1519 (time 1777000044.68 total 20.01 last 75.50) - open/close 2241 (time 1777000054.68 total 30.01 last 72.19) - open/close 2993 (time 1777000064.69 total 40.01 last 75.16) - open/close 3774 (time 1777000074.70 total 50.02 last 78.01) total: 4500 open/close in 59.48 seconds: 75.65 ops/second - /unlink 1360 (time 1777000097.98 total 10.00 last 135.94) - /unlink 2821 (time 1777000107.99 total 20.01 last 145.99) - /unlink 4394 (time 1777000118.00 total 30.02 last 157.20) total: 4500 /unlink in 30.74 seconds: 146.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 857 (time 1777000131.74 total 10.01 last 85.58) - open/close 1701 (time 1777000141.75 total 20.02 last 84.35) - open/close 2510 (time 1777000151.75 total 30.02 last 80.88) - open/close 3308 (time 1777000161.75 total 40.02 last 79.78) - open/close 3999 (time 1777000171.76 total 50.03 last 69.05) total: 4500 open/close in 57.52 seconds: 78.23 ops/second - /unlink 1350 (time 1777000192.80 total 10.00 last 134.95) - /unlink 2686 (time 1777000202.81 total 20.01 last 133.55) - /unlink 4065 (time 1777000212.81 total 30.01 last 137.88) total: 4500 /unlink in 33.31 seconds: 135.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 851 (time 1777000229.82 total 10.00 last 85.06) - open/close 1567 (time 1777000239.83 total 20.02 last 71.50) - open/close 2287 (time 1777000249.84 total 30.03 last 71.95) - open/close 3010 (time 1777000259.85 total 40.03 last 72.25) - open/close 3763 (time 1777000269.86 total 50.04 last 75.22) total: 4500 open/close in 59.90 seconds: 75.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1337 (time 1777000293.21 total 10.00 last 133.69) - /unlink 2907 (time 1777000303.21 total 20.00 last 156.99) total: 4500 /unlink in 29.68 seconds: 151.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 819 (time 1777000326.82 total 10.01 last 81.81) - open/close 1655 (time 1777000336.82 total 20.01 last 83.57) - open/close 2434 (time 1777000346.83 total 30.02 last 77.85) - open/close 3195 (time 1777000356.83 total 40.03 last 76.06) - open/close 3921 (time 1777000366.84 total 50.04 last 72.52) total: 4500 open/close in 58.10 seconds: 77.45 ops/second - /unlink 1464 (time 1777000389.17 total 10.01 last 146.30) - /unlink 2970 (time 1777000399.17 total 20.01 last 150.55) total: 4500 /unlink in 29.93 seconds: 150.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 737 (time 1777000422.41 total 10.01 last 73.66) - open/close 1534 (time 1777000432.41 total 20.01 last 79.70) - open/close 2324 (time 1777000442.42 total 30.01 last 78.93) - open/close 3208 (time 1777000452.42 total 40.02 last 88.39) - open/close 4035 (time 1777000462.42 total 50.02 last 82.67) total: 4500 open/close in 55.10 seconds: 81.67 ops/second - /unlink 1649 (time 1777000481.16 total 10.00 last 164.88) - /unlink 3166 (time 1777000491.17 total 20.01 last 151.54) total: 4500 /unlink in 29.15 seconds: 154.39 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11189: 94062 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg411-server unloading modules via unload_modules_local on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing unload_modules_local oleg411-server: modules unloaded.