== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:33:40 (1788158020) Stopping clients: oleg657-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg657-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg657-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg657-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg657-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg657-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg657-server oleg657-server: oleg657-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg657-server' oleg657-server: oleg657-server.virtnet: executing load_modules_local oleg657-server: Loading modules from /home/green/git/lustre-release/lustre oleg657-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg657-server: detected 4 online CPUs by sysfs oleg657-server: MODOPTS_LIBCFS= oleg657-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg657-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg657-server: oleg657-server.virtnet: executing set_default_debug -1 all pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg657-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg657-server: oleg657-server.virtnet: executing set_default_debug -1 all pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg657-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg657-server: oleg657-server.virtnet: executing set_default_debug -1 all pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg657-client.virtnet: -o user_xattr,flock 192.168.206.157@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 755 (time 1788158232.51 total 10.01 last 75.43) - open/close 1544 (time 1788158242.52 total 20.03 last 78.77) - open/close 2224 (time 1788158252.53 total 30.04 last 67.93) - open/close 2941 (time 1788158262.54 total 40.04 last 71.69) - open/close 3671 (time 1788158272.55 total 50.05 last 72.93) - open/close 4440 (time 1788158282.55 total 60.05 last 76.85) total: 4500 open/close in 61.00 seconds: 73.77 ops/second - /unlink 1467 (time 1788158298.29 total 10.01 last 146.53) - /unlink 2945 (time 1788158308.30 total 20.01 last 147.78) - /unlink 4341 (time 1788158318.30 total 30.02 last 139.53) total: 4500 /unlink in 31.34 seconds: 143.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 703 (time 1788158334.01 total 10.00 last 70.29) - open/close 1530 (time 1788158344.02 total 20.01 last 82.63) - open/close 2307 (time 1788158354.02 total 30.02 last 77.65) - open/close 3096 (time 1788158364.03 total 40.03 last 78.81) - open/close 3912 (time 1788158374.03 total 50.03 last 81.60) total: 4500 open/close in 57.57 seconds: 78.17 ops/second - /unlink 1409 (time 1788158395.40 total 10.00 last 140.88) - /unlink 2862 (time 1788158405.40 total 20.00 last 145.26) - /unlink 4281 (time 1788158415.40 total 30.01 last 141.87) total: 4500 /unlink in 31.67 seconds: 142.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 742 (time 1788158431.59 total 10.00 last 74.17) - open/close 1435 (time 1788158441.60 total 20.02 last 69.21) - open/close 2205 (time 1788158451.60 total 30.02 last 77.00) - open/close 3035 (time 1788158461.61 total 40.03 last 82.93) - open/close 3741 (time 1788158471.63 total 50.05 last 70.44) - open/close 4468 (time 1788158481.64 total 60.06 last 72.63) total: 4500 open/close in 60.52 seconds: 74.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1241 (time 1788158497.18 total 10.00 last 124.07) - /unlink 2486 (time 1788158507.18 total 20.00 last 124.48) - /unlink 3747 (time 1788158517.18 total 30.01 last 126.07) total: 4500 /unlink in 36.09 seconds: 124.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 751 (time 1788158538.40 total 10.01 last 75.03) - open/close 1418 (time 1788158548.41 total 20.02 last 66.61) - open/close 2153 (time 1788158558.41 total 30.03 last 73.47) - open/close 2903 (time 1788158568.42 total 40.03 last 74.96) - open/close 3632 (time 1788158578.42 total 50.04 last 72.86) - open/close 4366 (time 1788158588.43 total 60.04 last 73.34) total: 4500 open/close in 61.90 seconds: 72.70 ops/second - /unlink 1349 (time 1788158605.27 total 10.00 last 134.86) - /unlink 2694 (time 1788158615.27 total 20.01 last 134.44) - /unlink 3867 (time 1788158625.27 total 30.01 last 117.25) total: 4500 /unlink in 35.18 seconds: 127.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 659 (time 1788158645.04 total 10.00 last 65.87) - open/close 1370 (time 1788158655.05 total 20.02 last 71.01) - open/close 2037 (time 1788158665.08 total 30.05 last 66.51) - open/close 2724 (time 1788158675.09 total 40.06 last 68.63) - open/close 3335 (time 1788158685.09 total 50.06 last 61.09) - open/close 3985 (time 1788158695.10 total 60.07 last 64.94) total: 4500 open/close in 67.37 seconds: 66.80 ops/second - /unlink 1372 (time 1788158717.16 total 10.00 last 137.13) - /unlink 2563 (time 1788158727.17 total 20.01 last 119.06) - /unlink 3850 (time 1788158737.17 total 30.01 last 128.64) total: 4500 /unlink in 35.31 seconds: 127.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 556 (time 1788158759.08 total 10.01 last 55.53) - open/close 1333 (time 1788158769.09 total 20.02 last 77.62) - open/close 2073 (time 1788158779.09 total 30.03 last 73.98) - open/close 2852 (time 1788158789.10 total 40.03 last 77.85) - open/close 3602 (time 1788158799.10 total 50.03 last 74.99) - open/close 4330 (time 1788158809.11 total 60.04 last 72.74) total: 4500 open/close in 62.13 seconds: 72.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1381 (time 1788158825.30 total 10.00 last 138.05) - /unlink 2622 (time 1788158835.30 total 20.01 last 124.04) - /unlink 3929 (time 1788158845.31 total 30.01 last 130.66) total: 4500 /unlink in 34.21 seconds: 131.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 682 (time 1788158864.14 total 10.02 last 68.08) - open/close 1396 (time 1788158874.15 total 20.02 last 71.37) - open/close 2116 (time 1788158884.15 total 30.03 last 71.96) - open/close 2899 (time 1788158894.17 total 40.04 last 78.17) - open/close 3692 (time 1788158904.18 total 50.05 last 79.22) - open/close 4446 (time 1788158914.18 total 60.06 last 75.37) total: 4500 open/close in 60.82 seconds: 73.99 ops/second - /unlink 1275 (time 1788158930.14 total 10.00 last 127.45) - /unlink 2470 (time 1788158940.15 total 20.01 last 119.41) - /unlink 3861 (time 1788158950.15 total 30.01 last 139.08) total: 4500 /unlink in 34.41 seconds: 130.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 725 (time 1788158968.62 total 10.01 last 72.46) - open/close 1470 (time 1788158978.63 total 20.01 last 74.45) - open/close 2199 (time 1788158988.63 total 30.02 last 72.86) - open/close 2895 (time 1788158998.64 total 40.03 last 69.52) - open/close 3656 (time 1788159008.65 total 50.03 last 76.09) - open/close 4386 (time 1788159018.67 total 60.05 last 72.86) total: 4500 open/close in 61.53 seconds: 73.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1419 (time 1788159034.40 total 10.01 last 141.81) - /unlink 2912 (time 1788159044.41 total 20.01 last 149.24) - /unlink 4338 (time 1788159054.41 total 30.02 last 142.53) total: 4500 /unlink in 31.28 seconds: 143.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 825 (time 1788159070.54 total 10.02 last 82.36) - open/close 1501 (time 1788159080.54 total 20.02 last 67.59) - open/close 2163 (time 1788159090.54 total 30.02 last 66.18) - open/close 2878 (time 1788159100.55 total 40.03 last 71.47) - open/close 3607 (time 1788159110.56 total 50.04 last 72.80) - open/close 4257 (time 1788159120.57 total 60.05 last 64.92) total: 4500 open/close in 63.51 seconds: 70.86 ops/second - /unlink 1399 (time 1788159138.40 total 10.00 last 139.87) - /unlink 2830 (time 1788159148.40 total 20.01 last 143.06) - /unlink 4314 (time 1788159158.41 total 30.01 last 148.33) total: 4500 /unlink in 31.41 seconds: 143.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 657 (time 1788159174.72 total 10.00 last 65.70) - open/close 1412 (time 1788159184.74 total 20.02 last 75.35) - open/close 2089 (time 1788159194.75 total 30.03 last 67.66) - open/close 2835 (time 1788159204.76 total 40.03 last 74.54) - open/close 3563 (time 1788159214.76 total 50.03 last 72.80) - open/close 4228 (time 1788159224.76 total 60.04 last 66.48) total: 4500 open/close in 64.05 seconds: 70.26 ops/second - /unlink 1269 (time 1788159244.03 total 10.00 last 126.84) - /unlink 2651 (time 1788159254.04 total 20.01 last 138.08) - /unlink 4070 (time 1788159264.04 total 30.02 last 141.83) total: 4500 /unlink in 33.26 seconds: 135.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 727 (time 1788159283.21 total 10.00 last 72.68) - open/close 1443 (time 1788159293.23 total 20.02 last 71.47) - open/close 2125 (time 1788159303.24 total 30.03 last 68.13) - open/close 2888 (time 1788159313.24 total 40.03 last 76.29) - open/close 3653 (time 1788159323.25 total 50.04 last 76.43) - open/close 4420 (time 1788159333.26 total 60.04 last 76.67) total: 4500 open/close in 60.92 seconds: 73.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1351 (time 1788159349.26 total 10.00 last 135.04) - /unlink 2708 (time 1788159359.26 total 20.01 last 135.67) - /unlink 4110 (time 1788159369.26 total 30.01 last 140.18) total: 4500 /unlink in 33.16 seconds: 135.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 793 (time 1788159387.12 total 10.01 last 79.21) - open/close 1582 (time 1788159397.12 total 20.02 last 78.86) - open/close 2291 (time 1788159407.12 total 30.02 last 70.88) - open/close 3036 (time 1788159417.12 total 40.02 last 74.49) - open/close 3758 (time 1788159427.14 total 50.03 last 72.10) total: 4500 open/close in 59.04 seconds: 76.22 ops/second - /unlink 1445 (time 1788159450.50 total 10.00 last 144.49) - /unlink 2924 (time 1788159460.50 total 20.00 last 147.87) - /unlink 4293 (time 1788159470.50 total 30.00 last 136.89) total: 4500 /unlink in 31.43 seconds: 143.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 850 (time 1788159485.32 total 10.01 last 84.94) - open/close 1628 (time 1788159495.33 total 20.01 last 77.76) - open/close 2411 (time 1788159505.35 total 30.03 last 78.13) - open/close 3178 (time 1788159515.35 total 40.04 last 76.66) - open/close 4013 (time 1788159525.35 total 50.04 last 83.49) total: 4500 open/close in 56.71 seconds: 79.35 ops/second - /unlink 1533 (time 1788159546.63 total 10.01 last 153.20) - /unlink 2930 (time 1788159556.63 total 20.01 last 139.67) - /unlink 4216 (time 1788159566.63 total 30.01 last 128.58) total: 4500 /unlink in 32.45 seconds: 138.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 695 (time 1788159583.78 total 10.01 last 69.43) - open/close 1374 (time 1788159593.78 total 20.01 last 67.89) - open/close 2156 (time 1788159603.78 total 30.01 last 78.19) - open/close 2933 (time 1788159613.79 total 40.03 last 77.61) - open/close 3680 (time 1788159623.81 total 50.04 last 74.59) - open/close 4353 (time 1788159633.81 total 60.04 last 67.27) total: 4500 open/close in 62.33 seconds: 72.20 ops/second - /unlink 1376 (time 1788159650.84 total 10.01 last 137.50) - /unlink 2509 (time 1788159660.84 total 20.01 last 113.26) - /unlink 3558 (time 1788159670.85 total 30.02 last 104.84) total: 4500 /unlink in 37.95 seconds: 118.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 512 (time 1788159693.76 total 10.01 last 51.16) - open/close 1039 (time 1788159703.76 total 20.02 last 52.66) - open/close 1631 (time 1788159713.77 total 30.03 last 59.14) - open/close 2301 (time 1788159723.78 total 40.03 last 66.94) - open/close 2950 (time 1788159733.79 total 50.04 last 64.85) - open/close 3657 (time 1788159743.80 total 60.05 last 70.60) - open/close 4354 (time 1788159753.81 total 70.06 last 69.67) total: 4500 open/close in 71.77 seconds: 62.70 ops/second - /unlink 1421 (time 1788159769.46 total 10.00 last 142.09) - /unlink 2822 (time 1788159779.46 total 20.00 last 140.05) - /unlink 4273 (time 1788159789.47 total 30.01 last 145.06) total: 4500 /unlink in 31.98 seconds: 140.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 786 (time 1788159805.56 total 10.01 last 78.53) - open/close 1584 (time 1788159815.56 total 20.01 last 79.78) - open/close 2360 (time 1788159825.56 total 30.01 last 77.59) - open/close 3112 (time 1788159835.57 total 40.02 last 75.13) - open/close 3874 (time 1788159845.58 total 50.03 last 76.13) total: 4500 open/close in 57.95 seconds: 77.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1360 (time 1788159867.64 total 10.00 last 135.95) - /unlink 2791 (time 1788159877.65 total 20.01 last 142.97) - /unlink 4132 (time 1788159887.65 total 30.02 last 134.06) total: 4500 /unlink in 33.05 seconds: 136.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 794 (time 1788159905.18 total 10.00 last 79.39) - open/close 1563 (time 1788159915.19 total 20.01 last 76.81) - open/close 2352 (time 1788159925.20 total 30.03 last 78.78) - open/close 3080 (time 1788159935.21 total 40.03 last 72.75) - open/close 3902 (time 1788159945.22 total 50.05 last 82.11) total: 4500 open/close in 57.59 seconds: 78.14 ops/second - /unlink 1470 (time 1788159966.66 total 10.00 last 146.98) - /unlink 2744 (time 1788159976.67 total 20.01 last 127.30) - /unlink 4232 (time 1788159986.67 total 30.01 last 148.75) total: 4500 /unlink in 31.97 seconds: 140.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 766 (time 1788160002.22 total 10.01 last 76.53) - open/close 1538 (time 1788160012.23 total 20.01 last 77.16) - open/close 2276 (time 1788160022.25 total 30.03 last 73.65) - open/close 3027 (time 1788160032.27 total 40.05 last 74.96) - open/close 3778 (time 1788160042.32 total 50.11 last 74.71) - open/close 4470 (time 1788160052.32 total 60.11 last 69.18) total: 4500 open/close in 60.52 seconds: 74.36 ops/second - /unlink 1359 (time 1788160068.00 total 10.00 last 135.88) - /unlink 2540 (time 1788160078.01 total 20.01 last 118.04) - /unlink 3594 (time 1788160088.01 total 30.02 last 105.30) total: 4500 /unlink in 37.76 seconds: 119.19 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95327 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg657-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg657-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg657-server stop mds service on oleg657-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg657-server stop mds service on oleg657-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg657-server unloading modules via unload_modules_local on: 'oleg657-server' oleg657-server: oleg657-server.virtnet: executing unload_modules_local oleg657-server: modules unloaded.