== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:01:59 (1787868119) Stopping clients: oleg332-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg332-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg332-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg332-server oleg332-server: oleg332-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg332-server: detected 4 online CPUs by sysfs oleg332-server: MODOPTS_LIBCFS= oleg332-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg332-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg332-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg332-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 698 (time 1787868332.27 total 10.00 last 69.80) - open/close 1451 (time 1787868342.30 total 20.03 last 75.09) - open/close 2168 (time 1787868352.30 total 30.03 last 71.66) - open/close 2936 (time 1787868362.31 total 40.04 last 76.73) - open/close 3641 (time 1787868372.32 total 50.05 last 70.44) - open/close 4315 (time 1787868382.33 total 60.06 last 67.32) total: 4500 open/close in 63.19 seconds: 71.21 ops/second - /unlink 1252 (time 1787868400.20 total 10.01 last 125.12) - /unlink 2507 (time 1787868410.21 total 20.01 last 125.45) - /unlink 3922 (time 1787868420.21 total 30.01 last 141.45) total: 4500 /unlink in 34.45 seconds: 130.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 708 (time 1787868439.32 total 10.00 last 70.77) - open/close 1425 (time 1787868449.33 total 20.01 last 71.66) - open/close 2086 (time 1787868459.34 total 30.02 last 66.04) - open/close 2752 (time 1787868469.34 total 40.02 last 66.59) - open/close 3413 (time 1787868479.35 total 50.03 last 66.04) - open/close 4077 (time 1787868489.35 total 60.03 last 66.37) total: 4500 open/close in 66.31 seconds: 67.86 ops/second - /unlink 1345 (time 1787868509.53 total 10.00 last 134.48) - /unlink 2519 (time 1787868519.54 total 20.00 last 117.38) - /unlink 3790 (time 1787868529.54 total 30.01 last 127.04) total: 4500 /unlink in 36.74 seconds: 122.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 582 (time 1787868552.09 total 10.01 last 58.16) - open/close 1237 (time 1787868562.09 total 20.01 last 65.48) - open/close 1920 (time 1787868572.10 total 30.02 last 68.25) - open/close 2615 (time 1787868582.10 total 40.02 last 69.49) - open/close 3303 (time 1787868592.11 total 50.03 last 68.73) - open/close 4006 (time 1787868602.13 total 60.05 last 70.18) total: 4500 open/close in 67.33 seconds: 66.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1347 (time 1787868623.72 total 10.00 last 134.69) - /unlink 2627 (time 1787868633.72 total 20.00 last 127.98) - /unlink 4002 (time 1787868643.73 total 30.01 last 137.33) total: 4500 /unlink in 33.96 seconds: 132.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 676 (time 1787868662.44 total 10.00 last 67.57) - open/close 1385 (time 1787868672.45 total 20.02 last 70.79) - open/close 2106 (time 1787868682.47 total 30.04 last 71.93) - open/close 2860 (time 1787868692.48 total 40.04 last 75.40) - open/close 3413 (time 1787868702.51 total 50.08 last 55.12) - open/close 4012 (time 1787868712.52 total 60.09 last 59.84) total: 4500 open/close in 67.36 seconds: 66.81 ops/second - /unlink 1286 (time 1787868734.53 total 10.00 last 128.60) - /unlink 2476 (time 1787868744.53 total 20.00 last 118.97) - /unlink 3815 (time 1787868754.53 total 30.00 last 133.90) total: 4500 /unlink in 35.42 seconds: 127.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 686 (time 1787868774.11 total 10.00 last 68.58) - open/close 1408 (time 1787868784.12 total 20.02 last 72.11) - open/close 2126 (time 1787868794.13 total 30.02 last 71.75) - open/close 2893 (time 1787868804.14 total 40.03 last 76.65) - open/close 3569 (time 1787868814.15 total 50.04 last 67.50) - open/close 4308 (time 1787868824.16 total 60.05 last 73.84) total: 4500 open/close in 62.61 seconds: 71.87 ops/second - /unlink 1330 (time 1787868840.92 total 10.00 last 132.95) - /unlink 2535 (time 1787868850.93 total 20.01 last 120.39) - /unlink 3813 (time 1787868860.93 total 30.01 last 127.77) total: 4500 /unlink in 34.93 seconds: 128.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 697 (time 1787868880.88 total 10.01 last 69.65) - open/close 1392 (time 1787868890.89 total 20.01 last 69.47) - open/close 2076 (time 1787868900.89 total 30.02 last 68.37) - open/close 2803 (time 1787868910.90 total 40.02 last 72.65) - open/close 3513 (time 1787868920.91 total 50.04 last 70.89) - open/close 4244 (time 1787868930.93 total 60.05 last 73.00) total: 4500 open/close in 63.46 seconds: 70.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1352 (time 1787868948.96 total 10.00 last 135.15) - /unlink 2595 (time 1787868958.97 total 20.01 last 124.24) - /unlink 3944 (time 1787868968.97 total 30.01 last 134.83) total: 4500 /unlink in 33.99 seconds: 132.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 666 (time 1787868987.82 total 10.01 last 66.57) - open/close 1341 (time 1787868997.83 total 20.01 last 67.44) - open/close 1951 (time 1787869007.83 total 30.02 last 60.99) - open/close 2573 (time 1787869017.85 total 40.03 last 62.09) - open/close 3260 (time 1787869027.86 total 50.04 last 68.65) - open/close 3952 (time 1787869037.87 total 60.05 last 69.13) total: 4500 open/close in 68.22 seconds: 65.96 ops/second - /unlink 1413 (time 1787869060.23 total 10.00 last 141.25) - /unlink 2852 (time 1787869070.23 total 20.01 last 143.83) - /unlink 4287 (time 1787869080.24 total 30.01 last 143.41) total: 4500 /unlink in 31.60 seconds: 142.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 715 (time 1787869095.71 total 10.00 last 71.48) - open/close 1402 (time 1787869105.71 total 20.00 last 68.69) - open/close 2117 (time 1787869115.71 total 30.01 last 71.49) - open/close 2801 (time 1787869125.73 total 40.03 last 68.26) - open/close 3519 (time 1787869135.74 total 50.03 last 71.75) - open/close 4271 (time 1787869145.74 total 60.04 last 75.16) total: 4500 open/close in 62.98 seconds: 71.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1257 (time 1787869163.09 total 10.00 last 125.68) - /unlink 2436 (time 1787869173.09 total 20.00 last 117.87) - /unlink 3678 (time 1787869183.09 total 30.00 last 124.19) total: 4500 /unlink in 36.67 seconds: 122.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 639 (time 1787869204.40 total 10.02 last 63.75) - open/close 1356 (time 1787869214.41 total 20.04 last 71.60) - open/close 2081 (time 1787869224.42 total 30.04 last 72.49) - open/close 2762 (time 1787869234.42 total 40.04 last 68.07) - open/close 3435 (time 1787869244.42 total 50.05 last 67.28) - open/close 4177 (time 1787869254.43 total 60.05 last 74.15) total: 4500 open/close in 64.93 seconds: 69.31 ops/second - /unlink 1375 (time 1787869274.29 total 10.00 last 137.44) - /unlink 2637 (time 1787869284.30 total 20.02 last 126.06) - /unlink 3793 (time 1787869294.30 total 30.02 last 115.57) total: 4500 /unlink in 35.95 seconds: 125.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 664 (time 1787869315.28 total 10.00 last 66.39) - open/close 1354 (time 1787869325.29 total 20.01 last 68.95) - open/close 2080 (time 1787869335.30 total 30.02 last 72.54) - open/close 2737 (time 1787869345.30 total 40.02 last 65.70) - open/close 3393 (time 1787869355.31 total 50.02 last 65.56) - open/close 4086 (time 1787869365.31 total 60.03 last 69.25) total: 4500 open/close in 65.57 seconds: 68.63 ops/second - /unlink 1405 (time 1787869385.05 total 10.00 last 140.46) - /unlink 2657 (time 1787869395.06 total 20.01 last 125.10) - /unlink 3903 (time 1787869405.06 total 30.01 last 124.57) total: 4500 /unlink in 34.67 seconds: 129.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 627 (time 1787869424.84 total 10.01 last 62.64) - open/close 1270 (time 1787869434.86 total 20.02 last 64.22) - open/close 1919 (time 1787869444.87 total 30.04 last 64.81) - open/close 2671 (time 1787869454.87 total 40.04 last 75.20) - open/close 3376 (time 1787869464.87 total 50.04 last 70.50) - open/close 4086 (time 1787869474.89 total 60.06 last 70.86) total: 4500 open/close in 65.85 seconds: 68.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1383 (time 1787869494.91 total 10.00 last 138.24) - /unlink 2758 (time 1787869504.92 total 20.01 last 137.37) - /unlink 4192 (time 1787869514.92 total 30.02 last 143.36) total: 4500 /unlink in 32.38 seconds: 138.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 690 (time 1787869531.74 total 10.00 last 68.98) - open/close 1403 (time 1787869541.74 total 20.01 last 71.25) - open/close 2216 (time 1787869551.75 total 30.02 last 81.24) - open/close 2893 (time 1787869561.76 total 40.02 last 67.66) - open/close 3578 (time 1787869571.76 total 50.02 last 68.50) - open/close 4264 (time 1787869581.76 total 60.03 last 68.57) total: 4500 open/close in 63.62 seconds: 70.73 ops/second - /unlink 1276 (time 1787869600.42 total 10.01 last 127.43) - /unlink 2620 (time 1787869610.42 total 20.01 last 134.37) - /unlink 3953 (time 1787869620.42 total 30.02 last 133.27) total: 4500 /unlink in 34.48 seconds: 130.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 697 (time 1787869639.16 total 10.01 last 69.60) - open/close 1414 (time 1787869649.16 total 20.02 last 71.66) - open/close 2134 (time 1787869659.17 total 30.03 last 71.94) - open/close 2809 (time 1787869669.18 total 40.03 last 67.47) - open/close 3523 (time 1787869679.19 total 50.04 last 71.32) - open/close 4220 (time 1787869689.19 total 60.05 last 69.65) total: 4500 open/close in 63.96 seconds: 70.35 ops/second - /unlink 1421 (time 1787869707.25 total 10.00 last 142.08) - /unlink 2671 (time 1787869717.26 total 20.01 last 124.91) - /unlink 3951 (time 1787869727.27 total 30.01 last 127.94) total: 4500 /unlink in 34.70 seconds: 129.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 667 (time 1787869746.45 total 10.00 last 66.67) - open/close 1373 (time 1787869756.45 total 20.01 last 70.58) - open/close 2028 (time 1787869766.46 total 30.02 last 65.45) - open/close 2603 (time 1787869776.47 total 40.02 last 57.46) - open/close 3321 (time 1787869786.49 total 50.04 last 71.67) - open/close 4076 (time 1787869796.49 total 60.04 last 75.48) total: 4500 open/close in 66.92 seconds: 67.25 ops/second - /unlink 1357 (time 1787869817.78 total 10.01 last 135.60) - /unlink 2700 (time 1787869827.79 total 20.01 last 134.26) - /unlink 4053 (time 1787869837.79 total 30.01 last 135.27) total: 4500 /unlink in 33.49 seconds: 134.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 729 (time 1787869856.37 total 10.00 last 72.87) - open/close 1369 (time 1787869866.37 total 20.01 last 63.98) - open/close 1885 (time 1787869876.39 total 30.03 last 51.51) - open/close 2381 (time 1787869886.41 total 40.04 last 49.51) - open/close 3057 (time 1787869896.41 total 50.05 last 67.58) - open/close 3785 (time 1787869906.41 total 60.05 last 72.79) total: 4500 open/close in 69.74 seconds: 64.53 ops/second - /unlink 1262 (time 1787869930.77 total 10.00 last 126.17) - /unlink 2348 (time 1787869940.77 total 20.00 last 108.60) - /unlink 3414 (time 1787869950.78 total 30.01 last 106.55) total: 4500 /unlink in 39.29 seconds: 114.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 773 (time 1787869974.60 total 10.00 last 77.27) - open/close 1435 (time 1787869984.63 total 20.03 last 66.06) - open/close 2130 (time 1787869994.64 total 30.04 last 69.42) - open/close 2872 (time 1787870004.64 total 40.04 last 74.17) - open/close 3549 (time 1787870014.64 total 50.04 last 67.70) - open/close 4220 (time 1787870024.64 total 60.04 last 67.08) total: 4500 open/close in 64.08 seconds: 70.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1352 (time 1787870042.90 total 10.02 last 134.97) - /unlink 2638 (time 1787870052.90 total 20.02 last 128.54) - /unlink 3924 (time 1787870062.90 total 30.02 last 128.58) total: 4500 /unlink in 34.08 seconds: 132.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 701 (time 1787870081.85 total 10.00 last 70.08) - open/close 1313 (time 1787870091.87 total 20.02 last 61.07) - open/close 1846 (time 1787870101.87 total 30.03 last 53.29) - open/close 2537 (time 1787870111.88 total 40.03 last 69.05) - open/close 3164 (time 1787870121.89 total 50.05 last 62.59) - open/close 3816 (time 1787870131.90 total 60.06 last 65.14) - open/close 4392 (time 1787870141.91 total 70.07 last 57.55) total: 4500 open/close in 71.54 seconds: 62.90 ops/second - /unlink 1296 (time 1787870157.31 total 10.01 last 129.52) - /unlink 2550 (time 1787870167.31 total 20.01 last 125.40) - /unlink 3824 (time 1787870177.32 total 30.01 last 127.35) total: 4500 /unlink in 35.71 seconds: 126.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 619 (time 1787870197.63 total 10.01 last 61.83) - open/close 1250 (time 1787870207.64 total 20.02 last 63.02) - open/close 1946 (time 1787870217.67 total 30.05 last 69.43) - open/close 2602 (time 1787870227.67 total 40.05 last 65.57) - open/close 3225 (time 1787870237.69 total 50.07 last 62.22) - open/close 3869 (time 1787870247.69 total 60.07 last 64.36) total: 4500 open/close in 69.80 seconds: 64.47 ops/second - /unlink 1166 (time 1787870272.42 total 10.01 last 116.52) - /unlink 2427 (time 1787870282.43 total 20.02 last 125.98) - /unlink 3622 (time 1787870292.44 total 30.03 last 119.34) total: 4500 /unlink in 38.01 seconds: 118.38 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11375: 95402 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg332-server unloading modules via unload_modules_local on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local oleg332-server: modules unloaded.