== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:37:35 (1786030655) Stopping clients: oleg218-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg218-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg218-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg218-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg218-server oleg218-server: oleg218-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing load_modules_local oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg218-server: Loading modules from /home/green/git/lustre-release/lustre oleg218-server: detected 4 online CPUs by sysfs oleg218-server: MODOPTS_LIBCFS= oleg218-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg218-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg218-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg218-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 599 (time 1786030886.31 total 10.01 last 59.82) - open/close 1194 (time 1786030896.31 total 20.02 last 59.47) - open/close 1919 (time 1786030906.33 total 30.03 last 72.41) - open/close 2695 (time 1786030916.33 total 40.04 last 77.55) - open/close 3234 (time 1786030926.45 total 50.15 last 53.29) - open/close 3593 (time 1786030936.47 total 60.17 last 35.82) - open/close 4109 (time 1786030946.48 total 70.18 last 51.56) total: 4500 open/close in 76.55 seconds: 58.78 ops/second - /unlink 1064 (time 1786030968.36 total 10.00 last 106.35) - /unlink 2044 (time 1786030978.38 total 20.02 last 97.88) - /unlink 2906 (time 1786030988.38 total 30.02 last 86.17) - /unlink 4240 (time 1786030998.38 total 40.02 last 133.36) total: 4500 /unlink in 42.65 seconds: 105.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 699 (time 1786031016.86 total 10.01 last 69.81) - open/close 1359 (time 1786031026.87 total 20.03 last 65.90) - open/close 1946 (time 1786031036.88 total 30.04 last 58.65) - open/close 2616 (time 1786031046.89 total 40.04 last 66.95) - open/close 3253 (time 1786031056.89 total 50.05 last 63.66) - open/close 3939 (time 1786031066.90 total 60.06 last 68.56) - open/close 4426 (time 1786031076.92 total 70.07 last 48.62) total: 4500 open/close in 71.50 seconds: 62.93 ops/second - /unlink 856 (time 1786031095.67 total 10.02 last 85.43) - /unlink 1834 (time 1786031105.69 total 20.04 last 97.60) - /unlink 2777 (time 1786031115.71 total 30.06 last 94.10) - /unlink 3703 (time 1786031125.72 total 40.07 last 92.54) total: 4500 /unlink in 46.10 seconds: 97.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 579 (time 1786031147.18 total 10.03 last 57.71) - open/close 1204 (time 1786031157.19 total 20.04 last 62.47) - open/close 1908 (time 1786031167.19 total 30.05 last 70.35) - open/close 2595 (time 1786031177.20 total 40.05 last 68.68) - open/close 3166 (time 1786031187.20 total 50.05 last 57.10) - open/close 3705 (time 1786031197.20 total 60.05 last 53.90) - open/close 4457 (time 1786031207.20 total 70.05 last 75.19) total: 4500 open/close in 70.59 seconds: 63.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1247 (time 1786031223.03 total 10.00 last 124.68) - /unlink 2376 (time 1786031233.03 total 20.00 last 112.89) - /unlink 3251 (time 1786031243.04 total 30.02 last 87.36) - /unlink 4311 (time 1786031253.05 total 40.02 last 105.98) total: 4500 /unlink in 41.59 seconds: 108.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 576 (time 1786031269.84 total 10.01 last 57.54) - open/close 1107 (time 1786031279.86 total 20.02 last 53.03) - open/close 1682 (time 1786031289.86 total 30.02 last 57.50) - open/close 2322 (time 1786031299.86 total 40.03 last 63.99) - open/close 3133 (time 1786031309.88 total 50.04 last 80.95) - open/close 3760 (time 1786031319.88 total 60.05 last 62.69) - open/close 4239 (time 1786031329.90 total 70.06 last 47.82) total: 4500 open/close in 75.25 seconds: 59.80 ops/second - /unlink 1154 (time 1786031351.09 total 10.01 last 115.25) - /unlink 2167 (time 1786031361.10 total 20.02 last 101.25) - /unlink 3268 (time 1786031371.10 total 30.02 last 110.07) total: 4500 /unlink in 39.31 seconds: 114.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 637 (time 1786031395.46 total 10.04 last 63.48) - open/close 1330 (time 1786031405.46 total 20.04 last 69.26) - open/close 1998 (time 1786031415.47 total 30.05 last 66.77) - open/close 2583 (time 1786031425.48 total 40.06 last 58.41) - open/close 3262 (time 1786031435.49 total 50.07 last 67.84) - open/close 4054 (time 1786031445.50 total 60.08 last 79.14) total: 4500 open/close in 65.09 seconds: 69.13 ops/second - /unlink 1240 (time 1786031465.96 total 10.00 last 124.00) - /unlink 2482 (time 1786031475.97 total 20.01 last 124.12) - /unlink 3800 (time 1786031485.97 total 30.01 last 131.78) total: 4500 /unlink in 35.84 seconds: 125.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 648 (time 1786031507.45 total 10.01 last 64.73) - open/close 1397 (time 1786031517.45 total 20.01 last 74.89) - open/close 2059 (time 1786031527.45 total 30.01 last 66.18) - open/close 2741 (time 1786031537.46 total 40.03 last 68.12) - open/close 3498 (time 1786031547.47 total 50.03 last 75.66) - open/close 4222 (time 1786031557.47 total 60.04 last 72.36) total: 4500 open/close in 63.65 seconds: 70.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1249 (time 1786031576.40 total 10.00 last 124.87) - /unlink 2528 (time 1786031586.41 total 20.00 last 127.88) - /unlink 3737 (time 1786031596.41 total 30.01 last 120.85) total: 4500 /unlink in 35.49 seconds: 126.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 558 (time 1786031617.85 total 10.00 last 55.80) - open/close 1227 (time 1786031627.85 total 20.00 last 66.88) - open/close 1812 (time 1786031637.86 total 30.01 last 58.45) - open/close 2688 (time 1786031647.86 total 40.02 last 87.54) - open/close 3381 (time 1786031657.87 total 50.02 last 69.27) - open/close 4012 (time 1786031667.87 total 60.02 last 63.09) total: 4500 open/close in 67.01 seconds: 67.15 ops/second - /unlink 1166 (time 1786031690.31 total 10.01 last 116.46) - /unlink 2518 (time 1786031700.32 total 20.02 last 135.16) - /unlink 3721 (time 1786031710.32 total 30.02 last 120.25) total: 4500 /unlink in 36.33 seconds: 123.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 736 (time 1786031732.30 total 10.01 last 73.55) - open/close 1419 (time 1786031742.32 total 20.02 last 68.20) - open/close 2223 (time 1786031752.32 total 30.03 last 80.36) - open/close 3007 (time 1786031762.32 total 40.03 last 78.39) - open/close 3852 (time 1786031772.33 total 50.04 last 84.42) total: 4500 open/close in 57.94 seconds: 77.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1425 (time 1786031794.87 total 10.00 last 142.50) - /unlink 2734 (time 1786031804.87 total 20.00 last 130.85) - /unlink 3900 (time 1786031814.88 total 30.01 last 116.52) total: 4500 /unlink in 35.59 seconds: 126.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 701 (time 1786031835.69 total 10.00 last 70.07) - open/close 1407 (time 1786031845.70 total 20.02 last 70.53) - open/close 2153 (time 1786031855.71 total 30.02 last 74.56) - open/close 2923 (time 1786031865.74 total 40.05 last 76.78) - open/close 3673 (time 1786031875.74 total 50.05 last 74.97) - open/close 4364 (time 1786031885.76 total 60.07 last 68.97) total: 4500 open/close in 61.91 seconds: 72.68 ops/second - /unlink 1319 (time 1786031902.65 total 10.00 last 131.85) - /unlink 2398 (time 1786031912.66 total 20.01 last 107.86) - /unlink 3484 (time 1786031922.67 total 30.02 last 108.51) total: 4500 /unlink in 38.64 seconds: 116.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 715 (time 1786031947.20 total 10.01 last 71.44) - open/close 1450 (time 1786031957.20 total 20.01 last 73.48) - open/close 2325 (time 1786031967.22 total 30.02 last 87.39) - open/close 3068 (time 1786031977.22 total 40.03 last 74.28) - open/close 3808 (time 1786031987.23 total 50.04 last 73.91) total: 4500 open/close in 59.19 seconds: 76.02 ops/second - /unlink 1435 (time 1786032011.13 total 10.00 last 143.46) - /unlink 2934 (time 1786032021.14 total 20.01 last 149.86) - /unlink 4254 (time 1786032031.14 total 30.01 last 131.94) total: 4500 /unlink in 31.60 seconds: 142.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 633 (time 1786032048.01 total 10.00 last 63.29) - open/close 1207 (time 1786032058.02 total 20.00 last 57.39) - open/close 1989 (time 1786032068.02 total 30.01 last 78.18) - open/close 2754 (time 1786032078.03 total 40.01 last 76.43) - open/close 3507 (time 1786032088.03 total 50.02 last 75.28) - open/close 4230 (time 1786032098.04 total 60.02 last 72.25) total: 4500 open/close in 63.83 seconds: 70.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1353 (time 1786032116.46 total 10.00 last 135.26) - /unlink 2360 (time 1786032126.47 total 20.01 last 100.67) - /unlink 3439 (time 1786032136.47 total 30.01 last 107.87) total: 4500 /unlink in 39.40 seconds: 114.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 720 (time 1786032161.30 total 10.01 last 71.95) - open/close 1378 (time 1786032171.31 total 20.02 last 65.73) - open/close 2023 (time 1786032181.31 total 30.02 last 64.46) - open/close 2716 (time 1786032191.32 total 40.03 last 69.25) - open/close 3412 (time 1786032201.36 total 50.07 last 69.36) - open/close 3989 (time 1786032211.37 total 60.08 last 57.63) total: 4500 open/close in 67.83 seconds: 66.35 ops/second - /unlink 1244 (time 1786032234.64 total 10.01 last 124.30) - /unlink 2405 (time 1786032244.66 total 20.02 last 115.93) - /unlink 3611 (time 1786032254.66 total 30.03 last 120.53) total: 4500 /unlink in 37.38 seconds: 120.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 667 (time 1786032277.64 total 10.01 last 66.63) - open/close 1337 (time 1786032287.64 total 20.01 last 66.98) - open/close 1989 (time 1786032297.65 total 30.02 last 65.14) - open/close 2649 (time 1786032307.65 total 40.02 last 66.00) - open/close 3360 (time 1786032317.67 total 50.04 last 70.97) - open/close 4055 (time 1786032327.68 total 60.06 last 69.39) total: 4500 open/close in 66.72 seconds: 67.45 ops/second - /unlink 1285 (time 1786032350.46 total 10.00 last 128.48) - /unlink 2493 (time 1786032360.47 total 20.01 last 120.72) - /unlink 3834 (time 1786032370.47 total 30.01 last 134.08) total: 4500 /unlink in 34.31 seconds: 131.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 672 (time 1786032390.07 total 10.00 last 67.18) - open/close 1268 (time 1786032400.08 total 20.01 last 59.54) - open/close 2004 (time 1786032410.10 total 30.03 last 73.44) - open/close 2572 (time 1786032420.12 total 40.05 last 56.69) - open/close 3091 (time 1786032430.14 total 50.07 last 51.82) - open/close 3674 (time 1786032440.16 total 60.09 last 58.19) - open/close 4310 (time 1786032450.17 total 70.10 last 63.54) total: 4500 open/close in 72.82 seconds: 61.80 ops/second - /unlink 1260 (time 1786032467.69 total 10.01 last 125.92) - /unlink 2472 (time 1786032477.69 total 20.01 last 121.19) - /unlink 3705 (time 1786032487.69 total 30.01 last 123.30) total: 4500 /unlink in 36.95 seconds: 121.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 735 (time 1786032510.16 total 10.00 last 73.48) - open/close 1388 (time 1786032520.18 total 20.02 last 65.16) - open/close 2065 (time 1786032530.18 total 30.03 last 67.66) - open/close 2765 (time 1786032540.22 total 40.06 last 69.79) - open/close 3386 (time 1786032550.23 total 50.08 last 62.00) - open/close 3971 (time 1786032560.24 total 60.08 last 58.48) total: 4500 open/close in 66.59 seconds: 67.58 ops/second - /unlink 1412 (time 1786032581.13 total 10.00 last 141.16) - /unlink 2609 (time 1786032591.14 total 20.01 last 119.63) - /unlink 3906 (time 1786032601.15 total 30.02 last 129.55) total: 4500 /unlink in 36.03 seconds: 124.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 642 (time 1786032623.66 total 10.00 last 64.20) - open/close 1302 (time 1786032633.66 total 20.00 last 65.99) - open/close 2052 (time 1786032643.67 total 30.01 last 74.94) - open/close 2810 (time 1786032653.67 total 40.01 last 75.77) - open/close 3519 (time 1786032663.69 total 50.03 last 70.76) - open/close 4265 (time 1786032673.70 total 60.04 last 74.54) total: 4500 open/close in 64.04 seconds: 70.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1035 (time 1786032692.83 total 10.00 last 103.48) - /unlink 2249 (time 1786032702.83 total 20.01 last 121.36) - /unlink 3384 (time 1786032712.84 total 30.01 last 113.41) total: 4500 /unlink in 40.01 seconds: 112.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 713 (time 1786032737.28 total 10.00 last 71.29) - open/close 1396 (time 1786032747.28 total 20.00 last 68.30) - open/close 2106 (time 1786032757.28 total 30.00 last 70.98) - open/close 2839 (time 1786032767.28 total 40.01 last 73.28) - open/close 3461 (time 1786032777.31 total 50.03 last 62.04) - open/close 4273 (time 1786032787.31 total 60.04 last 81.14) total: 4500 open/close in 63.11 seconds: 71.30 ops/second - /unlink 1299 (time 1786032804.95 total 10.00 last 129.85) - /unlink 2568 (time 1786032814.95 total 20.01 last 126.85) - /unlink 3833 (time 1786032824.96 total 30.01 last 126.48) total: 4500 /unlink in 35.94 seconds: 125.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 486 (time 1786032845.86 total 10.03 last 48.47) - open/close 922 (time 1786032855.87 total 20.03 last 43.58) - open/close 1476 (time 1786032865.88 total 30.04 last 55.36) - open/close 2115 (time 1786032875.88 total 40.05 last 63.84) - open/close 2919 (time 1786032885.89 total 50.05 last 80.38) - open/close 3742 (time 1786032895.90 total 60.07 last 82.17) - open/close 4499 (time 1786032905.91 total 70.07 last 75.64) total: 4500 open/close in 70.07 seconds: 64.22 ops/second - /unlink 1291 (time 1786032920.45 total 10.01 last 128.94) - /unlink 2469 (time 1786032930.45 total 20.01 last 117.79) - /unlink 3796 (time 1786032940.46 total 30.01 last 132.67) total: 4500 /unlink in 36.18 seconds: 124.36 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95448 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg218-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg218-server unloading modules via unload_modules_local on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing unload_modules_local oleg218-server: modules unloaded.