Help
RSS
API
Feed
Maltego
Contact
Domain > blog.healthchecks.io
×
More information on this domain is in
AlienVault OTX
Is this malicious?
Yes
No
DNS Resolutions
Date
IP Address
2019-06-13
185.93.1.61
(
ClassC
)
2024-10-04
169.150.221.147
(
ClassC
)
Port 80
HTTP/1.1 301 Moved PermanentlyDate: Fri, 04 Oct 2024 08:54:37 GMTContent-Type: text/htmlContent-Length: 162Connection: keep-aliveServer: BunnyCDN-SIL1-915CDN-PullZone: 54799CDN-Uid: 25e272f5-167d-4855-b99f-3c8d13b1518aCDN-RequestCountryCode: USLocation: https://blog.healthchecks.io/CDN-RequestTime: 0CDN-RequestId: cd83b4643deb75145fb97d011528f158 html>head>title>301 Moved Permanently/title>/head>body>center>h1>301 Moved Permanently/h1>/center>hr>center>nginx/center>/body>/html>
Port 443
HTTP/1.1 200 OKDate: Fri, 04 Oct 2024 08:54:37 GMTContent-Type: text/html; charsetutf-8Content-Length: 148991Connection: keep-aliveVary: Accept-EncodingServer: BunnyCDN-SIL1-915CDN-PullZone: 54799CDN-Uid: 25e272f5-167d-4855-b99f-3c8d13b1518aCDN-RequestCountryCode: USCache-Control: public, max-age0, s-maxage604800Last-Modified: Fri, 04 Oct 2024 06:45:40 GMTx-amz-id-2: 2u/flREi/F+PS9ySYz1mIrQaXlAw32nn+cibh8+jfHlHW7XBWrWskas/cGLscH+1DaGPwademdsx-amz-request-id: ME5AH1TZAFH166K3x-amz-version-id: IisHmXvcHcKTjLcanuBAuAvUEEUiFn7FCDN-ProxyVer: 1.04CDN-RequestPullSuccess: TrueCDN-RequestPullCode: 206CDN-CachedAt: 10/04/2024 07:34:06CDN-EdgeStorageId: 915CDN-Status: 200CDN-RequestTime: 0CDN-RequestId: c3ca977fd0e3d3471388e3e48b6a3750CDN-Cache: HITAccept-Ranges: bytes !DOCTYPE html>html langen-US prefixog: http://ogp.me/ns# fb: http://ogp.me/ns/fb#>head>meta charsetUTF-8>meta nameviewport contentwidthdevice-width, initial-scale1>link relprofile hrefhttps://gmpg.org/xfn/11>title>Healthchecks.io – The Joy of Building a Cron Monitoring Service/title>link relpreload href/wp-content/astra-local-fonts/lato/S6u9w4BMUTPHh6UVSwiPGQ.woff2 asfont typefont/woff2 crossorigin>link relpreload href/wp-content/astra-local-fonts/roboto/KFOlCnqEu92Fr1MmWUlfBBc4.woff2 asfont typefont/woff2 crossorigin>link relpreload href/wp-content/astra-local-fonts/roboto-mono/L0xuDF4xlVMF-BfR8bXMIhJHg45mwgGEFl0_3vq_ROW4.woff2 asfont typefont/woff2 crossorigin>meta namerobots contentmax-image-preview:large>link relalternate typeapplication/rss+xml titleHealthchecks.io » Feed href/feed/>link relalternate typeapplication/rss+xml titleHealthchecks.io » Comments Feed href/comments/feed/>link relstylesheet idastra-theme-css-css href/wp-content/themes/astra/assets/css/minified/style.min.css mediaall>style idastra-theme-css-inline-css>@font-face{font-family:Astra;src:url(/wp-content/themes/astra/assets/fonts/astra.woff) format(woff) , url(/wp-content/themes/astra/assets/fonts/astra.ttf) format(truetype) , url(/wp-content/themes/astra/assets/fonts/astra.svg#astra) format(svg);font-weight:normal;font-style:normal;font-display:fallback}.ast-no-sidebar .entry-content .alignfull{margin-left: calc( -50vw + 50%);margin-right: calc( -50vw + 50%);max-width:100vw;width:100vw}.ast-no-sidebar .entry-content .alignwide{margin-left: calc(-41vw + 50%);margin-right: calc(-41vw + 50%);max-width:unset;width:unset}.ast-no-sidebar .entry-content .alignfull .alignfull,.ast-no-sidebar .entry-content .alignfull .alignwide,.ast-no-sidebar .entry-content .alignwide .alignfull,.ast-no-sidebar .entry-content .alignwide .alignwide,.ast-no-sidebar .entry-content .wp-block-column .alignfull,.ast-no-sidebar .entry-content .wp-block-column .alignwide{width:100%;margin-left:auto;margin-right:auto}.wp-block-gallery,.blocks-gallery-grid{margin:0}.wp-block-separator{max-width:100px}.wp-block-separator.is-style-wide,.wp-block-separator.is-style-dots{max-width:none}.entry-content .has-2-columns .wp-block-column:first-child{padding-right:10px}.entry-content .has-2-columns .wp-block-column:last-child{padding-left:10px}@media (max-width:782px){.entry-content .wp-block-columns .wp-block-column{flex-basis:100%}.entry-content .has-2-columns .wp-block-column:first-child{padding-right:0}.entry-content .has-2-columns .wp-block-column:last-child{padding-left:0}}body .entry-content .wp-block-latest-posts{margin-left:0}body .entry-content .wp-block-latest-posts li{list-style:none}.ast-no-sidebar .ast-container .entry-content .wp-block-latest-posts{margin-left:0}.ast-header-break-point .entry-content .alignwide{margin-left:auto;margin-right:auto}.entry-content .blocks-gallery-item img{margin-bottom:auto}.wp-block-pullquote{border-top:4px solid #555d66;border-bottom:4px solid #555d66;color:#40464d}:root{--ast-container-default-xlg-padding:6.67em;--ast-container-default-lg-padding:5.67em;--ast-container-default-slg-padding:4.34em;--ast-container-default-md-padding:3.34em;--ast-container-default-sm-padding:6.67em;--ast-container-default-xs-padding:2.4em;--ast-container-default-xxs-padding:1.4em}html{font-size:100%}a,.page-title{color:#0091ea}a:hover,a:focus{color:#1e73be}body,button,input,select,textarea,.ast-button,.ast-custom-button{font-family:Roboto,sans-serif;font-weight:400;font-size:16px;font-size:1rem;line-height:1.7}blockquote{color:#000}h1,.entry-content h1,h2,.entry-content h2,h3,.entry-content h3,h4,.entry-content h4,h5,.entry-content h5,h6,.entry-content h6,.site-title,.site-title a{font-family:Lato,sans-serif;font-weight:700}.site-title{font-size:35px;font-size:2.1875rem;display:none}.ast-archive-description .ast-archive-title{font-size:36px;font-size:2.25rem}.site-header .site-description{font-size:15px;font-size:.9375rem;display:none}.entry-title{font-size:30px;font-size:1.875rem}h1,.entry-content h1{font-size:30px;font-size:1.875rem;font-family:Lato,sans-serif;line-height:1.2}h2,.entry-content h2{font-size:24px;font-size:1.5rem;font-family:Lato,sans-serif;line-height:1.2}h3,.entry-content h3{font-size:20px;font-size:1.25rem;font-weight:700;font-family:Lato,sans-serif;line-height:1.2}h4,.entry-content h4{font-size:25px;font-size:1.5625rem;line-height:1.2;font-family:Lato,sans-serif}h5,.entry-content h5{font-size:20px;font-size:1.25rem;line-height:1.2;font-family:Lato,sans-serif}h6,.entry-content h6{font-size:15px;font-size:.9375rem;line-height:1.2;font-family:Roboto Mono,monospace}.ast-single-post .entry-title,.page-title{font-size:30px;font-size:1.875rem}::selection{background-color:#22bc66;color:#000}body,h1,.entry-title a,.entry-content h1,h2,.entry-content h2,h3,.entry-content h3,h4,.entry-content h4,h5,.entry-content h5,h6,.entry-content h6{color:#3a3a3a}.tagcloud a:hover,.tagcloud a:focus,.tagcloud a.current-item{color:#fff;border-color:#0091ea;background-color:#0091ea}input:focus,inputtypetext:focus,inputtypeemail:focus,inputtypeurl:focus,inputtypepassword:focus,inputtypereset:focus,inputtypesearch:focus,textarea:focus{border-color:#0091ea}inputtyperadio:checked,inputtypereset,inputtypecheckbox:checked,inputtypecheckbox:hover:checked,inputtypecheckbox:focus:checked,inputtyperange::-webkit-slider-thumb{border-color:#0091ea;background-color:#0091ea;box-shadow:none}.site-footer a:hover+.post-count,.site-footer a:focus+.post-count{background:#0091ea;border-color:#0091ea}.single .nav-links .nav-previous,.single .nav-links .nav-next{color:#0091ea}.entry-meta,.entry-meta *{line-height:1.45;color:#0091ea}.entry-meta a:hover,.entry-meta a:hover *,.entry-meta a:focus,.entry-meta a:focus *,.page-links>.page-link,.page-links .page-link:hover,.post-navigation a:hover{color:#1e73be}#cat option,.secondary .calendar_wrap thead a,.secondary .calendar_wrap thead a:visited{color:#0091ea}.secondary .calendar_wrap #today,.ast-progress-val span{background:#0091ea}.secondary a:hover+.post-count,.secondary a:focus+.post-count{background:#0091ea;border-color:#0091ea}.calendar_wrap #today>a{color:#fff}.page-links .page-link,.single .post-navigation a{color:#0091ea}.widget-title{font-size:22px;font-size:1.375rem;color:#3a3a3a}.main-header-menu .menu-link,.ast-header-custom-item a{color:#3a3a3a}.main-header-menu .menu-item:hover>.menu-link,.main-header-menu .menu-item:hover>.ast-menu-toggle,.main-header-menu .ast-masthead-custom-menu-items a:hover,.main-header-menu .menu-item.focus>.menu-link,.main-header-menu .menu-item.focus>.ast-menu-toggle,.main-header-menu .current-menu-item>.menu-link,.main-header-menu .current-menu-ancestor>.menu-link,.main-header-menu .current-menu-item>.ast-menu-toggle,.main-header-menu .current-menu-ancestor>.ast-menu-toggle{color:#0091ea}.header-main-layout-3 .ast-main-header-bar-alignment{margin-right:auto}.header-main-layout-2 .site-header-section-left .ast-site-identity{text-align:left}.ast-logo-title-inline .site-logo-img{padding-right:1em}.site-logo-img img{transition:all .2s linear}.ast-header-break-point .ast-mobile-menu-buttons-minimal.menu-toggle{background:transparent;color:#22bc66}.ast-header-break-point .ast-mobile-menu-buttons-outline.menu-toggle{background:transparent;border:1px solid #22bc66;color:#22bc66}.ast-header-break-point .ast-mobile-menu-buttons-fill.menu-toggle{background:#22bc66}.ast-small-footer{color:#020202}.ast-small-footer>.ast-footer-overlay{background-color:#fff}.footer-adv .footer-adv-overlay{border-top-style:solid;border-top-color:#7a7a7a}.wp-block-buttons.aligncenter{justify-content:center}@media (max-width:782px){.entry-content .wp-block-columns .wp-block-column{margin-left:0}}.wp-block-image.aligncenter{margin-left:auto;margin-right:auto}.wp-block-table.aligncenter{margin-left:auto;margin-right:auto}@media (max-width:768px){.ast-separate-container #primary,.ast-separate-container #secondary{padding:1.5em 0}#primary,#secondary{padding:1.5em 0;margin:0}.ast-left-sidebar #content>.ast-container{display:flex;flex-direction:column-reverse;width:100%}.ast-separate-container .ast-article-post,.ast-separate-container .ast-article-single{padding:1.5em 2.14em}.ast-author-box img.avatar{margin:20px 0 0 0}}@media (min-width:769px){.ast-separate-container.ast-right-sidebar #primary,.ast-separate-container.ast-left-sidebar #primary{border:0}.search-no-results.ast-separate-container #primary{margin-bottom:4em}}.menu-toggle,button,.ast-button,.ast-custom-button,.button,input#submit,inputtypebutton,inputtypesubmit,inputtypereset{color:#000;border-color:#22bc66;background-color:#22bc66;border-radius:2px;padding-top:10px;padding-right:40px;padding-bottom:10px;padding-left:40px;font-family:inherit;font-weight:inherit}button:focus,.menu-toggle:hover,button:hover,.ast-button:hover,.ast-custom-button:hover .button:hover,.ast-custom-button:hover,inputtypereset:hover,inputtypereset:focus,input#submit:hover,input#submit:focus,inputtypebutton:hover,inputtypebutton:focus,inputtypesubmit:hover,inputtypesubmit:focus{color:#fff;background-color:#1e73be;border-color:#1e73be}@media (min-width:544px){.ast-container{max-width:100%}}@media (max-width:544px){.ast-separate-container .ast-article-post,.ast-separate-container .ast-article-single,.ast-separate-container .comments-title,.ast-separate-container .ast-archive-description{padding:1.5em 1em}.ast-separate-container #content .ast-container{padding-left:.54em;padding-right:.54em}.ast-separate-container .ast-comment-list li.depth-1{padding:1.5em 1em;margin-bottom:1.5em}.ast-separate-container .ast-comment-list .bypostauthor{padding:.5em}.ast-search-menu-icon.ast-dropdown-active .search-field{width:170px}.site-branding img,.site-header .site-logo-img .custom-logo-link img{max-width:100%}}@media (max-width:768px){.ast-mobile-header-stack .main-header-bar .ast-search-menu-icon{display:inline-block}.ast-header-break-point.ast-header-custom-item-outside .ast-mobile-header-stack .main-header-bar .ast-search-icon{margin:0}.ast-comment-avatar-wrap img{max-width:2.5em}.ast-separate-container .ast-comment-list li.depth-1{padding:1.5em 2.14em}.ast-separate-container .comment-respond{padding:2em 2.14em}.ast-comment-meta{padding:0 1.8888em 1.3333em}}body,.ast-separate-container{background-color:#fff;background-image:none}.ast-no-sidebar.ast-separate-container .entry-content .alignfull{margin-left:-6.67em;margin-right:-6.67em;width:auto}@media (max-width:1200px){.ast-no-sidebar.ast-separate-container .entry-content .alignfull{margin-left:-2.4em;margin-right:-2.4em}}@media (max-width:768px){.ast-no-sidebar.ast-separate-container .entry-content .alignfull{margin-left:-2.14em;margin-right:-2.14em}}@media (max-width:544px){.ast-no-sidebar.ast-separate-container .entry-content .alignfull{margin-left:-1em;margin-right:-1em}}.ast-no-sidebar.ast-separate-container .entry-content .alignwide{margin-left:-20px;margin-right:-20px}.ast-no-sidebar.ast-separate-container .entry-content .wp-block-column .alignfull,.ast-no-sidebar.ast-separate-container .entry-content .wp-block-column .alignwide{margin-left:auto;margin-right:auto;width:100%}@media (max-width:768px){.widget-title{font-size:22px;font-size:1.375rem}body,button,input,select,textarea,.ast-button,.ast-custom-button{font-size:16px;font-size:1rem}#secondary,#secondary button,#secondary input,#secondary select,#secondary textarea{font-size:16px;font-size:1rem}.site-title{display:none}.ast-archive-description .ast-archive-title{font-size:40px}.site-header .site-description{display:none}.entry-title{font-size:30px}h1,.entry-content h1{font-size:45px}h2,.entry-content h2{font-size:32px}h3,.entry-content h3{font-size:26px}h4,.entry-content h4{font-size:22px;font-size:1.375rem}h5,.entry-content h5{font-size:18px;font-size:1.125rem}h6,.entry-content h6{font-size:15px;font-size:.9375rem}.ast-single-post .entry-title,.page-title{font-size:30px}}@media (max-width:544px){.widget-title{font-size:22px;font-size:1.375rem}body,button,input,select,textarea,.ast-button,.ast-custom-button{font-size:16px;font-size:1rem}#secondary,#secondary button,#secondary input,#secondary select,#secondary textarea{font-size:16px;font-size:1rem}.site-title{display:none}.ast-archive-description .ast-archive-title{font-size:40px}.site-header .site-description{display:none}.entry-title{font-size:30px}h1,.entry-content h1{font-size:32px}h2,.entry-content h2{font-size:28px}h3,.entry-content h3{font-size:22px}h4,.entry-content h4{font-size:20px;font-size:1.25rem}h5,.entry-content h5{font-size:20px;font-size:1.25rem}h6,.entry-content h6{font-size:15px;font-size:.9375rem}.ast-single-post .entry-title,.page-title{font-size:30px}html{font-size:100%}}@media (min-width:769px){.ast-container{max-width:808px}}@media (max-width:921px){.main-header-bar .main-header-bar-navigation{display:none}}.ast-desktop .main-header-menu.submenu-with-border .sub-menu,.ast-desktop .main-header-menu.submenu-with-border .astra-full-megamenu-wrapper{border-color:#22bc66}.ast-desktop .main-header-menu.submenu-with-border .sub-menu{border-top-width:2px;border-right-width:0;border-left-width:0;border-bottom-width:0;border-style:solid}.ast-desktop .main-header-menu.submenu-with-border .sub-menu .sub-menu{top:-2px}.ast-desktop .main-header-menu.submenu-with-border .sub-menu .menu-link,.ast-desktop .main-header-menu.submenu-with-border .children .menu-link{border-bottom-width:0;border-style:solid;border-color:#eaeaea}@media (min-width:769px){.main-header-menu .sub-menu .menu-item.ast-left-align-sub-menu:hover>.sub-menu,.main-header-menu .sub-menu .menu-item.ast-left-align-sub-menu.focus>.sub-menu{margin-left:-0}}.ast-small-footer{border-top-style:solid;border-top-width:0;border-top-color:#eee}.ast-small-footer-wrap{text-align:center}.ast-header-break-point.ast-header-custom-item-inside .main-header-bar .main-header-bar-navigation .ast-search-icon{display:none}.ast-header-break-point.ast-header-custom-item-inside .main-header-bar .ast-search-menu-icon .search-form{padding:0;display:block;overflow:hidden}.ast-header-break-point .ast-header-custom-item .widget:last-child{margin-bottom:1em}.ast-header-custom-item .widget{margin:.5em;display:inline-block;vertical-align:middle}.ast-header-custom-item .widget p{margin-bottom:0}.ast-header-custom-item .widget li{width:auto}.ast-header-custom-item-inside .button-custom-menu-item .menu-link{display:none}.ast-header-custom-item-inside.ast-header-break-point .button-custom-menu-item .ast-custom-button-link{display:none}.ast-header-custom-item-inside.ast-header-break-point .button-custom-menu-item .menu-link{display:block}.ast-header-break-point.ast-header-custom-item-outside .main-header-bar .ast-search-icon{margin-right:1em}.ast-header-break-point.ast-header-custom-item-inside .main-header-bar .ast-search-menu-icon .search-field,.ast-header-break-point.ast-header-custom-item-inside .main-header-bar .ast-search-menu-icon.ast-inline-search .search-field{width:100%;padding-right:5.5em}.ast-header-break-point.ast-header-custom-item-inside .main-header-bar .ast-search-menu-icon .search-submit{display:block;position:absolute;height:100%;top:0;right:0;padding:0 1em;border-radius:0}.ast-header-break-point .ast-header-custom-item .ast-masthead-custom-menu-items{padding-left:20px;padding-right:20px;margin-bottom:1em;margin-top:1em}.ast-header-custom-item-inside.ast-header-break-point .button-custom-menu-item{padding-left:0;padding-right:0;margin-top:0;margin-bottom:0}.astra-icon-down_arrow::after{content:\e900;font-family:Astra}.astra-icon-close::after{content:\e5cd;font-family:Astra}.astra-icon-drag_handle::after{content:\e25d;font-family:Astra}.astra-icon-format_align_justify::after{content:\e235;font-family:Astra}.astra-icon-menu::after{content:\e5d2;font-family:Astra}.astra-icon-reorder::after{content:\e8fe;font-family:Astra}.astra-icon-search::after{content:\e8b6;font-family:Astra}.astra-icon-zoom_in::after{content:\e56b;font-family:Astra}.astra-icon-check-circle::after{content:\e901;font-family:Astra}.astra-icon-shopping-cart::after{content:\f07a;font-family:Astra}.astra-icon-shopping-bag::after{content:\f290;font-family:Astra}.astra-icon-shopping-basket::after{content:\f291;font-family:Astra}.astra-icon-circle-o::after{content:\e903;font-family:Astra}.astra-icon-certificate::after{content:\e902;font-family:Astra}blockquote{padding:1.2em}:root .has-ast-global-color-0-color{color:var(--ast-global-color-0)}:root .has-ast-global-color-0-background-color{background-color:var(--ast-global-color-0)}:root .wp-block-button .has-ast-global-color-0-color{color:var(--ast-global-color-0)}:root .wp-block-button .has-ast-global-color-0-background-color{background-color:var(--ast-global-color-0)}:root .has-ast-global-color-1-color{color:var(--ast-global-color-1)}:root .has-ast-global-color-1-background-color{background-color:var(--ast-global-color-1)}:root .wp-block-button .has-ast-global-color-1-color{color:var(--ast-global-color-1)}:root .wp-block-button .has-ast-global-color-1-background-color{background-color:var(--ast-global-color-1)}:root .has-ast-global-color-2-color{color:var(--ast-global-color-2)}:root .has-ast-global-color-2-background-color{background-color:var(--ast-global-color-2)}:root .wp-block-button .has-ast-global-color-2-color{color:var(--ast-global-color-2)}:root .wp-block-button .has-ast-global-color-2-background-color{background-color:var(--ast-global-color-2)}:root .has-ast-global-color-3-color{color:var(--ast-global-color-3)}:root .has-ast-global-color-3-background-color{background-color:var(--ast-global-color-3)}:root .wp-block-button .has-ast-global-color-3-color{color:var(--ast-global-color-3)}:root .wp-block-button .has-ast-global-color-3-background-color{background-color:var(--ast-global-color-3)}:root .has-ast-global-color-4-color{color:var(--ast-global-color-4)}:root .has-ast-global-color-4-background-color{background-color:var(--ast-global-color-4)}:root .wp-block-button .has-ast-global-color-4-color{color:var(--ast-global-color-4)}:root .wp-block-button .has-ast-global-color-4-background-color{background-color:var(--ast-global-color-4)}:root .has-ast-global-color-5-color{color:var(--ast-global-color-5)}:root .has-ast-global-color-5-background-color{background-color:var(--ast-global-color-5)}:root .wp-block-button .has-ast-global-color-5-color{color:var(--ast-global-color-5)}:root .wp-block-button .has-ast-global-color-5-background-color{background-color:var(--ast-global-color-5)}:root .has-ast-global-color-6-color{color:var(--ast-global-color-6)}:root .has-ast-global-color-6-background-color{background-color:var(--ast-global-color-6)}:root .wp-block-button .has-ast-global-color-6-color{color:var(--ast-global-color-6)}:root .wp-block-button .has-ast-global-color-6-background-color{background-color:var(--ast-global-color-6)}:root .has-ast-global-color-7-color{color:var(--ast-global-color-7)}:root .has-ast-global-color-7-background-color{background-color:var(--ast-global-color-7)}:root .wp-block-button .has-ast-global-color-7-color{color:var(--ast-global-color-7)}:root .wp-block-button .has-ast-global-color-7-background-color{background-color:var(--ast-global-color-7)}:root .has-ast-global-color-8-color{color:var(--ast-global-color-8)}:root .has-ast-global-color-8-background-color{background-color:var(--ast-global-color-8)}:root .wp-block-button .has-ast-global-color-8-color{color:var(--ast-global-color-8)}:root .wp-block-button .has-ast-global-color-8-background-color{background-color:var(--ast-global-color-8)}:root{--ast-global-color-0:#0170b9;--ast-global-color-1:#3a3a3a;--ast-global-color-2:#3a3a3a;--ast-global-color-3:#4b4f58;--ast-global-color-4:#f5f5f5;--ast-global-color-5:#fff;--ast-global-color-6:#e5e5e5;--ast-global-color-7:#424242;--ast-global-color-8:#000}:root{--ast-border-color:#ddd}.ast-breadcrumbs .trail-browse,.ast-breadcrumbs .trail-items,.ast-breadcrumbs .trail-items li{display:inline-block;margin:0;padding:0;border:none;background:inherit;text-indent:0}.ast-breadcrumbs .trail-browse{font-size:inherit;font-style:inherit;font-weight:inherit;color:inherit}.ast-breadcrumbs .trail-items{list-style:none}.trail-items li::after{padding:0 .3em;content:\00bb}.trail-items li:last-of-type::after{display:none}h1,.entry-content h1,h2,.entry-content h2,h3,.entry-content h3,h4,.entry-content h4,h5,.entry-content h5,h6,.entry-content h6{color:var(--ast-global-color-2)}.ast-header-break-point .main-header-bar{border-bottom-width:1px;border-bottom-color:#eee}@media (min-width:769px){.main-header-bar{border-bottom-width:1px;border-bottom-color:#eee}}.main-header-menu .menu-item,#astra-footer-menu .menu-item,.main-header-bar .ast-masthead-custom-menu-items{-js-display:flex;display:flex;-webkit-box-pack:center;-webkit-justify-content:center;-moz-box-pack:center;-ms-flex-pack:center;justify-content:center;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-moz-box-orient:vertical;-moz-box-direction:normal;-ms-flex-direction:column;flex-direction:column}.main-header-menu>.menu-item>.menu-link,#astra-footer-menu>.menu-item>.menu-link{height:100%;-webkit-box-align:center;-webkit-align-items:center;-moz-box-align:center;-ms-flex-align:center;align-items:center;-js-display:flex;display:flex}.ast-primary-menu-disabled .main-header-bar .ast-masthead-custom-menu-items{flex:unset}.header-main-layout-1 .ast-flex.main-header-container,.header-main-layout-3 .ast-flex.main-header-container{-webkit-align-content:center;-ms-flex-line-pack:center;align-content:center;-webkit-box-align:center;-webkit-align-items:center;-moz-box-align:center;-ms-flex-align:center;align-items:center}.main-header-menu .sub-menu .menu-item.menu-item-has-children>.menu-link:after{position:absolute;right:1em;top:50%;transform:translate(0,-50%) rotate(270deg)}.ast-header-break-point .main-header-bar .main-header-bar-navigation .page_item_has_children>.ast-menu-toggle::before,.ast-header-break-point .main-header-bar .main-header-bar-navigation .menu-item-has-children>.ast-menu-toggle::before,.ast-mobile-popup-drawer .main-header-bar-navigation .menu-item-has-children>.ast-menu-toggle::before,.ast-header-break-point .ast-mobile-header-wrap .main-header-bar-navigation .menu-item-has-children>.ast-menu-toggle::before{font-weight:bold;content:\e900;font-family:Astra;text-decoration:inherit;display:inline-block}.ast-header-break-point .main-navigation ul.sub-menu .menu-item .menu-link:before{content:\e900;font-family:Astra;font-size:.65em;text-decoration:inherit;display:inline-block;transform:translate(0,-2px) rotateZ(270deg);margin-right:5px}.widget_search .search-form:after{font-family:Astra;font-size:1.2em;font-weight:normal;content:\e8b6;position:absolute;top:50%;right:15px;transform:translate(0,-50%)}.astra-search-icon::before{content:\e8b6;font-family:Astra;font-style:normal;font-weight:normal;text-decoration:inherit;text-align:center;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;z-index:3}.main-header-bar .main-header-bar-navigation .page_item_has_children>a:after,.main-header-bar .main-header-bar-navigation .menu-item-has-children>a:after,.site-header-focus-item .main-header-bar-navigation .menu-item-has-children>.menu-link:after{content:\e900;display:inline-block;font-family:Astra;font-size:.6rem;font-weight:bold;text-rendering:auto;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;margin-left:10px;line-height:normal}.ast-mobile-popup-drawer .main-header-bar-navigation .ast-submenu-expanded>.ast-menu-toggle::before{transform:rotateX(180deg)}.ast-header-break-point .main-header-bar-navigation .menu-item-has-children>.menu-link:after{display:none}.ast-separate-container .blog-layout-1,.ast-separate-container .blog-layout-2,.ast-separate-container .blog-layout-3{background-color:transparent;background-image:none}.ast-separate-container .ast-article-post{background-color:var(--ast-global-color-5);background-image:none}@media (max-width:768px){.ast-separate-container .ast-article-post{background-color:var(--ast-global-color-5);background-image:none}}@media (max-width:544px){.ast-separate-container .ast-article-post{background-color:var(--ast-global-color-5);background-image:none}}.ast-separate-container .ast-article-single:not(.ast-related-post), .ast-separate-container .comments-area .comment-respond,.ast-separate-container .comments-area .ast-comment-list li, .ast-separate-container .ast-woocommerce-container, .ast-separate-container .error-404, .ast-separate-container .no-results, .single.ast-separate-container .ast-author-meta, .ast-separate-container .related-posts-title-wrapper, .ast-separate-container.ast-two-container #secondary .widget,.ast-separate-container .comments-count-wrapper, .ast-box-layout.ast-plain-container .site-content,.ast-padded-layout.ast-plain-container .site-content, .ast-separate-container .comments-area .comments-title{background-color:var(--ast-global-color-5);background-image:none}@media (max-width:768px){.ast-separate-container .ast-article-single:not(.ast-related-post), .ast-separate-container .comments-area .comment-respond,.ast-separate-container .comments-area .ast-comment-list li, .ast-separate-container .ast-woocommerce-container, .ast-separate-container .error-404, .ast-separate-container .no-results, .single.ast-separate-container .ast-author-meta, .ast-separate-container .related-posts-title-wrapper, .ast-separate-container.ast-two-container #secondary .widget,.ast-separate-container .comments-count-wrapper, .ast-box-layout.ast-plain-container .site-content,.ast-padded-layout.ast-plain-container .site-content, .ast-separate-container .comments-area .comments-title{background-color:var(--ast-global-color-5);background-image:none}}@media (max-width:544px){.ast-separate-container .ast-article-single:not(.ast-related-post), .ast-separate-container .comments-area .comment-respond,.ast-separate-container .comments-area .ast-comment-list li, .ast-separate-container .ast-woocommerce-container, .ast-separate-container .error-404, .ast-separate-container .no-results, .single.ast-separate-container .ast-author-meta, .ast-separate-container .related-posts-title-wrapper, .ast-separate-container.ast-two-container #secondary .widget,.ast-separate-container .comments-count-wrapper, .ast-box-layout.ast-plain-container .site-content,.ast-padded-layout.ast-plain-container .site-content, .ast-separate-container .comments-area .comments-title{background-color:var(--ast-global-color-5);background-image:none}}.ast-pagination .next.page-numbers{display:inherit;float:none}.ast-pagination a{color:#0091ea}.ast-pagination a:hover, .ast-pagination a:focus, .ast-pagination > span:hover:not(.dots), .ast-pagination > span.current{color:#1e73be}.ast-pagination .prev.page-numbers,.ast-pagination .next.page-numbers{padding:0 1.5em;height:2.33333em;line-height:calc(2.33333em - 3px)}.ast-pagination{display:inline-block;width:100%;padding-top:2em;text-align:center}.ast-pagination .page-numbers{display:inline-block;width:2.33333em;height:2.33333em;font-size:16px;font-size:1.06666rem;line-height:calc(2.33333em - 3px)}.ast-pagination .nav-links{display:inline-block;width:100%}@media (max-width:420px){.ast-pagination .prev.page-numbers,.ast-pagination .next.page-numbers{width:100%;text-align:center;margin:0}.ast-pagination-circle .ast-pagination .next.page-numbers,.ast-pagination-square .ast-pagination .next.page-numbers{margin-top:10px}.ast-pagination-circle .ast-pagination .prev.page-numbers,.ast-pagination-square .ast-pagination .prev.page-numbers{margin-bottom:10px}}.ast-pagination .prev,.ast-pagination .prev:visited,.ast-pagination .prev:focus,.ast-pagination .next,.ast-pagination .next:visited,.ast-pagination .next:focus{display:inline-block;width:auto}.ast-page-builder-template .ast-pagination{padding:2em}.ast-pagination .prev.page-numbers.dots,.ast-pagination .prev.page-numbers.dots:hover,.ast-pagination .prev.page-numbers.dots:focus,.ast-pagination .prev.page-numbers:visited.dots,.ast-pagination .prev.page-numbers:visited.dots:hover,.ast-pagination .prev.page-numbers:visited.dots:focus,.ast-pagination .prev.page-numbers:focus.dots,.ast-pagination .prev.page-numbers:focus.dots:hover,.ast-pagination .prev.page-numbers:focus.dots:focus,.ast-pagination .next.page-numbers.dots,.ast-pagination .next.page-numbers.dots:hover,.ast-pagination .next.page-numbers.dots:focus,.ast-pagination .next.page-numbers:visited.dots,.ast-pagination .next.page-numbers:visited.dots:hover,.ast-pagination .next.page-numbers:visited.dots:focus,.ast-pagination .next.page-numbers:focus.dots,.ast-pagination .next.page-numbers:focus.dots:hover,.ast-pagination .next.page-numbers:focus.dots:focus{border:2px solid #eaeaea;background:transparent}.ast-pagination .prev.page-numbers.dots,.ast-pagination .prev.page-numbers:visited.dots,.ast-pagination .prev.page-numbers:focus.dots,.ast-pagination .next.page-numbers.dots,.ast-pagination .next.page-numbers:visited.dots,.ast-pagination .next.page-numbers:focus.dots{cursor:default}@media (min-width:993px){.ast-pagination{padding-left:3.33333em;padding-right:3.33333em}}.ast-pagination .prev.page-numbers{float:left}.ast-pagination .next.page-numbers{float:right}@media (max-width:768px){.ast-pagination .next.page-numbers .page-navigation{padding-right:0}}@media (min-width:769px){.ast-pagination .prev.page-numbers.next,.ast-pagination .prev.page-numbers:visited.next,.ast-pagination .prev.page-numbers:focus.next,.ast-pagination .next.page-numbers.next,.ast-pagination .next.page-numbers:visited.next,.ast-pagination .next.page-numbers:focus.next{margin-right:0}}/style>link relstylesheet idastra-google-fonts-css href/wp-content/astra-local-fonts/astra-local-fonts.css mediaall>link relstylesheet idastra-menu-animation-css href/wp-content/themes/astra/assets/css/minified/menu-animation.min.css mediaall>link relstylesheet idwp-block-library-css href/wp-includes/css/dist/block-library/style.min.css mediaall>style idglobal-styles-inline-css>body{--wp--preset--color--black:#000;--wp--preset--color--cyan-bluish-gray:#abb8c3;--wp--preset--color--white:#fff;--wp--preset--color--pale-pink:#f78da7;--wp--preset--color--vivid-red:#cf2e2e;--wp--preset--color--luminous-vivid-orange:#ff6900;--wp--preset--color--luminous-vivid-amber:#fcb900;--wp--preset--color--light-green-cyan:#7bdcb5;--wp--preset--color--vivid-green-cyan:#00d084;--wp--preset--color--pale-cyan-blue:#8ed1fc;--wp--preset--color--vivid-cyan-blue:#0693e3;--wp--preset--color--vivid-purple:#9b51e0;--wp--preset--color--ast-global-color-0:var(--ast-global-color-0);--wp--preset--color--ast-global-color-1:var(--ast-global-color-1);--wp--preset--color--ast-global-color-2:var(--ast-global-color-2);--wp--preset--color--ast-global-color-3:var(--ast-global-color-3);--wp--preset--color--ast-global-color-4:var(--ast-global-color-4);--wp--preset--color--ast-global-color-5:var(--ast-global-color-5);--wp--preset--color--ast-global-color-6:var(--ast-global-color-6);--wp--preset--color--ast-global-color-7:var(--ast-global-color-7);--wp--preset--color--ast-global-color-8:var(--ast-global-color-8);--wp--preset--gradient--vivid-cyan-blue-to-vivid-purple:linear-gradient(135deg,rgba(6,147,227,1) 0%,#9b51e0 100%);--wp--preset--gradient--light-green-cyan-to-vivid-green-cyan:linear-gradient(135deg,#7adcb4 0%,#00d082 100%);--wp--preset--gradient--luminous-vivid-amber-to-luminous-vivid-orange:linear-gradient(135deg,rgba(252,185,0,1) 0%,rgba(255,105,0,1) 100%);--wp--preset--gradient--luminous-vivid-orange-to-vivid-red:linear-gradient(135deg,rgba(255,105,0,1) 0%,#cf2e2e 100%);--wp--preset--gradient--very-light-gray-to-cyan-bluish-gray:linear-gradient(135deg,#eee 0%,#a9b8c3 100%);--wp--preset--gradient--cool-to-warm-spectrum:linear-gradient(135deg,#4aeadc 0%,#9778d1 20%,#cf2aba 40%,#ee2c82 60%,#fb6962 80%,#fef84c 100%);--wp--preset--gradient--blush-light-purple:linear-gradient(135deg,#ffceec 0%,#9896f0 100%);--wp--preset--gradient--blush-bordeaux:linear-gradient(135deg,#fecda5 0%,#fe2d2d 50%,#6b003e 100%);--wp--preset--gradient--luminous-dusk:linear-gradient(135deg,#ffcb70 0%,#c751c0 50%,#4158d0 100%);--wp--preset--gradient--pale-ocean:linear-gradient(135deg,#fff5cb 0%,#b6e3d4 50%,#33a7b5 100%);--wp--preset--gradient--electric-grass:linear-gradient(135deg,#caf880 0%,#71ce7e 100%);--wp--preset--gradient--midnight:linear-gradient(135deg,#020381 0%,#2874fc 100%);--wp--preset--font-size--small:13px;--wp--preset--font-size--medium:20px;--wp--preset--font-size--large:36px;--wp--preset--font-size--x-large:42px;--wp--preset--spacing--20:.44rem;--wp--preset--spacing--30:.67rem;--wp--preset--spacing--40:1rem;--wp--preset--spacing--50:1.5rem;--wp--preset--spacing--60:2.25rem;--wp--preset--spacing--70:3.38rem;--wp--preset--spacing--80:5.06rem;--wp--preset--shadow--natural:6px 6px 9px rgba(0,0,0,.2);--wp--preset--shadow--deep:12px 12px 50px rgba(0,0,0,.4);--wp--preset--shadow--sharp:6px 6px 0 rgba(0,0,0,.2);--wp--preset--shadow--outlined:6px 6px 0 -3px rgba(255,255,255,1) , 6px 6px rgba(0,0,0,1);--wp--preset--shadow--crisp:6px 6px 0 rgba(0,0,0,1)}body{margin:0;--wp--style--global--content-size:var(--wp--custom--ast-content-width-size);--wp--style--global--wide-size:var(--wp--custom--ast-wide-width-size)}.wp-site-blocks>.alignleft{float:left;margin-right:2em}.wp-site-blocks>.alignright{float:right;margin-left:2em}.wp-site-blocks>.aligncenter{justify-content:center;margin-left:auto;margin-right:auto}:where(.wp-site-blocks) > * {margin-block-start:24px;margin-block-end:0}:where(.wp-site-blocks) > :first-child:first-child {margin-block-start:0}:where(.wp-site-blocks) > :last-child:last-child {margin-block-end:0}body{--wp--style--block-gap:24px}:where(body .is-layout-flow) > :first-child:first-child{margin-block-start:0}:where(body .is-layout-flow) > :last-child:last-child{margin-block-end:0}:where(body .is-layout-flow) > *{margin-block-start:24px;margin-block-end:0}:where(body .is-layout-constrained) > :first-child:first-child{margin-block-start:0}:where(body .is-layout-constrained) > :last-child:last-child{margin-block-end:0}:where(body .is-layout-constrained) > *{margin-block-start:24px;margin-block-end:0}:where(body .is-layout-flex) {gap:24px}:where(body .is-layout-grid) {gap:24px}body .is-layout-flow>.alignleft{float:left;margin-inline-start:0;margin-inline-end:2em}body .is-layout-flow>.alignright{float:right;margin-inline-start:2em;margin-inline-end:0}body .is-layout-flow>.aligncenter{margin-left:auto!important;margin-right:auto!important}body .is-layout-constrained>.alignleft{float:left;margin-inline-start:0;margin-inline-end:2em}body .is-layout-constrained>.alignright{float:right;margin-inline-start:2em;margin-inline-end:0}body .is-layout-constrained>.aligncenter{margin-left:auto!important;margin-right:auto!important}body .is-layout-constrained > :where(:not(.alignleft):not(.alignright):not(.alignfull)){max-width:var(--wp--style--global--content-size);margin-left:auto!important;margin-right:auto!important}body .is-layout-constrained>.alignwide{max-width:var(--wp--style--global--wide-size)}body .is-layout-flex{display:flex}body .is-layout-flex{flex-wrap:wrap;align-items:center}body .is-layout-flex>*{margin:0}body .is-layout-grid{display:grid}body .is-layout-grid>*{margin:0}body{padding-top:0;padding-right:0;padding-bottom:0;padding-left:0}a:where(:not(.wp-element-button)){text-decoration:none}.wp-element-button,.wp-block-button__link{background-color:#32373c;border-width:0;color:#fff;font-family:inherit;font-size:inherit;line-height:inherit;padding: calc(0.667em + 2px) calc(1.333em + 2px);text-decoration:none}.has-black-color{color:var(--wp--preset--color--black)!important}.has-cyan-bluish-gray-color{color:var(--wp--preset--color--cyan-bluish-gray)!important}.has-white-color{color:var(--wp--preset--color--white)!important}.has-pale-pink-color{color:var(--wp--preset--color--pale-pink)!important}.has-vivid-red-color{color:var(--wp--preset--color--vivid-red)!important}.has-luminous-vivid-orange-color{color:var(--wp--preset--color--luminous-vivid-orange)!important}.has-luminous-vivid-amber-color{color:var(--wp--preset--color--luminous-vivid-amber)!important}.has-light-green-cyan-color{color:var(--wp--preset--color--light-green-cyan)!important}.has-vivid-green-cyan-color{color:var(--wp--preset--color--vivid-green-cyan)!important}.has-pale-cyan-blue-color{color:var(--wp--preset--color--pale-cyan-blue)!important}.has-vivid-cyan-blue-color{color:var(--wp--preset--color--vivid-cyan-blue)!important}.has-vivid-purple-color{color:var(--wp--preset--color--vivid-purple)!important}.has-ast-global-color-0-color{color:var(--wp--preset--color--ast-global-color-0)!important}.has-ast-global-color-1-color{color:var(--wp--preset--color--ast-global-color-1)!important}.has-ast-global-color-2-color{color:var(--wp--preset--color--ast-global-color-2)!important}.has-ast-global-color-3-color{color:var(--wp--preset--color--ast-global-color-3)!important}.has-ast-global-color-4-color{color:var(--wp--preset--color--ast-global-color-4)!important}.has-ast-global-color-5-color{color:var(--wp--preset--color--ast-global-color-5)!important}.has-ast-global-color-6-color{color:var(--wp--preset--color--ast-global-color-6)!important}.has-ast-global-color-7-color{color:var(--wp--preset--color--ast-global-color-7)!important}.has-ast-global-color-8-color{color:var(--wp--preset--color--ast-global-color-8)!important}.has-black-background-color{background-color:var(--wp--preset--color--black)!important}.has-cyan-bluish-gray-background-color{background-color:var(--wp--preset--color--cyan-bluish-gray)!important}.has-white-background-color{background-color:var(--wp--preset--color--white)!important}.has-pale-pink-background-color{background-color:var(--wp--preset--color--pale-pink)!important}.has-vivid-red-background-color{background-color:var(--wp--preset--color--vivid-red)!important}.has-luminous-vivid-orange-background-color{background-color:var(--wp--preset--color--luminous-vivid-orange)!important}.has-luminous-vivid-amber-background-color{background-color:var(--wp--preset--color--luminous-vivid-amber)!important}.has-light-green-cyan-background-color{background-color:var(--wp--preset--color--light-green-cyan)!important}.has-vivid-green-cyan-background-color{background-color:var(--wp--preset--color--vivid-green-cyan)!important}.has-pale-cyan-blue-background-color{background-color:var(--wp--preset--color--pale-cyan-blue)!important}.has-vivid-cyan-blue-background-color{background-color:var(--wp--preset--color--vivid-cyan-blue)!important}.has-vivid-purple-background-color{background-color:var(--wp--preset--color--vivid-purple)!important}.has-ast-global-color-0-background-color{background-color:var(--wp--preset--color--ast-global-color-0)!important}.has-ast-global-color-1-background-color{background-color:var(--wp--preset--color--ast-global-color-1)!important}.has-ast-global-color-2-background-color{background-color:var(--wp--preset--color--ast-global-color-2)!important}.has-ast-global-color-3-background-color{background-color:var(--wp--preset--color--ast-global-color-3)!important}.has-ast-global-color-4-background-color{background-color:var(--wp--preset--color--ast-global-color-4)!important}.has-ast-global-color-5-background-color{background-color:var(--wp--preset--color--ast-global-color-5)!important}.has-ast-global-color-6-background-color{background-color:var(--wp--preset--color--ast-global-color-6)!important}.has-ast-global-color-7-background-color{background-color:var(--wp--preset--color--ast-global-color-7)!important}.has-ast-global-color-8-background-color{background-color:var(--wp--preset--color--ast-global-color-8)!important}.has-black-border-color{border-color:var(--wp--preset--color--black)!important}.has-cyan-bluish-gray-border-color{border-color:var(--wp--preset--color--cyan-bluish-gray)!important}.has-white-border-color{border-color:var(--wp--preset--color--white)!important}.has-pale-pink-border-color{border-color:var(--wp--preset--color--pale-pink)!important}.has-vivid-red-border-color{border-color:var(--wp--preset--color--vivid-red)!important}.has-luminous-vivid-orange-border-color{border-color:var(--wp--preset--color--luminous-vivid-orange)!important}.has-luminous-vivid-amber-border-color{border-color:var(--wp--preset--color--luminous-vivid-amber)!important}.has-light-green-cyan-border-color{border-color:var(--wp--preset--color--light-green-cyan)!important}.has-vivid-green-cyan-border-color{border-color:var(--wp--preset--color--vivid-green-cyan)!important}.has-pale-cyan-blue-border-color{border-color:var(--wp--preset--color--pale-cyan-blue)!important}.has-vivid-cyan-blue-border-color{border-color:var(--wp--preset--color--vivid-cyan-blue)!important}.has-vivid-purple-border-color{border-color:var(--wp--preset--color--vivid-purple)!important}.has-ast-global-color-0-border-color{border-color:var(--wp--preset--color--ast-global-color-0)!important}.has-ast-global-color-1-border-color{border-color:var(--wp--preset--color--ast-global-color-1)!important}.has-ast-global-color-2-border-color{border-color:var(--wp--preset--color--ast-global-color-2)!important}.has-ast-global-color-3-border-color{border-color:var(--wp--preset--color--ast-global-color-3)!important}.has-ast-global-color-4-border-color{border-color:var(--wp--preset--color--ast-global-color-4)!important}.has-ast-global-color-5-border-color{border-color:var(--wp--preset--color--ast-global-color-5)!important}.has-ast-global-color-6-border-color{border-color:var(--wp--preset--color--ast-global-color-6)!important}.has-ast-global-color-7-border-color{border-color:var(--wp--preset--color--ast-global-color-7)!important}.has-ast-global-color-8-border-color{border-color:var(--wp--preset--color--ast-global-color-8)!important}.has-vivid-cyan-blue-to-vivid-purple-gradient-background{background:var(--wp--preset--gradient--vivid-cyan-blue-to-vivid-purple)!important}.has-light-green-cyan-to-vivid-green-cyan-gradient-background{background:var(--wp--preset--gradient--light-green-cyan-to-vivid-green-cyan)!important}.has-luminous-vivid-amber-to-luminous-vivid-orange-gradient-background{background:var(--wp--preset--gradient--luminous-vivid-amber-to-luminous-vivid-orange)!important}.has-luminous-vivid-orange-to-vivid-red-gradient-background{background:var(--wp--preset--gradient--luminous-vivid-orange-to-vivid-red)!important}.has-very-light-gray-to-cyan-bluish-gray-gradient-background{background:var(--wp--preset--gradient--very-light-gray-to-cyan-bluish-gray)!important}.has-cool-to-warm-spectrum-gradient-background{background:var(--wp--preset--gradient--cool-to-warm-spectrum)!important}.has-blush-light-purple-gradient-background{background:var(--wp--preset--gradient--blush-light-purple)!important}.has-blush-bordeaux-gradient-background{background:var(--wp--preset--gradient--blush-bordeaux)!important}.has-luminous-dusk-gradient-background{background:var(--wp--preset--gradient--luminous-dusk)!important}.has-pale-ocean-gradient-background{background:var(--wp--preset--gradient--pale-ocean)!important}.has-electric-grass-gradient-background{background:var(--wp--preset--gradient--electric-grass)!important}.has-midnight-gradient-background{background:var(--wp--preset--gradient--midnight)!important}.has-small-font-size{font-size:var(--wp--preset--font-size--small)!important}.has-medium-font-size{font-size:var(--wp--preset--font-size--medium)!important}.has-large-font-size{font-size:var(--wp--preset--font-size--large)!important}.has-x-large-font-size{font-size:var(--wp--preset--font-size--x-large)!important}.wp-block-navigation a:where(:not(.wp-element-button)){color:inherit}.wp-block-pullquote{font-size:1.5em;line-height:1.6}/style>link relstylesheet idhardypress_search-css href/wp-content/plugins/hardypress/search.css mediaall>!--if IE>script srchttps://blog.healthchecks.io/wp-content/themes/astra/assets/js/minified/flexibility.min.js idastra-flexibility-js>/script>script idastra-flexibility-js-after>flexibility(document.documentElement);/script>!endif-->link relhttps://api.w.org/ hrefhttps://api.hardypress.com/wordpress/d06053a69abb99f57e4c1cb423bc2d5970292dad/>link relEditURI typeapplication/rsd+xml titleRSD href/hp-rewrite/c65f4311a305f3ad0208033a7f9ff7e2>meta namegenerator contentWordPress 6.5.5>link relicon href/wp-content/uploads/2018/12/favicon.png sizes32x32>link relicon href/wp-content/uploads/2018/12/favicon.png sizes192x192>link relapple-touch-icon href/wp-content/uploads/2018/12/favicon.png>meta namemsapplication-TileImage contenthttps://blog.healthchecks.io/wp-content/uploads/2018/12/favicon.png>style idwp-custom-css>article{padding:0 0 2em 0!important;margin-top:2em!important}.entry-meta,.entry-meta *{font-size:15px;color:#666!important}.ast-small-footer-section{font-size:17px;text-align:left;color:#666;padding-top:2em!important}.ast-separate-container #primary{margin:0!important}figcaption{color:#555!important;font-size:14px!important}figure.wp-block-image{text-align:center;margin-bottom:1.5em}.slide{border:8px solid #f8f8f8}blockquote{font-size:17px;color:#3a3a3a}pre.wp-block-code{overflow-x:auto}pre.wp-block-code code,table code{white-space:pre;background:transparent;padding:0;border-radius:0}code{font-size:14px;background:#eee;border-radius:1px;padding:1px}.ast-pagination .page-numbers{font-size:17px}pre{font-family:Roboto Mono;font-size:14px}.wp-block-table td{border:1px solid #ddd}/style>meta propertyog:locale contenten_US>meta propertyog:site_name contentHealthchecks.io>meta propertyog:title contentHealthchecks.io>meta propertyog:url contenthttps://blog.healthchecks.io/>meta propertyog:type contentwebsite>meta propertyog:description contentThe Joy of Building a Cron Monitoring Service>meta itempropname contentHealthchecks.io>meta itempropheadline contentHealthchecks.io>meta itempropdescription contentThe Joy of Building a Cron Monitoring Service>meta nametwitter:title contentHealthchecks.io>meta nametwitter:url contenthttps://blog.healthchecks.io/>meta nametwitter:description contentThe Joy of Building a Cron Monitoring Service>meta nametwitter:card contentsummary_large_image>/head>body data-hardypress1 itemtypehttps://schema.org/Blog itemscopeitemscope classhome blog wp-custom-logo ast-desktop ast-separate-container ast-no-sidebar astra-3.9.4 ast-header-custom-item-inside ast-mobile-inherit-site-logo ast-inherit-site-logo-transparent>a classskip-link screen-reader-text href#content rolelink titleSkip to content>Skip to content/a>div classhfeed site idpage>header classsite-header ast-primary-submenu-animation-fade header-main-layout-1 ast-primary-menu-disabled ast-no-menu-items ast-logo-title-inline ast-hide-custom-menu-mobile ast-menu-toggle-icon ast-mobile-header-inline idmasthead itemtypehttps://schema.org/WPHeader itemscopeitemscope itemid#masthead>div classmain-header-bar-wrap>div classmain-header-bar>div classast-container>div classast-flex main-header-container>div classsite-branding>div classast-site-identity itemtypehttps://schema.org/Organization itemscopeitemscope>span classsite-logo-img>a href/ classcustom-logo-link relhome aria-currentpage>img width256 height77 src/wp-content/uploads/2018/12/cropped-h-blog-2.png classcustom-logo altHealthchecks.io decodingasync srcset/wp-content/uploads/2018/12/cropped-h-blog-2.png 1x, /wp-content/uploads/2018/12/h-blog@2x.png 2x>/a>/span> /div>/div>/div>/div>/div>/div>/header>div idcontent classsite-content>div classast-container>div idprimary classcontent-area primary>main idmain classsite-main>div classast-row>article classpost-1525 post type-post status-publish format-standard has-post-thumbnail hentry category-uncategorized ast-col-sm-12 ast-article-post idpost-1525 itemtypehttps://schema.org/CreativeWork itemscopeitemscope>div classast-post-format- blog-layout-1>div classpost-content ast-col-md-12>header classentry-header>h2 classentry-title itempropheadline>a href/2024/10/how-healthchecks-io-sends-webhook-notifications/ relbookmark>How Healthchecks.io Sends Webhook Notifications/a>/h2> div classentry-meta>By span classposted-by vcard author itemtypehttps://schema.org/Person itemscopeitemscope itempropauthor> a titleView all posts by Pēteris Caune href/author/cuu508/ relauthor classurl fn n itempropurl>span classauthor-name itempropname>Pēteris Caune/span>/a>/span>/ span classposted-on>span classpublished itempropdatePublished> October 4, 2024/span>span classupdated itempropdateModified> October 4, 2024/span>/span>/div> /header>div classentry-content clear itemproptext>p>Webhooks are a powerful way to notify external systems about checks changing state in Healthchecks.io. Webhook notifications are available to all user accounts, paid and free./p>p>Webhooks were the second notification method supported by Healthchecks (the first one was email). The webhook delivery code started as a simple code>requests.get(user_supplied_url)/code> and evolved. Today, the webhook integration in Healthchecks supports:/p>ul>li>HTTP GET, POST, and PUT requests with user-defined request bodies./li>li>User-defined request headers./li>li>Placeholder values like $NAME and $STATUS that can be used in the URL, the headers, or the request body./li>li>Separate webhook configurations for “check goes up” and “check goes down” events./li>li>Retries when requests time out or return non-2xx status code./li>/ul>p>In terms of implementation, none of the above is super complicated. When the user sets up a webhook integration, we collect the webhook configuration. When it is time to send a notification, we assemble the URL, the headers, and the request body, and pass them to our HTTP client library of choice. But two security-related aspects are a little more interesting:/p>ul>li>We want to prevent webhook requests from accessing private IP addresses (10.x.x.x, 192.168.x.x, …)./li>li>Webhook targets can sometimes take a long time to respond. One user’s slow notifications should not block or delay another user’s normal notifications./li>/ul>h3 classwp-block-heading>Private IP Addresses/h3>p>Malicious users can set up webhook URLs to tamper with resources in the Healthchecks.io internal network. They can also set up DNS records that resolve to private IP addresses. So it is not enough to check for private IP ranges in webhook URLs using e.g. regular expressions./p>p>I switched Healthchecks to using a hrefhttp://pycurl.io/>pycurl/a> for making outbound HTTP requests. pycurl is a Python wrapper for libcurl, and libcurl lets you specify a a hrefhttps://curl.se/libcurl/c/CURLOPT_OPENSOCKETFUNCTION.html>CURLOPT_OPENSOCKETFUNCTION/a> callback function. This function receives an IP address em>after DNS resolution/em>, and can decide whether to connect to it or not./p>p>Healthchecks a hrefhttps://healthchecks.io/docs/self_hosted_configuration/#INTEGRATIONS_ALLOW_PRIVATE_IPS>has a site-wide configuration setting/a> for enabling/disabling webhook requests to private IP addresses. This setting is disabled on the hosted service at Healthchecks.io. Operators of self-hosted Healthchecks instances, on the other hand, sometimes specifically em>need/em> webhooks to access services running inside their internal network, and they can enable it./p>p>When migrating Healthchecks from requests to pycurl, I wrote a hrefhttps://github.com/healthchecks/healthchecks/blob/master/hc/lib/curl.py>a wrapper for pycurl that mimics the requests API/a>, and thus could be used as a drop-in replacement. It does not cover the full functionality of requests, but it does cover the functionality that Healthchecks uses./p>h3 classwp-block-heading>Slow Webhook Targets/h3>p>Users can set up webhooks to targets that take a long time to respond, and then generate frequent notifications to these targets. Doing so would keep the notification-sending process busy and delay notifications for all other users. Users could do this maliciously, but this could also happen (and has happened) unintentionally./p>p>The first obvious mitigation was to implement a time budget for each webhook delivery: if a webhook delivery (including retries) takes too long, we abort it./p>p>Another mitigation was to prioritize notifications to integrations with lower historic send times. If we have multiple deliveries lined up, start with the quick ones, and do the slow ones last./p>p>The notification sender is implemented as a Django management command (“manage.py sendalerts”). A simple way to increase sending capacity would be to run multiple “sendalerts” processes concurrently. This works, but each process needs at least one database connection. I am not running PgBouncer (and want to delay introducing new infrastructure pieces for as long as possible), so I cannot go too crazy with many concurrent “sendalerts” processes./p>p>A few weeks ago I completed work on another idea to increase the sending capacity. The “sendalerts” process now uses multiple worker threads to send notifications. The worker threads share database connections using a hrefhttps://www.psycopg.org/psycopg3/docs/api/pool.html>psycopg3 connection pool/a>, which a hrefhttps://docs.djangoproject.com/en/5.1/releases/5.1/#postgresql-connection-pools>Django recently added support for/a>. There can be more worker threads than database connections available in the pool, but strong>the worker threads are programmed to return DB connections to the pool before potentially long network IO operations/strong>, allowing other threads to advance. With an appropriately set worker count, this allows hundreds of in-progress webhook requests while using only a few DB connections./p>p>After implementing the worker threads, I removed the prioritization by historic send time. I also increased the timeout value for outbound HTTP requests as now I could afford to! The timeout is currently set to 30 seconds, and Healthchecks retries failed requests up to 2 times. So a single delivery can take up to 3 * 30 90 seconds./p>h3 classwp-block-heading>Closing Notes/h3>p>Healthchecks.io now uses the threaded notification sender for delivering all notification types, not just webhooks. There are integration types other than webhooks that are sometimes slow. For example, Signal and MS Teams notifications sometimes take multiple seconds to complete. The above changes benefit all integration types, not just webhooks. Webhooks, however, are the most risky, as they can be fully configured by users./p>p>Thanks for reading,br>–Pēteris/p>/div>/div>/div>/article>article classpost-1520 post type-post status-publish format-standard has-post-thumbnail hentry category-uncategorized ast-col-sm-12 ast-article-post idpost-1520 itemtypehttps://schema.org/CreativeWork itemscopeitemscope>div classast-post-format- blog-layout-1>div classpost-content ast-col-md-12>header classentry-header>h2 classentry-title itempropheadline>a href/2024/07/running-one-man-saas-9-years-in/ relbookmark>Running One-man SaaS, 9 Years In/a>/h2> div classentry-meta>By span classposted-by vcard author itemtypehttps://schema.org/Person itemscopeitemscope itempropauthor> a titleView all posts by Pēteris Caune href/author/cuu508/ relauthor classurl fn n itempropurl>span classauthor-name itempropname>Pēteris Caune/span>/a>/span>/ span classposted-on>span classpublished itempropdatePublished> July 29, 2024/span>span classupdated itempropdateModified> July 29, 2024/span>/span>/div> /header>div classentry-content clear itemproptext>p>Healthchecks.io launched in July 2015, which means this year we turn 9. Time flies!/p>p>Previous status updates:/p>ul>li>In 2018, a hrefhttps://blog.healthchecks.io/2018/08/my-one-person-saas-side-project-celebrates-its-third-birthday/>My One-person SaaS Side Project Celebrates its Third Birthday/a>/li>li>In 2021, a hrefhttps://blog.healthchecks.io/2021/07/healthchecks-turns-6-status-update/>Healthchecks Turns 6, Status Update/a>/li>/ul>h3 classwp-block-heading>Money/h3>p>Healthchecks.io currently has 652 paying customers, and the monthly recurring revenue is 14043 USD. MRR graph:/p>figure classwp-block-image size-large>img fetchpriorityhigh decodingasync width1024 height393 src/wp-content/uploads/2024/07/mrr-1024x393.png alt classwp-image-1521>/figure>p>Side note: to minimize the number of data sub-processors, I am not using revenue analytics services. I used a script and a spreadsheet to make the MRR graph!/p>p>I’m happy to see MRR gradually go up, but I’m not optimizing for it. Healthchecks.io is sustainable as-is, and so I’m optimizing for enjoyment and life/work balance./p>p>More stats (user count, check count, pings/day) are available on the a hrefhttps://healthchecks.io/about/>Healthchecks.io About page/a>./p>h3 classwp-block-heading>Still a one-man business?/h3>p>Yes, Healthchecks.io is still a one-man business. Until 2022, I was part-time contracting. Since January 2022 Healthchecks.io has been my only source of income, but I work on it part-time./p>p>At least for the time being I’m not looking to expand the team. A large part of why I’m a “solopreneur” is because I do not want to manage or be managed. A cofounder or employee would mean regular meetings to discuss what’s done, and what’s to be done. It would be awesome to find someone who just magically does great work without needing any attention. Just brief monthly summaries of high-quality contributions, better than I could have done. But I don’t think I can find someone like that, and I also don’t think I could afford them./p>h3 classwp-block-heading>Growth Goals/h3>p>I’m not planning to tighten the limits of the free plans. I started Healthchecks in 2015 because I thought the existing services (Dead Man’s Snitch and Cronitor) were overpriced. I started with “I think this can be done better and cheaper”, and I’m sticking with it./p>p>For the same reason, I’m also not planning to raise pricing for paid plans./p>p>I’m choosing not to pursue enterprise customers who ask about PO billing, payments by wire transfer, custom agreements, and signing up to vendor portals. “But you are leaving money on the table!” – yes, it is a conscious decision. In my situation, the extra money will not make a meaningful difference, but the additional burden will make me more busy and grumpy./p>p>Feature-wise, I am happy with the current scope and feature set of Healthchecks. I am em>not/em> planning to expand the scope and add e.g. active uptime monitoring, hosted status pages, or APM features./p>p>Healthchecks the product is a hrefhttps://hachyderm.io/@danderson/112766460393943288>hobbit software/a> and Healthchecks.io the business is a lifestyle business./p>h3 classwp-block-heading>Hosting Setup/h3>p>The hosting setup is mostly the same as in a hrefhttps://blog.healthchecks.io/2022/02/healthchecks-io-hosting-setup-2022-edition/>2022/a>. Just a few updates:/p>ul>li>Web servers upgraded to Hetzner’s AX42 (AMD 8700GE, 8 cores). On the old machines, saw a few nonsensical Python exceptions. A kernel update and a reboot didn’t fix it. Rather than messing with hardware troubleshooting, I upgraded to newer, faster, and more efficient machines./li>li>Database servers upgraded to Hetzner’s EX101 (Intel 13900, 8+16 cores). I was setting up new database replicas after a hrefhttps://status.healthchecks.io/en/incidents/m7Qv7s8KCsVdMVjGvMbpJb/>an outage and failover event/a> and took the opportunity to upgrade hardware./li>li>Healthchecks.io a hrefhttps://blog.healthchecks.io/2023/08/notes-on-self-hosted-transactional-email/>now sends its own email using maddy/a>./li>li>Healthchecks.io a hrefhttps://blog.healthchecks.io/2022/04/we-moved-some-data-to-s3/>now stores ping body data in S3-compatible object storage/a>. This keeps the PostgreSQL database size down but adds reliance on an external service./li>/ul>p>That’s it for now, thank you for reading! Here’s to another 9 years, and in the closing here’s a complimentary picture of me trying to fit through pull-up bars, and my kids, Nora and Alberts, cheering:/p>figure classwp-block-image size-large>img decodingasync width1024 height768 src/wp-content/uploads/2024/07/pull_up_bars-1024x768.jpg alt classwp-image-1522>/figure>p>Happy monitoring,br>Pēteris,br>Healthchecks.io/p>/div>/div>/div>/article>article classpost-1514 post type-post status-publish format-standard hentry category-uncategorized ast-col-sm-12 ast-article-post idpost-1514 itemtypehttps://schema.org/CreativeWork itemscopeitemscope>div classast-post-format- ast-no-thumb blog-layout-1>div classpost-content ast-col-md-12>header classentry-header>h2 classentry-title itempropheadline>a href/2024/07/data-breach-report-some-sms-notifications-sent-to-france-and-italy-were-exposed/ relbookmark>Data Breach Report: Some SMS Notifications Sent To France and Italy Were Exposed/a>/h2> div classentry-meta>By span classposted-by vcard author itemtypehttps://schema.org/Person itemscopeitemscope itempropauthor> a titleView all posts by Pēteris Caune href/author/cuu508/ relauthor classurl fn n itempropurl>span classauthor-name itempropname>Pēteris Caune/span>/a>/span>/ span classposted-on>span classpublished itempropdatePublished> July 19, 2024/span>span classupdated itempropdateModified> July 19, 2024/span>/span>/div> /header>div classentry-content clear itemproptext>p>On July 2, 2024 we received a notice from Twilio, our SMS provider, about a data leak involving IdentifyMobile, one of their downstream carriers. The downstream carrier had made an AWS S3 bucket public from May 10-15, 2024. The bucket contained message-related data sent between January 1, 2024, and May 15, 2024./p>p>After requesting additional information, Twilio informed us that the leak included strong>13 SMS notifications/strong> sent by Healthchecks.io. The leaked data includes strong>message body, recipient number, timestamp/strong>. Unfortunately Twilio could not determine which specific recipient numbers were impacted, but they knew only messages to strong>France and Italy /strong>were impacted. On July 5, we notified all users with phone numbers in the affected regions, 40 accounts./p>p>strong>Q: I received “Notice of Security Incident With SMS Notifications” from Healthchecks. Is there anything I should do?/strong>/p>p>Your Healthchecks.io account is not compromised, no need to change its password./p>p>You could consider switching from SMS to a different notification method which does not require your phone number, for example Pushover. No service is immune to security incidents, but if they do not have your phone number in the first place, they cannot leak it./p>p>strong>Q: Why did you notify 40 accounts if only 13 messages were exposed?/strong>/p>p>Twilio provided a list of exposed message IDs, but not the associated phone numbers. We cannot associate message IDs with phone numbers, because we have configured our Twilio account to retain message logs for only 7 days. We had selected the relatively short log retention period, ironically, to minimize the damage in case the message logs somehow leaked./p>p>We asked Twilio support to request the recipient phone number data from IdentifyMobile, as they presumably still have access to the data that was exposed. According to Twilio, IdentifyMobile are “currently unable to share the requested information due to the sensitive nature of it”./p>p>strong>Timeline/strong>/p>ul>li>May 10, 2024: IdentifyMobile makes AWS S3 bucket containing sensitive data public./li>li>May 15, 2024: IdentifyMobile fixes the leak./li>li>July 2, 2024: Twilio sends a notice of security incident to its customers./li>li>July 3, 2024: We request additional information from Twilio support./li>li>July 4, 2024: Twilio support clarifies what information was exposed, and provides a list of the 13 exposed message IDs./li>li>July 5, 2024: We send a notice of security incident to the 40 potentially affected users./li>li>July 5, 2024: We ask Twilio support to request recipient numbers from IdentifyMobile. On the 3rd attempt, Twilio agrees to do it./li>li>July 10, 2024: Twilio support informs us IdentifyMobile cannot share the requested information./li>li>July 11-16, 2024: We ask Twilio support followup questions about plans to audit their other carriers and sub-carriers, and receive non-specific answers./li>li>July 19, 2024: We publish this report./li>/ul>/div>/div>/div>/article>article classpost-1442 post type-post status-publish format-standard hentry category-uncategorized ast-col-sm-12 ast-article-post idpost-1442 itemtypehttps://schema.org/CreativeWork itemscopeitemscope>div classast-post-format- ast-no-thumb blog-layout-1>div classpost-content ast-col-md-12>header classentry-header>h2 classentry-title itempropheadline>a href/2023/12/oncalendar-schedules-monitor-systemd-timers-with-healthchecks-io/ relbookmark>OnCalendar schedules: Monitor Systemd Timers with Healthchecks.io/a>/h2> div classentry-meta>By span classposted-by vcard author itemtypehttps://schema.org/Person itemscopeitemscope itempropauthor> a titleView all posts by Pēteris Caune href/author/cuu508/ relauthor classurl fn n itempropurl>span classauthor-name itempropname>Pēteris Caune/span>/a>/span>/ span classposted-on>span classpublished itempropdatePublished> December 18, 2023/span>span classupdated itempropdateModified> May 1, 2024/span>/span>/div> /header>div classentry-content clear itemproptext>p>Healthchecks now supports a hrefhttps://www.man7.org/linux/man-pages/man7/systemd.time.7.html#CALENDAR_EVENTS>OnCalendar/a> schedules, used for a hrefhttps://wiki.archlinux.org/title/Systemd/Timers>scheduling tasks with systemd timers/a>. Here’s what’s new: when creating a check, you can now switch between “Simple”, “Cron” and “OnCalendar” schedules:/p>figure classwp-block-image size-large>img decodingasync width918 height1024 src/wp-content/uploads/2023/12/image-2-918x1024.png alt classwp-image-1448>/figure>p>You can also edit schedules (and switch schedule types) for existing checks:/p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height626 src/wp-content/uploads/2023/12/image-3-1024x626.png alt classwp-image-1450>/figure>p>The UI control for entering the schedule is a multi-line textbox, and yes, you can specify multiple schedules there – Healthchecks will expect a ping when em>any/em> schedule matches:/p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height626 src/wp-content/uploads/2023/12/image-4-1024x626.png alt classwp-image-1452>/figure>p>Note: the schedule field is currently limited to 100 characters. You will be able to enter 2-3 schedules, but probably not 10+ schedules./p>p>systemd allows you to specify a timezone inside the OnCalendar expression. So does Healthchecks:/p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height626 src/wp-content/uploads/2023/12/image-5-1024x626.png alt classwp-image-1454>/figure>p>The API now supports OnCalendar schedules as well. You can pass either cron schedule or OnCalendar expression(s) in the “schedule” field for the a hrefhttps://healthchecks.io/docs/api/#create-check>Create a new check/a> and a hrefhttps://healthchecks.io/docs/api/#update-check>Update an existing check/a> calls, and Healthchecks will detect the schedule type automatically:/p>pre classwp-block-code>code>$ curl -s https://healthchecks.io/api/v3/checks/ \ --header "X-Api-Key: fdYYw32ftDvYQoCe4C1JUgp7SlPbOYTI" \ --data '{"name": "Runs at 8AM", "schedule": "8:00"}' | jq .{ "name": "Runs at 8AM", "slug": "", "tags": "", "desc": "", "grace": 3600, "n_pings": 0, "status": "new", "started": false, "last_ping": null, "next_ping": null, "manual_resume": false, "methods": "", "subject": "", "subject_fail": "", "start_kw": "", "success_kw": "", "failure_kw": "", "filter_subject": false, "filter_body": false, "ping_url": "https://hc-ping.com/97f70e1c-bf2b-4244-ba44-de413c93fab4", "update_url": "https://healthchecks.io/api/v3/checks/97f70e1c-bf2b-4244-ba44-de413c93fab4", "pause_url": "https://healthchecks.io/api/v3/checks/97f70e1c-bf2b-4244-ba44-de413c93fab4/pause", "resume_url": "https://healthchecks.io/api/v3/checks/97f70e1c-bf2b-4244-ba44-de413c93fab4/resume", "channels": "", "schedule": "8:00", "tz": "UTC"}/code>/pre>p>Under the hood, the OnCalendar schedule parsing logic is implemented a hrefhttps://github.com/cuu508/oncalendar/>in a separate “oncalendar” library/a>. Feel free to use it in your own Python projects as well!/p>p>The OnCalendar schedule support is live on a hrefhttps://healthchecks.io>https://healthchecks.io/a> and available to all accounts. Happy monitoring!/p>p>–Pēteris/p>/div>/div>/div>/article>article classpost-1409 post type-post status-publish format-standard has-post-thumbnail hentry category-uncategorized ast-col-sm-12 ast-article-post idpost-1409 itemtypehttps://schema.org/CreativeWork itemscopeitemscope>div classast-post-format- blog-layout-1>div classpost-content ast-col-md-12>header classentry-header>h2 classentry-title itempropheadline>a href/2023/11/comparison-of-cron-monitoring-services-november-2023/ relbookmark>Comparison of Cron Monitoring Services (November 2023)/a>/h2> div classentry-meta>By span classposted-by vcard author itemtypehttps://schema.org/Person itemscopeitemscope itempropauthor> a titleView all posts by Pēteris Caune href/author/cuu508/ relauthor classurl fn n itempropurl>span classauthor-name itempropname>Pēteris Caune/span>/a>/span>/ span classposted-on>span classpublished itempropdatePublished> November 23, 2023/span>span classupdated itempropdateModified> January 19, 2024/span>/span>/div> /header>div classentry-content clear itemproptext>p>In this post I’m comparing cron monitoring features of four services: a hrefhttps://cronitor.io/>Cronitor/a>, a hrefhttps://healthchecks.io/>Healthchecks.io/a>, a hrefhttps://uptimerobot.com/>Uptime Robot/a>, a hrefhttps://sentry.io/welcome/>Sentry/a>./p>p>How I picked the services for comparison: I searched for “cron monitoring” on Google and picked the top results in their order of appearance./p>p>Disclaimer: I run Healthchecks.io, so I’m a biased source. I’ve tried to get the facts right, but choosing what features to compare, and what differences to highlight, is of course subjective. When in doubt, do your own research!/p>h3 classwp-block-heading>Business Stats/h3>p>strong>Cronitor/strong> launched in 2014, is registered in the United States and runs on AWS. Cronitor is a bootstrapped company, and a hrefhttps://cronitor.io/about>is operated by three friendly humans/a>. Cronitor started as a cron monitoring service, but has expanded to website uptime monitoring, real user monitoring, and hosted status pages. Cronitor is a proprietary product and uses the SaaS business model./p>p>strong>Healthchecks.io/strong> launched in 2015, is registered in Latvia and runs on Hetzner (Germany). Healthchecks.io is a bootstrapped company, run by a solo founder. Healthchecks.io focuses on doing one thing and doing it well: em>alerting when something does not happen on time/em>. Healthchecks.io is open source (a hrefhttps://github.com/healthchecks/healthchecks>source on GitHub/a>), users can use the hosted service, or run a self-hosted instance./p>p>strong>Uptime Robot/strong> launched in 2010, is registered in Malta, and runs on Limestone Networks, AWS, and DigitalOcean. UptimeRobot started as a free website uptime monitoring service and added cron monitoring and hosted status pages support in 2019. After a hrefhttps://uptimerobot.com/blog/uptimerobot-2020-update/>getting acquired/a> in late 2019, UptimeRobot accelerated development and reorganized its pricing structure. Uptime Robot is a proprietary product and uses the SaaS business model./p>p>strong>Sentry/strong> launched in 2012, is registered in the United States and runs on AWS and Google Cloud. Sentry is a VC-funded company and has 200+ employees. Sentry started as an error tracking service, grew into APM, and launched cron monitoring support in public beta in January 2023. Sentry uses the SaaS business model, but its source code is available under the a hrefhttps://fsl.software/>FSL license/a>. Sentry is a complex product with many moving parts. Self-hosting is possible but is not trivial./p>h3 classwp-block-heading>Pricing/h3>p>Each reviewed service except Healthchecks.io bundles several products under one account:/p>ul>li>Cronitor: strong>cron monitoring/strong>, website uptime monitoring, RUM, status pages./li>li>Uptime Robot: strong>website uptime monitoring/strong>, cron monitoring, status pages./li>li>Sentry: strong>error tracking/strong>, APM, code coverage./li>/ul>p>The total set of functionality you get from a paid account on each service is vastly different, so their pricing is not directly comparable. With that in mind, here is the pricing summary for each service, as of November 2023, for monitoring cron jobs specifically./p>p>strong>Cronitor/strong>/p>ul>li>Free plan: monitor up to 5 jobs./li>li>Business plan: $2/mo for 1 job./li>/ul>p>Monitoring 100 jobs with Cronitor would cost $200/mo./p>p>strong>Healthchecks.io/strong>/p>ul>li>Free plan: monitor up to 20 jobs./li>li>Business plan: $20/mo for 100 jobs./li>li>Business Plus plan: $80/mo for 1000 jobs./li>/ul>p>Monitoring 100 jobs with Healthchecks.io would cost $20/mo. Healthchecks.io offers sponsored accounts for non-profits and open-source projects (a hrefhttps://healthchecks.io/faq/#free-for-open-source>details/a>)./p>p>strong>Uptime Robot/strong>/p>ul>li>Solo plan: $8/mo for 10 jobs or $19/mo for 50 jobs./li>li>Team plan: $34/mo for 100 jobs./li>li>Enterprise plan: $64/mo for 200 jobs./li>/ul>p>Monitoring 100 jobs with Uptime Robot would cost $34/mo. Uptime Robot offers sponsored accounts for charities and other non-profits (a hrefhttps://uptimerobot.com/sponsorship-testimonials/>details/a>)./p>p>strong>Sentry/strong>/p>p>s>Sentry Cron Monitoring feature is currently in open beta. The limits for different pricing plans are not known yet./s> Sentry a hrefhttps://blog.sentry.io/cron-monitoring-is-now-generally-available/>announced/a> general availability and pricing in January 16, 2024. /p>ul>li>Free: monitor 1 cron job for free./li>li>Paid: $0.78/mo for 1 job./li>/ul>p>Monitoring 100 jobs with Sentry would cost $77/mo. Sentry offers sponsored accounts for non-profits, open-source, and students (a hrefhttps://sentry.io/for/good/>details/a>)./p>h3 classwp-block-heading>Timeout-based Schedules/h3>p>When using timeout-based schedules the user specifies a period (for example, one hour). The monitored system is expected to “check in” (send an HTTP request to a unique address) at least every period. When a check-in is missed, the monitoring system declares an outage and notifies you./p>p>This monitoring technique is also sometimes called strong>Heartbeat Monitoring/strong>. All four reviewed services support timeout-based schedules./p>h3 classwp-block-heading>Cron Expression Schedules/h3>p>The user specifies a cron expression (for example, “0/5 * * * *”) and a timezone. The monitoring system calculates expected “check in” deadlines based on the cron expression./p>p>Supported by: Cronitor, Healthchecks.io, Sentry./p>p>Not supported by: Uptime Robot./p>p>Cronitor and Sentry use the a hrefhttps://github.com/kiorky/croniter>croniter/a> library to evaluate cron expressions. Healthchecks.io uses the a hrefhttps://github.com/cuu508/cronsim>cronsim/a> library./p>h3 classwp-block-heading>Start and Fail Signals/h3>p>In addition to basic “I’m alive!” check-in messages, monitoring services typically support additional signal types:/p>ul>li>“job started” signal: allows the measurement of job durations, and alerting when a job takes too long./li>li>“job failed” signal: allows the job to explicitly declare itself as failed./li>/ul>p>Supported by: Cronitor (a hrefhttps://cronitor.io/docs/telemetry-api#parameters>docs/a>), Healthchecks.io (a hrefhttps://healthchecks.io/docs/http_api/>docs/a>), Sentry (a hrefhttps://docs.sentry.io/product/crons/getting-started/http/#check-ins-recommended>docs/a>)./p>p>Not supported by: Uptime Robot./p>h3 classwp-block-heading>Check-in Via Email/h3>p>With this feature, clients can “check in” by sending an email message to a job-specific email address. This comes in handy when integrating with services that only support status reports via emails, or when working in restrictive environments where only email is allowed through./p>p>Supported by: Cronitor (a hrefhttps://cronitor.io/docs/telemetry-api#email-integration>docs/a>), Healthchecks.io (a hrefhttps://healthchecks.io/docs/email/>docs/a>)./p>p>Not supported by: Uptime Robot, Sentry./p>h3 classwp-block-heading>Auto-Provisioning/h3>p>With auto-provisioning clients can perform check-ins for jobs that the monitoring system does not yet know about, and the monitoring service registers the new jobs on the fly. Auto-provisioning is handy in dynamic environments where the set of monitored jobs changes frequently./p>p>Supported by: Cronitor (a hrefhttps://cronitor.io/docs/telemetry-api#send-events>docs/a>), Healthchecks.io (a hrefhttps://healthchecks.io/docs/http_api/#success-slug>docs/a>), Sentry (a hrefhttps://docs.sentry.io/product/crons/getting-started/http/#creating-or-updating-a-monitor-through-a-check-in-optional>docs/a>)./p>p>Not supported by: Uptime Robot./p>h3 classwp-block-heading>Client SDKs and API/h3>p>strong>Cronitor/strong> provides first-party command-line client and SDKs for Java, JavaScript, Kubernetes, PHP, Python, Ruby, and Sidekiq. There are also third-party SDKs for Terraform and .Net./p>p>strong>Healthchecks.io/strong> does not provide first-party client SDKs. There are a number of a hrefhttps://healthchecks.io/docs/resources/>third-party client libraries/a>./p>p>strong>Sentry/strong> provides first-party command-line client and SDKs for Celery, Go, Java, JavaScript, Laravel, Node, PHP, Python, Quartz, Rails, Ruby, and Spring./p>p>strong>Uptime Robot/strong> does not provide first-party client SDKs./p>p>All four services provide HTTP API: a hrefhttps://cronitor.io/docs/api-overview>Cronitor API docs/a>, a hrefhttps://healthchecks.io/docs/api/>Healthchecks.io API docs/a>, a hrefhttps://uptimerobot.com/api/>Uptime Robot API docs/a>, a hrefhttps://docs.sentry.io/api/crons/>Sentry API docs/a>./p>h3 classwp-block-heading>Notification Methods/h3>p>Each reviewed service supports a number of different ways to deliver downtime notifications:/p>p>strong>Cronitor/strong>/p>ul>li>Free: email, webhooks (only GET requests), MS Teams, Slack, Telegram./li>li>Paid: Opsgenie, PagerDuty, SMS, Splunk On-Call./li>/ul>p>strong>Healthchecks.io/strong>/p>ul>li>Free: email, webhooks, Discord, LINE Notify, Matrix, Mattermost, MS Teams, Opsgenie, PagerDuty, PagerTree, Pushbullet, Pushover, Rocket.Chat, Signal, Slack, Spike.sh, Telegram, Trello, Splunk On-Call, Zulip./li>li>Paid: SMS, voice calls, WhatsApp./li>/ul>p>strong>Uptime Robot/strong>/p>ul>li>Free: Android and iOS app, email, Google Chat, Discord, Pushbullet, Pushover, Splunk On-Call./li>li>Paid: webhooks, MS Teams, PagerDuty, Slack, SMS, Telegram, voice calls, Zapier./li>/ul>p>strong>Sentry/strong>/p>ul>li>Free: email, webhooks./li>li>Paid: Amixr, Discord, MS Teams, Opsgenie, PagerDuty, Pushover, Rocket.Chat, Rootly, Slack, Spike.sh, SMS, TaskCall, Threads, Splunk On-Call./li>/ul>h3 classwp-block-heading>Project Management, User and Team Management, Authentication/h3>p>strong>Cronitor/strong>/p>p>Cronitor supports organizing jobs into Environments. Within each environment, jobs can be grouped into groups. Jobs can be annotated with tags./p>p>Cronitor supports multiple team members ($5/mo for each additional user). Team members can have “admin”, “user”, “readonly” roles./p>p>Cronitor supports SAML2 SSO, which costs an extra $5/mo for every team member. Cronitor does not support two-factor authentication./p>p>strong>Healthchecks.io/strong>/p>p>Healthchecks.io supports organizing jobs into Projects. Jobs can be annotated with tags./p>p>Healthchecks.io supports multiple team members with “owner”, “manager”, “user”, and “read-only” roles./p>p>Healthchecks.io does not support any form of SSO. Healthchecks.io supports two-factor authentication using WebAuthn and using one-time codes (TOTP)./p>p>strong>Uptime Robot/strong>/p>p>Uptime Robot does not support grouping or tagging jobs./p>p>Uptime Robot’s higher-priced plans support multiple team members with “admin”, “read”, and “write” roles./p>p>Uptime Robot does not support any form of SSO. Uptime Robot supports two-factor authentication using one-time codes (TOTP)./p>p>strong>Sentry/strong>/p>p>Sentry supports organizing jobs into Projects and Environments./p>p>Sentry supports multiple team members with “billing”, “member”, “admin”, “manager”, and “owner” roles./p>p>Sentry offers many options for SSO: Google, GitHub, Okta, SAML2, and others. All options except Google and GitHub require the Business ($80/mo) billing plan. Sentry supports two-factor authentication using U2F, one-time codes (TOTP), and recovery codes./p>h3 classwp-block-heading>Feature Matrix/h3>figure classwp-block-table is-style-regular>table>tbody>tr>td classhas-text-align-right data-alignright>/td>td classhas-text-align-center data-aligncenter>Cronitor/td>td classhas-text-align-center data-aligncenter>Healthchecks.io/td>td classhas-text-align-center data-aligncenter>Uptime Robot/td>td classhas-text-align-center data-aligncenter>Sentry/td>/tr>tr>td classhas-text-align-right data-alignright>Business registered in/td>td classhas-text-align-center data-aligncenter>🇺🇸/td>td classhas-text-align-center data-aligncenter>🇱🇻/td>td classhas-text-align-center data-aligncenter>🇲🇹/td>td classhas-text-align-center data-aligncenter>🇺🇸/td>/tr>tr>td classhas-text-align-right data-alignright>Servers hosted in/td>td classhas-text-align-center data-aligncenter>🇺🇸/td>td classhas-text-align-center data-aligncenter>🇩🇪/td>td classhas-text-align-center data-aligncenter>🇺🇸/td>td classhas-text-align-center data-aligncenter>🇺🇸/td>/tr>tr>td classhas-text-align-right data-alignright>Team size/td>td classhas-text-align-center data-aligncenter>3/td>td classhas-text-align-center data-aligncenter>1/td>td classhas-text-align-center data-aligncenter>10+/td>td classhas-text-align-center data-aligncenter>200+/td>/tr>tr>td classhas-text-align-right data-alignright>Founded in/td>td classhas-text-align-center data-aligncenter>2014/td>td classhas-text-align-center data-aligncenter>2015/td>td classhas-text-align-center data-aligncenter>2010/td>td classhas-text-align-center data-aligncenter>2012/td>/tr>tr>td classhas-text-align-right data-alignright>Jobs in the free plan/td>td classhas-text-align-center data-aligncenter>5/td>td classhas-text-align-center data-aligncenter>20/td>td classhas-text-align-center data-aligncenter>0/td>td classhas-text-align-center data-aligncenter>?/td>/tr>tr>td classhas-text-align-right data-alignright>Price/mo for 100 jobs/td>td classhas-text-align-center data-aligncenter>$200/td>td classhas-text-align-center data-aligncenter>$20/td>td classhas-text-align-center data-aligncenter>$34/td>td classhas-text-align-center data-aligncenter>?/td>/tr>tr>td classhas-text-align-right data-alignright>Self-hosting possible/td>td classhas-text-align-center data-aligncenter>❌/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>❌/td>td classhas-text-align-center data-aligncenter>✅/td>/tr>tr>td classhas-text-align-right data-alignright>Timeout-based schedules/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>/tr>tr>td classhas-text-align-right data-alignright>Cron expressions/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>❌/td>td classhas-text-align-center data-aligncenter>✅/td>/tr>tr>td classhas-text-align-right data-alignright>“start” and “fail” signals/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>❌/td>td classhas-text-align-center data-aligncenter>✅/td>/tr>tr>td classhas-text-align-right data-alignright>Check-in via email/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>❌/td>td classhas-text-align-center data-aligncenter>❌/td>/tr>tr>td classhas-text-align-right data-alignright>Auto-provisioning/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>❌/td>td classhas-text-align-center data-aligncenter>✅/td>/tr>tr>td classhas-text-align-right data-alignright>Client SDKs/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>❌/td>td classhas-text-align-center data-aligncenter>❌/td>td classhas-text-align-center data-aligncenter>✅/td>/tr>tr>td classhas-text-align-right data-alignright>API/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>/tr>tr>td classhas-text-align-right data-alignright>Projects/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>❌/td>td classhas-text-align-center data-aligncenter>✅/td>/tr>tr>td classhas-text-align-right data-alignright>Team access/td>td classhas-text-align-center data-aligncenter>💰/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>💰/td>td classhas-text-align-center data-aligncenter>✅/td>/tr>tr>td classhas-text-align-right data-alignright>Single sign-on/td>td classhas-text-align-center data-aligncenter>💰/td>td classhas-text-align-center data-aligncenter>❌/td>td classhas-text-align-center data-aligncenter>❌/td>td classhas-text-align-center data-aligncenter>💰/td>/tr>tr>td classhas-text-align-right data-alignright>Two-factor authentication/td>td classhas-text-align-center data-aligncenter>❌/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>/tr>tr>td classhas-text-align-right data-alignright>Notify via email/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>/tr>tr>td classhas-text-align-right data-alignright>Notify via webhooks/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>💰/td>td classhas-text-align-center data-aligncenter>✅/td>/tr>tr>td classhas-text-align-right data-alignright>Notify via Slack/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>💰/td>td classhas-text-align-center data-aligncenter>💰/td>/tr>tr>td classhas-text-align-right data-alignright>Notify via Telegram/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>✅/td>td classhas-text-align-center data-aligncenter>💰/td>td classhas-text-align-center data-aligncenter>❌/td>/tr>tr>td classhas-text-align-right data-alignright>Notify via SMS/td>td classhas-text-align-center data-aligncenter>💰/td>td classhas-text-align-center data-aligncenter>💰/td>td classhas-text-align-center data-aligncenter>💰/td>td classhas-text-align-center data-aligncenter>💰/td>/tr>/tbody>/table>/figure>h3 classwp-block-heading>In Closing/h3>p>If you notice any factual errors, please let me know (a hrefhttps://healthchecks.io/about/>contacts/a>), and I will get them fixed ASAP!/p>p>There are many more things to compare. If you are shopping for a cron monitoring service, you will have to decide what is important for you, and likely do some additional research./p>p>Happy monitoring,br>– Pēteris/p>/div>/div>/div>/article>article classpost-1402 post type-post status-publish format-standard has-post-thumbnail hentry category-uncategorized ast-col-sm-12 ast-article-post idpost-1402 itemtypehttps://schema.org/CreativeWork itemscopeitemscope>div classast-post-format- blog-layout-1>div classpost-content ast-col-md-12>header classentry-header>h2 classentry-title itempropheadline>a href/2023/08/notes-on-self-hosted-transactional-email/ relbookmark>Notes on Self-hosted Transactional Email/a>/h2> div classentry-meta>By span classposted-by vcard author itemtypehttps://schema.org/Person itemscopeitemscope itempropauthor> a titleView all posts by Pēteris Caune href/author/cuu508/ relauthor classurl fn n itempropurl>span classauthor-name itempropname>Pēteris Caune/span>/a>/span>/ span classposted-on>span classpublished itempropdatePublished> August 9, 2023/span>span classupdated itempropdateModified> August 9, 2023/span>/span>/div> /header>div classentry-content clear itemproptext>p>Since a little more than two months ago, Healthchecks.io has been sending transactional email (~300’000 emails per month) through its own SMTP server. Here are my notes on setting it up./p>h3 classwp-block-heading>The Before/h3>p>Before going self-hosted, Healthchecks sent email using 3rd-party SMTP relays: AWS SES and later a hrefhttps://elasticemail.com/>Elastic Email/a>./p>p>The reason for switching from AWS to Elastic Email was GDPR compliance: at the time United States did not have an EU adequacy decision, but Canada (the registration country of Elastic Email Inc.) did./p>p>The primary reason I kept looking for alternatives to Elastic Email was also GDPR compliance: a country with EU adequacy decision is good, but being based in the EU is even better. Another reason was their poor communication during service outages: some outages were not acknowledged on their status page, there were no timely updates via support chat or otherwise, and there were no post-mortems published after outages. To their credit, Elastic Email did fix the outages reasonably quickly, and I was overall happy with the service in terms of functionality and pricing./p>h3 classwp-block-heading>The EU-based SMTP Relay Options/h3>p>There are few EU-based SMTP relay services. None of the big names (AWS SES, Sendgrid, Mailgun, Mailchimp, Postmark) are EU-based. I tested a few options:/p>ul>li>a hrefhttps://emaillabs.io/en/>EmailLabs/a>: OK in terms of functionality and pricing. Judging by the mix of Polish and English in the user interface and documentation seemed geared primarily to the Polish market./li>li>a hrefhttps://www.smtpeter.com/en>SMTPeter/a>: OK in terms of functionality and pricing. It was probably just bad timing, but had a major outage while I was testing it. Small shop./li>li>a hrefhttps://www.brevo.com/>Brevo/a> (formerly Sendinblue): the most prominent EU SMTP relay service. a hrefhttps://toot.lv/@cuu508/110371325259048432>Has open and click tracking enabled by default/a>, and refused to turn it off before seeing live production traffic, so a non-starter for me./li>/ul>p>None of the options seemed like an upgrade over what I already had, and I kept circling back to the idea of self-hosting. The common wisdom is that self-hosting email means endless deliverability problems, but maybe-maybe?/p>h3 classwp-block-heading>The Self-Hosting Options/h3>p>In May 2023, I spent several weeks researching and trialing self-hosted SMTP servers: a hrefhttps://github.com/mjl-/mox>mox/a>, a hrefhttps://github.com/postalserver/postal>Postal/a>, a hrefhttps://github.com/haraka/Haraka>Haraka/a>, a hrefhttps://github.com/zone-eu/zone-mta>Zone MTA/a>, a hrefhttps://github.com/OpenSMTPD/OpenSMTPD>OpenSMTPD/a>, and a hrefhttps://github.com/foxcpp/maddy>maddy/a>. My brain was getting fried from jumping between documentation sites, trying to make sense of the feature sets, and the pros and cons of each project. One thing that helped immensely was reading a hrefhttps://explained-from-first-principles.com/email/>Email explained from first principles/a> – it filled many gaps in my knowledge of email delivery./p>h3 classwp-block-heading>Maddy/h3>p>After experimenting with and strongly considering OpenSMTPD, I ultimately picked maddy. I iterated on a test configuration until I got it to do the required things:/p>ul>li>Accept email on port 465 from authenticated users./li>li>Rewrite its envelope sender from “@healthchecks.io” to “@mail.healthchecks.io” (required for routing bounce messages back to the maddy server)./li>li>Sign it using DKIM protocol./li>li>Put outgoing messages in a queue, attempt to deliver them, and retry with exponential backoff./li>li>Deliver messages to remote MTAs from a single, specific IP./li>li>When delivery fails, send a webhook notification to a designated webhook handler. For permanent failures, the handler can take appropriate action – unsubscribe a specific user from email reports, or mark a specific email integration as disabled./li>li>Listen for incoming email on port 25./li>li>When a remote MTA sends a DSN (delivery status notification, “bounce message”), deliver it to the same webhook./li>li>Use an automatically provisioned LetsEncrypt certificate for TLS encryption on port 465 and port 25./li>/ul>p>I wrote the provisioning scripts for deploying Maddy and its configuration to a server. I added and updated the required DNS entries for SPF and DKIM. I implemented, tested, and deployed the webhook handler that would receive bounce notifications from maddy./p>h3 classwp-block-heading>IP Warm-Up/h3>p>I spent several weeks gradually switching outgoing email traffic from Elastic Email to the self-hosted maddy server. IP warm-up serves two purposes:/p>ul>li>Slowly builds up the reputation of the sending IP address. Switching the entire sending volume to a new IP address all at once risks getting blocked by the receiving servers./li>li>It lets me test email delivery in the production environment and fix any potential problems with fewer negative consequences./li>/ul>h3 classwp-block-heading>The Failover IP Oopsie/h3>p>One issue I discovered during the IP warm-up phase was that the brand-new mail server (a Hetzner AX41 dedicated server) experienced minute-long network hiccups a couple of times per day. The cause could be a faulty NIC, a faulty switch, or a noisy neighbor, and the easiest fix is ordering another server and hoping for better luck. In anticipation of such a scenario, I had ordered a hrefhttps://docs.hetzner.com/robot/dedicated-server/ip/failover/>a failover IP/a> so I could keep using the already warmed-up IP with the new server./p>p>I set up a new server, switched the failover IP to it, and after a few days of testing, no more network hiccups! So I went ahead and canceled the original machine. Then, a few days later, around 2 AM local time, my monitoring notifications went off: email delivery was broken. I had assumed that “failover IP” is more or less what other providers call “floating IP.” Dazed and confused in front of a blue screen in the middle of the night, I realized my misunderstanding and mistake: the failover IP is owned by a specific server. Canceling the server also cancels the failover IP with all its sender reputation./p>p>To fix the immediate problem, I temporarily switched the web servers back to using Elastic Email as the SMTP relay. I asked Hetzner support if there was any way I could get the released IP back. Minutes later, I got a reply stating in perfect German calmness that my request would need to be handled by a different department during business hours. The next morning, Hetzner added the lost failover IP back to my account. Phew!/p>h3 classwp-block-heading>Local Relay-Only MTAs for Reliability/h3>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height606 src/wp-content/uploads/2023/08/mail-1024x606.png alt classwp-image-1404>/figure>p>The internet-facing SMTP server (mail.healthchecks.io) runs on a single machine. Each app server also runs a local maddy instance which accepts outgoing email messages from local clients only, and hands them off to mail.healthchecks.io. If mail.healthchecks.io is unavailable (for example, during server restart), the local maddy instances queue the messages and retry them later./p>h3 classwp-block-heading>Summary, Pros, and Cons/h3>p>The self-hosted maddy server has been handling all Healthchecks.io transactional email for over two months. I am keeping an eye on bounce notifications, the outbound email queue size, and blocklists. So far, there have been no significant deliverability issues–fingers crossed!/p>p>Cons of going self-hosted:/p>ul>li>Self-hosted SMTP server is another service to maintain. It uses up the limited time and mental bandwidth I have./li>li>The inevitable deliverability problems will be my problems./li>li>In the case of maddy, no pre-built graphical management and monitoring dashboard./li>/ul>p>And pros:/p>ul>li>strong>Complete control of subprocessors with access to customer data/strong> (just Hetzner in my case)./li>li>Complete control over server configuration./li>li>Fixed direct costs (as long as a single server can keep up with the sending volume)./li>li>I learned a bunch of new stuff!/li>/ul>p>Thanks for reading,br>Pēteris/p>/div>/div>/div>/article>article classpost-1381 post type-post status-publish format-standard has-post-thumbnail hentry category-uncategorized ast-col-sm-12 ast-article-post idpost-1381 itemtypehttps://schema.org/CreativeWork itemscopeitemscope>div classast-post-format- blog-layout-1>div classpost-content ast-col-md-12>header classentry-header>h2 classentry-title itempropheadline>a href/2023/07/new-feature-check-auto-provisioning/ relbookmark>New Feature: Check Auto-Provisioning/a>/h2> div classentry-meta>By span classposted-by vcard author itemtypehttps://schema.org/Person itemscopeitemscope itempropauthor> a titleView all posts by Pēteris Caune href/author/cuu508/ relauthor classurl fn n itempropurl>span classauthor-name itempropname>Pēteris Caune/span>/a>/span>/ span classposted-on>span classpublished itempropdatePublished> July 3, 2023/span>span classupdated itempropdateModified> July 4, 2023/span>/span>/div> /header>div classentry-content clear itemproptext>p>Healthchecks recently gained a new feature: check auto-provisioning. When you send a ping request to a a hrefhttps://blog.healthchecks.io/2021/09/new-feature-slug-urls/>slug URL/a>, and a check with the specified slug does not exist, Healthchecks can now automatically create the missing check. This feature requires opt-in: to use it, add a code>?create1/code> query parameter to the ping URL./p>p>Here’s check auto-provisioning in action (the code>-I/code> parameter tells curl to send HTTP HEAD requests so that we can see HTTP response status codes easily):/p>pre classwp-block-code>code>$ curl -I https://hc-ping.com/fixme-ping-key/does-not-existHTTP/2 404...$ curl -I https://hc-ping.com/fixme-ping-key/does-not-exist?create1HTTP/2 201...$ curl -I https://hc-ping.com/fixme-ping-key/does-not-exist?create1HTTP/2 200 .../code>/pre>ul>li>The first request returns HTTP 404 (“Not Found”) because a check with a slug code>does-not-exist/code> does, in fact, not yet exist./li>li>The second request has a “?create1” added to the URL to enable auto-provisioning. The server creates a new check and returns HTTP 201 (“Created”)./li>li>The third request is the same as the second, but a matching check now exists. The server accepts the ping and returns HTTP 200 (“OK”)./li>/ul>p>When is this useful? Whenever you are working with a dynamic infrastructure, and want your monitoring clients to be able to register with Healthchecks.io automatically. If you distribute the Ping Key to monitoring clients, each client can pick its own slug (for example, derived from the server’s hostname), construct a ping URL (code>https://hc-ping.com/<ping-key>/<slug-chosen-by-client>?create1/code>), and Healthchecks.io will auto-create a new check on the first ping./p>h3 classwp-block-heading>Auto-Provisioned Checks Use Default Configuration/h3>p>With the current auto-provisioning implementation, clients can create new checks on the fly, but they cannot yet specify the period, the grace time, the enabled integrations, or any other parameters. The new checks will be created with default parameters (period 1 day, grace time 1 hour, all integrations enabled). If you need to change any parameters, you will need to do this either manually from the web dashboard, or from a script that calls a hrefhttps://healthchecks.io/docs/api/>Management API/a>./p>h3 classwp-block-heading>Auto-Provisioning and Account Limits/h3>p>Each account has a specific limit of how many checks it is allowed to create: 20 checks for free accounts; 100 or 1000 checks for paid accounts. To reduce friction and the risk of silent failures, the auto-provisioning functionality strong>is allowed to temporarily exceed the account’s check limit up to two times/strong>. Meaning, if your account is already maxed out, auto-provisioning will still be able to create new checks until you hit two times the limit. If your account goes over the limit, you will start to see warnings in the dashboard and email:/p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height788 src/wp-content/uploads/2023/07/image-1-1024x788.png alt classwp-image-1386>/figure>p>As soon as you get the number of checks in your account below the limit (either by upgrading to higher limits, or by removing unneeded checks), the warning will go away. If you do not resolve the warning for more than a month, you will start seeing an “Account marked for deletion” notice in the dashboard. After another month of inaction, the account will be deleted./p>h3 classwp-block-heading>Slugs and Names Are Now Separate/h3>p>In the initial slug implementation check slugs were tied to check names. Changing a check’s name also updated its slug. With the introduction of auto-provisioning, check names and slugs are now decoupled. You can hand-pick a custom slug for each check. You can also rename a check but keep its existing slug./p>p>The “Name and Tags” dialog has gained a new, editable “Slug” field:/p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height889 src/wp-content/uploads/2023/07/image-2-1024x889.png alt classwp-image-1387>/figure>p>Similarly, the a hrefhttps://healthchecks.io/docs/api/#create-check>Create a Check/a> and a hrefhttps://healthchecks.io/docs/api/#update-check>Update an Existing Check/a> API calls now support a new code>slug/code> field./p>p>Happy monitoring,br>–Pēteris/p>/div>/div>/div>/article>article classpost-1339 post type-post status-publish format-standard hentry category-uncategorized ast-col-sm-12 ast-article-post idpost-1339 itemtypehttps://schema.org/CreativeWork itemscopeitemscope>div classast-post-format- ast-no-thumb blog-layout-1>div classpost-content ast-col-md-12>header classentry-header>h2 classentry-title itempropheadline>a href/2023/05/walk-through-set-up-self-hosted-healthchecks-instance-on-a-vps/ relbookmark>Walk-through: Set Up Self-Hosted Healthchecks Instance on a VPS/a>/h2> div classentry-meta>By span classposted-by vcard author itemtypehttps://schema.org/Person itemscopeitemscope itempropauthor> a titleView all posts by Pēteris Caune href/author/cuu508/ relauthor classurl fn n itempropurl>span classauthor-name itempropname>Pēteris Caune/span>/a>/span>/ span classposted-on>span classpublished itempropdatePublished> May 15, 2023/span>span classupdated itempropdateModified> May 15, 2023/span>/span>/div> /header>div classentry-content clear itemproptext>p>In this guide, I will deploy a Healthchecks instance on a VPS. Here’s the plan:/p>ul>li>Use a hrefhttps://hub.docker.com/r/healthchecks/healthchecks>the official Docker image/a> and run it using Docker Compose./li>li>Store data in a managed PostgreSQL database./li>li>Use LetsEncrypt certificates initially, and load-balancer-managed certificates later for a HA setup./li>li>Use an external SMTP relay for sending email./li>/ul>p>Prerequisites:/p>ul>li>A domain name (and access to its DNS settings)./li>li>A payment card (for setting up a hosting account)/li>li>Working SMTP credentials for sending emails./li>/ul>h3 classwp-block-heading>Hosting Setup/h3>p>For this exercise, I’m using a hrefhttps://upcloud.com/>UpCloud/a> as the hosting provider. I’m choosing UpCloud because they are a European cloud hosting provider that I have not used before, and they offer managed databases. /p>p>I registered for an account, deposited €10, and launched the cheapest server they offer (1 core, 1GB RAM, €7/mo) with Ubuntu 22.04 as the OS. On the new server, I:/p>ul>li>Installed OS updates (code>apt update && apt upgrade/code>)./li>li>Disabled SSH password authentication/li>li>Installed Docker by following a hrefhttps://docs.docker.com/engine/install/ubuntu/>the official instructions/a>./li>li>Created a non-root user, set up SSH authentication for it, and added it to the “docker” group./li>/ul>h3 classwp-block-heading>Basic docker-compose.yml/h3>p>On the server, logged in as the non-root user, I created a code>docker-compose.yml/code> file with the following contents:/p>pre classwp-block-code>code>version: "3"services: web: image: healthchecks/healthchecks:v2.8.1 restart: unless-stopped environment: - DB_NAME/tmp/hc.sqlite/code>/pre>p>I then ran code>docker compose up/code>. The Healthchecks container started up, but I could not access it from the browser yet: it does not expose any ports, it has no domain name, and there is no TLS terminating proxy yet./p>h3 classwp-block-heading>Add DNS records, Add caddy, Add ALLOWED_HOSTS, SITE_ROOT/h3>p>I own a domain name “monkeyseemonkeydo.lv”, and for this Healthchecks instance I used the subdomain “hc.monkeyseemonkeydo.lv”. I created two new DNS records:/p>pre classwp-block-code>code>hc.monkeyseemonkeydo.lv A 94.237.80.66hc.monkeyseemonkeydo.lv AAAA 2a04:3542:1000:910:80a5:5cff:fe7f:0a17/code>/pre>p>(These are of course the IPv4 and IPv6 addresses of the UpCloud server)./p>p>In code>docker-compose.yml/code> I added a new “caddy” service to act as a TLS terminating reverse proxy, and I added code>ALLOWED_HOSTS/code> and code>SITE_ROOT/code> environment variables in the “web” service:/p>pre classwp-block-code>code>version: "3"services: caddy: image: caddy:2.6.4 restart: unless-stopped command: caddy reverse-proxy --from https://hc.monkeyseemonkeydo.lv:443 --to http://web:8000 ports: - 80:80 - 443:443 volumes: - caddy:/data depends_on: - web web: image: healthchecks/healthchecks:v2.8.1 restart: unless-stopped environment: - ALLOWED_HOSTShc.monkeyseemonkeydo.lv - DB_NAME/tmp/hc.sqlite - SITE_ROOThttps://hc.monkeyseemonkeydo.lvvolumes: caddy:/code>/pre>p>Note: Caddy needs a persistent “/data” volume for storing TLS certificates, private keys, OCSP staples, and other information./p>p>After running code>docker compose up/code> again, the site loads in the browser:/p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height708 src/wp-content/uploads/2023/05/image-10-1024x708.png alt classwp-image-1356>/figure>h3 classwp-block-heading>Add DEBUGFalse and SECRET_KEY/h3>p>Next, I added code>DEBUG/code> and code>SECRET_KEY/code> environment variables. code>DEBUGFalse/code> turns off the debug mode, which should always be off on public-facing sites. code>SECRET_KEY/code> is used for cryptographic signing and should be set to a unique, secret value. Do not copy the value I used!/p>pre classwp-block-code>code>environment: ... - DEBUGFalse - SECRET_KEYb553f395-2aa1-421a-bcf5-d1c1456776d7 .../code>/pre>h3 classwp-block-heading>Launch PostgreSQL Database, Add Database Credentials/h3>p>I created a managed PostgreSQL database in the UpCloud account. I selected PostgreSQL 15.1, and the lowest available spec (1 node, 1 core, 2GB RAM, €30/mo). I made sure to select the same datacenter that the web server is in./p>p>After the database server started up, I took note of the connection parameters: host, port, username, password, and database name. Since I was planning to use this database server for the Healthchecks instance and nothing else, I used the default database user (“upadmin”) and the default database (“defaultdb”). Here is the database configuration:/p>pre classwp-block-code>code>environment: ... - DBpostgres - DB_HOSTpostgres-************.db.upclouddatabases.com - DB_PORT11550 - DB_NAMEdefaultdb - DB_USERupadmin - DB_PASSWORDAVNS_******************* .../code>/pre>p>After another code>docker compose up/code>, I created a superuser account:/p>pre classwp-block-code>code>docker compose run web /opt/healthchecks/manage.py createsuperuser/code>/pre>p>I tested the setup by signing in as the superuser:/p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height708 src/wp-content/uploads/2023/05/image-11-1024x708.png alt classwp-image-1358>/figure>h3 classwp-block-heading>Configure Outgoing Email/h3>p>The Healthchecks instance needs valid SMTP credentials for sending email./p>p>For a production site, I would sign up for an SMTP relay service. Since I’m setting this instance up only for demonstration purposes, and the volume of sent emails will be very low, I used my personal mail (hosted by Fastmail) SMTP credentials./p>p>Here are the new environment variables:/p>pre classwp-block-code>code>environment: ... - ADMINSmeow@monkeyseemonkeydo.lv - DEFAULT_FROM_EMAILmeow@monkeyseemonkeydo.lv - EMAIL_HOSTsmtp.fastmail.com - EMAIL_HOST_USERmeow@monkeyseemonkeydo.lv - EMAIL_HOST_PASSWORD**************** .../code>/pre>p>The code>ADMINS/code> setting sets the email addresses that will receive error notifications. The code>DEFAULT_EMAIL_FROM/code> sets the “From:” address for emails from this Healthchecks instance./p>h3 classwp-block-heading>Disable New User Signups/h3>p>The new Healthchecks instance currently allows any visitor to sign up for an account. This will be a private instance, so I disabled new user registration via the code>REGISTRATION_OPEN/code> environment variable:/p>pre classwp-block-code>code>environment: ... - REGISTRATION_OPENFalse .../code>/pre>h3 classwp-block-heading>Add Pinging by Email/h3>p>Healthchecks supports pinging (sending heartbeat messages from clients) via HTTP and also via email. To enable pinging via email, I set the code>PING_EMAIL_DOMAIN/code> and code>SMTPD_PORT/code> environment variables, and exposed port 25:/p>pre classwp-block-code>code>environment: ... - PING_EMAIL_DOMAINhc.monkeyseemonkeydo.lv - SMTPD_PORT25 ...ports: - 25:25 /code>/pre>p>After another code>docker compose up/code>, I sent a test email and verified its arrival:/p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height708 src/wp-content/uploads/2023/05/image-12-1024x708.png alt classwp-image-1360>/figure>h3 classwp-block-heading>Add Logo and Site Name/h3>p>The default logo image is located at code>/opt/healthchecks/static-collected/img/logo.png/code> inside the “web” container. To use a custom logo, one can either set the a hrefhttps://healthchecks.io/docs/self_hosted_configuration/#SITE_LOGO_URL>SITE_LOGO_URL/a> environment variable or mount a custom logo over the default one. I used the latter method./p>p>I used a hrefhttps://github.com/googlefonts/noto-emoji/blob/main/png/128/emoji_u1f408_200d_2b1b.png>an image from the Noto Emoji font/a> as the logo, placed it next to code>docker-compose.yml/code> on the server, and picked a site name:/p>pre classwp-block-code>code>environment: ... - SITE_NAMEMeowOps ...volumes: - $PWD/logo.png:/opt/healthchecks/static-collected/img/logo.png/code>/pre>p>The result:/p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height708 src/wp-content/uploads/2023/05/image-13-1024x708.png alt classwp-image-1362>/figure>h3 classwp-block-heading>The Complete docker-compose.yml/h3>p>Putting it all together, here is the complete code>docker-compose.yml/code>:/p>pre classwp-block-code>code>version: "3"services: caddy: image: caddy:2.6.4 restart: unless-stopped command: caddy reverse-proxy --from https://hc.monkeyseemonkeydo.lv:443 --to http://web:8000 ports: - 80:80 - 443:443 volumes: - caddy:/data depends_on: - web web: image: healthchecks/healthchecks:v2.8.1 restart: unless-stopped environment: - ADMINSmeow@monkeyseemonkeydo.lv - DEBUGFalse - ALLOWED_HOSTShc.monkeyseemonkeydo.lv - DBpostgres - DB_HOSTpostgres-************.db.upclouddatabases.com - DB_PORT11550 - DB_NAMEdefaultdb - DB_USERupadmin - DB_PASSWORDAVNS_******************* - DEFAULT_FROM_EMAILmeow@monkeyseemonkeydo.lv - EMAIL_HOSTsmtp.fastmail.com - EMAIL_HOST_USERmeow@monkeyseemonkeydo.lv - EMAIL_HOST_PASSWORD**************** - PING_EMAIL_DOMAINhc.monkeyseemonkeydo.lv - REGISTRATION_OPENFalse - SECRET_KEYb553f395-2aa1-421a-bcf5-d1c1456776d7 - SITE_NAMEMeowOps - SITE_ROOThttps://hc.monkeyseemonkeydo.lv - SMTPD_PORT25 ports: - 25:25 volumes: - $PWD/logo.png:/opt/healthchecks/static-collected/img/logo.pngvolumes: caddy:/code>/pre>h3 classwp-block-heading>HA/h3>p>With the current setup, the web server and the database are both single points of failure. For a production setup, it would be desirable to have as few single points of failure as possible./p>p>The database part is easy, as UpCloud-managed databases support HA configurations. I changed the database plan from 1 node to 2 HA nodes (2 cores, 4GB RAM, €100/mo) and that was that. I did not even need to restart the web container./p>p>The web server part is more complicated: launch a second web server, put a managed load balancer in front of both web servers, and move TLS termination to the load balancer. I updated code>docker-compose.yml/code> yet again:/p>pre classwp-block-code>code>version: "3"services: web: image: healthchecks/healthchecks:v2.8.1 restart: unless-stopped environment: - ADMINSmeow@monkeyseemonkeydo.lv - DEBUGFalse - DBpostgres - DB_HOSTpostgres-************.db.upclouddatabases.com - DB_PORT11550 - DB_NAMEdefaultdb - DB_USERupadmin - DB_PASSWORDAVNS_******************* - DEFAULT_FROM_EMAILmeow@monkeyseemonkeydo.lv - EMAIL_HOSTsmtp.fastmail.com - EMAIL_HOST_USERmeow@monkeyseemonkeydo.lv - EMAIL_HOST_PASSWORD**************** - PING_EMAIL_DOMAINhc.monkeyseemonkeydo.lv - REGISTRATION_OPENFalse - SECRET_KEYb553f395-2aa1-421a-bcf5-d1c1456776d7 - SITE_NAMEMeowOps - SITE_ROOThttps://hc.monkeyseemonkeydo.lv - SMTPD_PORT25 ports: - 10.0.0.2:8000:8000 - 10.0.0.2:25:25 volumes: - $PWD/logo.png:/opt/healthchecks/static-collected/img/logo.png/code>/pre>ul>li>I removed the “caddy” service since the load balancer will now be terminating TLS./li>li>I removed the ALLOWED_HOSTS setting. This was required to get the load balancer health checks to work (UpCloud’s load balancer does not send the code>Host/code> request header)./li>li>I exposed port 8000 of the “web” service on a private IP that the load balancer will connect through./li>li>I updated the port 25 entry to bind only to the private IP./li>/ul>p>The following steps are UpCloud-specific, not Healthchecks-specific, so I will only summarize them:/p>ul>li>I launched a second web server and set it up identically to the existing one./li>li>I created a managed load balancer (2 HA nodes, €30/mo)./li>li>I replaced the “A” and “AAAA” DNS records for hc.monkeyseemonkeydo.lv with a CNAME record that points to the load balancer’s hostname./li>li>I configured the load balancer to terminate TLS traffic on port 443, add code>X-Forwarded-For/code> request headers, and proxy the HTTP requests to the web servers./li>li>I configured the load balancer to proxy TCP connections on port 25 to port 25 on the web servers./li>/ul>h3 classwp-block-heading>Costs/h3>p>For the single-node setup:/p>ul>li>Web server: €7/mo./li>li>Database: €30/mo./li>li>Total: €37/mo./li>/ul>p>For the HA setup:/p>ul>li>Web servers: 2 × €7/mo./li>li>Database: €100/mo./li>li>Load balancer: €30/mo./li>li>Total: €144/mo./li>/ul>h3 classwp-block-heading>Monitoring, Automation, Documentation/h3>p>At this point, the Healthchecks instance is up and running and the walk-through is complete. For real-world deployment, also consider the following tasks:/p>ul>li>Set up uptime monitoring using your preferred uptime monitoring service./li>li>Set up CPU / RAM / disk / network monitoring using your preferred infrastructure monitoring service./li>li>Set up monitoring for notification delivery./li>li>Move secret values out of code>docker-compose.yml/code>, and store code>docker-compose.yml/code> under version control./li>li>Document the web server setup and update procedures./li>li>Automate the setup and update tasks if and where it makes sense./li>/ul>p>Thanks for reading, and good luck in your self-hosting adventures,br>–Pēteris/p>/div>/div>/div>/article>article classpost-1317 post type-post status-publish format-standard hentry category-uncategorized ast-col-sm-12 ast-article-post idpost-1317 itemtypehttps://schema.org/CreativeWork itemscopeitemscope>div classast-post-format- ast-no-thumb blog-layout-1>div classpost-content ast-col-md-12>header classentry-header>h2 classentry-title itempropheadline>a href/2023/05/monitor-disk-space-on-servers-without-installing-monitoring-agents/ relbookmark>Monitor Disk Space on Servers Without Installing Monitoring Agents/a>/h2> div classentry-meta>By span classposted-by vcard author itemtypehttps://schema.org/Person itemscopeitemscope itempropauthor> a titleView all posts by Pēteris Caune href/author/cuu508/ relauthor classurl fn n itempropurl>span classauthor-name itempropname>Pēteris Caune/span>/a>/span>/ span classposted-on>span classpublished itempropdatePublished> May 8, 2023/span>span classupdated itempropdateModified> May 8, 2023/span>/span>/div> /header>div classentry-content clear itemproptext>p>Let’s say you want to get an email notification when the free disk space on your server drops below some threshold level. There are many ways to go about this, but here is one that does not require you to install anything new on the system and is easy to audit (it’s a 4-line shell script)./p>h3 classwp-block-heading>The code>df/code> Utility/h3>p>code>df/code> is a command-line program that reports file system disk space usage, and is usually preinstalled on Unix-like systems. Let’s run it:/p>pre classwp-block-code>code>$ df -h /Filesystem Size Used Avail Use% Mounted on/dev/mapper/ubuntu--vg-ubuntu--lv 75G 23G 51G 32% //code>/pre>p>The “-h” argument tells code>df/code> to print sizes in human-readable units. The “/” argument tells code>df/code> to only output stats about the root filesystem. The “Use%” field in the output indicates the root filesystem is 32% full. If we wanted to extract just the percentage, code>df/code> has a handy “–output” argument:/p>pre classwp-block-code>code>$ df --outputpcent /Use% 32%/code>/pre>p>We can use code>tail/code> to drop the first line, and code>tr/code> to delete the space and percent-sign characters, leaving just the numeric value:/p>pre classwp-block-code>code>$ df --outputpcent / | tail -n 1 | tr -d '% '32/code>/pre>h3 classwp-block-heading>The Disk Space Monitoring Script/h3>p>Here is a shell script that looks up the free disk space on the root filesystem, compares it to a defined threshold value (75 in this example), then does some action depending on the result:/p>pre classwp-block-code>code>pct$(df --outputpcent / | tail -n 1 | tr -d '% ')if $pct -gt 75 ; then // FIXME: the command to run when above the threshold else // FIXME: the command to run when below the thresholdfi/code>/pre>p>We can save this as a shell script, and run it from cron at regular intervals. Except the script does not yet handle the alerting part of course. Some things to consider:/p>ul>li>Sending emails directly from servers is a little fiddly: you need to a hrefhttps://blog.healthchecks.io/2021/11/how-to-send-email-from-cron-jobs/>install an MTA and configure it with working SMTP credentials/a>./li>li>Rather than sending a notification after every check it would be much better to send a notification only when the available disk space crosses the threshold value./li>/ul>h3 classwp-block-heading>Healthchecks.io/h3>p>a hrefhttps://healthchecks.io>Healthchecks.io/a>, a cron job monitoring service, can help with the alerting part:/p>ul>li>You can send monitoring signals to Healthchecks.io via HTTP requests using curl or wget./li>li>Healthchecks.io handles the email delivery (as well as Slack, Telegram, Pushover, and many other options)./li>li>Healthchecks.io sends notifications only on state changes – when something breaks or recovers. It will not spam you with ongoing reminders unless you tell it to./li>li>It will also detect when your monitoring script goes AWOL. For example, when the whole system crashes or loses the network connection./li>/ul>p>In your Healthchecks.io account, create a new Check, give it a descriptive name, set its Period to “10 minutes”, and copy its Ping URL./p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height372 src/wp-content/uploads/2023/05/image-3-1024x372.png alt classwp-image-1328>/figure>p>The monitoring API is super-simple. To signal success (disk usage is below threshold), send an HTTP request to the Ping URL directly:/p>pre classwp-block-code>code>curl https://hc-ping.com/your-uuid-here/code>/pre>p>And, to signal failure, append “/fail” at the end of the Ping URL:/p>pre classwp-block-code>code>curl https://hc-ping.com/your-uuid-here/fail/code>/pre>p>Let’s integrate this into our monitoring script:/p>pre classwp-block-code>code>urlhttps://hc-ping.com/your-uuid-herepct$(df --outputpcent / | tail -n 1 | tr -d '% ')if $pct -gt 75 ; then url$url/fail; ficurl -fsS -m 10 --retry 5 -o /dev/null --data-raw "Used space on /: $pct%" $url/code>/pre>p>The curl call here has a few extra arguments:/p>ul>li>“-fsS” tells curl to suppress output except for error messages/li>li>“-m 10” sets a 10-second timeout for HTTP requests/li>li>“–retry 5” tells curl to retry failed requests up to 5 times/li>li>“-o /dev/null” sends the server’s response to code>/dev/null/code>/li>li>“–data-raw …” specifies a log message to include in an HTTP POST request body/li>/ul>p>Save this script in a convenient location, for example, in code>/opt/check-disk-space.sh/code>, and make it executable. Then edit crontab (code>crontab -e/code>) and add a new cron job:/p>pre classwp-block-code>code>*/10 * * * * /opt/check-disk-space.sh/code>/pre>p>Cron will run the script every 10 minutes. On every run, the script will check the used disk space, and signal success (disk usage below or at threshold) or failure (disk usage above threshold) to Healthchecks.io. Whenever the status value flips, Healthchecks.io will send you a notification:/p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height892 src/wp-content/uploads/2023/05/image-4-1024x892.png alt classwp-image-1332>/figure>p>You will also see a log of the monitoring script’s check-ins in the Healthchecks.io web interface:/p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height971 src/wp-content/uploads/2023/05/image-5-1024x971.png alt classwp-image-1334>/figure>h3 classwp-block-heading>Closing Comments/h3>p>If your use case involves handling millions of small files, at least on ext4 filesystems, the filesystem can also run out of inodes. Run code>df -i/code> to see how many inodes are in use and how many are available. If inode use is a potential concern, you could update the code>check-disk-space.sh/code> script to track it too. /p>p>The shell script + Healthchecks.io pattern would work for monitoring other system metrics too. For example, you could have a script that checks the system’s 15-minute load average, the number of files in a specific directory, or a temperature sensor’s reading./p>p>If you are looking to monitor more than a couple of system metrics though, look into purpose-built system monitoring tools such as a hrefhttps://www.netdata.cloud/>netdata/a>. The shell script + Healthchecks.io approach works best when you have a few specific metrics you care about, and you want to avoid installing full-blown monitoring agents in your system./p>p>Thanks for reading and happy monitoring,br>–Pēteris./p>/div>/div>/div>/article>article classpost-1276 post type-post status-publish format-standard has-post-thumbnail hentry category-uncategorized ast-col-sm-12 ast-article-post idpost-1276 itemtypehttps://schema.org/CreativeWork itemscopeitemscope>div classast-post-format- blog-layout-1>div classpost-content ast-col-md-12>header classentry-header>h2 classentry-title itempropheadline>a href/2023/03/making-http-requests-with-arduino-and-esp8266/ relbookmark>Making HTTP requests with Arduino and ESP8266/a>/h2> div classentry-meta>By span classposted-by vcard author itemtypehttps://schema.org/Person itemscopeitemscope itempropauthor> a titleView all posts by Pēteris Caune href/author/cuu508/ relauthor classurl fn n itempropurl>span classauthor-name itempropname>Pēteris Caune/span>/a>/span>/ span classposted-on>span classpublished itempropdatePublished> March 30, 2023/span>span classupdated itempropdateModified> April 3, 2023/span>/span>/div> /header>div classentry-content clear itemproptext>p>A Healthchecks user sent me a code snippet for sending HTTP pings from Arduino. This prompted me to do some Arduino experimenting on my own. I ordered Arduino Nano 33 IoT board:/p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height663 src/wp-content/uploads/2023/03/nano-1024x663.jpg alt classwp-image-1278>figcaption classwp-element-caption>Arduino Nano 33 IoT/figcaption>/figure>p>I picked this board because I wanted an easy entry into Arduino development. As a first-party Arduino hardware, it should be easy to get it working with Arduino IDE. It has an on-board WIFI chip, so I would not need to hook up additional WiFi or Ethernet hardware./p>p>The Nano 33 IoT has a micro USB port. After connecting to my PC running Ubuntu, Arduino’s power LED lit up, and on the computer side a code>/dev/ttyACM0/code> device appeared. Arduino IDE detected the connected board, but my initial attempt to upload a sketch failed. This turned out to be a permissions issue. After I added my OS user to the code>dialout/code> group, I could upload a “Hello World” sketch to the board:/p>figure classwp-block-image alignleft size-large>img loadinglazy decodingasync width1024 height793 src/wp-content/uploads/2023/03/hello_world-5-1024x793.png alt classwp-image-1299>/figure>p>/p>h3 classwp-block-heading>Sending a Raw HTTP Request/h3>p>Arduino Nano 33 IoT has an on-board WiFi module. To use it, Arduino provides the WiFiNINA library. The library comes with example code snippets. a hrefhttps://docs.arduino.cc/tutorials/communication/wifi-nina-examples#wifinina-wifi-web-client>One of the examples/a> shows how to connect to a WiFi network and make an HTTP request. I adapted it to make an HTTPS request to hc-ping.com:/p>pre classwp-block-code>code>#include <WiFiNINA.h>#include "arduino_secrets.h"char ssid SECRET_SSID;char pass SECRET_PASS; int status WL_IDLE_STATUS; WiFiSSLClient client;void setup() { pinMode(LED_BUILTIN, OUTPUT); Serial.begin(9600); while (!Serial); Serial.print("Connecting ..."); WiFi.begin(ssid, pass); while (WiFi.status() ! WL_CONNECTED) { delay(500); Serial.print("."); } Serial.print("\nConnected, IP address: "); Serial.println(WiFi.localIP()); }void ping() { Serial.println("Pinging hc-ping.com..."); if (client.connect("hc-ping.com", 443)) { Serial.println("Connected to server."); client.println("GET /da840100-3f58-405e-a5ee-e7e6e4303e82 HTTP/1.0"); client.println("Host: hc-ping.com"); client.println("Connection: close"); client.println(); Serial.println("Request sent."); } while (client.connected()) { while (client.available()) { char c client.read(); Serial.write(c); } } Serial.println("\nClosing connection."); client.stop();}void loop() { ping(); // Blink LED for 10 seconds: Serial.print("Waiting 10s: "); for (int i0; i<10; i++) { Serial.print("."); digitalWrite(LED_BUILTIN, HIGH); delay(500); digitalWrite(LED_BUILTIN, LOW); delay(500); } Serial.println();}/code>/pre>p>After uploading this sketch to Arduino, here’s the output on serial console:/p>pre classwp-block-code>code>Connecting ...Connected, IP address: 192.168.1.77Pinging hc-ping.com...Connected to server.Request sent.HTTP/1.1 200 OKserver: nginxdate: Thu, 30 Mar 2023 12:33:25 GMTcontent-type: text/plain; charsetutf-8content-length: 2access-control-allow-origin: *ping-body-limit: 100000connection: closeOKClosing connection.Waiting 10s: ..........Pinging hc-ping.com...Connected to server.Request sent.HTTP/1.1 200 OKserver: nginxdate: Thu, 30 Mar 2023 12:33:41 GMTcontent-type: text/plain; charsetutf-8content-length: 2access-control-allow-origin: *ping-body-limit: 100000connection: closeOKClosing connection.Waiting 10s: ........../code>/pre>p>Quite impressively, this works over HTTPS out of the box – the WiFiNINA library and the chip takes care of performing TLS handshake and verifying the certificates. All I had to do was specify port 443, and the rest was handled automagically./p>h3 classwp-block-heading>ArduinoHttpClient/h3>p>After getting the minimal example working, I found the a hrefhttps://github.com/arduino-libraries/ArduinoHttpClient>ArduinoHttpClient/a> library. It offers a higher-level interface for making GET and POST requests, and for parsing server responses. It works with several different network libraries, including WifiNINA./p>pre classwp-block-code>code>#include <ArduinoHttpClient.h>#include <WiFiNINA.h>#include "arduino_secrets.h"char ssid SECRET_SSID;char pass SECRET_PASS; int status WL_IDLE_STATUS; WiFiSSLClient wifi;char host "hc-ping.com";char uuid UUID;HttpClient client HttpClient(wifi, host, 443);void setup() { pinMode(LED_BUILTIN, OUTPUT); Serial.begin(9600); while (!Serial); Serial.print("Connecting ..."); WiFi.begin(ssid, pass); while (WiFi.status() ! WL_CONNECTED) { delay(500); Serial.print("."); } Serial.print("\nConnected, IP address: "); Serial.println(WiFi.localIP()); }void loop() { client.get("/" + String(uuid)); Serial.print("Status code: "); Serial.println(client.responseStatusCode()); Serial.print("Response: "); Serial.println(client.responseBody()); // Blink LED for 10 seconds: Serial.print("Waiting 10s: "); for (int i0; i<10; i++) { Serial.print("."); digitalWrite(LED_BUILTIN, HIGH); delay(500); digitalWrite(LED_BUILTIN, LOW); delay(500); } Serial.println();}/code>/pre>p>Output in the serial console:/p>pre classwp-block-code>code>Connecting ...Connected, IP address: 192.168.1.77Status code: 200Response: OKWaiting 10s: ..........Status code: 200Response: OKWaiting 10s: ............./code>/pre>h3 classwp-block-heading>ESP8266/h3>p>After having good results with Arduino Nano 33 IoT, I wanted to try the same on an ESP8266 board I had lying around:/p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height783 src/wp-content/uploads/2023/03/esp-1024x783.jpg alt classwp-image-1281>figcaption classwp-element-caption>ESP8266 on a carrier board/figcaption>/figure>p>This board from AliExpress has a few goodies in addition to the ESP8266 chip: a relay, and multiple powering options: 220V AC, 7-12V DC, 5V DC. It has a USB port, but this port can be used for supplying power only, there is no USB-UART interface onboard. There are clearly labeled GND, 5V, RX, TX pins that I can hook a USB-UART converter (also from AliExpress) to:/p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height934 src/wp-content/uploads/2023/03/esp-usb-1024x934.jpg alt classwp-image-1283>figcaption classwp-element-caption>ESP8266 with a USB-serial converter hooked up/figcaption>/figure>p>The yellow jumper connects GPIO 0 to the ground, this puts ESP8266 in programming mode. At this point I can plug the USB-UART converter in the PC and check for signs of life using code>esptool/code>:/p>pre classwp-block-code>code>$ apt-get install esptool$ esptool chip_idesptool.py v2.8Found 2 serial portsSerial port /dev/ttyUSB0Connecting...Detecting chip type... ESP8266Chip is ESP8266EXFeatures: WiFiCrystal is 26MHzMAC: a8:48:fa:ff:15:45Enabling default SPI flash mode...Chip ID: 0x00ff1545Hard resetting via RTS pin.../code>/pre>p>Arduino IDE does not support ESP8266 chips out of the box, but there is a hrefhttps://github.com/esp8266/Arduino>esp8266/Arduino/a> project which adds support for different flavors of ESP boards./p>figure classwp-block-image size-large>img loadinglazy decodingasync width1024 height572 src/wp-content/uploads/2023/03/arduino-esp-2-1024x572.png alt classwp-image-1298>figcaption classwp-element-caption>esp8266 library in Arduino IDE’s Board Manager view/figcaption>/figure>p>The esp8266/Arduino project also a hrefhttps://arduino-esp8266.readthedocs.io/en/latest/esp8266wifi/readme.html>comes with a WiFi library/a>, which provides an interface to the WiFi functionality on the chip. For simple use cases, the esp8266wifi library is a drop-in replacement for the WiFiNINA library:/p>pre classwp-block-code>code>#include <ArduinoHttpClient.h>#include <ESP8266WiFi.h>#include "arduino_secrets.h"char ssid SECRET_SSID;char pass SECRET_PASS; WiFiClient wifi;char host "hc-ping.com";char uuid UUID;HttpClient client HttpClient(wifi, host, 80);void setup() { pinMode(LED_BUILTIN, OUTPUT); Serial.begin(115200); while (!Serial); Serial.println(); WiFi.begin(ssid, pass); Serial.print("Connecting ..."); while (WiFi.status() ! WL_CONNECTED) { delay(500); Serial.print("."); } Serial.print("\nConnected, IP address: "); Serial.println(WiFi.localIP());}void loop() { client.get("/" + String(uuid)); Serial.print("Status code: "); Serial.println(client.responseStatusCode()); Serial.print("Response: "); Serial.println(client.responseBody()); // Blink LED for 10 seconds: Serial.print("Waiting 10s: "); for (int i0; i<10; i++) { Serial.print("."); digitalWrite(LED_BUILTIN, HIGH); delay(500); digitalWrite(LED_BUILTIN, LOW); delay(500); } Serial.println(); }/code>/pre>p>Although the esp8266wifi library a hrefhttps://arduino-esp8266.readthedocs.io/en/latest/esp8266wifi/bearssl-client-secure-class.html>does support TLS/a>, the documentation also mentions significant CPU and memory requirements. To keep things simple and quick, I went with port 80 and unencrypted HTTP for this experiment./p>p>I uploaded the sketch, removed the yellow jumper, reset the board, and got this output on the serial console:/p>pre classwp-block-code>code>Connecting ..........Connected, IP address: 192.168.1.78Status code: 200Response: OKWaiting 10s: ..........Status code: 200Response: OKWaiting 10s: ............./code>/pre>p>Success!/p>p>In summary, my first steps in Arduino development left me with positive impressions. The network libraries provide an easy to use, high-level interface for working with network hardware. They have uniform interfaces, so can be used in sketches interchangeably, with minimal code changes. After the initial hump of getting a board recognized by Arduino IDE, and getting the first sketch to upload and run, the development went smoothly. To be fair, the “development” in my case was mostly copying and tweaking code samples. But it was still good!/p>p>Happy tinkering,br>–Pēteris/p>/div>/div>/div>/article>/div> /main>div classast-pagination>nav classnavigation pagination rolenavigation aria-labelPosts>span classscreen-reader-text>Posts navigation/span>div classnav-links>span aria-currentpage classpage-numbers current>1/span>a classpage-numbers href/page/2/>2/a>span classpage-numbers dots>…/span>a classpage-numbers href/page/6/>6/a>a classnext page-numbers href/page/2/>Next Page span classast-right-arrow>→/span>/a>/div>/nav>/div> /div>/div>/div>footer classsite-footer idcolophon itemtypehttps://schema.org/WPFooter itemscopeitemscope itemid#colophon>div classast-small-footer footer-sml-layout-1>div classast-footer-overlay>div classast-container>div classast-small-footer-wrap>div classast-small-footer-section ast-small-footer-section-1>div classast-footer-widget-1-area>aside idcustom_html-3 classwidget_text widget widget_custom_html>div classtextwidget custom-html-widget>div stylebackground: #eee; padding: 2.5em 2.5em>h2 stylemargin-bottom: 0.5em>Get alerted when your cron jobs don't run on time/h2>p stylecolor:#333>Healthchecks.io is a free and open-source cron monitoring service. Setting up monitoring for your cron jobs only takes minutes. Start sleeping better at nights!/p>div styletext-align: center>a styledisplay: inline-block; color: #fff; background: #22bc66; font-weight: bold; padding: 0.4em 1.3em; border-radius: 3px hrefhttps://healthchecks.io>Get Started/a>/div>/div>/div>/aside>/div> /div>div classast-small-footer-section ast-small-footer-section-2>Copyright © 2024 span classast-footer-site-title>Healthchecks.io/span>a href/feed/ stylefloat: right; color: #0091ea >RSS feed/a> /div>/div>/div>/div>/div>/footer>/div>script idastra-theme-js-js-extra>var astra{break_point:921,isRtl:};/script>script src/wp-content/themes/astra/assets/js/minified/style.min.js idastra-theme-js-js>/script>script idhardypress_search-js-extra>var hardypressSearch{searchUrl:https:\/\/api.hardypress.com\/dawn-cloud-562};/script>script src/wp-content/plugins/hardypress/search.js idhardypress_search-js>/script>script>/(trident|msie)/i.test(navigator.userAgent)&&document.getElementById&&window.addEventListener&&window.addEventListener(hashchange,function(){var t,elocation.hash.substring(1);/^A-z0-9_-+$/.test(e)&&(tdocument.getElementById(e))&&(/^(?:a|select|input|button|textarea)$/i.test(t.tagName)||(t.tabIndex-1),t.focus())},!1);/script>/body>/html>
View on OTX
|
View on ThreatMiner
Please enable JavaScript to view the
comments powered by Disqus.
Data with thanks to
AlienVault OTX
,
VirusTotal
,
Malwr
and
others
. [
Sitemap
]