Help
RSS
API
Feed
Maltego
Contact
Domain > blog.sunshineonacloudy.net
×
More information on this domain is in
AlienVault OTX
Is this malicious?
Yes
No
DNS Resolutions
Date
IP Address
2018-01-07
172.217.19.243
(
ClassC
)
2025-01-11
142.251.215.243
(
ClassC
)
Port 80
HTTP/1.1 200 OKContent-Type: text/html; charsetUTF-8Expires: Sat, 11 Jan 2025 19:05:15 GMTDate: Sat, 11 Jan 2025 19:05:15 GMTCache-Control: private, max-age0Last-Modified: Wed, 28 Aug 2024 14:09:54 GMTX-Content-Type-Options: nosniffX-XSS-Protection: 1; modeblockServer: GSEAccept-Ranges: noneVary: Accept-EncodingTransfer-Encoding: chunked !DOCTYPE html>html classv2 dirltr langen>head>link hrefhttps://www.blogger.com/static/v1/widgets/3566091532-css_bundle_v2.css relstylesheet typetext/css/>meta contentwidth1100 nameviewport/>meta contenttext/html; charsetUTF-8 http-equivContent-Type/>meta contentblogger namegenerator/>link hrefhttp://blog.sunshineonacloudy.net/favicon.ico relicon typeimage/x-icon/>link hrefhttp://blog.sunshineonacloudy.net/ relcanonical/>link relalternate typeapplication/atom+xml titleSunshine on a Cloudy Net - Atom hrefhttp://blog.sunshineonacloudy.net/feeds/posts/default />link relalternate typeapplication/rss+xml titleSunshine on a Cloudy Net - RSS hrefhttp://blog.sunshineonacloudy.net/feeds/posts/default?altrss />link relservice.post typeapplication/atom+xml titleSunshine on a Cloudy Net - Atom hrefhttps://www.blogger.com/feeds/6220320961972054840/posts/default />link relme hrefhttps://www.blogger.com/profile/09177647876962492314 />!--Cant find substitution for tag blog.ieCssRetrofitLinks-->meta contentCloud computing and data science from an Economist's perspective. namedescription/>meta contenthttp://blog.sunshineonacloudy.net/ propertyog:url/>meta contentSunshine on a Cloudy Net propertyog:title/>meta contentCloud computing and data science from an Economist's perspective. propertyog:description/>title>Sunshine on a Cloudy Net/title>style typetext/css>@font-face{font-family:Corsiva;font-style:normal;font-weight:400;font-display:swap;src:url(//fonts.gstatic.com/l/font?kitzOL84pjBmb1Z8oKb8rCr7A&skeyb9a50c359455acd3&vv19)format(woff2);unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD;}/style>style idpage-skin-1 typetext/css>!--/*-----------------------------------------------Blogger Template StyleName: SimpleDesigner: BloggerURL: www.blogger.com----------------------------------------------- *//* Content----------------------------------------------- */body {font: normal normal 14px Arial, Tahoma, Helvetica, FreeSans, sans-serif;color: #cccccc;background: #292929 none repeat scroll top left;padding: 0 40px 40px 40px;}html body .region-inner {min-width: 0;max-width: 100%;width: auto;}h2 {font-size: 22px;}a:link {text-decoration:none;color: #dd7700;}a:visited {text-decoration:none;color: #cc6600;}a:hover {text-decoration:underline;color: #cc6600;}.body-fauxcolumn-outer .fauxcolumn-inner {background: transparent none repeat scroll top left;_background-image: none;}.body-fauxcolumn-outer .cap-top {position: absolute;z-index: 1;height: 400px;width: 100%;}.body-fauxcolumn-outer .cap-top .cap-left {width: 100%;background: transparent none repeat-x scroll top left;_background-image: none;}.content-outer {-moz-box-shadow: 0 0 40px rgba(0, 0, 0, .15);-webkit-box-shadow: 0 0 5px rgba(0, 0, 0, .15);-goog-ms-box-shadow: 0 0 10px #333333;box-shadow: 0 0 40px rgba(0, 0, 0, .15);margin-bottom: 1px;}.content-inner {padding: 10px 10px;}.content-inner {background-color: #333333;}/* Header----------------------------------------------- */.header-outer {background: transparent none repeat-x scroll 0 -400px;_background-image: none;}.Header h1 {font: normal normal 60px Corsiva;color: #ffff00;text-shadow: -1px -1px 1px rgba(0, 0, 0, .2);}.Header h1 a {color: #ffff00;}.Header .description {font-size: 140%;color: #aaaaaa;}.header-inner .Header .titlewrapper {padding: 22px 30px;}.header-inner .Header .descriptionwrapper {padding: 0 30px;}/* Tabs----------------------------------------------- */.tabs-inner .section:first-child {border-top: 1px solid #404040;}.tabs-inner .section:first-child ul {margin-top: -1px;border-top: 1px solid #404040;border-left: 0 solid #404040;border-right: 0 solid #404040;}.tabs-inner .widget ul {background: #222222 none repeat-x scroll 0 -800px;_background-image: none;border-bottom: 1px solid #404040;margin-top: 0;margin-left: -30px;margin-right: -30px;}.tabs-inner .widget li a {display: inline-block;padding: .6em 1em;font: normal normal 14px Arial, Tahoma, Helvetica, FreeSans, sans-serif;color: #999999;border-left: 1px solid #333333;border-right: 0 solid #404040;}.tabs-inner .widget li:first-child a {border-left: none;}.tabs-inner .widget li.selected a, .tabs-inner .widget li a:hover {color: #ffffff;background-color: #000000;text-decoration: none;}/* Columns----------------------------------------------- */.main-outer {border-top: 0 solid #404040;}.fauxcolumn-left-outer .fauxcolumn-inner {border-right: 1px solid #404040;}.fauxcolumn-right-outer .fauxcolumn-inner {border-left: 1px solid #404040;}/* Headings----------------------------------------------- */div.widget > h2,div.widget h2.title {margin: 0 0 1em 0;font: normal bold 11px Arial, Tahoma, Helvetica, FreeSans, sans-serif;color: #ffffff;}/* Widgets----------------------------------------------- */.widget .zippy {color: #999999;text-shadow: 2px 2px 1px rgba(0, 0, 0, .1);}.widget .popular-posts ul {list-style: none;}/* Posts----------------------------------------------- */h2.date-header {font: normal bold 11px Arial, Tahoma, Helvetica, FreeSans, sans-serif;}.date-header span {background-color: transparent;color: #cccccc;padding: inherit;letter-spacing: inherit;margin: inherit;}.main-inner {padding-top: 30px;padding-bottom: 30px;}.main-inner .column-center-inner {padding: 0 15px;}.main-inner .column-center-inner .section {margin: 0 15px;}.post {margin: 0 0 25px 0;}h3.post-title, .comments h4 {font: normal normal 22px Arial, Tahoma, Helvetica, FreeSans, sans-serif;margin: .75em 0 0;}.post-body {font-size: 110%;line-height: 1.4;position: relative;}.post-body img, .post-body .tr-caption-container, .Profile img, .Image img,.BlogList .item-thumbnail img {padding: 0;background: #111111;border: 1px solid #111111;-moz-box-shadow: 1px 1px 5px rgba(0, 0, 0, .1);-webkit-box-shadow: 1px 1px 5px rgba(0, 0, 0, .1);box-shadow: 1px 1px 5px rgba(0, 0, 0, .1);}.post-body img, .post-body .tr-caption-container {padding: 1px;}.post-body .tr-caption-container {color: #cccccc;}.post-body .tr-caption-container img {padding: 0;background: transparent;border: none;-moz-box-shadow: 0 0 0 rgba(0, 0, 0, .1);-webkit-box-shadow: 0 0 0 rgba(0, 0, 0, .1);box-shadow: 0 0 0 rgba(0, 0, 0, .1);}.post-header {margin: 0 0 1.5em;line-height: 1.6;font-size: 90%;}.post-footer {margin: 20px -2px 0;padding: 5px 10px;color: #888888;background-color: #303030;border-bottom: 1px solid #444444;line-height: 1.6;font-size: 90%;}#comments .comment-author {padding-top: 1.5em;border-top: 1px solid #404040;background-position: 0 1.5em;}#comments .comment-author:first-child {padding-top: 0;border-top: none;}.avatar-image-container {margin: .2em 0 0;}#comments .avatar-image-container img {border: 1px solid #111111;}/* Comments----------------------------------------------- */.comments .comments-content .icon.blog-author {background-repeat: no-repeat;background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABIAAAASCAYAAABWzo5XAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEgAACxIB0t1+/AAAAAd0SU1FB9sLFwMeCjjhcOMAAAD+SURBVDjLtZSvTgNBEIe/WRRnm3U8RC1neQdsm1zSBIU9VVF1FkUguQQsD9ITmD7ECZIJSE4OZo9stoVjC/zc7ky+zH9hXwVwDpTAWWLrgS3QAe8AZgaAJI5zYAmc8r0G4AHYHQKVwII8PZrZFsBFkeRCABYiMh9BRUhnSkPTNCtVXYXURi1FpBDgArj8QU1eVXUzfnjv7yP7kwu1mYrkWlU33vs1QNu2qU8pwN0UpKoqokjWwCztrMuBhEhmh8bD5UDqur75asbcX0BGUB9/HAMB+r32hznJgXy2v0sGLBcyAJ1EK3LFcbo1s91JeLwAbwGYu7TP/3ZGfnXYPgAVNngtqatUNgAAAABJRU5ErkJggg);}.comments .comments-content .loadmore a {border-top: 1px solid #999999;border-bottom: 1px solid #999999;}.comments .comment-thread.inline-thread {background-color: #303030;}.comments .continue {border-top: 2px solid #999999;}/* Accents---------------------------------------------- */.section-columns td.columns-cell {border-left: 1px solid #404040;}.blog-pager {background: transparent none no-repeat scroll top center;}.blog-pager-older-link, .home-link,.blog-pager-newer-link {background-color: #333333;padding: 5px;}.footer-outer {border-top: 0 dashed #bbbbbb;}/* Mobile----------------------------------------------- */body.mobile {background-size: auto;}.mobile .body-fauxcolumn-outer {background: transparent none repeat scroll top left;}.mobile .body-fauxcolumn-outer .cap-top {background-size: 100% auto;}.mobile .content-outer {-webkit-box-shadow: 0 0 3px rgba(0, 0, 0, .15);box-shadow: 0 0 3px rgba(0, 0, 0, .15);}.mobile .tabs-inner .widget ul {margin-left: 0;margin-right: 0;}.mobile .post {margin: 0;}.mobile .main-inner .column-center-inner .section {margin: 0;}.mobile .date-header span {padding: 0.1em 10px;margin: 0 -10px;}.mobile h3.post-title {margin: 0;}.mobile .blog-pager {background: transparent none no-repeat scroll top center;}.mobile .footer-outer {border-top: none;}.mobile .main-inner, .mobile .footer-inner {background-color: #333333;}.mobile-index-contents {color: #cccccc;}.mobile-link-button {background-color: #dd7700;}.mobile-link-button a:link, .mobile-link-button a:visited {color: #ffffff;}.mobile .tabs-inner .section:first-child {border-top: none;}.mobile .tabs-inner .PageList .widget-content {background-color: #000000;color: #ffffff;border-top: 1px solid #404040;border-bottom: 1px solid #404040;}.mobile .tabs-inner .PageList .widget-content .pagelist-arrow {border-left: 1px solid #404040;}h2 { font-size: 1.2em; }-->/style>style idtemplate-skin-1 typetext/css>!--body {min-width: 1180px;}.content-outer, .content-fauxcolumn-outer, .region-inner {min-width: 1180px;max-width: 1180px;_width: 1180px;}.main-inner .columns {padding-left: 0px;padding-right: 0px;}.main-inner .fauxcolumn-center-outer {left: 0px;right: 0px;/* IE6 does not respect left and right together */_width: expression(this.parentNode.offsetWidth -parseInt(0px) -parseInt(0px) + px);}.main-inner .fauxcolumn-left-outer {width: 0px;}.main-inner .fauxcolumn-right-outer {width: 0px;}.main-inner .column-left-outer {width: 0px;right: 100%;margin-left: -0px;}.main-inner .column-right-outer {width: 0px;margin-right: -0px;}#layout {min-width: 0;}#layout .content-outer {min-width: 0;width: 800px;}#layout .region-inner {min-width: 0;width: auto;}body#layout div.add_widget {padding: 8px;}body#layout div.add_widget a {margin-left: 32px;}-->/style>link hrefhttps://www.blogger.com/dyn-css/authorization.css?targetBlogID6220320961972054840&zx61903cca-7e92-48e6-a04d-ab434698a35c medianone onloadif(media!'all')media'all' relstylesheet/>noscript>link hrefhttps://www.blogger.com/dyn-css/authorization.css?targetBlogID6220320961972054840&zx61903cca-7e92-48e6-a04d-ab434698a35c relstylesheet/>/noscript>meta namegoogle-adsense-platform-account contentca-host-pub-1556223355139109/>meta namegoogle-adsense-platform-domain contentblogspot.com/>/head>body classloading variant-dark>div classnavbar section idnavbar nameNavbar>div classwidget Navbar data-version1 idNavbar1>script typetext/javascript> function setAttributeOnload(object, attribute, val) { if(window.addEventListener) { window.addEventListener(load, function(){ objectattribute val; }, false); } else { window.attachEvent(onload, function(){ objectattribute val; }); } } /script>div idnavbar-iframe-container>/div>script typetext/javascript srchttps://apis.google.com/js/platform.js>/script>script typetext/javascript> gapi.load(gapi.iframes:gapi.iframes.style.bubble, function() { if (gapi.iframes && gapi.iframes.getContext) { gapi.iframes.getContext().openChild({ url: https://www.blogger.com/navbar.g?targetBlogID\x3d6220320961972054840\x26blogName\x3dSunshine+on+a+Cloudy+Net\x26publishMode\x3dPUBLISH_MODE_HOSTED\x26navbarType\x3dLIGHT\x26layoutType\x3dLAYOUTS\x26searchRoot\x3dhttp://blog.sunshineonacloudy.net/search\x26blogLocale\x3den\x26v\x3d2\x26homepageUrl\x3dhttp://blog.sunshineonacloudy.net/\x26vt\x3d9001293784408960451, where: document.getElementById(navbar-iframe-container), id: navbar-iframe, messageHandlersFilter: gapi.iframes.CROSS_ORIGIN_IFRAMES_FILTER, messageHandlers: { blogger-ping: function() {} } }); } }); /script>script typetext/javascript>(function() {var script document.createElement(script);script.type text/javascript;script.src //pagead2.googlesyndication.com/pagead/js/google_top_exp.js;var head document.getElementsByTagName(head)0;if (head) {head.appendChild(script);}})();/script>/div>/div>div itemscopeitemscope itemtypehttp://schema.org/Blog styledisplay: none;>meta contentSunshine on a Cloudy Net itempropname/>meta contentCloud computing and data science from an Economist's perspective. itempropdescription/>/div>div classbody-fauxcolumns>div classfauxcolumn-outer body-fauxcolumn-outer>div classcap-top>div classcap-left>/div>div classcap-right>/div>/div>div classfauxborder-left>div classfauxborder-right>/div>div classfauxcolumn-inner>/div>/div>div classcap-bottom>div classcap-left>/div>div classcap-right>/div>/div>/div>/div>div classcontent>div classcontent-fauxcolumns>div classfauxcolumn-outer content-fauxcolumn-outer>div classcap-top>div classcap-left>/div>div classcap-right>/div>/div>div classfauxborder-left>div classfauxborder-right>/div>div classfauxcolumn-inner>/div>/div>div classcap-bottom>div classcap-left>/div>div classcap-right>/div>/div>/div>/div>div classcontent-outer>div classcontent-cap-top cap-top>div classcap-left>/div>div classcap-right>/div>/div>div classfauxborder-left content-fauxborder-left>div classfauxborder-right content-fauxborder-right>/div>div classcontent-inner>header>div classheader-outer>div classheader-cap-top cap-top>div classcap-left>/div>div classcap-right>/div>/div>div classfauxborder-left header-fauxborder-left>div classfauxborder-right header-fauxborder-right>/div>div classregion-inner header-inner>div classheader section idheader nameHeader>div classwidget Header data-version1 idHeader1>div idheader-inner>div classtitlewrapper>h1 classtitle>Sunshine on a Cloudy Net/h1>/div>div classdescriptionwrapper>p classdescription>span>Cloud Computing and Data Science from an Economists Perspective/span>/p>/div>/div>/div>/div>/div>/div>div classheader-cap-bottom cap-bottom>div classcap-left>/div>div classcap-right>/div>/div>/div>/header>div classtabs-outer>div classtabs-cap-top cap-top>div classcap-left>/div>div classcap-right>/div>/div>div classfauxborder-left tabs-fauxborder-left>div classfauxborder-right tabs-fauxborder-right>/div>div classregion-inner tabs-inner>div classtabs no-items section idcrosscol nameCross-Column>/div>div classtabs no-items section idcrosscol-overflow nameCross-Column 2>/div>/div>/div>div classtabs-cap-bottom cap-bottom>div classcap-left>/div>div classcap-right>/div>/div>/div>div classmain-outer>div classmain-cap-top cap-top>div classcap-left>/div>div classcap-right>/div>/div>div classfauxborder-left main-fauxborder-left>div classfauxborder-right main-fauxborder-right>/div>div classregion-inner main-inner>div classcolumns fauxcolumns>div classfauxcolumn-outer fauxcolumn-center-outer>div classcap-top>div classcap-left>/div>div classcap-right>/div>/div>div classfauxborder-left>div classfauxborder-right>/div>div classfauxcolumn-inner>/div>/div>div classcap-bottom>div classcap-left>/div>div classcap-right>/div>/div>/div>div classfauxcolumn-outer fauxcolumn-left-outer>div classcap-top>div classcap-left>/div>div classcap-right>/div>/div>div classfauxborder-left>div classfauxborder-right>/div>div classfauxcolumn-inner>/div>/div>div classcap-bottom>div classcap-left>/div>div classcap-right>/div>/div>/div>div classfauxcolumn-outer fauxcolumn-right-outer>div classcap-top>div classcap-left>/div>div classcap-right>/div>/div>div classfauxborder-left>div classfauxborder-right>/div>div classfauxcolumn-inner>/div>/div>div classcap-bottom>div classcap-left>/div>div classcap-right>/div>/div>/div>!-- corrects IE6 width calculation -->div classcolumns-inner>div classcolumn-center-outer>div classcolumn-center-inner>div classmain section idmain nameMain>div classwidget Blog data-version1 idBlog1>div classblog-posts hfeed> div classdate-outer> h2 classdate-header>span>Monday, June 8, 2015/span>/h2> div classdate-posts> div classpost-outer>div classpost hentry uncustomized-post-template itempropblogPost itemscopeitemscope itemtypehttp://schema.org/BlogPosting>meta contenthttp://www.clipular.com/c/6414704193830912.png?k5DzvisnrkXNBmuDd2HCfxj1PaKg itempropimage_url/>meta content6220320961972054840 itempropblogId/>meta content3697602316357310533 itemproppostId/>a name3697602316357310533>/a>h3 classpost-title entry-title itempropname>a hrefhttp://blog.sunshineonacloudy.net/2015/06/introduction-to-apache-spark_64.html>Introduction to Apache Spark/a>/h3>div classpost-header>div classpost-header-line-1>/div>/div>div classpost-body entry-content idpost-body-3697602316357310533 itemproparticleBody>h1 idintroduction-to-apache-spark>Introduction to Apache Spark/h1>These are some rough notes for a talk to be given at the June 2015 Chadoop meeting. br />h2 idwhat-is-apache-spark>What is a hrefhttp://spark.apache.org/>Apache Spark/a>?/h2>ol>li>Enterprise parallel processing framework based on resilient distributed datasets (RDD)/li>li>for java, scala, python and sql/li>li>with memory caching (memory reusable, used as progressive enhancement)/li>/ol>h2 idhow-can-i-get-spark-up-and-running-quickly>How can I get Spark up and running quickly?/h2>–> 1. Docker on collection of various firewalled hardware (described here) br />2. Amazon EC2 script (included in Spark distribution)br />h2 idhow-does-it-perform-scale>How does it perform / scale?/h2>ol>li>Resilient against (some) worker failures/li>li>Can add / remove workers Amazon spot market compatible br />–> 3. Can perform well or poorly depending on classical parallel processing concerns (cpu vs memory vs disk vs net bottlenecks)/li>/ol>h2 idquirks>Quirks/h2>ul>li>strong>Transformations/strong>, such as reading a file, mapping data over a function, etc. do not instantiate. /li>li>strong>Actions/strong> do – this is good and allows you to be deliberate about exactly how large intermediate results are to be handled (cached in memory, on disk, recomputed each time). /li>li>strong>Spark is not currently hierarchical./strong> Networking is not layered as App/Driver <-> Master <-> Workers. strong>Important/strong> make sure all workstations/masters/workers can connect to each other. /li>li>Random port numbers. /li>li>strong>Networking is not-NAT friendly/strong> and the documentation says little about this issue. This can be a blocker for em>simple/em> combining your cluster of 3-4 home PCs with extra Amazon EC2 machines, as the home machines may be on the 192.168 LAN and the Amazon EC2 machines on a difference Amazon LAN. If the spark master has a local IP like 192.168.1.100 and a router exposes it to the internet as 1.2.3.4, packets to the spark master at 1.2.3.4 will be read and discarded with logged errors. /li>li>Ideally, jobs that don’t finish timeout and are rescheduled. But that wasn’t always my experience. Spark still has a ways to go on robustness/reliability. There are unsolved issues. Workers can hang. A malformed csv file import was one way. I suspect though, reliability will improve over time./li>/ul>hr />h1 idusing-pipework-and-docker-to-run-apache-spark>Using Pipework and Docker to Run Apache Spark/h1>strong>My cluster at home experience:/strong> For Apache Spark I learned the hard way not to use docker’s hostname flag code>-h/code>, but instead use default hostnames and use code>pipework/code> to establish an alternate network interface in each container in order to place them all with their own IPs on a shared firewalled LAN. By default they can not be reached from outside the home, and would require planned holes in the firewall for outside access probably via ssh to a shell account inside the home with access to the cluster.br />a hrefhttps://github.com/jpetazzo/pipework>Pipework/a> is a free utility to route an additional ethernet bridge into a docker containerbr />a hrefhttp://www.docker.com/>Docker/a> is a Virtualization Technology for Linux based on Containers. br />To install in Ubuntu use code>apt-get install docker.io/code> where the name is code>docker.io/code> due to name conflict with an older project also called docker (system tray for KDE3/Gnome2 applications).br />h2 iddocker-vs-traditional-virtual-machine>Docker vs. Traditional Virtual Machine/h2>ul>li>Docker uses Linux Kernel namespace and cgroup capabilities to isolate and restrict containers/li>li>All the host and the docker containers share the hosts’ Linux kernel, memory, swap and hardware/li>li>the container sees its own file tree and process list, and can only access devices as permitted by docker/li>/ul>h3 idin-a-traditional-vm-resources-are-wasted>In a Traditional VM, resources are wasted:/h3>ul>li>booting and running multiple kernels/li>li>duplicate drivers, system utlities and cron jobs, etc. /li>li>by requiring hard allocations of memory for each running VM that might turn out to be incorrect later, like giving a workers too little or too much memory versus the master or driver. /li>/ul>h2 idthe-traditional-vm-is-probably-safermore-secure-docker-is-lighter-weight-in-a-trusted-environment>The traditional VM is probably safer/more secure. Docker is lighter weight in a trusted environment./h2>a hrefhttps://hub.docker.com/>Dockerhub/a> is a public repository of layered images for container filesystemsbr />A a hrefhttp://docs.docker.com/reference/builder/>code>Dockerfile/code>/a> is a script for building a container filesystem from the repository, docker directives and shell scripting.br />pre>code>FROM drpaulbrewer/spark-roasted-elephant:latestMAINTAINER drpaulbrewer@eaftc.comADD my-spark-worker.sh /spark/CMD /spark/my-spark-worker.sh/code>/pre>The a hrefhttp://docs.docker.com/reference/commandline/cli/>code>docker/code> command/a>, in conjunction with a docker daemon, builds, pushes, pulls, runs, stops, kills, etc… containers. br />shell script ./run-docker-spark for master and 1st worker AMD FX 8150br />pre>code>#!/bin/bashsudo -vMASTER$(docker run --namemaster --expose1-65535 --env SPARK_MASTER_IP192.168.1.10 --env SPARK_LOCAL_IP192.168.1.10 \ -v /data:/data -v /tmp:/tmp -d drpaulbrewer/spark-master:latest)sudo pipework eth0 $MASTER 192.168.1.10/24@192.168.1.1SPARK1$(docker run --namespark1 --expose1-65535 --env mem10G \--env masterspark://192.168.1.10:7077 \--env SPARK_LOCAL_IP192.168.1.11 \--env SPARKDIR/spark/spark-1.3.1 \-v /data:/data -v /tmp:/tmp -d drpaulbrewer/spark-worker:latest)sudo pipework eth0 $SPARK1 192.168.1.11/24@192.168.1.1/code>/pre>shell script for additional worker on the strong>same wired LAN/strong>br />pre>code>#!/bin/bashsudo -vSPARK$(docker run --namespark --expose1-65535 \--env mem10G \--env masterspark://192.168.1.10:7077 \--env SPARK_LOCAL_IP192.168.1.13 \-v /data:/data -v /tmp:/tmp -d drpaulbrewer/spark-worker:latest)sudo pipework eth0 $SPARK 192.168.1.13/24@192.168.1.1/code>/pre>shell script for additional worker on strong>wireless LAN/strong>. br />Issue: wireless LAN apparently not compatible with pipework’s bridging designed for eth0 br />Solution: –net”host” runs on hosts’ network stack; no pipework runbr />pre>code>#!/bin/bashsudo -vSPARK$(docker run --nethost --expose1-65535 \--env SPARKDIR/spark/spark-1.3.1 \--env mem10G \--env masterspark://192.168.1.10:7077 \-v /data:/data -v /tmp:/tmp -d drpaulbrewer/spark-worker:latest)/code>/pre>To reset or on system restart: br /> need to docker stop (or docker kill) and docker rm each container before running againbr />hr />h1 idparallel-processing-toy-example>Parallel Processing Toy example:/h1>h2 idcalculate-the-leibniz-formula-for-pi>Calculate the a hrefhttp://en.wikipedia.org/wiki/Leibniz_formula_for_%CF%80>Leibniz formula for PI/a>/h2>h2 idπ4-1-13-15-17-19-111>(π/4) 1 - (1/3) + (1/5) - (1/7) +(1/9) - (1/11) + …/h2>h2 idlets-use-denominators-out-to-4-billion-and-1>Let’s use denominators out to 4 Billion and 1./h2>h1 ida-without-apache-spark-python>A. Without Apache Spark – Python/h1>pre>code>total1.0for n0 in xrange(1000000000): n 1+n0 total + (1.0/(4.0*n+1.0)) - (1.0/(4.0*n-1.0))print 4.0*total/code>/pre>h1 ida-run-time-409-sec-on-1-core-of-amd-fx8150>A. Run Time: 409 sec on 1 core of AMD-FX8150/h1>hr />h1 idb-without-apache-spark-python-numpy>B. Without Apache Spark – Python + Numpy/h1>pre>code>import numpybillionInt 1+numpy.arange(1000000000, dtypenumpy.int32) total 1.0 + ((1.0/(4.0*billionInt+1.0)) - (1.0/(4.0*billionInt-1.0))).sum() print 4.0*total/code>/pre>h1 idissue-wants-26g-ram>Issue: wants 26G ram/h1>h2 idsolution-rent-amazon-ec2-xeon-e5-2680-28ghz-32gb-c3-4xlarge>Solution: Rent Amazon EC2 Xeon E5-2680 2.8Ghz 32GB (c3-4xlarge)/h2>h1 idb-run-time-25-sec-on-one-core>B. Run Time 25 sec on one core/h1>hr />h1 idc-without-apache-spark-c>C. Without Apache Spark – C/h1>pre>code>#include <stdio.h>#include <stdlib.h>#include <string.h>int main(int argc, char ** argv){ long int n; if (argc!3){ fprintf(stderr, usage: LeibnizPi1G <fromidx> <toidx_uninclusive>\n); exit(EXIT_FAILURE); } long int loop_from strtol(argv1,NULL,10); long int loop_to strtol(argv2, NULL, 10); long double pisum 0.0L; const long double one 1.0L; const long double four 4.0L; for(nloop_from;n<loop_to;++n) pisum+(one/(four*n+one))-(one/(four*n-one)); long double LeibnizPi 4.0*(1.0+pisum); printf(%.20Lf\n, LeibnizPi);}/code>/pre>h1 idc-run-time-amd-fx8150-1-core-9-sec>C. Run Time (AMD FX8150, 1 Core): 9 sec/h1>hr />h1 idd-without-apache-spark-hand-written-mapreduce>D. Without Apache Spark – hand written map/reduce/h1> br />pre>code>#!/bin/bashX./LeibnizPi1G$X 1 100000001 >/tmp/x1 &$X 100000001 200000001 >/tmp/x2 &$X 200000001 300000001 >/tmp/x3 &$X 300000001 400000001 >/tmp/x4 &$X 400000001 500000001 >/tmp/x5 &$X 500000001 600000001 >/tmp/x6 &$X 600000001 700000001 >/tmp/x7 &$X 700000001 800000001 >/tmp/x8 &$X 800000001 900000001 >/tmp/x9 &$X 900000001 1000000001 >/tmp/x10 &waitcat /tmp/x* | gawk BEGIN {sum0.0} {sum+($1-4.0)/4.0} END { printf(%0.17g\n,4.0+4.0*sum) }/code>/pre>h1 idd-run-time-2-3-sec-on-8-core-amd-fx-8150>D. Run Time: 2-3 sec on 8 core AMD FX 8150/h1>hr />h1>JS. Browser-side Javascript/h1>a hrefhttps://jsfiddle.net/DrPaulBrewer/q3vebu7a/>Leibniz π on jsfiddle/a>h2>Run Time: 8.88 sec in Chromium Browser on i5-4570/h2>hr />h1 idhome-apache-spark-cluster>Home Apache Spark Cluster/h1>h2 idbox-1-master-worker1-and-workstation-amd-fx-8150-8-cores-16gb>Box 1 (master, worker1, and workstation) AMD FX-8150 8 cores 16GB/h2>h2 idbox-2-worker2-i5-3470s-hp-ultra-slim-4-cores-16-gb>Box 2 (worker2) i5-3470S HP Ultra Slim 4 cores 16 GB/h2>h2 idbox-3-worker3-i5-4570r-gigabyte-brix-cube-4-cores-16gb-wlan>Box 3 (worker3) i5-4570R Gigabyte Brix Cube 4 cores 16GB WLAN/h2>After the master is running and the workers connected, we can go to the master WebUI to monitor br />the cluster and check stdout/stderr on any of the workersbr />img altSpark Cluster Management Console srchttp://www.clipular.com/c/6414704193830912.png?k5DzvisnrkXNBmuDd2HCfxj1PaKg title width1000/>br />hr />h1 idspark-python>Spark – Python/h1>pre>code>from operator import addfrom pyspark import SparkContextdef term(n): return (1.0/(4.0*n+1.0))-(1.0/(4.0*n-1.0))def LeibnizPi(Nterms): sc SparkContext(appNameLeibnizPI) piOver4Minus1 sc.parallelize(xrange(1,Nterms+1),30).map(term).reduce(add) return 4*(1+piOver4Minus1)print LeibnizPi(1000*1000*1000)/code>/pre>h2 idto-run-spark-submit-master-master-pathtocodepy-or-pyspark>To run: spark-submit –master $MASTER /path/to/code.py or pyspark/h2>h2 idrun-time-36-sec>Run Time 36 sec/h2>hr />h3 ididea-make-the-map-task-more-substantial>Idea: Make the map task more substantial/h3>h1 idspark-python-2>Spark – Python #2/h1>pre>code>from operator import addfrom pyspark import SparkContextdef Msum(n): megaN 1000000*n s 0.0 for k in xrange(1000000): d0 4*(k+1+megaN)+1 d1 d0-2 s + ((1.0/d0)-(1.0/d1)) return sdef LeibnizPi(Nterms): sc SparkContext(appNameLeibnizPI) piOver4Minus1 sc.parallelize(xrange(0,Nterms+1), 20).map(Msum).reduce(add) return 4*(1+piOver4Minus1)print LeibnizPi(999)/code>/pre>h2 idrun-time-24-sec>Run Time: 24 sec/h2>hr />h1 idspark-scala>Spark – Scala /h1>pre>code>def term(n: Int) (1.0/(4.0*n+1.0))-(1.0/(4.0*n-1.0))def LeibnizSum(Nterms: Int) sc.parallelize(1 to Nterms).map(term).reduce(_ + _)def LeibnizPi(Nterms: Int) 4.0*(1.0+LeibnizSum(Nterms)) println(LeibnizPi(1000*1000*1000))/code>/pre>h2 idto-run-spark-shell-master-master-then-load-codesc>To run: spark-shell –master $MASTER, then :load code.sc/h2>h2 idrun-time-19-sec>Run Time: 19 sec/h2>hr />h1 idspark-scala-2>Spark – Scala #2/h1>pre>code>def LeibnizMSum(k: Int) : Double { var sum:Double 0.0; var n:Long 0; sum 0.0 for( n <- (1000000*k.toLong+1) to (1000000*(k.toLong+1)) ){ sum sum + ((1.0/(4.0*n+1.0))-(1.0/(4.0*n-1.0))) ; } return sum;}def LeibnizSum(Nterms: Int) sc.parallelize(0 to Nterms). map(LeibnizMSum).reduce(_ + _)def LeibnizPi(Nterms: Int) 4.0*(1.0+LeibnizSum(Nterms)) println(LeibnizPi(999))/code>/pre>h2 idrun-time-215-225-sec>Run Time: 2.15 - 2.25 sec/h2>h3>Run time for 10 Billion terms, run as 64 map tasks: 17 sec /h3>hr />h1 idspark-java-1>Spark – Java #1/h1>h2 idfailed>Failed/h2>h3 idparallelize-required-a-listinteger-to-split-up-among-workers-but-it-first-had-to-be-initialized-to-1-2-3-1-billion-which-was-problematic>code>parallelize()/code> required a code>List<Integer>/code> to split up among workers, but it first had to be initialized to 1, 2, 3, …, 1 Billion, which was problematic./h3>hr />h1 idspark-java-2>Spark – Java #2/h1>pre>code>import org.apache.spark.api.java.*;import org.apache.spark.api.java.function.Function;import java.util.ArrayList;class LeibnizMSum implements Function<Integer,Double> { public Double call(Integer k){ double s0.0; int n 0; int limit 1000000*(k+1); for(n1000000*k+1; n<limit; ++n){ s s + ((1.0/(4.0*n+1.0))-(1.0/(4.0*n-1.0))); } return s; }}public class LeibnizPIApp { public static void main (String args){ int billion 1000*1000*1000; String masterURLspark://192.168.1.10:7077; String appName LeibnizPIJava; String home /spark/spark-1.3.1; String jarfile /Z/pi/LeibnizJava/target/LeibnizPiJava-1.0.jar; System.out.println(Hello this is LeibnizPIApp. Initializing...); ArrayList<Integer> gigalist new ArrayList<Integer>(1000); int i; for(i 0; i<1000; ++i) gigalist.add(i); System.out.println(Initialized gigalist); JavaSparkContext sc new JavaSparkContext(masterURL, appName, home, jarfile); System.out.println(have spark context); JavaRDD<Integer> gigaRDD sc.parallelize(gigalist, 32); System.out.println(have paralellized gigalist, starting map/reduce); Double pisum gigaRDD.map(new LeibnizMSum()).reduce( (a,b) -> a+b ); Double pi 4.0 + 4.0*pisum; System.out.println(pi); }}/code>/pre>h1 idrun-time-314-sec>Run Time 3.14 sec/h1>hr />h1 idspark-sql>Spark SQL/h1>ul>li>Prepare a file with entries 1,2,3,4,5…1 Billion. One int per line./li>li>Load it as a temporary SQL table with a scala script/li>li>Transformations, such as reading a file, mapping data over a function, etc. do not instantiate. strong>Actions/strong> do/li>li>A cache request stores the object em>the first time an action instantiates it/em>/li>li>Must use –executor-memory option to set available memory to workers/li>/ul>h2 idsql-table-load-script>SQL Table Load script/h2>pre>code>val sqlContext new org.apache.spark.sql.SQLContext(sc)import sqlContext.implicits._case class Gig(n: Int)val gigcsv sc.textFile(/data/gig.csv) .map(x > Gig(x.trim.toInt)).toDF().cache()gigcsv.registerTempTable(gig)/code>/pre>h2 idsql-count-statement>SQL count statement/h2>pre>code>sqlContext.sql(select count(*) from gig).collect().foreach(println)/code>/pre>h3 idfirst-run-59-secs-to-read-1-billion-records-from-ssd>First Run: 59 secs to read 1 billion records from SSD/h3>h3 idsecond-run-cached-in-memory-35-secs>Second Run: Cached in memory – 3.5 secs/h3>h2 idsql-pi-calculation>SQL Pi Calculation/h2>pre>code>sqlContext.sql(select 4.0*(1.0+sum( (1.0/(4.0*n+1.0))-(1.0/(4.0*n-1.0)) )) as LeibnizPI FROM gig).collect().foreach(println)/code>/pre>h3 idrun-time-20-secs-from-previous-cached-result>Run Time: 20 secs (from previous cached result)/h3>div styleclear: both;>/div>/div>div classpost-footer>div classpost-footer-line post-footer-line-1>span classpost-author vcard>byspan classfn itempropauthor itemscopeitemscope itemtypehttp://schema.org/Person>meta contenthttps://www.blogger.com/profile/09177647876962492314 itempropurl/>a classg-profile hrefhttps://www.blogger.com/profile/09177647876962492314 relauthor titleauthor profile>span itempropname>Paul Brewer/span>/a>/span>/span>span classpost-timestamp>/span>span classpost-comment-link>a classcomment-link hrefhttp://blog.sunshineonacloudy.net/2015/06/introduction-to-apache-spark_64.html#comment-form onclick>No comments: /a>/span>span classpost-icons>span classitem-control blog-admin pid-1033706227>a hrefhttps://www.blogger.com/post-edit.g?blogID6220320961972054840&postID3697602316357310533&frompencil titleEdit Post>img alt classicon-action height18 srchttps://resources.blogblog.com/img/icon18_edit_allbkg.gif width18/>/a>/span>/span>div classpost-share-buttons goog-inline-block>a classgoog-inline-block share-button sb-email hrefhttps://www.blogger.com/share-post.g?blogID6220320961972054840&postID3697602316357310533&targetemail target_blank titleEmail This>span classshare-button-link-text>Email This/span>/a>a classgoog-inline-block share-button sb-blog hrefhttps://www.blogger.com/share-post.g?blogID6220320961972054840&postID3697602316357310533&targetblog onclickwindow.open(this.href, _blank, height270,width475); return false; target_blank titleBlogThis!>span classshare-button-link-text>BlogThis!/span>/a>a classgoog-inline-block share-button sb-twitter hrefhttps://www.blogger.com/share-post.g?blogID6220320961972054840&postID3697602316357310533&targettwitter target_blank titleShare to X>span classshare-button-link-text>Share to X/span>/a>a classgoog-inline-block share-button sb-facebook hrefhttps://www.blogger.com/share-post.g?blogID6220320961972054840&postID3697602316357310533&targetfacebook onclickwindow.open(this.href, _blank, height430,width640); return false; target_blank titleShare to Facebook>span classshare-button-link-text>Share to Facebook/span>/a>a classgoog-inline-block share-button sb-pinterest hrefhttps://www.blogger.com/share-post.g?blogID6220320961972054840&postID3697602316357310533&targetpinterest target_blank titleShare to Pinterest>span classshare-button-link-text>Share to Pinterest/span>/a>/div>/div>div classpost-footer-line post-footer-line-2>span classpost-labels>/span>/div>div classpost-footer-line post-footer-line-3>span classpost-location>/span>/div>/div>/div>/div> /div>/div> div classdate-outer> h2 classdate-header>span>Wednesday, March 25, 2015/span>/h2> div classdate-posts> div classpost-outer>div classpost hentry uncustomized-post-template itempropblogPost itemscopeitemscope itemtypehttp://schema.org/BlogPosting>meta contenthttp://www.clipular.com/c/6234404281122816.png?kIVn4dAktiX8k7qzjbCkXmAvkV-M itempropimage_url/>meta content6220320961972054840 itempropblogId/>meta content5983073946663804532 itemproppostId/>a name5983073946663804532>/a>h3 classpost-title entry-title itempropname>a hrefhttp://blog.sunshineonacloudy.net/2015/03/loanmodificationapprovalrates_25.html>LoanModificationApprovalRates/a>/h3>div classpost-header>div classpost-header-line-1>/div>/div>div classpost-body entry-content idpost-body-5983073946663804532 itemproparticleBody>h1 idinside-the-us-treasury-hamp-201501-data-dump-fun-with-loan-modification-approval-rates>Inside the US Treasury HAMP 2015.01 Data Dump: Fun with Loan Modification Approval Rates /h1>p>It’s no secret that many people who applied for a loan modification were unable to obtain one. Some people received the a hrefhttp://www.propublica.org/article/loan-mod-profiles-runaround>run around as banks lost their paperwork over and over/a>. Others received a trial modification but not a permanent one. Some people simply weren’t eligible. /p>p>The fields code>ln_st_nme/code> and code>ln_mdfc_mode_nme/code> in the data dump tell this story:/p>p>img altoutcomes of loan modification requests srchttp://www.clipular.com/c/6234404281122816.png?kIVn4dAktiX8k7qzjbCkXmAvkV-M title>/p>p>Only rows 2 and 6 are good outcomes. This article concentrates on the rate of row 2 “Active Payment - Official Modification” outcomes, about 1 million out of the 6+ million total requests. /p>p>A recent public data dump, together with the capabilities of Google BigQuery, allows a deeper look at the approval rates for loan modification requests. /p>h1 idloan-modification-approvals-percentages-by-msa-state>Loan Modification Approvals Percentages by MSA State/h1>p>img alttop 10 states by loan modification approval rate srchttp://www.clipular.com/c/6316456477196288.png?kpdDWJ9pwRSBhRBGzCVFRRHsN838 title>/p>p>em>Active/em> is the number of official, permanent modifications with active payment/p>p>em>Total/em> is the number of all requests, including rejected requests, cancellations, trials and permanent./p>p>em>Approval percentage/em> is a percentage calculated as 100 x Active / Total/p>p>Puerto Rico is the surprising entry on this list . Even though the number of applications and official modifications is small, there’s no obvious explanation why it should be easier for someone to get a loan modification approval in Puerto Rico, as opposed, to say, Hawaii, which had about than twice as many applications but fewer finished official, permanent modifications. /p>p>DC is not exactly a surprise. Putting cynicism aside for a moment, the place is full of professionals who can navigate, or help others navigate, the government red tape involved in the loan modification program./p>h1 idbottom-states>Bottom States/h1>p>img altBottom States by Loan Modification Approval Rate (Metro Areas) srchttp://www.clipular.com/c/6625542120931328.png?kwH87pVj02VjeRBAIHPGGboRWOcA title>/p>p>Among the bottom approval rates we find Texas, with a ~ 12% approval percentage for loan modifications. Texas has about as many active, permanent, official modifications as Arizona, about 33k. However, 272k Texans applied as opposed to 200k Arizonans. /p>h1 idwhats-an-msa>What’s an MSA?/h1>p>Practically, an MSA is an area around a city or town, identified by a code number such as 31080 for Los Angeles, or 12060 for Atlanta. /p>p>Technically, an em>MSA/em> is a em>Metropolitan/Micropolitan Statistical Area/em>, a group of counties identified by the federal government (probably the census bureau) as representing a named area. /p>p>In the HAMP data dump, a row is tagged with em>either/em> an MSA code number (~5.8 million records) or a em>state/em> (~419,000 records) or nothing (8 records) . Knowing both the state and MSA would be useful in state border regions such as New York City, Memphis, or Chicago as the MSA can cross state borders./p>h1 idbest-approval-rates-by-msa>Best Approval Rates by MSA/h1>p>img altenter image description here srchttp://www.clipular.com/c/6182327349673984.png?kTe-NMHHZMQ7zho7iescxZgwz_SQ title>/p>p>Once again, Puerto Rico takes the lead, followed by the LA metro area and various California towns./p>h1 ida-bit-of-fun-with-politics>A Bit of Fun with Politics/h1>p>Going back to our state-level data, we can also fetch whether Obama (D) or Romney(R) won the state in the 2012 Election and put that alongside the loan modification approval percentage./p>h2 id2012-presidential-election-state-by-state>2012 Presidential Election State by State/h2>p>img alt2012 election map from Wikipedia srchttp://upload.wikimedia.org/wikipedia/commons/thumb/4/44/ElectoralCollege2012.svg/500px-ElectoralCollege2012.svg.png title> br>source: a hrefhttp://en.wikipedia.org/wiki/United_States_presidential_election,_2012>Wikipedia/a>/p>p>img alttop loan modification approval percentage with 2012 election info srchttp://www.clipular.com/c/6496047112126464.png?kmKsWoflO-1WNKfpELMYCFc6PQLY title>/p>p>Although the US political scene is not so simple or rigid as to put an entire state in the Democratic or Republican bucket based on the 2012 election, for the sake of argument we see 8 (D) states getting the top 10 loan modification approval rates and only 2 (R) states in the top 10./p>p>And if we look at the bottom 10 states by loan modification approval percentage…/p>p>img altbottom loan modification approval percentages wth 2012 election info srchttp://www.clipular.com/c/5056328339816448.png?kIcF_HgNnXNEODX_s4NQgb9ta3UA title>/p>p>we see 9 (R) states at the bottom of the rankings and only 1 (D) state./p>p>From this, what can a hypothetical talk-radio host conclude? Probably not much. Certainly none of these things:/p>ul>li>that the (D) states are being paid off for voting (D), at the expense of the (R) states **? /li>li>that (R) folks have a better chance a hrefhttp://en.wikipedia.org/wiki/IRS_targeting_controversy>attracting a tax audit/a> than receiving a loan modification**?/li>li>that the (R) folks are sabotaging the loan modification process in some states **?/li>li>that people who needed a loan modification voted, in 2012, for the party that would give them one**? br>** except Utah and Arizona of course. /li>/ul>p>After all, a hrefhttp://stats.stackexchange.com/q/36/87>correlation does not prove causation/a>. /p>p>There can be unknown factors creating these effects that are also correlated with politics. /p>p>Speaking of correlation, if we use 2012 popular vote, % Romney - % Obama, to put each state on a left-right axis, we get this pretty plot:/p>p>img altMSA HAMP Loan Modification Approval Rate vs 2012 Popular Vote srchttps://raw.githubusercontent.com/DrPaulBrewer/loan-modification-politics/master/approvalRate/fig1.png title>/p>p>The R^2 for the linear fit is around 0.40, meaning that about 40% of the variance is explained by the linear model (leaving 60% unexplained…) /p>p>Once again, remember, correlation does not prove causation. /p>p>R provided this summary of the linear model shown in the chart. /p>pre classprettyprint>code class hljs asciidoc>Call:lm(formula hampspan classhljs-emphasis>_approval_/span>rate ~ 1 + popularspan classhljs-emphasis>_vote2012, data df)/span>Residuals:span classhljs-code> Min 1Q Median 3Q Max /span>-3.6433 -0.9806 -0.3023 1.1444 6.2260 Coefficients:span classhljs-code> Estimate Std. Error t value Pr(>|t|) /span>(Intercept) 14.76049 0.26599 55.492 < 2e-16 ***span classhljs-header>popular_vote2012 -0.07342 0.01307 -5.618 9.54e-07 ***---/span>Signif. codes: 0 ‘**span classhljs-strong>*’ 0.001 ‘*/span>span classhljs-strong>*’ 0.01 ‘*/span>’ 0.05 ‘.’ 0.1 ‘ ’ 1Residual standard error: 1.875 on 48 degrees of freedomMultiple R-squared: 0.3967, Adjusted R-squared: 0.3842 F-statistic: 31.57 on 1 and 48 DF, p-value: 9.544e-07/code>/pre>p>The queries, raw data, and R scripts for this are in a hrefhttps://github.com/DrPaulBrewer/loan-modification-politics>this github repository/a> for those who would like to use these files as a basis for other explorations, commentary, or improvement. /p>h1 idtotalling-savings-to-borrowers>Totalling Savings to Borrowers/h1>p>The data dump contains fields for borrower monthly housing expense before and after the modification. These fields do not always contain any numbers, or numbers that make sense, so we restrict the data to permanent modifications where these numbers take on somewhat reasonable values (I used up to span>$/span>10,000/mo, which might seem a bit high for a house payment plus miscellaneous – but there are hundreds of entries that are higher) and eliminate any reports showing zero or negative savings. /p>p>img altMonthly Savings from Loan Modification srchttp://www.clipular.com/c/5379121707220992.png?kHX0Nepw7TBIJyuQyY8-4gRJLneQ title>/p>p>The California savings of span>$/span>209 million/month dwarfs the #2 and #3 states of Florida and New York that are tied at around span>$/span>72 million/month saved on housing expenses. Other states in the top 12 come in from span>$/span>10-span>$/span>30 million/month saved./p>p>The top average savings per borrower/mod studied goes to HI with span>$/span>1008/mo (off chart), then NY (span>$/span>910/mo), CA (span>$/span>857/mo), DC (span>$/span>769/mo)/p>p>img altBottom states, Monthly Savings from Loan Modification srchttp://www.clipular.com/c/6211941954486272.png?kF4Q7oIwe2DxI2OKW0xHTLzUI1xM title>/p>p>The bottom end of the savings list dips under span>$/span>100K/month total saved, with average savings in the span>$/span>300/mo-span>$/span>500/mo range./p>h1 iddata-issues>Data Issues/h1>p>Ive heard that there are data issues with some fields in the HAMP data dump. Some of these are mentioned in the data dictionary and users guide. Ive also seen reports of higher HAMP approval rates than the measly 10-20% we see here. /p>p>Heres a document that says the number is more like 30% but it includes, in the fine print, trials that do not become permanent modifications: br/> a hrefhttp://www.treasury.gov/initiatives/financial-stability/reports/Documents/HAMP Application Activity by Servicer Jan 2015.pdf>HAMP Application Activity by Servicer Jan 2015/a>/p>p>So, take what you read here with a grain of salt. If anyone has further insight about the data issues, please, explain in the comments section. /p>div styleclear: both;>/div>/div>div classpost-footer>div classpost-footer-line post-footer-line-1>span classpost-author vcard>byspan classfn itempropauthor itemscopeitemscope itemtypehttp://schema.org/Person>meta contenthttps://www.blogger.com/profile/09177647876962492314 itempropurl/>a classg-profile hrefhttps://www.blogger.com/profile/09177647876962492314 relauthor titleauthor profile>span itempropname>Paul Brewer/span>/a>/span>/span>span classpost-timestamp>/span>span classpost-comment-link>a classcomment-link hrefhttp://blog.sunshineonacloudy.net/2015/03/loanmodificationapprovalrates_25.html#comment-form onclick>No comments: /a>/span>span classpost-icons>span classitem-control blog-admin pid-1033706227>a hrefhttps://www.blogger.com/post-edit.g?blogID6220320961972054840&postID5983073946663804532&frompencil titleEdit Post>img alt classicon-action height18 srchttps://resources.blogblog.com/img/icon18_edit_allbkg.gif width18/>/a>/span>/span>div classpost-share-buttons goog-inline-block>a classgoog-inline-block share-button sb-email hrefhttps://www.blogger.com/share-post.g?blogID6220320961972054840&postID5983073946663804532&targetemail target_blank titleEmail This>span classshare-button-link-text>Email This/span>/a>a classgoog-inline-block share-button sb-blog hrefhttps://www.blogger.com/share-post.g?blogID6220320961972054840&postID5983073946663804532&targetblog onclickwindow.open(this.href, _blank, height270,width475); return false; target_blank titleBlogThis!>span classshare-button-link-text>BlogThis!/span>/a>a classgoog-inline-block share-button sb-twitter hrefhttps://www.blogger.com/share-post.g?blogID6220320961972054840&postID5983073946663804532&targettwitter target_blank titleShare to X>span classshare-button-link-text>Share to X/span>/a>a classgoog-inline-block share-button sb-facebook hrefhttps://www.blogger.com/share-post.g?blogID6220320961972054840&postID5983073946663804532&targetfacebook onclickwindow.open(this.href, _blank, height430,width640); return false; target_blank titleShare to Facebook>span classshare-button-link-text>Share to Facebook/span>/a>a classgoog-inline-block share-button sb-pinterest hrefhttps://www.blogger.com/share-post.g?blogID6220320961972054840&postID5983073946663804532&targetpinterest target_blank titleShare to Pinterest>span classshare-button-link-text>Share to Pinterest/span>/a>/div>/div>div classpost-footer-line post-footer-line-2>span classpost-labels>/span>/div>div classpost-footer-line post-footer-line-3>span classpost-location>/span>/div>/div>/div>/div> /div>/div> /div>div classblog-pager idblog-pager>span idblog-pager-older-link>a classblog-pager-older-link hrefhttp://blog.sunshineonacloudy.net/search?updated-max2015-03-25T23:29:00-07:00&max-results2 idBlog1_blog-pager-older-link titleOlder Posts>Older Posts/a>/span>a classhome-link hrefhttp://blog.sunshineonacloudy.net/>Home/a>/div>div classclear>/div>div classblog-feeds>div classfeed-links>Subscribe to:a classfeed-link hrefhttp://blog.sunshineonacloudy.net/feeds/posts/default target_blank typeapplication/atom+xml>Posts (Atom)/a>/div>/div>/div>/div>/div>/div>div classcolumn-left-outer>div classcolumn-left-inner>aside>/aside>/div>/div>div classcolumn-right-outer>div classcolumn-right-inner>aside>/aside>/div>/div>/div>div styleclear: both>/div>!-- columns -->/div>!-- main -->/div>/div>div classmain-cap-bottom cap-bottom>div classcap-left>/div>div classcap-right>/div>/div>/div>footer>div classfooter-outer>div classfooter-cap-top cap-top>div classcap-left>/div>div classcap-right>/div>/div>div classfauxborder-left footer-fauxborder-left>div classfauxborder-right footer-fauxborder-right>/div>div classregion-inner footer-inner>div classfoot section idfooter-1>div classwidget BlogArchive data-version1 idBlogArchive1>h2>Blog Archive/h2>div classwidget-content>div idArchiveList>div idBlogArchive1_ArchiveList>ul classhierarchy>li classarchivedate expanded>a classtoggle hrefjavascript:void(0)>span classzippy toggle-open> ▼ /span>/a>a classpost-count-link hrefhttp://blog.sunshineonacloudy.net/2015/>2015/a>span classpost-count dirltr>(5)/span>ul classhierarchy>li classarchivedate expanded>a classtoggle hrefjavascript:void(0)>span classzippy toggle-open> ▼ /span>/a>a classpost-count-link hrefhttp://blog.sunshineonacloudy.net/2015/06/>June/a>span classpost-count dirltr>(1)/span>ul classposts>li>a hrefhttp://blog.sunshineonacloudy.net/2015/06/introduction-to-apache-spark_64.html>Introduction to Apache Spark/a>/li>/ul>/li>/ul>ul classhierarchy>li classarchivedate collapsed>a classtoggle hrefjavascript:void(0)>span classzippy> ► /span>/a>a classpost-count-link hrefhttp://blog.sunshineonacloudy.net/2015/03/>March/a>span classpost-count dirltr>(4)/span>/li>/ul>/li>/ul>/div>/div>div classclear>/div>/div>/div>/div>table border0 cellpadding0 cellspacing0 classsection-columns columns-2>tbody>tr>td classfirst columns-cell>div classfoot no-items section idfooter-2-1>/div>/td>td classcolumns-cell>div classfoot section idfooter-2-2>div classwidget Profile data-version1 idProfile1>h2>About Me/h2>div classwidget-content>dl classprofile-datablock>dt classprofile-data>a classprofile-name-link g-profile hrefhttps://www.blogger.com/profile/09177647876962492314 relauthor stylebackground-image: url(//www.blogger.com/img/logo-16.png);>Paul Brewer/a>/dt>/dl>a classprofile-link hrefhttps://www.blogger.com/profile/09177647876962492314 relauthor>View my complete profile/a>div classclear>/div>/div>/div>/div>/td>/tr>/tbody>/table>!-- outside of the include in order to lock Attribution widget -->div classfoot section idfooter-3 nameFooter>div classwidget Attribution data-version1 idAttribution1>div classwidget-content styletext-align: center;>Copyright Dr Paul Brewer 2015- - Economic and Financial Technology Consulting LLC. Simple theme. Powered by a hrefhttps://www.blogger.com target_blank>Blogger/a>./div>div classclear>/div>/div>/div>/div>/div>div classfooter-cap-bottom cap-bottom>div classcap-left>/div>div classcap-right>/div>/div>/div>/footer>!-- content -->/div>/div>div classcontent-cap-bottom cap-bottom>div classcap-left>/div>div classcap-right>/div>/div>/div>/div>script typetext/javascript> window.setTimeout(function() { document.body.className document.body.className.replace(loading, ); }, 10); /script>script typetext/javascript srchttps://www.blogger.com/static/v1/widgets/91768132-widgets.js>/script>script typetext/javascript>window__wavt AOuZoY5LSy31Jgw6lu0pT36899LAekqMFQ:1736521198700;_WidgetManager._Init(//www.blogger.com/rearrange?blogID\x3d6220320961972054840,//blog.sunshineonacloudy.net/,6220320961972054840);_WidgetManager._SetDataContext({name: blog, data: {blogId: 6220320961972054840, title: Sunshine on a Cloudy Net, url: http://blog.sunshineonacloudy.net/, canonicalUrl: http://blog.sunshineonacloudy.net/, homepageUrl: http://blog.sunshineonacloudy.net/, searchUrl: http://blog.sunshineonacloudy.net/search, canonicalHomepageUrl: http://blog.sunshineonacloudy.net/, blogspotFaviconUrl: http://blog.sunshineonacloudy.net/favicon.ico, bloggerUrl: https://www.blogger.com, hasCustomDomain: true, httpsEnabled: false, enabledCommentProfileImages: true, gPlusViewType: FILTERED_POSTMOD, adultContent: false, analyticsAccountNumber: , encoding: UTF-8, locale: en, localeUnderscoreDelimited: en, languageDirection: ltr, isPrivate: false, isMobile: false, isMobileRequest: false, mobileClass: , isPrivateBlog: false, isDynamicViewsAvailable: true, feedLinks: \x3clink rel\x3d\x22alternate\x22 type\x3d\x22application/atom+xml\x22 title\x3d\x22Sunshine on a Cloudy Net - Atom\x22 href\x3d\x22http://blog.sunshineonacloudy.net/feeds/posts/default\x22 /\x3e\n\x3clink rel\x3d\x22alternate\x22 type\x3d\x22application/rss+xml\x22 title\x3d\x22Sunshine on a Cloudy Net - RSS\x22 href\x3d\x22http://blog.sunshineonacloudy.net/feeds/posts/default?alt\x3drss\x22 /\x3e\n\x3clink rel\x3d\x22service.post\x22 type\x3d\x22application/atom+xml\x22 title\x3d\x22Sunshine on a Cloudy Net - Atom\x22 href\x3d\x22https://www.blogger.com/feeds/6220320961972054840/posts/default\x22 /\x3e\n, meTag: \x3clink rel\x3d\x22me\x22 href\x3d\x22https://www.blogger.com/profile/09177647876962492314\x22 /\x3e\n, adsenseHostId: ca-host-pub-1556223355139109, adsenseHasAds: false, adsenseAutoAds: false, boqCommentIframeForm: true, loginRedirectParam: , view: , dynamicViewsCommentsSrc: //www.blogblog.com/dynamicviews/4224c15c4e7c9321/js/comments.js, dynamicViewsScriptSrc: //www.blogblog.com/dynamicviews/d2c988fc3a56b1b1, plusOneApiSrc: https://apis.google.com/js/platform.js, disableGComments: true, interstitialAccepted: false, sharing: {platforms: {name: Get link, key: link, shareMessage: Get link, target: }, {name: Facebook, key: facebook, shareMessage: Share to Facebook, target: facebook}, {name: BlogThis!, key: blogThis, shareMessage: BlogThis!, target: blog}, {name: X, key: twitter, shareMessage: Share to X, target: twitter}, {name: Pinterest, key: pinterest, shareMessage: Share to Pinterest, target: pinterest}, {name: Email, key: email, shareMessage: Email, target: email}, disableGooglePlus: true, googlePlusShareButtonWidth: 0, googlePlusBootstrap: \x3cscript type\x3d\x22text/javascript\x22\x3ewindow.___gcfg \x3d {\x27lang\x27: \x27en\x27};\x3c/script\x3e}, hasCustomJumpLinkMessage: false, jumpLinkMessage: Read more, pageType: index, pageName: , pageTitle: Sunshine on a Cloudy Net, metaDescription: Cloud computing and data science from an Economist\x27s perspective.}}, {name: features, data: {}}, {name: messages, data: {edit: Edit, linkCopiedToClipboard: Link copied to clipboard!, ok: Ok, postLink: Post Link}}, {name: template, data: {name: Simple, localizedName: Simple, isResponsive: false, isAlternateRendering: false, isCustom: false, variant: dark, variantId: dark}}, {name: view, data: {classic: {name: classic, url: ?view\x3dclassic}, flipcard: {name: flipcard, url: ?view\x3dflipcard}, magazine: {name: magazine, url: ?view\x3dmagazine}, mosaic: {name: mosaic, url: ?view\x3dmosaic}, sidebar: {name: sidebar, url: ?view\x3dsidebar}, snapshot: {name: snapshot, url: ?view\x3dsnapshot}, timeslide: {name: timeslide, url: ?view\x3dtimeslide}, isMobile: false, title: Sunshine on a Cloudy Net, description: Cloud computing and data science from an Economist\x27s perspective., url: http://blog.sunshineonacloudy.net/, type: feed, isSingleItem: false, isMultipleItems: true, isError: false, isPage: false, isPost: false, isHomepage: true, isArchive: false, isLabelSearch: false}});_WidgetManager._RegisterWidget(_NavbarView, new _WidgetInfo(Navbar1, navbar, document.getElementById(Navbar1), {}, displayModeFull));_WidgetManager._RegisterWidget(_HeaderView, new _WidgetInfo(Header1, header, document.getElementById(Header1), {}, displayModeFull));_WidgetManager._RegisterWidget(_BlogView, new _WidgetInfo(Blog1, main, document.getElementById(Blog1), {cmtInteractionsEnabled: false, lightboxEnabled: true, lightboxModuleUrl: https://www.blogger.com/static/v1/jsbin/599103511-lbx.js, lightboxCssUrl: https://www.blogger.com/static/v1/v-css/1964470060-lightbox_bundle.css}, displayModeFull));_WidgetManager._RegisterWidget(_BlogArchiveView, new _WidgetInfo(BlogArchive1, footer-1, document.getElementById(BlogArchive1), {languageDirection: ltr, loadingMessage: Loading\x26hellip;}, displayModeFull));_WidgetManager._RegisterWidget(_ProfileView, new _WidgetInfo(Profile1, footer-2-2, document.getElementById(Profile1), {}, displayModeFull));_WidgetManager._RegisterWidget(_AttributionView, new _WidgetInfo(Attribution1, footer-3, document.getElementById(Attribution1), {}, displayModeFull));/script>/body>/html>
View on OTX
|
View on ThreatMiner
Please enable JavaScript to view the
comments powered by Disqus.
Data with thanks to
AlienVault OTX
,
VirusTotal
,
Malwr
and
others
. [
Sitemap
]