Help
RSS
API
Feed
Maltego
Contact
Domain > www.kr41.net
×
More information on this domain is in
AlienVault OTX
Is this malicious?
Yes
No
DNS Resolutions
Date
IP Address
2025-09-29
3.5.139.75
(
ClassC
)
2025-11-01
3.5.137.110
(
ClassC
)
Port 80
HTTP/1.1 200 OKx-amz-id-2: GSNe2TYi8XvGlA+KCf86FcZrPQWekYFO4G+tvLc7CW+47BJyOc9Qk9ZBgKKkXlyvuTKaFj+x/Ewx-amz-request-id: E1MMZENZE1JK8DHCDate: Sat, 01 Nov 2025 19:39:46 GMTLast-Modified: Tue, 17 Oct 2017 16:20:44 GMTETag: deca80f29aa82e51739e3b4a1508fd59Content-Type: text/htmlContent-Length: 77391Server: AmazonS3 !DOCTYPE html>html langen-us>head> meta charsetutf-8> meta http-equivX-UA-Compatible contentIEedge,chrome1> meta nameviewport contentwidthdevice-width> title>kr41/title> meta namedescription contentDmitry Vakhrushev about programming and related things> meta namewot-verification content2fa7df96f65f8ede9fee> link relstylesheet href/assets/css/styles.css> link relstylesheet href//fonts.googleapis.com/css?familyNoto+Serif:400,400italic,700,700italic|Droid+Sans+Mono> link relshortcut icon href/favicon.ico /> meta namerobots contentnoindex, follow> link relalternate typeapplication/rss+xml titleRSS hrefrss.xml> link relprev titleOlder href/older-1.html>/head>body> div classsite-container itemscope itemtypehttp://schema.org/Blog> header classsite-header> h1 itempropname>kr41/h1> div itempropalternativeHeadline>Dmitry Vakhrushev about programming and related things/div> hr> nav classmain-menu> span classmenu-item> a href/ classnav>Blog/a> /span> span classmenu-item> a href/about.html classnav>About/a> /span> span classmenu-item> a href/hireme.html classnav>Hire Me/a> /span> span classmenu-item> a href/archive.html classnav>Archive/a> /span> span classmenu-item> a href/tags/ classnav>Tags/a> /span> span classmenu-item> a href/rss.xml classnav>RSS/a> /span> /nav> /header> section classsite-content> nav classsiblings> small> span classprevious>a href/older-1.html>← Older/a>/span> /small> /nav> article itempropblogPost itemscope itemtypehttp://schema.org/BlogPosting> header> h1 itempropname>a itempropurl href/2016/05-03-shit_driven_development.html>Shit Driven Development/a>/h1> div classdate> small> time itempropdatePublished datetime2016-05-03>May 03, 2016/time> /small> /div> /header> div itemproparticleBody> p>strong>TL;DR/strong> Software developer should cover each fault case with appropriateerror handler, even if the case is impossible. Because even impossible caseworks sometimes. Because shit happens./p>p>Let’s see an example./p>p>I develop an authentication system based on a classreference external hrefhttps://jwt.io/>JSON Web Tokens/a>. There arefollowing steps./p>ol classarabic simple>li>Client application sends device information to backend./li>li>Backend saves the information into database and issues a device token./li>li>Client sends the token with user credentials on login./li>li>Backend decodes the token and binds device to the user./li>/ol>p>There is an impossible scenario. Backend successfully decodes device token,but cannot find device information in the database. This scenario is impossible,because the token issued only after successful database write, moreoverit contains a device ID generated by the database. And client cannotforge the token (theoretically) because it doesn’t have a cryptographic key.So the case is impossible, and I don’t have to cover it by special error handler.Correct? No./p>p>Let’s see what could happen here./p>ol classarabic simple>li>Backend saves a device and issues a token./li>li>Database corrupts./li>li>Administrator rollbacks the database to a previous snapshot, that doesn’tcontain the device information./li>li>Client owns the valid device token, but the device information doesn’texist in the database. The impossible case works!/li>/ol>p>If I don’t cover the case by an error handler, the backend will return a vague“500 Internal Server Error.” But it isn’t the server error, it’s the client error,because the client sends invalid token. And the backend must inform itby an appropriate error code. So the client will be able to throw awayinvalid token and reregister the device, instead of showing useless error message./p>p>Therefore, adding error handlers for impossible fault cases increasessustainability of the system./p>p>You can say: “Well, all correct, but it happens so rare. Why do we needto care about it? These efforts will never pay off.” And you won’t be right.It happens much more frequently than you expect. People are optimists.We suck at estimating risks. Every time we thinkabout something bad, we think it won’t happen, at least with us. A lot ofpeople dies every day of lung cancer and atherosclerosis. But a lot ofpeople keeps on smoking and eating fast-food. They are optimists.Every day we stumble with poorly developed software, and keep on developfragile systems, because we’re optimists too. We all think that shit won’thappen with us, it could happen with someone else. But it isn’t true.The true is that shit will definitely happen with us./p>p>Here is my top three./p>ol classarabic simple>li>Firefox had been updated and switched off incompatible extensions.LastPass was one of them. It had happened when I had to pay a bill.So I wasn’t able to login into online bank, because of browser update!Needless to say I don’t use Firefox anymore./li>li>Trello lost connection to its server and had silently lost the changesI made on a board. Communication between my teammates was broken./li>li>Twitter CDN was dead, and my browser wasn’t able to load JavaScript.Thus I wasn’t able to write a tweet for two days. Nobody got harmed, butit wasn’t good anyway./li>/ol>p>Devil in the details. You program could work well in the ideal conditions,but remember that there are no ones. So, next time, when you develop software,please, switch you brain in paranoid mode. It will help your system to be robustand the world to be better./p>p>P.S. Hey, look mom, I’ve invented a cool buzzword!/p> /div> footer> small> span classdisqus_thread> span classfa fa-comments>/span> a hrefhttp://www.kr41.net/2016/05-03-shit_driven_development.html#disqus_thread>Leave a comment/a> /span> span classtags> span classfa fa-tags>span classicon-alt>Tags:/span>/span> span itempropkeywords> a href/tags/shit_driven_development/>Shit Driven Development/a> a href/tags/sustainability/>Sustainability/a> a href/tags/error_handler/>Error handler/a> /span> /span> /small> /footer> /article> article itempropblogPost itemscope itemtypehttp://schema.org/BlogPosting> header> h1 itempropname>a itempropurl href/2016/03-23-dont_inherit_python_builtin_dict_type.html>Don’t inherit Python built-in dict type/a>/h1> div classdate> small> time itempropdatePublished datetime2016-03-23>March 23, 2016/time> /small> /div> /header> div itemproparticleBody> p>I have been asked to interview Python programmers for our team recently.And I gave them a task—implement dictionary-like structure tt classdocutils literal>Tree/tt> withthe following features:/p>pre classcode pycon literal-block>span classn>/span>span classgp>>>> /span>span classn>t/span> span classo>/span> span classn>Tree/span>span classp>()/span>span classgp>>>> /span>span classn>t/span>span classp>/span>span classs>a.x/span>span classp>/span> span classo>/span> span classmi>1/span>span classgp>>>> /span>span classn>t/span>span classp>/span>span classs>a.y/span>span classp>/span> span classo>/span> span classmi>2/span>span classgp>>>> /span>span classn>t/span>span classp>/span>span classs>b/span>span classp>/span>span classs>x/span>span classp>/span> span classo>/span> span classmi>3/span>span classgp>>>> /span>span classn>t/span>span classp>/span>span classs>b/span>span classp>/span>span classs>y/span>span classp>/span> span classo>/span> span classmi>4/span>span classgp>>>> /span>span classn>t/span> span classo>/span> span classp>{/span>span classs>a.x/span>span classp>:/span> span classmi>1/span>span classp>,/span> span classs>a.y/span>span classp>:/span> span classmi>2/span>span classp>,/span> span classs>b.x/span>span classp>:/span> span classmi>3/span>span classp>,/span> span classs>b.y/span>span classp>:/span> span classmi>4/span>span classp>}/span>span classgo>True/span>span classn>/span>span classgp>>>> /span>span classn>t/span>span classp>/span>span classs>a/span>span classp>/span> span classo>/span> span classp>{/span>span classs>x/span>span classp>:/span> span classmi>1/span>span classp>,/span> span classs>y/span>span classp>:/span> span classmi>2/span>span classp>}/span>span classgo>True/span>span classnb>/span>span classgp>>>> /span>span classnb>list/span>span classp>(/span>span classn>t/span>span classo>./span>span classn>keys/span>span classp>())/span>span classgo>a.x, a.y, b.x, b.y/span>span classnb>/span>span classgp>>>> /span>span classnb>list/span>span classp>(/span>span classn>t/span>span classp>/span>span classs>a/span>span classp>/span>span classo>./span>span classn>keys/span>span classp>())/span>span classgo>x, y/span>/pre>p>“It’s quite simple task,” you may think at a glance. But it isn’t, in fact it’stricky as hell. Any implementation has its own trade-offs and you can neverclaim that one implementation better another—it depends on context.There is also a lot of corner cases that have to be covered with tests.So I expected to discuss such tricks and trade-offs on the interview. I think,it is the best way to learn about interviewee problem solving skills./p>p>However, there is one line of code that gives away bad solution./p>pre classcode Python literal-block>span classk>class/span> span classnc>Tree/span>span classp>(/span>span classnb>dict/span>span classp>):/span>/pre>p>Inheritance from built-in tt classdocutils literal>dict/tt> type. Let’s see why you shouldn’t do thatand what you should do instead./p>p>Python dictionary interface has number of methods that seems to use one another.For example, reading methods:/p>pre classcode pycon literal-block>span classn>/span>span classgp>>>> /span>span classn>d/span> span classo>/span> span classp>{/span>span classs>x/span>span classp>:/span> span classmi>1/span>span classp>}/span>span classgp>>>> /span>span classn>d/span>span classp>/span>span classs>x/span>span classp>/span>span classgo>1/span>span classn>/span>span classgp>>>> /span>span classn>d/span>span classo>./span>span classn>get/span>span classp>(/span>span classs>x/span>span classp>)/span>span classgo>1/span>span classn>/span>span classgp>>>> /span>span classn>d/span>span classp>/span>span classs>y/span>span classp>/span> span classc># ``__getitem__`` raises KeyError for undefined keys/span>span classgt>Traceback (most recent call last):/span> File span classnb>"<stdin>"/span>, line span classm>1/span>, in span classn><module>/span>span classgr>KeyError/span>: span classn>y/span>span classn>/span>span classgp>>>> /span>span classn>d/span>span classo>./span>span classn>get/span>span classp>(/span>span classs>y/span>span classp>)/span> span classc># whereas ``get`` returns None/span>span classgp>>>> /span>span classn>d/span>span classo>./span>span classn>get/span>span classp>(/span>span classs>y/span>span classp>,/span> span classmi>2/span>span classp>)/span> span classc># or default value passed as second argument/span>span classgo>2/span>/pre>p>So you can expect that tt classdocutils literal>dict.get()/tt> method is implemented like this:/p>pre classcode Python literal-block>span classk>def/span> span classnf>get/span>span classp>(/span>span classbp>self/span>span classp>,/span> span classn>key/span>span classp>,/span> span classn>default/span>span classo>/span>span classbp>None/span>span classp>):/span> span classk>try/span>span classp>:/span> span classk>return/span> span classbp>self/span>span classp>/span>span classn>key/span>span classp>/span> span classk>except/span> span classne>KeyError/span>span classp>:/span> span classk>return/span> span classn>default/span>/pre>p>And you can also expect that overriding tt classdocutils literal>dict.__getitem__()/tt> behavior you willoverride tt classdocutils literal>dict.get()/tt> behavior too. But it doesn’t work this way:/p>pre classcode pycon literal-block>span classk>/span>span classgp>>>> /span>span classk>class/span> span classnc>GhostDict/span>span classp>(/span>span classnb>dict/span>span classp>):/span>span classgp>... /span> span classk>def/span> span classnf>__getitem__/span>span classp>(/span>span classbp>self/span>span classp>,/span> span classn>key/span>span classp>):/span>span classgp>... /span> span classk>if/span> span classn>key/span> span classo>/span> span classs>ghost/span>span classp>:/span>span classgp>... /span> span classk>return/span> span classs>Boo!/span>span classgp>... /span> span classk>return/span> span classnb>super/span>span classp>()/span>span classo>./span>span classn>__getitem__/span>span classp>(/span>span classn>key/span>span classp>)/span>span classgp>.../span>span classgp>>>> /span>span classn>d/span> span classo>/span> span classn>GhostDict/span>span classp>()/span>span classgp>>>> /span>span classn>d/span>span classp>/span>span classs>ghost/span>span classp>/span>span classgo>Boo!/span>span classn>/span>span classgp>>>> /span>span classn>d/span>span classo>./span>span classn>get/span>span classp>(/span>span classs>ghost/span>span classp>)/span> span classc># returns None/span>span classgo>>>>/span>/pre>p>It happens, because Python built-in tt classdocutils literal>dict/tt> is implemented on C and itsmethods are independent of one another. It is done for performance, I guess./p>p>So what you really need is tt classdocutils literal>Mapping/tt> (read-only) or tt classdocutils literal>MutableMapping/tt>abstract base classes from a classreference external hrefhttps://docs.python.org/3/library/collections.abc.html>collections.abc/a> module. The classesprovide full dictionary interface based on a handful of abstract methodsyou have to override and they work as expected./p>pre classcode pycon literal-block>span classkn>/span>span classgp>>>> /span>span classkn>from/span> span classnn>collections.abc/span> span classkn>import/span> span classn>Mapping/span>span classgp>>>> /span>span classk>class/span> span classnc>GhostDict/span>span classp>(/span>span classn>Mapping/span>span classp>):/span>span classgp>... /span> span classk>def/span> span classnf>__init__/span>span classp>(/span>span classbp>self/span>span classp>,/span> span classo>*/span>span classn>args/span>span classp>,/span> span classo>**/span>span classn>kw/span>span classp>):/span>span classgp>... /span> span classbp>self/span>span classo>./span>span classn>_storage/span> span classo>/span> span classnb>dict/span>span classp>(/span>span classo>*/span>span classn>args/span>span classp>,/span> span classo>**/span>span classn>kw/span>span classp>)/span>span classgp>... /span> span classk>def/span> span classnf>__getitem__/span>span classp>(/span>span classbp>self/span>span classp>,/span> span classn>key/span>span classp>):/span>span classgp>... /span> span classk>if/span> span classn>key/span> span classo>/span> span classs>ghost/span>span classp>:/span>span classgp>... /span> span classk>return/span> span classs>Boo!/span>span classgp>... /span> span classk>return/span> span classbp>self/span>span classo>./span>span classn>_storage/span>span classp>/span>span classn>key/span>span classp>/span>span classgp>... /span> span classk>def/span> span classnf>__iter__/span>span classp>(/span>span classbp>self/span>span classp>):/span>span classgp>... /span> span classk>return/span> span classnb>iter/span>span classp>(/span>span classbp>self/span>span classo>./span>span classn>_storage/span>span classp>)/span> span classc># ``ghost`` is invisible/span>span classgp>... /span> span classk>def/span> span classnf>__len__/span>span classp>(/span>span classbp>self/span>span classp>):/span>span classgp>... /span> span classk>return/span> span classnb>len/span>span classp>(/span>span classbp>self/span>span classo>./span>span classn>_storage/span>span classp>)/span>span classgp>.../span>span classgp>>>> /span>span classn>d/span> span classo>/span> span classn>GhostDict/span>span classp>(/span>span classn>x/span>span classo>/span>span classmi>1/span>span classp>,/span> span classn>y/span>span classo>/span>span classmi>2/span>span classp>)/span>span classgp>>>> /span>span classn>d/span>span classp>/span>span classs>ghost/span>span classp>/span>span classgo>Boo!/span>span classn>/span>span classgp>>>> /span>span classn>d/span>span classo>./span>span classn>get/span>span classp>(/span>span classs>ghost/span>span classp>)/span>span classgo>Boo!/span>span classn>/span>span classgp>>>> /span>span classn>d/span>span classp>/span>span classs>x/span>span classp>/span>span classgo>1/span>span classnb>/span>span classgp>>>> /span>span classnb>list/span>span classp>(/span>span classn>d/span>span classo>./span>span classn>keys/span>span classp>())/span>span classgo>y, x/span>span classnb>/span>span classgp>>>> /span>span classnb>list/span>span classp>(/span>span classn>d/span>span classo>./span>span classn>values/span>span classp>())/span>span classgo>1, 2/span>span classnb>/span>span classgp>>>> /span>span classnb>len/span>span classp>(/span>span classn>d/span>span classp>)/span>span classgo>2/span>/pre>p>Type checking also works as expected:/p>pre classcode pycon literal-block>span classnb>/span>span classgp>>>> /span>span classnb>isinstance/span>span classp>(/span>span classn>GhostDict/span>span classp>(),/span> span classn>Mapping/span>span classp>)/span>span classgo>True/span>span classnb>/span>span classgp>>>> /span>span classnb>isinstance/span>span classp>(/span>span classnb>dict/span>span classp>(),/span> span classn>Mapping/span>span classp>)/span>span classgo>True/span>/pre>p>P.S. You can see a classreference external hrefhttps://bitbucket.org/kr41/configtree/src/tip/configtree/tree.py>my own implementation/a> of the task in the sources of a classreference external hrefhttp://configtree.readthedocs.org/en/latest/>ConfigTree/a>package. As I said above, it isn’t perfect, it’s just good enoughfor the context it is used in. And its a classreference external hrefhttps://bitbucket.org/kr41/configtree/src/tip/tests/tree_test.py>tests/a>... well, I have no idea whathappens there now. I just don’t touch them./p> /div> footer> small> span classdisqus_thread> span classfa fa-comments>/span> a hrefhttp://www.kr41.net/2016/03-23-dont_inherit_python_builtin_dict_type.html#disqus_thread>Leave a comment/a> /span> span classtags> span classfa fa-tags>span classicon-alt>Tags:/span>/span> span itempropkeywords> a href/tags/python/>Python/a> a href/tags/dict/>dict/a> a href/tags/dictionary_interface/>dictionary interface/a> a href/tags/mapping/>Mapping/a> a href/tags/mutablemapping/>MutableMapping/a> /span> /span> /small> /footer> /article> article itempropblogPost itemscope itemtypehttp://schema.org/BlogPosting> header> h1 itempropname>a itempropurl href/2016/02-29-legacy.html>Legacy/a>/h1> div classdate> small> time itempropdatePublished datetime2016-02-29>February 29, 2016/time> /small> /div> /header> div itemproparticleBody> p>The most miserable being in the world is a lost dog. Number two is a programmerwho got legacy. But not the legacy of rich childless dead uncle you never know.No, I mean legacy code of the guy who worked before you. You are smart, youuse agile, test driven development, continuous integration and other cool things...But it doesn’t matter anymore. Because the guy preferred to apply hot fixes onproduction using Vim and SSH. The repository keeps outdated broken code.The live code accidentally crashes, but nothing useful could be found in thelogs. And you have to deal with it. My condolences, you got legacy./p>p>And the most frustrating thing is that you cannot start from scratch, because theproduct is alive, too much time and money had been invested, you know, you aresmart, fix it, please. And then you get paralysis. You have to do something,but you cannot force yourself to start coding. You take a cup of coffee,check your inbox, check for updates on Redit, Facebook, Twitter, then checkyour inbox again, then another cup of coffee, then lunchtime, then updates... andthat never ends. But what the heck? You can write code all day long, and all nightlong. You love it. What happens here? Why you cannot just start?/p>p>The answer is chaos. You don’t know what to do. And you must have a plan./p>ol classarabic simple>li>Explain to your family, that the bad mood you got is not because of them.It’s really important./li>li>Explain to your customer, that you are going to fix stuff, but some workingthings might accidentally be broken. It’s really important too.The paralysis you got includes the fear to break something, because oflack of understanding how things work together./li>li>Create new repository and place the code from production into it. You reallydon’t need to find out, which hot fixes had been applied on the live code andwhy code in the current repository is outdated and broken. Just throw itaway./li>li>Make working development environment. It shall give you some inside ofhow the code works and will help to eliminate the fear./li>li>Make test plan. Some day you will make automated tests. But for now,simple checklist would be enough. It will help you to control process.More control—less fear./li>li>Setup staging environment and continuous integration. No comments, youmust have it./li>/ol>p>At the moment you make these six steps, you will have understandingof what you have to do and how you have to do it. The paralysis will go away,and the confidence will get back. Go ahead, and may the force be with you./p> /div> footer> small> span classdisqus_thread> span classfa fa-comments>/span> a hrefhttp://www.kr41.net/2016/02-29-legacy.html#disqus_thread>Leave a comment/a> /span> span classtags> span classfa fa-tags>span classicon-alt>Tags:/span>/span> span itempropkeywords> a href/tags/legacy/>Legacy/a> a href/tags/paralysis/>Paralysis/a> a href/tags/fear/>Fear/a> /span> /span> /small> /footer> /article> article itempropblogPost itemscope itemtypehttp://schema.org/BlogPosting> header> h1 itempropname>a itempropurl href/2015/09-15-wunderwaffe.html>Wunderwaffe/a>/h1> div classdate> small> time itempropdatePublished datetime2015-09-15>September 15, 2015/time> /small> /div> /header> div itemproparticleBody> p>This story had to be published on 13th September, on Day of the Programmer.But I always forget about this holiday. So I did it again this year./p>p>It happened about 15 years ago. I was a student and worked on a gamewith my friends. The game was multiplayer sci-fi turn-based strategy. There wasno graphics, just text mode. And its gameplay was endless. The game was run onTurbo Pascal on PC with Intel 80386 processor. Golden times!/p>p>Once we had to implement a function that generates names for new weapons.Because of endless gameplay, upgrade process was endless too. So the functionhad to generate a new name for each call.The idea was to join a couple of random prefixes, random string, and random suffix.Prefixes and suffix had to be selected from predefined lists.The prefix list looked like this: hyper, mega, plasma, etc. And the suffix listwas like this: gun, cannon, blaster, rifle, etc. The middle string was randomcombination of consonant syllables./p>p>So the function had been done, and we ran a test. And in the first dozen ofnames, it printed out a name of probably the most powerful weapon in whole knownuniverse: Super Megadick Launcher./p>p>The test had been passed. The game unfortunately had not been finished./p> /div> footer> small> span classdisqus_thread> span classfa fa-comments>/span> a hrefhttp://www.kr41.net/2015/09-15-wunderwaffe.html#disqus_thread>Leave a comment/a> /span> span classtags> span classfa fa-tags>span classicon-alt>Tags:/span>/span> span itempropkeywords> a href/tags/story/>Story/a> a href/tags/day_of_the_programmer/>Day of the Programmer/a> /span> /span> /small> /footer> /article> article itempropblogPost itemscope itemtypehttp://schema.org/BlogPosting> header> h1 itempropname>a itempropurl href/2015/09-06-making_validatation_code_looks_better.html>Making validation code looks better/a>/h1> div classdate> small> time itempropdatePublished datetime2015-09-06>September 06, 2015/time> /small> /div> /header> div itemproparticleBody> p>If you develop applications that use untrusted input, you deal with validation.No matter which framework or library you are using. It is a common task.So I am going to share a recipe that neatly integrates validation layer withbusiness logic one. It is not about what data to validate and how to validatethey, it is mostly about how to make code looks better using Python decoratorsand magic methods./p>p>Let’s say we have a class tt classdocutils literal>User/tt> with a method tt classdocutils literal>login/tt>./p>pre classcode python literal-block>span classk>class/span> span classnc>User/span>span classp>(/span>span classnb>object/span>span classp>):/span> span classk>def/span> span classnf>login/span>span classp>(/span>span classbp>self/span>span classp>,/span> span classn>username/span>span classp>,/span> span classn>password/span>span classp>):/span> span classo>.../span>/pre>p>And we have a validation schema tt classdocutils literal>Credentials/tt>. I use a classreference external hrefhttp://docs.pylonsproject.org/projects/colander/en/latest/>Colander/a>,but it does not matter. You can simply replace it by your favorite library:/p>pre classcode python literal-block>span classkn>import/span> span classnn>colander/span>span classk>class/span> span classnc>Credentials/span>span classp>(/span>span classn>colander/span>span classo>./span>span classn>MappingSchema/span>span classp>):/span> span classn>username/span> span classo>/span> span classn>colander/span>span classo>./span>span classn>SchemaNode/span>span classp>(/span> span classn>colander/span>span classo>./span>span classn>String/span>span classp>(),/span> span classn>validator/span>span classo>/span>span classn>colander/span>span classo>./span>span classn>Regex/span>span classp>(/span>span classs>r^a-z0-9\_\-\.{1,20}$/span>span classp>)/span> span classp>)/span> span classn>password/span> span classo>/span> span classn>colander/span>span classo>./span>span classn>SchemaNode/span>span classp>(/span> span classn>colander/span>span classo>./span>span classn>String/span>span classp>(),/span> span classn>validator/span>span classo>/span>span classn>colander/span>span classo>./span>span classn>Length/span>span classp>(/span>span classnb>min/span>span classo>/span>span classmi>1/span>span classp>,/span> span classnb>max/span>span classo>/span>span classmi>100/span>span classp>),/span> span classp>)/span>/pre>p>Each time you call tt classdocutils literal>login/tt> with untrusted data, you have to validate the datausing tt classdocutils literal>Credentials/tt> schema:/p>pre classcode python literal-block>span classn>user/span> span classo>/span> span classn>User/span>span classp>()/span>span classn>schema/span> span classo>/span> span classn>Credentials/span>span classp>()/span>span classn>trusted_data/span> span classo>/span> span classn>schema/span>span classo>./span>span classn>deserialize/span>span classp>(/span>span classn>untrusted_data/span>span classp>)/span>span classn>user/span>span classo>./span>span classn>login/span>span classp>(/span>span classo>**/span>span classn>trusted_data/span>span classp>)/span>/pre>p>The excessive code is a trade-off for flexibility. Such methods are also canbe called using trusted data. So we can’t just put validation into the methoditself. However, we can bind the schema to the method without loss of flexibility./p>p>Firstly, create tt classdocutils literal>validation/tt> package using the following structure:/p>pre classliteral-block>myproject/ __init__.py ... validation/ __init__.py schema.py/pre>p>Then add the following code into tt classdocutils literal>myproject/validation/__init__.py/tt>(again, usage of a classreference external hrefhttps://github.com/pydanny/cached-property>cached_property/a> is inessential detail, you can use the same decoratorprovided by your favorite framework):/p>pre classcode python literal-block>span classkn>from/span> span classnn>cached_property/span> span classkn>import/span> span classn>cached_property/span>span classkn>from/span> span classnn>./span> span classkn>import/span> span classn>schema/span>span classk>def/span> span classnf>set_schema/span>span classp>(/span>span classn>schema_class/span>span classp>):/span> span classk>def/span> span classnf>decorator/span>span classp>(/span>span classn>method/span>span classp>):/span> span classn>method/span>span classo>./span>span classn>__schema_class__/span> span classo>/span> span classn>schema_class/span> span classk>return/span> span classn>method/span> span classk>return/span> span classn>decorator/span>span classk>class/span> span classnc>Mixin/span>span classp>(/span>span classnb>object/span>span classp>):/span> span classk>class/span> span classnc>Proxy/span>span classp>(/span>span classnb>object/span>span classp>):/span> span classk>def/span> span classnf>__init__/span>span classp>(/span>span classbp>self/span>span classp>,/span> span classn>context/span>span classp>):/span> span classbp>self/span>span classo>./span>span classn>context/span> span classo>/span> span classn>context/span> span classk>def/span> span classnf>__getattr__/span>span classp>(/span>span classbp>self/span>span classp>,/span> span classn>name/span>span classp>):/span> span classn>method/span> span classo>/span> span classnb>getattr/span>span classp>(/span>span classbp>self/span>span classo>./span>span classn>context/span>span classp>,/span> span classn>name/span>span classp>)/span> span classn>schema/span> span classo>/span> span classn>method/span>span classo>./span>span classn>__schema_class__/span>span classp>()/span> span classk>def/span> span classnf>validated_method/span>span classp>(/span>span classn>params/span>span classp>):/span> span classn>params/span> span classo>/span> span classn>schema/span>span classo>./span>span classn>deserialize/span>span classp>(/span>span classn>params/span>span classp>)/span> span classk>return/span> span classn>method/span>span classp>(/span>span classo>**/span>span classn>params/span>span classp>)/span> span classn>validated_method/span>span classo>./span>span classn>__name__/span> span classo>/span> span classs>validated_/span> span classo>+/span> span classn>name/span> span classnb>setattr/span>span classp>(/span>span classbp>self/span>span classp>,/span> span classn>name/span>span classp>,/span> span classn>validated_method/span>span classp>)/span> span classk>return/span> span classn>validated_method/span> span classnd>@cached_property/span> span classk>def/span> span classnf>validated/span>span classp>(/span>span classbp>self/span>span classp>):/span> span classk>return/span> span classbp>self/span>span classo>./span>span classn>Proxy/span>span classp>(/span>span classbp>self/span>span classp>)/span>/pre>p>There are three public objects: tt classdocutils literal>schema/tt> module, tt classdocutils literal>set_schema/tt> decorator,and tt classdocutils literal>Mixin/tt> class. The tt classdocutils literal>schema/tt> module is a container for all validationschemata. Place tt classdocutils literal>Credentials/tt> class into this module. The tt classdocutils literal>set_schema/tt>decorator simply adds passed validation schema to decorating method astt classdocutils literal>__schema_class__/tt> attribute. The tt classdocutils literal>Mixin/tt> class adds proxy object tt classdocutils literal>validated/tt>.The object provides access to the methods with tt classdocutils literal>__schema_class__/tt> attributeand lazily creates their copies wrapped by validation routine. This is how it works:/p>pre classcode python literal-block>span classkn>from/span> span classnn>myproject/span> span classkn>import/span> span classn>validation/span>span classk>class/span> span classnc>User/span>span classp>(/span>span classnb>object/span>span classp>,/span> span classn>validation/span>span classo>./span>span classn>Mixin/span>span classp>):/span> span classnd>@validation.set_schema/span>span classp>(/span>span classn>validation/span>span classo>./span>span classn>schema/span>span classo>./span>span classn>Credentials/span>span classp>)/span> span classk>def/span> span classnf>login/span>span classp>(/span>span classbp>self/span>span classp>,/span> span classn>username/span>span classp>,/span> span classn>password/span>span classp>):/span> span classo>.../span>/pre>p>Now, we can call validated tt classdocutils literal>login/tt> method within a single line of code:/p>pre classcode python literal-block>span classn>user/span> span classo>/span> span classn>User/span>span classp>()/span>span classn>user/span>span classo>./span>span classn>validated/span>span classo>./span>span classn>login/span>span classp>(/span>span classn>untrusted_data/span>span classp>)/span>/pre>p>So what we get: the code is more compact; it is still flexible, i.e. we can callthe method without validation; and it is more readable and self-documenting./p> /div> footer> small> span classdisqus_thread> span classfa fa-comments>/span> a hrefhttp://www.kr41.net/2015/09-06-making_validatation_code_looks_better.html#disqus_thread>Leave a comment/a> /span> span classtags> span classfa fa-tags>span classicon-alt>Tags:/span>/span> span itempropkeywords> a href/tags/architecture/>Architecture/a> a href/tags/python/>Python/a> /span> /span> /small> /footer> /article> article itempropblogPost itemscope itemtypehttp://schema.org/BlogPosting> header> h1 itempropname>a itempropurl href/2015/08-13-how-to-write-good-documentation.html>How to write good documentation/a>/h1> div classdate> small> time itempropdatePublished datetime2015-08-13>August 13, 2015/time> /small> /div> /header> div itemproparticleBody> p>I hate writing documentation, but I have to. Good actual documentation significantlydecreases efforts for introduction new teammates. And of course, nobody woulduse perfect open source code without documentation. So I have written plentyof documents. The most of them are miserable. But I tried to find a wayto make them better. And it seems, I have found the general mistake I did./p>p>Typical documentation consists of three parts:/p>ul classsimple>li>Getting started guide describes main features and principles./li>li>Advanced usage guide describes each feature in details./li>li>Internals or API documentation describes low level things, i.e. particularmodules, classes, and functions. It is usually generated from doc-strings ofthe sources./li>/ul>p>I used to write documentation in the direct order: getting started tutorial,advanced section, and finally internals. Don’t do that. If you want to writegood documentation, you have to write it in the opposite order./p>p>This is how it works. The most important thing of any documentation is cross-linking.When you describe a feature that consists of a number of smaller ones, you haveto link each mention of the smaller feature to its full description. That is whyinternal documentation generated from doc-strings is your foundation. It isquite easy to document particular function or class (lazy developersguess it is enough). So when you describe how the things work together, youcan link mentions of the particular thing to its own documentation, instead ofoverburden the entire description by the details. The same works for gettingstarted tutorial. It must be concise, but there must be links to the fulldescription of each feature it mentions./p>p>There is no magic. This technique just makes documentation writing process more productive and fun.Use it and make your documents better and your users happier./p> /div> footer> small> span classdisqus_thread> span classfa fa-comments>/span> a hrefhttp://www.kr41.net/2015/08-13-how-to-write-good-documentation.html#disqus_thread>Leave a comment/a> /span> span classtags> span classfa fa-tags>span classicon-alt>Tags:/span>/span> span itempropkeywords> a href/tags/documentation/>Documentation/a> /span> /span> /small> /footer> /article> article itempropblogPost itemscope itemtypehttp://schema.org/BlogPosting> header> h1 itempropname>a itempropurl href/2015/08-11-pastedeploy_with_custom_configuration_format.html>PasteDeploy with custom configuration format/a>/h1> div classdate> small> time itempropdatePublished datetime2015-08-11>August 11, 2015/time> /small> /div> /header> div itemproparticleBody> p>a classreference external hrefhttp://pythonpaste.org/deploy/>PasteDeploy/a> is a great tool for managing WSGI applications. Unfortunately,there is no support of configuration formats other than INI-files. a classreference external hrefhttps://github.com/inklesspen/montague>Montague/a>is going to solve the problem, but its documentation is unfinished and saysnothing useful. Hope, it will be changed soon. But if you don’t want to wait,as me do, the following recipe is for you./p>p>Using a classreference external hrefhttp://configtree.readthedocs.org/en/latest/>ConfigTree/a> on my current project, I stumbled with the problem: how toserve a classreference external hrefhttp://docs.pylonsproject.org/projects/pyramid/en/latest/>Pyramid/a> applications (I got three ones) from the custom configuration?Here is how it looks like in YAML:/p>pre classcode yaml literal-block>span classl-Scalar-Plain>app/span>span classp-Indicator>:/span> span classl-Scalar-Plain>use/span>span classp-Indicator>:/span> span classs>"egg:MyApp#main"/span> span classc1># Application local settings goes here/span>span classl-Scalar-Plain>filters/span>span classp-Indicator>:/span> span classp-Indicator>-/span> span classl-Scalar-Plain>use/span>span classp-Indicator>:/span> span classs>"egg:MyFilter#filter1"/span> span classc1># Filter local settings goes here/span> span classp-Indicator>-/span> span classl-Scalar-Plain>use/span>span classp-Indicator>:/span> span classs>"egg:MyFilter#filter2"/span>span classl-Scalar-Plain>server/span>span classp-Indicator>:/span> span classl-Scalar-Plain>use/span>span classp-Indicator>:/span> span classs>"egg:MyServer#main"/span> span classc1># Server local settings goes here/span>/pre>p>The easy way is to build INI-file and use it. The hard way is to make my ownloader. I chose the hard one./p>p>PasteDeploy a classreference external hrefhttp://pythonpaste.org/deploy/#basic-usage>provides public functions/a> tt classdocutils literal>loadapp/tt>, tt classdocutils literal>loadfilter/tt>, and tt classdocutils literal>loadserver/tt>.However, these functions don’t work, because they don’t accept local settings. Only globalconfiguration can be passed into./p>pre classcode python literal-block>span classn>app/span> span classo>/span> span classn>loadapp/span>span classp>(/span>span classs>egg:MyApp#main/span>span classp>,/span> span classn>global_conf/span>span classo>/span>span classn>config/span>span classp>)/span>/pre>p>But the most of PasteDeploy-based applications simply ignore tt classdocutils literal>global_conf/tt>.For example, here is the paste factory of a classreference external hrefhttp://waitress.readthedocs.org/en/latest/>Waitress/a>:/p>pre classcode python literal-block>span classk>def/span> span classnf>serve_paste/span>span classp>(/span>span classn>app/span>span classp>,/span> span classn>global_conf/span>span classp>,/span> span classo>**/span>span classn>kw/span>span classp>):/span> span classn>serve/span>span classp>(/span>span classn>app/span>span classp>,/span> span classo>**/span>span classn>kw/span>span classp>)/span> span classc># global_conf? Who needs this shit?/span> span classk>return/span> span classmi>0/span>/pre>p>I dug around the sources of PasteDeploy and found tt classdocutils literal>loadcontext/tt> function.It is kind of low level private function. But who cares? So here is thesource of loader, that uses the function./p>pre classcode python literal-block>span classkn>from/span> span classnn>paste.deploy.loadwsgi/span> span classkn>import/span> span classn>loadcontext/span>span classp>,/span> span classn>APP/span>span classp>,/span> span classn>FILTER/span>span classp>,/span> span classn>SERVER/span>span classk>def/span> span classnf>run/span>span classp>(/span>span classn>config/span>span classp>):/span> span classk>def/span> span classnf>load_object/span>span classp>(/span>span classn>object_type/span>span classp>,/span> span classn>conf/span>span classp>):/span> span classn>conf/span> span classo>/span> span classn>conf/span>span classo>./span>span classn>copy/span>span classp>()/span> span classn>spec/span> span classo>/span> span classn>conf/span>span classo>./span>span classn>pop/span>span classp>(/span>span classs>use/span>span classp>)/span> span classn>context/span> span classo>/span> span classn>loadcontext/span>span classp>(/span>span classn>object_type/span>span classp>,/span> span classn>spec/span>span classp>)/span> span classc># Loading object/span> span classn>context/span>span classo>./span>span classn>local_conf/span> span classo>/span> span classn>conf/span> span classc># Passing local settings/span> span classk>return/span> span classn>context/span>span classo>./span>span classn>create/span>span classp>()/span> span classn>app/span> span classo>/span> span classn>load_object/span>span classp>(/span>span classn>APP/span>span classp>,/span> span classn>config/span>span classp>/span>span classs>app/span>span classp>)/span> span classk>if/span> span classs>filters/span> span classow>in/span> span classn>config/span>span classp>:/span> span classk>for/span> span classn>filter_conf/span> span classow>in/span> span classn>config/span>span classp>/span>span classs>filters/span>span classp>:/span> span classn>filter_app/span> span classo>/span> span classn>load_object/span>span classp>(/span>span classn>FILTER/span>span classp>,/span> span classn>filter_conf/span>span classp>)/span> span classn>app/span> span classo>/span> span classn>filter_app/span>span classp>(/span>span classn>app/span>span classp>)/span> span classn>server/span> span classo>/span> span classn>load_object/span>span classp>(/span>span classn>SERVER/span>span classp>,/span> span classn>config/span>span classp>/span>span classs>server/span>span classp>)/span> span classn>server/span>span classp>(/span>span classn>app/span>span classp>)/span>/pre>p>But it is not the end. Pyramid comes with its own command a classreference external hrefhttp://docs.pylonsproject.org/projects/pyramid/en/latest/narr/project.html#what-is-this-pserve-thing>pserve/a>, thatuses PasteDeploy to load and start up application from INI-file.And there is an option of the command that makes development fun.I mean tt classdocutils literal>span classpre>--reload/span>/tt> one. It starts separate process with a file monitor thatrestarts your application when its sources are changed. The following codeprovides the feature. It depends on Pyramid, because I don’t want to reinventthe wheel. But if you use another framework, it won’t be hard to write yourown file monitor./p>pre classcode python literal-block>span classkn>import/span> span classnn>sys/span>span classkn>import/span> span classnn>os/span>span classkn>import/span> span classnn>signal/span>span classkn>from/span> span classnn>subprocess/span> span classkn>import/span> span classn>Popen/span>span classkn>from/span> span classnn>paste.deploy.loadwsgi/span> span classkn>import/span> span classn>loadcontext/span>span classp>,/span> span classn>APP/span>span classp>,/span> span classn>FILTER/span>span classp>,/span> span classn>SERVER/span>span classkn>from/span> span classnn>pyramid.scripts.pserve/span> span classkn>import/span> span classn>install_reloader/span>span classp>,/span> span classn>kill/span>span classk>def/span> span classnf>run/span>span classp>(/span>span classn>config/span>span classp>,/span> span classn>with_reloader/span>span classo>/span>span classbp>False/span>span classp>):/span> span classk>def/span> span classnf>load_object/span>span classp>(/span>span classn>object_type/span>span classp>,/span> span classn>conf/span>span classp>):/span> span classn>conf/span> span classo>/span> span classn>conf/span>span classo>./span>span classn>copy/span>span classp>()/span> span classn>spec/span> span classo>/span> span classn>conf/span>span classo>./span>span classn>pop/span>span classp>(/span>span classs>use/span>span classp>)/span> span classn>context/span> span classo>/span> span classn>loadcontext/span>span classp>(/span>span classn>object_type/span>span classp>,/span> span classn>spec/span>span classp>)/span> span classn>context/span>span classo>./span>span classn>local_conf/span> span classo>/span> span classn>conf/span> span classk>return/span> span classn>context/span>span classo>./span>span classn>create/span>span classp>()/span> span classk>def/span> span classnf>run_server/span>span classp>():/span> span classn>app/span> span classo>/span> span classn>load_object/span>span classp>(/span>span classn>APP/span>span classp>,/span> span classn>config/span>span classp>/span>span classs>app/span>span classp>)/span> span classk>if/span> span classs>filters/span> span classow>in/span> span classn>config/span>span classp>:/span> span classk>for/span> span classn>filter_conf/span> span classow>in/span> span classn>config/span>span classp>/span>span classs>filters/span>span classp>:/span> span classn>filter_app/span> span classo>/span> span classn>load_object/span>span classp>(/span>span classn>FILTER/span>span classp>,/span> span classn>filter_conf/span>span classp>)/span> span classn>app/span> span classo>/span> span classn>filter_app/span>span classp>(/span>span classn>app/span>span classp>)/span> span classn>server/span> span classo>/span> span classn>load_object/span>span classp>(/span>span classn>SERVER/span>span classp>,/span> span classn>config/span>span classp>/span>span classs>server/span>span classp>)/span> span classn>server/span>span classp>(/span>span classn>app/span>span classp>)/span> span classk>if/span> span classow>not/span> span classn>with_reloader/span>span classp>:/span> span classn>run_server/span>span classp>()/span> span classk>elif/span> span classn>os/span>span classo>./span>span classn>environ/span>span classo>./span>span classn>get/span>span classp>(/span>span classs>master_process_is_running/span>span classp>):/span> span classc># Pass your configuration files here using ``extra_files`` argument/span> span classn>install_reloader/span>span classp>(/span>span classn>extra_files/span>span classo>/span>span classbp>None/span>span classp>)/span> span classn>run_server/span>span classp>()/span> span classk>else/span>span classp>:/span> span classk>print/span>span classp>(/span>span classs>"Starting subprocess with file monitor"/span>span classp>)/span> span classn>environ/span> span classo>/span> span classn>os/span>span classo>./span>span classn>environ/span>span classo>./span>span classn>copy/span>span classp>()/span> span classn>environ/span>span classp>/span>span classs>master_process_is_running/span>span classp>/span> span classo>/span> span classs>true/span> span classn>childproc/span> span classo>/span> span classbp>None/span> span classk>try/span>span classp>:/span> span classk>while/span> span classbp>True/span>span classp>:/span> span classk>try/span>span classp>:/span> span classn>childproc/span> span classo>/span> span classn>Popen/span>span classp>(/span>span classn>sys/span>span classo>./span>span classn>argv/span>span classp>,/span> span classn>env/span>span classo>/span>span classn>environ/span>span classp>)/span> span classn>exitcode/span> span classo>/span> span classn>childproc/span>span classo>./span>span classn>wait/span>span classp>()/span> span classn>childproc/span> span classo>/span> span classbp>None/span> span classk>if/span> span classn>exitcode/span> span classo>!/span> span classmi>3/span>span classp>:/span> span classk>return/span> span classn>exitcode/span> span classk>finally/span>span classp>:/span> span classk>if/span> span classn>childproc/span> span classow>is/span> span classow>not/span> span classbp>None/span>span classp>:/span> span classk>try/span>span classp>:/span> span classn>kill/span>span classp>(/span>span classn>childproc/span>span classo>./span>span classn>pid/span>span classp>,/span> span classn>signal/span>span classo>./span>span classn>SIGTERM/span>span classp>)/span> span classk>except/span> span classp>(/span>span classne>OSError/span>span classp>,/span> span classne>IOError/span>span classp>):/span> span classk>pass/span> span classk>except/span> span classne>KeyboardInterrupt/span>span classp>:/span> span classk>pass/span>/pre>p>That’s it. Wrap the code with a console script and don’t forget to initializethe logging./p> /div> footer> small> span classdisqus_thread> span classfa fa-comments>/span> a hrefhttp://www.kr41.net/2015/08-11-pastedeploy_with_custom_configuration_format.html#disqus_thread>Leave a comment/a> /span> span classtags> span classfa fa-tags>span classicon-alt>Tags:/span>/span> span itempropkeywords> a href/tags/python/>Python/a> a href/tags/pastedeploy/>PasteDeploy/a> a href/tags/pyramid/>Pyramid/a> a href/tags/configuration/>Configuration/a> a href/tags/configtree/>ConfigTree/a> /span> /span> /small> /footer> /article> article itempropblogPost itemscope itemtypehttp://schema.org/BlogPosting> header> h1 itempropname>a itempropurl href/2015/06-15-about_configtree.html>About ConfigTree/a>/h1> div classdate> small> time itempropdatePublished datetime2015-06-15>June 15, 2015/time> /small> /div> /header> div itemproparticleBody> p>I have just released a classreference external hrefhttp://configtree.readthedocs.org/en/latest/>ConfigTree/a>. It is the longest project of mine.It took more than two and a half years from the first commit to the release.But the history of the project is much longer./p>p>The idea came from “My Health Experience” project. It was a great project Iworked on, unfortunately it is closed now. My team started from a small forumand ended up with a full featured social network. We got a single server atthe start and a couple of clusters at the end. A handful of configuration filesgrew up to a directory with dozens of ones, which described all subsystems inall possible environments. Each module of the project had dozens of callsto the configuration registry. And we developed a special tool to manage thesettings./p>p>This is how it worked. An environment name was a dot-separated string in formattt classdocutils literal>group.subgroup.environment/tt>. For instance, tt classdocutils literal>span classpre>prod.cluster-1.server-1/span>/tt> wasan environment name of the first server from the first cluster of the productionenvironment; and tt classdocutils literal>dev.kr41/tt> was the name of my development environment.The configuration directory contained a tree of subdirectories, where each ofthe subdirectory was named after a part of some environment name. For example:/p>pre classliteral-block>config/ prod/ cluster-1/ server-1/ dev/ kr41//pre>p>The most common configuration options were defined at the root of the tree,the most specific ones—at the leafs. For example, tt classdocutils literal>config/prod/tt> directorycontained files with common production settings; tt classdocutils literal>span classpre>config/prod/cluster-1/span>/tt>—commonsettings for all servers of the first cluster; and tt classdocutils literal>span classpre>config/prod/cluster-1/server-1/span>/tt>—concretesettings for the first server. The files were merged by a loader on startupinto a single mapping object using passed environment name. Some of the commonsettings were overridden by the concrete ones during the loading process.So that we did not use copy-paste in our configuration files. If there was anoption for a number of environments, this option had been defined within groupsettings. There we also post-loading validation, that helped us to use safe defaults.For instance, when each server had to use its own cryptographic key, such keyhad been defined on the group level with an empty default value, which wasrequired to be overridden. So that validator raised an exception onstartup, when it had found this empty value in the result configuration.Because of this we never deployed our application on production with unsafesettings./p>p>The tool was so useful, so when I started to use Python I had tried to find somethingsimilar. Yep, “My Health Experience” had been written on PHP, and it was thelast PHP project I worked on. My search was unsuccessful, and I reinventedsuch tool working on each my project. So I eventually decided to rewriteand release it as an open-source project. And here it is./p>p>I added some flexibility and extensibility to the original ideas. Each stepof configuration loading process can be customized or replaced by your own implementation.It also comes with command line utility program, which can be used to buildconfiguration as a single JSON file. So you can even use it within a non-Pythonproject—JSON parser is all what you need. I hope, the tool is able to solve a lotof problems and can be useful for different kind of projects. Try it outand send me your feedback. As for me, I am going to integrate it into my currentproject right now./p> /div> footer> small> span classdisqus_thread> span classfa fa-comments>/span> a hrefhttp://www.kr41.net/2015/06-15-about_configtree.html#disqus_thread>Leave a comment/a> /span> span classtags> span classfa fa-tags>span classicon-alt>Tags:/span>/span> span itempropkeywords> a href/tags/python/>Python/a> a href/tags/configuration/>Configuration/a> a href/tags/configtree/>ConfigTree/a> /span> /span> /small> /footer> /article> article itempropblogPost itemscope itemtypehttp://schema.org/BlogPosting> header> h1 itempropname>a itempropurl href/2015/06-08-occasional_connectionerror_in_requests.html>Occasional ConnectionError in Requests/a>/h1> div classdate> small> time itempropdatePublished datetime2015-06-08>June 08, 2015/time> /small> /div> /header> div itemproparticleBody> p>a classreference external hrefhttp://docs.python-requests.org/>Requests/a> library is a de facto standard for handling HTTP in Python.Each time I have to write a crawler or REST API client, I know what to use.I have made a dozen of ones during the last couple of years. And each timeI stumbled one frustrating thing. I mean tt classdocutils literal>requests.exceptions.ConnectionError/tt>which is unexpectedly raised with the message tt classdocutils literal>error(111, Connection refused)/tt>after 3–5 hours of client uptime, when remote service works well and staysavailable./p>p>I don’t know for sure why it happens. I have a couple of versions, butessentially they all about unideal world we live in. Connection may die or hang.Highly loaded web server may refuse request. Packets may be lost.Long story short—shit happens. And when it happens, default Requestssettings will not be enough./p>p>So if you are going to make long-live process, which will use some services viaRequests, you should change its default settings in this way:/p>pre classcode python literal-block>span classkn>from/span> span classnn>requests/span> span classkn>import/span> span classn>Session/span>span classkn>from/span> span classnn>requests.adapters/span> span classkn>import/span> span classn>HTTPAdapter/span>span classn>session/span> span classo>/span> span classn>Session/span>span classp>()/span>span classn>session/span>span classo>./span>span classn>mount/span>span classp>(/span>span classs>http:///span>span classp>,/span> span classn>HTTPAdapter/span>span classp>(/span>span classn>max_retries/span>span classo>/span>span classmi>5/span>span classp>))/span>span classn>session/span>span classo>./span>span classn>mount/span>span classp>(/span>span classs>https:///span>span classp>,/span> span classn>HTTPAdapter/span>span classp>(/span>span classn>max_retries/span>span classo>/span>span classmi>5/span>span classp>))/span>/pre>p>tt classdocutils literal>HTTPAdapter/tt> performs only one try by default and raisestt classdocutils literal>ConnectionError/tt> on fail. I started from two tries, and empiricallygot that five ones gives 100% resistance against short-term downtimes./p>p>I am not sure is it a bug or a feature of the Requests. But I never see thatthese default settings are changed in some Requests-based library likeTwitter or Facebook API client. And I got such errors using these librariestoo. So if you are using such library, examine its code. Now you know how tofix it. Thanks Python design, there are no true private members./p>p>Unfortunately, I cannot reproduce this bug (if it is a real bug) in laboratorialconditions for now. So I will be grateful, if somebody suggests me how to./p> /div> footer> small> span classdisqus_thread> span classfa fa-comments>/span> a hrefhttp://www.kr41.net/2015/06-08-occasional_connectionerror_in_requests.html#disqus_thread>Leave a comment/a> /span> span classtags> span classfa fa-tags>span classicon-alt>Tags:/span>/span> span itempropkeywords> a href/tags/python/>Python/a> a href/tags/requests/>Requests/a> a href/tags/connectionerror/>ConnectionError/a> /span> /span> /small> /footer> /article> article itempropblogPost itemscope itemtypehttp://schema.org/BlogPosting> header> h1 itempropname>a itempropurl href/2015/04-05-greenrocket_gets_testing_friendly.html>GreenRocket gets testing friendly/a>/h1> div classdate> small> time itempropdatePublished datetime2015-04-05>April 05, 2015/time> /small> /div> /header> div itemproparticleBody> p>I released a classreference external hrefhttps://pypi.python.org/pypi/GreenRocket/>GreenRocket/a> library on October 2012. It is a dead simple implementationof Observer design pattern, which I use in almost all of my projects.I thought, there was nothing to improve. But my recent project heavily usesthe library. And I get tired to write tests that checks signals. This is howthem look like:/p>pre classcode python literal-block>span classkn>from/span> span classnn>nose/span> span classkn>import/span> span classn>tools/span>span classc># I use Nose for testing my code/span>span classkn>from/span> span classnn>myproject/span> span classkn>import/span> span classn>MySignal/span>span classp>,/span> span classn>some_func/span>span classc># ``MySignal`` inherits ``greenrocket.Signal``/span>span classc># ``some_func`` must fire ``MySignal`` as its side-effect/span>span classk>def/span> span classnf>test_some_func/span>span classp>():/span> span classn>log/span> span classo>/span> span classp>/span> span classc># Create log for fired signals/span> span classnd>@MySignal.subscribe/span> span classc># Subscribe a dummy handler/span> span classk>def/span> span classnf>handler/span>span classp>(/span>span classn>signal/span>span classp>):/span> span classn>log/span>span classo>./span>span classn>append/span>span classp>(/span>span classn>signal/span>span classp>)/span> span classc># Put fired signal to the log/span> span classn>some_func/span>span classp>()/span> span classc># Call the function to test/span> span classc># Test fited signal from the log/span> span classn>tools/span>span classo>./span>span classn>eq_/span>span classp>(/span>span classnb>len/span>span classp>(/span>span classn>log/span>span classp>),/span> span classmi>1/span>span classp>)/span> span classn>tools/span>span classo>./span>span classn>eq_/span>span classp>(/span>span classn>log/span>span classp>/span>span classmi>0/span>span classp>/span>span classo>./span>span classn>x/span>span classp>,/span> span classmi>1/span>span classp>)/span> span classn>tools/span>span classo>./span>span classn>eq_/span>span classp>(/span>span classn>log/span>span classp>/span>span classmi>0/span>span classp>/span>span classo>./span>span classn>y/span>span classp>,/span> span classmi>2/span>span classp>)/span> span classc># ...and so on/span>/pre>p>There are four lines of utility code. And it is boring. So I added helperclass tt classdocutils literal>Watchman/tt> to the library to make it testing friendly. This is howit works:/p>pre classcode python literal-block>span classkn>from/span> span classnn>greenrocket/span> span classkn>import/span> span classn>Watchman/span>span classkn>from/span> span classnn>myproject/span> span classkn>import/span> span classn>MySignal/span>span classp>,/span> span classn>some_func/span>span classk>def/span> span classnf>test_some_code_that_fires_signal/span>span classp>():/span> span classn>watchman/span> span classo>/span> span classn>Watchman/span>span classp>(/span>span classn>MySignal/span>span classp>)/span> span classc># Create a watchman for MySignal/span> span classn>some_func/span>span classp>()/span> span classn>watchman/span>span classo>./span>span classn>assert_fired_with/span>span classp>(/span>span classn>x/span>span classo>/span>span classmi>1/span>span classp>,/span> span classn>y/span>span classo>/span>span classmi>2/span>span classp>)/span> span classc># Test fired signal/span>/pre>p>Just one line of utility code and one line for actual test! I have alreadyrewritten all of my tests. So if you are using the library, it’s time toupgrade. If you don’t, then a classreference external hrefhttps://pypi.python.org/pypi/GreenRocket/>try it out/a>./p> /div> footer> small> span classdisqus_thread> span classfa fa-comments>/span> a hrefhttp://www.kr41.net/2015/04-05-greenrocket_gets_testing_friendly.html#disqus_thread>Leave a comment/a> /span> span classtags> span classfa fa-tags>span classicon-alt>Tags:/span>/span> span itempropkeywords> a href/tags/python/>Python/a> a href/tags/greenrocket/>GreenRocket/a> a href/tags/testing/>Testing/a> /span> /span> /small> /footer> /article> nav classsiblings> small> span classprevious>a href/older-1.html>← Older/a>/span> /small> /nav> /section> footer classsite-footer> hr> div> small> © 2013–2016, span itempropauthor itemscope itemtypehttp://schema.org/Person> a itempropurl relauthor href/about.html>span itempropname>Dmitry Vakhrushev/span>/a> /span> br> Content is licensed under a itemproplicense hrefhttp://creativecommons.org/licenses/by-sa/3.0/deed.en_US>CC BY-SA 3.0/a> /small> /div> /footer> /div> script typetext/javascript src/assets/js/scripts.js>/script>/body>/html>
View on OTX
|
View on ThreatMiner
Please enable JavaScript to view the
comments powered by Disqus.
Data with thanks to
AlienVault OTX
,
VirusTotal
,
Malwr
and
others
. [
Sitemap
]