2012-01-30 02:44:02
Author: Michael Van Veen
From The Dorm To The Monastery
Steve Jobs in his living room
I really enjoyed maciej's post on the various hosting options available to developers these days.
Those of us familiar with the various offerings will find ourselves nodding along with the unique advantages and disadvantages each platform offers.
If you're new to web development and want some good advice on hosting, I highly recommend reading his post, as it comes from someone whose dealt with scalability first hand.
Pinboard started as a small project hosted on a Mac Mini and has since grown up quite a bit!
Incidentally, if you're in the market for a hosted bookmarking service, I highly recommend pinboard. I have nothing but good things to say about it.
The Monastery
I mostly develop in the monastery of Google App Engine these days, but it wasn't too long ago that the photo sharing company I co-founded, Zooomr, was running on co-located hardware.
Sometimes, on dreary days where I find myself wrapping my architecture around App Engine's quirks, I look back on the LAMP stack days longingly. Keeping idempotency in mind within your architecture at all times can be frustrating occaisionally. These platforms often force the developer to build for scalability from day one.
If you've architected your application successfully, then your application will handle increasing load with much less difficulty, but it can often come at the cost of lost flexibility and velocity that a standard LAMP stack can afford.
Overall, I think it is worth the pain. Deploys become stupid simple. You never have to worry about load balancing. Rolling back your application is trivial. The data model is kept in native Python primitives, so working with dates, phone numbers, etc. is much nicer than a standard RBDMS.
Not everything is as simple as it should be, however. Logging (in App Engine) is a blessing and a curse in that logs aggregated automatically, but only kept around temporarily. Getting your logs out is harder than it should be.
Just being able to ride on top of Google's infrastructure is enough reason to consider App Engine seriously. The amount of energy they are putting into the platform is impressive, and it is maturing at a rapid rate to address many of the frustrations people have.
The Dark Ages
While the quirks of the platform-as-a-service is looming in my head, I haven't forgotten the horrors which dedicated hardware can bring.
If you've ever had a RAID controller go haywire and start writing random bits, found yourself hustling over to your co-location center to reboot your box, or been DDoS'ed hours before a release date then you understand why hosted platforms like App Engine and Heroku are so appealing.
Having actual hardware fail on you in catastrophic ways is a good
The Dormitory
"Dormitory" services such as EC2, linode and Rackspace provide more flexibility than PaaS offerings, but the flexibility comes with the cost of the hassle which comes from administrating your own system.
Unless I need a special purpose system configuration or more flexibility, I usually stick to App Engine these days. But it's wonderful having the flexibility that "dormitory" providers offer.
freedom.txt
2012-01-23 02:01:45
Author: Michael Van Veen
freedom.txt
I couldn't login to Disqus for some reason, so I'm linking to Inderpreet's blog post here.
My freedom.txt is available here.
Update: I've created a github repo to make it easy to update the url
lists at the bottom of the various freedom.txt's.
Full text:
Hi, my name is Michael Van Veen and I own this website. I no longer wish to stay Anonymous in the fight against internet censorship and those who want to close down our internet.
An open internet is more important than the interests of nation states. Governments have failed us because of ignorance & corruption. They must be fought.
An open internet is more important than corporations. They have failed us because of their natural greed. They must be changed.
An open internet is more important than security, copyright infringement, terrorism or child pornography. We will not be fooled by the strategies of fear employed by those who wish to censor us.
A closed internet is a tool of oppression. An open internet is a weapon of mass creation. Pick your sides. Create your own freedom.txt on your websites.
Python Option Parsing
2012-01-21 09:28:26
Author: Michael Van veen
Python Option Parsing
Note: These are some quick notes I wrote up on the optparse module a
long time ago. Definitely a work in progress.
Adding Options
Initialization
To create a new option parser just do the following:
from optparse import OptionParser
parser = OptionParser()
Setting A Destination
You can specify which key the option will write to with the dest argument. For example, the following will send results to the 'destination' key.
parser.add_option('--dest', dest='destination')
When you look at the results of parse_args
Specifying Argument Type
The type argument can be set to 'complex', 'choice','float', 'int',
'long', or `'string''.
The optparse module will perform type checking on all types except string,
which is left alone. It will also perform automatic type conversion. Please
consult the documentation() for more information.
parser.add_option('--integer-option', type='int')
Handling Standalone Arguments
Sometimes you need an option which doesn't take any parameters. You can add such an option to your option parser like so:
parser.add_option('--boolean', default=False, action='store_true')
This will set the default parsed option to False, and will set the option to
True when it is invoked on the command line.
Variadic Arguments
The nargs option allows you to specify the number of arguments to include
when inputting a particular option.
parser.add_option('--twoargs', nargs=2)
Grabbing Input
Run the following line to grab the output from your parsed options:
(options, args) = parser.parse_args()
options.dest will be the destination setting you made for a given option by
supplying a dest parameter to the add_option method.
e.g.
-
When you add an option with:
parser.add_option('--some_opt', dest='dest_parameter')
-
...and grab the parsed output by running:
(options, args) = parser.parse_args()
options.dest_parameter will contain your parsed parameter.
Help Output
Printing Help
The optparse module will automattically generate help output for you. You
can add help messages to options by specifying a help argument:
parser.add_option('--help-option' help='this is some help text')
Running your script with the -h or --help options will display help.
Additionally, you can call any OptionParser object's print_help method to
print help output.
parser.print_help()
Setting Usage
You can optionally set the usage string of the help output generated. The usage string will be show up at the top of any of help output that gets printed.
parser.set_usage('Usage: some_script.py [options] [args]')
pydoc.py
2012-01-19 08:56:17
Author: Michael Van Veen
Get Pydoc
Do yourself a favor and try out pydoc.
It isn't the prettiest documentor, but using it as a code browser is a wonderful (albeit minimalistic) experience.
If you ever checked out something with the builtin help function, then you've
used it!
This is the module that implements help. Check out
help(help) and help(pydoc) for more information. ;-)
If you have Python installed then you already have it!
Just do the following to install it into your local path:
Happy code browsing!
Python Class Decorator Notes
2012-01-19 06:19:58
Author: Michael Van Veen
Class(Y) Decorators:
Warning IN PROGRESS. Notes are incomplete and currently have some unchecked facts because my internet was out.
Introduction
I only have seldom occaision to introduce a class decorator into my workflow. Therefore, it is common for me to forget the exact specifics of how class decorators operate.
I think class decorators are still a great addition to the language. It is hard to think of a pattern implemented in Python that isn't either absolutely neccessary or so tremendously useful that living without it seems unbearable.
The class decorator, under the right circumstances, falls neatly into the latter category.
Background/Compatibility
Python class decorators were introduced in PEP 3129 and are supported in Python 2.6 and above.
Why Would I Use A Class Decorator?
Anytime you want to have a method foo which wraps a class like so:
class A:
pass
A = foo(A)
You might want to use a class decorator.
Why Not A Metaclass?
Good questsion. In particular, a class decorator is much more flexible than a
metaclass because you don't have to specify the Class's metaclass with
__meta__.
Rationale
Class Decorators don't allow you to do anything new, but they make it easier to do certain things.
From PEP 3129:
While it is possible to express class decorator-like functionality using metaclasses, the results are generally unpleasant and the implementation highly fragile [3]. In addition, metaclasses are inherited, whereas class decorators are not, making metaclasses unsuitable for some, single class-specific uses of class decorators. The fact that large-scale Python projects like Zope were going through these wild contortions to achieve something like class decorators won over the BDFL.
Wikipedia Blackout
2012-01-17 07:14:02
Author: Michael Van Veen
The Wikipedia Blackout
Wikipedia will be going dark Wednesday. Big props to the Wikimedia Foundation for deciding to pull the trigger and show the Western world what the chilling effects of legislation like SOPA and PIPA are.
From "the three Wikimedia administrators who formally facilitated the community's discussion:"
It is the opinion of the English Wikipedia community that both of these bills, if passed, would be devastating to the free and open web.
Over the course of the past 72 hours, over 1800 Wikipedians have joined together to discuss proposed actions that the community might wish to take against SOPA and PIPA. This is by far the largest level of participation in a community discussion ever seen on Wikipedia, which illustrates the level of concern that Wikipedians feel about this proposed legislation. The overwhelming majority of participants support community action to encourage greater public action in response to these two bills. Of the proposals considered by Wikipedians, those that would result in a blackout of the English Wikipedia, in concert with similar blackouts on other websites opposed to SOPA and PIPA, received the strongest support.
On careful review of this discussion, the closing administrators note the broad-based support for action from Wikipedians around the world, not just from within the United States. The primary objection to a global blackout came from those who preferred that the blackout be limited to readers from the United States, with the rest of the world seeing a simple banner notice instead. We also noted that roughly 55% of those supporting a blackout preferred that it be a global one, with many pointing to concerns about similar legislation in other nations.
I posted this HN thread asking people what they're planning to do, if anything, in light of this news.
One great way to circumvent censorship is by archiving the data in question for your personal use. You can check out a great guide on how to set up your own wikipedia archive here.
I find Wikipedia tremendously useful for finding a quick reference defintion on a topic subject, or as a resource for good external references to find more information.
Having this indepsensible resource go dark all of a sudden would be a tremendously pitiful waste of precious human effort, collaboration, and organization. Wikipedia is a gem.
This exercise has helped me realize the importance of fighting against legislation like SOPA and PIPA, and helping to raise awareness about the consequences our society will have in such a world.
As technologists, I think we all have a duty to inform the public and try to do what we can to encourage decentralization as a means to mitigate the consequences of legislation from people who don't understand, or aren't interested in the consequences of the laws they put into motion.
I'm planning to setup a true Wikipedia mirror on app engine in the near future, but no promises (I am thinking I might do some image processing or programming languages in my spare time instead).
For more information about SOPA & PIPA, check out the EFF website.
Ctrlp.vim
2012-01-05 09:42:16
Author: Michael Van Veen
Ctrl-p
Ctrlp.vim is a super sexy new plugin for vim. It blows any other fuzzy-finder/file-explorer/magic-file-wand I've used out of the water (with the possible exception of NERDTree, but the two plugins go great together).
Hopefully this will kill my habit of typing :Ex all the time.
It seems to have a boatload of features I've yet to completely explore, but so far I really like:
- Using
<c-y>to create a new file from a specified path (with any neccessary folders) - Fuzzy-finding that just works.
- Marking/opening multiple files with
<c-z>and<c-o>, respectively
If you use pathogen, you can install it with:
$ cd ~/.vim/bundle
$ git clone https://github.com/kien/ctrlp.vim.git
You can install it using the pathogen package manager or by
copying over all the content to your .vim directory directory.
If you don't have pathogen yet, I highly reccomend it. It's the package manager for vim that makes managing vim plugins not awful.
You can install it fresh from the github it's github repo with two easy commands (borrowed from the pathogen website).
$ mkdir -p ~/.vim/autoload ~/.vim/bundle; \
$ curl -so ~/.vim/autoload/pathogen.vim \
https://raw.github.com/tpope/vim-pathogen/HEAD/autoload/pathogen.vim
Next, add this to your vimrc:
call pathogen#infect()
You should be ready to rock.
mvv.io is now live!
2012-01-03 01:35:27
Author: Michael Van Veen
mvv.io is now live!
My trendy new three letter .io domain name, mvv.io, is
finally live after a few minor hitches. Setting up app engine with custom DNS
is some trick!
Please Note: In order to setup a custom domain, you must have the domain set as your primary account in the Google Apps panel.
This handly little piece of knowledge is freely availble to anyone clever enough to be able to find this gem of a StackOverflow thread, where you can find a lot of good knowledge about how to set up custom root-level DNS entries for your app (i.e. you don't have to specify a specific domain name).
I also had to add A records for Google Apps in order to route the naked URL to
www.mvv.io in order to get the routing to work right.
I couldn't find this info anywhere in the goole apps docs!
Blog Architecture
2012-01-01 08:18:06
Author: Michael Van Veen
Warning: this is a work in progress. Some of the proposed features have not been implemented, may not be final, and should be interpreted as speculatory.
Blog Architecture
Dropblog is a tiny Dropbox CMS I wrote while recovering from pneumonia. It runs on top of App Engine, is inspired by Jekyll, but isn't statically generated.
It reads yaml-compliant metadata, parses markdown posts, has a built-in JSON API, and is generally pretty badass.
What's more, it's hosted entirely from your Dropbox folder!
Just copy over posts and static content to Dropbox and relax. Your changes will automatically deploy.
Remember the days when writing this sort of CRUD meant implementing a heavyweight RDBMS solution? What the hell were we smoking?
Plain text and flat files keep things simple, stupid. The very unix foundation we stand on has plain text publishing firmly in its roots. From Kerika's blog:
Typesetting is closely tied to the history of Unix, and, indeed, provided the raison detre for Unixs existence. In 1971, when Ken Thompson and Dennis Ritchie (and others) at Bell Labs wanted to get funding for developing the Unix operating system, their business case was based upon the rather tenuous argument that developing this new operating system (Unix) would help them develop a better typesetting program (troff), which could be used by Bell Labs to file patents.
If it's good enough for Ken and Dennis then it's good enough for me.
Architecture In A Nutshell
File structure
The basic Dropbox file structure just contains posts and templates. You can see an example below:
.
|-- Icon
|-- posts
| |-- blog_architecture.md
| `-- first_post.md
`-- templates
|-- blog.html
|-- footer.html
|-- header.html
|-- index.html
`-- posts.html
The posts/ directory contains all the blog entries that go live to the site.
Similarly, the templates/ directory is where the rest of the CMS pages are
kept.
Routing
Routing Posts
URLs for posts follow a date convention.
For example, this posts URL is /2012/01/first-post. I looked around at all
the blogs I could find in half an hour and it seemed to be a pretty consistent
URL naming pattern.
The title (i.e the first-post part of the above URL) is gleaned from this
post's metadata front matter.
Template Routing
With a few important exceptions, URLs in templates map to their static equivalents with the file extension dropped.
e.g. templates/about.html will be visible at /about off of the URL root.
Posts
Markdown
I use Markdown at work for just about all the things, and it is good. I think Markdown is a wonderfully simple syntax and I have it burned into my brain.
It seems pretty ubiquitous these days, it's simple, aesthetically pleasing (one persons's opinion), and mostly just gets out of the way.
What more could you ask for in a markup language?
YAML Front Matter
Our YAML Front Matter works nearly identically to how the front matter [works in
Jekyll][frontmatter]. The main difference is that we seperate the post from
the front matter with a double-newline instead of triple --- quotes.
We might optionally support this syntax in the future if we find that we need double newlines in our YAML.
What's the Deal With Dropbox?
What, you got a problem with Dropbox I for one like having my files
synced, encrypted, and archived seamlessly and welcome our new cloud storage
overlords.
I am well aware that Dropbox is no silver bullet. I even have an angry rant on HN to prove it. But I think blogging and small site hosting is the perfect candidate for a Dropbox-based architecture.
We aren't talking about a massively complex, distributed use case rife with dreadful concurrency issues. We're talking about a fucking homebrewed personal blog.
This shit ain't rocket science, folks. I'm qualified to say that. I once made a CMS for rocket scientists. But that's another story.
First Post
2011-12-30 03:09:13
Author: Michael Van Veen
First Post!
I've been bed-ridden with pneumonia for the last three days and I am itching to code something. Why not a blog engine?
When I was adolescent and more naive, I loved dreaming up blog architectures as I was figuring out the basics of web development. There must be something irresistable about blog engines, because it seems like just about everyone has written one.
Someone wittier than me once wrote that "rolling your own" blog engine should be a sort of rite of passage for hackers, akin to making your own light saber.
They're probably right.
What's more, it's getting so much eaier to do these days. I think a big
motivator behind this new blog is simply that the tech behind it is so
damn small now. If you're interested, I have the gory details in another
post, but it's mostly just a small layer on top of Dropbox.
I can stay at home in my vim cave and work solely with flat files and the changes are automagically pushed live.
Why A Blog?
I want somewhere to post all these thoughts I have all the time in a place that isn't Facebook, Twitter, G+, etc. We're rapidly eroding the common grounds of the Internet and we're going to suffer because of it.
Also, I want to practice my writing. A blog seems like a great fit.
But Why A Blog Now?
What, a guy can't start a blog out of the blue? For what it's worth, I've got some projects in the works and want a space to write about them. Also, as I said earlier I have a million thoughts I need to archive somewhere, so I think it makes a lot of sense.