Good Devs Don’t Like Magic
February 11th, 2012
I recently saw a tweet from someone complaining about Visual Studio database projects:
Visual Studio db projects – sounds like a decent idea until you discover it was designed by people who’ve never built a real-world db app.
It reminded me of our own experience with this technology. We tried using it, but ultimately chose db migrations instead. And while there were many reasons not to choose it, probably the most compelling one was that its approach was too magical.
What Exactly Do You Mean by “Magical”?
Solutions of the “magical” variety go out of their way to hide (obfuscate?) the underlying implementation. They present the developer with a Faustian bargain: I’ll make it easy for you to get things done so long as you don’t question how I work.
VS db project is a classic case study in magical solutions. Its job is to help you manage database changes and it does that by comparing your current schema (i.e. development) against some baseline version (i.e. production). That comparison results in a delta file, which you could then apply to the baseline.
The problem is that it’s very difficult to predict with certainty what will be in the delta file (especially on projects with many database changes). So, to use Visual Studio’s db project, you just have to trust that it always does the right thing. You have to believe in magic.
Ok, But What’s Wrong With Magic?
Well, good devs* don’t like magic. They want to understand exactly what’s happening under the hood. They want to know how the machine got from point A to point B. Most importantly, they want to be in control of their code (instead of the other way around).
Good devs also know that magic comes at a cost. That cost is paid when things don’t work exactly as you expected. Suddenly, you have to dig through a maze of your framework’s abstractions to understand what went wrong.
This happens to a man when the blood that is accumulated can flow out through the penile veins. viagra vs generic Other reasons for weak erection in viagra price http://www.glacialridgebyway.com/windows/Kerkhoven%20Heritage%20Room.html men include excessive intake of such medicinal drugs. But this conception is entirely wrong and this medicine is best viagra in uk glacialridgebyway.com helpful and effective on impotent men. cialis overnight delivery Ranking Member of the Africa Subcommittee is congresswoman Karen Bass, and she was the events honorary host. This is why db migrations are so dev friendly. There’s no mystery here: sequentially numbered files are applied, one by one, to the database. You can see exactly what changes are being made at each step, you can see exactly what version your database is currently at, etc. No magic.
Yes, But Isn’t Everything Magical to Some Extent?
I realize that any abstraction could be considered magical. For instance, a compiler is magical: you blindly rely on it to transform your source into a binary executable.
The thing is, for certain abstractions the tradeoff between transparency and magic makes sense. Sure, you could figure out what exactly the compiler is doing behind the scenes, but why would you want to? (other than to get Vietnam-style flashbacks from your undergrad compilers course).
Furthermore, it’s not always obvious when that tradeoff makes sense.Take CoffeeScript. On one hand, a lot of good devs find it to be a very useful abstraction of JavaScript. On the other hand, some make a compelling case that the cost of obfuscation is too high.
* By the way, I don’t think this is true for all devs. In my experience, mediocre / bad devs are usually a lot more willing to accept magical solutions.
You may also like:
Did you love / hate / were unmoved by this post?
Then show your support / disgust / indifference by following me on
Twitter!
This post got 44 comments so far. Care to add yours?
[…] Good developers don't like solutions which go out of their way to hide the underlying implementation. They want to understand exactly what’s happening under the hood. They want to know how the machine got from point A to point B. Most importantly, they want to be in control of their code (instead of the other way around). Database Read the original post on DZone… […]
Lack of magic is actually the reason why I like CoffeeScript. It’s very clear what the code does and it compiles cleanly to JavaScript. Compare that with things like GWT or Objective-J.
Thank you very much for your comment iain. I agree that, compared to GWT (or Objective J), CoffeeScript is a lot less magical.
This idea is a bit silly. Taken to its fullest conclusion it mean we should program in binary/assembler. New tools aren’t hiding underlying functionality, they are making knowledge of those mechanics irrelevant to the achievement of the goals of the developer. That’s how all the tools of developers have progressed.
Hi Jon, thanks for reading. Your point is well taken: abstraction is often necessary to make progress.
I can not zoom in on your blog on my iPhone unless I use the system zoom.
This means I won’t read your blog, I am not going to squint my way through, or scroll around using the system zoom, nor am I going to save it for later.
For us visually impaired the Internet is full of unreadable websites. It’s sad but I suppose it helps me prune.
Personally I consider the sign of a good developer to be how usable his or her product is. Which arguably covers both magic and accessibility.
Hi Tina, thanks for your comment. I’m sorry that the site doesn’t work well enough for you on an iPhone. I actually use a special theme that is supposed to be optimized for mobile, but it clearly needs more work. Thanks again.
See … too much magic!
I worked at Ticketmaster, but learned nothing to speak of about their database stuff. I learned some stuff.
With LoseThos I thought I might make a professional database server in the beginning. But, I’m stuck with ATA port I/O which is much lower bandwidth than PCI DMA. I no longer dream any professional grade uses for LoseThos, but see it as a simpler linux for user tinkering. I’m sure many people who dive into Linux code find it complicated. Now, that I have a vision–simple code — and I have made decisions to support that.
Until I checked your site, I was convinced you were a Markov Chain bot. What exactly are you trying to say??
Sorry to disappoint dawnmail333, but I’m not eloquent enough to explain myself any more clearly. Thanks for reading anyway!
I think dawmail333 was saying that Terry A Davis was a bot, not you, Alex!
(And I agree: his/her comment is incomprehensible.)
Alex: Very nice article. Regarding Jon’s comment that abstraction is necessary: yes, abstraction is good, but this kind of ‘magic’ is generally unpredictable and leaky abstraction.
Ah, yes I see that now. Good lesson learned about approving blog comments using WordPress’s iPhone app (hard to tell what the comment was in response to). Sorry dawmail333, and again thanks for reading.
Also, thank you very much for your comment Andrew, really appreciate it!
@Jon, Alex acknowledges that some “magical” abstractions are worthwhile.
Perhaps a good way to measure/define the value of a magical abstraction is by comparing the level of abstraction it provides, with the reliability of its output, in terms of either succeeding as desired, or failing in a way that doesn’t require breaking the abstraction.
A compiler provides a large amount of abstraction, and a useful one will consistently succeed, or it will fail for high-level reasons, with clear error output. When your build fails, you don’t need to dig around in machine code to fix the issue, you can read high-level errors and stay in the same language as your project to fix it. Thus, it wins the tradeoff analysis.
A high level database schema management tool like VS provides less abstraction, and yet can fail in ways that require breaking the abstraction and digging around in SQL and/or your database. Someone competent enough to use a compiler is (roughly) competent and aware enough to fix compiler errors, but someone competent enough to click around in a VS db project is not necessarily going to understand its likely failure modes or how to fix them. (Not to mention the possibility of data loss… which immediately adds its own bad flavor to the magic.) Thus, the value provided may not be worth the cost to some developers.
Hi Ben, thank you very much for such a thoughtful reply. I completely agree about the tradeoff you so eloquently described.
Magic == unknown science. It’s the “unknown” part that is troublesome.
Thanks Emanuel!
Magic sometimes works. I mean that the “like” is as serious wording as another anecdotal parts. In fact, in most of times, when complexity of your problem increase, you will have to use some “magic” to do your task done. Maybe all good developers know all magic items, from how compiler generates, code, which assembler and how the branch prediction of a specific CPU works, to how every high level detail component works, but most of them would know just the relevant average of the topics depend on the topic.
When you go on working with higher level platforms like MVC3, Rails (Ruby), or an extensibility framework (i.e. MEF 2) or a simple PLinq statement, I think that is better to trust the magic most of the times, than to really understand all the gory details, that would make us more geeky but not necessarily better programmers.
Using the Visual Studio database project, you can build the changes between version to SQL scripts, which you can review and tweek. You don’t have to go from the project to direct deployment, and that is probably not recommended. Pair this up with TFS, and you have very strong database design and change control system. I’m not sure I agree there is something magical about it other than it works and helps organize your scripts and projects.
Thanks for your comment, Magical Horse. I understand your point, but we found Visual Studio’s approach to be quite problematic (you can find more details here: http://tatiyants.com/database-source-control-revisited/).
You mentioned tweaking the scripts it generates. Well, we found that essentially every script it generated required tweaking. This is problematic if you have to regenerate the script for any reason (like to fix a bug for example). Whatever tweaks you applied to it last time would need to be redone, which is time consuming and error prone.
I respectfully disagree.
The script produced by VSDB projects is an artifact of the build process and not of the project itself. As soon as you start making ad-hoc changes to the SQL you lose the ability to reproduce your build, because the change won’t be part of your project in source control.
And if it isn’t in source control, it doesn’t exist:
http://www.troyhunt.com/2011/05/10-commandments-of-good-source-control.html
Agreed, I’m becoming more hesitant of solutions that use a lot of magic. My favorite is when documentation shows you how easy it is to get something running, and assume everything Just Works. Of course it never does, and then you’re stuck debugging cryptic error messages as you dig through the abstractions. In the end you waste more time learning how they built it than you would using a less magical product.
completely agree, thank you very much for reading Chris.
Oh I see this one all the time, it demos great, but when you actually have to use it for a real world project, things come crashing down.
The worst I’ve seen for this is the “Oracle Application Development Framework”… It demos great, which is how it gets into businesses because the people attending the demos buy the stuff before consulting or evaluating it with their developers… Once you try to use it, the world comes crashing to a halt as nothing works as it’s supposed to unless you follow the rails they show in their demos and limit your products to exactly that.
Hi Jimmy, thank you very much for reading.
Agreed.
I hate it when code acts magically. However I think there’s a finer point to be made. The problem isn’t necessarily the magic itself so much as how hard it is to figure out how it works. or how easy it is to find solutions.
For example. a compiler may be magical but it’s also 99.99% of the time going to either generate accurate compile errors or generate an executable that works. It’s also the case that the final product – the executable is not going to be modified further in most cases. so you rarely care what it produces.
databases are a little different because people will look at the database tables directly and wonder why it does things strangely. DBAs will want to get in there and start applying indexes, renaming things and organize things to optimize queries etc. As soon as that happens the assumptions made to make the magic break down and if it wasn’t documented or flexible enough to handle it then your SOL.
A separate point to make is that the number of abstractions makes a difference too. The human brain can only comprehend so many moving pieces. One of my biggest gripes with Java is that it is typical to get massive stack traces when bugs occur. Wrappers upon abstractions, upon proxies and interfaces. it all compounds into a level of complexity where the value of abstracting things has been lost to the added complexity required to debug.
excellent points, thank you very much for the insightful comment Matt.
The greatest problem with these so-called magic solutions is that they are an enormous risk. Like you say, you have to trust them to do the right thing all the time… and if they don’t, you have no clue how to fix it. Your managers and customers are not going to accept, “it’s Microsoft’s/ Oracle’s/ other vendor’s fault!” as an excuse when an errant tool tears the production system a new one… though usually they are the first to be sold on these “magic” solutions in the first place!
Sorry you found it so magical. Some of us understand it and like it. Database projects are great.
Good to know it’s working for you, thanks for reading!
Thanks for vocalizing what I’ve felt for years. My experience with the “magic” has been that you either naively and blindly accept it or you find another way without it. Magical things provide the illusion of increased productivity and on small or self contained projects it can indeed do just that but when things grow or special cases arise where the magic code goes left, the increased efficiency turns into hours of frustration trying to decipher what is going on.
This is one of the reasons why I jumped ship and moved to open source based projects. I formerly worked on a large, high visibility MS project and grew tired of the magical tools that were supposed to make life easier. Working with open source tools, at least for me, forces me to think much more cleanly about what I am doing. The benefit of this is that when things go wrong I know where to look for the problem.
Great article, I’ll have to check back more often!
Glad you enjoyed it john, thank you very much for reading.
I agree with this: I hate “magic” things, too… but I like Visual Studio in general because it’s so damned helpful all the time: there isn’t an open source development environment which can hold a cande to it. This being said, I use it to develop straight programs, and I never use any of the “magic” tools in it, like the database stuff: to be perfectly honest, I just don’t trust them: I’d rather write my own stuff and know what it does. The other thing with the magic tools is that you then have to write everything else to work the way the tools want to work. I don’t like most of the “helpful” data handling classes: really, I won’t go much higher level than the DataReader, because you start to get locked into having to trust that the inbuilt classes are as efficient and adaptive as you’re ever going to need them to be… and in reality, they’re not. I think VS is an excellent environment, but you have to be aware of when to stop using what it has to offer: using magic tools just because they’re there is like never eating anything but ice cream – yeah – the stuff tastes great, but your body’s going to fall apart if you keep doing it.
Thank you for your comment Dan!
I think I agree with you, and not at the same time. It’s a question of definition…
I like magic. I like it a lot… When I am able to understand how it works, what it does and what it doesn’t do. You could argue that it’s not magical anymore if you understand it… But there is some beauty in magic. Elegance. I agree however that DB projects are in fact magic. But the issue with this technology is that it removes so much control from you that it become practically useless. Lack of control is not magic. It’s uselessness.
I am using some frameworks that are so much magical that I call them “high elves magic”. But they are magic because of a “it just works” that leaves surprised at first, but also because they are elegantly extensible. Good IoC implementations using Autofac are magic. SignalR or NodeJS are magic too. But WE are the Wizards!
Thanks for your comment Jean-Michel. I agree, some frameworks do a pretty good job abstracting away unnecessary detail and in those instances the magic is real.
What about object oriented programming in general? The whole paradigm is in essence “magic” as you describe it – you are actively encouraged to hide your implementation from the consumers of your code! What makes this different from any other magic? Or is OOP “evil” as well?
Hi Ed, I don’t consider OO magical in-and-of-itself, though one can of course write magical code using OO techniques. Abstraction done poorly (leaky, unstable, etc) whether through encapsulation or otherwise is what I was complaining about.
Man, you got that one right. That’s the only reason why I hate those wizards. Unless I get a thorough explanation of every single code, I don’t want to have anything to do with black boxes.
Thank you very much for your comment Phil!
I’ve been having a very hard time wrapping my brain around MVC for this very reason. The way it maps a URL to a controller class and method, then to a view, then down to partial views — much of which seems to be based on naming conventions that Visual Studio isn’t explicit about — is all too magical for my taste. Trying to find the one file that’s rendering one particular part of the page I’m interested in involves wading through many files and directories (or, most often, just running a grep on the whole project looking for some identifier that is hopefully uncommon); and while that could certainly be blamed on the way the team has organized (or, more accurately, failed to organize) the project, it doesn’t help that the framework requires nothing in the code to explicitly identify the file.
And then when you discover that Cassini allows some things in processing URLs that IIS doesn’t, so when some magic works in development but not deployed, it leaves you scratching your head wondering why, when there’s nothing technically wrong with anything *you* did, the magic just doesn’t work anymore.
Very nice article. It worries me, however, that the trend is leaning heavily toward “magic” solutions. I’ve actually worked with “programmers” who develop much of their code using copy, past, and google. That’s a very scary prospect. Many of these same developers think I’m completely nuts for doing things like reading an ISO or delving into the methods taken for specific tools…they consider it a waste of time….but as this article eluded to, I can usually find what breaks and why in my code. I also have a better understanding of the architecture being implemented…
Personally, I’ve been developing since the early 90’s…I don’t miss 90% of the environments. However, I do miss the fact that I knew what every piece of my code was doing. So, am I just the next old generation of developers yelling at the copy-pasters to turn the music down???
Finally…Yes, I like the use of “…” and I use it frequently 😛
-Joe At Work x7
Thank you very much for your comment Joe, glad you enjoyed the post.
Hi, Alex!
I’m the web editor at iMasters, one of the largest developer communities in Brazil. I´d like to talk to you about republishing your article at our site. Can you contact me at rina.noronha@imasters.com.br?
Bests,
Rina Noronha
Journalist – web editor
http://www.imasters.com.br
redacao@imasters.com.br
rina.noronha@imasters.com.br
+55 27 3327-0320 / +55 27 9973-0700