One Amongst a Multitude

Happiness, by Max Ehrmann

2010-09-07T00:18:00.000-07:00

- one of my favorite poems, and a personal philosophy.

To be without desire is to be content.
But contentment is not happiness.
And in contentment there is no progress.
Happiness is to desire something, to work for it, and to obtain at least a part of it.
In the pursuit of beloved labor the busy days pass cheerfully employed, and still nights in peaceful sleep.
For labor born of desire is not drudgery, but manly play.
Success brings hope, hope inspires fresh desire, and desire gives zest to life and joy to labor.
This is true whether your days be spent in the palaces of the powerful or in some little green by-way of the world.

Therefore, while yet you have the strength, cherish a desire to do some useful work in your little corner of the world, and have the steadfastness to labor.
For this is the way to the happy life; with health and endearing ties, it is the way to the glorious life.

Generating all subsets

2010-04-01T00:50:00.000-07:00

is a breeze with itertools

Maiden hands

2010-03-20T16:08:00.000-07:00

Time turns
faster,
slower...
and at times he stood still,
quiet, unmoving.
Like the dawn of winter mornings
in snowy Wisconsin.

When he passes in the hands of a maiden.
--
Tim

Gray

2010-03-14T21:50:00.001-07:00

Black and white, so cut and dried..
unflinchingly looked me in the eye.

"Don't give in, to the heat.
Don't be marching to the beat."

Vows, sworn, under the light.
When all was clear, horizon bright.

On the peaks of destiny,
in the valleys of ignominy,

Seem made to be broken in the numbing grey,
of life's choices that we face.

-- May08
Tim

Over and Over

2010-01-05T21:53:00.001-08:00

Over and over,
I come to forks in the road,
a path splits to two,
sometimes three.

Over and over,
mind indecisive,
past weighing on me,
future unseen.

To go one way,
is to not see the other.
yet to stand at the crossroads,
is to be left without life and memories.

Over and over,
I choose,
to walk on one path,
to open one door, and close the others.

Yet.... over and over,
I move forward.

Sometimes with trepidation,
sometimes with caution,
sometimes with relief,
with joy, with a touch of bitterness, with regret, with prescience, with ignorance,
but forward.

And that has made all the difference.

--
Tim

quote from shimon peres

2009-11-29T22:16:00.000-08:00

The most careful thing is to dare.
- Shimon Peres

Marcus Aurelius's Meditations

2009-11-02T23:33:00.000-08:00

The person who loves reputation supposes that his own good depends on the activity of others; the lover of pleasures finds his own good in being affected by his emotions. But the person who has Intelligence understands the good to be in his own actions.

from dilbert creator

2009-10-19T23:06:00.000-07:00

an artist listens to his inner calling and hopes the public agrees. A business person listens to the audience and gives them what they want

Book of Changes

2009-10-18T00:31:00.000-07:00

Take not gain or loss to heart.
What man holds high comes to nothing.

Do not waste even a moment on gains and losses when death is plucking your ears saying, "Live! I am coming."

ayn rand -hope

2009-10-17T19:12:00.000-07:00

Do not let your fire go out, spark by irreplaceable spark, in the hopeless swamps of the approximate, the not-quite, the not-yet, the not-at-all. Do not let the hero in your soul perish in lonely frustration for the life you deserved, but have never been able to reach. Check your road and the nature of your battle. The world you desired can be won. It exists, it is real, it is possible, it is yours.

Your Work...

2009-08-02T23:51:00.001-07:00

your work is to keep cranking the flywheel that turns the gears
that spin the belt in the engine of belief that keeps you and your desk in midair

-Ann Dillard in the Writing Life

Clojure recursive accumulator

2009-07-22T16:49:00.000-07:00

I'm trying to extract all values with a certain key in an arbitrarily nested value in clojure.
I couldn't find this functionality right off the bat in core/contrib, so I decided to write it, if there's a better way to do this, leave a comment! I'm not completely happy with my implementation.

Ayn Rand - show me your happiness. From The Fountainhead.

2009-07-19T16:24:00.000-07:00

Men have not found the words for it, nor the deed nor the thought, but they have found the music. Let me see that in one single act of man on earth. Let me see it made real. Let me see the answer to the promise of that music. Not servants nor those served; not altars and immolations, but the final, the fulfilled, the innocent of pain. Don't help me or serve me, but let me see it once, because I need it. Don't work for my happiness, my brothers - show me yours - show me that it is possible - show me your achievement - and the knowledge will give me courage for mine.

apply-fnmap - clojure

2009-07-07T20:06:00.000-07:00

I'm not really sure what to call this, but we have to do some data analysis using hadoop, and in order to be efficient, we apply mutiple classifiers at once to the dataset per job.

There might be stuff for this already somewhere in the builtin libs, but I wanted to apply a map of functions to a bunch of data (variable number of them) and have the result be the key and the transformed data/function result.
Here's the src...

My Emacs setup (for clojure, ecb, etc)

2009-07-02T20:28:00.000-07:00

To check out the src, go to github.

So I've been working recently in Clojure with a colleague of mine, and being a Vim user, it's been kind of daunting initially to get started with emacs.

I was looking for a standard install that comes with sane default plugins already installed. I saw ClojureBox, but I wanted it to not be such a black box in case I wanted to put some of my own customizations.

It's a good thing I found Jonas Boner's emacs config on github,
It was a pretty modular setup, and came with some stuff that I wanted like ECB (code browsing).
The third-party plugins are all in a folder, with the startup and loading all in a different one, and you can put an emacs lisp file for whatever you want to happen during emacs startup.

(mapcar 'load-directory
'("$EMACS_LIB/startup"))

I just sanitized and generalized a lot of it, and made the clojure stuff work better, and also grabbed some stuff from Brad Cross's clojure emacs config like clojure-pom, which has slime-project. (It starts with clojure-contrib) It's really easy to get started, go on ahead and take a look at the README to see what I mean. (just 2 steps!)

Some startup keys:
M-x ecb-activate
M-x slime-project (then enter path to project)
C-c C-, (run test-is for clojure on current file)

If You Ever Need to Bootstrap Clojure in a non-Clojure environment

2009-07-01T20:07:00.000-07:00

Read this thread. http://bit.ly/HocNx

I'm using clojure as a wrapper over cascading, and reading this has been really helpful in finding out how clojure works under the hood.

Art of War for Business....

2009-06-26T12:29:00.001-07:00

Turn their strengths against them.
Or find weaknesses in their strengths, or core competencies.

Clojure to Java interop gotchas so far

2009-06-21T18:25:00.000-07:00

1.) When your package name has a dash (-), in the filesystem, it should be an underscore. (_) .

2.) When calling vararg Java methods or constructors: ie.

public transient Fields(java.lang.Comparable... comparables) { /* compiled code */ }

you have to call it by using into-array in clojure.

(Fields. (into-array Comparable ["1" "2" "3"]))

These couple of things just bit me recently, and they took some time to resolve.

Please post more in the comments or in your own blogs if you find any gotchas!

Good words to work by.

2009-06-21T11:01:00.000-07:00

So I have just one wish for you — the good luck to be somewhere where you are free to maintain the kind of integrity I have described, and where you do not feel forced by a need to maintain your position in the organization, or financial support, or so on, to lose your integrity. May you have that freedom.

-Richard Feynman

Lazy Windowed View into collection for clojure

2009-06-20T21:22:00.000-07:00

There's a frequent need in time-series/financial applications and statistical analysis to do a windowed view into a collection of values.
Say you want to compute the mean of every 4 overlapping values of a huge collection/stream, it would look like this:

stream: [1, 2, 3, 4, 5, 6, 7, 8.....n]
result: [(mean [1 2 3 4]) (mean [2 3 4 5]) .... (mean [n-3 n-2 n-1 n])]

you can do this in clojure really easily, in a lazy and memoized way as well to prevent a stackoverflow in case the source stream is too big.

Below is the code:

When dealing with Non-Linear situations

2009-04-04T20:31:00.000-07:00

A few things to remember:

Thought is influenced by emotions, context, values, and motivations. Man is no logical machine.

As time goes on, (as we get used to environment) we become more people of action than reflection.

The problem can sometimes be unintentionally redefined by participants

Helplessness generates cynicism.

Consistency is not a virtue, at least in problem solving.

Watch for secondary effects. (munger)

Set useful goals =)

Hard situation - complexity, lack of transparency, internal dynamics, inadequate understanding of the system.

Learning and discovery flow:
1.) Goal formulation

General goals can be unclear, which trigger "repair service" behavior. (just solve immediate problem, instead of getting to root cause, solving underlying issues)
After "repair service" behavior, sometimes people will change their goal to solving the immediate problem. This happens when there is a desire for safety, or a lack of confidence from the problem-solver.
Beware conflicting goals.

2.) Models

Positive feedback - increase in one, increases the other
Negative feedback - increase in one, decrease in other.
Systems thinking, secondary/third effects

3.) Time Sequence

nonlinear rates of growth
watch out for non-immediate effects, we always overcompensate.

4.) Summary

state goals clearly, we all know we should do that but we rarely encounter the necessity.
can't realize goals all at once because different goals may conflict with each other. need compromise sometimes
establish priorities but can't hang on to them forever. may have to change them
when dealing with a a given configuration, establish a model to deal with it, must anticipate side effects and long term repercussions.
adapt information gathering to task at hand, not go into excessive detail or stop too short.
learn consequences of excessive abstraction.
watch out for hastily ascribing all events in a certain field to one cause.
when to continue gathering information and when to stop.
we sometimes act because we want to prove to ourselves that we can act. watch out for this. No acting for the sake of action/pride/insecurity.
Watch out for knee-jerk methodology/cargo-culting.
draw lessons from errors.

Negatives

acted without prior analysis of the situation
failed to anticipate side effects and long term repercussions.
assumed that the absence of immediately obvious negative effects meant that correct measures have been taken.
let over involvement in projects blind them to emerging needs and changes in situation.
prone to cynical reactions.
very ad-hoc, easily distracted, reactionary
no reflection
blames others for failure
solve problems they can solve rather than what they ought to.
regulate one instance/situation instead of taking a higher level view. (process, etc).

Positives

more decisions. felt they could influence fate more.
more decisions/goal or intention. acted more complexly.
tested their hypothesis.

Planning: Don't Use Forecasts, Instead Use Scenarios and PreMortems

2009-01-15T19:51:00.000-08:00

I'm a little peeved after hearing the word "forecast" for the umpeenth time today. I guess since it's the beginning of the year, it comes with the territory.
My beef with forecasts is that they aim to provide answers and to justify the PLAN or the SPREADSHEET when no one can really know the future. It also usually draws too much on the immediate past. So when the forecast is wrong, it is often wrong in a bad way. Forecasts are also written to be consumed, they don't encourage critical thinking of the different factors and risks that could come into play.
Companies, groups, and even individuals base their plans and expectations on forecasts, (esp. the ones staffed full of MBAs) which makes them blind to whatever signals and signs that they should have been paying attention to. People presenting forecasts often present it with certainty and a crapload of data, giving the impression that they've done their due diligence, when actually what they've done is to retrofit evidence and statistics to justify the status quo (most of the time) or whatever theory they're pushing. (hello consultants)

So given that I think forecasts are useless, what should be used instead?
Well, it can be anything, as long as it encourages the participants to think critically about what could affect their future. It should also encourage a mindset to disprove things, not just to justify them.
One method that I like is premortems, where you imagine that the worst has come true that year, and come up with possible reasons why this might be the case.
This encourages participants to detach emotionally from the plan that they've come up with.
It also helps to have a diverse group of people doing the exercise, so you can approach things from different perspectives and come up with different scenarios.

I think doing this over forecasts gives you most of the value that you want out of planning. You don't plan to have certainty; I think the value in planning is so that you've done a good mental simulation of things that could happen, including the more implausible ones, so that when they crop up, at least you've already given some thought to them.
Doing planning this way also makes you aware of the different signals and factors that could give you an early warning about possible shifts and disruptions, so maybe at the beginning of the year you can set up and automate for certain flags to be thrown to make you aware of them.

Anyhoo, just a little rant about planning meetings, MBAs, and their insanity.

Blindly Right

2008-09-12T00:08:00.000-07:00

I once took pride
in being right,
in knowing truth,
in fooling sight.

But lately I've been thinking...
there's much a cost
to this pride.

Time changes
today's truth.
and to be proud,
is to be blind.

So from today on,
I'll take away the mask of pride.
And promise
to see things from your side.

From HBO's the Wire...

2008-08-21T21:36:00.001-07:00

"You can hold back from the suffering of the world. You have free permission to do so, and it is in accordance with your nature. But perhaps the holding back is the one suffering you could have avoided."

The Wheel Of Time

2008-08-19T22:52:00.001-07:00

(with apologies to Robert Jordan)

"Time is money",
the old adage goes,
so I busied and hurried,
with no thought for repose.

Right out of school,
with Wall street as my shrine,
Hoping to retire,
when I'm thirty-nine.

I worked day and night,
with nary a bite,
to secure financial freedom,
at the cost of my life.

With a house on the lake,
four cars and a yatch,
To have money enough,
to buy a dot on the map.

As the years went by,
they came and went quicker.
Yet my life marched on,
to the beat of the stock ticker.

Until one evening,
with my head full of wine,
the clock struck twelve,
I was forty-nine...

The best years of my life,
have come and gone,
my wife and kids,
strangers they've become.

Love for money,
Lust and greed,
have blinded my eyes,
to what I really need.

What profit did I get,
from losing family and soul,
though I gained the world,
my life was the toll.

Regret and emptiness,
for the years of my life,
forever will be,
companions of mine.

For in eternity,
they could've echoed sublime...
but they've been grinded to dust,
by the wheel of time.

It Takes More Than Hard Work ....

2008-07-22T00:25:00.000-07:00

to deliver good software consistently. (by good, I mean functional defect-free software that delivers business value and is maintainable)

I think there is a tendency for a lot of technical people to not push back on the business, and say yes to everything that gets thrown to them. I used to have this tendency in the past. I always looked at tight schedules and unrealistic expectations as a personal challenge, and having someone question the validity of my "yes, I can do it" answer as an affront to my technical ability. As a result, it creates unrealistic expectations, forcing developers to go into hero mode.

Going into hero mode should be the last resort. It piles on a lot of technical debt, creates silos among the developers in terms of knowledge of the application, and prevents the code from achieving low coupling and high cohesion. It also makes developers go into "firefighting" mode (emergency bug-fixing) all the time, since you don't really have the time to step back and think about what you're doing, or *shudder* write tests to verify your code, which will prevent emergencies in the future.

Organizations with a hero/fire-fighting culture sprints faster in the beginning, and tend to think that things are going well until too late. By all accounts, things are getting done, business needs are being met, and operational metrics are not too far off expectation.
But as the code base grows larger, and the "heros" get tired of the big ball of mud architecture that came out of their "rockstar hacking", it becomes a lot harder for the application the change and evolve along with the needs of the business and the marketplace.

A better culture for writing software is one that promotes bug-prevention, code maintainability, and "getting things smart" over bug-fixing and just plain-old "getting things done".
This sounds really obvious, but a lot of times, it seems to be treated as a nice-to-have instead of something essential to delivering software.
I think this is due to a couple of reasons:
1.) establishing this kind of culture is harder and has more upfront cost.

You need to build up and maintain automated tests to catch errors, developers actually need to think about where they should put their code, instead of having classes called "CommonUtil", "CommonHelper", and their many variants.
Sadly, this kind of culture only seems to exist in technical/software companies, because as far as business people are concerned, they only care that it's cheap and it works, not about the code and the how.
I think a good analogy for this is stock-picking. Some people only look at the financial data sheet and the numbers when picking stocks, while others look at the whole picture; the management, the industry, the company's position in the market, etc.
I'm no expert in stock-picking to say which way is better, so I will defer to Charlie Munger on this one:
"A great business at a fair price is superior to a fair business at a great price."
Similarly, "a well-written and maintained application at a fair price is better than a "well it works" codebase at a great price."

2.) A culture of prevention is more team-oriented, so individual developers don't get the lion's share of the credit for a successful application as much as a "hero" culture.
Having a culture of prevention in software is actually more beneficial to the business than developers.
In a "hero" culture, developers have more job security, because individual developers are typically the only ones who know certain areas of an application. It also promotes knowledge-hoarding, since you want to be the one to fix the next bug instead of the other guy.
In contrast, a culture of prevention is typically more team oriented..(a good description I found was "egoless programming"). You also find most programming mistakes by running tests, instead of having a pile of defects and developers incessantly setting the debugger all over the place.

While having a culture of prevention makes more sense than having a culture of fire-fighting, it's definitely not the norm. I think a lot of organizations have realized that this is the case, but it is really hard to change the culture once it is set.
The best way, in my opinion is to change the way individuals and projects are measured and judged on whether they are successes or failures.
After all, you will only get what you measure and reward.

Career Advice From Charlie Munger

2008-07-20T22:16:00.000-07:00

- Don't sell anything you wouldn't buy yourself.
- Don't work for anyone you don't respect or admire.
- Work only with people you enjoy.

From the book, Poor Charlie's Almanack. I highly, highly recommend reading it.

It sounds so simple but getting simple down and the principles right is the most important, and often the most-ignored part of the way we do things nowadays....

If...

2008-07-19T00:49:00.001-07:00

what a fantastic poem!

People Can Rationalize Anything...

2008-07-16T23:16:00.001-07:00

but having a reason does not make a certain choice the right one.
We all think we make rational decisions based on good information... and all our actions make sense.

The truth is, we are ruled by our emotions, biases, beliefs, and history more than we realize, or might want to admit.

It's hard enough to act on uncertainty, but if we can't take in data and information unfiltered, it makes things even worse. We end up not maximizing our probabilities, and even though we might have a short run at success due to luck; luck, as they say, is a fickle mistress.

The best skill to learn is to first, take in information for the choices we care about without filters and prejudice, then to learn to evaluate that information.

It sounds simple, but it really is so hard. I find myself having set reactions to so many triggers, a lot of them so pavlovian that it borders on the comical.

Cynical or Idealistic?

2008-05-21T16:51:00.000-07:00

Idealistic about strategy (vision). Cynical in terms of execution. (code and process)

A lot of people are either idealistic or cynical, and they take pride in being one and not the other. I think you need to be both.

You need to be idealistic about your vision. It requires many leaps of faith to think big and create new opportunities.

You need to be fully commited to your vision to have any shot at success.

But for code and process, you need to be cynical. Like a colleague of mine says, you need to look at this with a balance sheet.

You have to realize when an asset (a codebase) is worth investing more money into, or when it's time to cut your losses.

At a certain point, a codebase can turn from being an asset to a liability, (once it accrues too much technical debt) especially if it hinders the business from evolving against competition.

Thoughts on Hiring: Why You Shouldn't Judge People Solely On Outcomes

2008-05-16T16:12:00.000-07:00

History and track records are important. They establish credibility and inspire confidence.
We see this in many things: best-selling authors, recommendations in Linkedin, suggestions for doctors from someone you know, or people's resumes and references.
Track records can also be kind of a self-fulfilling prophecy for success; the more successful you were in the past, more people will want to work with you and the more opportunities will come up.

The problem that I see with track records is, most people, especially when desperate for more help, never look beyond it and just take everything people with good track records say at face value, and don't dig any deeper. This is especially dangerous for non-deterministic situations.

Fortune's Role
Track records are made out of outcomes. The thing about outcomes is a lot of it is due to chance. People generally think probability is only important in activities like gambling and finance, but it also plays a very important role in business, software development, new ventures, etc.
A long string of successes often blinds us to luck's effect on those outcomes, and we start feeling an aura of invincibility and treating future ventures with an inevitability of success, which is very dangerous.

Not All Successes are Made Equally
This sounds really obvious. For example, in professional baseball, if I have a hitter at the minor leagues who hits .300, and someone in the majors who hits .290, it's fairly obvious that the 2 are not comparable, and that you have more of an appreciation for the major leaguer's batting skills. (at least I do) Even though it's really obvious, a lot of times we seem to ignore this fact.
Let's say for example, you are interviewing 2 people for a job. The first one has successfully delivered in all 10 of his projects, while the other guy has 4 failed ventures, and 6 successes. Given that information, who would you choose? I think most people will pick guy #1, without any more questions asked.
What they failed to investigate, was that guy #1 has mostly been working on similar looking projects that are easier to deliver, while guy #2 took on some hard stuff early on and took some failures, learned from them, and has since had a higher success rate. Guy #2 also has took on a wide variety of jobs, giving him a lot of perspective on what tradeoffs for the choices that he has had to make. Guy #1 in the meantime, just follows a "tried-and-true" formula for everything that he does. (remember, there are no silver bullets!)

Halo Effect
A corollary to this is people who rationalize their actions by saying, "so-and-so also does it".

It's a common reaction to treat everything that a successful person does as good or worthy of duplication. Successful people have a halo effect. An example was: puzzles as interview questions from Microsoft. Because Microsoft was the hot new success back in the 80's and early 90's, everyone started trying to copy their interviewing techniques. People have since questioned the direct correlation between puzzle-solving and software development skills, but this practice has spread so much that virtually all software developers have encountered puzzle questions in interviews. (I guess maybe if you are programming in Haskell (read Smullyan's to Mock a MockingBird), some sort of logical reasoning skills are important)
The point is, you have to beware of cargo-culters who blindly follow the people with good track records.

What You Should Look For: Process/Approach and an Appreciation for Risk
Process can sometimes be a touchy subject, because to some extent, everyone does things a little differently, and different situations call for slightly different processes.
I like the word approach better, I think there are some common themes that you should look for in a person's reasoning and thinking when they talk about how they do things.

Do they try to get quick feedback?

Feedback gives you data so that you can make more informed decisions. This is especially important when there are a lot of unknowns. You have to drive those unknowns out.

2. Are they "philosophical" about both their failures and successes?

I don't know if philosophical is the right word, but people often think a lot about their failures and try to learn from them and turn them into a positive experience. A lot of people don't really think deeply about their good outcomes; they mostly attribute good things to good strategy and execution, instead of thinking about what could have gone wrong that went right, and whether the decisions that they made were the best, and what part luck and coincidence played in their success.
You want this type of person because they appreciate the fact that success is never a given, and the best chance for succeeding in any non-deterministic task is ability to get feedback, and to act on the feedback when appropriate.
When hiring people with a lot of experience, this is doubly important. You mostly hire experienced people for their insight, and because they should have a lot of different perspectives. But if they don't look back and think/internalize about what they have done, they won't have this. Instead they will mostly likely be set in their ways, and it will be hard to get them to adjust to new things and ideas.

3. Do they take risk into account?

I have always hated platitudes that try to stir up emotion and try to show determination. Sayings like, "Failure is not an option", or "We will prevail", "we will not fail", etc. A lot of people who say these kinds of things seldom have a plan to hedge for risk, and they always flame out spectacularly in the end. People who don't take risk into account are dangerous for you to be associated with, especially if they have a history of successes. Those successes have probably made them too confident in the way they do things, and are too arrogant to admit mistakes and change.

The Question That You Have To Ask:
Which person/choice improves my probabilities the most for what I want to do?
At the end of the day, that is all that matters. Track record is only important in the context that it will improve the odds of future success.
You can't answer this question until you probe and find out about the person's decision-making approach, and his perspective on his previous experiences.

On Choices, and Making Decisions

2008-03-30T19:32:00.000-07:00

People always say that it's good to have choices, and most of the time, they're right.
But sometimes, having to choose can be a curse, especially when there is no obvious option.... it becomes an exercise of weighing tradeoffs and second-guessing yourself. This process can get torturous, and some people go to great lengths to avoid thinking about it. They tend to either pick a choice randomly after pushing off the decision until the latest moment, or go with the popular vote.

For some of the decisions we have to make, (picking which brand to buy, where to eat) we don't really care about what choice we make, but for others, we care a lot. (where to move to, who to marry, what to do for a living, etc.)

I've recently had to make an important decision, and it was hard to pick among the choices. Below are some of the things I did and questions that I asked to come to my choice.

1.) What do you really want out of this decision?
This sounds really obvious, but I did not ask myself this until later, because I was swayed by the popular vote and blinded by the pontential dazzle and prestige of some of the choices. Answering this question gave me a baseline to work with, and ruled out some choices quickly. It also made me realize which benefits were the real meat and potatoes and which ones were just gravy.

2.) What can be realistically expected out of X choice?
A lot of times, we get blinded by potential. The problem with potential, is that of course, there is a big chance that it will go unfulfilled. (as a lot of fantasy sports team players know =)) When making decisions, it's important to take note of what each choice can give us for sure, and which benefits are reaches.

3.) What's the worst that could happen if I make X choice? (downside)
Asking this question helped me to understand and deal with my fears behind each of my choices, and also gave me a clearer picture of my risk tolerance. A lot of times, we are afraid of riskier choices, and other people's advice often steer us to the status quo. What we have to realize is that risky choices and big changes can be good for us at certain points of our lives. Opportunity only knocks a couple of times, and if we don't take it, we will often regret it.

4.) What would the future me on my deathbed think?
A little dramatic, but a very useful question nonetheless. This helps to put things in perspective. It gives us a long-term outlook over what is more convenient now, and often helps us realize what is really important to us.

5.) What does your gut tell you?
This is a little harder to quantify, but we often get a "feeling" about some particular choice. I was hesitant to put this in, because bad things have happened when people followed their gut/heart instead of their head. (stocks, life partners, etc.) I consider this more to be a tie-breaker. If everything else is equal, or pretty close, then this will probably be the thing to push one choice over the top.

6.) Realize that the future is always uncertain.
Sometimes I get paralyzed by some particular choice that I have to make. I drag my feet hoping for more certainty to emerge, but most of the time it doesn't. This makes me afraid to make a choice and I often decide to take the easy way out and keep the status quo. I've regretted it everytime... a lot of times, the only way to improve is to take a risk. Life is about probabilities, and the only thing you can do is to improve your odds.

And..... that's it for now. All of us face decisions everyday, some of us never learn how to do it. Hopefully this helps.

Buying Advice

2008-03-01T11:23:00.000-08:00

"It is far more profitable to sell advice, than to take it."

- Steve Forbes, publisher of Forbes magazine

Being a cynic, and a consultant, I can really see the humor of that quote! =)

Idiomatic Python Resources

2008-02-03T00:03:00.000-08:00

I've finally found a good resource on how to write idiomatic Python/Pythonic code. It's a presentation from Pycon 2007.
Check out the references at the end of the page.

These are really useful because coming from a Java background, I don't want to keep writing in a Java or just functional style. (I went crazy with this after reading the Little Schemer and the Seasoned Schemer books.)
In Python, you can do both, which seems to make Pythonistas more pragmatic (uses both styles) than the purely functional programmers, while not being tied to just nouns like Java.

Reality Is...

2007-12-23T19:20:00.000-08:00

"Reality is like fine wine, it will not appeal to children."

- Don Miller

A little snobby but true... great quote.

Some Geek Humor

2007-10-19T20:44:00.000-07:00

Programming language humor

A Prime Question

2007-10-17T20:04:00.001-07:00

What's the fastest/most efficient way to get the common prime factors of 2 numbers?

I thought about getting the gcd (greatest common divisor) using Euclid's algorithm, then find the prime factors for the gcd

Not sure if there is a more efficient method.

Guess gotta ask some more math-y people...

Python is very succinct

2007-10-16T06:40:00.000-07:00

So I decided to pick up Python the other day, just to see what it was all about.
Normally when I learn a new language, I read about the basics and dive right into it by using it to write a program.

For Python, I decided to write a solution for one of the Facebook puzzles, Prime Bits.

It was great. I finished the problem quicker than I thought I would, (1 day) considering I'm just learning the language.
And it was under a 100 lines of code as well. (not counting unittests of course, which are more than 100 lines by themselves)

I've been thinking about why that is, and I think it's because of all the built-in list operations at least in this program's case.
In Java, the equivalent of a list comprehension in Python might take 10 more lines of code.

I wonder if it's the same for Ruby...

Two Boys From Manila

2007-08-28T19:51:00.000-07:00

Two boys growing up,
Two minutes from each other.

One eats three meals a day, sometimes more.
The other is lucky to eat anything at all.

One sleeps on a bed, in an air-conditioned room.
The other on cardboard, under the stars and the moon.

One has a car to take him places,
The other wishes for shoes to help his paces.

One has three maids, for his every want and need.
The other has eight siblings to clothe and to feed.

One has parents, who would give him the world,
Yet proud and spoiled, he rebels at their every word.
The other's father, gambles everyday,
Money for food, he takes to play.

One gets money, and doesn't appreciate,
"I want more than this", he shouts irate.
The other in dumpsters, he scraps and collects.
To feed his family with the one's rejects.

Bent down in hunger, on his knees,
The other sees one on the street.
"Palimos po", the other pleads.
But one only spits on the concrete.

Forgive me Father, for I was the one.
Who had been given everything by your Son.
But selfish and blind, failed to see the other,
Who in need, was my brother.

written Jan '06

Learning to Find the Right Solution

2007-08-26T15:41:00.000-07:00

All through my years in school, from prep school in Manila to a Comp Sci degree at UW-Madison, I was taught and told to memorize many different formulas and solutions.

Math, science, economics, programming, etc. all followed a similar trend. We were taught formulas and theories in lecture, and then were expected to apply these new learnings to the problems that we faced in projects and tests.

This was supposed to serve as training for a job in the working world.

I think there is something fundamentally wrong with this pattern, specifically in the lecture then test part. As a student, when you are facing a question on a test, you know that the expected answer can be constrained to something that you just learned.
So in order to do well on tests, most students end up just memorizing the text and regurgitating what was commited to memory.

The solutions and formulas end up becoming useless detail, a hoop that students have to jump through in order to get the piece of paper that says they are fit and knowledgable enough to work.

Because the problems/requirements are driven by the solution, and not the other way around, we know that we only need to blindly apply the solution to whatever problem we need to solve in the test/project.

This pattern of learning does not really encourage creativity and real thinking. Worse, it does not teach students on how to solve problems and how to deal with ambiguity, which is a very essential skill in the working world.

Students also do not get the answer to their frequently asked question, "Why do we need to learn this?"

As a result, the majority of graduates from school are only fit to work in the old factory model, where you are told exactly what to do and to follow the solution blindly.

One of the worse things that come out as a result is that if the solution doesn't solve the problem completely, the problem gets reframed so that it fits the solution! I've observed that this is particularly true of a lot of MBAs. (not all of them, but a lot, I think this is a result of how much they had to pay to learn those solutions =))

How can we change this? After all, there is a lot of value in knowing different solutions and formulas, and knowing them are usually requirements to getting jobs and delivering value.

Well, from personal experience, I've found that I learn best when I am faced with a hard problem that I want to solve.
I learn to research, and to know the pros and cons of solutions, and how to deal with one-offs, and to mix and match when appropriate.

Still, the old mentality crops up sometimes, especially when I just learn about a new way of doing things. I find myself trying to apply what I just learned to all the problems that I face right after.

"If you only have a hammer, you tend to see every problem as a nail." -Abraham Maslow.

There is a definite disjoint in what school teaches us and what we have to face in the real world. School gets half of it right, you do need to know the solutions and formulas and ways of doing things.
But we also need to know when it is appropriate to apply what we learned, and the current model of learning just does not cut it, when evaluated in this regard.

Performance Testing Java Code

2007-08-25T17:43:00.000-07:00

I'm currently reading the book, Java Concurrency in Practice, and there is a great section in there about performance testing programs written in Java. (If you haven't read the book and you are a programmer of any language having to deal with concurrency, I highly recommend it!)

There are some "gotchas" in there that I don't think I would've thought of that might produce variances in your performance testing numbers: (some of these is java specific, but most of them can be generalized)

1.) Remember to tune your app with all production configurations for your VM set. If you don't and don't use the -server flag, you wil most likely be surprised at the bottlenecks and problems when your app is on production and under heavy use, since the bottlenecks that you encountered might be done away by optimizations done by the -server setting.
Worse, the code that you wrote to get around the bottleneck during testing might become the bottleneck.

2.) Dynamic Compilation - Java has 2 ways of running your code, it will either interpret the bytecode, or after a while, the dynamic compiler will convert the bytecode to machine code.
I don't think you can really know when this will kick in, but this could cause a lot of variance in your performance testing results because machine code will run faster than bytecode. (the chart in the book shows more than half gain) Also, if the compiler decides to do its thing when your test is running, that will also use up resources, etc. which could also skew the numbers.
The solution the book proposes is to "warm up" the code a few times to avoid this.

3.) Usage patterns - Trying to simulate lot of threads hitting a part of your code is not enough to be an accurate indicator of bottlenecks. There is also the element of how much computation the threads are crunching, and the size of the payloads/data.
For example, for simplified purposes, you might decide to use test data or shorten the tasks that are being executed by the threads when you are performance testing, and try to make up for it by the volume of concurrent tasks that you throw at it.
Thing is, this could be a different usage pattern, a lot of short-running tasks versus fewer long-running computations call for different optimizations.
With a lot of short-running tasks, there could be a lot of contention for shared resources, so trying to solve that bottleneck would be a big benefit.
Fewer long-running tasks do not have this problem. Instead there is more a liveness concern with the application; how do you know whether the thread is running slow or blocking/waiting for resources? How do you know it's not deadlocked?, etc.
Optimizing for one usage pattern will most likely not be applicable for another, that's why it's important to know this kind of background information before starting to optimize.

4.) Static/Unchanging Test Data - if your test always pushes the same input data through to be computed/operated on by your code, the compiler might compute it once and store/cache or replace the code that you have with a precomputed result. This will cause in long-running first passes through the code, and really fast execution after awhile.
This is something to watch out for especially if the data your code is computing comes from users, make sure you use dynamic data.

There are many other good points in the book, some parts of it require re-reading a couple of times, but I think it's because of the complexity of the subject, not the writing/language. (it is very readable, plus, there are a lot of good code samples) I think it is one of those books where you get something different out of it everytime you read it, since the pages are so information-packed that you can only focus on certain things the first pass through.

I'm finally getting close to the end of it, and right now, my brain is close to bursting, so I guess it's time to empty it again with a few beers. Ciao!

The Thing About Higher-Level Programming Languages

2007-08-23T19:58:00.000-07:00

I just read Paul Graham's "Holding a Program in One's Head". The premise is pretty much, the programmer becomes his utmost best and gets into the "flow" when he can think of the whole program at once.

In my experience, this is very true, when I program for a while, I get to a certain point where my brain just flows and I can clearly see what I have to do to accomplish whatever the program's purpose was.

There are many things that could interrupt this flow. One of them is not being able to transcribe the stuff in my brain to my IDE/editor. For those of you not familiar with programming software, this is analogous to someone trying to write a speech, but the transcriber is a slow typer and can't keep up with the flood of words, as a result, the speechwriter's flow is interrupted and is forced to repeat whatever he just said, causing him to lose his train of thought.

Programming is the same way. You want to know your tools so well that they become an extension of your brain. The problem with lower-level languages is kind of like having someone not fluent in your language as your transcriber; you have to spell out more complicated words for the transcriber to understand.

Using a good programming IDE/editor helps a lot with the tediousness. With a few keystrokes, you can automate away a lot of it. The problem is, sometimes it is still not enough. You will always encounter instances where you have to stop and spell out explicitly what you want the program to do. (fight with type system, deal with syntax idiosyncracy, etc.)

That, I think, sums up why programming should mostly be done in higher-level languages. It interrupts your flow less often, you come up with elegant and simple solutions that are straight to the point because you can see more clearly. It increases productivity and creativity by a wide margin, because it abstracts away the low-level details that are not relevant. (except maybe for performance) If performance starts becoming a bottleneck, then you can always substitute the particular piece of code with a lower-level language.

Learning to Learn the Right Things

2007-08-19T16:03:00.000-07:00

Everyday as we go about our business, we encounter many new and unfamiliar things and situations. This is good, since unfamiliarity often leads to a new experience and learning.

Too much of this however, can become a bad thing; our minds become saturated, or we might struggle and lose confidence and get overloaded with details, we end up not learning much of anything at all.

It's probably the same with other fields, but in software development, there is always something new to learn. A new language, a new framework, a new library, and sometimes, a totally new paradigm. A programmer who actively learns his craft for 10 years could have barely just scratched the surface. That's one of the reasons that led to the traditional IT shop becoming so siloed and specialized. (which is a bad thing)

Given the sheer amount of details, it is important to step back and set goals, and realize what you want to learn and do. If you just follow what you need for your job, you end up learning only about a particular vendor's product or your company's home-brewed solution. When that vendor (or your company) goes belly-up or if you get laid-off (or bangalored/shanghaied) then all you can do is get on your knees and start praying because the knowledge that you have accumulated becomes useless detail. (unless of course, you work for a company like Amazon, Google, or MS, then your company's infrastructure might actually be more interesting than what's out there)

Since we only have so much capacity in our brains, we should be more selective of what we learn. The best things to learn are the standardized things so your knowledge can be transferred. Or better yet, learn the principles of creation and design behind a particular implementation or a product.

When I first started in the software industry, I thought given time, I would become proficient in the majority of the areas, but as I dig deeper, I am finally starting to realize how much I don't know yet and how much there is to learn.

I've decided to scale back and be realistic, and focus on just a few fields. (compilers, networking/distributed systems) Otherwise, I want to learn too many things and end up just spinning my wheels.

My Learning Goals as a Software Developer

2007-08-13T07:04:00.000-07:00

1.) The compiler is not a black box.
(ANTLR, compiler theory, Programming Language Pragmatics

2.) Multicore processors
(multi-threaded programming, Erlang)

3.) Networking/Distributed Systems
(communication protocols, scaling gracefully, algorithms, web servers)

4.) Computer Science/Math
(Modern Heuristics, Graph Theory, Computational Geometry, Combinatorial Math)

Thoughts After First Project (Written in May '07)

2007-08-06T19:34:00.000-07:00

It's been really hectic the past few weeks, and next week will be a grind, but before dumping my head in and jumping back into the fray, I wanted to take a little time and make sure that I catalog a few things that I have learned over the past couple of months. After all, the whole point of doing projects, (other than business value for the client and revenue for us of course) is to grow and make sure that pain points experienced now will hopefully be prevented in the future.

Before I start, I should give a little background and general thoughts. (without getting into legal implications about confidentiality and stuff =))

The project is for a very well-known client in Chicago. Work is being done co-sourced, (with client devs and delivery/pm managers) and had a fairly aggressive timeline. It's close but I think we are going to make it.
Overall, it's an awesome first project to be in. A lot of senior devs, great team "chemistry", open source tools used, interesting work, and a lot of learning for me.

The following list are things that I have learned and observed during the course of the project.

1.) Technical debt is sometimes necessary

Every project accumulates technical debt. For this project, because of wanting to stay on track and maintain/increase velocity, decisions sometimes had to be made to move forward and cut a few corners.
Whether it's less than optimal solutions or not taking more time to investigate different options, the decision that will increase the best business value might be to drop it and move forward.
Just make sure that you, the rest of the team, and the client knows the pros and cons of the decisions.

2.) Don't try to be a hero

I thought about this more after reading DHH's recent blog post. (Ruby on Rails creator)

There were quite a few times over the course of the project where senior devs would push back on riskier features that I thought might've been implementable given the amount of time. While everything might've come out OK, it should be the client's decision to decide whether a certain feature is core or a nice-to-have, given the risk and the potential cost.

I can see that some of the pushing back is paying off now, and we are able to put in certain nice-to-have's as well once the core has been done.

3.) Falling ice, beware!

Projects will always have risk. And just as signs on the sidewalk are important to make us aware of potential hazards, risks on a project should also be identified and flagged.

So who should be flagging risks? Well, technically you could say the project manager, since he is the logical choice to coordinate a solution or present it to clients. But the right answer is everyone on the team. On this particular project, a lot of the risks were flagged by developers and BAs, and a lot of times, they were one step ahead of the risks with potential/implemented solutions for them already.

4.) "Walk softly, and carry a big gun." - dawn of war

Performance matters. And since it does, it should be addressed earlier.
Note that I am not talking about premature optimization. But once certain areas of the application are done, it seems to be a good idea to start measuring and getting performance numbers to have a rough inkling about where the bottleneck might be, instead of waiting for everything to be done and the cement to set before doing it. Once you start optimizing, make sure that performance does not degrade by putting some performance tests in your integration/deployment/daily build.

5.) Put Ajax/JS effects last

If there was anything painful about this project, it's the "view" part of the application. The client specifically wanted Ajax on certain parts of the site, so developers (including me) implementing the features would put the Ajax right away as the de facto choice.

Looking back, this was a bad idea for a couple of reasons, this made Ajax implementation and the user experience inconsistent across different areas, and the application does not degrade gracefully.
I recently finished the book Bulletproof Ajax , and the author Jeremy Keith discusses a good way to do it.

First make sure the site works in normal HTML with standard get and post reloads without Javascript, then hijack user behavior with Javascript/Ajax. This way, the Javascript is unobtrusive and the application has a default before to fall back on if Javascript is not working/enabled.

This also has the advantage of being easier to combine/reuse Javascript across multiple pages.

6.) Don't leave the view for the end

We got our "display" stuff (HTML/CSS) from a third party design firm for this project. There was a lot of pain experienced integrating their work with ours, parts of some pages broke, etc. because of different CSS class names, HTML element id's, pure HTML vs JSP, etc.
Looking back, I think integration between design firms and dev teams might be better if HTML (just the HTML, CSS not necessary) was delivered up front before the dev team started so that the id's, classes, and names of HTML elements will not have to change much.

7.) Be consistent, enforce agreements

I saw the importance of consistency in this project. While things are not perfect, I can say that I can reasonably expect a certain way of doing things when looking at Java code, like naming conventions, etc.
The same cannot be said for the Javascript. But I've talked enough about that in #5.

8.) Minimize risks from external dependencies

We had to integrate with certain external services for our application.
It's important to

a.) have a separate module deal with the outside source and have the rest of your application deal with that one module.

b.) expect the external service to be done and plan accordingly. Also to stub things out for testing!

9.) Know (and learn) good tools

Tools like selenium, cruise control, (a given on Thoughtworks projects) checkstyle, cpd (copy paste detector) and firebug were very useful for the project. I found firebug to especially be a lifesaver for doing Javascript and browser work. I've used it before this project, just not to the extent that I am using it now. Having the ability to inspect an HTML DOM element, having a debugger for Javascript, and changing CSS and HTML directly in the browser have been great time savers.

10.) Refactor beyond the scope of the "story"

Stories is an agile development term for feature. Usually developers get assigned to do stories.

There are times were a bunch of stories will be in play that are all related and are similar to each other, so different dev pairs will be churning out pretty similar looking code.

Once the stories are done, we usually move on to do other stories. But I think there should be some time spent refactoring and removing duplication in the Java code, as well as the view area. (jsp's, javascript, html, etc.)

This should be done as early as possible because once people start building more code on top of the duplication, it gets really really hard to refactor it.

So for some later requirement being played, (for example: guest users need to be prompted to login when clicking a save button) instead of changing Javascript in one place, you will have to go page by page, since changing the jsp's at this time will mean editing the css, the java code, the xml files, etc.

I guess that's why small teams are preferable, it's easier to have good cohesion and less duplication in the code.

The More Things Change...

2007-08-06T18:04:00.000-07:00

the more they feel the same.

Sure, everytime something new comes along, there is inevitably a feeling of shininess, a sense of wonder and excitement at something different.
But lately, it seems like new things are just old things rehashed, there is really nothing new under the sun.

New things become old, and as a person, my reaction to the different happenings in my life fall into a pattern. Is this what it means to grow up?

I miss the feeling of school, where your future looks unbound, and your current place is just a step to what seems like a great journey.

Lately, work has given my outlook a feeling of finality. Like, "this is pretty much how it's going to be, for the forseeable future."

I guess there I'm feeling a sense of disillusionment at the failure of new things to deliver on their promise.

I hope this is not true, that there are things out there that will get me up and motivated again, I guess we'll see. I'm not ready to give up yet. In the meantime, I should keep looking.