Quality assurance
Filed under: technologies — andraz @ 17:08 24. Nov, 2007
How do you test a complex system that is trying to mimic being smart?
You want to automate testing, so you have your quality meter available for every little change you make. While having unit tests helps catching classical programming regressions, the major part of the challenge is having ’smart part’ under control. Unfortunately the only way to tell if the system is doing a good job or not is to have human check the results. The trick is that if you could automate testing in general, you would already have solved the hardest problem.So what you basically do is generate a set of evaluation data, manually. And have a system that does something like unit tests, but instead of giving you fail/pass results, you get statistics. Now you would think that the problem is solved, but that’s far away from truth.
There is changing of the dataset - when you have new content in the system, you get completely new related stories and you have to go back and have a human judge them. There is expansion of the evaluation data - as you add new tests you generally can’t send them through previous versions of your algorithms, since that would be prohibitely expansive. And there is statistics that hardly gives you overview over what exactly your changes caused, just few final numbers. And then there is the problem of pipelining the processing. Even if you improve the first stage, end results might be worse, since you’ve already adapted the second stage to previous first one. So you need to actually evaluate each part of the system in isolation and then together.
At the end you actually find out that you spend disproportional amount of time evaluating even the smallest changes. So you are in danger to just skip that evaluation which naturally you shouldn’t.
Ok, so much for today, now I think the evaluation run has just ended and I should be checking the results, again.
Related articles by Zemanta:
How do you spell Zemanta?
Filed under: zemanta — andraz @ 1:35 21. Nov, 2007
Some internal jokes that really aren’t:
Q: How do you know you are stretching your tools to the limit?
A: You find reproducable segfault inside Perl. In its regex code.
Q: How do you know you surely have enough RAM?
A: “free” command causes a line break in a terminal. Everything below is doubtful.
Q: How do you know your server specs are not balanced?
A: You have more memory than disk space.
Someone testing one of the most hyped technologies in the Valley: Who are “The other guys” and where can I use them?
Q: What does a developer say to CEO at 8am in the morning?
A: Goodnight.
Q: How many developers does it take to change a lightbulb?
A: Wait, wait, I am already looking it up in Wikipedia!
Q: How do you properly spell Zemanta?
A: Did you mean Zemanova? Google cannot be wrong!
Related articles by Zemanta:
Optimization
Filed under: technologies — andraz @ 16:22 17. Nov, 2007
It is interesting to see how during development we are moving back and forth between ‘better job of suggesting’ to ‘faster suggesting’. A cycle seems to last around two weeks. Currently we are in optimization part of it. Just yesterday Tomaz had a breakthrough getting one component that took 10 seconds per request (the biggest time spender) to just under one second.
The interesting thing is that we were on the verge of falling into premature optimization trap. We’ve moved the most time-critical part of the code from Python to C, which at first didn’t seem to help much. And we’ve almost went for moving even more code to C, but in a moment of doubt I’ve fired up amazing oprofile profiler.
Profiling showed that we spend most of the time in an inner loop measuring the length of strings by wcslen() which is called from the Boost library that we use. Now we are back on track with our performance.